ALY6015_FinalProject_Patel

FINAL REPORT ALY 6015 Intermediate Analytics Hardh Patel Date: December 4 th , 2023 Instructor: Sergiy Shevchenko

INTRODUCTION The U.S. Census Bureau carries out a monthly inquiry known as the Current Population Survey (CPS), gathering data on a plethora of demographic and economic characteristics that influence the American populace. This trove of information from the CPS is instrumental in shedding light on the nation's social and economic dynamics, serving as an indispensable tool for both decision-makers and scholars. This preliminary assessment aims to lay out our early observations and offer a detailed examination of the CPS data collection. Our objective is to deepen our comprehension of the prevailing conditions affecting individuals in the U.S. by exploring a variety of demographic and economic elements, including but not limited to age, sex, educational background, earnings, and employment circumstances. In addition, our examination will delve into various population cohorts to pinpoint any imbalances and monitor evolution over periods. Our present scrutiny is anchored in the data gleaned from the CPS for November 2022, encapsulating details on upwards of 123,000 individuals. This dataset encapsulates an array of demographic and economic attributes, spanning age, sex, ethnicity, educational achievements, financial status, professional classifications, and sectoral engagement. To dissect the data, we employed exploratory methods, scrutinizing the likelihood distributions and condensed metrics for the assorted variables. To further elucidate our findings, we crafted graphical representations such as histograms and point diagrams, which serve to underscore recurring patterns and trajectories within the dataset. EXPLORATORY DATA ANALYSIS DESCRIPTION Employing descriptive statistical methods has underscored the importance of refining the data and deepening our understanding of the involved metrics. Within this dataset, we found 123,009 entries across 388 distinct variables. Our initial step in extracting pertinent conclusions and insights was to purify the dataset. This process entailed the elimination of incomplete entries and the enhancement of the variables at hand. For instance, employment status was subdivided into several groups, including those who are retired, employed, or unable to work. Furthermore, we distinguished variables relating to geographic region and cultural background to facilitate a more granular examination and utilized summary tables in our exploratory data analysis. The income range

female participants compared to male participants. It was also observed that urban areas had a higher representation in the dataset compared to rural areas. Moreover, the urban status of certain records remained unspecified. SUBSET 2: Employment Status and Net Income Table 2: Descriptive Statistics of Employment Status and Family Income Employment Status Min q1 Median Mean q3 Max Disabled 2,009 13,873.0 29,049.0 45,083.62 57,154.00 299,936 Employed - Absent 2,025 43,870.0 83,466.0 102,292.02 139,974.50 298,709 Employed - At Work 2,032 52,739.0 89,868.0 110,859.50 148,164.00 299,974 Other 2,009 38,080.5 73,363.0 95,847.31 134,490.00 299,976 Retired 2,017 28,881.0 51,678.5 70,097.58 88,759.75 299,852 Unemployed - Looking 2,003 24,120.0 49,802.0 71,111.90 92,639.50 299,961 Unemployed - On Layoff 3,311 28,139.0 50,643.0 68,954.27 94,178.00 292,466 Table 2 presents the aggregate income levels in relation to the employment status of household members. From this table, it's clear that those who are employed report the highest average income in comparison to other employment categories. On the other hand, individuals who are disabled report the lowest average income. However, the data also indicates that those with disabilities, while having the lowest average income, still receive the highest incomes within their category when compared to peers in other job statuses.

SUBSET 3: Education Status and Net Income Table 3: Descriptive Statistics on Education Status and Family Income Education Staus min q1 median mean q3 max College 2,002 34,008.25 59,836.5 78,655.07 99,648.5 299,963 Doctorate 2,017 45,689.00 87,418.0 108,170.59 151,540.5 299,991 Elementary Education 2,043 19,841.00 37,425.0 55,823.03 68,732.0 299,916 Graduation 2,044 63,904.25 107,603.5 125,568.09 179,661.5 299,994 High School 2,014 26,750.00 51,699.0 77,136.54 101,440.0 299,889 Table 3 delineates the correlation between educational achievements and the net earnings of the population. It is discernible from the data that individuals holding master's, professional, and bachelor's degrees are at the higher end of the earnings spectrum compared to those with other levels of education. This pattern of income distribution across different educational qualifications of family members provides a predictive framework for household net income. SUBSET 4: Occupation and Total working hours Table 4: Descriptive Statistics on Occupation and number of total working hours Occupation q1 median mean q3 max Construction and Extraction 40.00 40 34.80478 40 134 Farming, Fishing and Forestry 6.00 40 28.84741 40 99 Installation, Maintenance and Repair 40.00 40 38.30394 40 85 Management and Business 40.00 40 38.39611 45 120 Office and Administrative Job 35.00 40 34.42317 40 100 Production Occupation 40.00 40 37.29208 40 99

Research Questions 1. How are individuals dispersed across the various regions of the United States according on their gender and metropolitan status? 2. How does employment status affect net family income in household of US? 3. What are the good predictor variables for predicting net family income? 4. What is the relationship between net family income and its predictor variables? ANALYSIS Following an in-depth analysis, we gained insights into the interconnections between variables and their impact on other aspects within the dataset. The subsequent visuals provide a deeper interpretation of these relationships.: Figure 1: Visualization on average income based on gender.  The study of the gender-based average income in the United States census statistics for November 2022 shows that men dominate the income landscape.  Males made an average of $98000 while females made $90000.

 The gender pay gap may indicate that there is still substantial gender-based wage disparities in the employment. Figure 2: Visualization on relation of education level with employment status  Based on an analysis of census data, the graph above shows that education level seems to have a substantial impact on job position. Compared to other degree holders, diploma graduates have the best job rates. The most popular degree among people is a graduate diploma, followed by a doctoral degree.  The two job statuses with the highest prevalence in the community are employed-at- work and retired. People with better education levels, such as bachelor's degrees, graduate diplomas, and master's degrees, are more likely to have the employed-at- work classification.  According to the statistics, people with doctorates are mostly classified as having "Other" job statuses, which is unclear and calls for more research.  These results imply that diploma graduates may have more employment possibilities because of their better degrees of education.

Figure 4: Visualization on average income based on marital status and region.  The above graph shows how the typical income is distributed across the Midwest, Northeast, South, and West areas based on marital status. The plot supports the census data's finding that married people who can see their partner earn more on average than people with other marriage statuses.  Widowed people have the lowest incomes, which makes sense given that it takes more than one earner to lead the revenue charts. The dispersal of people across regions is almost similar, as evidenced by the data, which was evenly gathered from the four distinct regions, but married people with spouses present again have predominated in every region by outnumbering other classifications.

MODELS LINEAR REGRESSION To predict the total income of U.S. households, we employed a linear regression model. The dependent variable in our model was the household's net income (HEFAMINC), and we used a variety of independent variables to predict it: the highest level of education attained (PEEDUCA), primary occupation (PRMJOC1), racial identity (PTDTRACE), regional location (PEREG), status as an urban or rural resident (GTMETAT), and the total number of people in the household (HRNUMHOU), among Hypothesis : H0: Whether different six factors defined above have no impact on net income. H1: Whether different six factors defined above have impact on net income. Table 5: Linear Regression Results summary Estimate Std. Error t-value Pr(>|t|) (Intercept) 28254.47 3219.62 8.776 < 2e-16 Edu_LevelHigh School -4460.11 691.71 -6.448 1.14e-10 occupationSalesDepartment -877.14 3478.16 -0.252 0.800900 ethnicityAsian 29151.13 2067.001 4.103 < 2e-16 ethnicityBlack 1660.39 1988.39 0.835 0.403695 ethnicityHawaiian 9934.45 3497.88 2.840 0.004510 ethnicityWhite 20524.79 1880.35 10.915 < 2e-16 regionWest 2280.80 630.19 3.619 0.000296 metro_statusNon-Metropolitian -17402.73 554.28 -31.397 < 2e-16 metro_statusNot Identified -16629.58 2092.51 -7.947 1.93e-15. prtage -261.78 12.76 -20.523 < 2e-16 genderMale 3995.88 437.24 9.139 < 2e-16 marital _ statusSeparated -2994.38 2738.78 -1.093 0.274252 marital_statusWidowed 18847.95 2505.96 7.521 5.47e-14 healthNot-Healthy 1926.00 1363.95 1.412 0.157930 R-squared/ Adjusted R-squared: 0.2094/ 0.209 F - statistic : 498.3 p - value : < 2.2e-16

 It can be clearly observed from above plot that since ridge regression does not eliminates the variable it considers all the predictors to predict the model fit. In this case, all 52 variables are considered in the model fit and results are shown below for some of the predictors. Table 5: Ridge Regression Results Summary

s0 (Intercept) 46738.41475 Edu_LevelHigher Education 30561.13414 OccupationSales Department 16989.40402 EthnicityAsian 21459.94807 EthnicityHawaiian 1360.60239 EthnicityWhite 13187.00637 RegionSouth -4416.05436 RegionWest 2416.21647 Metro_statusNon-Metropolitian -17198.62418 Age -264.44731 GenderMale 4305.75837 Marital_StatusWidowed 8135.14909 DisabilityNot-Disabled 12438.36708 Total_Paid_Employees 1132.38484 HealthNot-Healthy 2317.80400  The R-Squared Value for the model was 21.4% which means that using these predictors variation of 21.4% can be explained in predicting family income. Also, as there are total 388 variables the certain variation can be explained by other variables. LASSO REGRESSION We then proceeded to apply Lasso Regression to our dataset for a comparative analysis with the outcomes from linear and Ridge regression models. • The dataset was initially partitioned into training and testing segments in a 70:30 ratio. • For Lasso Regression, it was necessary to convert the dataset into a matrix format. Following this, we concentrated on identifying the optimal Lambda value, which is key in penalizing the coefficients to effectively reduce the slope. Lambda.Min 17.30025 Lambda.1se 1037.123

s0 (Intercept) 32977.860 Edu_LevelHigher Education -5386.74337 OccupationSales Department -54372.83241 EthnicityAsian 25929.54673 EthnicityHawaiian 5187.3698 EthnicityWhite 18013.32298 RegionSouth -4385.03818 RegionWest 2382.04990 Metro_statusNon-Metropolitian -14493.13258 Age -265.44731 GenderMale 4319.75837 Marital_StatusWidowed 15910.14909 DisabilityNot-Disabled 12385.36708 Total_Paid_Employees 1152.38484 HealthNot-Healthy 2150.80400 The R-Squared value for our model stands at 25.20%. This figure signifies that the predictors we have utilized can explain approximately 25.20% of the variation in predicting family income. However, it's essential to acknowledge that there are a total of 388 variables in our dataset. This suggests that there may be other variables outside of our model that can explain additional variations in family income.

MODEL COMPARISON Table 7: Model Comparison Table Linear Regression Model Ridge Regressio n Model Lasso Regression Model R- Square d 21.2% 21.4% 25.20% RMSE 66216.98 66009.61 63672.01 Upon conducting an analysis of the three modeling techniques employed, we computed the R-squared and RMSE (Root Mean Squared Error) values for each model to determine the most suitable approach for predicting family net income. Among these models, the Lasso Regression model emerged as the top performer, boasting the highest R-squared value at 25.20% and the lowest RMSE value of 63672.01. These results suggest that the Lasso Regression model is the optimal choice for predicting family net income.

REFERENCE  United States Census Data. Retrieved on Apr 02, 2023. https://www.census.gov/data/datasets/time-series/demo/cps/cps-basic.html  Top 50 visualizations with ggplot. Retrieved on Apr 02, 2023. http://r-statistics.co/Top50-Ggplot2-Visualizations-MasterList-R-Code.html  The data tho. Retrieved on Apr 02, 2023. How to write descriptive statistics. http://r- statistics.co/Top50-Ggplot2-Visualizations-MasterList-R-Code.html APPENDIX LIBRARY(TIDYVERSE) LIBRARY(PSYCH) LIBRARY(READR) LIBRARY(DPLYR) LIBRARY(RCOLORBREWER) LIBRARY(GGPLOT2) LIBRARY(GGALLY) LIBRARY(GGPUBR) LIBRARY(FASTDUMMIES) LIBRARY(GGIRAPHEXTRA) LIBRARY(RESHAPE2) LIBRARY(CORRPLOT) LIBRARY(CATOOLS) LIBRARY(CAR) LIBRARY(GGCORRPLOT) LIBRARY(MASS)

LIBRARY(VTABLE) LIBRARY(OFFICER) LIBRARY(FLEXTABLE) LIBRARY(PATCHWORK) LIBRARY(STARGAZER) LIBRARY(MASS) LIBRARY(LEAPS) LIBRARY(FURNITURE) LIBRARY(KNITR) LIBRARY(GLMNET) LIBRARY(METRICS) LIBRARY(SJPLOT) LIBRARY(SJMISC) LIBRARY(SJLABELLED) LIBRARY(TIDYR) OLD_CENSUS <- READ.CSV("/USERS/KUSHAGRABUBNA/DOWNLOADS/NOV22PUB.CSV") OLD_CENSUS <- OLD_CENSUS %>% DROP_NA() CENSUS_DATA <- READ.CSV("/USERS/KUSHAGRABUBNA/DOWNLOADS/NOV22PUB.CSV") CENSUS_DATA <- NA.OMIT(CENSUS_DATA)