Last month we blogged about the work that New Growth did for Skills for Chicagoland’s Future (Skills). Skills is a public-private partnership working to find jobs for unemployed and underemployed job seekers in the Chicago area. They have adopted a “demand driven” approach – working with businesses to determine their hiring needs and then finding qualified individuals to fill those jobs. Skills wanted to measure how well their model was working in the real world, so they hired New Growth to evaluate the project.
The research goal was to determine if individuals placed by Skills had better employment outcomes (like earnings) than comparable individuals who were not placed by Skills. Our design had to balance this goal against the availability of data. We started with Skills’ existing database where we found 1,197 people who were placed by Skills during the 6-quarter study period (Q1 of 2014 through Q2 of 2015). Because earnings was the key outcome, we worked with the Illinois Department of Employment Security (IDES), which is the state agency responsible for tracking quarterly earnings records. IDES was able to find 600 of these people in their database, and these became the study participants.
Then, the key was to try to identify comparison individuals who were as similar as possible to the participants. We wanted comparisons to be matched on demographics and prior earnings (earnings from the period before the participant was placed by Skills). Unfortunately, the earnings records in IDES do not have demographic information. To get demographics, we had to utilize the Unemployment Insurance (UI) data at IDES. Of course, not every participant appeared in the UI system. Ultimately, we were able to get age, race, gender, highest level of education, and zip code location for 306 of the 600 participants.
- For each participant in the UI system, we identified 14 other individuals in the system who had the same demographics. Then we acquired the earnings data for all of these potential comparison group members. We chose those who would land in the final comparison group based on matching on prior earnings, keeping up to the best 4 matches per participant.
- For each participant not in the UI system, we identified up to the best 4 matches based on prior earnings alone. Therefore, about half of the participants were matched to comparison individuals who had the same demographics and same prior earnings, whereas the other half of the participants were matched based just on prior earnings.
At the end of this process, we checked the comparability of the participants with the comparison group and found the matching to be excellent. Of course, there is still the possibility of un-measured variables differing between the two groups – no match is perfect – but we feel we came as close as possible given the constraints of time, money, and availability.
After constructing the study groups, we proceeded to compare them on our outcomes of interest, such as earnings after placement with Skills. We found that in the 2 years post placement, the participants earned $6,120 per year more than the individuals in the comparison group, on average. More results can be found in our prior blog post and in the final evaluation report.
