Hawaii Food Stamp Employment and Training/JOBS Conformance Demonstration:

Impact Evaluation Final Report


 
 
 
 

Deanna T. Schexnayder

Jerome A. Olson
June 1997

 

Center for the Study of Human Resources

Lyndon B. Johnson School of Public Affairs
The University of Texas at Austin

This report was prepared with funds provided through Interagency Contract No. DHS-96-SSSSD-4925 from the Hawaii Department of Human Services to the Center for the Study of Human Resources at the University of Texas at Austin. The views expressed here are those of the authors and do not represent the positions of the agency or of The University.

 

Executive Summary


The Center for the Study of Human Resources (CHR) of the LBJ School of Public Affairs at The University of Texas at Austin conducted an evaluation of the Hawaii Food Stamp Employment and Training (FSE&T)/Job Opportunities and Basic Skills (JOBS) Program Conformance Demonstration under contract to the Hawaii Department of Human Services (HDHS). The demonstration and the evaluation were sponsored by the Food and Nutrition Service of U.S. Department of Agriculture. The evaluation was designed to assess the impact of the demonstration on participation patterns, service delivery, client outcomes, and costs. The evaluation encompassed the period from January 1992 to June 1995. This report presents the results from the impact component of the evaluation.
 
Overview of the Demonstration. Hawaii’s Food Stamp Employment and Training/JOBS Conformance Demonstration was in operation on the island of Oahu between November 1993 and September 1996. The Oahu program was given the name PRIDE (Positive Response in Developing Employment), and its goals were to:
• Maximize the employability of Food Stamp recipients and reduce Food Stamp dependency through improved consistency and coordination between the JOBS and FSE&T programs (primary goal).
• Coordinate across programs to reduce administrative and service costs.
• Enhance FSE&T services through the use of a new case management system, the addition of new barrier removal and family social support services, and an expanded emphasis on educational activities.
• Decrease program errors due to reduced complexity of and conflicts between program regulations for the FSE&T and JOBS programs.
The demonstration also was intended to increase overall fairness by offering all public assistance recipients “the same realistic and meaningful opportunities to achieve self-sufficiency.” The key features of PRIDE’s strategy included the introduction of a comprehensive case-management approach designed to link families and individuals to needed support services for the removal of psycho-social barriers to employment, followed by the provision of needed employment preparation training, basic education, and vocational training services.
Impact Analysis Research Questions. The impact analysis was designed to address the following research questions about client participation patterns, services and employment outcomes:

Participation patterns

• Were there any significant changes in the number of Food Stamp recipients who were mandatory work registrants?
 
• How did the demonstration affect the number of mandatory work registrants who would have been exempt had exemption criteria not been altered for the demonstration?
• How did the demonstration affect the number of mandatory work registrants who responded to the initial appointment invitation?
• Did the demonstration affect the number of mandatory work registrants who were issued notice of adverse action for failure to participate in required services?
• What was the number of individuals sanctioned 1st, 2nd, and 3rd times?
• How did the demonstration affect the number of mandatory work registrants who actually participated in FSE&T services?
• Did the number of Food Stamp recipients who are voluntary participants increase as a result of the demonstration?

Services

• What types of services were selected by the participants?
 
• Did the demonstration increase participation in different services?
• Did the demonstration change the duration and intensity of E&T services received?
• Did the demonstration change/increase the number or type of supportive services received?
• Did the number of participants receiving services from other agencies through non-financial arrangements increase as a result of the demonstration?

Employment Outcomes

• Did the number/percent of participants who entered employment increase?
 
• Was the participant able to obtain employment in the area trained?
• Did the average wage rate at placement increase?
• Did employment retention rates increase?
Summary of Research Results.
Participation patterns. Many of the research questions concerning participation patterns could not be answered because the administrative data files needed to answer these questions were not archived. This data difficulty affected all questions that relied on the use of historical data on exemption status and mandatory and volunteer status. Attempts to retrieve the history of these variables through the manipulation and matching of a number of administrative files were unsuccessful.
Response to being called in to participate in an employment and training program dropped 25 percentage points during the PRIDE demonstration after adjusting for differences in demographic and economic differences between the demonstration and comparison sites. However, this decrease in response rates was not accompanied by a corresponding increase in the issuing of Notice of Adverse Actions (sanctions). Use of this tool was very rare in the baseline period and did not increase in the demonstration period. The decline in response to call-in possibly resulted from the longer length of time that persons in the PRIDE pool had to wait before being called in to participate in the program. Differences in the data sources used to calculate this statistic also may have accounted for some of the response rate decline.
Actual participation in a component among those who had been called-in also dropped during the PRIDE demonstration. After adjusting for demographic and economic differences, participation rates declined by 58 percentage points. Differences in the manner in which ‘participation’ was defined between the two data sources may have accounted for some of this decline.
Services. Participation in individual components changed significantly as a result of the PRIDE demonstration. Far more emphasis was placed on an upfront assessment, with a total of 779 individuals served in this component for an average duration of 1.69 months. Assessment was not delineated as a separate component in the regular FSE&T program.
Among the other components, a major decline in the average monthly participation occurred in Individual Job Search (-40 percent), accompanied by a small decline in Basic Education (-5 percent). These reductions in participation were balanced by increases in other activities, with positive adjusted net effects of 17 percent and 25 percent in Vocational Training and Work Experience, respectively. These changes correspond to the intended design of the PRIDE demonstration to increase participation in components that improved participants’ competitiveness in the labor market.

Some major shifts in the average number of months persons were enrolled in individual components also occurred at a result of the PRIDE demonstration. Regression-adjusted results indicate that the monthly duration of job search skills increased by 1.2 months, probably the result of adding the Ho’ala curriculum to this component. Surprisingly, the length of time in basic education declined significantly during the demonstration, with a net reduction of 1.6 months. This seems contrary to the expectation that more emphasis would be placed on education and training in the PRIDE demonstration. Other changes in duration were not statistically significant.

The JOBS (and PRIDE) program instituted a major change in its participation requirements shortly before the end of the study period that required persons to work or look for work while also participating in education or training. This change in the participation requirements strongly affected the percent of persons still enrolled in selected components at the end of each time period. The regression-adjusted rates of persons completing independent job search declined by 36 percentage points while the rates of persons no longer enrolled in vocational training increased by 40 percentage points. Obviously, this shift in program rules affected individuals’ participation in a major way.

While far more individuals enrolled in post-secondary education as a result of the PRIDE demonstration, these enrollments did not result in a significant increase in the share of persons receiving post-secondary degrees or certificates by June of 1995. This result occurred both because of the short time in which to observe this outcome and the change in program rules cited above.

Employment outcomes. Effects of the demonstration were measured for employment rates immediately following program participation, quarterly earnings immediately following job placement, and job retention rates six months following employment. Almost all of the differences in both unadjusted and adjusted net effects were statistically insignificant. Employment entry rates ranged from 42-47 percent for persons no longer enrolled in the PRIDE or regular FSE&T programs. Quarterly earnings averaged $1,700 - $1,900 in the quarter immediately following placement, or $566 - $633 per month.

The only measure which showed a significant net effect from the demonstration was employment retention. Approximately 50-59 percent of persons who obtained jobs were still employed six months later. Regression-adjusted net employment retention rates dropped by 12 percentage points during the PRIDE demonstration, a statistically signficant decline. However, incomplete demographic data resulted in some observations being dropped from this regression, meaning that this statistic should not be taken too seriously.

The insignificant differences in most of the employment outcomes seem to indicate that the PRIDE demonstration has not achieved its objective of improving employment outcomes by offering more intensive program treatments. The short postprogram time for which outcomes were observed and the change in program rules that caused many individuals to drop out of their postsecondary components prior to completion may have adversely affected these results.

Conclusions. While the impact analysis of the PRIDE demonstration was able to document fairly large shifts in the share of activities in which the participants engaged, the analysis of many of the other outcomes is clouded by data issues. One clear message from this analysis is that the handling of administrative data by HDHS needs to be improved. Historical data needs to be archived on a regular basis —usually monthly or quarterly — so that key variables needed for a longitudinal analysis are not overwritten. Prior to embarking on another research demonstration for which program operators are interested in outcomes, HDHS should thoroughly review its data collection and archiving procedures. Working with an evaluator at the beginning of such an endeavor also would enable the agency to ascertain that data are being maintained in a manner that will allow the research questions to be answered.

Even with the data difficulties, the PRIDE demonstration does appear to have resulted in major shifts in the types of components in which persons enrolled. Unfortunately, testing of the major premise of this demonstration — that investing in longer-term treatments would improve participants’ employability — was short-circuited when program rules in the PRIDE program were changed to require that education and training participants also work or participate in job search.