Hawaii Food Stamp Employment and Training/JOBS
Impact Evaluation Final Report
Deanna T. Schexnayder
Jerome A. Olson
Center for the Study of Human Resources
Lyndon B. Johnson School of Public Affairs
This report was prepared with funds
provided through Interagency Contract No. DHS-96-SSSSD-4925 from the Hawaii
Department of Human Services to the Center for the Study of Human Resources
at the University of Texas at Austin. The views expressed here are those
of the authors and do not represent the positions of the agency or of The
The Center for the
Study of Human Resources (CHR) of the LBJ School of Public Affairs at The
University of Texas at Austin conducted an evaluation of the Hawaii Food
Stamp Employment and Training (FSE&T)/Job Opportunities and Basic Skills
(JOBS) Program Conformance Demonstration under contract to the Hawaii Department
of Human Services (HDHS). The demonstration and the evaluation were sponsored
by the Food and Nutrition Service of U.S. Department of Agriculture. The
evaluation was designed to assess the impact of the demonstration on participation
patterns, service delivery, client outcomes, and costs. The evaluation
encompassed the period from January 1992 to June 1995. This report presents
the results from the impact component of the evaluation.
Overview of the Demonstration. Hawaii’s Food Stamp Employment
and Training/JOBS Conformance Demonstration was in operation on the island
of Oahu between November 1993 and September 1996. The Oahu program was
given the name PRIDE (Positive Response in Developing Employment), and
its goals were to:
• Maximize the employability
of Food Stamp recipients and reduce Food Stamp dependency through improved
consistency and coordination between the JOBS and FSE&T programs (primary
• Coordinate across programs
to reduce administrative and service costs.
• Enhance FSE&T services
through the use of a new case management system, the addition of new barrier
removal and family social support services, and an expanded emphasis on
• Decrease program errors
due to reduced complexity of and conflicts between program regulations
for the FSE&T and JOBS programs.
also was intended to increase overall fairness by offering all public assistance
recipients “the same realistic and meaningful opportunities to achieve
self-sufficiency.” The key features of PRIDE’s strategy included the introduction
of a comprehensive case-management approach designed to link families and
individuals to needed support services for the removal of psycho-social
barriers to employment, followed by the provision of needed employment
preparation training, basic education, and vocational training services.
Research Questions. The impact analysis was designed to address the
following research questions about client participation patterns, services
and employment outcomes:
• Were there any significant
changes in the number of Food Stamp recipients who were mandatory work
• How did the demonstration affect the number of mandatory work registrants
who would have been exempt had exemption criteria not been altered for
• How did the demonstration
affect the number of mandatory work registrants who responded to the initial
• Did the demonstration
affect the number of mandatory work registrants who were issued notice
of adverse action for failure to participate in required services?
• What was the number
of individuals sanctioned 1st, 2nd, and 3rd times?
• How did the demonstration
affect the number of mandatory work registrants who actually participated
in FSE&T services?
• Did the number of Food
Stamp recipients who are voluntary participants increase as a result of
• What types of services
were selected by the participants?
• Did the demonstration increase participation in different services?
• Did the demonstration
change the duration and intensity of E&T services received?
• Did the demonstration
change/increase the number or type of supportive services received?
• Did the number of participants
receiving services from other agencies through non-financial arrangements
increase as a result of the demonstration?
• Did the number/percent
of participants who entered employment increase?
• Was the participant able to obtain employment in the area trained?
• Did the average wage
rate at placement increase?
• Did employment retention
Summary of Research
of the research questions concerning participation patterns could not be
answered because the administrative data files needed to answer these questions
were not archived. This data difficulty affected all questions that relied
on the use of historical data on exemption status and mandatory and volunteer
status. Attempts to retrieve the history of these variables through the
manipulation and matching of a number of administrative files were unsuccessful.
Response to being
called in to participate in an employment and training program dropped
25 percentage points during the PRIDE demonstration after adjusting for
differences in demographic and economic differences between the demonstration
and comparison sites. However, this decrease in response rates was not
accompanied by a corresponding increase in the issuing of Notice of Adverse
Actions (sanctions). Use of this tool was very rare in the baseline period
and did not increase in the demonstration period. The decline in response
to call-in possibly resulted from the longer length of time that persons
in the PRIDE pool had to wait before being called in to participate in
the program. Differences in the data sources used to calculate this statistic
also may have accounted for some of the response rate decline.
in a component among those who had been called-in also dropped during the
PRIDE demonstration. After adjusting for demographic and economic differences,
participation rates declined by 58 percentage points. Differences in the
manner in which ‘participation’ was defined between the two data sources
may have accounted for some of this decline.
in individual components changed significantly as a result of the PRIDE
demonstration. Far more emphasis was placed on an upfront assessment, with
a total of 779 individuals served in this component for an average duration
of 1.69 months. Assessment was not delineated as a separate component in
the regular FSE&T program.
Among the other components,
a major decline in the average monthly participation occurred in Individual
Job Search (-40 percent), accompanied by a small decline in Basic Education
(-5 percent). These reductions in participation were balanced by increases
in other activities, with positive adjusted net effects of 17 percent and
25 percent in Vocational Training and Work Experience, respectively. These
changes correspond to the intended design of the PRIDE demonstration to
increase participation in components that improved participants’ competitiveness
in the labor market.
Some major shifts in
the average number of months persons were enrolled in individual components
also occurred at a result of the PRIDE demonstration. Regression-adjusted
results indicate that the monthly duration of job search skills increased
by 1.2 months, probably the result of adding the Ho’ala curriculum to this
component. Surprisingly, the length of time in basic education declined
significantly during the demonstration, with a net reduction of 1.6 months.
This seems contrary to the expectation that more emphasis would be placed
on education and training in the PRIDE demonstration. Other changes in
duration were not statistically significant.
The JOBS (and PRIDE)
program instituted a major change in its participation requirements shortly
before the end of the study period that required persons to work or look
for work while also participating in education or training. This change
in the participation requirements strongly affected the percent of persons
still enrolled in selected components at the end of each time period. The
regression-adjusted rates of persons completing independent job search
declined by 36 percentage points while the rates of persons no longer enrolled
in vocational training increased by 40 percentage points. Obviously, this
shift in program rules affected individuals’ participation in a major way.
While far more individuals
enrolled in post-secondary education as a result of the PRIDE demonstration,
these enrollments did not result in a significant increase in the share
of persons receiving post-secondary degrees or certificates by June of
1995. This result occurred both because of the short time in which to observe
this outcome and the change in program rules cited above.
Effects of the demonstration were measured for employment rates immediately
following program participation, quarterly earnings immediately following
job placement, and job retention rates six months following employment.
Almost all of the differences in both unadjusted and adjusted net effects
were statistically insignificant. Employment entry rates ranged from 42-47
percent for persons no longer enrolled in the PRIDE or regular FSE&T
programs. Quarterly earnings averaged $1,700 - $1,900 in the quarter immediately
following placement, or $566 - $633 per month.
The only measure which
showed a significant net effect from the demonstration was employment retention.
Approximately 50-59 percent of persons who obtained jobs were still employed
six months later. Regression-adjusted net employment retention rates dropped
by 12 percentage points during the PRIDE demonstration, a statistically
signficant decline. However, incomplete demographic data resulted in some
observations being dropped from this regression, meaning that this statistic
should not be taken too seriously.
The insignificant differences
in most of the employment outcomes seem to indicate that the PRIDE demonstration
has not achieved its objective of improving employment outcomes by offering
more intensive program treatments. The short postprogram time for which
outcomes were observed and the change in program rules that caused many
individuals to drop out of their postsecondary components prior to completion
may have adversely affected these results.
the impact analysis of the PRIDE demonstration was able to document fairly
large shifts in the share of activities in which the participants engaged,
the analysis of many of the other outcomes is clouded by data issues. One
clear message from this analysis is that the handling of administrative
data by HDHS needs to be improved. Historical data needs to be archived
on a regular basis —usually monthly or quarterly — so that key variables
needed for a longitudinal analysis are not overwritten. Prior to embarking
on another research demonstration for which program operators are interested
in outcomes, HDHS should thoroughly review its data collection and archiving
procedures. Working with an evaluator at the beginning of such an endeavor
also would enable the agency to ascertain that data are being maintained
in a manner that will allow the research questions to be answered.
Even with the data difficulties,
the PRIDE demonstration does appear to have resulted in major shifts in
the types of components in which persons enrolled. Unfortunately, testing
of the major premise of this demonstration — that investing in longer-term
treatments would improve participants’ employability — was short-circuited
when program rules in the PRIDE program were changed to require that education
and training participants also work or participate in job search.