Report on Evaluation Strategies for the Program

 The success of instructional programs depends not only on learning outcomes, but also on the overall satisfaction of all vested stakeholders. For the EnterTech Project, it is important that the program satisfy an array of constituents who include the learners, instructors, enrolling educational institutions, employers, and the project coalition members.

To measure the value, worth, merit or effectiveness of a program, information about individual learners must be aggregated to provide a picture of the program as a whole. Quality assurance measures built into the instructional design process evaluate effectiveness as the program is developed. Indicators such as implementation and maintenance costs for training sites, employer and learner satisfaction measures, labor market outcomes of job placements and earnings
portray overall program effectiveness.

To plan for instructional program evaluation, the following activities are necessary:

• Research and analyze current strategies for determining the effectiveness of a training program of this type.
• Design documents to obtain information from customers as to the training program's effectiveness.
• Design documents to collect data pertaining to enrollment, retention, completion, job placement, job retention or continued education.
• Develop strategies for continued evaluation and enhancements.

Section 1. Research and analyze current strategies for determining the effectiveness of a training program

Two types of educational program evaluation are in standard use: formative evaluation and summative evaluation. Formative evaluation occurs as part of the development process. Feedback from real learners using early versions of the developed materials help improve the final program. When the program has been completed and implemented as planned, a summative evaluation examines the program, not for revision, but for acceptance and general effectiveness.

Formative Evaluation

The main function of formative evaluation is to identify and correct any design flaws during the process of instructional development. (Gentry) By collecting data and information during instructional development, designers can test and improve program effectiveness. There are many approaches to performing a formative evaluation.

Instructional system designers use a three-step process: 1:1 talk-throughs, small group testing, and field trials. Each of the three steps serves specific purposes and, cumulatively, addresses the on-going revision of a sound instructional design process. (Dick & Carey)

The first step, 1:1 talk-throughs, involves a single learner interacting with the instructional materials and an instructional designer. At the same time the learner is using the instructional materials, she talks with the instructional designer, asking questions or giving reactions. The instructional designer records the learner’s observations, and may also ask the learner questions about how she is interacting with the materials. Talk-throughs also include use of the same assessment instruments that are connected to the instructional module.

After changes and revisions suggested by the talk-throughs have been incorporated into the instruction, a small group of 6-10 typical learners uses the materials. A facilitator, rather than an instructional designer, guides the learners in use of the materials. The designer observes with minimal interruption or interaction with the facilitator and learners. Again, the assessment instruments designed or selected for the module are used. Assessment data is aggregated and analyzed. Additionally, learners’ reactions are measured in some fashion, usually through a brief attitude questionnaire, often referred to as a "smiley sheet." (Gentry)

Field trials are conducted when the materials are at final revision stage. For a field trial, no facilitator interruption is allowed as a facilitator interacts with a group similar in size to the eventual audience. Assessment instruments and reaction measures are conducted, as in the small group testing. Final revisions are made upon the data from the field trials.

Alpha sites for the EnterTech Project have been chosen to provide the learners for 1:1 talk-throughs and small group testing. Selected beta sites will act as field trial sites for Phase 2 materials. Final revisions will be made as data from the field trials is analyzed.

An additional step to strengthen the EnterTech program is to host a one-day workshop for potential program providers. During this workshop, the group will review the materials, the processes, and assessment procedures. Suggestions for improvements will be generated and reviewed. This additional input from professionals will not only serve to pinpoint potential weaknesses, but also to provide further validity for the formative evaluation itself.

Summative Evaluation

Summative evaluation occurs when the program is in use as intended. Summative evaluation often addresses the informational needs of an array of program stakeholders. EnterTech summative evaluation will have multiple audiences:

• Employers
• Learners
• Instructional program providers
• EnterTech coalition members, and
• Texas Governor’s Office and the Texas Workforce Commission.

Coalition members want to ensure the program is producing new workers for employment. Employers want to know if the program is helping learners acquire the targeted skills needed for their entry-level jobs. EnterTech program providers need to know if the program is acceptable to potential learners as customers. The potential learners need to know that the program will help them acquire the skills required for the targeted jobs.

Providing all of these audiences with all of the answers they would like is not feasible under reasonable budgetary constraints. Instead, a synthesis of needed information can provide the most reasonable summative evaluation approach. A multiple measure approach is required.

The Kirkpatrick model of evaluation provides a useful framework for EnterTech program evaluation planning. The Kirkpatrick model examines four levels of program effectiveness:

Level 1—Reaction
Measures learner reactions to the program and their opinions about effectiveness, often using questionnaires, surveys, and interviews.

Level 2—Learning
Measures student learning. Changes in knowledge, skills and abilities are measured with achievement tests and performance checklist. By correlating pre-assessment data with post-assessments, these changes can be measured and analyzed.

Level 3—Behavior
Measures whether the learning resulted in behavioral change. Measurements of actual behavioral changes in learners is difficult. In the Kirkpatrick model, this is measured after the program when the worker has returned to the work setting. EnterTech learners hired into targeted positions can be evaluated for appropriate changes in job-related behaviors. This could be performed with employer surveys sometime after the learner becomes the worker.

Level 4—Results
Measures whether the program improved the targeted situation. To many, the bottom line of the entire EnterTech project supersedes all other concerns: how many of the targeted learners have gotten targeted jobs? Answering this question will require a tracking system to count the EnterTech learners entering the program, exiting the program, and success in landing a job. Combined with an accurate accounting off project costs, these figures can also provide a rough return on investment (ROI) figure.
(Kaufman & Keller)

 

Section 2. Design documents to obtain information from customers as to the training program's effectiveness.

The informational needs of the multiple audiences suggest use of several methodologies for data collection:

• Achievement Tests
• Surveys/Questionnaires
• Interviews
• Logical analysis
• Panel review

Combining these analyses of audiences, informational needs, and levels of evaluation, a multiple-pronged approach to performing a summative evaluation is recommended. The summative evaluation process should include:

• Analysis of learner reactions gathered through attitude, surveys, and interviews,
• Analysis of aggregated pre-assessment/post-assessment data produced as part of learner assessment, including achievement tests, performance checklists, and interviews.
• Review of the program by a panel of professionals, including program providers.

This choice of multiple methodologies also addresses the design principle of mixed-methodologies by including both quantitative data and qualitative data.

In cooperation with EnterTech Project subcontractors and with the assistance of an evaluation instrument designer provided by the University of Texas at Austin, measurement tools will be developed to gather the aforementioned summative evaluation information.

Section 3. Design documents to collect data pertaining to enrollment, retention, graduation, job placement, job retention or continued education.

Follow-along studies are one of the most difficult types of evaluation to implement. Follow-along is the process of gathering data to describe learner outcomes after the individual learner has completed the program. A tracking and information gathering system must be devised to follow-up on the individuals in incremental time episodes: every 6 weeks, once-a-month, even years for longitudinal studies. And, the types of data to be measured must be chosen.

Some measures worthy of collection for the EnterTech Project include number of job placements, number of individuals seeking continued education, job wages and retention averages, and employer satisfaction.

Data gathered from follow-along studies can provide information relevant to various program stakeholders.

The Texas State Occupational Information Coordination Committee (TSOICC), which has developed a field guide for providing follow-up services, identifies the following purposes for outcomes data collection:

• Service providers use them in several ways. They help providers explain the benefits of the services they deliver. They also help providers make management decisions about which services to offer, which to discontinue, and/or which ones in their current repertoire should be expanded or improved.
• They may help eligible customers make informed choices. Where alternatives are open to them, customers need (and have a right to know) information about the outcomes achieved by prior exit cohorts in order to determine: a) if they want a service and can benefit from it; and b) how to choose from available alternatives based on reasonable expectations inferred from service providers' performance histories.
• They help stewards of public funds (e.g., local decision-makers, state administrators, legislators, advisory council members, etc.) make more rational, data-driven decisions when delivering services or procuring services on behalf of those eligible to participate in publicly funded programs.
• They help private citizens and third parties -- such as non-profit public interest research groups, community-based organizations, academic researchers, and commercial enterprises -- to determine their policy stances or to render advice to other stakeholders about programs and service providers.
  (Anderberg & Pfeiffer)

Various EnterTech stakeholders already use follow-along evaluation. For example, the Department of Human Services tracks individuals served and total caseload statistics. Tech-Prep programs receiving funding from the Carl Perkins Vocational and Applied Technology Education Act must report follow-along data. Public schools keep records of Work-Study student outcomes. One-stop career centers can receive financial incentives for improved outcome measures among their clientele.

Wherever the EnterTech program is offered, any pre-existing follow-along services should be connected to the EnterTech learners. Pilot sites selected for the EnterTech Program will be encouraged to perform a one-year follow-along service for their EnterTech program participants. Additionally, the Center for the Study of Human Resources at the LBJ School of Public Affairs, University of Texas at Austin, has inquired about conducting a longitudinal survey of EnterTech program participants statewide. Survey dimensions are currently under consideration.

Section 4. Develop strategies for continued evaluation and enhancements.

A strategic plan or a five-year business plan must be developed to address issues of sustainability and future enhancements of the EnterTech program. Whereas the scope of the grant to produce the EnterTech Project does not encompass support beyond piloting the instructional program, funding for distribution and maintenance of the program must be chartered. Two options are currently under consideration:

A "cost recovery selling price" for organizations within Texas. Cost recovery could include minimal overhead expenses plus the price of materials.

A "for profit selling price" for organizations outside of Texas. The cost should be determined for a fair market pricing comparable to other educational program products. A price point that covers overhead expenses, price of materials, plus funding for product enhancements.

Possible enhancements to the instructional product include continued migration to improved networking and multimedia delivery systems. For example, whereas Quicktime Video is a standards media delivery format via the Internet, newer technologies such as MPEG standards for the Internet will drastically improve the quality and speed of multimedia delivery within the next three years.

Likewise, the technology manufacturing industry will continue to improve its processes, adding newer automation and computerization into its manufacturing. As entry-level job tasks change, the EnterTech program must be able to incorporate new competencies into its instruction.

A committee of Texas technology companies representatives -- or an established industry group such as the Technology Initiatives Working Group (TWIG) -- should be designated to periodically evaluate the curriculum and make recommendations for improvements.

The EnterTech Project can be placed within a governmental, educational or non-profit institution as a permanent service/product for their upkeep. State universities and agencies should be surveyed for interest and service match to the EnterTech Project, starting with the project's own coalition membership.

About this Report

EnterTech Project Coordinator Melinda Jackson prepared this report. Substantial research and writing were contributed by Interactive Architex and Top Drawer Productions.

Anderberg, Marc and Pfeiffer, Jay. The Basics of Follow-Up: Excerpts from a Field Guide to Automated Follow-Up. Texas State Occupational Information Coordination Committee. June 1998.

Dick, Walter and Carey, Lou. The Systematic Design of Instruction. Harper Collins Publishers. 1990.

Gentry, Castelle G. Introduction to Instructional Development: Process and Techniques. Wadsworth Inc. 1994.

Imel, Susan. Vocational Education Performance Standards. ERIC Digest No. 96. 1990.

Kaufman, R. and Keller, J. Levels of evaluation: Beyond Kirkpatrick in Human Resource Development Quarterly. 5(4), 371-80. 1994.