Report on Assessment and Evaluation Strategies for the Learner

Assessment is a key success factor for any curriculum. The application of reliable and accurate pre-assessment tools assesses a learner’s appropriate "fit" into a curriculum, having both the pre-requisite skills necessary to complete the curriculum and the personal interest and aptitude to do so. Ongoing and iterative assessment within the instructional modules can inform student of progress as well as provide feedback for selecting individual learning strategies and achieving program learning goals. Likewise, proper post-instructional assessment assures learners have demonstrated the knowledge, skills and abilities of the program's terminal objectives. For the EnterTech Project, learners should be able to demonstrate the knowledge, skills and abilities necessary to be successful in the entry-level job workplace or in a continuing education program.

The following activities were conducted to inform instructional design decisions on learner assessment:

 

Section 1. Research and analyze current tests, tools and instruments used to assess knowledge, skills, abilities, aptitudes, and interest.

Learner assessment can occur before, during, and after instruction. Pre-assessment activities occur before instruction and may inform program entry decisions. Pre-assessment can also help determine placement within a program that offers multiple learner levels. On-going assessment occurs during the program. This assessment can be performed at specific, distinct assessment opportunities, such as progress tests. Or assessment may be performed constantly, embedded within the curriculum to assess a student’s evolving level of understanding. Final assessment, or post-assessment, occurs at the end of the instructional program. Post-assessment may determine program completion or certification.

Assessment can range from paper-and-pencil tests to observing a learner’s performance to reviewing a portfolio of work prepared by the learner. All of these methods are valid for different purposes for different programs. Much of traditional assessment centers on achievement tests. Achievement tests are designed to measure a learner’s knowledge and skills in a particular content area. For programs serving large populations, standardized achievement tests predominate, as these also serve to aggregate data for program evaluation. However, many concerns revolve around the use of standardized achievement tests, especially in measuring higher-order thinking skills, or in specialized learner populations. Alternative assessment approaches, such as assessing performance through portfolios, provide real work examples, or "authentic assessment," of student’s demonstrated knowledge, skills and abilities.

The EnterTech Project will include a combination of assessment strategies: authentic assessments, portfolio assessments, embedded assessments, and achievement-test assessments.

 

Authentic Assessment

Assessment activities can be considered "authentic" when the learning they measure has "value beyond the classroom and is meaningful to the learner." (Kerka) Traditional assessment activities such as multiple-choice tests may measure stored knowledge and knowledge retrieval, but often do not "measure up" when applied to such essential workplace skills as critical thinking and individual responsibility. Authentic assessment attempts to approximate the real world performance that will be expected. Within the EnterTech curriculum, students will be assessed through their interactions and choices made in work-related situations. From selecting possible solutions to resolve production problems on the assembly line, to correctly completing an inventory count and recording results into a spreadsheet for the shipping department, student achievement will be measured by their performance of real workplace skills.

Well-designed authentic assessments demonstrate an array of what learners know and can do; they display products and processes of learning; and they are accumulative, depicting learner growth over time. (Kerka) When aligned to the curriculum, authentic assessments connect thinking and doing into real contexts and provide self-monitoring and reflection for students. (ibid)

 

Portfolio Assessment

The most widely used technique in authentic assessment is student portfolios. Portfolios can include samples of class work, checklists rating progress in workplace skills, and journals recording student reactions to instruction and activities. (Burt & Saccomano) Through portfolios, students can create ongoing records of their achievements, skills and goals. The information students add to portfolios can create a resource for use after instruction too. From reflecting on and accentuating personal qualities, to developing career and education checklists, from keeping a current resume on hand, to including sample cover letters, the portfolio becomes a personal tool valuable to life and career decisions.

 

Embedded Assessment

As the student navigates through the computer-based scenarios, their skills are assessed as they interact with the instruction. This is known as embedded assessment. Rather than periodically stop for a true-false, multiple choice or other traditional type of assessment exercise, the student is assessed continually as they problem-solve and select or input solutions into the computer. The computer provides a means of storing the assessment results that are administered within the program. Even more, the computer can calculate student progress and "prescriptively branch" the student into increasingly complex skill levels or remediate unproven skills. The computer-based performance results will be stored in a report that students can print for their individual portfolios.

 

Standardized Achievement Test

The use of standardized achievement tests has been problematic with the targeted learner population for EnterTech, as previously detailed in the Targeted Learner Characteristic Report and later in this report. However, such achievement tests remain standard evaluation tools for Texas schools and service providers. For example, the Texas Department of Human Services assesses client skills through the Test of Adult Basic Education (TABE) whereas Texas public schools are mandated to test student skills with the Texas Academic Achievement Systems (TAAS). Additionally, one intended outcome of the EnterTech instruction is that students be able to pass Work Key assessment exams at the skill levels determined appropriate for the entry-level jobs profiled for the curriculum.

Whereas standardized achievement tests will not be used as an iterative testing method within the EnterTech instructional program, they can be useful for measures of pre-requisite skills and post-instructional outcomes. Since many of the organizations referring learners into the EnterTech program already use standardized tests, it is beneficial - in terms of cost and time savings - to correlate these measures to EnterTech entry criteria and to use the pre-collected results for assessing learner’s appropriate fit into the program. Likewise, if students so choose to take Work Keys assessments after program completion, these assessment results can be used to verify both learner skills attainment and program effectiveness.

The following sections of this report more fully articulate possible learner assessment strategies for the EnterTech Project. The ongoing assessment activities must be selected or developed in tandem with the instructional materials and learning activities. This report outlines different types of assessments and links to competency areas. The appendices contain descriptions of numerous, stand-alone assessment tools already in use by Texas schools and public service entities; these can aid in obtaining pre-assessment and post-assessment results.

 

Section 2. Determine prerequisite criteria to enter the Entertech Project training program, and select or develop appropriate pre-assessment instrument(s).

Pre-assessment for EnterTech should include:

This process will be implemented with many candidates who have historically responded poorly to traditional testing. EnterTech will be offered in diverse settings where it is possible that candidates may go through an existing battery of tests for pre-screening. Thus, the EnterTech pre-assessment process must strike a fine balance between thoroughness and the need to avoid overburdening potential candidates with testing.

The primary purpose of pre-assessment for EnterTech is to provide information about the eligibility of an individual for entry into the program. A secondary, but also important, purpose for pre-assessment is to provide data to help measure program effectiveness. Several considerations impact the selection of appropriate pre-assessment activities for EnterTech.

 

Prerequisite Skills Assessment Instruments

Several instruments considered potentially suitable for pre-assessment use with EnterTech have been reviewed. The list is not exhaustive, but represents the instruments most commonly used to test the stated prerequisite skills for EnterTech. Stated prerequisite skills for EnterTech are:

The pre-assessment instruments reviewed were:

More comprehensive information on each of these instruments is contained in the Appendix to this report.

Another aspect potentially appropriate for pre-assessment would be the individual’s career interests and aptitudes. Career counseling should accompany pre-assessment testing for EnterTech. This can include live interaction with a counselor as well as career interest/aptitude instruments.

All of the instruments reviewed test the appropriate prerequisite skills: 8th grade reading level and 6th grade level mathematics ability. Criteria considered in this review of pre-assessment instruments are listed below.

Appropriate audiences are targeted—Young adults to adults, of varying ethnic backgrounds, predominately female, overwhelmingly unsuccessful in traditional academic settings, including standardized testing situations

Length of test taking is limited—Preference should be given to tests that yield the essential information in less time. Tests lasting more than an hour are probably less effective in reaching the target learners. This is particularly important since additional program entry procedures (career aptitude/interest measures, career counseling interviews) also must be scheduled.

Flexible test administration—Pre-assessment should be available on an individualized, "walk-in" basis to encourage participation. Group administration should still be possible to help EnterTech program providers to maximize test administrators.

External review of reliability & validity—Publishers make many claims about the reliability and validity of their instruments. In making a selection, program planners need to have some sort of external review of these claims. This is more difficult on more recently developed instruments. The Buros Institute publishes the highly regarded Mental Measurements Yearbook series that features scholarly reviews of various tests. The Mental Measurement Yearbooks series, however, lags behind the newest developments. For example, the 1998 version (13th Mental Measurement Yearbooks) reports on "new instruments" from 1994. This gap of 4-5 years prevents newly introduced instruments from competing on a level field.

Other considerations—Additional factors impact the effectiveness or appropriateness of pre-assessment instruments. Some instruments severely restrict conditions for administering the instrument. Some instruments report scores on their own scale rather than providing a method for determining grade equivalency. Any instruments selected for use should also be screened for cultural bias, as the target learner population represents great cultural and ethnic diversity.

 

Prerequisite Skills Recommendations

Pre-assessment activities for EnterTech need to measure the prerequisite skills of reading and math to determine if candidates will be able to benefit from the program. These skills can be measured with standardized achievement tests already in use by Texas schools and service organizations. Eight standard assessment instruments have been reviewed for this report. The matrix (Table 1) shows information about each assessment in several key criteria.

Table 1. Prerequisite Skills Assessment Instruments

Name Appropriate Audience Length of Test Test administration External Review Other considerations
ABLE X 175-215 minutes Group or individual 11th MMY  
SelectABLE X 15 minutes Group or individual 11th MMY Locator test
ASVAB Young adults 3.5 hours Administered only at MEPS and METS 9th MMY Available only from Armed Services
CAT-ASVAB Young adults 2 hours Administered only at MEPS and METS 9th MMY Computer administration
CASAS X 60 minutes per assessment Must be performed by CASAS trained staff 13th MMY Linked to CASAS curriculum
TABE X 154-209 minutes Group or individual 13th MMY  
TABE, Locator X 45 minutes Group or individual 13th MMY Locator test
TABE, Work-Related Foundation Skills X 120 minutes Group or individual 13th MMY  
TAAS Through 12th grade No time limits within sections Under TEA guidelines Technical report available "Live" TAAS available only under restricted conditions
TALS X 40 minutes per subtest, 120 minutes for full battery Group or individual 13th MMY  
Work Keys X 40 minutes per test Group or individual Limited Scores not linked to grade equivalency
WRAT-3 X 15-30 minutes Group or individual 10th MMY:

"not advisable"

 

 

Of these, both CASAS and Work Keys are designed to report scores within a proprietary system. Linkage to grade equivalencies is not clear.

Only the TABE Locator tests can be used in a stand-alone manner to report grade equivalencies for reading and math. The additional research that demonstrates the effectiveness of the TABE locator tests for program placement support the use of these briefer tests as assessment instruments within the EnterTech program.

Even if EnterTech selects a specific pre-assessment instrument, other indications of prerequisite skills should be considered when making program entry decisions. If an EnterTech candidate has other proof of prerequisite skills, i.e., recent TAAS proficiency ratings for 6th grade math and 8th grade reading, the candidate should be considered for program entry. EnterTech providers may in fact, choose to accept test results other than TABE Locator tests, for program entry decisions.

As the EnterTech curriculum is developed, instructional designers may suggest crafting a pre-requisite assessment customized to the EnterTech program. One issue for consideration arises around the efficacy of all assessment measures. Recent research demonstrates that results on general aptitude (ability) tests parallel the results on skill focused tests (reading, math, etc.) The greater construct validity (i.e. the closer the test items reflect the tasks/skills being measured) the more meaningful the results. While these statements seem at odds, it appears to be a matter of degree. Unless resources allow creation of customized instruments with extremely high construct validity, instruments may be selected based on any factors (cost, ease of administration, etc.) with minimal difference in results. (Journal of Applied Psychology).

Other pre-assessment activities should be performed as well to legitimize program entry decisions. Career counseling can help ensure that candidates are motivated to complete the EnterTech program and enter the high-tech work force. This career counseling would likely include a personal interview as well as the administration of a career interest/aptitude instrument.

 

Career Interest/Ability Assessment Instruments

A broad range of career interest/ability measures have been examined in an effort to facilitate EnterTech’s goal of developing a job skills training program specific to entry-level positions in Texas technology industries. Profiles of these instruments do not represent an exhaustive list, but offer a diverse group of instruments that show potential toward providing assessment information that will address EnterTech’s job profiles.

The current broad search around career interest/ability measures includes instruments in numerous areas (Appendix B):

Instruments that meet EnterTech’s desire to incorporate screening for true determinants of motivation are difficult to locate. EnterTech may select from the many interest, attitude and values measures available on the market. A number are listed in this report. Because of the importance of academic and vocational guidance, commercially available interest inventories have been almost as popular as tests of general intelligence and special abilities. Compared with cognitive measures, however, interest inventories are not very accurate predictors of school grades or occupational performance. Scores on interest inventories contribute to the prediction of occupational selection, persistence, and satisfaction, but job success is usually more closely related to ability than to interests (Kuder; Campbell & Hansen). Because people are more likely to avoid occupations that they dislike than they are to enter occupations that they like, low scores on interest inventories tend to be more predictive of what a person will not do than high scores are predictive of what he or she will do (Dolliver, Irvin, & Bigley; Sytowski).

When considering the use of interest, aptitude and values measures, other factors can also affect the outcome:

The socioeconomic status of the respondent is one demographic that is significantly related to the responses, and consequently to the validity of interest inventories. Working-class people do not always have an opportunity to cultivate their interests or to train for and enter occupations that are appealing to them. To these individuals, monetary security in a more important factor in employment decisions than satisfying one’s interest. This is one reason why, for many years, psychologists showed little inclination to construct inventories to measure the vocational interests of people who were planning to enter unskilled, semiskilled, or even skilled occupations. Because money seemed to be a more important occupational determinant than satisfying one’s vocational interests, the development of interest inventories for nonprofessional occupations was viewed as unproductive. In any event, it would seem that most occupations in the modern workplace do not provide for satisfaction of the interests of most workers (Warnath). So what do workers do when they find that there are large discrepancies between what they would like to do and what they have to do in order to survive? In most cases, rather than jeopardizing their security in relentless pursuit of their vocation interests and ambitions, they are much more likely to adjust their aspirations to be closer to what they can actually attain (Gottfredson & Becker).

 

Career Interest/Choice Recommendations

Instruments and measures appropriate to EnterTech needs should address the factors previously listed as criteria, as well as concerns of construct validity and how best to narrow the assessment focus given the current broadly defined EnterTech job profiles. Instruments recommended for further investigation follow under their respective header.

 

Learner Employability

The Employee Aptitude Survey. This survey is a multiple-aptitude battery designed specifically for personnel selection and placement in business and industry. The battery is broken down into tests with short time limits, which may appeal to possible EnterTech candidates. Because it examines a wide range of aptitudes, it presents a diverse picture of candidate strengths.

Jackson Vocational Interest Survey (JVIS). The JVIS is one of the most carefully designed and validated of all general-interest inventories. The survey is based on results of extensive research and consists of 289 forced-choice pairs of statements describing job-related activities. The instrument requires 45-60 minutes to complete. The JVIS has received praise for its validity, careful construction and factorially pure scale. More evidence for validity is needed (Davidshofer; Thomas)

TABE Work Related Foundation Skills and TABE Work Related Problem Solving, 1994. This instrument is recommended for further review due to its wide use in testing along with other TABE instruments. It yields data in the two EnterTech pre-requisite skill areas of reading and math, as well as a range of work-related applications.

Industrial/Technical

Bennett Test of Mechanical Comprehension, Form W1. If EnterTech job profiles require understanding of mechanical principles involved in a range of practical situations, the Bennett is one of the most popular tests in this category. The items on this test consist of drawings and questions concerning the operation of mechanical relationships and physical laws in practical situations. Separate norms are provided by sex. Evidence for the validity of the test in found in modest correlations with performance in a variety of occupations requiring mechanical ability (Aiken).

Applied Technology Series (ATS). With its focus on applied and high technology fields, ATS warrants further review. This test may be applicable if job profiles include skills related to areas of new technology such as fault finding, spatial checking and diagrammatic thinking.

Other

Logic eXtension Resources. The customized approach offered by Logic eXtension appears to have strong potential when job profiles become more specific. This instrument and the customized approach EnterTech might achieve, speaks strongly to the concern for construct validity as well as tailoring instrumentation for the population, length of administration and flexibility.

 

Interests, Attitudes And Values

Strong and Kuder. Both the Strong-Campbell Interest Inventory and the Kuder Occupational Surveys are widely used, highly regarded and have substantial documentation regarding validity.

Myers Briggs Type Indicator (MBTI). MBTI is widely used and is composed of a series of two-choice items concerning preferences or inclinations in feelings and behavior. The Myers-Briggs inventory is associated with Jung’s theory of personality.

Each of these assessments must be evaluated in the context of its purpose within the EnterTech assessment plan. If EnterTech will use scores to make personnel decisions, the assessments must meet strict reliability and validity standards, sufficient to provide a sound legal defense. This will require painstaking attention to the psychometric quality of the instrument, to the standardization of the administration, and to the accuracy of scoring. Instruments used by EnterTech should be of immediate use to job candidates in making the transition to the workforce and should support the information needs of not only EnterTech, but also the adults in transition (Saterfiel & McLarty).

 

Section 3. Incorporate iterative and concluding testing in the instructional design

During the instructional program, the learner needs to be assessed on a regular, on-going basis. This sort of assessment can be as simple as informal observations communicated to the learner by the facilitator or as involved as achievement tests and performance checklists. Because the EnterTech targeted learner has historically performed poorly on traditional tests, the emphasis for on-going assessment in EnterTech must shift focus to the more authentic assessment possibilities.

Suggested on-going assessment techniques include:

This section lists various on-going assessment techniques and links each to competency areas and instructional goals appropriate for assessment by the techniques. The design principle of emergent design indicates that selection or design of the specific instruments to be used must be deferred until specific curriculum is developed. On-going assessment procedures will be developed during the instructional design phase. Furthermore, these assessment procedures will be tested for appropriateness during the formative evaluation phase.

 

Achievement Tests

Achievement tests form the backbone of traditional learner assessment procedures. Many areas of recalled knowledge or cognitive skills could be tested through various forms of achievement tests, such as multiple choice tests. Possible areas within the proposed EnterTech curriculum for assessing with achievement test instruments include:

Another form of achievement tests, such as short answer/essay test, might be needed for assessing:

Rating scales might be used for personal stress level and attendance records.

 

Oral Exams/Interviews

Oral exams and interviews might be used to assess the following areas:

 

Performance Checklists

Several performance objectives are best measured in true or simulated performance. A performance checklist generally uses a rubric with set criteria to assess the learner’s performance. Some of the areas best assessed with performance checklist include:

Some checklists would be based on observations: self-observation, observations from the facilitator, and observations from peers. These competency areas might be best assessed with observational ratings:

 

Facilitator’s Evaluation

Some areas can best be evaluated and rated by the facilitator:

 

Work Samples/Products

Review of work samples or products can suffice for assessing many critical competencies. Some of these work samples/products include:

 

Projects

Several of these products will be produced independently, but many will result from projects. Projects can also provide means of assessing additional competencies. Some of the possible project-related assessment could include:

The on-going assessment of the learner during the instructional program is essential to guiding the learner towards the goal of completing the program. They will be inextricably linked to the curriculum as developed. Development of the on-going assessment instruments and procedures will necessarily be completed at the same time of instructional content design.

Scoring Rubrics

Rubrics are scoring devices or tools that specify performance expectations at various levels. (Kerka) They provide a framework for consistent assessment among students, as well as establish the benchmarks for documenting progress. (ibid) Whereas authentic assessments can involve a degree of subjective evaluation on the part of the student (in selecting which materials to put into a portfolio) and instructor (in indicating degree of skill level), a rubrics provides a fair framework to measure the performance of many students to the same task or skill set. A typical rubric should include the following:

For example, a student is asked to develop an improved inventory system. The rubric lists criteria such as number and quality of sources of information used, ingenuity and creativity, quality documentation, and workability of the system, etc. Each criterion could be rated as exemplary, acceptable or not yet acceptable. (ibid)

As the rubrics are designed, feedback from learners and instructors - as well as coalition employers - should be collected to verify real-world authenticity of the tasks and their effects on achieving learning outcomes. (ibid)

 

Section 4. Select or develop appropriate post assessment instrument based on requisite knowledge, skills and abilities required for entry-level jobs or continuing education.

As the learner completes the instructional program, assessment of learning is conducted. Within the EnterTech program, this final assessment will provide information determining the awarding of a certificate of completion. This certificate gains in importance by the use of authentic assessment procedures.

While there are many ways to perform authentic assessment, the preferred method for many programs is the use of portfolio assessment.

Portfolio assessment more closely approximates the process whereby an applicant is selected for a job. Such selection is rarely based upon a single measure, and often includes a review of many materials presented by the applicant. Portfolio assessment also addresses the need for multiple measures and mixed methodology.

Portfolios come in many shapes and sizes. The final make-up of the EnterTech certification portfolio will need to be determined as the content of the instructional program is finalized.

The EnterTech portfolio can and likely should include:

Most, if not all of these portfolio pieces will have already been produced during the learner’s journey through the EnterTech instructional program. Some, such as test results, can be produced as part of a final assessment process. The final portfolio should further be presented and explained by the learner, much as it would be for a potential employer. The facilitator likewise will evaluate this final portfolio presentation.

Additionally, students will be prepared to take standardized achievement tests upon completing the EnterTech program. TABE locator tests can be used to measure changes in the reading and math levels of EnterTech learners. And most significantly, learners will be prepared to take Work Keys assessments. Whereas learners can progress to various levels, the Work Keys skill levels targeted by the EnterTech curriculum are:

Assessment Area Level
Reading for Information 3-4
Locating Information 3-4
Listening 3
Writing 3-4
Applied Mathematics 3-4
Teamwork 3-6
Applied Technology 3-5
Observation 3-5

             

Conclusion

Assessment and evaluation are essential in the development of any effective instructional program. EnterTech, in seeking to serve a difficult target learner population, has a greater need of assessment and evaluation atypical of traditional testing. This report has examined issues involved in learner assessment and includes recommendations not only for specific pre-assessment instruments and procedures, but also guidelines for developing on-going assessment and post-assessment processes. Assessment and curriculum development are inextricably linked and therefore continuing analysis, identification and creation of learner assessment strategies will occur during instructional design.

 

About this Report

EnterTech Project Coordinator Melinda Jackson prepared this report. Substantial research and writing were contributed by Interactive Architex and Top Drawer Productions.

Bibliography and Appendices