home
Evaluate programs

Analyzing products

For program evaluation, a sample of participant or client products is typically used unless the program is small and it is feasible to examine all participant products. Your sampling strategy will depend on the nature of the population and resources available. [more on sampling] Once you draw your sample, use quality standards, narrative descriptions, a combination of the two, or program objective tables to compare the content of products with established outcome objectives for the program.

If practical, use multiple trained evaluators. To improve consistency between evaluators or reliability, train them in a group, have them practice using the same form, and compare results. If there is not at least 80% agreement between evaluators, discuss differences and repeat the process until there is satisfactory agreement.

Quality standards

One way to use quality standards is to record the number or percent of participants who are meeting each program objective for an assignment or activity, or to record the extent to which participants are meeting objectives. Use rubrics to get consistent scores across the sample. [more]

In the example below, each participant’s multimedia program would be evaluated using the rubric. Data for each program would then be entered into a database for analysis using Excel, Access, or a statistical software package. Calculate the mean rating for each objective after entering the data.


Example

Rubric for analysis of a participant multimedia product:
Key:
Not present = no use or demonstration of objective
Below criteria = little use or demonstration of objective or use is frequently inaccurate
Meets criteria = consistent use or demonstration of objective
Exceeds criteria = consistent and skillful use or demonstration of objective

Objective Not present
(0 pts)
Below criteria
(1 pt)
Meets criteria
(2 pts)
Exceeds criteria
(3 pts)

Participants will understand the five basic multimedia forms.

    X  

Participants will learn basic multimedia techniques.

      X

Participants will learn how to identify and articulate instructional goals for a course.

  X    

Participants will understand which multimedia form fits best with which instructional goals.

  X    

Narrative descriptions

Use narrative descriptions to clarify how participant demonstrate program objectives and document the skills, concepts, and resources that participants used to create the product. Focus your descriptions on the elements of the participant products that best illustrate program objectives. Use terms consistently to make comparisons easier. For example, always use "good" to describe above average performance and "excellent" to describe superior performance.


Example

Narrative for analysis of a multimedia product:

Objective Narrative Comments

Participants will understand the five basic multimedia forms.

Demonstrated understanding of four forms correctly and one incorrectly. Applications were based on those presented in class but went beyond the basic idea. Good elaboration.

Participants will learn basic multimedia techniques.

Demonstrated evidence of correct use of all basic multimedia techniques. Excellent technical skills.

Participants will learn how to identify and articulate instructional goals for a course.

Instructional goals identified but not always articulated clearly. Also, goals were not always directly connected to application.

Participants will understand which multimedia form fits best with which instructional goals.

Some original thought and use of multimedia forms, but application to instructional goals correct for only one form.

To analyze narrative descriptions, enter them into a word processing program and code them into meaningful categories. Coding enables you to organize large amounts of text and discover patterns that would be difficult to detect by just reading the narratives. Bogdan and Biklin (1998) suggest first ordering descriptions by some meaningful criteria. Next, conduct initial coding by generating numerous category codes as you read commentary, labeling data that are related without worrying about the variety of tegories. Because codes are not always mutually exclusive, a phrase or section might be assigned several codes. Last, use focused coding to eliminate, combine, or subdivide coding categories and look for repeating ideas and larger themes that connect codes. Repeating ideas are the same idea expressed by different evaluators, while a theme is a larger topic that organizes or connects a group of repeating ideas. Try to limit final codes to between 30 and 50. After you have developed coding categories, make a list that assigns each code an abbreviation and description [more]

Berkowitz (1997) suggests considering these questions when coding qualitative data:

Software programs can help with coding qualitative data, understanding conceptual relationships, or counting key words. They facilitate systematic, efficient coding and complex analyses. Three popular software packages for qualitative coding and data analysis are Atlas.ti and NVivo7 and XSight.
Once you have identified repeating ideas and larger themes, relate them to your central questions. How do the ideas and themes help to answer your central questions? What additional questions do the data raise? What conclusions and recommendations do the narrative findings suggest? How do the narrative findings relate to findings from other study components such as surveys or interviews?

Quality standards and narratives

You can also combine narrative descriptions with quality standard comparisons for a more comprehensive product analysis:

Example

Narrative with quality standards for analysis of a multimedia product:
Key:
0 = no use or demonstration of objective
1 = little use or demonstration of objective or use is frequently inaccurate
2 = consistent use or demonstration of objective
3 = consistent and skillful use or demonstration of objective

Objective Narrative Quality Ratings

Participants will understand the five basic multimedia forms.

Demonstrated understanding of four forms correctly and one incorrectly. Applications were based on those presented in class but went beyond the basic idea. Good elaboration.

2

Participants will learn basic multimedia techniques.

Demonstrated evidence of correct use of all basic multimedia techniques. Excellent technical skills.

3

Participants will learn how to identify and articulate instructional goals for a course.

Instructional goals identified but not always articulated clearly. Also, goals were not always directly connected to application.

1

Participants will understand which multimedia form fits best with which instructional goals.

Some original use of multimedia forms, but application to instructional goals correct for only one form.

1

Using both quality ratings and narrative comments can help to provide additional insight. Narrative comments can help you interpret numeric ratings for an objective. For example, if the average of quality ratings for the third program objective was 1.45, you would likely conclude that this objective is not being met. By systematically analyzing the narrative comments associated with the objective, you are better able to understand why this objective is not being met or to identify specific barriers to meeting the objective

Program objective tables

Another way to use participant products in program evaluation is to identify program objectives and then examine participant products from all program courses to determine whether courses are meeting program objectives. Use the following four steps to create your program objectives table.

  1. Gather and rate participant products using a standards rubric or narrative description (see above). Choosing participant products for a single assignment from each program course is one way to draw a sample. [more on sampling]
  2. List all program objectives in the first column of your table.
  3. Across the top of your table list each course in a separate column.
  4. Examine the results for each course and determine whether program objectives are being met for that course. Use a checklist (the objective is being met Yes/No) or a rating scale (not at all, a little, moderately, extensively) to evaluate each course.

Example

Program objectives table

Are courses in the Applying Multimedia Instructional Techniques Yourself (AMITY) program meeting program objectives?
Key:
X = Yes
O = No
NA = Not Applicable

Program objective Course 1 Course 2 Course 3 Course 4 Course 5
Participants will understand the five basic multimedia forms. X X O X O
Participants will learn basic multimedia techniques. X X X X X
Participants will learn how to identify and articulate instructional goals for a course. NA X NA NA O
Participants will understand which multimedia form fits best with which instructional goals. NA X X X O

Additional information

Patton, M.Q. (1990). Qualitative evaluation and research methods, 2nd ed. Newbury Park, CA: Sage.

Russ-Eft, D. & H. Preskill (2001) Evaluation In Organizations: A Systematic Approach To Enhancing Learning, Performance, and Change. New York: Basic Books.

Page last updated: Sep 21 2011
Copyright © 2007, The University of Texas at Austin