home
Evaluate programs

Product analysis conclusions

Evaluate your results based on how well they answer your central questions and base conclusions on the objectives you set for your analysis. For a program evaluation use multiple methods to triangulate findings whenever possible.

In the example below, an evaluator notices from the analysis of Essay 1 that nearly all participants are meeting objective 2 but almost no participants are meeting objectives 3 and 4 and a disappointing percentage are meeting objective 1. The evaluator might conclude that the instructor needs to rewrite the directions for Essay 1 to emphasize the requirements for originality and providing persuasive arguments and that she should spend more class time having participants apply key concepts. Conducting a document analysis and observations would help verify these conclusions.

Holistically comparing objectives across assignments is also helpful. For example, participants met objective 3 better for Essay 2 than for Essay 1. If the intent of both essays is to incorporate all learning objectives, this finding could lead to the conclusion that the Essay 1 assignment needs revision. However, if objective 3 was more central to Essay 2 than to Essay 1, revision may not be necessary. The results below could also be viewed positively, because they show general improvement across assignments. Thus, if a primary focus of the course is to teach participants to write more persuasively, these results show some success in meeting that objective.

Establishing what performance standards are acceptable prior to conducting a product analysis is important when making conclusions. For example, the table below shows that over 90% of participants are meeting objective 2 for each assignment. Whether this is acceptable depends on the instructor's standard. If the instructor has set the standard that all participants (100%) will be able to use examples of key concepts correctly, then the evaluator would conclude that the instructor needs to make adjustments to instruction or assignments.

The priority of each objective should also guide conclusions. An instructor may set a lower standard of success for an objective of medium priority compared to one of high priority. For example, you might view 90% of participants meeting a high priority objective as successful and view 70% of participants meeting a medium priority objective as successful.

By organizing information in this way, it is easier to formulate possible explanations for the results and identify solutions. Are learning objectives not being communicated clearly? Is the material too challenging? Are participants capable of doing the work but need more information or resources? A follow-up survey could help answer such questions, and analyses of the syllabus, lecture notes, and handouts (document analysis) could reveal whether the instructor is clearly communicating objectives and needed information.

Example

Percentage of participants meeting objectives for multimedia assignment in the Applying Multimedia Instructional Techniques Yourself (AMITY) program

Program Objective Priority Essay 1 Essay 2 Term Paper
1. Application of concepts

high

65%

88%

75%

2. Use of examples

high

92%

95%

90%

3. Persuasive arguments

medium

20%

75%

82%

4. Originality

medium

10%

27%

38%

Additional information

Patton, M.Q. (1990). Qualitative evaluation and research methods, 2nd ed. Newbury Park, CA: Sage.

Russ-Eft, D. & H. Preskill (2001) Evaluation In Organizations: A Systematic Approach To Enhancing Learning, Performance, and Change. New York: Basic Books.

Page last updated: Sep 21 2011
Copyright © 2007, The University of Texas at Austin