home
Evaluate programs

Document analysis conclusions

Draw conclusions based on the results of your analysis. To what degree are program objectives being met based on your analysis of course or program documents

In the first example below, you might conclude that instruction is based primarily on written and aural approaches to learning and recommend that program instructors receive training on how to incorporate kinesthetic and visual approaches.

Example

Document analysis rubric

Are multiple learning modes being used in PowerPoint presentations for abnormal psychology courses?
Ratings Key:
none = not used
little = used less than 5 minutes
medium = used more than 5 minutes but less than half the time
extensive = used half the time or more

Lecture Reading Aural Visual Kinesthetic

Lecture 1: Introduction to course: What is Abnormal Psychology?
Comments: extensive class discussion and interaction, presentation of multiple perspectives

medium

extensive

none

medium

Lecture 2: Historical perspectives on abnormal psychology
Comments: reading assigned in preparation; some class discussion

medium

extensive

none

little

Lecture 3: The Brain and behavior
Comments: This lecture relied heavily on figures and diagrams

medium

extensive

medium

none

Lecture 4: Diagnosis and assessment
Comments: Some charts presented, handouts of diagnostic criteria and cases as examples

extensive

extensive

little

little

Lecture 5: Research methods
Comments: extensive lecturing on concepts, comparison of methods

medium

extensive

little

none

In the second example below, you might conclude that courses 1, 2, and 4 are fulfilling program objectives, but course 3 is not meeting one objective and course five is concentrating only on techniques and their application. In this case, you could recommend that course five be substantially revised.

Example

Objectives table

Are courses in the Applying Multimedia Instructional Techniques Yourself (AMITY) program meeting program objectives?
Key:
X = Yes
O = No
NA = Not Applicable

Program objectives Course 1 Course 2 Course 3 Course 4 Course 5

Students will understand the five basic multimedia forms.

X

X

0

X

0

Students will learn basic multimedia techniques.

X

X

X

X

X

Students will learn how to identify and articulate instructional goals for a course.

N/A

X

N/A

N/A

0

Students will understand which multimedia form fits best with which instructional goals.

N/A

X

X

X

0

Students will produce a 20-minute multimedia instructional presentation that uses at least three basic multimedia forms.

N/A

N/A

N/A

N/A

X

In the final example below, it appears that the program is successful in teaching multimedia techniques and their application but is not successful in teaching best practices instructional techniques, and program participants may need more time to practice the techniques they are learning. These findings could also be used to help explain observational findings.

Example

Program documents

Is the Applying Multimedia Instructional Techniques Yourself (AMITY) program effective in getting faculty to integrate technology into their instruction?
Ratings Key:
Poor = dimension present in 0-19% of documents reviewed
Fair = dimension present in 20-39% of documents reviewed
Average = dimension present in 40-59% of documents reviewed
Good = dimension present in 60-79% of documents reviewed
Excellent = dimension present in 80-100% of documents reviewed

Success dimension Poor Fair Average Good Excellent

Program instructional materials include evidence of teaching multimedia instructional techniques.

X

Program instructional materials include evidence that students are required to complete at least one multimedia assignment.

X

Program instructional materials provide evidence of student time to learn and practice multimedia techniques.

X

Program instructional materials include evidence of teaching best practices as they relate to multimedia techniques.

X

To verify conclusions, examine results from other assessment methods such as observations, focus groups, interviews, or surveys. Finding similar results using different methods strengthens conclusions.

On the other hand, results that differ may suggest further analysis. If, for example, surveys and interviews indicate that instructors have redesigned courses to meet program objectives, but the analysis of course syllabi provides no evidence of this, examine each method for reliability. Is document analysis measuring the same behavior that surveys and interviews are measuring? Do instructors or program leaders believe they have incorporated changes when they have not? If results from different methods are contradictory, you may need to gather more information to be confident in your conclusions.

Additional information

Russ-Eft, D. & H. Preskill (2001) Evaluation In Organizations: A Systematic Approach To Enhancing Learning, Performance, and Change. New York: Basic Books.

Page last updated: Sep 21 2011
Copyright © 2007, The University of Texas at Austin