home
Evaluate programs

Analyzing documents

Document analysis for program evaluation may focus on analyzing course documents of instructors participating in a program or on documents relating to the program itself such as policy and procedures, mission statements, reports, or strategic plans. In some cases, both types of documents may be used to evaluate a program.

Course documents

How deeply you analyze program documents depends on your central question(s). Examine documents to identify repeating ideas and larger themes using guidelines adapted from McNamara (1998):

On the other hand, to evaluate a specific aspect of instruction, such as the extent different learning modes are used to deliver content, use a rubric to rate your documents on various domains. Make sure you establish clear criteria for ratings such as "none," "little," "medium," or "extensive" and concretely define the relative importance of different criteria.

Example

Document analysis rubric

Are multiple learning modes being used in PowerPoint presentations for abnormal psychology courses?

Ratings Key:
none = not used
little = used less than 5 minutes
medium = used more than 5 minutes but less than half the time
extensive = used half the time or more

Lecture Reading Aural Visual Kinesthetic

Lecture 1: Introduction to Course: What is Abnormal Psychology?
Comments: extensive class discussion and interaction, presentation of multiple perspectives

medium

extensive

none

medium

Lecture 2: Historical Perspectives on Abnormal Psychology
Comments: reading assigned in preparation; some class discussion

medium

extensive

none

little

Lecture 3: The Brain and Behavior
Comments: This lecture relied heavily on figures and diagrams

medium

extensive

medium

none

Lecture 4: Diagnosis and Assessment
Comments: Some charts presented, handouts of diagnostic criteria and cases as examples

extensive

extensive

little

little

Lecture 5: Research methods
Comments: extensive lecturing on concepts, comparison of methods

medium

extensive

little

none

A second way to use course documents for program evaluation is identifying program objectives and then examining course documents such as syllabi and assignments to determine whether courses are meeting program objectives. Use the following four steps to create your course objectives table:

  1. Gather and read relevant documents. For large programs you may want to use a sample instead of reading all documents from all courses. [more] Choosing only one type of document to analyze for all courses is one way to accomplish this.
  2. List all objectives in the first column of your table.
  3. Across the top of your table list each course in a separate column.
  4. Examine the selected document for each course and determine whether program objectives are being met. Use a checklist (the objective is being met Yes/No) or a rating scale (not at all, a little, moderately, extensively) to evaluate each document and course.

Example

Objectives table

Are courses in the Applying Multimedia Instructional Techniques Yourself (AMITY) program meeting program objectives?

Key:
X = Yes
O = No
NA = Not Applicable

Program objectives Course 1 Course 2 Course 3 Course 4 Course 5
Students will understand the five basic multimedia forms. X X 0 X 0
Students will learn basic multimedia techniques. X X X X X
Students will learn how to identify and articulate instructional goals for a course. N/A X N/A N/A 0
Students will understand which multimedia form fits best with which instructional goals. N/A X X X 0
Students will produce a 20-minute multimedia instructional presentation that uses at least three basic multimedia forms. N/A N/A N/A N/A X

Program documents

A program document is any document used by program staff to operate the program or by program participants to participate in the program. Program documents may include:

At minimum, it is useful to review these types of documents to better understand the program you are evaluating because they provide excellent background information and can inform other parts of your evaluation. They are especially useful for identifying program objectives and expected outcomes.

It is also sometimes useful to systematically examine program documents to determine if the program is operating as intended. Used in conjunction with systematic observations, analyzing program documents allows you to compare how the program is supposed to operate (document analysis) and what is actually occurring (observation).

While an analysis of program documents alone cannot always answer evaluation questions, it is often a key component of the evaluation. Examine program documents to determine:

The document analysis portion of the evaluation would look something like this:

Example

Program documents

Is the Applying Multimedia Instructional Techniques Yourself (AMITY) program effective in getting faculty to integrate technology into their instruction?

Ratings Key:

Poor = dimension present in 0-19% of documents reviewed
Fair = dimension present in 20-39% of documents reviewed
Average = dimension present in 40-59% of documents reviewed
Good = dimension present in 60-79% of documents reviewed
Excellent = dimension present in 80-100% of documents reviewed

Success dimension Poor Fair Average Good Excellent

Program instructional materials include evidence of teaching multimedia instructional techniques.

        X

Program instructional materials include evidence that students are required to complete at least one multimedia assignment.

        X

Program instructional materials provide evidence of student time to learn and practice multimedia techniques.

    X    

Program instructional materials include evidence of teaching best practices as they relate to multimedia techniques.

X        

After analyzing program documents to determine what faculty were taught, you would then conduct classroom observation to see if faculty are implementing their training as well as interviews, focus groups, and/or a survey to assess faculty satisfaction with the program and help identify successful and problematic program elements.

Additional information

Russ-Eft, D. & H. Preskill (2001) Evaluation In Organizations: A Systematic Approach To Enhancing Learning, Performance, and Change. New York: Basic Books.

Page last updated: Sep 21 2011
Copyright © 2007, The University of Texas at Austin