Analyzing documents

Document analysis for program evaluation may focus on analyzing course documents of instructors participating in a program or on documents relating to the program itself such as policy and procedures, mission statements, reports, or strategic plans. In some cases, both types of documents may be used to evaluate a program.

Course documents

How deeply you analyze program documents depends on your central question(s). Examine documents to identify repeating ideas and larger themes using guidelines adapted from McNamara (1998):

• Start with evaluation goals.
• Read all the documents.
• Label ideas (e.g., instances of one type of teaching approach) as you go along.
• Organize ideas by similar content and identify repeating ideas.
• Identify themes-patterns or associations and causal relationships among repeating ideas.

On the other hand, to evaluate a specific aspect of instruction, such as the extent different learning modes are used to deliver content, use a rubric to rate your documents on various domains. Make sure you establish clear criteria for ratings such as "none," "little," "medium," or "extensive" and concretely define the relative importance of different criteria.

Example

A second way to use course documents for program evaluation is identifying program objectives and then examining course documents such as syllabi and assignments to determine whether courses are meeting program objectives. Use the following four steps to create your course objectives table:

1. Gather and read relevant documents. For large programs you may want to use a sample instead of reading all documents from all courses. [more] Choosing only one type of document to analyze for all courses is one way to accomplish this.
2. List all objectives in the first column of your table.
3. Across the top of your table list each course in a separate column.
4. Examine the selected document for each course and determine whether program objectives are being met. Use a checklist (the objective is being met Yes/No) or a rating scale (not at all, a little, moderately, extensively) to evaluate each document and course.

Example

Program documents

A program document is any document used by program staff to operate the program or by program participants to participate in the program. Program documents may include:

• mission statements
• policy and procedures manuals
• strategic plans
• reports
• logs
• timesheets
• memos
• emails
• meeting minutes
• publicity materials
• forms
• brochures
• staff training manuals
• client instructional material

At minimum, it is useful to review these types of documents to better understand the program you are evaluating because they provide excellent background information and can inform other parts of your evaluation. They are especially useful for identifying program objectives and expected outcomes.

It is also sometimes useful to systematically examine program documents to determine if the program is operating as intended. Used in conjunction with systematic observations, analyzing program documents allows you to compare how the program is supposed to operate (document analysis) and what is actually occurring (observation).

While an analysis of program documents alone cannot always answer evaluation questions, it is often a key component of the evaluation. Examine program documents to determine:

• What technology integration strategies are being taught (client instructional materials)
• The objectives of the program (mission statement)
• Reasons for program success or failure (training materials, policy and procedures, staff training materials)

The document analysis portion of the evaluation would look something like this:

• Identify and define key dimensions such as technology integration strategies, program objectives, and program success/failure, and then develop an objectives table to evaluate each dimension.
• Review each document and determine whether the dimension is present (yes/no).
• Compute the percentage of the documents reviewed that show the dimension.
• Evaluate overall program success by using the rubric in the example below:

Example

After analyzing program documents to determine what faculty were taught, you would then conduct classroom observation to see if faculty are implementing their training as well as interviews, focus groups, and/or a survey to assess faculty satisfaction with the program and help identify successful and problematic program elements.

Additional information

Russ-Eft, D. & H. Preskill (2001) Evaluation In Organizations: A Systematic Approach To Enhancing Learning, Performance, and Change. New York: Basic Books.