Document analysis conclusions
Draw conclusions based on the results of your analysis. To what degree are program objectives being met based on your analysis of course or program documents
In the first example below, you might conclude that instruction is based primarily on written and aural approaches to learning and recommend that program instructors receive training on how to incorporate kinesthetic and visual approaches.
Example
Document analysis rubric
Are multiple learning modes being used in PowerPoint presentations for abnormal psychology courses?
Ratings Key:
none = not used
little = used less than 5 minutes
medium = used more than 5 minutes but less than half the time
extensive = used half the time or more
| Lecture | Reading | Aural | Visual | Kinesthetic |
|---|---|---|---|---|
Lecture 1: Introduction to course: What is Abnormal Psychology? |
medium |
extensive |
none |
medium |
Lecture 2: Historical perspectives on abnormal psychology |
medium |
extensive |
none |
little |
Lecture 3: The Brain and behavior |
medium |
extensive |
medium |
none |
Lecture 4: Diagnosis and assessment |
extensive |
extensive |
little |
little |
Lecture 5: Research methods |
medium |
extensive |
little |
none |
In the second example below, you might conclude that courses 1, 2, and 4 are fulfilling program objectives, but course 3 is not meeting one objective and course five is concentrating only on techniques and their application. In this case, you could recommend that course five be substantially revised.
Example
Objectives table
Are courses in the Applying Multimedia Instructional Techniques Yourself (AMITY) program meeting program objectives?
Key:
X = Yes
O = No
NA = Not Applicable
| Program objectives | Course 1 | Course 2 | Course 3 | Course 4 | Course 5 |
|---|---|---|---|---|---|
Students will understand the five basic multimedia forms. |
X |
X |
0 |
X |
0 |
Students will learn basic multimedia techniques. |
X |
X |
X |
X |
X |
Students will learn how to identify and articulate instructional goals for a course. |
N/A |
X |
N/A |
N/A |
0 |
Students will understand which multimedia form fits best with which instructional goals. |
N/A |
X |
X |
X |
0 |
Students will produce a 20-minute multimedia instructional presentation that uses at least three basic multimedia forms. |
N/A |
N/A |
N/A |
N/A |
X |
In the final example below, it appears that the program is successful in teaching multimedia techniques and their application but is not successful in teaching best practices instructional techniques, and program participants may need more time to practice the techniques they are learning. These findings could also be used to help explain observational findings.
Example
Program documents
Is the Applying Multimedia Instructional Techniques Yourself (AMITY) program effective in getting faculty to integrate technology into their instruction?
Ratings Key:
Poor = dimension present in 0-19% of documents reviewed
Fair = dimension present in 20-39% of documents reviewed
Average = dimension present in 40-59% of documents reviewed
Good = dimension present in 60-79% of documents reviewed
Excellent = dimension present in 80-100% of documents reviewed
| Success dimension | Poor | Fair | Average | Good | Excellent |
|---|---|---|---|---|---|
Program instructional materials include evidence of teaching multimedia instructional techniques. |
X |
||||
Program instructional materials include evidence that students are required to complete at least one multimedia assignment. |
X |
||||
Program instructional materials provide evidence of student time to learn and practice multimedia techniques. |
X |
||||
Program instructional materials include evidence of teaching best practices as they relate to multimedia techniques. |
X |
To verify conclusions, examine results from other assessment methods such as observations, focus groups, interviews, or surveys. Finding similar results using different methods strengthens conclusions.
On the other hand, results that differ may suggest further analysis. If, for example, surveys and interviews indicate that instructors have redesigned courses to meet program objectives, but the analysis of course syllabi provides no evidence of this, examine each method for reliability. Is document analysis measuring the same behavior that surveys and interviews are measuring? Do instructors or program leaders believe they have incorporated changes when they have not? If results from different methods are contradictory, you may need to gather more information to be confident in your conclusions.
Additional information
Russ-Eft, D. & H. Preskill (2001) Evaluation In Organizations: A Systematic Approach To Enhancing Learning, Performance, and Change. New York: Basic Books.
