home
Assess teaching

Analyzing documents

How deeply you analyze course documents depends on your central question(s). To better understand your teaching approach and how you communicate it to students, examine documents to identify repeating ideas and larger themes using these guidelines adapted from McNamara (1998):

Document analysis rubric

On the other hand, to evaluate a specific aspect of your instruction, such as the extent that you use different learning modes when making presentation or the effectiveness of a course website, use a rubric to rate your documents on various domains. Make sure you establish clear criteria for ratings such as "none," "little," "medium," or "extensive" and concretely define the relative importance of different criteria.

Example

Use of learning modes in PowerPoint presentations for abnormal psychology

Ratings key

  • none = not used
  • little = used less than 5 minutes
  • medium = used more than 5 minutes but less than half the time
  • extensive = used half the time or more
# Lecture title Comments Reading Aural Visual Kinesthetic
1 Introduction to course: What is abnormal psychology?

extensive class discussion and interaction, presentation of multiple perspectives

medium

extensive

none

medium

2 Historical perspectives on abnormal psychology

reading assigned in preparation; some class discussion.

medium

extensive

none

little

3 The brain and behavior

This lecture relied heavily on figures and diagrams

medium

extensive

medium

none

4 Diagnosis and assessment

Some charts presented, handouts of diagnostic criteria and cases as examples

extensive

extensive

little

little

5 Research methods

extensive lecturing on concepts, comparison of methods

medium

extensive

little

none

Example

Document analysis rubric for a course Web site

A second type of rubric, adapted from Vandervelde (2005) is used to evaluate the content of a course Web site and assigns points to ratings to enable comparison to a standard:

Criteria Exemplary Proficient Partially Proficient Incomplete Points

Focus

6 points
The content has a clear main idea or theme.

4 points
The theme or main idea of the content is evident but could be clearer.

2 points
The theme or main idea is vague and does not create a strong sense of purpose

0 points
The website lacks a clear purpose or central theme.

4

Quality of content

6 points
Provides useful and engaging educational information

4 points
Provides useful educational information, but could be more engaging

2 points
Does not include educational information that students can easily use

0 points
Is not useful to students

4

Links and references

6 points
The content points readers to high quality, up to date pertinent resources. The content provides essential information to the reader.

4 points
The content points readers to quality information resources. The content provides useful information to the reader.

2 points
The content points readers to information that does not relate to the purpose or theme of the page. Information is incomplete or inaccurate.

0 points
The content points readers to some information resources that are inaccurate or misleading or inappropriate for the intended audience.

6

Use of photos, graphics, sound and video

3 points
All of the photographs, graphics, sound and/or video enhance the content and create interest.

2 points
Most of the photographs, graphics, sound and/or video enhance the content and create interest.

1 points
A few of the photographs, graphics, sound and/or videos are inappropriate for the content and do not create interest.

0 points
The photographs, graphics, sounds, and/or videos are inappropriate for the content or there are no photos, graphics, sound, or videos.

0

Internal and external navigation

3 points
All pages connect back to the home page and/or sitemap.
All external links to connecting websites are active and functioning.
A sitemap/index is provided.

2 points
Most pages connect back to the home page and/or sitemap.
Almost all of the external links to connecting websites are active and functioning.
A limited sitemap/index is provided.

1 point
Some pages connect back to the home page and/or sitemap.
Some of the external links to connecting websites are not active and functioning.
No sitemap/index is provided.

0 points
There are few or no connections back to the preceding pages or to the original index page.
Many external links to connecting websites are not active and functioning.
No sitemap/index is provided

1

Grammar

3 points
The text has no errors in grammar.

2 points
The text has very few grammatical errors requiring minor editing

1 point
The text has between four and seven errors in grammar requiring editing.

0 points
The text has more than seven errors in grammar and requires major editing.

3

 

 

 

 

Total Points

18/30

Overall rating (maximum = 30):

Exemplary = 26 or more
Proficient = 16 - 25
Partially proficient = 7 - 15
Incomplete = 6 or fewer

Document analysis table or matrix

Another approach to document analysis is using a table or matrix to align exams, assignments, lectures, or labs to course objectives. Such an analysis not only helps you to identify explicit course objectives but can also identify implicit ones not contained in the syllabus. Once course objectives are identified, you can then examine course documents to identify which exams, lectures, assignments or labs meet each objective. Use the following process to create your course objectives table:

  1. Gather and read relevant documents.
  2. Beginning with your syllabus, identify each course objective. Some objectives may be contained in the course description or reading description and not be explicitly identified as a course goal.
  3. Examine assignments and labs for additional course objectives.
  4. List all the identified objectives in the first column of your table.
  5. Across the top of your table list each exam, assignment, lecture, or lab in separate columns.
  6. Examine each course document and identify which objectives are being met in each.

Example

Course objectives table

Course objectives Exam 1 Exam 2 Exam 3 Assignment 1 Assignment 2

Students will understand the various forms of contemporary journalism.

1, 3, 7, 16, 22, 26, 29, 30

4, 7, 20

14, 36

X

 

Students will learn to critique the various forms of contemporary journalism.

2, 3, 8, 22, 23, 26, 28, 33, 40

1, 4, 7, 18, 20

14, 23, 36, 45

 

X

Students will learn basic journalistic writing techniques.

 

 

 

X

X

Students will learn the basic elements of a news story.

43, 44, 45, 50

 

 

X

 

Note: The numbers in each exam column represent the question numbers that relate to the course objective for that exam. Exam questions and assignments may relate to more than one course objective.

Additional information

Vandervelde, J. (2005) Rubric for Classroom Web Pages. Retrieved  June 21, 2006 from the University of Wisconsin--Stout School of Education Web site: http://www.uwstout.edu/soe/profdev/webpagerubric.html

McNamara, C. (1998). Basic guide to program evaluation. Retrieved   June 28, 2006 from http://www.mapnp.org/library/evaluatn/fnl_eval.htm

Page last updated: Sep 21 2011
Copyright © 2007, The University of Texas at Austin