Observation
Overview
Observation refers to the systematic examination of instructional technology with the goal of identifying challenges and practices. Observations typically incorporate a prescribed protocol containing specific measures of observable behavior and the narrative recording of the instruction technology in use and its context. Types of observational approaches include: check lists, scaled ratings, interval observations, and narrative comments. [more]
Suggested uses of observations:
- Observation is especially useful when combined with the use of other assessment methods.
- Gaining insight by describing the instructional technology in use and identifying any needs or challenges.
- Developing ideas for future assessment by using observational data to help focus other assessment instruments such as surveys.
Limitations of observations:
- Not suitable for a summative evaluation of an instructional technology.
- Largely descriptive.
Resource requirements
Implementing this method requires moderate resources: developing an observational protocol, hiring an experienced assistant, and compiling and analyzing the data gathered. [more]
Plan your observation
STEP 1. Describe the instructional technology and context
Include the purpose of the instructional technology: the need it addresses, its expected effects, current resources, and resources needed to implement. Describe the users (education, motivation, skill levels), learning objectives in relation to the technology, and the learning context. A worksheet is available to help you through this step.
STEP 2. Identify stakeholder needs and develop central questions
Central questions identify what you and the stakeholders would want to learn from the observation(s). For example, "What features are challenging when watching webcasts?" A worksheet is available to help you identify stakeholder needs and develop central questions.
STEP 3. Determine the purpose of the observation
Your observational study should have a clear purpose and focus. Using your central questions as a guide, specify how your observation will help you gain insight, change practices for use, or measure the effects of the instructional technology. A worksheet is available to help you develop and refine your study’s purposes.
STEP 4. Determine how you will use the results
How you intend to use results should also guide the focus of your observations. Focus observations on only those aspects of the instructional technology you are interested in. A worksheet is available to help exemplify how to use results after determining the purpose of a study.
STEP 5. Determine the observational types you will use
Use checklists, scaled ratings, and interval observations if you know the behavior you want to observe. An advantage of using these observational types is that your results are highly focused and categories are rarely overlooked. However, a disadvantage is failing to observe unforeseen categories or to report critical details. For these observational types, you need to create an observation form.
Observation forms
To create an observational form you need to know what behavior to observe to answer the purpose and central questions of your assessment. Begin this process with top-down planning. Start with the big picture (“Why are you assessing this technology?”), and systematically reduce this question level by level or category by category.
Categories
Refer back to your purpose and central questions of the assessment, as well as the learning objectives of the course. Why are you assessing this technology (e.g., to improve learning, student engagement, help with presentation of material, etc)? Once you determine broad categories continue top-down planning within each category to determine specific observable behavior. Another strategy to help establish broad categories and measurable behavior is to review prior research that describes observation forms measuring a similar educational technology that you are assessing.
Example
Category: Effectiveness as an instructional (or learning) tool
Measurable behavior: Improves presentation of instruction, easily integrated into current course/curriculum, conducive for writing good learning assignments, promotes and/or uses effective learning strategies.
Category: Ease of use
Measurable behavior: Easy first time use,
consistent appearance, clear instructions, not limited by technical
resources, needs
technical support, ease of navigation, perceived errors.
Category: Likeability
Measurable behavior: Student engagement, faculty’s enthusiasm, frustration or negative comments, etc.
Once you establish the categories and measurable behavior, convert the behavior into observation types by simply adding a rating scale, a simple yes/no for checklists, or a time interval format. [more]
Use narrative comment observations if you are unsure of the categories you want to find or want to use more of an explorative approach. In this approach, also known as bottom-up planning, the observational data will inform you of the most important or salient categories. However, a disadvantage is that the level of detail and time required might exhaust your available resources.
Additional information
Center for Teaching Excellence. (undated) Preparing for Peer Observation in, TX: The University of Texas at Austin.
Chism, N.V.N. (1999). Peer Review of Teaching: A Sourcebook. Bolton, MA: Anker Publishing Co., Inc.
