Assess technology



Observation refers to the systematic examination of instructional technology with the goal of identifying challenges and practices. Observations typically incorporate a prescribed protocol containing specific measures of observable behavior and the narrative recording of the instruction technology in use and its context. Types of observational approaches include: check lists, scaled ratings, interval observations, and narrative comments. [more]

Suggested uses of observations:

Limitations of observations:

Resource requirements

Implementing this method requires moderate resources: developing an observational protocol, hiring an experienced assistant, and compiling and analyzing the data gathered. [more]

Plan your observation

STEP 1. Describe the instructional technology and context

Include the purpose of the instructional technology: the need it addresses, its expected effects, current resources, and resources needed to implement. Describe the users (education, motivation, skill levels), learning objectives in relation to the technology, and the learning context.   A worksheet is available to help you through this step. 

STEP 2. Identify stakeholder needs and develop central questions

Central questions identify what you and the stakeholders would want to learn from the observation(s). For example, "What features are challenging when watching webcasts?" A worksheet is available to help you identify stakeholder needs and develop central questions.

STEP 3. Determine the purpose of the observation

Your observational study should have a clear purpose and focus. Using your central questions as a guide, specify how your observation will help you gain insight, change practices for use, or measure the effects of the instructional technology. A worksheet is available to help you develop and refine your study’s purposes.

STEP 4. Determine how you will use the results

How you intend to use results should also guide the focus of your observations. Focus observations on only those aspects of the instructional technology you are interested in. A worksheet is available to help exemplify how to use results after determining the purpose of a study.

STEP 5. Determine the observational types you will use

Use checklists, scaled ratings, and interval observations if you know the behavior you want to observe.  An advantage of using these observational types is that your results are highly focused and categories are rarely overlooked.   However, a disadvantage is failing to observe unforeseen categories or to report critical details.  For these observational types, you need to create an observation form.

Observation forms

To create an observational form you need to know what behavior to observe to answer the purpose and central questions of your assessment.  Begin this process with top-down planning.  Start with the big picture (“Why are you assessing this technology?”), and systematically reduce this question level by level or category by category. 


Refer back to your purpose and central questions of the assessment, as well as the learning objectives of the course.  Why are you assessing this technology (e.g., to improve learning, student engagement, help with presentation of material, etc)?  Once you determine broad categories continue top-down planning within each category to determine specific observable behavior.  Another strategy to help establish broad categories and measurable behavior is to review prior research that describes observation forms measuring a similar educational technology that you are assessing. 


Category: Effectiveness as an instructional (or learning) tool

Measurable behavior: Improves presentation of instruction, easily integrated into current course/curriculum, conducive for writing good learning assignments, promotes and/or uses effective learning strategies.

Category: Ease of use

Measurable behavior: Easy first time use, consistent appearance, clear instructions, not limited by technical resources, needs
technical support, ease of navigation, perceived errors.

Category: Likeability

Measurable behavior: Student engagement, faculty’s enthusiasm, frustration or negative comments, etc.

Once you establish the categories and measurable behavior, convert the behavior into observation types by simply adding a rating scale, a simple yes/no for checklists, or a time interval format. [more]

Use narrative comment observations if you are unsure of the categories you want to find or want to use more of an explorative approach.  In this approach, also known as bottom-up planning, the observational data will inform you of the most important or salient categories.  However, a disadvantage is that the level of detail and time required might exhaust your available resources. 

Additional information

Center for Teaching Excellence. (undated) Preparing for Peer Observation in, TX: The University of Texas at Austin.

Chism, N.V.N. (1999). Peer Review of Teaching: A Sourcebook. Bolton, MA: Anker Publishing Co., Inc.

Page last updated: Sep 21 2011
Copyright © 2007, The University of Texas at Austin