home
Assess technology

Technology assessment process

Instructional technology assessment involves iterative cycles of planning, gathering data, and reporting.It is best to plan the assessment at the same time that the technology introduction is being coordinated, so that you can take necessary steps such as collection of baseline data.

All program assessments follow a planning, gathering data and reporting results process consisting of the following nine steps:

Planning

STEP 1 Describe the technology and context

Describe the instructional technology to be used in terms of its function or purpose, expected effects, and resources needed. Also, describe the instructional context, such as the program or course, learner characteristics, and learning objectives in relation to the technology. 

STEP 2 Identify stakeholders and their needs

Stakeholders are the individuals and organizations involved in developing and supporting the technology, those served or affected by the technology, and the intended users of the assessment.  The needs of stakeholders generally reflect the central questions the stakeholders have about the instructional technology.

Determining stakeholder needs helps to focus the assessment process so that the results are of the greatest utility. 

STEP 3 Determine the assessment purpose

Identifying a clear purpose helps determine how to conduct the assessment. Three general purposes for instructional assessments are to gain insight, change practices, and/or measure effects.

    • Gain Insight - to clarify how instructional activities should be designed to bring about expected changes.
    • Change Practice - to improve the quality, effectiveness, or efficiency of instructional technology.
    • Measure Effects – to examine the relationship between instructional technology, teaching and learning
STEP 4 Identify intended uses

Intended uses are the specific ways assessment results will be applied. They are the underlying goals of the assessment and are linked to the central questions of the study.

STEP 5 Create an assessment plan

The assessment plan is a detailed description of how the assessment will be implemented that includes identification of the resources available for implementing the plan, what information is to be gathered, the research method(s) to be used, a description of the roles and responsibilities of the stakeholders, instructor and/or assessor and a timeline for accomplishing tasks. [more]


Example plan

Gathering Data

STEP 6 Gather data

Data gathering focuses on information acquisition that will convey a well-rounded picture of the instructional technology

Data gathering includes consideration about what indicators, data sources and methods to use, the quality and quantity of the information, human subject protections, and the context in which the data gathering occurs. [more]

Reporting Results

STEP 7 Analyze data

Data analysis involves identifying patterns in the data, either by isolating important findings (analysis) or by combining sources of information to reach a larger understanding (synthesis), and making decisions about how to organize, classify, interrelate, compare, and display information.  These decisions are guided by the questions being asked, the types of data available, and input from stakeholders.

STEP 8 Make conclusions and recommendations

Conclusions are linked to the evidence gathered and judged against agreed-upon standards set by stakeholders. Recommendations are actions for consideration that are based on conclusions but go beyond simple judgments about efficacy or interpretation of the evidence gathered.

STEP 9 Report results

Factors to consider when reporting results, or dissemination, include tailoring a report content for the audience, explaining the focus of the study and its limitations, and listing both the strengths and weaknesses of the technology. [more]

Adapted from the following sources:

Center for Disease Control Evaluation Working Group. Steps in Program Evaluation. http://www.cdc.gov/eval/steps.htm Retrieved June, 30, 2006.

Joint Committee on Educational Evaluation, James R. Sanders (chair). 1994. The Program Evaluation Standards: How to assess assessment of educational programs, 2nd ed. Sage Publications: Thousand Oaks, CA.

Popham, James W. 1993. Educational Evaluation, 3rd ed. Allyn & Bacon: Boston.

Taylor-Powell E., Rossing B., Geran J. 1998. Evaluating Collaboratives: Reaching the potential. Madison, WI: University of Wisconsin Cooperative Extension.

University of Washington, Office of Educational Assessment. Evaluation Planning Guide. http://www.washington.edu/oea/evaluatn.htm Retrieved March 26, 2004. <Note: site no longer available>

Page last updated: Sep 21 2011
Copyright © 2007, The University of Texas at Austin