Assess technology

Usability testing


Usability testing is the process of measuring how well people can use a product (e.g., a Web site, software, hardware, etc.) by directly observing and talking to actual or potential users of the product.  Usability is defined as the ease of use, learnability, efficiency, and error tolerability of a particular product.

There are also other methods for assessing usability of a product besides usability testing with users, such as inspection methods. [more]   However, testing actual users is recommended for instructional technology because most skilled designers and developers cannot completely predict how users will respond to products. One study showed that user involvement is the number one factor in the success of software development.

There are many different types of usability testing you can use depending upon your assessment purpose. [more] If you are developing a product, the most important requirement is to conduct iterative testing, usability testing that is repeated multiple times during different attempts and phases of the product development process. [more]

Suggested uses of usability testing:

When developing a product:

When selecting a product to adopt:

Limitations of usability testing:

Resource requirements

This method requires a moderate level of knowledge about many tasks including how to create surveys, generate valid survey questions, write measurable testing objectives, and how to gather data from a variety of qualitative methods (e.g., observations, focus groups, or interviews).  You should also understand how to use and interpret basic statistics (e.g. frequencies, means, weighting) as well as qualitative data (e.g., theme-coding). Data gathering and analysis may be time consuming requiring additional staff. [more]

Plan your usability test

STEP 1. Describe the instructional technology and context

Include the purpose of the instructional technology: the need it addresses, its expected effects, current resources, and resources needed to implement. Describe the users (education, motivation, skill levels), learning objectives in relation to the technology, and the learning context.   If possible, create a usability testing project team with people who represent different viewpoints (e.g., information and system developers, usability specialist, etc.). A worksheet is available to help you through this step. 

STEP 2. Identify stakeholder needs and develop central questions

Central questions identify what you and the stakeholders would want to learn from the usability testing. For example, "Is it easy to find information in the course website?" The most important stakeholders to consider during usability testing are the targeted users.  A worksheet is available to help you identify stakeholder needs and develop central questions.

STEP 3. Determine the purpose of the usability testing

There are two possible purposes and uses for usability testing: to improve the design of the products being developed and to inform the product selection process.  With the former, determine what stage of the product development testing should be assessed, what design feature to focus on, and how to integrate the user feedback to the design.  For product selection, the results are used to compare multiple products, to assess the overall effectiveness and satisfaction of the technology, or to understand what assistance users require after product adoption.

STEP 4. Determine how you will use the results

The purpose of your usability testing will guide how you intend to use your results.  Only conduct usability tasks that will help you design or choose a product.

STEP 5. Plan the usability testing

If the testing is for a product being developed, use iterative testing.  When deciding which product to select, use comparison or assessment testing. For either case, do the following:

Create an assessment plan that considers:

  • The time available to perform the study
  • The resources available to perform the study (e.g., people, facilities, software, hardware, etc)
  • When and where to conduct the study (field testing, lab testing, or remote testing). [more]
  • How to find potential participants (the participants should be the target users of the product)
  • How many participants to invite (five participants can find most critical problems).
  • If developing a product, which stages do you want to test?

Create testing objectives and tasks

An important aspect when planning your usability testing is to create specific and measurable objectives, followed by specific tasks that measure these objectives. [more]

Create a usability test protocol

A protocol explains the purpose of the test, the product being tested, the test environment, and testing procedures.  Click here for an example of an usability test protocol.

Determine what data collection methods you will use

Some methods are post-test surveys and post-task surveys.  They are given immediately after a certain task is completed in order to gain specific feedback of the task or how participants’ perceptions change over time, observational methods, debriefing sessions (similar to an informal interview) or narrative descriptions for usability testing– test participants are asked to describe their experience about a certain task or the overall product via writing or speaking, interviews,  or focus groups

Plan how to analyze the results

Determine how you will analyze your data:

  • Top-down planning: Planning top-down means starting with the big picture and systematically reducing results level by level.  Use this approach when you have established the categories you want to find, such as feedback, navigation, terminology, and design issues.  An advantage of this approach is that your results are highly focused and categories are rarely overlooked.   However, a disadvantage is failing to observe unforeseen categories or to report critical details. 
  • Bottom-up planning: Planning bottom-up means starting with the data collected and systematically combining data into broader and common themes or categories.  Use this approach when you are unsure of the categories you want to find or when you want the data to inform the categories, similar to explorative testing.  An advantage of this approach is that the data (i.e., test participants) will inform you of the most important or salient categories.  However, a disadvantage is that the level of detail and time required for this approach might exhaust your available resources. 

Prepare for usability testing

Preparation for usability testing includes recruiting participants, creating pre-test surveys, creating a consent/release form, and conducting informal walkthroughs. [more]

Additional information

Barnum, C. M. (2002) Usability Testing and Research. New York: Longman.

Dumas, J.S., & Redish J. C. (1999) A practical Guide to Usability Testing. Portland: Intellect.

Miami University - Students in the Masters of Technical and Scientific Communication Program (2002).  Usability Testing: Developing useful and usable products.  www.units.muohio.edu/mtsc/usabilitytestingrevisedFINAL.pdf

Nielsen, J. (1994). “Gerrilla HCI: Using Discount Usability Engineering to Penetrate the Intimidation Barrier.” Cost-Justifying Usability. Eds. Randolph G. Bias and Deborah J. Mayhew. Boston: Academic Press (242-272. )

Nielsen, J. (n.d.) How to conduct a heuristic evaluation. http://www.useit.com/papers/heuristic/heuristic_evaluation.html

Page last updated: Sep 21 2011
Copyright © 2007, The University of Texas at Austin