Reporting usability testing results
STEP 7. Analyze the data
Data analysis for usability testing consists of transforming both qualitative and quantitative data into results useful for making recommendations about a technology’s usability.
Quantitative data is usually analyzed through descriptive statistics to identify user performance measures, errors and difficulties, and tasks that do not meet acceptable criteria. [more]
Qualitative data is usually analyzed through thematic coding, which identifies general trends or categories of users’ behavior such as how well users can complete a task and where they are encountering problems. In addition, qualitative data can identify users’ general perceptions and opinions of the technology’s usability. [more]
Steps for analyzing qualitative trends will depend on the method(s) you selected for data collection. From the list below, select the assessment method(s) you used in data collection and learn how to analyze qualitative data:
- Focus groups
- Interviews and debriefing sessions
- Observational methods
- Post-surveys and narrative descriptions
STEP 8. Make conclusions
After analyzing qualitative data, begin to draw conclusions from the themes that emerged in the data. [more] Once complete, compare the results from all data sources. This process, called triangulation, can improve the reliability and validity of conclusions.
If the observation results and other information (e.g., survey responses) conflict, go with the results from the observation(s) because directly observed behavior is more valid than the remembered behaviors and attitudes captured by a survey.
Evaluate the results by how well they answer the study's central questions. Your conclusions should be based on the findings of your data analysis and should inform any recommendations. [more]
STEP 9. Report results
In general, prioritize recommendations by error severity or problem frequency. For product development, all the recommendations should be included, even though only some will be implemented. For recommendations that are implemented, follow-up testing should be conducted to confirm if the changes actually improve the usability.
The report should include both qualitative and quantitative results organized by theme.
To report qualitative results, present repeating ideas that lead to major themes that, in turn, inform conclusions and recommendations. Quote one or two responses that exemplify a repeating idea. [more]
To report quantitative results, present frequencies graphically and with the level of detail useful to the audience (e.g., a frequency table indicating 37% rather than 37.33333546%).
Report results informally to stakeholders or more formally in a presentation or written report. Reports may include a cover letter, executive summary, introduction (including needs for and goal of the test, overview of the product), methodology (including user profile, number of users, tasks, place, time, length and procedure of test, facilities and equipment used for the test, development stage of the product being developed, measurement instrument, analysis method), results, recommendations, and appendices. Use tables, graphs, and diagrams where possible. Pictures or screenshots that show where users had problems can make a strong case for recommendations. [more]