home
Evaluate programs

Survey conclusions

Determine whether your response rate was adequate based on the size and accessibility of the population you are sampling and the survey mode you used. Check if response rates for subgroups in your sample are comparable. If response rates vary greatly for subgroups, it may shed light on other results. For instance, if participants from one school have a low response rate and also report spending the least amount of time collaborating on a program, their time may be very limited.

Critically examine the content of questions, especially when you obtain unexpected results, to see if their wording might be biased or unclear or if the question order might have created bias.

Use care when interpreting cross-tabulation results. For example, survey responses of staff participants in a program to incorporate technology into instruction indicate that those who worked on several projects at once were more likely to experience problems than those who worked on one project:

Example

Challenge Single projects
(% yes, with 25 respondents)
Multiple projects
(% yes, with 5 respondents)

Technical difficulties resulting in delays

24

40

Getting started late

41

60

One explanation is that staff attempting multiple projects became overwhelmed with the work involved in managing several projects at once. Another possibility, however, is that the odds of encountering a problem increased with the number of projects attempted, and that, per project, those involved in multiple projects were not more likely to have technical difficulties or start late than those working on one project. In addition, because of the small number of respondents in the multiple project group, results are more likely to be due to chance and might change with a larger sample. To determine the most reasonable explanation, examine responses to other survey questions or findings from other data sources, such as interviews, focus groups, and any previous surveys. This process, called triangulation, can improve the reliability and validity of conclusions.

Additional information

American Association for Public Opinion Research (2000). Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys, Second Edition [Electronic version], Ann Arbor, MI: AAPOR http://www.aapor.org

Page last updated: Sep 21 2011
Copyright © 2007, The University of Texas at Austin