Survey findings

Determining the findings of your survey involves more than simply reporting initial results. Instead, it is important to critically examine survey questions and check for statistical pitfalls to develop accurate findings upon which you can make reliable conclusions.

Critically examine questions

• Critically examine questions, especially when you obtain unexpected results, to see if their wording might be biased or unclear or if the question order might have created bias . Wording also impacts how well a question captures actual differences between respondents. Make sure you have avoided ceiling and floor effects, which occur when respondents frequently encounter upper or lower limits.

Check for statistical pitfalls

• Determine whether the response rate was reasonable based on the size and accessibility of the population you sampled and the survey mode you used.   The average response rate is usually greater than 60%. [more]
• Use care when interpreting cross-tabulation results. For example, survey responses of staff participants in a program to incorporate technology into instruction indicate that those who worked on several projects at once were more likely to experience problems than those who worked on one project:

  Example

  Challenge	Single Projects (% yes, with 25 respondents	Multiple Projects (% yes, with 5 respondents)
  Technical difficulties resulting in delays	24	40
  Getting started late	41	60

  One explanation is that staff attempting multiple projects became overwhelmed with the work involved. Another possibility, however, is that the odds of encountering a problem increased with the number of projects attempted, and that, per project, those involved in multiple projects were not more likely to have technical difficulties or start late than those working on one project. In addition, because of the small number of respondents in the multiple project group, results are more likely to be due to chance and might change with a larger sample.

• Could there be any errors due to sample size? If you have fewer than 25 cases per group, you may lack adequate statistical power to detect differences between groups. On the other hand, if you have very large groups, almost any difference, even a trivial one, will be statistically significant, and could lead you to make unwarranted conclusions.  For this reason, you should indicate effect sizes, which allow the readers to judge how meaningful the differences are between/among groups.  [more]

• Are you making multiple comparisons between variables? Each additional comparison between groups increases the chance of finding an erroneous relationship due to chance.  Decrease errors resulting from multiple comparisons by using a more stringent significance level, adjusting for the number of comparisons made, or using multiple comparison techniques that account for this issue.

• Other statistical pitfalls

Make conclusions

• Evaluate your results based on how well they answer your research questions or confirm your hypotheses.
• Statistically significant and/or practically significant findings should form the basis of your main conclusions. Emphasize your strongest findings.
• Consider all possible explanations for results before making conclusions.
• Verify (triangulate) findings from your survey with results from other data sources such as interviews or focus groups that can provide additional insight. Finding similar results using different methods strengthens conclusions. On the other hand, differing results call for further analysis.

Additional information

American Association for Public Opinion Research (2000). Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys, Second Edition [Electronic version], Ann Arbor, MI: AAPOR

Arsham, H. (2005). Questionnaire Design and Surveys Sampling, 9th ed. Retrieved June 21, 2006 from: http://home.ubalt.edu/ntsbarsh/stat-data/Surveys.htm

Bogdan R. B. & Biklin, S. K. (1998). Qualitative Research for Education: An Introduction to Theory and Methods, Third Edition. Needham Heights, MA: Allyn and Bacon.

Coe, R. (2000). What is an 'effect size'? A guide for users. Retrieved June 21, 2006 from the University of Durham, Curriculum, Evaluation and Management Centre, Evidence-Based Education-UK Web site: http://www.cemcentre.org/renderpage.asp?linkid=30325015

Darlington, R.B. (1997). Factor analysis. Retrieved June 212006from: http://comp9.psych.cornell.edu/Darlington/factor.htm

Helberg, C. (1995). Pitfalls of data analysis. Retrieved June 21, 2006from: http://my.execpc.com/4A/B7/helberg/pitfalls/

Lane, D. M. (2003). Tests of linear combinations of means, independent groups. Retrieved June 21, 2006from the Hyperstat Online textbook: http://davidmlane.com/hyperstat/confidence_intervals.html

Linear regression. (n.d.) Retrieved June 21, 2006from the Yale University Department of Statistics Index of Courses 1997-98 Web site: http://www.stat.yale.edu/Courses/1997-98/101/linreg.htm

Lowry, R. P. (2005). Concepts and Applications of Inferential Statistics. Retrieved June 21, 2006from: http://faculty.vassar.edu/lowry/webtext.html

Spector, P. A. (1992). Summated Rating Scale Construction: An Introduction. Newbury Park, CA, Sage.