Analyzing survey data
Enter closed-ended survey responses into a program
Enter closed-ended survey responses into a program like Excel, Access, SPSS, or SAS, which can perform statistical analyses and create tables and graphs. Often, you can import data from electronic surveys directly into the analysis software. Code the data by assigning a number for each response (for example, "definitely no" = 1, "no" = 2, etc.) and create a key explaining the coding for each question.
Reverse score responses if necessary
Some survey questions may be worded so that a given response (i.e. “definitely no” = 1) represents an unfavorable rating for one question, but a favorable rating for another. An example would be two questions that asked, “The instructor communicated effectively” and “The instructor communicated poorly.” In order to compare or aggregate these survey responses, the inconsistent survey question should be reverse scored.
To reverse score, switch the highest and lowest numerical values of a response, then substitute the next highest and lowest values, and so on. Non-numerical responses, such as “other” or “non-applicable”, should not be included in reverse scoring.
Question: "The instructor communicated poorly"
"definitely yes" =4
"yes" = 3"
"no" = 2
"definitely no" = 1
"definitely yes" = 1
"yes" = 2
"no" = 3
"definitely no" = 4
Open-ended survey responses
For open-ended survey responses, identify general themes and noteworthy exceptions to trends. If you have a large amount of text, you may want to code responses to organize content and reveal patterns [more about coding]. With short-answer questions, you can obtain basic information about response frequencies by categorizing open-responses, assigning a numerical code to each category, and then entering the codes into a statistical analysis program.
You may need to recode some answers to questions that have an open-ended "other" response option. For example, one person may answer the question, "Do you consider yourself African American, Caucasian, Asian, Hispanic, or other?" by circling "other" and writing, "Chinese." To maintain consistency, you would code the answer as "Asian" rather than "other."
Inspect the data
Inspect the data for errors that occur during data entry or when respondents provide inconsistent answers. For large databases, check at least five percent of entered data for accuracy. If you find any errors, check the remainder of the data and correct the errors.
Calculate the response rate
Calculate the response rate by dividing the number of people who submitted a completed survey (80% or more of questions answered) by the number of people you attempted to contact. If 185 program participants were asked to complete the survey, and 107 responded, the response rate was 107/185 or 58%. Consider other formulas for calculating responses rates, such as counting partially completed surveys as responses.
Response rates below 70 percent
If the response rate is below 70 percent, determine if the sample is representative of the target population by comparing sample and target population means for characteristics-such as race, age, major, or grade point average-that would likely affect responses. An unrepresentative sample may have response bias. For example, if 40% of faculty participants in a program are from the School of Engineering, yet only 10% of evaluation survey respondents are from that school, results will not represent the concerns of this subgroup. To address the problem, you might arrange additional interviews with engineering faculty participants to supplement your results. Often, the best solution is to weight results so that the attitudes of important subgroups are not underrepresented. For example, you could multiply the limited responses from engineering faculty by four. Weighting, however, is problematic if the people who responded differ in important ways from those who did not respond.
Calculate response frequencies and percentages for each question
Count the number of respondents who selected each response choice for a question to obtain frequencies and divide these frequencies by the total number of responses to the question to compute percentages. For example, for the question below, the "strongly disagree" response percentage would be calculated: 10/107 = .09 or 9%.
To create more meaningful categories, combine the "agree" and "strongly agree" categories to obtain the percentage of agreement (65%) and the "disagree" and "strongly disagree" categories for the percentage disagreement (24%).
Compute cross-tabulations to see relationships between responses for two survey questions. For example, you may ask students to provide their race in the first question and rate their overall satisfaction with a program in the second question. Display response choices for the first question as column labels and choices for the second question as row labels. For example, a cross-tabulation might reveal that the percentage of participants who were satisfied with a program depended on their race:
Cross-tabulations highlight differences between groups of participants or findings that are consistent across groups. Deciding which cross-tabulations to compute is guided by your questions at the onset of the evaluation or patterns you notice when reviewing the data.
American Association for Public Opinion Research (2000). Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys, Second Edition [Electronic version], Ann Arbor, MI: AAPOR [http://www.aapor.org]