Designing an effective experiment
Outline tasks and create a timeline
Write a step-by-step procedure detailing how you will implement interventions, measure effects, and analyze data, to understand the preparation, time commitment, and outside assistance required. Outline the responsibilities of students, clients, volunteers, sponsors, or other staff.
Whenever possible, choose accepted measureswhose validity is supported by evidence, such as agreement with other established measures of the same variable (Bordens & Abbott, 1996). Measures should be reliable, producing similar results at different times, and accurate, yielding results that agree with an established standard. Use measures that are sensitive to an effect when one is present. Avoid decreased range effects, which occur when participants frequently encounter upper or lower limits when responding.
For example, an exam that is extremely easy will result in a ceiling effect if half of a class obtains a perfect score, while a question that asks how many times participants have experienced glossolalia (nonmeaningful speech associated with a trance state or religious fervor) will result in a floor effect if only one person has done so. Having ceiling or floor effects limits your ability to find an effect.
Anticipate alternative explanations and design methods to rule them out
As much as possible, the conditions for each group should be the same except for your intervention. If you plan to compare two course sessions, choose sessions that meet for the same duration at approximately the same time of day, are around the same size, and have roughly similar students.
For example, if you compare honors and regular sections in an experiment testing the impact of student feedback on essay writing, you may wonder if your intervention worked or if the effect was due to stronger motivation among the honors students.
Choose a design based on your resources and the study's purpose
- A single-group experiment is the easiest design to implement and creates the least disruption to a course or program. [more]
- A field experiment compares groups of participants assigned nonrandomly to different treatments. For example, if you compare two sections of a course, students assign themselves by enrolling in a particular section. [more]
- Controlled experiments, in which participants are randomly assigned to an intervention or control group, are uniquely capable of producing accurate measures of the effectiveness of an intervention. [more]
Bordens, K.S. and Abbott, B.B. (1996). Research Design and Methods: A Process Approach. 3rd ed. Mountain View, CA: Mayfield.