Controlled experimental studies are hard to design, implement, and maintain, but may be worth the trouble when substantial resources are being invested in an innovation or program. They should not be used when programs are rapidly evolving (St. Pierre, 2004).
Group 1: Random assignment Pretest ----> Treatment ----> Posttest
Group 2: Random assignment Pretest ----------------------> Posttest
Randomly assigning participants to groups is often not feasible or unethical because random assignment removes control from students, clients, instructors, and staff, and denies some individuals immediate access to what may be a highly effective intervention. Carrying out random assignment, therefore, requires coordination, understanding, and commitment from everyone involved. [more]
Are resources available?
Conducting a controlled experiment usually requires substantial time, money, and political support. [more]
What will you compare?
You may want to compare the performance of people who participate in a program with that of people who do not participate in any program. On the other hand, it may be more realistic to compare the performance of participants in a new program with that of participants in an existing effective program.
What is the unit of random assignment?
You can randomly assign individual students, specific sections, or even campuses. [more]
Do I have a good chance of detecting differences?
In order to find statistically significant differences between two groups, you must have a good estimate of the expected difference between the control and intervention groups on outcome variables. Reviewing similar interventions from previous research can help you make this estimate and make your intervention more potent. Larger samples are more likely to reveal differences. The number of interventions being tested and the significance level of statistical tests also affect your ability to detect differences. If your knowledge of statistics is somewhat limited, consult with an expert.
How will you deal with nonparticipation and attrition?
Participants who do not complete a course or don't fully participate create a problem if their level of participation or attrition follows a pattern different from the rest of the group (i.e., is non-random). For example, if under-achieving students are more likely to drop out of the intervention group than the control group, the intervention may appear to be more effective. Gathering background information for all participants, such as previous achievement records or socioeconomic status, can help you estimate bias that is introduced and adjust for it.
How will you elicit cooperation?
Collaborate with program staff, instructors, and students or clients to gain their support and cooperation by explaining and addressing objections to randomized assignment.
Completing a pre-test measure may make participants aware of a deficit that they then address. If you use identical measures, students or clients may do better the second time because of practice.
How you measure the outcome and who measures it may change from pre- to post-test and can affect whether students or clients appear to improve. It is important that pre and post measures be equivalent.
St. Pierre, R. G. (2004). Using randomized experiments. In J.S. Wholey, J. P. Hatry, & K. E. Newcomber (Eds.) Handbook of Practical Program Evaluation, Second Edition, San Francisco: Jossey-Bass.