Designs for Experiments (2024)

PPA 696 RESEARCH METHODS

EXPERIMENTAL DESIGNS FOR RESEARCH

Causality
Experimental Designs
Control Group Pre-test/Post-testDesign
Threats toInternal Validity
Threats to ExternalValidity
Post-Test only Control GroupDesign

CAUSALITY

To establish whether two variables are causally related, that is, whethera change in the independent variable X results in a change in the dependentvariable Y, you must establish:

1) time order--The cause must have occurred before the effect;

2) co-variation (statistical association)-- Changes in the valueof the independent variable must be accompanied by changes in the valueof the dependent variable;

3) rationale-- There must be a logical and compelling explanationfor why these two variables are related;

4) non-spuriousness-- It must be established that the independentvariable X, and only X, was the cause of changes in the dependent variableY; rival explanations must be ruled out.

To establish causality, one must use an experimental or quasi-experimentaldesign. Note that it is never possible to prove causality, but onlyto show to what degree it is probable.

EXPERIMENTAL DESIGNS

True experimental designs include:

Pre-test/Post-test control group design

Solomon Four-Group design

Post-test only control group design

Pre-test/Post-testcontrol group design
This is also called the classic controlled experimental design, andthe randomized pre-test/post-test design because it:

1) Controls the assignment of subjects to experimental (treatment) andcontrol groups through the use of a table of random numbers.

This procedure guarantees that all subjects have the same change of beingin the experimental or control group. Because of strict random assignmentof subjects, it is assumed that the two groups are equivalent on all importantdimensions and that there are no systematic differences between the twogroups. Researchers may substitute matching for random assignment.Subjects in the two groups are matched on a list of characteristics thatmight affect the outcome of the research (e.g., sex, race, income).This may be cheaper but matching on more than 3 or 4 characteristics isvery difficult. And if the researcher does not know which characteristicsto match on, this compromises internal validity.

2) Controls the timing of the independent variable (treatment) and whichgroup is exposed to it.

Both group experience the same conditions, with the exception ofthe experimental group, which receives the influence of the independentvariable (treatment) in addition to the shared conditions of the two groups.

3) Controls all other conditions under which the experiment takes place.

Nothing but the intervention of the independent (treatment) variable isassumed to produce the observed changes in the values of the dependentvariable.

The steps in the classic controlled experiment are:

1) randomly assign subjects to treatment or control groups;

2) administer the pre-test to all subjects in both groups;

3) ensure that both groups experience the same conditions except that inaddition the experimental group experiences the treatment;

4) administer the post-test to all subjects in both groups;

5) assess the amount of change on the value of the dependent variable fromthe pre-test to the post-test for each group separately.

These steps are diagramed as follows:

R O₁X O₂
R O₁O₂

This diagram can be expanded upon as in the following table:

Scientific Random Assignment of Subjects to:	1st observation (measurement) of the dependent variable O1 = Pre-test	Exposure to the Treatment (X) (independent variable)	2nd observation (measurement) of the dependent variable O2 = Post-test
Experimental Group	Experimental Group's average score on the dependent variable	X	Experimental Group's average score on the dependent variable
Control Group	Control Group's average score on the dependent variable		Control Group's average score on the dependent variable

The difference in the control group's score from the pre-test to thepost-test indicates the change in the value of the dependent variable thatcould be expected to occur without exposure to the treatment (independent)variable X.

Control group - control group= control group difference
pre-test score post-testscoreon the dependent variable

The difference in the experimental group's score from the pre-test tothe post-test indicates the change in the value of the dependent variablethat could be expected to occur with exposure to the treatment (independent)variable X.

Experimental group - experimentalgroup = experimental group difference
pre-test scorepost-test scoreon the dependent variable

The difference between the change in the experimental group and thechange in the control group is the amount of change in the value of thedependent variable that can be attributed solely to the influence of theindependent (treatment) variable X.

Control group difference - experimentalgroup difference = difference attributableto X

This can be illustrated by the following experiment to see whether participationin small group discussions would improve medical students' ability to respondto emotional needs of patients:

Scientific Random Assignment of Medical Students to:	How many times did students use emotional words to describe patients	Exposure to the Treatment (X) (independent variable)	How many times did students use emotional words to describe patients
Small group discussions (experimental group)	Average of .68 times per student in 3 case studies	Attended small group discussions plus regular course work	Average of 2.02 times per student in 3 case studies
Control Group	Average of .89 times per student in 3 case studies	Regular course work only	Average of 1.13 times per student in 3 case studies

The control group used emotional words an average of .89 times per student(in three case studies) on the pre-test and an average of 1.13 times perstudent (in three case studies) on the post-test. The differencein the control group's score from the pre-test to the post-test is +.24times per student. This indicates the change in using emotional wordsthat could be expected to occur with regular course work only.

The experimental group used emotional words an average of .68 timesper student (in three case studies) on the pre-test and an average of 2.02times per student (in three case studies) on the post-test. The differencein the experimental group's score from the pre-test to the post-test is+1.34 times per student. The experimental group's score from thepre-test to the post-test indicates the change in using emotional wordsthat could be expected to occur with regular course work plus the smallgroup discussions.

The difference between the change in the experimental group (+1.34)and the change in the control group (+.24) is +1.10. This is theamount of change in using emotional words that can be attributed solelyto the influence of the small group discussions.

The controlled or true experimental design allows the researcher tocontrol for threats to the internal and external validity of the study.Threats to internal validity compromise the researcher's ability to saywhether a relationships exists between the independent and dependent variables.Threats to external validity compromise the researcher's ability to saywhether this study's findings are applicable to any other groups.

Controllingfor Threats to Internal Validity

1) History: did some other current event effect the changein the dependent variable? No, because both groups experienced thesame current events.

2) Maturation: were changes in the dependent variable dueto normal developmental processes? No, because both groups experiencedthe same developmental processes.

3) Statistical Regression: did subjects come from low orhigh performing groups? Differences between the two groups that couldinfluence the dependent variable would be controlled for as subjects weregenerally equivalent at the beginning of the research.

4) Selection: were the subjects self-selected into experimentaland control groups, which could affect the dependent variable? No,the subjects were assigned by strict random selection and all had equalchance of getting the treatment or control condition.

5) Experimental Mortality: did some subjects drop out?did this affect the results? About the same number of students madeit through the entire study in both the experimental and control groups,so there appears to be no bias.

6) Testing: Did the pre-test affect the scores on the post-test?Both groups got a pre-test; but a pre-test may have made the experimentalgroup more sensitive to the treatment.

7) Instrumentation: Did the measurement method change duringthe research? The measurement method and instruments did not change.

8) Design contamination: did the control group find outabout the experimental treatment? did either group have a reasonto want to make the research succeed or fail? The researcher mustdo some qualitative investigation to find out if there was design contamination.

Controlling for Threatsto External Validity

1) Unique program features: There may have been an unusuallymotivated set of facilitators for the small group discussions.

2) Effects of Selection: Probably applicable to othermedical students.

3) Effects of Setting: Medical schools have their own cultures;doubtful if this would be applicable to other types of students.

4) Effects of History: No information given

5) Effects of Testing: No information given

6) Reactive effects of experimental arrangements:It would be best to replicate the results in other medical schools.

Post-Test Only Control GroupDesign

This design follows all the same steps as the classic pre-test/post-testdesign except that it omits the pre-test. There are many situationswhere a pre-test is impossible because the participants have already beenexposed to the treatment, or it would be too expensive or too time-consuming.For large enough groups, this design can control for most of the same threatsto internal and external validity as the classic controlled experimentaldesign. For example, it eliminates the threat to internal validityof pre-testing by eliminating the pre-test. It may also decreasethe problem of experimental mortality by shortening the length of the study(no pre-test).

For small groups, however, a pre-test is necessary. Also, a pre-testis necessary if the researcher wants to determine the exact amount of changeattributable to the independent variable alone.
Public administrators would like to be able to use experimental designsfor policy and program evaluation. Did a regional economic developmentpolicy bring more business to the economically depressed region?Did the Women-Infants-and-Children (WIC) program lower the rate of malnutritionin young children?

Unfortunately, it is difficult for public administrators to meet the requirementsof the classic controlled experimental design.

-It is difficult to conduct program evaluations in a laboratory,where other influences can be controlled.

-It is difficult to achieve random assignment, due to political andethical concerns.

-Policies may not be specific as to what changes they intend to produce.

-Often funds or other resources for a large-scale research projectare lacking.

-Decision makers often operate on short time frames and cannot waitfor an experimental study to run its course.

-Research is not always begun before a program is implemented, andtherefore cannot take measurements of the "before" condition.

-Programs are not always implemented all at once, but rather gradually,over time, which may diminish their effects.

-Programs may not be implemented as originally intended, or may notproduce the effects that researchers are looking for.

-Small treatments with modest goals are more amenable to a controlledexperimental design than large scale social programs with ambitious goals.