|
Comparison Group Issues:
How to evaluate comparison group study designs
Researchers need to select the comparison and control groups so
that they are as equivalent as possible to ensure that any differences
between the groups are caused by the use of the software and not
to some other outside factor.
There are three key ways that researchers select groups for a
research study: random assignment, matching,
and the use of generic comparison groups.
Matching and the use of generic comparison groups are the most common
procedures used. Random assignment is the most reliable way to create
equivalent groups. The next most reliable procedure would be matching.
The least reliable procedure is the use of generic comparison groups.
Watch out for factors that could weaken the distinctions between
the treatment and comparison groups, such as having students in
the comparison group who use the technology outside of school. If
this issue is not adequately addressed in the study, the results
may be misleading. In many cases this may lead to spurious finding
of "ineffectiveness."
Where can you find this information?
Information about how the treatment and comparison groups were selected
and how comparable they are is usually available in the Methods
section of a published report.
|
Example
Study
|
Methods
Since students were already participating
in Computer Reading Fun in the district when the evaluation
began, researchers needed to select a matched sample
of nonparticipating students to create a comparison
group. The students from nonparticipating schools were
drawn from schools that matched participating schools
in prior standardized test achievement and socioeconomic
and demographic composition.
|
|
[top]
Random Assignment. The use of the random assignment
procedure to create equivalent groups is based on the statistical
assumption that if students are assigned to the treatment and comparison
groups purely at random, students with similar abilities, motivation
levels, and family backgrounds will have an equal chance of being
assigned to either the treatment or comparison group. If the two
groups are large enough, researchers can assume that the different
types of students or classrooms will be evenly distributed among
the groups. When random assignment is used properly, it is the most
effective procedure for creating equivalent treatment and comparison
groups. If the groups are equivalent at the start of the study,
we can be more confident that any difference in student performance
measured at the end of the study was caused by the use of the software
and not some other factor.
However, if differential
sample attrition occurs after random assignment is completed,
you can no longer be assured of the comparability of the two groups.
If the groups are no longer comparable, it will be difficult to
attribute any differences in student performance to the use of the
software alone.
[top]
Matching. Unfortunately, it is often difficult
in an educational setting to carry out random assignment because
school district officials might decline to withhold new technology
from some classrooms or schools. As an alternative, researchers
try to make sure that the students in comparison and treatment groups
are as similar as possible. This type of comparison group is called
a matched comparison group. Students in the groups may be matched
according to such characteristics as their ethnicity, grade level,
academic achievement, motivation, or attitudes toward school. In
many studies, the comparison groups used are other classrooms in
the school or schools in the district that did not use the technology
but are similar to the treatment group in average test scores and
other student characteristics. Unfortunately, researchers can never
be sure that they have compared the groups on all the important
variables that might influence the outcome of a study. However,
the use of careful matching procedures to create equivalent groups
is better than no matching at all.
[top]
Generic Comparison Group.
Sometimes researchers who have no comparison groups available to
them try to compare the change in a treatment group's performance
to the change reported for the "average" student or school
in the district, state, or nation. This average group is called
a generic comparison group. The use of generic comparison groups
or "controls" is almost always problematic because the
sample of students or schools participating in the intervention
might be different in important ways from the "average"
school or student in the district, state, or nation that is used
as the comparison case. These differences make it extremely difficult
to attribute differences in student performance to the use of the
software alone and not to some other outside factor. For this reason,
the use of generic comparison groups is the least reliable procedure
to create equivalent treatment and comparison groups.
[top]
|
