A Meta-Analysis of the Effectiveness of Teaching and Learning With Technology on Student Outcomes

Previous | Contents | Next

Method

Search and Selection Criteria

A systematic search of research published from 1997 through 2003 investigating the effects of technology on student outcomes was conducted by accessing several sources. For this review, we used selection criteria and review methods that are similar to other recent major national reviews conducted in areas such as teacher preparation (Wilson, Floden, & Ferrini-Mundy, 2001) and reading (National Reading Panel, 2001).

Several criteria were established for inclusion in this synthesis. The synthesis included quantitative, experimental, and quasi-experimental research and evaluation studies that have been published in refereed journals during a six-year period (1997–2003). In order to be included, the study also needed to: (a) focus on teaching and learning with technology in K–12 classroom contexts where students and their teachers interact primarily face-to-face (> 50 percent of the time); (b) compare a technology group to a nontechnology comparison group, or compare the group at the beginning of the intervention (pretest) to a posttest measure; and (c) have reported statistical data (e.g., t tests or F tests) that allowed the calculation of effect sizes.

We identified studies by examining database searches, using relevant keywords, and searching the Education Resources Information Center (ERIC). We located additional studies by examining the reference lists of relevant literature reviews and reports. We specifically examined several major journals in the field of educational technology, such as:

In addition, other education journals such as American Educational Research Journal, Journal of Educational Research, British Journal of Educational Psychology, Journal of Education for Teaching, Educational Psychology, Educational Psychologist, Journal of Educational Psychology, Contemporary Educational Psychology, Teaching and Teacher Education, Learning and Instruction, Research in Education, and Elementary School Journal were examined. Also, several Web sites provided comprehensive lists of technology-based journals with links to journal Web sites. Some were links to specific journals, and others that were only print-based provided a fairly comprehensive index to their journals. Entering the keywords educational technology, evaluation, and instruction and research into a search engine (e.g., Metacrawler and Google) provided a number of other sites (e.g., dmoz.org) that were searched.

Certain types of studies and reports were excluded from the synthesis. Many studies were eliminated because they did not report the appropriate statistics necessary to calculate effect sizes. Some of these studies, for example, provided raw scores for a few "select" participants in the treatment group, but they did not report aggregate scores for both groups (i.e., experimental and control groups). Other studies were eliminated because students in the control group either had access to or used computers. There were many studies, for example, that used research designs where technology was held as a constant and comparisons were made between factors such as differential feedback or instructional approaches. Most of these studies, however, were eliminated because all the students in the control groups had access to and used technology.

Procedure

To calibrate the studies' results, or place them on a common scale, effect sizes were calculated. These effect sizes consist of the treatment group mean minus the control mean divided by the control standard deviation. Effect sizes can be considered a standardized estimate of where the treatment group stands in comparison with the control group distribution. In the case of articles examined for this study, a positive effect size indicated that the instructional technology group received higher (i.e., more desirable) scores than the control group. The formulas of Glass, McGaw, and Smith (1981) were employed for studies that did not report group means or standard deviations but contained F or t values, correlations, or other statistics from which effect sizes could be calculated.

For this synthesis, three investigators recorded 69 codable characteristics and other data for each of the 282 effect sizes from the 42 studies. The 69 categorical variables were employed as factors in an analysis of variance (ANOVA). Each investigator independently coded three studies from each of two investigators. The intercoder agreement for each

The coding categories are listed in the Appendix. The methodological threats to validity were adapted from Cook and Campbell (1979). Most of the technology characteristics were adapted from other meta-analyses in the area. The teaching variables were adapted from the Five Standards for Effective Pedagogy developed by the Center for Research on Education, Diversity, and Excellence (2002; see Dalton, 1998; Tharp, 1997). The five standards are: (1) Teachers and Students Producing Together (Joint Productive Activity), (2) Developing Language and Literacy Across the Curriculum (Language Development), (3) Making Meaning: Connecting School to Students' Lives (Contextualization), (4) Teaching Complex Thinking (Challenging Activities), and (5) Teaching Through Conversation (Instructional Conversation). These standards are based on the best theoretical and empirical knowledge in the field, and there is ample evidence that their use in classrooms may lead to dramatic improvements for the education of all students (Tharp, Estrada, Dalton, & Yamauchi, 2000).

The studies varied by the number of comparisons they reported. Therefore, those studies with a greater number of comparisons (e.g., those that reported separate results by ability level, sex, or race) would have weighted more heavily than others if each comparison had been given equal weight. To give all studies the same unit weight in the analysis, each comparison was weighted in inverse proportion to the number of comparisons in the study from which it was taken (i.e., 1/n where n = number of comparisons in the study). Each of the three comparisons of Michael (2001), for example, received a weight of .333. For studies in which multiple comparisons were made by the percentages of computer use or number of computers, the comparisons between the high and low categories were used to calculate the effect size. Most of the studies had multiple outcomes, but the only comparisons included were those that had the appropriate statistics to calculate effect sizes.

Previous | Contents | Next