Technology in Education

A Meta-Analysis of the Effectiveness of Teaching and Learning With Technology on Student Outcomes

Discussion

The results of this quantitative synthesis show a modest, positive effect of teaching and learning with technology on student outcomes. The mean effect size of .410 is higher than the median of other recent meta-analyses in the area of instructional technology in education (see Table 1). Furthermore, the findings from the present meta-analysis revealed no significant differences across the contextual categories of study quality, teaching, and technology characteristics. In other words, the results can be generalized across a wide variety of conditions that have been investigated as well as across student, school, and study characteristics.

Research Quality Issues

One of the most important issues related to teaching and learning with technology that needs to be addressed is the soundness of the research for the implementation and improvement of technology programs. First, there were few quantitative studies published in the last five years that included relevant data to permit a meta-analysis and calculation of effect sizes. Scientific journals that use independent peer review in deciding what research merits publication are generally considered to be the highest standard of research, yet much of the work in the field of teaching and learning with technology does not meet that standard. The lack of quality, refereed quantitative studies points to a serious problem of research in the field.

Second, there were few studies that used a randomized, experimental design. Only 25 percent of the studies included in the meta-analysis used randomized, experimental designs. Furthermore, it is somewhat surprising that there are still many recent articles published in technology journals that are merely descriptive in nature and just report anecdotes from "selected" teachers or students who enjoy using the technology application. Other published studies explicitly state that their work is "exploratory" in nature, which might explain why they do not report specific findings.

A final concern regarding the quality of research in the field pertains to the lack of details that were included in many of the published articles included in this meta-analysis. Many of the studies lacked the specificity that was needed for us (and potentially others) to code all of the teaching and technology characteristics that we were specifically interested in. About 20 percent of the studies, for example, did not even specify what software was being using in their study. Researchers and journal editors need to make sure that all the relevant details about the classroom processes (e.g., teaching and technology components) are included in articles. Without that explicit information, we will return to the past decades of research on instructional technology, where we were considered to be in a "black box" stage in which we had no idea why instructional technology was effective (Waxman & Bright, 1993).

Limitations of the Present Study

The present meta-analysis, like most others, has several limitations. First, meta-analysis findings are correlational in nature and, therefore, do not warrant strong causal inferences. Second, meta-analysts do not have experimental control over data that reduces the sensitivity of the analysis. Third, the overall findings from the meta-analysis often are limited by the quality of the primary studies, a problem we have previously discussed.

Another perceived limitation of this meta-analysis may be that we included only published articles in refereed journals. Our justification for doing that is threefold. First, one of the critical scientific principles of educational research is that "scientific studies do not contribute to a larger body of knowledge until they are widely disseminated and subjected to professional scrutiny by peers" (National Research Council, 2002, p. 5). In recent years, a growing number of educators and researchers have become concerned about the quality of work that is posted and disseminated on the Internet. The Committee on Scientific Principles for Education Research (National Research Council, 2002), for example, maintains that the "extent to which the principles of science are met in some electronically posted work is often unclear" (p. 72). In this era of evidence-based and scientifically based research, one of the critical characteristics of a study is that it is refereed (i.e., approved for publication) by a panel of independent reviewers (International Reading Association, 2002).

A second explanation of why we excluded Web-based reports is that they often are too broad in nature yet not specific enough to allow meaningful coding. For example, we carefully examined about 20 potential sources from the major national research labs, regional support services, policy institutes, and government institutions. Some of those reports were quantitative studies and even included effect sizes; but they covered, for example, statewide programs over a 10-year intervention period. This report and others clearly do not feature the singularity and clarity of focus that one needs for inclusion in a meta-analysis.

We excluded books, chapters, dissertations, conference proceedings, and technical reports because they are unevenly reviewed. Furthermore, a final explanation of why we excluded nonpublished reports is that there is some evidence that nonpublished Web-based reports in technology have dramatically higher effect sizes than published reports (Niemiec, Sikorski, & Walberg, 1996). Also, there is evidence that many Web-based technical reports are sponsored by agencies that have obvious conflict of interests associated with the results (Wilson, Floden, & Ferrini-Mundy, 2001). For example, in a recent meta-analysis investigating the use of technology to enhance connections between home and school, Penuel et al. (2002) examined the relationship between the researchers who conducted the evaluation studies included in the meta-analysis and the programs they evaluated. They found that in more than half of the studies, the researcher was hired to do the research by either the vendor or school district involved in the study.

A final limitation of our study relates to the recentness of the review. Although we were interested in recent applications of technology, a few of the studies included in the meta-analysis stated that their article was based on projects completed in the early 1990s. In other words, even though we chose only articles that were published within the last five years, some of those articles are still based on technology (i.e., software and hardware) that is nearly a decade old. Future research syntheses may want to include either more rigorous criteria to ensure that only recent technology projects are included or expand the criteria to include older studies in order to examine if there are any secular trends.

Previous | Contents | Next