Viewpoints Vol. 11

The Challenge and Hope of
Scientifically Based Research

By Margaret A. Trybus, Ed.D.
Assistant Superintendent, Curriculum and Instruction, Thornton Fractional Township High School District 215

Previous | Contents | Next

Teachers and administrators alike are challenged by the No Child Left Behind Act of 2001 to incorporate scientifically based research (SBR) into their decision making for programs and practices that will improve student achievement. Educational researchers also are challenged to produce studies that are faithful to scientifically based principles. Now more than ever before, practitioners and researchers need to link their efforts to address student learning as a result of the No Child Left Behind legislation. The challenge is to base practice on rigorous evidence that specific programs will work to guide teaching and learning and, at the same time, to understand the reality that this type of research is not readily available or understood by most administrators and classroom teachers. Finding new opportunities for educators, policymakers, and researchers to work together on behalf of schools affected by the new legislation is a challenge, and it instills hope that SBR will provide better direction and evidence that student achievement can improve.

This essay outlines the place of SBR in the No Child Left Behind Act of 2001, provides details about the mandate for SBR in the Comprehensive School Reform legislation, and explains the rationale and challenges for using SBR in making education decisions. It also provides important definitions and outlines tools for translating research into practice.

Prioritizing SBR

The No Child Left Behind Act redefines the federal role in K–12 education by stipulating that federally funded programs and practices must be grounded in SBR. Funding is determined in part by whether programs and practices have a basis in "scientific research" (Beghetto, 2003). The No Child Left Behind (NCLB) Act mentions the term scientifically based research 111 times, and SBR is mentioned extensively in Title I: "to promote school wide reform and ensure the access of children to effective, scientifically based instructional strategies and challenging academic content" (NCLB Act, 2002, Title I, Section 1001 [9]). Title I, the largest federally funded program for educationally disadvantaged children, requires both state and local education agencies to use SBR for targeted-assistance schools to strengthen the core academic program. It also calls for school improvement efforts that seek to "identify and implement professional development, instructional strategies, and methods of instruction that are based on scientifically based research and that have proven effective in addressing the specific instructional issues that caused the school to be identified for school improvement" (NCLB Act, 2002, Title I, Part A, Section 1116 [4] [B] [ii]).

All technical assistance—whether provided by a state agency, higher education, or a state-approved professional development provider—also must comply with the SBR mandate by guaranteeing that new curriculum, instructional strategies, or specific initiatives that promise to improve the learning of low-achieving students meet SBR requirements. Subparts of Title I also cite SBR under Reading First, Early Reading First, Even Start Literacy Program, Improving Literacy through School Libraries, and Comprehensive School Reform (NCLB Act, 2002). Whether funds are used for procuring instructional materials such as software or reading programs, or funds are used for establishing partnerships between professional development providers and local school agencies, all efforts need to be screened through the SBR mandate.

SBR in Comprehensive School Reform

The SBR legislation has posed a challenge to schools seeking to implement reform strategies through the Comprehensive School Reform (CSR) Program. With the availability of additional funding, schools can make a commitment to work with an outside provider who must demonstrate that a particular reform program has proven SBR results and is replicable to the school using CSR. This initiative has far-reaching implications since it addresses every aspect of the school: all grades and key subjects (primarily English and mathematics curricula and instructional practices), school management, parental involvement, community involvement, and school organization (Herman, 2002).

CSR is usually initiated when individual school improvement efforts are not successful and assessment of data indicates that students are not meeting standards. Making Good Choices: A Guide for Schools and Districts (Hassel, 1998) offers a process for selecting a CSR model that requires identifying two or three models to find the best match between the model provider and the local school needs. Since 2001 and the reauthorization of the Elementary and Secondary Education Act, the standard for being an approved CSR model, however, has been changed from using "innovative strategies and proven methods for student learning, teaching, and school management based on reliable research and effective practices" to a call for comprehensive reform programs that "employ proven strategies and proven methods for student learning, teaching, and school management that are based on scientifically based research and effective practices and have been replicated successfully in schools" (Comprehensive School Reform [CSR] Program Office, & Office of Elementary and Secondary Education, 2002, p. 2).

Of the 11 components schools must implement to receive CSR funding, two specifically require a high standard of SBR. One component that requires a high standard is the selection of a model that uses proven strategies and methods for learning, teaching, and school management based on SBR and effective practices, and that has been used successfully in multiple schools. Another component requires that an approved CSR model has "been found, through scientifically based research, to significantly improve the academic achievement of students… or has been found to have strong evidence that [it] will" (CSR Program Office & Office of Elementary and Secondary Education, 2002, p. 2). For the other nine components (see The 11 Components of a Comprehensive School Reform Program), the U.S. Department of Education recognizes "school leaders will need to rely on the best available empirical evidence and some degree of professional judgment in creating their programs" (CSR Program Office & Office of Elementary and Secondary Education, 2002, p. 3).

In 2000, the American Institutes for Research (AIR, 2000) published the Educators' Guide to School Reform, reviewing the research on 24 of the most promising CSR models in the country. Even though the research was limited, each model was analyzed against criteria including the research design, controls, validity, sample size, measures, outcomes, and duration of the study (Herman, 2002). The American Association of School Administrators (AASA, 2003) has provided an online summary of the guide for the 24 "whole-school," "comprehensive," or "schoolwide" approaches reviewed by AIR. This guide provides a review of the relative strengths of each approach in three areas:

• Evidence of positive effects on student achievement.
• Support provided to schools as they adopt the approaches.
• First-year adoption costs. (AASA, 2003)

This guide will help decision makers determine areas of strength and weakness, as well as the amount of evidence available to assess the effects of a model on student achievement.

The Logic of SBR

Traditionally, relying on anecdotal evidence and professional judgment has been the approach many schools have taken with respect to school improvement initiatives, especially those schools not engaged in CSR. Since the passage of the NCLB Act, schools must be more diligent in their use of research and alter their decision-making practices accordingly. According to Assistant Secretary Grover Whitehurst, we have to create a culture and practice that demands more from our educational system by using evidence-based practices in order to transform education in the same order of magnitude as in medicine and agriculture (U.S. Department of Education, 2002).

The sense that education is moving to evidence-based practices poses some concerns for school leaders who are skeptical that they will have access to appropriate research that is replicable and solid (Beghetto, 2003). Some fear that the "heart and soul" of education may be lost, and along with it the interpersonal relationships between administrators, teachers, and students. Not so according to Valerie Reyna (2002), former deputy of the Office of Educational Research and Improvement (now the Institute of Education Sciences), who emphatically believes that there is no dichotomy between science and values or science and emotion:

"Evidence does not determine our decision solely. It is not just the facts. It's the facts plus values. But without the facts, we might make the wrong decision, even based on our values. Because we don't know what's true and what's not true. The facts, the evidence is necessary to make decisions that affect students' lives, but it's not sufficient. But it is necessary. That is what we're promulgating, that, at least, it be part of the discussion so that we can base practice on it. So, we're talking about science with a human face, and that's a person." (p. 10)

As we bring scientific reasoning into educational decision making, it is critical that we balance hard evidence with human judgment (Feuer, 2002). Together policymakers and practicing educators will have to find ways to address the balance between research and practical wisdom (Shavelson & Towne, 2002). To make this happen will require building partnerships between researchers and practitioners to address both improving practice through research and improving research through a better understanding of what happens on a daily basis in schools (Towne, 2002).

Defining SBR

According to the NCLB Act (2002), the term scientifically based research:

"(A) means research that involves the application of rigorous, systematic, and objective procedures to obtain reliable and valid knowledge relevant to education activities and programs and

(B) includes research that—

1. employs systematic, empirical methods that draw on observation or
experiment;
2. involves rigorous data analyses that are adequate to test the stated hypotheses and justify the general conclusions drawn;
3. relies on measurements or observational methods that provide reliable and valid data across evaluators and observers, across multiple measurements and observations, and across studies by the same or different investigators;
4. is evaluated using experimental or quasi-experimental designs in which individuals, entities, programs, or activities are assigned to different conditions and with appropriate controls to evaluate the effects of the condition of interest, with a preference for random-assignment experiments, or other designs to the extent that those designs contain within-condition or across-condition controls;
5. ensures that experimental studies are presented in sufficient detail and clarity to allow for replication or, at a minimum, offer the opportunity to build systematically on their findings; and
6. has been accepted by a peer-reviewed journal or approved by a panel of independent experts through a comparably rigorous, objective, and scientific review." (Title IX, Part A, Section 9101 [37])

This definition is intended to encourage researchers to provide better and more useful evidence of what works and to challenge practitioners to make good decisions based on evidence (Feuer, 2002). The difficulty is that few studies of education programs meet this definition in its entirety.

In 2002, AIR introduced two standards against which education research can be judged. The gold standard is research that meets all the requirements of SBR; the silver standard is research that meets the requirements but does not employ random sampling (AIR, 2002). The institute's work, which was prepared for the U.S. Department of Education, also includes some guidelines that can be used by school staff and others to review education research. These guidelines include:

• The theoretical base of the program or practice, explaining specific goals followed by implementation activities.
• The evidence of effects, stating how the practice has demonstrated improved student learning.
• Implementation and replicability, explaining the degree to which the program has been successfully implemented in diverse settings. (AIR, 2002)

The following graphic, developed by the Iowa Department of Education, illustrates the potential range of methodologies used in education research.

Figure 1. Iowa Content Network: Continuum for Reviewing Research

From the Iowa Professional Development Model: Selecting Content
(www.state.ia.us/educate/ecese/tqt/tc/prodev/definitions.html)

Reprinted with permission of the Iowa Department of Education.

According to Slavin (2003), scientific research traditionally has played a relatively minor role in education reform since many innovative practices and programs are untested. When reform efforts fail, educators and policymakers move to implement a different set of innovations that also have untested claims, instead of adopting well-researched programs and practices that have been proven to work. Shifting to a new paradigm will mean changing practice to look more deeply to research-based programs rather than following a new trend.

"Unlike most other fields of scientific inquiry, education places extraordinary emphasis on the new and the novel. Believing that the most recent theory—at whatever level of research—is also the most important, education leaders may lose sight of the value of seminal research and proven practices." (Grossen, 1996, p. 22)

Both Congress and the U.S. Department of Education are hopeful that with the introduction of new research standards and federal mandates, "evidence- based reform" will become the norm and set an expectation for using rigorous, experimental research to justify programs and practices.

Broadening SBR Within NCLB

In addition to Comprehensive School Reform and Title I legislation, SBR also is cited in Title II, Preparing, Training and Recruiting High Quality Teachers and Principals; Title III, Language Instruction for Limited-English-Proficient and Immigrant Students; Title IV, 21st Century Schools; Title V, Promoting Informed Parental Choice and Innovative Programs; Title VI, Flexibility and Accountability; Title VII, Indian, Native Hawaiian, and Alaska Native Education; and Title IX, General Provisions (NCLB Act, 2002).

Each applicant for funding must demonstrate efforts to address SBR by describing how each activity will be based on a review of SBR and will be tied to evidence-based results. This has far-reaching implications for professional development in all content areas, teacher preparation programs in higher education, English language acquisition programs, safe and drug-free programs, parent involvement, mentoring programs, and all programs designed to address state and local student academic achievement standards (NCLB Act, 2002).

Because of the extent of this legislation, all schools will be affected—not just those that have been in CSR programs or identified as in need of improvement. Using SBR will require establishing a culture of inquiry regarding how decisions are made to improve student learning. Leadership will matter at both the system and school level, and must include teachers with high-quality professional learning to improve practice (Sparks, 2003). Infusing SBR into school culture will require enhanced professional learning to increase understanding of the meaning and usefulness inherent in compelling research to drive practice.

Learning a New Vocabulary

In order to become a critical consumer of research, one must understand the language and terminology of research. In addition to understanding the definition of SBR and various standards for research, practitioners need a basic vocabulary with which they can interpret and communicate research studies to one another, to parents, and even to students when appropriate. The following definitions are a few suggestions.

As one becomes familiar with the terminology and the concepts undergirding SBR, balancing the use of professional wisdom and empirical evidence will determine the extent to which evidence-based programs can be useful. Figure 2 illustrates the relationships between individual experience, consensus, scientifically based research, and empirical evidence in executing evidence-based education programs.

Figure 2. Evidence-Based Education

Source: Whitehurst, G. J. (2002). Evidence-based education (slide presentation).

A basic understanding of research terminology and methods will help practitioners develop skills to make better judgments regarding educational research. It will also help to develop an understanding of the work being done by professional panels, such as the What Works Clearinghouse, that are identifying appropriate research for replication. There is no doubt, however, that the movement to create a more evidence-based environment will grow as researchers work to meet the needs of thousands of schools throughout our country who are challenged to improve learning for all students. Educators will need to be critical in judging whether the claims made by research make sense based on research design and methodology coupled with practitioners' experience in the field (Carnahan & Fitzpatrick, 2003). The more practitioners can do to increase their understanding of the findings, the more the intent of the law can be met and improvements in education addressed.

Research Challenges

How will the work of educational researchers adjust to meet the "evidence-based standard" and at the same time recognize the need to increase the availability of research for practitioners? How will education research be guided by scientific methodologies, as in the field of medicine, yet not limit the advancement of the education research field? One issue is the extent to which education research can use a medical research model that implements a "randomized-control" approach. Using this approach, researchers randomly assign students to treatment and control groups in order to determine the effectiveness of what is being studied (Reyna, 2002; Lewis, 2003). Some claim this approach is too narrow a focus for education research and ultimately will limit growth in the field (Laitsch, 2003). There is also a concern that qualitative studies will be dismissed in favor of more sophisticated statistical analyses of measurements and deemed unworthy of serious consideration because they do not meet the SBR requirements (Davis, 2002; Flinders, 2003).

Over the last 20 years, researchers such as Eisner (1997) have worked to justify qualitative methods that have broadened the field of research by recognizing the need to include social sciences as well as arts and humanities as sources of research methodology. Some fear that the move to formalize quantitative research as the gold standard for education research may diminish the pursuit of qualitative forms of data. There are also issues around the difficulty of using randomized studies in public schools. Randomized studies often are more controversial since parents may be uncomfortable with researchers identifying students who will and who will not receive a particular treatment (Flinders, 2003; Shavelson & Towne, 2002). Some compromise may be necessary to consider quasi-experimental studies, that is, "studies that do not truly randomize sampling or assignments, but that try to approximate these procedures" (Flinders, 2003, p. 384). According to Valerie Reyna, quasi-experimental studies provide a lower level of evidence, but are "second best" to randomized experimental trials, which allow for more stringent statistical control of the variables. "It's not as good, but at least you know that something is probably true based on some form of evidence" (Reyna, 2002).

Founded in 1916, the American Educational Research Association (AERA, n.d.), "strives to improve the educational process by encouraging scholarly inquiry related to education." In response to the mandate for SBR, AERA (2003) adopted a resolution on January 26, 2003, which reasserted:

"that there are multiple components of quality research, including well specified theory, sound problem formulation, reliance on appropriate research designs and methods, and integrity in the conduct of research and the communication of research findings. A fundamental premise of scientific inquiry is that research questions should guide the selection of inquiry methods. …However, the Council of the Association expresses dismay that the Department of Education through its public statements and programs of funding is devoting singular attention to this one tool of science, jeopardizing a broader range of problems best addressed through other scientific methods. The Council urges the Department of Education to expand its current conception of scientifically-based research." (p. 1)

The hope that sound, rigorous educational research will reach practitioners raises the issue of "usability." Researchers and practitioners alike agree that "to be effective, education research needs to be both credible and usable" (Viadero, 2003). With the focus on scientific experimentation, there is a concern that federal funding will not be available to support other kinds of research that practitioners also may find useful. Research studies may become "too academic," not practical enough for replication and not in a language accessible to practitioners. Educators may "view those writings as inaccessible, arcane, and irrelevant to their everyday jobs" (Viadero, 2003). On the other hand, education research has been criticized for securing scant evidence of affecting practice and improving student achievement, and skepticism regarding the use of federal funds to support research and development efforts has existed for over a decade (Kaestle, 1993).

Access to Evidence-Based Research

Realizing both researchers' challenge to produce SBR, and practitioners' challenge to find and utilize SBR, the U.S. Department of Education, through the Institute of Education Sciences, has established the What Works Clearinghouse (WWC). It will become a resource to assist in educational decision making regarding scientifically based programs and practices (Beghetto, 2003). The WWC was created in August 2002 as a national project to provide the information decision makers need to make choices based on high-quality scientific research (WWC, 2002a). This will be done through Web-based databases that will contain specific reviews of "potentially replicable interventions (programs, products, and practices)" that have been identified by synthesizing the scientific evidence that indicates effectiveness in improving student outcomes (WWC, 2002a). Other components will include an educational approaches and policies registry for evidence-based research reviews, a test instruments registry, and an evaluator registry that will identify individuals and organizations that are willing to conduct quality evaluation of education interventions (What Works Clearinghouse, 2002a; Rentner, Chudowsky, Fagan, Gayler, Hamilton, & Kober, 2003). The WWC has an ambitious agenda, which includes the following:

• Establishing standards for research aiming to draw causal inferences and to establish the validity of tests and other assessments.
• Creating an independent group of technical advisors.
• Selecting topics to focus research.
• Conducting syntheses of research studies. (WWC, 2002b)

The standards, which were approved in June 2003, will be used to guide the WWC Evidence Reports, which will be prepared by teams of analysts who will create systematic reviews of evidence. "Each WWC Evidence Report will examine the effects of programs, practices, products, and policies that are designed to improve student outcomes within a topic area" (WWC, 2002c). The first-year topic areas will include:

• Interventions for beginning reading
• Curriculum-based interventions for increasing K–12 mathematics achievement
• Programs for preventing high school dropouts
• Programs for increasing adult literacy
• Peer-assisted learning in elementary schools: reading, mathematics, and science gains
• Interventions to reduce delinquent, disorderly, and violent behavior in middle and high schools
• Interventions for elementary school English language learners: increasing English language acquisition and academic achievement

Another feature is the What Works Clearinghouse's Cumulative Research Evidence Assessment Device (CREAD), which is meant "to provide an expression of the confidence with which a conclusion can be drawn about the existence of causal effects of an intervention based on an entire body of accumulated evidence" (WWC, 2003, p. 1). Using the evidence-base approach will allow for the review of multiple studies that address the same topic, thus increasing the confidence that an intervention had a large, small, or no effect. These are important issues for practitioners when determining the degree to which the program or practice is appropriate for a particular school and population of students.

Practitioners are seeking assistance with implementing quality research. The Association for Supervision and Curriculum Development (ASCD) recently began publishing Research Brief. This Web-based publication will help educators and policymakers translate high-quality research into usable decision-making tools. In addition, to support the use of evidence-based practices, ASCD has recently published two books: (1) What Works in Schools: Translating Research into Action and (2) Classroom Instruction that Works: Research-Based Strategies for Increasing Student Achievement. In Classroom Instruction That Works,researchers at Mid-continent Research for Education and Learning used a meta-analysis technique to analyze selected research studies to determine the effect of a particular intervention (Marzano, Pickering, & Pollock, 2001). With this methodology, the effect size—which was also equated to percentiles—determined the increase or decrease in achievement of the experimental group of students who were involved with a specific instructional technique. By identifying specific instructional strategies that have a high effect size, Marzano, Pickering, and Pollock were able to establish categories of strategies with which practitioners can identify. Even though the authors admit that not all instructional strategies will work well in all situations, they have presented useful tools for practitioners to consider when implementing instructional strategies intended to improve student achievement.

Categories of Effective Instructional Strategies

Identifying similarities and differences Summarizing and note taking Reinforcing effort and providing recognition Homework and practice Nonlinguistic representations Cooperative learning Setting objectives and providing feedback Generating and testing hypotheses Questions, cues, and advance organizers (Marzano, Pickering, & Pollock, 2001)

In What Works in Schools: Translating Research into Action, Marzano (2003) contends that 35 years of research should not be overlooked in determining guidance on effective schooling. His work synthesizes this research and claims that evidence exists to support three general factors that influence student academic achievement: school-level factors, teacher-level factors, and student-level factors (Marzano, 2003, p. 10). Marzano identifies a process for implementation of his school reform model, which takes these factors into consideration. He defines steps for schools to identify the elements of the model that are applicable and identifies actions to take and steps to implement those actions and to determine the effects of the actions. What Works in Schools is one way practitioners can think about applying research-based factors to the process of school reform.

Moving Forward

Is it possible to improve the education of children by following the guidance of scientific methodology? Can we help students read better, do mathematics better, and achieve better by focusing efforts on research that has been proven effective? Will research become the basis upon which practitioners make decisions so that the most effective practices will be used to help all children learn?

The challenge and hope of accepting, understanding, and using SBR is now part of educational reform and accountability measures to see that no child will be left behind. It will take a concerted effort on the part of all stakeholders—administrators, teachers, parents, researchers, business leaders, and policymakers—to move this vision into action. It will take both scientific and professional wisdom to create a knowledge base, as well as a belief system, that this educational mandate has the power to transform teaching and learning. It will require building a data-driven system in which teachers look for results and continually challenge themselves to make decisions for improvement based on the integration of meaningful and relevant data and research. It will require educators to let go of past practices that are no longer justifiable because they are not achieving identified goals. It will require risk taking to understand and apply research methods in classrooms so that practitioners will not only be guided by evidence, but also informed of their own results. It will require a new vision of staff development that seeks to empower educators to be the agents of change and to try new ideas that offer to improve student learning. It will challenge educators to defend school improvement decisions so that instructional practices are grounded in results. The impetus to move forward must come from the willingness to use SBR and apply it to classroom settings. After all, children deserve no less than access to that which is proven effective and that which offers hope that improvements in achievement can be made.

Previous | Contents | Next