NCREL's Learning Point (Spring 2003)

Contents | Previous | Next

Digging Out

How to Avoid Getting Buried Under a Mountain of Research

By Danielle Carnahan and Michele Fitzpatrick
Reprinted with permission of the National Staff Development Council. Originally appeared in JSD, Spring 2003 (Vol. 24, No. 2) as “Don't Get Buried Under a Mountain of Research.”

Consider this scenario: It is school improvement planning time—again. While preparing a professional development plan based on literacy, you begin an intense search for articles and information that you can use to align this plan with both your state and national goals. A priority in this search is material that will meet the Reading First stipulations of the No Child Left Behind Act of 2001, the $900 million program aimed at ensuring that the nation's children are proficient readers by the end of third grade. As you begin digging into the pile of articles you have accumulated, the sheer volume of information seems overwhelming. Where do you start?

Strategies to find the information you need

Becoming a sophisticated consumer of research on literacy is a continuing process, not a one-time accomplishment. It begins—as does any journey into new territory—with adjustments and attempts to comprehend new information that may seem a foreign language. Then it requires verbalizing what you don't know, asking questions. Finally, it requires a keen sense of inquiry, sniffing out the best sources of information to address those questions.

Here are some practical tips for your journey of applying the right type of research to translating research into classroom practice:

1. Savvy readers comprehend research language.
   First, learn the core definition of scientifically based research, which appears in the No Child Left Behind (2002) legislation, under Reading First (Title I, Part B, Subpart 1, Section 1208):

   The term “scientifically based reading research” means research that:

   1. Applies vigorous, systematic, and objective procedures to obtain valid knowledge relevant to reading development, reading instruction, and reading difficulties; and
   2. Includes research that:
      1. Employs systematic, empirical methods that draw on observation or experiment;
      2. Involves rigorous data analyses that are adequate to test the stated hypotheses and justify the general conditions drawn;
      3. Relies on measurements or observational methods that provide reliable and valid data across evaluators and observers and across multiple measurements and observations; and
      4. Has been accepted by a peer-reviewed journal or approved by a panel of independent experts through a comparably rigorous, objective, and scientific review.

   Within and related to this definition are a number of terms that warrant further translation:

   Observational methods: Methods researchers use to document events, behaviors, and surroundings.

   Data analyses: Using the artifacts of instruction to help see more clearly where students are, where they are going, and what patterns are emerging.

   Hypothesis: A statement of prediction of the results of a research project.

   Professional wisdom: “The judgment that individuals acquire through experience...Increased professional wisdom is reflected in numerous ways, including the effective identification and incorporation of local circumstances into instruction” (Whitehurst, 2002).

   Empirical evidence: “Scientifically based research from fields such as psychology, sociology, economics, and neuroscience, and especially from research in educational settings...Objective measure of performance used to compare, evaluate, and monitor progress” (Whitehurst, 2002).

   Evidence-based education: ”The integration of professional wisdom with the best available empirical evidence in making decisions about how to deliver instruction” (Whitehurst, 2002).

2. Sophisticated readers of research ask the right questions.
   A dozen years ago, S. Jay Samuels and Alan E. Farstrup offered an insight about research that remains valid today. They wrote, “Research is not a collection of ready-made answers to instruction-related questions, waiting to be claimed by eager and trusting teachers. It is a resource that can provide direction and substance for making instructional decisions when it is approached with purpose and caution” (Farstrup & Samuels, 1992, p. 1).

   As consumers of research, educators must critically inspect materials that claim to be research based. Educators must bring to the table qualities that enhance this process—qualities such as a well-developed sense of inquiry, experience in the field, and commitment to authentic learning. Evaluating research as a smart consumer means knowing how to interpret the claims made related to being “scientifically based.” When reviewing materials that purport to provide research-based evidence, take the following steps:

   • First, determine what kind of document it is. Generally, there are three categories of documents:
     • How-to documents, which describe, step-by-step, the process of implementing a strategy or program.
     • Descriptive documents, which generally describe the unqualified success of one program or another. Such documents may include a scenario, individual program, initiative, curriculum, strategy, plan, or method.
     • Research documents, which generally include some or all of the following: literature review, results of a study or survey, thesis supported by evidence. Such documents likely will include graphs, tables, charts, statistics, or other data to support the thesis.

     As informative as how-to, descriptive, and some research documents are, a closer look is needed to interpret whether the evidence provided is sufficient and valid, and whether the evidence is transferable to your classroom or school. Sound research should be able to be used in the classroom to assist with curriculum, instruction, and assessment.

     Feuer, Towne, and Shavelson (2002) state,“No method is good, bad, scientific, or unscientific in itself: Rather, it is the appropriate application of method to a particular problem that enables judgments about scientific quality” (p. 8).

   • Second, ask these questions:
     • Who conducted this research? Does the organization or individual possess credible background to address the research topic with an authoritative voice? Is there reason to believe that the organization would have any bias regarding the topic that might make the research results questionable?
     • What is being researched? Is the research topic focused sufficiently so that the data gathered can support feasible results? Is the information interpreted without bias?
     • When was the research conducted? Classic research studies can have as much validity as recent research studies. However, the timeliness of the results is important, and it should influence how the reader applies the results of the research. If the research is brand new, does it pay attention to the studies and findings that came before?
     • Where was the research done? Is the number of individuals involved in the study enough for a credible measure, given the topic studied? Does the research cover people and places like your classroom or school?
     • Why was the research conducted? Remember: context, context, context. If the research was conducted to prove the effectiveness of a program, or sponsored by persons who could benefit from positive results, this reason needs to be taken into consideration. Is there any other research that is relevant to this piece? If so, are the researchers speaking with the same voice? If not, why do the pieces disagree? Research is a community enterprise. Communal work and expertise is needed to validate findings. Research that provides answers to questions should not be carried out in isolation, but in a community of inquiry.
     • How was the data gathered and analyzed? Is the methodology sound? What process was used to develop appropriate questions and to determine how to organize the data and how to analyze the data? Is the solution connected to the research? Is it clear that the instrument or intervention made the changes? Again, there are resources that can help you with this.
     • Hmmm? Trust your own instincts. Remember, you bring useful perspective and knowledge to the table. If the research seems tainted or suspect, that is reason to pause and ask, “Why?”
3. Inquisitive readers find reliable information.
   The search for reliable sources begins with taking a close look at the existing research base. Credible sources are very important. Many of the same questions suggested to examine research also can be asked of information sources.

   If you are beginning to investigate a specific content area, such as reading, here are some supplemental questions: What information can be found? What information is missing? What information is debated within the community? If you ask yourself these questions, you will begin to feel comfortable that you have identified the most current and relevant research.

   Starting Points:

   • The newly established What Works Clearinghouse (www.w-w-c.org) is a Web site that helps educators make choices based on scientifically based research. The site, sponsored by the U.S. Department of Education, is being updated continuously and will offer information on various topics. (Please see the article Get REAL.)
   • Handbooks of research are available in many content areas. These publications are typically edited by the leading scholars of the discipline, and they feature powerful and relevant research.
   • Scholarly journals typically have a review process for material published. The complexity and thoroughness of this process varies from journal to journal. The description of these processes can be found in the journals themselves. The articles published in journals typically offer descriptions of new research and the studies' results.
   • A list of “Places to Begin Building Knowledge” organized by topic appears at the bottom of this article and is a suggested starting point.
4. Reading professionals take action.
   Scientifically based reading research provides a framework through which we can continue to engage in investigation— and in instruction based on evidence. Begin to learn the language, ask the questions, and seek reliable sources for answers. Collect data to support or refute assumptions. As we bring more research into our schools and classrooms, this framework assists us in implementing new practices and helps us become more sophisticated consumers of literacy materials.

Educators, policymakers, and professional development providers are in the position—now more than ever—to contribute to the knowledge base and further change the landscape of American education. Although there are no quick steps for translating research into practice, it's time to begin the journey.

Collection of Reading Materials

State-Specific Reading First Committees. Under the new legislation, states submit applications to the federal government for funding. In these applications, the states must lay out a comprehensive plan that details program information, professional development information, and assessment information. Because each state's plan differs, educators must pay close attention to their state's specific plan. Check your state's Web site to see what resources are available.

NRP Report. The National Reading Panel report, Teaching Children to Read, is a meta-analysis of reading research. It offers specific information that fits the scientifically based reading research criteria. Both an executive summary and the full report are available online at www.nationalreadingpanel.org/Publications/summary.htm.

CIERA and NIFL Report. Put Reading First is a publication co-sponsored by the Center for Improvement of Early Childhood Reading Achievement (CIERA) and the National Institute for Literacy. It details specific instructional implications. Strategies are suggested for teaching the five essential elements of reading. This publication is available online at www.nifl.gov/partnershipforreading/publications/Cierra.pdf (Adobe^® Reader^® PDF).

Other Helpful Books. The International Reading Association (IRA) has published many helpful books. Two suggestions are:

• What Research Has to Say About Reading Instruction (3rd edition), edited by Alan E. Farstrup and S. Jay Samuels, is a collection of articles on research and reading instruction.
• Evidence-Based Reading Instruction—Putting the National Reading Panel Report to Work is a collection of articles published in various IRA journals.

Places to Begin Building Knowledge

Whitehurst, G. J. (2002). Evidence-based education [Slide presentation]. Retrieved December 20, 2020, from http://www.ed.gov/offices/OESE/SASA/eb/evidencebased.pdf (Adobe Reader PDF).

Contents | Previous | Next

Back to Top

Adobe Reader FAQ