Critical Issue: Monitoring the School Literacy Program
ISSUE: Teachers make many educational decisions about how reading and writing should be taught in the classroom. Some teachers make these decisions intuitively, but a systematic process of monitoring is more effective in promoting students' literacy development. When teachers monitor the school's literacy program, they track the reading progress of their students, the results of different teaching methods, and the value of various materials used. The results of their monitoring activities allow teachers to judge the value of their program--in other words, to evaluate its effectiveness--and to make necessary improvements.
Careful and continuous monitoring is an integral part of an effective literacy program because it enables teachers to determine the most effective strategies for teaching reading and writing and because it helps teachers address students' learning problems before these problems seriously affect student achievement. The careful planning of literacy monitoring involves addressing four questions: Why are we evaluating? What are we evaluating? How are we evaluating? How well have we evaluated? An additional question is implied: At what points along the way will the process be monitored?
OVERVIEW: Monitoring a school literacy program is a systematic process of examining students' reading progress and teachers' instructional strategies in order to improve students' reading and writing achievement. Any monitoring effort has three basic components: collecting information on a regular basis, analyzing and evaluating that information, and taking action to improve student performance (Richards, 1988). These components may be preceded by other activities, such as determining gaps in practice and articulating questions on which to focus the monitoring.
Monitoring is a continuous process. When teachers monitor the school's literacy program, they keep a watchful eye on students' achievement and successes in reading and writing activities. They collect literacy-focused assessment data, including standardized tests and alternative assessments (such as samples of students' work and observations of students' reading behavior and performance). They also look beyond assessment data to children's attitudes toward reading, comments from families, and any other information that sheds light on whether the goals of the literacy program are being met. Next, they review and analyze this information to determine the value of various elements of the literacy program and how well these elements foster students' success. Finally, they make adjustments in areas that need improvement. Then the cycle of monitoring begins again.
Effective monitoring practices are necessary for maintaining a quality literacy environment. "All teachers need ways to determine what students are learning and the progress they are making. This information provides the basis for making decisions, planning instructional activities and experiences, and distinguishing effective from ineffective procedures," states Cooper (1997, p. 513). Effective monitoring helps teachers take ownership of the teaching-learning process and enables them to implement new strategies to foster students' literacy growth.
When teachers articulate their literacy goals for students and the types of evidence they need to determine if students are successful in meeting those goals, teachers begin to build a foundation for the monitoring process. Monitoring a literacy program requires a process for tracking and evaluating current procedures and outcomes. Teachers need to know what is working, what isn't, and how well students' efforts are being supported. Program monitoring through the assessment, and later evaluation, of teaching strategies and student learning is essential. It aids informed decision making in the classroom and school, and it contributes to each teacher's learning.
Concerns in Monitoring Literacy Programs
Before the monitoring process can be started, teachers and administrators should be aware of some concerns in monitoring literacy programs. More efforts are made to evaluate reading programs than any other curriculum area. In spite of these efforts, the evaluation of reading remains shrouded in misunderstanding and misinterpretation. Attempts to evaluate reading programs and students' reading achievement often confuse the issue rather than clarify it. There are several reasons for this confusion.
First, evaluation often is set up as an afterthought. A plan is not in place to answer initial evaluation questions before the process of teaching reading begins in the classroom. Such evaluation questions include: Why are we evaluating? What are we evaluating? How are we evaluating? How well have we evaluated? Instead of being used as a process for judging and improving program quality, evaluation becomes something teachers or administrators are required to do in order to satisfy externally determined needs. When this situation occurs, teachers have no ownership of the evaluation plan or the generated data. The result is that often no one seems to know why the assessment information was gathered or how it can be used to improve instruction.
Second, the evaluation may be hampered by unclear objectives. If little attention has been devoted to why the evaluation is occurring, what is being evaluated may be unclear. Is the evaluation trying to determine how well a particular group of students can read? If so, what definition of reading is being used to determine success? And what will be the evidence that the students can read? Perhaps, alternatively, teachers are interested in how well the literacy program is succeeding and how well their students as a whole are doing. The evaluation of literacy programs should span more than individual children; instead, it addresses questions and judgments of the literacy program itself. The challenges, however, can remain the same: The definition of reading and the criteria for acceptable reading performance may not be clear. To be valid, evaluation demands a standard against which data or information can be judged. Without it, no judgment of worth can be made. It is impossible for teachers to determine whether the program is working if they have not determined what the program should do.
A third problem is confusion about the meaning of the term evaluation. Evaluation is the process used to determine the worth of something; it is an attempt to determine if some product, process, activity, or procedure is of value or is satisfactory. Some educators equate evaluation with assessment, although the terms are not synonymous. These terms have been used interchangeably for some years, yet their meanings are quite different. "Assessment is the process of gathering information about something (getting students to respond), and evaluation is the process of judging that information (judging students' responses) to determine how well individuals are achieving," notes Cooper (1997, p. 512). Evaluation decisions are based on assessment information and data. Testing, often defined more narrowly as standardized testing, is but one way to gather data. When deciding how they will evaluate the literacy program, teachers should consider alternative assessment strategies, such as portfolios, projects, teacher observations, surveys, and interviews. As teachers clarify their goals for students (based on their definition of reading) and specify what evidence they need to determine whether the students are meeting these goals, they realize that there are many ways to gather evidence and use it to make a judgment about the literacy program.
When educators miss the opportunity to answer the first three questions--Why are we evaluating? What are we evaluating? How are we evaluating?--they probably will be unable to respond positively to the fourth question, How well have we evaluated? The data or information produced may be insufficient or inappropriate to use in evaluating the reading program and, if published, could be misinterpreted by the community. (For additional information on assessment reporting, refer to the Critical Issue "Reporting Assessment Results.") For example, results of a single test that bears no relationship to the literacy program or results of a homegrown survey of attitudes with unknown reliability and questionable validity are inappropriate for judging the overall effectiveness of a literacy program. How well the task is carried out really depends on the preparation involved in planning the evaluation.
Four preliminary activities are necessary before the formal monitoring process can begin. First, the school establishes a literacy committee that can direct the monitoring efforts. Second, the faculty is informed by research and best practice about literacy, evaluation, and assessment. Third, if gaps between desired practice and ideal practice are not known, the school conducts a needs assessment. Finally, the faculty formulates questions to focus the monitoring. These activities are followed by the formal monitoring activities: collecting information, organizing and analyzing information, and taking action.
Establishing a Literacy Committee. One of the first activities in monitoring a literacy program is establishing a literacy committee whose ultimate goal is to improve the school's literacy program (Patty, Maschof, & Ransom, 1996). This committee is made up of teachers, administrators, and interested parents and community members, but the majority should be school personnel (Patty, Maschof, & Ransom, 1996). The committee is responsible for planning and initiating the monitoring process, gathering data and resources, coordinating with teachers who are monitoring the literacy program in their individual classes, leading the decision-making process to chart a new direction for the literacy program, and communicating the findings to school personnel and community members.
Mary Foertsch, former director of the Center for Literacy and Assessment at North Central Regional Educational Laboratory, discusses the formation of the school's literacy committee and its important tasks [101 K audio file]. Excerpted from a videotaped interview with Mary Foertsch (North Central Regional Educational Laboratory, 2000). A text transcript is available.
Building a Knowledge Base. The literacy committee is responsible for building a knowledge base about literacy, evaluation, and assessment. This knowledge base will be used by the committee and by the rest of the faculty. The committee may compile information about reading research, best practices in reading instruction, different ways to monitor children's reading progress, and examples of schools that have model literacy programs. It may wish to determine the definition of reading that drives the school's literacy program and develop an awareness of characteristics of outstanding literacy programs and characteristics of literacy-rich classrooms. Also important is knowledge of literacy standards for students, such as the language arts standards for grades K-12 and the Continuum of Children's Development in Early Reading and Writing for children in preschool through second grade. Finally, the knowledge base should include information on the principles of effective literacy assessment as well as effective strategies for integrating assessment and instruction in ways that support learning and ensuring equity with alternative assessments.
Conducting a Needs Assessment. After building a knowledge base, the literacy committee may wish to conduct a needs assessment for the literacy program. Generally, a needs assessment is a means of examining the overall quality of a program, identifying strengths and weaknesses of the program, and recognizing the current needs of students and teachers.
The needs assessment for monitoring a literacy program is a little more focused. Its purpose is to pose questions that help teachers identify information they need in order to make instructional or curricular decisions. By answering these questions in terms of their own program, teachers first determine what information is valuable and then identify gaps between the ideal and reality. The discrepancy between some goal and the reality of a present situation is defined by Gottman and Clasen (1972) as a need. The needs assessment identifies these discrepancies and helps provide a focus for monitoring and improvement efforts.
The needs assessment attempts to identify whether teachers and administrators need additional information to analyze decisions currently being made about reading instruction. When educators define what information is needed, they also clarify the issues to be evaluated. A discrepancy between the information required to make professional decisions and the information that is available can be determined by asking a series of questions about educational decisions made in the school.
First are questions about who makes literacy decisions. Answering these questions helps teachers pinpoint responsibilities for making specific decisions, such as when children are ready for reading instruction, what reading curriculum will be used, and how children's reading progress will be communicated to parents.
Next are questions about information and processes currently used to make literacy decisions. Answering these questions enables teachers to determine what specific information about instructional practice and curriculum should be known and what information currently is known. For example, teachers' answers at the primary level might reveal the strategies and skills that children should acquire before beginning formal reading instruction, teachers' criteria for determining these skills, and the background and previous experiences that help prepare children for formal reading instruction. Teachers' answers to these questions also help the literacy committee and the faculty evaluate the worth of issues they are examining.
Finally, there are questions about external factors that affect children's reading achievement. Answering these questions reminds teachers to consider children's diverse backgrounds and needs. As part of the needs assessment, the literacy committee may wish to identify the external factors that affect the success of the literacy program. Factors related to students' cultural backgrounds, second-language learning, socioeconomic status, and special needs, as well as factors related to teacher effectiveness have been found to affect students' reading achievement (Allington & Cunningham, 1996; Au, 1993; Bialystock, 1991; Foertsch, 1998; Garcia, E., 1994; Heath & Mangiola, 1991; Snow, Burns, & Griffin, 1998). Identification of these factors helps ensure that all aspects of a literacy program are included in the monitoring process and that steps eventually will be taken to address weak areas.
Teachers' answers to all these questions help the literacy committee clarify the information currently used to make those decisions and the gaps in information that may exist. The level of information available becomes critical. Through this questioning procedure, the literacy committee can identify what available information is appropriate or necessary for making professional decisions about the literacy program and determine if that information currently is being used. If current information is different from what ideally is needed, there is a discrepancy. Any discrepancy indicates a need to collect better information and improve program monitoring.
The needs assessment may be conducted through questionnaires, surveys, or worksheets for school staff. The literacy committee is responsible for collecting the responses or data, assisting the faculty in analyzing and prioritizing the concerns, and helping teachers formulate questions based on those concerns to focus the monitoring.
Formulating Questions to Focus the Monitoring. After analyzing the needs assessment, the literacy committee works with teachers to formulate specific questions for the monitoring process. These questions help teachers determine an area of focus before they begin collecting data on children's literacy performance. The area of focus should involve teaching and learning and be within the control of the school (Mills, 1999; Calhoun, 1994). Farr and Prichard (1989) state, "Formulation of clear and explicit evaluation questions is the most important step in planning and implementing effective assessment of the reading program" (p. 164). When formulating questions, the literacy committee can follow rules of thumb for determining focus questions.
The focus questions are based on the gaps found in the needs assessment and should emphasize curricular and instructional issues. For example, the needs assessment may indicate that not enough information is available to determine if the primary literacy program is developmentally appropriate, or if the upper-elementary students have the skills and strategies required to make text-based inferences. Examining these needs will enable teachers to formulate specific evaluation questions on which to focus. In this situation, an evaluation question for the primary program might be: Do the activities we use with emergent readers in first grade help them meet the literacy goals established for grade-level achievement? For the upper-elementary program, an evaluation question might be: Does our instruction help the students make increasingly sophisticated connections between sentences in a passage and use evidence from the text to draw logical conclusions?
For each question that is formulated, teachers need to determine two types of information: what is and what should be. What is relates to what the students currently are able to do in the classroom. What should be relates to what the students should be able to do, according to a standard or expectation of student performance. To be successful, the monitoring process should have a standard against which data or information can be measured. Without a standard, it is impossible for teachers to determine how well students are achieving.
After the focus questions have been determined, it is the teachers' turn to drive the monitoring process. Teachers now assess the literacy curriculum, instructional practices, and students' literacy performance. First, they determine what information is valuable in answering the focus questions. Then, they collect data and pertinent information in their classrooms. Next, they organize and analyze that data (with the assistance of the literacy committee). Finally, they take action to improve students' literacy achievement.
Collecting Information. Data collection must relate specifically to the questions that were formulated to focus the monitoring (Farr & Pritchard, 1989; Padak & Padak, 1994). The challenge is ensuring that data is collected on "enough important variables to be certain all major and relevant considerations are included in the evaluation," note Worthen and Sanders (1987, p. 233). The information should provide a clear picture of how the school environment affects student learning; it also should help identify factors affecting the program that are within the control of the school (Calhoun, 1994).
Teachers must identify the information needed to respond to the questions. They determine what concrete and anecdotal information will be collected about students and what information will be collected about the environment (Calhoun, 1994). For example, if the needs analysis indicates that a focus of the monitoring process should be children's readiness to read and how to begin formal instruction, teachers can collect information on readiness skills specified in the curriculum and standards being used, experiences that children have had with literacy prior to entering the class, and approaches currently being used to determine children's literacy skills and awareness levels. "Not all information needs can be identified in advance, but surely many can and should be. Identifying information needs at the outset ... offer[s] the advantage of following a plan for collecting information that continues to be viewed as important throughout the evaluation," state Worthen and Sanders (1987, p. 232).
Three sources of information help provide an accurate picture of the school literacy program: existing archival sources of data (such as student grades and standardized test results), conventional sources of data (such as samples of students' work, observations, and anecdotal records), and inventive sources of data (such as portfolios, exhibits, and videotapes). Teachers may wish to collect archival data first and then move to conventional and inventive data sources. The use of multiple sources of data provides a more complete picture of students and the learning environment than does a single source (Calhoun, 1994; Padak & Padak, 1994). Because a single data source is limited, it may provide inaccurate information and lead to false conclusions (Calhoun, 1994; Worthen & Sanders, 1987). These sources can be plotted along a continuum, with existing sources at one end and inventive sources at the other. As one moves across the continuum, the collected information gains depth; as a result, additional time is necessary to analyze and interpret the data (Calhoun, 1994).
Teachers can keep in mind that data collection should occur regularly, should be as simple as possible, and should promote collective ownership of data. Following these suggestions will improve the quality of data collected. Also, the data-collection process may need to be monitored until procedures for collecting become established among the teachers who are collecting the data (Calhoun, 1994).
Organizing and Analyzing Information. After collecting the data, teachers work with the literacy committe to organize it in a clear format. The manner of organization depends upon whether the data is quantitative or qualitative. Organization of quantitative data involves the use of charts or tables to provide clear information for short-term and long-term decision making. Organization of qualitative data involves working with words rather than numbers, so this data can be arranged in paragraph descriptions or summaries. If there is a great deal of qualitative information, it initially can be sorted into piles according to questions. Data not directly related to questions should be set aside (Padak & Padak, 1994). Some qualitative information may need to be reread or revisited several times (Mills, 1999).
If various staff members have collected information relating to a specific question, the data should be shared when it is being organized. For example, classroom teachers may collect information on students' literacy experiences outside of school, office staff may collect information on attendance or grades, and the reading specialist may review the reading curriculum to determine skills and concepts to be taught.
After organizing the data, teachers and the literacy committee analyze it. Using guidelines for data analysis, they study the data and try to determine patterns, trends, and areas of strength and weakness. They also compare data from various sources and try to formulate conclusions, such as possible causes or reasons for students' performance. Calhoun (1994) states, "Time needs to be provided for the faculty to study, discuss, and question the data as a professional collective; to determine priority area(s) for action; and to decide what can be celebrated" (p. 80).
Data can be organized and analyzed on three levels: school, grade level, and classroom, but "it is the grade- and class-level data (and for the teacher it is individual student data) that provide the primary diagnostic data needed to guide schoolwide collective action," notes Calhoun (1994, p. 81). An example of data analysis provides a glimpse of one school's efforts at analyzing and interpreting data.
Taking Action. The final step of monitoring a literacy program is taking action to improve student performance. During this step, teachers apply what they have learned and develop an action plan. The information provided through data analysis and the information collected for the knowledge base help teachers focus collective action on those options most likely to yield positive benefits for students. The action plan should include both short-term and long-term actions.
The selected actions are influenced by the school's literacy goals for all students and should be based on best teaching practices, research, and current literature on literacy. Actions should focus on curriculum and instruction and should be specific. Because teachers have been involved in the ongoing monitoring process, they understand the importance of taking action and are willing to implement specific changes into their daily lessons and teaching strategies. The action plan needs to include techniques for studying implementation of new curricula and methods of instruction (Calhoun, 1994). Tracking the effectiveness of the selected actions continues the monitoring cycle.
Ongoing monitoring of the school literacy program is an effective means to ensure the continuing literacy development of students. Use of the Literacy Program Evaluation Tool, which outlines the steps in the monitoring process, provides guidelines for building-level personnel to examine students' reading progress and teachers' instructional strategies. When teachers are involved in the monitoring process, they stay informed about students' literacy achievements in relation to school goals. They also are more willing to implement improvement strategies in the classroom. The end result is a more effective literacy program for all students.
ACTION OPTIONS: The literacy committee and classroom teachers can take the following steps to monitor and evaluate the school's literacy program.
IMPLEMENTATION PITFALLS: Even though monitoring the school's literacy program is of critical importance, some teachers who are faced with the day-to-day workload of preparing lessons, instructing, keeping records, correcting papers, and disciplining may consider ongoing monitoring to be a luxury for which they do not have time. The principal or superintendent should emphasize the importance of monitoring the literacy program and ensure that teachers have adequate time to collect and interpret data. (For additional information on providing time for teachers, refer to the Critical Issue "Finding Time for Professional Development.")
Some educators may suggest using data already available instead of collecting new data. In such situations, the questions for monitoring can be manipulated to focus only on data that is readily available. Although this approach may appear to be time-efficient, it will not yield reliable or valid results (Worthen & Sanders, 1987) and will not be as helpful in meeting the goal of improved student learning. Teachers need to keep in mind the goals of the monitoring process as they collect new data.
Sometimes decisions are based on limited data or no data. Instead of making a premature analysis, educators need to be patient and allow the evaluation process to fully unfold (Mills, 1999). It also is important for educators to avoid over-reliance on perceptual data, such as surveys and attitudinal measures. Although the use of surveys and questionnaires provides some useful information, the most important information is based on actual student performance in the classroom. Calhoun (1994) advises educators to "collect behavioral data whenever possible ... and keep your initial surveys short and of high quality" (p. 56).
The school may have policies that restrict data collection or limit data-collection procedures. Such restrictions may occur in the form of teacher contractual agreements, state requirements and approvals, rights of privacy, or district policy. If restrictions are identified early in the planning process, accommodations may be worked out upfront and save valuable time later (Worthen & Sanders, 1987).
Some teachers and principals may not know how to go about collecting data, or they may have set ways of analyzing and interpreting the data (Patty, Maschof, & Ransom, 1996). This situation can be addressed by providing staff with professional development on data-collection strategies so they understand the process of monitoring and the value of data collected. Professional development activities may take the form of reviewing monitoring plans from other schools and discussing current literature on monitoring programs to determine which methods will work best for the school's literacy program.
Teachers and administrators may express reluctance to begin program monitoring simply because they do not know where to begin. Although they sense why they should be monitoring student progress and understanding assessment strategies, they may be unable to translate philosophy into action. They also may be unsure of how to determine a focus that would make the monitoring process meaningful and manageable. Under those circumstances, a needs assessment can play an important role in getting started.
Mills (1999) notes that obstacles such as "lack of resources, resistance to change, reluctance to interfere with others' professional practices, reluctance to admit difficult truths, and the challenges of finding a forum to share what you have learned" may impede teachers from taking action to effect change.
DIFFERENT POINTS OF VIEW: Some administrators and educators lightly regard program monitoring because they mistrust data. Some have been stung by data gathered by someone else. Many are confused by the jargon of the evaluator. Others are discouraged because past results often seem inconclusive. Still others insist that because teaching reading is more an art than a science, it cannot be scientifically measured.
Some parents and educators believe that standardized tests are the most valid representation of the value of a literacy program and the bottom line in measuring students' reading performance. They may not consider alternative assessment data to be valuable to collect and evaluate during the monitoring process.
Center for the Improvement of Early Reading Achievement (CIERA)
University of Michigan, School of Education
610 E. University Ave., Room 1600 SEB
Ann Arbor, MI 48109-1259
(734) 647-6940; fax (734) 763-1229
Contact: Elfrieda Hiebert, Director
ERIC Clearinghouse on Reading, English, and Communication
Smith Research Center, Suite 150
Bloomington, IN 47408-2698
(800) 759-4723 or (812) 855-5847; fax (812) 855-4220
International Reading Association
800 Barksdale Road
P.O. Box 8139
Newark, DE 19714-8139
(302) 731-1600; fax: (302) 731-1057
National Center for Research on Evaluation, Standards, and Student Testing
University of California at Los Angeles
Graduate School of Education and Information Studies
300 Charles E. Young Drive North
Los Angeles, CA 90095-1522
(310) 206-1532; fax 310-825-3883
National Council of Teachers of English
1111 West Kenyon Road
Urbana, IL 61801
(2l1) 328-3870 or (800) 369-6283; fax (217) 328-9645
National Institute for Literacy
1775 I St. N.W., Suite 730
Washington, DC 20006-2401
(202) 233-2025; fax (202) 233-2050
Contact: Andrew Hartman, Director
National Research Center on English Learning and Achievement
Albany Institute for Research in Education
University at Albany, ED-B9
1400 Washington Ave.
Albany, NY 12222
(518) 442-5026; fax (518) 442-5944
Contact: Janet Angelis, Associate Director
This Critical Issue was written by by Debra Johnson, a freelance writer who also is a multiage resource teacher at Lincoln School in Mundelein, Illinois, and Mary Foertsch, former director of the Center for Literacy and Assessment at North Central Regional Educational Laboratory.
Date posted: 2000
Copyright © North Central Regional Educational Laboratory. All rights
Disclaimer and copyright information.