Evaluation

A 1993 review conducted by the General Accounting Office of 10 school-linked human service efforts found impact evaluations for six. Of these, five reported positive effects on student dropout rates, absenteeism, and academic achievement. Because so few evaluations have been done, many questions still remain about the costs and benefits of various types of school linked programs and which are most appropriate in given situations (U.S. General Accounting Office, 1993).

Evaluation is difficult and results are often ambiguous for several reasons:

• Conventional, experimental research designs are not well-suited to collaborative initiatives. Interventions are seldom uniform. The midcourse correction and fine-tuning that are part of an effective initiative create a constantly moving target for the evaluator. Similar interventions at various sites within the same community often look very different. And, for practical and ethical reasons it is seldom possible to establish the requisite matched control groups. Initiatives are often started in schools and/or neighborhoods that by virtue of their high incidence of overlapping problems are unique. Sufficiently similar sites are hard to find. Nor is creating control groups from among at-risk students at the same intervention site generally considered acceptable given how much all similarly situated students need what is being provided.

• Collaborative initiatives operate at many levels (individual, family, community, institutional and cross-system). Evaluators are hard-pressed to sort out effects within and across each level. In large part this is because the social science research tools needed to manage this complexity are not well developed.

• The time frame of most traditional evaluation and the patience of funders and politicians is often too short to show a measurable impact on large outcomes.

For a comprehensive discussion of issues surrounding the evaluation of comprehensive initiatives, refer to Connell, Kubish, Schorr, and Weiss (1995). The Center for the Study of Social Policy (1995) describes how these factors shaped the evaluation of a large foundation-funded effort to integrate services in five cities. Useful work has also been done to begin designing alternative outcome evaluation and accountability frameworks.

Young, Gardner, Coley, Schorr, and Bruner (1994) have developed the six-level framework presented below. It starts at the family-worker level and systematically moves toward community-level impacts.

A. Service Penetration

• What are the characteristics of families most likely to benefit from the comprehensive service strategy being developed?

• How do these correspond to the families who are receiving services and supports through that strategy?

• What proportion of families within the community with these characteristics are being reached by the service strategy?

B. Family Engagement

• To what extent do families initially reached by the service strategy 'join' and begin participation?

• How long do they participate?

• What are their reasons for ceasing to participate?

C. Family Growth

• What growth occurs for families during their time of participation?

• To what extent do families attribute growth and achievement of goals to some support provided by the service strategy?

• For what proportion of the families do the workers feel needed change has occurred, and in what ways has the service contributed to that change?

D. Community Embeddedness

• What visibility do the services/strategy have within the community?

• What proportion of families in the community are aware of the services and, whether or not they participate, what is their orientation to them?

• How do other community institutions and networks of community support view the services, and to what extent do they see them fulfilling a need community role?

• To what extent has the services/strategy helped community residents define community need and set community goals?

E. System Response, Climate for Reform, and Change

• To what extent have the providers of comprehensive services identified and advocated for changes and new responses within other public service systems?

• What changes have (and have not) occurred within other service systems that may be attributed to this advocacy?

• To what extent have community residents identified and advocated for changes and new responses from other public service systems and from society as a whole?

• What changes have (and have not) occurred within other service systems and within society as a result of this advocacy?

F. Community-Wide Family Well-Being

• What changes in indicators of family well-being on a community-wide (both within and across neighborhoods within a community) basis have occurred since the initiation of new service strategies?

• Given the penetration of the service strategy and the extent to which other services have been able to respond to identified family needs, how much impact could the strategy realistically be expected to make?

• To what extent do community residents attribute any changes in the quality of life within the neighborhood and community to the changes in services?" (p. 37)

  References

  info@ncrel.org
  Copyright © North Central Regional Educational Laboratory. All rights reserved.
  Disclaimer and copyright information.