Stopping the Evaluation Tail From Wagging the Prevention Dog

By Roger Chesswas
Associate Director, Evaluation

New visions for prevention are emerging. They suggest a significant shift in the way we think about young people, their inclination to use alcohol, tobacco, and other drugs (ATOD), and what prevention programs should look like. These visions stress a new definition of prevention, as explained in Judson Hixson's article, that focuses more on what we want young people to become, rather than simply what we don't want them to do. What we have emerging now is a prevention approach that focuses first on recognizing the basic needs of young people and then on what we need to develop in them and for them to address those needs. We are no longer talking about looking at problem behaviors, but rather their root causes. Experts have different names for this process: Bonnie Benard calls it human development; Karen Pittman, youth development.

These new comprehensive prevention approaches require more than just building the competencies of young people, however. They require policies, professional development, and broad-based parent and community involvement. Clearly, prevention must move beyond the walls of the school. For this evaluator, this is an exciting perspective and provides for many new evaluation challenges and opportunities.

Evaluation studies have traditionally focused (mostly through student self-report ATOD surveys) on trying to measure problem behaviors and personalities. We have become used to trying to connect prevention efforts with patterns of use obtained from these surveys. These measures have been used to justify the need for prevention programs and as the outcome or impact measures for prevention program effectiveness. Unfortunately, they tell us nothing about the real needs of young people or the root causes of the behaviors that students are reporting. In none of the evaluation studies I have read has anybody explained why and how the prevention programs being evaluated are directly related to student reports of changes in ATOD behaviors. There is no proof of any connection; therefore, how do we know that prevention had any effect? Youth and human development proponents present an alternative: focus less on problem behaviors and more on identifying and addressing the basic needs of youth and then developing the appropriate skills, knowledge, and competencies in youth to address those needs. What is so appealing about this approach is that it requires us to look at a completely different set of needs than student self-reports on behaviors and to connect programs to meet those needs. It also requires us to look not only at decreasing the likelihood of starting to use substances but also changing the conditions, reasons, and factors for use. The evaluation question then becomes: How well did programs and interventions address those needs? And, by implication, by meeting those needs have we stemmed the use of substances and changed the conditions and reasons for use?

The focus of prevention now becomes the whole child, the conditions and environments of young people, their basic and most fundamental needs, and ongoing development activity that is inextricably linked with education. To my mind, Karen Pittman in her work on youth development provides us with a new point of departure for prevention. Her new vision suggests that we look first at seven basic human needs fundamental for survival and healthy development (safety, belonging, self-worth, independence, closeness, competence and mastery, and self-awareness) by developing in all young people five basic sets of competencies (health/physical well-being, personal and social, cognitive and creative, vocational, and citizenship).

This view requires us to drastically alter our needs assessment and evaluation activities. We now have to look at actual and basic needs and the various contexts of those needs and not use student self-reports as proxies. Program evaluation will focus more on the ongoing development of the basic sets of competencies in conjunction with the more traditional and new forms of (academic) assessment found in schools. The portfolios of learning will now show not only academic progress but also the development of additional competencies designed to address basic needs for survival and healthy development.

This will require a number of different things of evaluation, including needs assessment. First, the focus will change:

1. Evaluators will become facilitators in discussions among community members, parents, and young people about what they want young people to become and be able to do.

2. We will need to learn the new content of the development-focused prevention (and, for that matter, education) paradigm--what is meant by the whole child, the environments of young people and their basic needs and key competencies.

3. We will need to get agreement about, or identify the different perspectives about, the "bigger and better" portrait of the development of young people and what they look like at certain stages of development.

4. We will need to reflect on current prevention practices and philosophies and see if they "fit" with the new visions of youth and youth development.

5. We will need to develop the new touchstone (what is ideal, desired) with which to examine what we find in schools and communities.

6. We are going to have to ask the tough questions about what to keep, what to start doing because it's needed, and what to stop doing because it does not fit.

Second, evaluation will have to change:

1. Outcomes evaluation should focus on (a) assessment of the whole array of basic competencies that are acquired or mastered and (b) the extent to which these competencies meet the entire constellation of needs and how.

2. The whole child and comprehensive youth development approaches will require much broader systems evaluation (multiprogram, multiprovider) versus singular program evaluation.

3. Evaluation will play a significant role in social construction as we develop visions and portraits of what we want young people to become.

4. Needs assessment will focus on the basic needs of youth and the contexts of those needs, not just self-reports of young people.

5. More time and labor will be required for evaluation.

6. Prevention practitioners will need to become more reflective about their work and to play a significant role in evaluation.

7. Evaluations will require a broadening of methods and sources.

8. All evaluators will need to develop new interpretive skills around the data generated by multiple methods and perspectives.

The days of current evaluation and needs assessment practices depending on student self-report ATOD surveys as both assessment of needs and program outcomes should be numbered. We also need to reconsider the natural tendency to try to find the one instrument to do the job of evaluation. Both these practices have resulted in the evaluation tail wagging the prevention dog, and as a result, we are still struggling to provide strong evidence that prevention works. Many prevention professionals know from their experience that it does. Reflective and responsive evaluation can lead the way, but we must be clear about what the dog is and how and why the dog wags its tail!

Copyright © 1995:

Midwest Regional Center for
Drug-Free Schools & Communities


North Central Regional Educational Laboratory

Contact: info@ncrel.org

Posted on March 27, 1995

URL: http://www.ncrel.org/sdrs/areas/issues/envrnmnt/sa/4-1stop.htm