be investigated. As a result, we recommend that policy makers continue to support the development of new approaches to test-based incentives but with a realistic understanding of the limited knowledge about how to design such programs so that they will be effective.
Recommendation 1: Despite using them for several decades,
incen-tives and accountability. This should include exploration of the effects of key features suggested by basic research, such as who is targeted for incentives; what performance measures are used;
what consequences are attached to the performance measures and how frequently they are used; what additional support and options are provided to schools, teachers, and students in their efforts to improve; and how incentives are framed and communicated. Choices among the options for some or all of these features are likely to be critical in determining which—if any—incentive programs are successful.
In general, the design of test-based incentives should begin with a clear description and delineation of the most valued educational goals that the incentive program is meant to promote, as well as recognition of the tradeoffs among these goals. Those goals should shape the features of the incentive program, even though experience shows that the effects of a program may not always occur in the ways intended.
The performance measures used in an incentive system are likely to be critical. The tests and indicators used for performance measures should be designed to reflect the most valued educational goals, and their rela-tive weights in the incenrela-tive system should reflect the tradeoffs across educational goals that designers of the system are prepared to accept.
Although any test will necessarily be incomplete, it should be designed to emphasize the most important learning goals in the subject domain and to measure students’ attainment of the goals through the use of various test item formats.
A test that asks very similar questions from year to year and uses a limited set of item formats will become predictable and encourage nar-row teaching to the test. The test scores are likely to become distorted as a result, even if they were initially an excellent measure. To reduce the inclination for teachers to inappropriately teach to high-stakes tests, the tests themselves should be designed to sample the subject domain broadly and include continually changing content and item formats. And test items should be reused only rarely and unpredictably.
Performance targets should be challenging while also being attain-able. Data should be used to determine attainable targets. Psychological research shows that unrealistically high goals undermine motivation. The ideal goals provide optimal challenge—ones that encourage people to stretch themselves and are attainable with effort.
The indicators used to summarize test results should match the goals of the test-based incentives policy, both in terms of the level of student achievement expected and the students or subgroups that are the focus of attention. Because any system of tests and indicators is necessarily
incom-plete, the system should be designed to emphasize the most important goals, and progress toward those goals should be measured in varied and diverse ways. Policy makers should recognize that goals that are not mea-sured are likely to be deemphasized during instruction. Test-based incen-tive systems should be dynamic, responding to current goals as well as to indications of whether incentives are aligned to these goals in practice.
Given that tests are necessarily incomplete measures of valued educa-tional goals, designers of incentive systems should recognize the potential problems inherent in having strong consequences based on test scores alone and should experiment with the use of systems of multiple mea-sures that reflect desired outcomes. One way of incorporating multiple measures would be to use the results of large-scale tests as triggers for more focused evaluation of struggling schools and teachers, rather than as final evaluations on their own.
It is possible that the weak effects of the test-based incentive programs we reviewed may be due in part to the use of performance measures based primarily on tests that encourage narrow test preparation rather than broader instruction that can produce more general learning gains that are not tied to a particular test. We note, however, that the one program we reviewed that used multiple measures—the Teacher Advancement Program, which uses classroom observations in addition to test scores in evaluating teachers—produced a near-zero average effect with a number of negative effects in the upper grades. Again, this result underlines how much is still unknown about using test-based incentives effectively.
The nature of the support provided in conjunction with a test-based incentives system is also likely to prove important to success. If the capac-ity to bring about change is limited, successful implementation will require that the incentives system include provisions to promote the development of that capacity. In any system of incentives—whether focused on schools, teachers, or students—the people who are most in need of improvement and therefore usually the focus of the incentives are often specifically those who lack the capacity to bring about change on their own. The research to date does not suggest what kinds of support could be paired with test-based incentives to increase program effectiveness.
It is beyond the committee’s charge to suggest how to build capacity in school systems, but there is a growing literature on resources that are most useful in helping schools improve. Some of that work is brought together in two reports from the National Research Council, Engag-ing Schools: FosterEngag-ing High School Students’ Motivation to Learn (National Research Council and Institute of Medicine, 2004) and America’s Lab Report: Investigations in High School Science (2006a). A recent report by the Center on Reinventing Public Education (Hill et al., 2008) suggests new approaches to finance, governance, and accountability that would foster
the kinds of competitive experimentation that could produce empirically grounded understandings of what works under what circumstances and for different groups.
RESEARCH ON TEST-BASED INCENTIVES
Substantial research needs to be conducted in order to understand the effects of test-based incentives well enough for policies to be designed that will consistently result in meaningful educational improvement. The committee recognizes that it is difficult and time-consuming to conduct definitive—or even credible—studies of the effects of test-based incen-tives in educational settings. However, there is a strong initial body of work that can serve as a foundation. Chapter 4 provides examples of the kind of research that will be needed to identify successful ways of design-ing test-based incentive policies.
Recommendation 3: Research about the effects of incentive