What role does a needs assessment play?

Being an informed consumer requires an understanding of a district’s or school’s capacity, needs, and goals and, more important, an understanding of the barriers to meeting these needs and attaining these goals (Fashola, 2004). Thorough needs assessments can provide both of these by looking at student demographics and achievement; classroom processes and supports such as curriculum and instruction, assessment, professional development, and classroom management; and school management and resources (Hale, 2000).

Under the Schoolwide Title I Program, schools in the planning process are required to conduct a comprehensive needs assessment. The needs assessment guides the development of the comprehensive improvement plan and suggests benchmarks for its evaluation. As a result, the needs assessment is closely linked to all aspects of schoolwide program implementation (U.S. Department of Education, 2006). The non-regulatory guidance for the Schoolwide Title I Program suggests school and district staff take the following steps in conducting a needs assessment:

Needs assessments allow school and district officials to focus on what is most important at school sites. They also help them to identify whether staff, administration, and students at a particular school have the motivation and capacity to implement a given reform.

What types of reform approaches are available?

Schools and districts have a variety of reform programs to consider for integration into a school’s comprehensive improvement plan. Some approaches provide their own curricula and instructional methods while others take schools through a process of decision-making centered on teaching and learning. Perhaps the best way to think about reform programs is along a continuum: from approaches focused primarily on curriculum and instruction to those focused primarily on processes and school organization (Hale, 2000).

Curricular or instructional programs are designed to increase student academic achievement in specific curricular areas. They typically provide level- and grade-specific curriculum and testing materials, training sessions and manuals, and many other forms of technical assistance. Because of their academic focus, these programs tend to be more structured than the broader organizational programs. They most often cover basic curriculum areas such as reading and math, but may also address other subjects. To address their academic needs, schools often implement two or more different curricular programs concurrently (Fashola, 2004).

Process or organizational programs focus on the organizational and administrative needs of the school rather than directly address academic achievement of specified subject areas. Instead of providing student materials and specific instructional approaches as curricular models often do, these programs restructure the organizational makeup of the school and improve the school climate through goal setting and similar activities. Process or organizational reform programs are rooted in the finding that schools are sometimes able to improve the academic achievement their students by restructuring the school organization (Fashola, 2004).

As schools and districts explore different reform programs, they should think about how a particular approach fits into the school’s overall vision and how it might be integrated with other strategies to form a coherent and comprehensive improvement plan that will help the school meet its needs and objectives.

What qualifies as methodological rigor?

In recent years, rigor of research design methodology has referred to two designs in particular: experimental and quasi-experimental designs. Thought to be the more rigorous of the two, experimental designs set conditions in which students are randomly assigned to comparison groups (minimally one that receives the treatment or program and one that does not). Students in each group are tested on a valid and reliable measure before and after exposure to the treatment/ program. Quasi-experimental designs also assign students to control and treatment groups, but instead of being assigned randomly, students are matched according to a number of relevant student characteristics. This design allows for analysis of historical data and retrospective analysis while also statistically isolating the effects of the treatment/program on outcomes (Slavin & Fashola, 2003).

What qualifies as evidence of effectiveness?

Straightforward descriptive analyses may show differences in student achievement like that of gain scores or the mean change in scores from one year to another, however, what is not so clear is whether there are true differences in the scores, or whether they are the result of chance (Slavin & Fashola, 1998). Before beginning analysis, researchers typically set levels of significance (symbolized by p) at p <= .05 or p<= .01, which signifies that there is a five- or one-percent probability, respectively, that the differences are a result of chance. Establishing significance, however, only tells us with great certainty that the differences between two sets of mean scores are not the result of chance; they say nothing of the extent to which the means are different—for example between pre and post test scores comparing the performance between the control and treatment groups. The effect size has been described as “a standard means of expressing achievement gains and losses across studies, showing differences between experimental and control groups in terms of standard deviation” Slavin, & Fashola 1998). In practice, +0.20 to +0.30 is considered a small effect, +0.50 a medium effect, and +0.80 a large effect (Cohen, 1988). While programs evaluated with experimental designs and with strong evidence of improving student outcomes are considered the gold standard in educational research, few programs meet the rigorous standards set by NCLB. Schools may need to consider programs without evidence of effectiveness that have individual components with strong links to proven research.

What other factors should be considered?

While scientifically-based criteria for rigorous research is indeed helpful in guiding schools’ choices of the programs they will adopt, there are still newer programs that have not yet been evaluated or have not been evaluated enough. Slavin has called such programs “promising, but not proven” (Slavin & Fashola, 1998). Besides being new, other programs that may fit this category are those that were found to have a positive influence on student outcomes in research. These outcomes may either be qualitative in nature or quantitative studies that did not use an experimental or quasi-experimental design.

Although rigor of research design is important, consumers should be careful not to rely solely on statistical generalizability when deciding on a match with a particular program. Even if the research supporting an a reform program meets all the criteria of methodological rigor, shows positive effects, results in moderate to strong effects sizes, and seems to perfectly match the needs identified in the comprehensive assessment, it is still necessary to study the contexts in which the program was implemented. In other words, factors such as school culture, support internal and external to the school, or conditions of school facilities and resources typically do not show up in quantitatively driven evaluations. Rather than the decisive factor, generalizability should serve as further evidence when considering a model.

Do we have the human resources and capacity to support the reform program?

All teachers and administrators, especially ones at low-performing schools, need a coherent system of professional development. However, many districts lack support systems to provide teachers and principals with the training they need. Most research-based programs require training prior to and during implementation. These trainings help to focus professional development across grades and subjects and should be the center of the professional development component of the school’s comprehensive improvement plan. Every effort should be made to ensure that other forms of professional development complement these trainings in terms of content, philosophy, time required, and costs. To do otherwise will result in a fragmented system of professional development and will be less effective in changing classroom practices.

Closely related to the need for professional development is the need for external assistance. Research has shown that program effectiveness is largely dependent on implementation support (Newmann, & Sconzert 2000). Effective implementation assistance should involve a blend of initial training and ongoing, on-site follow-up support for all teachers. It is essential that this support be provided by highly qualified model staff or by a local trainer who themselves has received sufficient training to be considered an expert in the research-based approach. Capacity building is also vital, as it helps schools develop local expertise and establish structures that nurture continued learning (Northwest Regional Educational Laboratory & The National Clearinghouse for Comprehensive School Reform, 2003).

According to a recent American Institutes for Research report (American Institutes for Research. 2005), questions schools and districts should ask before selecting a research-based approach include:

Do we have the financial resources needed to adopt the reform program?

School staffs have to make hard choices about their budgets. Start-up costs for implementing a research-based program can carry a hefty price tag. There are costs associated with materials, trainings, and teacher release time. Some may choose a particular program largely because it is inexpensive. But choosing a program based on costs without regard for fit or rigor is unlikely to produce effective schools or may even be harmful.

Schoolwide Title I program schools have the flexibility to coordinate and consolidate resources and services to improve the achievement of all students in a school. One key benefit of operating a Schoolwide Title I program school is the ability to integrate federal, state, and local education funds. Federal discretionary grants and formula grants, with a few exceptions, can be consolidated and used to raise student achievement across an entire school. Moreover, many statutory and regulatory reporting requirements are waived for Title I schoolwide program schools, however schools must demonstrate that the intent and purposes of the individual federal programs are met. For example, a school operating a schoolwide program may demonstrate that it meets the intent and purpose of the Title II, Part A program (Preparing, Training, and Recruiting High Quality Teachers and Principals) if a school’s comprehensive improvement plan identifies activities that increase teacher quality by providing professional development trainings associated with a research based program.

Whenever possible, school officials should consider integrating funds to finance the following costs associated with adopting research-based programs:

Since the amount and quality of human resources is dependent on the availability of financial resources, school officials operating schoolwide programs should carefully consider opportunities to integrate costs associated with adopting a research-based approach.

How much time will the reform program take to produce results (and do we have that much time)?

Schools and school districts feel pressure from parents, community stakeholders, and local and state accountability systems to act quickly and produce results. In today’s high stakes accountability environment, parents and the general public pressure schools to raise test scores. Superintendents feel pressure to hastily implement new programs to show that they are doing “something” during their tenure. Moreover, in the era of NCLB, low-performing schools are pressured to produce gains in the narrow timeline from initial identification as a school in need of improvement to school restructuring.

Often, districts respond to this pressure by mandating reforms before evidence of their effectiveness has been proven. Research demonstrates that reforms require 3 to 5 years to become fully implemented and to show improvement in student performance. Yet district leaders feel that they can’t wait years to produce gains in student outcomes (Corcoran, 2003). The need for expediency rather than proven results drives decision-making.

The need for scientifically based research becomes even more important in light of the pressure schools and districts face to act quickly to improve student achievement. Without readily available, rigorous research about education practices, educators and policymakers are more likely to make rushed decisions that are ineffective or even harmful (Lauer, 2004).