Los líderes de la educación y mucha literatura exhortan que maestros y educan a líderes para utilizar los datos más a menudo y más indicar efectivamente planeando y la toma de decisiones – llamado, "los datos manejaron la toma de decisiones". Este término es ubicuo en el discurso de la literatura y la reforma, pero "en el suelo," tan hablar, los facultativos encaran desafíos significativos a analizar, a la comprensión, y a aplicar los datos para mejorar la práctica. Los obstáculos encarados por facultativos incluyen pericia insuficiente, las herramientas, y el tiempo; también, las culturas organizativas en escuelas crean generalmente pocos estímulos para el análisis de datos y tan a menudo como no sostiene normas hostiles a la colaboración y acción colectiva requirió para datos manejó la toma de decisiones. El caso informado aquí ilustra estos desafíos por las acciones de un director que identifica un problema de la cultura organizativa y la práctica y de dirigir instruccional una iniciativa para promover colaboración, el análisis, y la reflejo para ayudar a mejorar escribiendo instrucción en su escuela.

Education leaders and much literature exhort teachers and school leaders to use data more to guide planning and decision making – called, “data driven decision making.” A substantial literature has emerged with theoretical models and practical prescriptions. Yet typical practice as shown by empirical studies still falls well short of theory-based conceptions embraced by scholars and reformers. Practitioners still face many challenges in analyzing, understanding, and applying data to improve practice.

This case illustrates an application of data analysis in service of standards-based instruction. A school principal is concerned about variable academic standards in his school, particularly in literacy instruction. Teachers have avoided for the most part collaborative planning and there has been little scrutiny or discussion of practice. Seeking a mechanism to change this culture, he organizes a benchmarking activity to examine and rate student writing in 5^th grade – an activity he hopes will stimulate teacher conversations about writing, help create greater consistency among teachers in assessing writing, and show that instruction can and should be subject to systematic empirical inquiry.

This module strengthens knowledge and skills in using and analyzing assessment data and applying data analysis toward the aim of standards-based writing instruction.

Section I. presents theory and research on data-based decision making and school leadership; the first part provides background on data-based decision making and the second part discusses challenges of school leadership aimed at supporting data inquiry to improve practice. Section I. can be read and discussed with or without supplementary reading (see reference list). Students should discuss ways to connect data with writing instruction, trying to be specific about how to achieve what they propose (e.g., how would you actually do this in your school?).

The first part of Section II. presents the methods and the case – an account of one school’s approach to data analysis through a benchmarking activity in writing. The principal is concerned about a multi-year pattern of mediocre writing assessment results at his school. He wants to stimulate inquiry into practice and motivate change. This case describes a data-based benchmarking process.

This module can be read in its entirety followed by discussion, or the discussion leader can treat each section separately.

The second part of Section II. presents the analyses and results. The discussion leader should review each table and figure thoroughly so students understand each one. Some are just descriptive snapshots requiring little interpretation; others contain more information, require more interpretation, and raise discussion questions about their implications or limitations.

Section III. presents discussion questions and exercises to deepen students’ understanding of the data (quantitative literacy) and to reflect on implications for leadership, professional development, and instructional change.

Section IV. provides notes for the instructor on the questions and exercises of Section III. as well as a rubric to help evaluate the module’s largest assignment: proposing an analysis to supplement or expand on the analysis depicted in the case.

From US Secretary of Education Arne Duncan addressing the Fourth Annual IES Research Conference (IES, 2009), “Robust Data Gives Us The Roadmap to Reform:”

Secretary Duncan’s hope for data-driven schools is shared by many (Bernhardt, 2004; Boudett, City, & Murnane, 2005; DQC, 2009; Kowalski, Lasley, & Mahoney 2008; Mills, 2007). A large literature has emerged on data-based decision making along with annual conferences (e.g., DQC, 2009; MIS, 2010) and a variety of foundation-sponsored initiatives around the country helping strengthen districts’ data systems and personnel training. Media accounts with headlines like, “Data-driven schools see rising scores,” (Hechinger, 2009) help fuel high hopes for data driven decision making as do portrayals of model schools or districts (Henke, 2005; Dattnow, Park, & Wohlstetter, 2007; Zavadsky & Dolejs, 2007). There is no doubt that the role of data in teachers’ and principals’ practice has grown over the last decade along with improvements in data quality and data access technologies.

As Kerr, Marsh, Ikemoto, Darilek, and Barney (2006, p. 498) note, there are high expectations and much potential for data use, and many ways data can be brought to bear on planning and practice in schools.

A persisting gap exists between theory and typical practice as shown by empirical studies (Bruner et al., 2005; Coburn & Talbert, 2006; Coburn, Toure, and Yamashita, 2009; Ingram, Louis, & Schroeder, 2004; Means, Padilla, & Gallagher, 2010; Wayman 2005). Most schools still find significant challenges in trying to use data effectively. It is not easy to transform teachers’ roles that have never in the history of the profession entailed widespread expectations of proficiency and participation in data analysis to examine practice, evaluate outcomes, and guide decision making and planning – especially in collaborative groups as commonly espoused today.

The challenge, for the most part, is not lack of data. It is not that districts do not have data. Indeed, modern data collection and information technology have filled districts’ databases with test scores, grading records, conduct records, health information, demographic data, student transcripts, personnel records, finance and budgeting data, survey data, parent information, and more. Districts, generally speaking, have data. What districts generally don’t have are enough staff in each school with the needed expertise, initiative, and tools to turn data into actionable information.

Part of the challenge is capacity and part of the challenge is school culture. Capacity issues include limited expertise, data access, analytical tools, and available time. Beyond these issues of capacity, are school culture issues: many teachers are ambivalent about examining their practice with objective data. While no single “attitude” characterizes all teachers’ stance toward data, many teachers are apprehensive about the prospect of spotlighting classroom results and many do not fully understand uses of data for guiding instructional planning and evaluation. In any given school, if enough staff have skeptical or resistant attitudes, it will be challenging for the school’s leaders to build what is widely advocated in the literature: a culture of inquiry, reflection, and collaboration in which data plays a key role in planning and decision making (Zavadsky & Dolejs, 2007; Newman, 2006).

School leadership is the key to building a staff culture willing to examine practice and collaborate for improvement. Leadership, ideally, should come from a team – the principal and teacher leaders – and it should be driven by clear and specific purposes. Research shows that leadership teams united by a common purpose can be powerful agents of school improvement (McLaughlin & Talbert, 2006; Vescio, Rossa, & Adams, 2008) and that among the many improvement-focused purposes such teams can serve, focusing on data is an important priority (Chrispeels, Castillo, & Brown, 2000; Young, 2006). For instance, Young (2006), based on case studies of four schools, found that leadership effectiveness was a major variable in teacher buy-in and participation. Here is a description of a school with effective leadership:

It is neither simple nor easy for teachers in a school to transition from roles of autonomy to teamwork and to face heightened expectations about using data to guide decisions. New roles require new skills and may bring changed routines. For leaders it requires identifying opportunities that will create staff buy-in, but that also promote staff learning and improved practice (Chen, Heritage, & Lee, 2005; Chrispeels et al., 2000; Copland, 2003). At the same time, an initiative can go badly if leaders assign staff to tasks or roles they perceive as unproductive or that threaten pride or professional efficacy. As described above, there are many ways of bringing data analysis into practice and many challenges the leader must recognize and anticipate.

One strategy with potential is assessment benchmarking. As used here, this refers to teachers reflecting on and calibrating their assessment criteria and standards against a pre-established standard. Benchmarking can be as simple as a group of teachers discussing a particular scoring rubric and relating it to their individual assessment criteria and standards. Benchmarking can also be more elaborate, involving systematic procedures to rate student work, record assessment scores, analyze results, and develop action plans. The case presented here shows a method to examine writing assessment scores to identify variation and consistency among teachers and to guide teachers’ discussion of data, writing instruction, and assessment. It is an approach that is feasible with typical school data and tools and that does not depend on analyses beyond what is reasonable to expect from professional educators. Managed well, it is a productive learning experience with the potential to improve practice.

(Comments on Exercise #2, Section I.) Theoretical justification for PLCs: (a) strengthen teachers’ sense of ownership over curriculum and professional development; (b) improve quality of decision-making by pooling expertise; (c) raise level of accountability to colleagues. For more information, see DuFour, DuFour, and Eaker (2008), Kilbane (2009), McLaughlin and Talbert (2006), and Mullen & Hutinger (2008).

(Comments on Discussion question #1, Section II.) That academic expectations in writing in some classrooms (notably, classrooms 1, 5, and 7) need to be higher is a reasonable conclusion. While the evidence from the benchmarking exercise alone does not constitute incontrovertible proof, the evidence is definitely strong enough to warrant concern about a gap between what the three teachers view as “at standard” writing and the level of proficiency prescribed in state standards as specified in rubrics. It should be emphasized that these data – the evidence generated from an exercise like this – must be interpreted cautiously and discourse among participations should use appropriately qualified language. The evidence should stimulate further discussion about writing instruction, assignments, expectations, and grading.

(Comments on Discussion question #2, Section II.) Some experts believe that standardized writing assessments (like the one described here) may have the effect of constraining the range of performance. Imagine, for instance, if the students of an average grade class of 5th were given 24 hours to write an 800 word evidence-based argument on a particular topic (some issue of relevance to 5th graders). The low end papers would not look a lot different from the low end papers in a standardized writing assessment, but the high end papers would be sophisticated essays – fully developed arguments with evidence, definitions, explication of assumptions, examples, and possibly even rebuttals to counter positions. Thus, the range of papers from worst to best would grow because the most able and motivated students would not be constrained by a relatively short time limit and by the five paragraph structure imposed by standardized writing assessment rubrics. Being less constrained, the top students would have more freedom for reading, writing, revising and creating their own argument. So in this sense, relative to the standardized “one period” writing assessment, the observed range in quality of papers submitted would grow. However, what this means is less clear: the 24 hour essay allows more room for other attributes to factor into performance, namely motivation, substantive background knowledge, and information processing skill that arguably fall outside the domain of writing. This raises the question, then, would an assessment based on the 24 hour essay be just a measure of writing, or is it assessing some broader construct. There is no simple answer to this question because it depends on how we think of and define writing. While there is no simple answer, it is instructive to contemplate this question and examine our own conceptions of the construct, “writing.”

(Comments on Discussion question #3, Section II.) The answer to the question is “No,” the two averages being similar does not mean the teacher and the benchmark rater are consistent with each other in how they apply the rubric. It is essential to compare not just the mean scores of one classroom to another, but also to examine and compare deviation scores. For example, the teacher could grade three papers 5, 3, and 1, while the benchmark scores for those same three papers are 1, 3, and 5. Both sets of ratings have an average of 3.0, but, clearly, the teacher is not interpreting the rubric in the same way as the benchmark rater.

(Comments on Exercise #1, Section II.) The classroom level (not student level) deviation scores could be shown to teachers either as a group in a meeting or individually in one-on-one conferences; and, if in a group, can be viewed either with teachers identified or not. The leader would want to consider teachers’ level of concern about being identified and about whether there would be excessive discomfort in openly discussing individual assessment results. In some schools this would not be a problem, but if a school’s leadership and teachers are not practiced with such discussions, it may be better to design the process to avoid inter-teacher comparisons. For instance, code numbers could be substituted for teachers’ names in viewing tables showing the full set of classroom results, with later individual conversations to discuss individual results. Ideally, professionals should be comfortable with discussing practice and be accountable to supervisors, colleagues, and clients for performance outcomes; the reality is that culture and practice in many schools does not reflect this ideal.

(Comments on Exercise #2, Section II.) Suggested workshop activity: Collaborative grading of papers can be a very productive activity if well planned, organized, and led by an experienced workshop leader with expertise in writing instruction and assessment. The activity involves reading and discussing a range of student papers, discussing the paper’s strengths and weaknesses, comparing the attributes of weak papers to better ones, and individual participants explaining to others their grading criteria with examples from papers illustrating these criteria. The workshop can include mini-benchmarking sessions with a small number of papers (i.e., individual reading and rating of papers followed by comparing and discussing results – discussions should be connected with local or state documents containing approved writing standards). These sessions would be well served by creating tables like Table 7 so participants can examine deviation scores for ratings on individual papers and discuss disparities and outlier scores among the participants. The workshop should culminate with a focus on instructional practice to improve the writing.

(Comments on Exercise #3, Section II.) Many additional analyses are possible. Here are a few examples: (a) Probe more deeply into “error” patterns (deviation scores) in teachers’ application of the assessment rubric. (Table 7 shows a deviation analysis for one teacher.) Are teachers more likely to be off randomly or in predictable ways? (b) How big are the differences in performance scores (writing, test scores, grades) among the different demographic groups and do these gaps differ depending upon the measure? (c) Do students with higher standardized test scores also do better on the writing assessment? Get better grades? How strong is the correlation? Do relationships between achievement variables vary by demographic category?

Andrade, H, Buff, C., Terry, J., Erano, M., & Paolino, S. (2009). Assessment-driven improvements in middle school students' writing. Middle School Journal, 40 (4), March, 4-12

Bernhardt, V. (2003). No schools left behind. Educational Leadership 60 (5): 26–30.

Bernhardt, Victoria L. (2004). Data analysis for continuous school improvement. 2nd Edition. Larchmont, NY: Eye on Education.

Boudett, K., City, E., & Murnane, R. (2005). Data wise: A step-by-step guide to using assessment results to improve teaching and learning. Cambridge, MA: Harvard University Press.

Brunner, C., C. Fasca, J. Heinze, M. Honey, D. Light, E. Mandinach, and D. Wexler. (2005). Linking data and learning: The Grow Network study. Journal of Education for Students Placed At Risk, 10, no. 3: 241–267.

Chen, E., Heritage, M., & Lee, J. (2005). Identifying and monitoring students’ learning needs with technology. Journal of Education for Students Placed at Risk, 10 (3), 309–32.

Choppin, Jeffrey. 2002. Data use in practice: examples from the school level. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, April.

Chrispeels, J., Castillo, S. & Brown, J. (2000). School leadership teams: Factors that influence their development and effectiveness. Advances in Research and Theories of School Management and Educational Policy, 4:39–73.

Coburn, C. E., J. Toure, & M. Yamashita. (2009). Evidence, interpretation, and persuasion: Instructional decision making at the district central office. Teachers College Record, 111 (4), April, 1115–1161.

Coburn, C. & Talbert, J. (2006). Conceptions of evidence-based practice in school districts: Mapping the terrain. American Journal of Education, 112 (4), 469–495.

Copland, M. (2003). Leadership of inquiry: Building and sustaining capacity for school improvement. Educational Evaluation and Policy Analysis, 25 (4), Winter, 375 – 395.

Data Quality Campaign. (April 2009b). DQC brings together education leaders to urge states to use data systems for continuous improvement. DQC Newsletter, 3, Issue 7. http://www.dataqualitycampaign.org/files/DQCNewsletterApr09.pdf (accessed May 1, 2009).

Datnow, A., Park, V., & Wohlstetter, P. (2007). Didn’t see in the paper… could have missed it? Achieving with data: How high-performing school systems use data to improve instruction for elementary students. Los Angeles, CA: Center on Educational Governance Rossier School of Education, University of Southern California.

DuFour, R. (2011). Work together: But only if you want to. Phi Delta Kappan, 92 (5), February, 57-61

DuFour, R., DuFour, R. & Eaker, R. (2008). Revisiting professional learning communities at work: New insights for improving schools. Bloomington, IN: Solution Tree.

Feldman, J. & Tung, R. (2001). Whole school reform: How schools use the data-based inquiry and decision making process. Paper presented at the annual meeting of the American Educational Research Association, Seattle, April.

Gere, A. (2010). Taking initiative on writing. Principal Leadership, November, 37 – 42.

Hechinger, J. (2009, June 12) Data-driven schools see rising scores. Wall Street Journal.

Henke, K. (2005). From vision to action: How school districts use data to improve performance. Washington DC: Consortium for School Networking (CoSN), 1025 Vermont Avenue NW, Suite 1010.

IES (2009). Robust Data Gives Us The Roadmap to Reform. Address by Secretary Arne Duncan to the Fourth Annual IES Research Conference. Washington, DC: U. S. Department of Education and Institute for Education Sciences. Retrieve from: http://www2.ed.gov/news/speeches/2009/06/06082009.html

Ingram, D., Louis, K. S., & Schroeder, R. (2004). Accountability policies and teacher decision making: Barriers to the use of data to improve practice. Teachers College Record , 106 (6), June,1258–1287.

Kerr, K., Marsh, J., Ikemoto, G., Darilek, H., & Barney, H. (2006). Strategies to promote data use for instructional improvement: Actions, outcomes, and lessons from three urban districts. American Journal of Education, 112 (4), 496-520.

Kilbane, J. (2009). Factors in sustaining professional learning community. NASSP Bulletin, 93 (3), September, 184-205.

Kowalski, T.J., Lasley, T.J. II, & Mahoney, J.W. (2008). Data-driven decisions and school leadership: Best practices for school improvement. New York: Pearson.

Mason, S. (2002). Turning data into knowledge: Lessons from six Milwaukee public schools. Madison: Wisconsin Center for Education Research.

Means, B., Padilla, C., & Gallagher, L. (2010). Use of education data at the local level from accountability to instructional improvement. Washington, D.C.: U.S. Department of Education, Office of Planning, Evaluation, and Policy Development.

McLaughlin, M. & Talbert, J. (2006). Building school-based teacher learning communities: Professional strategies to improve student achievement. New York: Teachers College Press.

Mills, G. (2007). Action research: A guide for the teacher researcher. Upper Saddle River, NJ: Pearson.

Mullen, C. & Hutinger, J. (2008). The principal's role in fostering collaborative learning communities through faculty study group development. Theory Into Practice, 47 (4,), 276 — 285.

MIS (2010). Annual Management Information Systems conference, sponsored by National Center for Education Statistics and US Department of Education, Austin, TX.

Newman, J. (2006). Alabama district improves by sharpening data & goals. Journal of Staff Development,27 (2), Spring, 10-14.

Rowan, B., Harrison, D. & Hayes, H. (2004). Using instructional logs to study mathematics curriculum and teaching in the early grades. Elementary School Journal, 105 (1), 103-127.

Smith, J., Lee, V., & Newmann, F. (2001). Instruction and achievement in Chicago elementary schools. Chicago: IL: Consortium for Chicago School Research

Spillane, J. (2004). Standards deviation: How schools misunderstand education policy. Cambridge, MA: Harvard University Press.

Supovitz, J. & Klein, V. (2003). Mapping a course for improved student learning: How innovative schools systematically use student performance data to guide improvement. Report by Consortium for Policy Research in Education, University of Pennsylvania, Philadelphia, PA.

Wayman, J. C. (2005). Involving teachers in data-driven decision making: Using computer data systems to support teacher inquiry and reflection. Journal of Education for Students Placed At Risk, 10, (3), 295–308.

Vescio, V., Rossa, D. & Adams, A. (2008). A review of research on the impact of professional learning communities on teaching practice and student learning. Teaching and Teacher Education, 24, (1), January, 80-91.

Young, V. (2006). Teachers’ use of data: Loose coupling, agenda setting, and team norms. American Journal of Education, 112 (4), 521 – 548

Zavadsky, H. & Dolejs, (2007). Data: Not just another four letter word. Principal Leadership, October, 32 – 36.