Dorothy C. Rea is assistant professor of Educational Leadership at Florida Gulf Coast University. Dr. Rea was previously a teacher and an elementary principal in the Shawnee Mission, Kansas as well as teaching as an adjunct in Educational Leadership at the University of Missouri at Kansas City.

Cecil F. Carter is an associate professor of Educational Leadership at Florida Gulf Coast University where he has served since 1997. Prior to joining the professoriate, he served in three states as a classroom teacher, principals, state education official, district administrator and superintendent.

Judy R. Wilkerson is a Professor of Measurement, Research, and Evaluation at Florida Gulf Coast University where she teaches undergraduate and graduate courses in measurement, assessment, and research. She has co-authored two books on assessing teacher competencies and dispositions and has consulted widely to state departments of education and universities in the areas of accreditation and assessment.

Thomas C. Valesky is currently a professor in the Educational Leadership program at Florida Gull Coast University and the coordinator of post master’s degree programs in the College of Education. Dr. Valesky was previously chair of Educational Administration and Supervision at the University of Memphis, member of the Executive Board of NCPEA from 1995-98 and President of the Southern Regional Council of Educational Administration (SRCEA), 2001-2002.

William Steve Lang is a Professor of Educational Measurement and Research at the University of South Florida St Petersburg where he teachers graduate courses in measurement, statistics and research. He has taught in the K-12 system as well as the universities in South Carolina and Georgia and co-authored two books on dispositions-- Assessing Teacher Dispositions – Five Standards-Based Steps to Valid Measurement Using the DAATS Model and its teacher competency corollary, using the DAATS model.

El Concilio de Oficiales Principales de Escuela de Estado (CCSSO), por la Escuela Interestatal Líderes Licensure Consorcio (ISLLC), estándares desarrollados para el conocimiento, para las habilidades, y para las disposiciones necesarias para la práctica efectiva por líderes educativos (CCSSO, 1996).) Estos estándares proporcionan un dominio contento viable de que valorar a líder aprender cognoscitivo y afectiva. El Líder Educativo Candidato Creencia Escala (ELCBS) es utilizado para la medición disposiciones del liderazgo de candidatos educativas. ELCBS fue construido basado en un proceso sistemático de muestreo de las disposiciones de esperanzas de desempeño de ISLLC. La evidencia inicial de la validez y la certeza es presentada, utilizando el modelo de Rasch de teoría de respuesta de artículo. La escala del intervalo-nivel para ser producida, junto con medidas adicionales ser desarrollada, proporciona el potencial para valorar el impacto de líderes en el logro de niños.

As colleges of education are faced with NCATE requirements to assess dispositions in addition to knowledge and skills, preparation programs across the country are looking for ways to assess dispositions through valid and reliable measures. In this article we describe the development of a survey instrument to assess the dispositions of master’s degree Educational Leadership candidates. We began by using the dispositions enumerated in the document developed as a companion piece to the 2008 national educational leadership policy standards (Council of Chief State School Officers, 2008a) titled Performance Expectations and Indicators for Education Leaders (Council of Chief State School Officers, 2008b).

The gap in the literature of leader dispositions assessment research is twofold. First, there is limited attention to building a scale that systematically samples from the content domain needed for accountability and accreditation (i.e., the ISLLC standards). Second, the measurement process is largely reliant on statistics that fail to address the assumptions for their use and/or do not lead to research designs that take advantage of pairing dispositions results with interval level achievement scores. The question explored here is whether leader dispositions can be scientifically measured using a standards-based instrument and modern measurement/statistical techniques, as is being reported for teacher dispositions assessment (Wilkerson & Lang, 2011).

The Educational Leader Candidate Belief Scale (ELCBS) instrument discussed herein made use of the ISLLC standards, affective measurement literature, psychometric standards, and Rasch measurement. Each is presented briefly below.

The DAATS model is being followed in developing the ELCBS. Purpose was defined as both remediation of individual candidates and program improvement efforts; the content domain was defined as the ISLLC standards (Step 1). The sampling plan was based on coverage of most ISLLC indicators (Step 2). The first instrument (ELCBS) was developed using the Thurstone technique (Step 3). Technology (Angel) will be used to manage the data (Step 4). The current study provides beginning evidence of validity and reliability (Step 5.)

We created a series of 53 statements, eight to ten per Performance Expectation (PE). Each statement was classified based on our expectation of its difficulty, with the goal of ensuring variability and the expectation that the classifications would change with empirical data. Without variability there is no measurement, only confirmation. Existing measures, such as the one proposed by Brown, Kin, & Herron (2008), showing virtually no variability, are less likely to explain differences in performance. To avoid respondents agreeing without thought to all items, a mix of items with expected “agree” and “disagree” responses was included. Table 1 provides the number of items on the instrument by performance indicator, level of expected difficulty, and agree/disagree distribution.

Table 2 provides sample items, one for each PE. The scaled scores (Rasch measures) are reported for each of these sample items, showing that in some instances the expected difficulty matched the observed difficulty, whereas, for other items it did not. As calibration continues, these values are likely to shift with more respondents. The point here is that the scaling process is working. We have chosen the most difficult item as the example for PE 6, demonstrating the use of an item that pushes commitments to an extreme level.

Responses on this first pilot test of ELCBS meet all of the Rasch statistical requirements for modeling (or measuring) the construct of consistency with the ISLLC dispositions, so there is psychometric reason to have confidence in the measures. The rank order of leaders is as expected, providing judgmental evidence that the scale is working. The construct is well measured; the scores are valid and reliable for these 26 leaders. Reliability is acceptable and should improve with attention to the misfitting items and addition of respondents.

The items on the scale can be related clearly to issues of fairness and the belief that all children can learn, meeting the NCATE requirement for assessing dispositions. For example, leaders who respond appropriately to questions about using data for assessment, equality of resources, diversity, collegiality, and other values tapped provide evidence of their likelihood of making fair decisions. The instrument, then, provides an operational definition of the NCATE requirements as well as the ISLLC standards.

Based on the results of this study, the ELCBS can be implemented in the Educational Leadership program as a valid and reliable measure of candidate dispositions. The items on the scale can be related clearly to issues of fairness and the belief that all children can learn, meeting the NCATE requirement for assessing dispositions. For example, leaders who respond as expected to questions about data use for outcomes assessment, consensus in vision setting, equality of resources, non-exclusive use of test scores, diversity issues, collegiality, and other values tapped in this instrument have provided evidence that they are likely to make fair decisions benefitting all children in the schools. The instrument, then, provides an operational definition of the NCATE requirements.

Given the psychometric properties of the instrument, combined with raw and scaled scores and distribution, we can conclude that leaders are learning the values associated with the ISLCC standards in this program. The mix of current leaders and future leaders on the scale, with most students slightly lower than practicing leaders confirms that students are making progress toward the acquisition of these learned values.

Because the instrument is technically sound, it also provides useful information to monitor and guide the learning of candidates in the program. Unexpected values surface at an individual item and total score level, providing valuable feedback and opportunities for improving students, who are basically on track.

Similarly, items that were unexpectedly difficult can provide important opportunities influencing the program. For example, one of the most difficult items on the scale was item 2, “Consensus is critical in setting a vision, but sometimes one or two people can get in the way.” Lively discussions can result from whether or not those with dissenting opinions are “getting in the way.” One of the most difficult items, item 10, asked “With appropriate learning opportunities, most children can learn.” This item challenged respondents to choose between most children and all children, again a good topic for discussion. The vast majority of respondents were satisfied with “most” – something we expected but hoped would not happen.

As we progress with the refinement and implementation of the instrument, five items need to be monitored and possibly rewritten to improve the scale. It would also be useful to add some additional difficult items. Leaders with the lowest scores (more than one standard deviation from the mean) should be monitored to determine needs for intervention, especially if their score was different from what was anticipated. Leaders with unexpected incorrect responses on easy items should also be asked about their beliefs on those items.

Most important we also recognize that one instrument is insufficient to measure anything, so we need to continue to explore other instruments to measure dispositions, including the current observation instrument. It may also be useful to review a combination of selected and constructed response items, as recommended in the DAATS model, to provide a more diagnostic picture of future leaders’ dispositions toward the nationally accepted standards.

We have addressed the concerns expressed by Lindahl (2009). The care with which we developed and tested the instrument is the result of a well thought-out process. Passing scores are not typically set for dispositional measures; however, using the Rasch model, which produces interval level data it is possible to set cut-scores. We could use traditional standard setting models should there be a programmatic justification to do so. Selected response items avoid the bias of a constructed response or rating form. We are able to produce sub-scale scores for each of the performance indicators as well as a total score from the scale, looking both holistically and individually at the standards.

Most important we also recognize that one instrument is insufficient to measure well, so we will continue developing this and other assessments. The dispositions checklist (an observation instrument) is next. We also hope to develop a scoring system that may be of use to other groups measuring leader dispositions.

American Educational Research Association, American Psychological Association, and National Council of Measurement in Education (1999).  Standards for educational and psychological testing.  Washington, D.C.:  American Association of Educational Research.

Bond, T. & Fox, C. (2007). Applying the Rasch model: Fundamental measurement in the human sciences (2nd). Mahwah, NJ: Lawrence Erlbaum Associates.

Brown, S., King, G., & Herron, S. (2008). Assessing dispositions of administrative intern candidates. AASAJournal of Scholarship & Practice,5(1), 27-34.

Council of Chief State School Officers (1992). Interstate New Teacher Assessment and Support Consortium standards for teachers. Washington, D.C.: Author.

Council of Chief State School Officers (1996). Interstate School Leaders Licensure Consortium standards for school leaders. Washington, D.C.: Author.

Council of Chief State School Officers (2008a). Educational leadership policy standards. Washington, D.C.: Author.

Council of Chief State School Officers (2008b). Performance expectations and indicators for educational leaders. Washington D.C.: Author.

Damon, W. (2007). Dispositions and teacher assessment: The need for a more rigorous definition. Journal of Teacher Education,58(5), 365-369. doi: 10.1177/0022487107308732

Diez, M. E., & Raths, J. (2007). Dispositions in teacher education. Charlotte, NC: Information Age Publishing.

Dottin, E. (2009). Professional judgment and dispositions in teacher education. Teaching and Teacher Education: An International Journal of Research and Studies, 25(1), 83-88.

Katz, L.G. (1993). Dispositions: Definitions and implications for early childhood practice http://ceep.crc.uiuc.edu/eecearchive/books/disposit.html.

Linacre, J. M. (2003). A user’s guide to WINSTEPS: Rasch-model computer programs. Chicago: MESA Press.

Lindahl, R. (2009). Teaching and assessing dispositions in principal-preparations programs: A conundrum. In C.M. Achilles, B.J. Irby, B. Alford, & G. Perreault (Eds.), Remembering our mission: Making education and schools better for students (pp. 15-38). Lancaster, PA: Pro>activePublications.

National Council for Accreditation of Teacher Education (2008), Professional standards for the accreditation of teacher preparation institutions. Washington, D.C.: Author.

Perkins, D. (1995). Outsmarting IQ: The emerging science of learnable intelligence. New York: The Free Press.

Richardson, D. & Onwuegbuzie, A. (2003). Attitudes toward dispositions related to the teaching of pre-service teachers, in-service teachers, administrators, and college/university professors. (ERIC Document Reproduction Service ED482689).

Ritchhart,R. (2001).From IQ to IC: A dispositional view of intelligence. Roper Review, 23(3),143-50.

Schulte, L., Edwards, S., & Edick, N. (2008). The development and validation of the Diversity Dispositions Index. AASAJournal of Scholarship & Practice,5(3), 11-19.

Smith, E. & Smith, R., (Eds.) (2004). Introduction to Rasch measurement. Maple Grove, MN: Journal of Applied Measurement Press.

Smith, R. M. (2003). Rasch measurement models: Interpreting WINSTEPS and FACETS output. Maple Grove, MN: Journal of Applied Measurement Press.

Theoharis, G. & Causton-Theoharis, J. (2008). Oppressors or emancipators: Critical dispositions for preparing inclusive school leaders. Equity & Excellence in Education, 41(2), 230-246.

Thurstone, L.L. (1928). Attitudes can be measured. American Journal of Sociology, 33, 529-554.

Wasonga, T. A., & Murphy, J. (2007). Co-creating leadership dispositions. International Studies in Educational Administration, 35(2), 20-31.

Wilkerson, J. R. & Lang, W. S. (2011). Standards-based teacher dispostions as a necessary and measurable construct. The International Journal of Educational and Psychological Assessment 7 (2), 34-55.

Wilkerson, J. R. & Lang, W. S. (2008). Measuring teacher dispositions: An application of the Rasch Model to a complex accreditation requirement. Resources in Education. (ERIC Document Reproduction Service No. ED502872).

Wilkerson, J. R. & Lang, W. S. (2007). Assessing teacher dispositions: Five standards-based steps to valid measurement using the DAATS model. Thousand Oaks, CA: Corwin Press.

Wilson, M. (2005). Constructing measures: An item response modeling approach. Mawah, NJ: Lawrence Ehrlbaum Associates.