Many institutions (and individuals) shy away from assessing dispositions of administrative candidates and from using the results of those assessments for any summative (Scriven, 1991) purposes for fear of legal reprisal (Lindahl, 2009). However, courts are unlikely to intervene if the institution has taken appropriate measures to ensure that the assessment processes are valid, reliable, provide sufficient due process, and do not violate the civil rights of applicants or candidates (Ginzberg & Whaley, 2003; Lindahl, 2009).
The first condition courts would consider would be the extent to which the dispositions assessed are clearly related to job requirements (Griggs v Duke, 1971), which is known as predictive validity. Institutions of higher education may look to professional standards to determine those dispositions most essential for the profession. This is referred to as content validity. For educational administration programs, the most common source (Lindahl, 2009) of such standards is the 1996 Standards for School Leaders (Council of Chief State School Officers), promulgated by the Interstate School Leadership Licensure Consortium (ISLLC) under the aegis of the Council of Chief State School Officers, later adopted by the National Policy Board for Educational Administration and the Educational Leadership Constituent Consortium. These standards identified 43 specific dispositions that all school leaders should demonstrate. The revision of these standards in 2008 (Council of Chief State School Officers) maintained the same basic six standards with only a minor change in wording. Instead of explicating the specific knowledge, skills, and dispositions underlying each standard, the new standards substituted performance-based language. However, personal communication with the head of the revision committee, Joe Simpson, revealed that the revision committee in no way intended to suggest that the original skills, knowledge, and dispositions no longer applied (personal communication, April 20, 2008). The performance-based language was substituted to facilitate the assessment of school leaders via the standards (Sanders & Keaney, 2007). Professional organizations, states, and institutions were left to ensure that both potential and acting school leaders were equipped with the necessary knowledge, skills, and dispositions. Consequently the 43 dispositions of the original ISLLC Standards may still be considered valid. Some caution may need to be taken if an institution chooses to assess all 43 dispositions, for this places a considerable burden on the multiple assessors of these dispositions and may be superfluous, as there is considerable conceptual overlap among the dispositions and auto-correlation is likely. It may be prudent to categorize them and to reduce the number of specific dispositions assessed.
Other sources of essential dispositions may include the NCATE Unit Standards (2008) and each individual school or college of education’s conceptual model, as required by NCATE. Because these standards and conceptual models were developed through formal processes involving university faculty and highly respected P-12 practitioners, they may be considered to have face, or even expert, validity. This provides the foundation for further research by the institution on the extent to which the assessed dispositional levels of candidates later translates into predictive validity for their performance as school leaders.
Establishing the content validity of a set of standards is merely the first step in the process. Next, proper and varied instrumentation should be developed to ensure the reliability of the assessments. Because dispositions tend to be somewhat amorphous, assessors may hold differing perceptions on what they are assessing. Consequently, all instruments must be extremely clear as to their definition of each disposition. From one assessor to another, there should be very little need for interpretation; this should help to ensure the stability of the assessments over time, allowing for them to measure any true growth (or decline) relative to the disposition, but greatly reducing the “noise” in the measurement process.
Multiple assessment instruments are advised (Lang & Wilkerson, 2006). There is a plethora of types of instruments to be selected from, e.g., scenario-based, portfolio rubrics, reflective journaling, interview rubrics, observation rubrics, Thurstone Attitude Scales, Semantic Differential Scales, focus groups, projective techniques, qualitative text analyses, constructed response methods, perceptual evaluation protocols, human relations incidents and subsequent interviews, biographical and metaphorical self-assessment, and paragraph completion methods. This allows for the various instruments to be checked for concurrent validity and reliability. It also enhances the reliability and validity of the overall assessment process.
In addition to the use of multiple instruments, the assessment of dispositions should be done from multiple perspectives, by multiple assessors. For this reason, self-assessment and feedback from peers, instructors, and mentors are recommended. Clearly, for the assessment of dispositions to be a learning process, it should be heavily based on the candidate’s reflection and self-assessment (Wasicsko, 1977; 2007). However, self-assessment often does not match the assessments of others; consequentially, for a candidate to continue to develop dispositionally, external feedback at various stages of the preparation program is essential. Such external feedback can come from colleagues specifically tasked with providing such feedback on the candidate’s dispositions, full cohorts or classes of peers, professors, and field-based mentors. This feedback may come from the use of common instruments, which can facilitate the triangulation of the assessors’ perspectives, or from instruments uniquely designed to suit the circumstances of each group of assessors to the candidate.
Any time multiple assessors are involved, regardless of how valid and reliable the instruments may be, attention must be paid to ensuring inter-rater agreement (reliability). This is best accomplished by careful training of each assessor on the use of the instrument(s) and trial evaluations of that assessor’s scores compared to the scores of other assessors evaluating the same candidate’s performance. This can be done by using sample video or audio tapes of candidates, portfolio assessments, or joint examination of survey instrument results. Such training and evaluation of assessors must be done prior to any formal assessment by that individual and must be periodically re-checked over time in order to prevent an assessor deviating from the group norms.
Messick (1989) introduced the concept of consequential validity, or the concern for the proper use of assessment results. The assessment of dispositions may be done for formative or summative purposes (see Scriven, 1991). When used formatively, the assessments should guide the candidate for further development of specific dispositions which are not yet sufficient to meet the institution’s standards. This calls for individualized dispositional growth plans and mentoring along the preparation program. It would violate Messick’s concept of consequential validity to fail to develop and monitor the implementation of such growth plans.
However, violation of consequential validity is more likely to occur in the summative uses of the assessment of dispositions. For example, dispositions should be assessed as part of the admissions process. Because at least some dispositions can be developed over time with proper circumstances and feedback, applicants who do not display the required level of dispositional development in the admissions process should have their admissions deferred, rather than rejected. When used summatively at the conclusion of the candidate’s program of study, or prior to admission to the field-based internship, candidates failing to demonstrate the required levels of dispositions should be provided with remediation opportunities rather than being summarily dismissed.
These processes tie closely to the court’s likely next concern – procedural due process. Institutions carry such responsibilities as advising applicants and candidates of the institutional expectations regarding specific dispositions, as well as of the processes by which these will be periodically assessed, from the admission process until program completion. They must be informed of how assessment data will be used in decision-making regarding the candidate and the potential consequences of those decisions. Similarly, after admission, candidates must be provided planned opportunities to develop those dispositions deemed to be their personal strengths and weaknesses. Finally, appeals processes regarding the assessment of dispositions must be available, without penalty, to the applicants and candidates.
The court’s likely final, and overriding, concern would be that no individual’s or group’s civil rights be violated in the assessment of dispositions. In short, institutions must ensure that all assessment practices are equitably implemented and are without bias toward any individual or group. For example, institutions are obligated to ascertain if their assessment practices, including the specific dispositions identified, are gender, race, culture, handicap, or sexual preference-neutral. Confidentiality of assessment results must be maintained among the program faculty and administration.
Although these challenges may seem daunting, they are by no means insurmountable. As the next section of this article highlights, there is a growing research base to support the efforts of institutions in assessing dispositions. In addition, the Internet provides access to much of the work that many institutions of higher education are doing in this regard, which may not yet be part of the refereed knowledge base.