Improving the Evaluation of Professional Development Presentations Using Retrospective Pre-testing of Existing Knowledge and Self-Efficacy

Module by: Wendy Siegel, Camille Yates

Summary: Professional development is expensive, difficult to implement successfully, and time-consuming to provide (Noyce, 2006), therefore only the most effective programs should be chosen. Unfortunately, although the need for evaluation of professional development presentations is receiving increased attention (Jerald, 2000; Lewis & Shaha, 2003), little has been done to examine the effectiveness of current practices (Lowden, 2005; Noyce, 2006). Current presentation assessments focus on the skills of the presenter rather than on whether or not participants increase their level of knowledge and apply what they learn from attending presentations (Lowden, 2005). The authors of this paper present the argument that the current practice of focusing on satisfaction surveys does not address the real goal of presentation evaluation, which is to determine if skills learned are eventually applied (Belzer, 2003; Elmore, 2002; Guskey, 2002; Killion, 2002) and ultimately benefit students. Two suggestions for improving presentation evaluations are presented: 1) clarifying the purpose of the evaluation, and 2) using the retrospective pretest method (Campbell & Stanley, 1963) to assess participant’s existing knowledge and self-efficacy.

Rigorous evaluation of professional development presentations is becoming a greater concern partially due to the increased focus on student achievement included in legislation such as No Child Left Behind (Killion, 2003), and partially because professional development is expensive, time consuming, and difficult to implement successfully (Noyce, 2006). In a study of 5 urban district, Miles, Odden, Fermanich, and Archibald (2004) found that spending on professional development (including teacher time, coaching, materials, facilities, tuition, and fees) ranged between 2 to more than 5 percent of total district expenditures, amounting to an average of more than $4,000 per teacher. This implies that as a nation we may be spending $5 to $12 billion annually on professional development. Furthermore, there is an additional, potentially more harmful cost – teachers’ absence from the classroom. These considerations make a strong case for the argument that only the most productive professional development should be utilized.

Careful assessment is needed to identify which professional development presentations are producing the greatest benefits. Well planned assessment is essential for improving instruction of any kind, and extensive research has been conducted to improve assessment methodology. However, although much of the research is applicable for administering multiple assessments across time (such as in classroom settings), little has been done to improve assessments given after one–time conference and other professional development presentations.

In fact, although evaluation is identified as a critical component in the delivery of professional development, it is also often considered the weakest link (Tibbetts, Kutner, Hemphill, & Jones, 1991). This may be because current presentation evaluations tend to focus on the skills of the presenter and surface measures of participants’ satisfaction. The goals of these evaluations are usually: 1) to determine if a presenter will be asked to return, and 2) to help a presenter hone her/his skills. Current assessments most often include questions such as whether or not a presenter speaks clearly and audibly, and if the presenter is organized. However, questions of this nature do not indicate whether or not the presenter has successfully imparted knowledge in such a way that attendees can understand and utilize the information in applied settings, which is the ultimate goal of a professional development presentation (Belzer, 2003; Elmore, 2002; Guskey, 2002; Killion, 2002). Many researchers, including Killian (2002), Noyce (2006), and Garet, Birman, Porter, Desimore, Herman, & Kwang, (1999) have recognized the weakness of current evaluation practices, and proposed that evaluation of professional development is in need of improvement from the practice of depending heavily upon participation counts and teacher satisfaction surveys.

This paper will identify the difference between assessment and evaluation, and compare and contrast the differences between procedures for assessing and evaluating presentations in the classroom as opposed to assessing and evaluating one-time presentations. After reviewing current practices, two potential ways to strengthen presentation evaluations are suggested: 1) clarifying the purpose of the evaluation, and 2) using the retrospective pretest method (Campbell & Stanley, 1963) to assess participant’s existing knowledge and self-efficacy. An example assessment is provided.

Assessment vs. Evaluation

Although the terms are often used interchangeably, there is an important difference between assessment and evaluation. Chittenden (1991) defines assessment as the process of collecting and organizing information or data so that learning may be evaluated or judged. Evaluation uses the results of assessment to make judgments about growth (how much an individual has changed) and progress (Owen, 1994). Yet assessment and evaluation are closely related. Assessment is possible without evaluation, however evaluation is not possible without assessment.

Multiple forms of assessment exist for different purposes. Assessments conducted after presentations differ significantly from those given in the classroom. Classroom instruction occurs during multiple occasions over an extended period of time, while many professional development presentations are one-time activities. Although best practices call for multiple presentations and assessments for optimal learning to occur, there are many occasions when one time presentations are difficult to avoid. Because of this, presentation assessment is often a one-time activity whereas classroom instruction usually incorporates multiple forms and instances of assessment over time, enabling the use of pre- and post testing to determine knowledge growth. Pre- post testing is rarely used in presentation assessment, yet pre-testing is essential for identifying participants’ existing knowledge in order to identify what additional learning was absorbed as a result of attending the presentation.

Another major difference between classroom and presentation assessment involves what is measured. In a classroom, growth in students’ knowledge and ability are assessed, keeping the focus on students’ learning, which is the ultimate goal of classroom instruction. On the other hand, assessments given after presentations most often focus on the presenters’ skills rather than participant’s learning and application of knowledge. Best practices include clarifying the purpose of the evaluation and how the results will be used before an assessment is developed (Mullins, 1994). If the ultimate goal of professional development presentations is that participants will apply what they have learned to impact student learning, then the current practice of measuring presenter’s skills is not addressing that goal.

Levels of Evaluation

In order to reach the goal of connecting attendance at professional development presentations with improved practice, Guskey (1998) proposed five levels of professional development evaluation. The five levels include: 1) participant’s reactions, 2) participant’s learning, 3) organizational support and change, 4) participant’s use of new knowledge and skills, and 5) student learning outcomes (the ultimate goal of educational professional development). Each of these levels will be examined in greater detail below.

Data on participant’s reactions (first level) is the easiest to collect and analyze, and therefore the simplest and most commonly used form of professional development evaluation. Typical questions included on these assessments ask participants to determine the quality of various items such as content, vocal clarity, activities, and overall presentation (see table 1 for an example). Questions regarding the conference/seminar as a whole may be included at this level (i.e.: Was the room temperature comfortable?). Commonly, this data is collected from questionnaires that are passed out to participants at the end of a presentation. These questionnaires usually include some combination of a rating scale and open ended questions. Although measuring participants’ satisfaction is important, this type of evaluation provides very little information on how much participants learned from a presentation, or how much they actually implement.

The second level of assessment examines participant’s learning by measuring the knowledge, skills, or changes in attitudes teachers have developed after attending a presentation. To accurately determine this, some form of pre- and post-assessment should be considered to determine if the participants already possess the knowledge and skills that will be presented, and if not, whether they possess the knowledge and the skills necessary to understand the content of the presentation.

Third level assessment examines organizational support for the skills gained in professional development. This data must be collected after a sufficient period of time has passed, and thus cannot be gathered at the end of a presentation. Furthermore, although this level may easily be assessed by school administration, obtaining this type of data requires access to information that is difficult if not impossible to obtain for the purposes of many one-time presentations not conducted at the school in question.

The fourth level involves assessing participant’s application of new knowledge and skills. A number of alternative sources are acceptable, such as direct observation, interviews, reflections, and questionnaires. Once again, this information would normally be collected after the fact, making it logistically difficult for presenters or conference organizers to obtain, but very achievable for school administrators. However, although direct observation is preferred for obtaining this information, it can be expensive and time-consuming.

The fifth level examines student learning. Participants’ students are observed or interviewed to determine if they benefited in some way (academically or behaviorally) from their teacher’s attendance at the presentation. However, because multiple other life events also have an impact, proving a direct link between student learning and professional development activities is problematic (Lockwood, 1999).

Meeting each of these levels of evaluation is optimal, however, the expense, manpower, and time needed to do so may make it logistically difficult. Therefore, in most cases presentations at conferences and in-services are evaluated using only the first level of Guskey’s model, which measures what the participant liked most and least about the presentation (Sherman, Kutner, Tibbetts, & Weidler, 1998), and focuses largely on the presenting skills and knowledge of the speaker rather than what participants will actually implement in practice. Obtaining participants’ reactions is easily addressed, but obtaining higher levels of evaluation, including level of learning, organizational support, use of skills, and student outcomes, is much more problematic.

Level of Learning

Determining how much knowledge participants gained from a presentation is impossible to measure without first determining their pre-existing knowledge of the subject, yet current assessments do not commonly address this concern. In fact, questions like the first question in Table 1 may actually lower the scores on a presenter’s evaluation if the attendee had advanced knowledge of the topic before attending the presentation. There are at least two potential ways for this to happen: 1) the level of the presentation (beginner, intermediate, advanced) is not accurately defined and an attendee with advanced knowledge mistakenly attends a presentation meant for beginners, or 2) the attendee misjudges his/her own level as “beginner”, only to find that he/she already knows the material being offered by the presenter. In both of these cases, in order for the attendee to truthfully answer the evaluation, he/she is forced to mark “Strongly disagree” for question #1, thus skewing the data and making it appear that the presenter did a poor job of conveying information, even if he/she gave an excellent presentation.

According to Guskey (1998), one way around the problem is to include some form of pre-assessment (illustrated by question 1a in Table 2) to determine if the participants already possess the knowledge and skills that will be presented. In an extensive literature search, only two articles suggested using pre-assessment for professional development presentations. Lamb and Tschillard (2005) suggested using the retrospective pretest method developed by Campbell & Stanley (1963) to measure the pre and post knowledge of participants. This method asks for both pre and post knowledge after a presentation is concluded. In this manner, response shift bias is avoided. Response shift bias is the change in perspective that an attendee experiences after a presentation. For instance, what a person considers “advanced” knowledge may change after hearing a presentation. This shift in perspective changes how they would mark their level of understanding after the presentation from what they would have marked their level of understanding of the material prior to the presentation. Response shift bias can contribute to underestimating the effects of a presentation (Lamb & Tschillard, 2005). One other article tried to address the problem of pre-existing knowledge by including the question: “Had you heard about the ideas presented in this workshop before?”. Unfortunately, the wording of that question does not get at the level of participants’ prior knowledge of the topic, merely that they had “heard” of it before.

Application of Skills

Observing participants might be the most accurate way of determining whether or not participants apply what they learn from attending a presentation, however, it is hardly the most efficient way, and would in many cases be difficult to implement for logistical reasons. The possibility also exists that if the participant knows the purpose of the observation they might “put on a show” to meet what they think is expected of them. Measuring participant’s motivation and attitudes may be an alternative way to determine if they are likely to apply the skills learned from a presentation. According to Killion (2003), when participant’s knowledge, attitudes, skills, aspirations, and behaviors are targeted for change, their students’ achievement is maximized. Furthermore, Shaha, Lewis, O’Donnell, and Brown (2004) assert that measures of attitudes should be standard practice for making decisions regarding educational programs. In order to support a link between professional development and student achievement, one of the things an evaluator must know is whether teachers were motivated to implement the strategies (Killion, 2003). Although obtaining data on whether or not teachers actually implement strategies is difficult, teachers’ motivation and the probability that they will implement the strategies can effectively be assessed by including pre- post questions about teachers’ self-efficacy.

Self-efficacy has been defined by Bandura (1986) as the belief in one’s own abilities to organize and execute the actions required to manage prospective situations. Bandura (1977) theorized thatmotivation, affective states, and actions are based more on what people believe to be true than on what is objectively the case.What people know and what they have previously accomplished are often poor predictors of future performance because their beliefs about their abilities will strongly influence their actual behaviors. As a result, their actions can often be more accurately predicted by what they believe they will do than what they are actually capable of doing. In other words, the individuals’ view of how capable they are at a given task will help determine what they will actually do with the knowledge and skills they have attained. The bottom line is that people will perform tasks with which they feel competent and avoid tasks when they are unsure of their ability.Graham and Weiner (1996) stated that self-efficacy has proven to be the most consistent predictor of actual behavior, particularly in the fields of education and psychology. A 1998 meta-analysis conducted by Stajkovic and Luthans illustrated that multiple researchers in various fields have noted a high correlation between self-efficacy beliefs and behavior changes, further supporting self-efficacy as an excellent predictor of behavior. A good self-efficacy assessment should include both a measurement of how much the individual believes they are capable of performing the task in question, and also the strength of that belief (Bandura,1997).

Advantages of Including Pre-Post Existing Knowledge and Self-Efficacy Questions

According to O'Day and Smith (1993), the format of assessment is important because it influences the format of instruction. The format for presentation assessment illustrated in Table 2 integrates levels 1, 2, and 4 of Guskey’s model with Bandura’s theory of self-efficacy to provide a number of valuable advantages over existing presentation evaluations.

The first advantage is that it gives not only an assessment of a presenter’s skills, but also assesses participants’ pre- and post knowledge of the topic (see table 2, question 1a) which helps to determine how well a presenter is conveying information. This format takes participants’ prior knowledge into consideration and allows the data to be provided in terms of gain scores, which are more appropriate and accurate for the purpose of determining how much participants learned, and for purposes of score comparison between presenters. This method emphasizes the increase in the participant’s knowledge that is directly attributable to attending the presentation, and keeps the focus on the presenter’s effectiveness as opposed to his/her presenting skills.

The second advantage is that retrospective pretest questioning regarding self-efficacy is utilized (See Table 2, 1a and 1b). According to Bandura (1997) and Killion (2002), this method will successfully predict whether or not participants will apply what they have learned. Another advantage of this format is that item #7 on Table 2 can serve as a needs assessment for future presentations.

Yet another advantage of using this format is that the evaluator may easily recognize when an assessment has not been completed appropriately. For instance, if someone circles all “excellent” responses, the evaluator may question the validity of the assessment. If that is the case, it is likely that the participant didn’t read the questions or was trying to save time in filling out the assessment, leading to inaccurate, skewed data. This ability to recognize cavalier responses provides evaluators with the opportunity to identify and have the option of discarding potentially invalid responses.

A final advantage of this format for assessment is that it incorporates an element of self-assessment. Research suggests that self-evaluation plays an important role in promoting learning (Arter, Spandel, Culham, & Pollard, 1994; Maehr & Stallings, 1972; Rolheiser, 1996) Positive self-evaluations encourage individuals to set higher goals and commit more effort to achieving them. Increased effort leads to higher levels of achievement. Higher levels of achievement can result in higher levels of self-confidence. Higher self-confidence predicts higher self-evaluation, and the cycle continues.

Conclusion

Because assessment drives instruction, the success of professional development presentations can be more accurately determined by ensuring that presentations are assessed effectively. Many professional development presentations are currently being evaluated through the use of assessments which focus on the presenter’s skills. Because the resulting data do not reveal whether or not student’s learning is ultimately impacted, there is a need to change the focus of these assessments from measuring presenters skills to measuring participants’ knowledge and application of skills. The suggested format for assessment included in this paper integrates levels 1, 2, and 4 of Guskey’s model with Bandura’s theory of self-efficacy to produce data which illustrates the degree of knowledge that participants gain from a presentation, and the likelihood of whether or not they will apply what they have learned. This change in purpose can help both presenters and decision makers. Presenters can use the data to determine if they are effectively teaching participants, and to make appropriate adjustments if they are falling short of their goals. Decision makers can use the data to identify presentations which are most likely to produce results, which in turn allows them to invite only the most effective presenters to return. Research is needed to evaluate the reliability and validity of assessments produced with the goal of measuring whether or not participants apply the knowledge they gain from attending professional development presentations.

References

Arter, J., Spandel, V., Culham, R., & Pollard, J. (1994, April).The impact of training students to be self-assessors of writing. Paper presented at the Annual Meeting of the American Educational Research Association, New Orleans, LA.

Bandura, A. (1977). Self-efficacy: Toward a unifying theory of behavioral change. Psychological Review, 84, 191-215.

Bandura, A. (1986). Social Foundations of Thought and Action: A Social Cognitive Theory. Englewood Cliffs, NJ: Prentice-Hall.

Bandura, A. (1997). Self-efficacy: The exercise of control. New York: Freeman.

Belzer, A. (2003). Toward broadening the definition of impact of professional development for ABE practitioners. Adult Basic Education, 13(1), 44-59.

Campbell, D. T., & Stanley, J. C. (1963). Experimental and quasi-experimental designs for research. Chicago: Rand-McNally College Publishing Company.

Chittenden, E. (1991). Authentic assessment, evaluation, and documentation of student performance. In Vito Perrone, (Ed.), Expanding Student Assessment. Alexandria, VA: Association for Supervision and Curriculum Development.

Elmore, R. F. (2002). Bridging the gap between standards and achievement: The imperative for professional development in education. Washington, D.C. Albert Shanker Institute.

Garet, M. S., Birman, B. F., Porter, A. C., Desimore, L., Herman, R., & Kwang, S. Y. (1999). Designing effective professional development: Lessons from the Eisenhower program. National Evaluation of the Eisenhower Professional Development Program, U.S. Department of Education. Retrieved October 12, 2006 from http://www.ed.gov/inits/teachers/eisenhower/index.html)

Graham, S., & Weiner, B. (1996). Theories and principles of motivation. In Handbook of Educational Psychology. Berliner & Calfee (Eds.). New York, NY: Simon & Schuster Macmillan.

Guskey, T. R. (2002). Does it make a difference? Educational Leadership, 59(6), 45.

Guskey, T. R. (1998). The age of our accountability. Journal of Staff Development, 19(4), 36-44.

Jerald, C. D. (2000). The state of states (Quality counts). Education Week.

Killion, J. (2002). Assessing impact: Evaluating staff development. Oxford, OH: National Staff Development Council.

Killion, J. (2003). Solid Footwork Makes Evaluation of Staff Development Programs a Song. Journal of Staff Development, 24(3), 14-26.

Lamb, T. A., & Tschillard, R. (2005). Evaluating learning in professional development workshops: Using the retrospective pretest. The Journal of Research in Professional Learning. Retrieved on October 13, 2006 from http://nsdc.org/library/publications/research/lowden.pdf

Lewis, V. K., & Shaha, S. H. (2003). Maximizing learning and attitudinal gains through intergrated curricula. Education 123(3).

Lockwood, A. T. (1999). The promise and potential of professional development. Unpublished manuscript.

Lowden, C. (2005). Evaluating the impact of professional development. The Journal of Research in Professional Learning. Retrieved on October 13, 2006 from http://nsdc.org/library/publications/research/lowden.pdf

Maehr, M., & Stallings, R. (1972). Freedom from external evaluation, Child Development, 43, 177-185.

Miles, K. H., Odden, A., Fermanich, M., & Archibald, S. (2004). Inside the black box of school district spending on professional development: Lessons from comparing five urban districts. Journal of Education Finance, 30(1), 1-26.

Mullins, T.W. (1994). Staff development programs: A guide to evaluation. Thousand Oaks, CA: Corwin Press.

Noyce, P. (2006, September 13). Professional development: How do we know if it works? [commentary]. Education Week, 26(3), 36-37, 44.

O'Day, J. A., & Smith, M. S. (1993). Systemic Reform and Educational Opportunity. In S. H. Fuhrman. (Ed.), Designing Coherent Educational Policy (pp.250-312). San Francisco: Jossey-Bass.

Owen, J. M. (1994). Genuine Reward: Community Inquiry into Connecting Learning, Teaching, and Assessing. Andover, MA: Regional Laboratory for Educational Improvement of the Northeast & Islands.

Rolheiser, C. (Ed.). (1996). Self-evaluation...Helping students get better at it!Ajax, ON: Visutronx.

Shaha, S. H., Lewis, V. K., O’Donnell, T. J., & Brown, D. H. (2004). Evaluating professional development: An approach to verifying program impact on teachers and students. The Journal of Research in Professional Learning, National Staff Development Council. Retrieved October 2, 2006, from http://www.nsdc.org/library/publications/research/shaha.pdf

Sherman, R., Kutner, M., Tibbetts, J., & Weidler, D. (1998). Professional development resource guide for adult educators and professional development resources supplement: Improving instruction, organization, and learner outcomes through professional development. Washington, DC: Office of Vocational and Adult Education, Washington DC division of Adult Education and Literacy. Retrieved on October 1, 2006 from http://www.eric.ed.gov/ERICDocs/data/ericdocs2/content_storage_01/0000000b/80/23/59/2e.pdf

Stajkovic, A. D., & Luthans, F. (1998). Self-efficacy and work-related performance: A meta-analysis. Psychological Bulletin, 124, 240-261.

Tibbetts, J., Kutner, M., Hemphill, D., & Jones, E. (1991). The Content and Delivery of Training for Adult Education Teachers and Volunteer Instructors. Washington, DC: Pelavin Associates, Inc.

Table 1

Example Presentation Assessment

Strengths of this presentation:

Areas for improvement:

Table 2

Suggested Presentation Assessment Form

Please circle the number that indicates your evaluation of the item in question. Be sure to read each question carefully, as this form may vary from traditional evaluations.

I particularly liked:

Suggestions for improvement:

Comments, questions, feedback, criticisms?

Send feedback

• E-mail the authors of the module, Improving the Evaluation of Professional Development Presentations Using Retrospective Pre-testing of Existing Knowledge and Self-Efficacy

Footer

More about this content: Metadata | Version History

• How to reuse and attribute this content
• How to cite and attribute this content

This work is licensed by Wendy Siegel and Camille Yates under a Creative Commons Attribution License (CC-BY 2.0).

Last edited by National Council of Professors of Educational Administration on Mar 14, 2007 12:38 pm GMT-5.