William E. Kelly

The Catholic University of America

Washington, DC 20064

Introduction

In 2005, the Association for Institutional Research (AIR) published two volumes on assessment in the disciplines; these two volumes deal with the business disciplines.1 A third volume on mathematics has recently been published and a volume on engineering is scheduled for 2007. The purpose of this paper is to provide an introduction to the AIR engineering volume.

The vision of the engineer of 2020 presented by the National Academy of Engineering (NAE) in their recent report goes well beyond the ABET outcomes as it should.2 One of the keys, according to the NAE authors is life-long learning – one of the current ABET outcomes and one that has been particularly difficult to assess and evaluate. The NAE authors state that “...to be individually/personally successful, the engineer of 2020 will learn continuously throughout his or her career, not just about engineering but also about history, politics, business, and so forth.”3

Assessment is increasingly focusing on student learning including the ability of students to assess their own learning. Ultimately, a professional must have the ability to assess where there are gaps in their knowledge and skills and develop, implement, and assess appropriate learning strategies.

An overall goal of accreditation is to assure minimum levels of quality in graduates and to promote continuous quality improvement in programs. The preface to the current ABET Engineering Criteria states that

These criteria are intended to assure quality and to foster the systematic pursuit of improvement in the quality of engineering education that satisfies the needs of constituencies in a dynamic and competitive environment.4

The current ABET criteria are not prescriptive as to what students should learn but they do require a process of goal setting, evaluation of achievement of goals, assessment of outcomes, and quality improvement.

ABET recognizes that to realize its vision, it has to be a leader in promoting assessment and continuous improvement. To this end, ABET recently completed a longitudinal study of the impact of EC 2000 on engineering education and published “Engineering Change: A Study of the Impact of EC 2000” that documents the positive impacts that the new engineering criteria are already having on graduates.

The ABET longitudinal study also documents the broad support that the new criteria and the philosophy of assessment and improvement have in the engineering education community. The issue now is increasingly how best to do assessment and how to do it efficiently and efficiently.

ABET and others recognize that the processes for assessment and improvement must be sustainable for a range of educational institutions. Therefore, assessment must not be too burdensome for faculty and must also be perceived by faculty and administrators as adding value to their programs. Faculty members may also expect that good assessment will help them improve their teaching effectiveness.

ABET provides the criteria but it has been somewhat reluctant to define what constitutes good practice for assessment and improvement. It does, however, provide guidance through activities such as workshops for faculty on engineering assessment practice.5 It also provides guidance on assessment planning through a section on the ABET web site maintained by Gloria Rogers.6

There have been numerous sessions and papers at ASEE regional and national meetings dealing with assessment. ASEE started early with its report on assessment – still a good reference.7 In 2005, Olds et al published a review of the state of assessment in engineering education as reflected in articles published in the ASEE Journal of Engineering Education.8 Bottom line - there is no shortage of materials faculty and program administrators can consult on assessment.

The situation is somewhat different with quality improvement. There has been some experimentation with ISO 9001 and with Malcolm Baldridge in higher education but there appear to have been few or no attempts to apply either to engineering programs in the US.

One of the issues that engineering programs face is workload; thus faculty and program administrators are looking for assessment methods that are both reliable and efficient and of course acceptable to ABET. There is also the issue of data collection and processing and this is where institutional-wide support services such as institutional research offices can and are helping. Institutions have a need for and interest in defining and managing assessment and improvement processes that will serve a wide variety of accrediting agencies.

In 2002, ABET looked at some of the potential barriers to full implementation of EC 2000. One of the conclusions was that “Workload, documentation, and assessment tools continue to provide frustration for constituents. Sustaining the change relies on sustaining the level of commitment and enthusiasm – the level of momentum – both on campus and at ABET. We understand this and are working to continually improve it.”9

Standards and assessment are a fact of life for elementary and middle school teachers and increasingly for high school teachers as states set and enforce learning standards. The Joint Committee on Standards for Educational Evaluation recently (2003) published “The Student Evaluation Standards.”10 One purpose of standards is to document and promote best practice and the International Organization for Standardization (ISO) working agreement on applying ISO 9001 to higher education should be useful to programs in organizing their improvement processes. At this point, there are no international standards for assessment in higher education.

In the remainder of this paper I will outline some of what has been learned about assessment and improvement as reported in the business and mathematics AIR volumes and what is planned for the AIR engineering volume.

Previous Association for Institutional Research (AIR) volumes

The Association for Institutional Research (AIR) supports institutional research efforts in post secondary education. Campus institutional research offices collect and manage data used for a variety of campus planning and management activities. Increasingly, they are providing support, particularly for data collection and management, for campus-wide and program-level assessment activities. AIR instituted the series “Assessment in the Disciplines” specifically to support institutional researchers and faculty in campus assessment activities.11

AIR has published three volumes on assessment practice. The first two volumes deal with assessment in business schools and the third volume with assessment in mathematics. The engineering volume will be the fourth.

Accreditation in business schools is at the school or college level in contrast to engineering programs which are accredited at the program level. However accreditation of general engineering programs at schools where there are multiple tracks in engineering probably comes very close to the situation in business schools.

In 2003 the Association to Advance Collegiate Schools of Business (AACSB) approved and began to implement new standards for accrediting business schools.12 These standards require that business schools be able to provide direct evidence of student progress in meeting learning goals.13 The operative words here are “direct evidence.”

Martell and Calderon present what is intended to be primer for assessment for business schools. In their introduction to the first AIR volume, they provide a summary of where the business schools are with assessment today.14 Some of their comments should resonate with engineering faculty. For example, they advise business deans to focus on direct assessment methods for assessing student learning; they note that surveys have their place in academic planning and management but not as evidence of student learning. They also advise deans to keep things simple. They point out that good program assessment does not have to meet the standards of academic rigor expected for peer-reviewed publication but it does have to effective; the judges of effectiveness ultimately being the users and the accrediting team.

Since the AACSB standards are new, it is to be expected that business schools would be and are struggling with how to do direct assessment and provide appropriate evidence – a situation not too different from what engineering programs are dealing with. In the introductory chapter to the first business volume, Martell and Calderon note that requirements for assessment data for business schools are consistent with regional and other professional accreditation (e.g. ABET) and some state legislatures.15 There are many good examples in the two business volumes that would be useful to faculty and administrators charged with organizing assessment and improvement processes at the school and college level in engineering.

Anyone familiar with the evolution of engineering accreditation over the last ten years would find the discussion in business familiar. Martell and Calderon’s comment that the AACSB requirements for assessment data are consistent with regional and other professional accrediting bodies and some state legislatures suggests, as suggested earlier, that universities will increasingly be looking to define best assessment practices to demonstrate student learning to multiple audiences.

Mathematics knowledge and skills are extremely important in engineering. The ABET requirement for mathematics and science is 32 credits and a number of the program criteria imply a high-level of mathematical performance for graduates. Assessment of mathematics readiness – relation to performance - is also increasingly recognized for its importance in advising students in and potentially interested in studying engineering . According to Adelman, “the highest level of mathematics studied in secondary school is strongly correlated with bachelor’s completion in any field.”16 This is particularly true with respect to persistence and success in engineering.

Mathematics and the AIR mathematics volume are different in that mathematics programs themselves are not separately accredited. However, mathematics is an important part of all engineering programs and thus there would be expected to be common assessment issues. Several of the papers in the mathematics volume are directly applicable to engineering. Also, the Mathematical Association of America is actively supporting assessment with its “Supporting Assessment in Undergraduate Mathematics” (SAUM) program and has case histories available online.17

Assessment in Engineering

As indicated earlier, ABET itself is a good source of assessment materials and a resource to check frequently is the ABET web page.18 For programs undergoing a review there is no substitute for participation in the ABET annual meetings. The fall meeting is now largely devoted to assisting programs in preparing excellent self-study reports and the summer EAC meeting for deans to providing the latest information on practices in place for the current visit cycle; what are the latest issues, how evaluators are looking at materials, and so on.

There is no substitute for experience and all engineering administrators should consider volunteering as an ABET evaluator.19 One of the keys to ABETs’ past and future success is having a cadre of effective evaluators. ABET is focusing a great deal of its current improvement efforts on improving the performance of program evaluators.20

ABET has a long history of conducting faculty workshops first to introduce faculty to EC 2000 and today to provide information on how to do assessment.

Rose Hulman Institute of Technology has been holding an annual symposium on assessment since 1997. The 2007 symposium is entitled “Best Assessment Practices IX.” This title suggests that the engineering community is beginning to define what is accepted as best assessment practice.

The ASEE annual and regional meetings are excellent places to find out what faculty members in engineering and technology are actually doing on campus. As mentioned earlier, the ASEE white paper on assessment is still a good resource.21 The ASEE Journal on Education is another resource. However, as suggested by Martell and Calderon, the key is direct, effective, and simple assessment.

What specific need will the AIR engineering volume fill? Clearly there is no shortage of materials that faculty could look to for guidance on assessment. What the AIR engineering volume will attempt to do is capture some good approaches that faculty are actually using in a range of institutional settings.

The AIR asked that the engineering volume focus on lower division course work. However, in soliciting papers, this was found to be too limiting for authors. In soliciting papers, the editor sought coverage of the range of what faculty and programs are actually doing. Past and current EAC members were asked to suggest assessment activities that should be highlighted in the volume.

The ABET longitudinal study is unique in the assessment literature. A summary of the study is available on the ABET web site and a copy of the complete report can be purchased from ABET.22 The full report is over 400 pages long. There is quite a bit that can be gleaned from the report about what seems to be working for programs with respect to assessment and improvement. Two of the papers in the AIR engineering volume will analyze what was learned from the study about best assessment practices and suggest ways that this information can be used to improve program and campus-level assessment.

The most prescriptive statement of what engineering graduates should know is embodied in the Fundamentals of Engineering (FE) examination that many engineering schools are using for outcomes assessment. One of the papers in the engineering volume will deal with the use of the FE exam for outcomes assessment.

Standards and assessment are a fact of life for elementary and middle school teachers and increasingly for high school teachers as states set standards. There has also been an effort by the Joint Committee on Standards for Educational Evaluation “to develop standards to help ensure ethical, useful, feasible, and accurate evaluation of students. The Joint Committee is accredited by the American National Standards Institute (ANSI) to develop standards and the “Student Evaluation Standards” are an American National Standard.23 One of the papers in the volume will deal with these standards and how they can be related to the ABET requirements for assessment and evaluation. The use of the standards will be illustrated with a case history.

Not all of what students learn is learned in the classroom and student competitions can be an integral part of a program’s strategy to demonstrate achievement of ABET outcomes. There are a number of design competitions that have the potential to contribute to the overall educational experience. One of the papers in the volume will deal with assessment of the EPA P3 program itself and secondly what participation in the program can contribute to outcomes for engineering programs.24

Although cooperative education is no longer separately accredited by ABET, it is still a distinctive part of many engineering programs. ABET’s attitude has been that the outcomes for cooperative education programs are the same as for traditional programs and therefore no special accommodation is necessary. On the other hand, there are unique learning and assessment opportunities and one of the papers will deal with assessment of a cooperative education program. This paper will include information on the use of an on-line system for collecting and analyzing information from students’ cooperative education experience to document achievement of ABET outcomes.

Data collection and management is an issue for all programs and innovative approaches to using technology for assessment are of wide interest. One of the papers will deal with a flexible, data-driven tool for program level assessment.

Embedded assessment is a technique that takes advantage of what faculty are already doing to improve student learning to collect direct evidence to demonstrate achievement of ABET outcomes. One of the papers in the volume will deal with how a large state university is using embedded assessment to improve the effectiveness of its assessment and at the same time, manage faculty workload.

Although grades per se are not generally accepted as assessment, all faculty members know intuitively that assessment and grading are closely related. Thus the question is how to best piggyback assessment on the existing requirement to grade student work and performance. Embedded assessment is one way and one of the papers in the volume will outline how to use an existing grading system to support direct assessment.

The remaining papers in the volume present a series of case history experiences in different institutional and program settings. The goal is to share and promote good assessment practice across the spectrum of engineering program types and fields of engineering.

Summary

Assessment and improvement methodologies for engineering education are still in the early stages of development. The ABET Criteria have provided a driver and a motivator for assessment and improvement. However, there are many other pressures to improve engineering education and real success will only come when assessment and improvement have become integral to the culture of engineering education. Hopefully, the AIR engineering volume will contribute in some small way to this transition.