Theoretically, as has become clear through empirical studies in language testing, there has been ‘a shift from using assessment as a way to keep students in their place to using assessment as a way to help students find their place in school and in the world community of language users’ (Cohen, 1996, p. 3). In this popular tendency of treating language tests, language tests have been considered extremely helpful for both students and teachers, and even for administrators. Madsen (1983, p. 4-5) points out the importance of language testing by demonstrating that properly made tests can

‘1. help create positive attitudes towards instruction by giving students a sense of accomplishment and a feeling that the teacher’s evaluation of them matches what he has taught them.

2. help students learn the language by requiring them to study hard, emphasizing course objectives, and showing them where they need to improve.

3. help teachers and administrators by confirming progress that has been made and showing how they can best redirect their future efforts.’

Therefore, being competent in language testing, particularly in oral language testing under review in this thesis, is claimed to be crucial for language teachers to properly develop language tests. This thesis is an attempt to provide a clear discussion of how to become competent in oral language testing. An answer to this question will explicitly help to evaluate TNU current oral testing practices.

Apart from the above theoretical concern, this thesis also grows out of a practical consideration regarding the researcher’s work at TNU as an English teacher and assessor of students’ oral test performance. The problem identified in this thesis has taken root from the existing oral testing practices at TNU. The following are two tables of oral test results of the second-year students (School Year 2001-2002) and of the third-year students (School Year 2002-2003).

Table 1.1: The second-year students’ oral test results

Table 1.2: The third-year students’ oral test results

The two tables reveal that nearly half of the second-year students (47%) and more than half of the third-year students (67,5%) get high marks (7,8 and 9). However, in a talk with the researcher about their speaking ability, preferably their results of the former speaking tests, the majority of those students who got high marks in the tests seemed very reluctant to agree that their test results really revealed their actual ability to use English for communication. For example, they still found it hard to use English to either satisfactorily communicate their ideas or make themselves fully understood in a real instance of communication. So why did they get such high scores?

The same question was put forwards for discussion with the teachers and the assessors of speaking skill. Most of them believed that those students deserved high scores because they actually performed their tasks fluently and were able to answer the questions of the examiners. Nevertheless, when the author asked these teachers what particular speaking abilities they expected to see in the students’ test performance, what exact criteria their assessment was based on, and what the detailed procedure for test design was, they did not all provided any specific and clear answers. They could not point out speaking abilities expected, they have assessed students’ test performance intuitively, and they have constructed oral tests in their own way. Thus, as can be concluded that there seems to have never been any official detailed guidance for the construction and administration of oral tests at this institution.

Most of the staff members whom I have discussed this matter with have shared the same worry and showed interest in how to gain a scientific approach to assessing our students’ oral ability properly and fairly. In other words, we all would like to find out how to appropriately measure the students’ speaking ability and how to write useful speaking tests. This aims at helping to ensure fairness for the students, and improve and maintain the training quality of the institution.

All the concerns described above indicate an urgent need to evaluate the reality of oral language testing at TNU in the light of language testing theory. A thorough study is carried out as an attempt to help the staff give the oral testing an adequate position in their training program.

The Design stage involves six activities all aiming at producing a design statement as a principled basis for the other two stages (Bachman & Palmer, 1996, p. 88). The six activities are as follows:

(1) Description of the test purpose(s). First specific inferences about language ability and capacity for language use made from the test takers’ performance are explicitly stated, and then specific decisions based on these inferences are provided.

(2) Identification and description of test tasks in the target language use (TLU) domain. A set of the TLU task types is characterized as the basis for developing actual test tasks.

(3) Description of the characteristics of the test takers. The characteristics to be believed to be particularly relevant to test development involve personal characteristics, topical knowledge, general level and profile of language ability, and predictions about test takers’ potential affective responses to the test.

(4) Definition of the construct/ability to be measured. The components of language ability to be assessed through the test task(s) are critically determined.

(5) Development of a plan for evaluating test usefulness. This plan consists of three parts as follows:

an initial consideration of the appropriate balance among the six qualities of usefulness and the setting of minimum acceptable levels for each,

the logical evaluation of usefulness, and

procedures for collecting qualitative and quantitative evidence during the administration stage.

(Bachman & Palmer, 1996, p.133-134)

(6) Identification of resources and development of a plan for their allocation and management. The resources refer to the people, material and time involved in test development. The balance between the resources available and required for test development should be taken into account in order to provide a good plan for how to allocate and manage them.

The Operationalization stage need to be closely examined with the purpose of helping the concerned staff of TNU to equip themselves with a thorough understanding of the stage and then to gradually improve their practices of oral testing.

As mentioned above, this stage focuses on the structure of a test – the blueprint involving the number of test tasks/parts and specifications of each test task.

Test task specifications are described as follows:

1. The purpose of the test task.

2. The definition of the construct to be measured.

3. The characteristics of the setting of the test task

4. Time allotment.

5. Instructions for responding to the task.

6. Characteristics of input, response, and relationship between input and response.

7. Scoring method.

(Bachman & Palmer, 1996, p.172-173)

These test task specifications can be interpreted in the context of oral testing as:

In order to well operationalize or produce test task specifications, test writers should make informed judgements of major considerations in oral test operationalization reviewed in detail in Section 2.4

The administration of a test mainly involves three activities such as giving the test to a particular group of test takers, gathering test results and analyzing the results (Bachman & Palmer, 1996, p. 91). In particular, procedures for administering a test include preparing the testing environment, communicating the instructions, maintaining a supportive test taking environment, and collecting the test papers. These all aim at ‘guiding test takers through the process of taking the test in accordance with the procedures specified in the test blueprint.’

The explicit classification of test takers’ language knowledge levels helps to grade test tasks according to the communicative stress. It is displayed on a formal document including established ‘criterion levels of oral language proficiency based on the goals and objectives of classroom instruction’ (O’Malley & Pierce, 1996, p.65). This document, called a level scale or rating scale by Underhill (1987), is a series of short descriptions of different levels of language ability in terms of test takers’ or students’ language knowledge. It describes in brief what a typical learner at each level can do so that teachers and assessors can analytically select or grade test tasks that best fit each level, and can easily decide on the score to give each student in a test.

The following is an example of a level scale with four major levels (Table 2.1 on page 17) based on the level scale introduced in ‘Hệ thống Định chuẩn Trình độ Ngoại ngữ’ của Hội đồng Châu Âu by Vũ Thị Phương Anh and Nguyễn Thị Kim Thư (2003).

Table 2.1: Level Scale of Language Proficiency Based on the Global Scale by Council of Europe

When test takers’ proficiency level is explicitly identified on the level scale, adequate oral test types and proper elicitation techniques must be critically selected to fit the test takers’ level and the testing situation. This sub-section reviews types of oral test in combination with elicitation techniques of this kind of testing, for these two aspects have an interrelated relationship. An elicitation technique involves the procedures of performance for each test task, and a test task itself represents a test type.

Underhill (1987) classifies oral tests/test tasks into four main types: (1) the direct interview type, (2) the pre-arranged information gap tests, (3) tests where the learner prepares in advance, and (4) mechanical/entirely predictable tests. Each type requires some specific techniques to elicit test takers’ language performance named elicitation techniques. The four following sub-sections respectively summarize these four oral test types in combination with the involved elicitation techniques

During the operationalization process of oral tests, the classification of test takers’ level of language proficiency is to be carefully considered with the purpose of choosing adequate test types and elicitation techniques in which the test takers’ language performance can be best shown. While designing particular test task(s), test writers or teachers should consider and decide how to mark each test task, and therefore build up a guideline helping the assessors to mark each task. This guidline is called a marking key or marking protocol by Underhill (1987).

A marking key is a set of procedures specified in advance that tells assessors what they are supposed to do step by step in the process of marking each test task/question. Test writers can make the marking quicker and more reliable by drawing up a detailed marking guide that tells the marker how to mark each question.

Underhill (1987, p. 95) identifies the aims of a marking key as follows:

- To anticipate problems that the marker is likely to face, and to suggest how to cope with them.

- To maintain the aims of the test by directing the marker’s attention to the language areas that are most important, and by giving general guidelines for dealing with unusual responses.

- To describe the purpose of each question/task.

A marking key revealing such aims thus surely helps to increase the consistency of measurement, that is reliability (See 2.5.2). In fact, oral tests are a kind calling for subjective judgement on the part of assessors, and thus do not have as a high degree of reliability as those that require objective judgement such as multiple-choice or cloze tests with either completely right or completely wrong answers. In order to help assessors achieve the highest possible degree of reliability, it is essential to provide them with a comprehensible marking key conveying the three aims identified by Underhill.

The most important factor concerned in a marking key is the distribution of marks to specific speaking sub-skills that are intended to be measured. These speaking sub-skills are named mark categories by Underhill (1987). The kind of categories defined in a test should be based on the teaching program and be cited by the way in which the teaching syllabus expresses the aims of the program (Underhill, 1987). There are two models that mark categories base on: the traditional model of language components and the more recent model of performance criteria. The former refers to the components of language proficiency (grammar, vocabulary, pronunciation and intonation, style and fluency, content, etc.) while the latter mentions the components of language performance or performance criteria (flexibility, accuracy, appropriacy, independence, hesitation, etc.).

The focus on a number of different language sub-skills or categories can also help to improve marker reliability; the assessor is supposed to give each test taker a separate mark for each category. All these separate marks are then combined to give the overall score, which is related to the process of weighting. In most oral tests or test tasks some categories are more emphasized than others according to the test purpose(s), so a weighting system is used as shown in the following example taken from Underhill (1987, p. 97).

Grammarmarked out of 10 then multiplied by 3

Vocabularymarked out of 10 then multiplied by 3

Pronunciationmarked out of 10 then multiplied by 2

Fluencymarked out of 10 then multiplied by 1

Contentmarked out of 10 then multiplied by 1

Total score 10

In sum, it can be asserted that the marking key plays a very essential role in the design of language tests in general, of oral language tests in particular to ensure the quality of reliability. It must be involved in the whole process of test development from the beginning. Language teachers or test developers should thus take the ways to mark test performance into sound consideration throughout the test construction process.

In the oral test operationalization process, consequently, language teachers or test designers must take great care over not only the selection test types and proper elicitation techniques for the intended test tasks but also the design of a marking key for each test task.

Test validity generally is concerned with the degree to which a test actually measures what it is supposed to measure. In other words, it refers to the correspondence between abilities to be assessed and real indication of these abilities in a test, so a test is said to be invalid when there is no relationship between them. The concept of validity includes such detailed aspects as content validity, construct validity and predictive validity. A test is said to have content validity if its content represents a sample of the language skills, structures, etc. with which it is meant to be concerned ( Hughes, 1989). When embarking on the test construction, a test writer should first draw up a table of test specifications, describing in very clear and precise terms the particular language skills and areas to be included in the test. Not less important is the construct validity of a test. A test with construct validity is capable of measuring certain specific characteristics in accordance with a theory of language behaviour and learning. In other words, construct validity ‘examines whether the instrument permit inferences about underlying abilities.’ (Cohen, 1996). According to Hughes (1989), the word ‘construct’ refers to ‘any underlying ability or trait which is hypothesised in a theory of language ability’. This ability or trait is defined by Bachman and Palmer (1996) as ‘the domain of generalisation to which our score interpretations generalize’. Certain learning theories or constructs are believed to underlie the acquisition of abilities and skills. Another approach to test validity is to measure the degree of the agreement between results of the test and those provided by some important task at some future point.

If test validity is defined as accuracy of measurement, test reliability is related to consistency of measurement. A reliable test score will be consistent across different characteristics of the testing situation. Unless test scores are relatively consistent, they cannot give any information at all about the ability measured. Another aspect of overall test reliability is rater reliability. Raters must maintain consistency in their own marking standards. This kind of reliability is called intra-marker reliability (Underhill, 1987). Or the same work marked by different raters should produce similar results, which is named inter-marker also by Underhill. If some raters rate more severely than others, the ratings of different raters are not consistent, and the scores obtained could not be considered to be reliable. Oral tests belong to the kind calling for subjective judgement on the part of the marker, so the scores awarded in an oral test cannot be believed to always have such high reliability.

It is also necessary to recognize that inconsistencies cannot be eliminated entirely. Nevertheless, it is possible to minimize the effects of the potential sources of inconsistencies under control in test design (Bachman and Palmer, 1996). Amongst factors affecting test performance, the characteristics of the test tasks are partly under control. In language test design and development, thus, it is possible to minimize variations in the test task characteristics that do not correspond to variations in target language tasks.

Test administration also involved in the concept of reliability has not been given proper attention at some universities at the present time. Administrating a test involves exam invigilators and such test conditions as classrooms, equipment, materials, exam rules and procedures dealing with test takers’ cheating.

Test practicality pertains to ‘the ways in which a test will be implemented, and, to a large degree, whether it will be developed and used at all’ (Bachman & Palmer, 1996, p. 35). It concerns practical matters such as the amount of time, human and material resources available for constructing a test, administering it, marking it, and interpreting the results. If the test resources required for implementing a test exceed the resources available, the test will be impractical. Human resources are a crucial component of test construction and administration involving such individuals as test writers, scorers or raters, and test administrators as well as clerical and technical support personnel. In fact, not all institutions have sufficient staff to be in charge of all these well-defined roles. One person may be in charge of several functions. Test writers, key personnel in the process of test development, are involved not only in writing tests but also in collecting materials, editing and recording. Material resources include space (the number of classrooms, language labs needed), equipment (typewriters, computers, cassette players, overhead projectors), test materials (test booklets, answer sheets, audiotapes). Time consists of test development time and the time required to complete the parts of each stage of the test development process.

Moreover, the specific types and amounts of resources required may differ according to the design of a specific test, and available resources may vary from one situation to another. Test practicality thus can only be determined for a specific testing situation. Obviously, to determine the practicality of a given test, test developers must take into account of the resources required to develop a test, and the management and allocation of the resources available.

The situation analysis with a checklist (as seen on page 31) that has been developed based on Bachman and Palmer’s framework for test development reviewed in 2.3 involved such four factors as (1) the test design stage, (2) the test operationalization stage, (3) the test administration stage, and (4) the use of test results.

Observation of a particular achievement speaking test is expected to help collect evidence that supplements the analysis of the current oral test development process, namely the operationalization and administration stages. The observation enables the researcher to directly collect data firsthand, and the data gathered describes the observed phenomena as they take place in their natural settings (Nachmias, 1996, p.206); therefore, this kind of information is surely of great reliability.

The observation focused on the oral test type(s) with underlying elicitation techniques being used (See Appendix 2), time spent on the students’ test performance, and interaction between the assessors and test takers recorded in a tape and then transcribed (See Appendix 3).

In order to elicit TNU staff’s perceptions of oral language testing, questionnaires were formulated, and then delivered to the staff members. Questionnaires, in this thesis, are chosen as an adequate way to well elicit respondents’ knowledge, for the respondents are not put under pressure of time, i.e. they answer questions in their own time and at their own pace, and in an anonymous style of responding they undoubtedly feel free and comfortable to answer questions (Gillham, 2000; Nachmias, 1996).

The respondents include 12 English language teachers of TNU, who are involved in the development process of oral tests. They are at the age of 27 to 50. They all had tertiary training in language teaching in different educational institutions throughout Vietnam.

The questionnaire consists of 10 questions, 8 of which are used to elicit the teaching staff’s perceptions of oral testing. These 8 questions are developed based on the theory in oral language testing reviewed in Chapter 2. The other two concern the staff’s working experience, and their qualifications in language testing. The questions are as follows:

Questions 1 and 2 relate to the functions of spoken language and the communicative approach to measuring speaking skill.

Questions 3 and 4 relate to oral test types and elicitation techniques used in a test of speaking ability.

Question 5 relates to grading test tasks and validity.

Question 6 relates to the design and operationalization process of oral tests, and reliability.

Questions 7 and 8 relate to reliability of a test.

In general, questions 1 to 8 are close questions, and questions 9 & 10 are open-ended.

For questions 1 and 2, the respondents are required to arrange the options according to their level of priority. Question 3 requires 1 out of 4 options. Questions 4 and 5 require the respondents to tick the level(s) of language proficiency that the elicitation techniques and particular questions fit. Question 6 and question 8 allow more than one choice. For question 7, the respondents are required to choose 1 out of 3 options.

Questions 9 and 10 are meant to investigate possible sources of the differences in the subjects’ perceptions of oral language testing.

Information regarding the existing practices of developing English speaking tests at TNU described in this sub-section plays an essential role in the critical evaluation of these practices. TNU current oral test development process is described in relation to such four main factors as (1) the test design stage, (2) the test operationalization stage, (3) the test administration stage, and (4) the use of test results.

First of all, the oral tests used at this institution have been formally administered at the end of each term in order to measure what the students have actually achieved after one particular time of learning. Such kind of test is called a final achievement test by Hughes (1989) and McNamara (2000), progress and grading test by Bachman & Palmer (1996), course test by Davies (2000), or final or attainment test by Heaton (1990). All the teachers or test writers as well as assessors have always known that they should elicit the students’ actual ability to use the language in real communication, and especially their language knowledge and ability they are required to grasp by the end of one particular term. Therefore, oral tests at this institution are explicitly identified as achievement ones from the very start, and both the teachers and the assessors have been aware of this.

However, neither the Department nor the English Section has produced a formal document including explicit classification of students’ language proficiency levels, particular areas of language ability or construct to be assessed and sets of TLU tasks identified for all the levels. Also, they have not established any criteria for test quality evaluation.

Secondly, as regards the operationalization process, the test designers have not kept in hand the official document mentioned above with descriptions of different levels of language proficiency in terms of the students’ language knowledge, i.e. a level scale (discussed in 2.3.1) and with areas of language ability to be tested. Additionally, the teachers or test writers have not received any detailed guidelines or instructions on number of test tasks included in the test(s) as well as test task specifications. Nevertheless, they have been informed about the administration time of the test(s) in advance in order to ensure punctual test production and submission.

Therefore, the teachers or test designers have freely produced the speaking tests in their own way, and most of the oral tests conducted make use of merely one oral test type – Tests where the learner prepares in advance – and one elicitation technique – oral report – and consist of one test task/part (See Appendix 1- these three tests were used for the same class for three terms in succession). Furthermore, as shown in these three tests, none of the test task/question is attached with neither external prompts helping the students make a structured presentation nor explicit instructions quantifying language knowledge and ability needed to perform the task.

Thirdly, concerning the administration process, there has been no detailed guidance in the form of either a meeting among the group of administrators and assessors or an official document regarding the students’ language proficiency level, i.e. level scale mentioned above, and no guidelines on method(s) of marking students’ performance on each test task. In short, the assessors are never informed of or provided with these two important things before test administration.

At the end of one term the oral test administration takes place with three classes of 60 students on average. The test administration for one class is allotted half a day and each class is usually divided into two groups of about 30 students in two separate rooms. The time for test performance of each student is about 4 to 5 minutes, hence. During test administration every 5 students are called into to draw lots for test questions or tasks to make a preparation for 5 to 10 minutes. Then each student presents his/her preparation in front of two assessors. The two assessors often raise some questions related or perhaps unrelated to the student’s presentation. Sometimes the assessors do not ask any questions. The student’s final score is taken from the average of marks given by the two assessors. Meanwhile, the other students waiting for their turn are standing along the corridor and talking, that is to say a supportive testing environment is not maintained.

Last but not least, after all the students finish their performance, test results are analysed to grade the students. However, the teachers are not provided with and are not allowed to keep the list of students’ test scores. The oral test results at this institution are not thus used to either evaluate the effectiveness of the instructional programs or modify the teaching methods and materials.

TNU oral test development described above can be reviewed by means of the checklist (Table 4.1 on page 39), specially designed with the purpose of highlighting the strong and weak points of the current practices. The answer ‘Yes’ is ticked ‘’ and ‘No’ is crossed ‘×’.

Table 4.1: A checklist for Oral Test Development

Table 4.1 partially speaks for TNU current oral testing practices with many ‘×s’, which reveals impropriety in the speaking test development at this institution and undoubtedly indicates a big gap between practice and theory.

Table 4.2 displays oral test types used during the administration of the end-of-term speaking test for the second-year students (Term 2 – School Year 2002-2003 – Appendix 2). The information from this table is intended for evaluation of oral test types in use in the next sub-section 4.1.3

Table 4.2: Summary of Oral Test Types Used in the Achievement Speaking Test for the Second-Year Students (School Year 2002-2003)

Table 4.2 indicates that only one test type was in use, yet as discussed in 2.4.2 (Chapter 2), a speaking test, namely an achievement one intended to measure overall oral proficiency, that can be believed to be valid should be a combination of various oral test types, at least two.

The following is Table 4.3 presenting elicitation technique(s) employed to elicit the 10 students’ ability during the achievement test mentioned above, their topic number or test question, duration of their test performance, their interaction with the assessors. All these details were recorded and transcribed in Appendix 3.

Table 4.3: Summary of the Students’ Oral Test Performance in the Achievement Speaking Test for the Second-Year Students

As can be seen in table 4.3, oral report, one of the three main elicitation techniques used to elicit test takers’ speaking ability through their performance on this kind of test – Tests where the learner prepares in advance (See 2.4.2.3, Chapter 2), was the only elicitation technique employed throughout this achievement test. Additionally, after most of the students finished their presentation, the assessors did not raise any questions except for students 3 and 9.

The evaluation of TNU current oral testing practices is carried out in relation to four factors described in 4.1.1: (1) test design stage, (2) test operationalization stage, (3) test administration stage, and (4) use of test results.

As can be easily seen in Table 4.1, oral tests are explicitly identified as achievement ones from the very start. Obviously, clear identification of test type at the beginning of a course proves to be beneficial because the teachers can integrate the test content into the teaching program. As pointed out by Brown (1994), Heaton (1990), Hughes (1989) and Ur (1996), achievement tests should be integrated into the teaching program and related directly to the classroom lessons or units, the syllabus or curriculum. Therefore, information or indication of students’ performance on an achievement test reveals their achievement or progress at the end of a course of study (Bachman & Palmer, 1996), and an achievement test of speaking skill is of course a means of eliciting students’ progress in overall speaking ability after a course/term of study.

However, a product of this stage involving such four crucial things as students’ profile of language ability, construct/ability to be measured, sets of test tasks in the TLU domain and a plan for test quality evaluation, as described in 4.1.1, has never been produced and presented to the teachers as a principled basis or guidelines for the other two stages. This undoubtedly indicates that the first stage of oral test development at TNU is far from being consistent with the theoretical framework reviewed in 2.3.1 – Chapter 2. As a result, this big mismatch leads to the staff’s improper practices in the other two stages.

Apart from the mismatch between practice and theory at this institution mentioned above, a remarkably essential fact shown in Table 4.1 is that the Department and English Section have not provided any specific guidance, i.e. a blueprint, for speaking test construction process, namely (1) the number of test tasks to be included in a speaking test, and (2) specifications of each test task. These two factors are critically analysed respectively.

Firstly, as previously discussed, an achievement test of speaking skill is a means of eliciting students’ progress in overall speaking ability after a course of study, yet most of the achievement speaking tests in use at TNU can be asserted to fail to serve this purpose because they make use of merely one type of oral test or one test task – Tests where the learner prepares in advance (Tables 4.2) - combined with only one elicitation technique - Oral Report (Table 4.3). This is partially because no blueprint is presented. Underhill (1987) points out that an oral test rarely consists of only one elicitation technique but it is usual that it involves several techniques placed in a sequence. The reasons he provides for including more than one technique in an oral test are as follows

(Underhill, 1987, p.38)

Probably, such test tasks have been carefully discussed in class, and the students are expected to produce ‘well-prepared’ talk, even predictable questions can also be prepared in advance. Of course, ‘the task(s) on which the student has to perform may be generally familiar in form to the student, but the student cannot ‘prepare’ a written version of what he will say’ (Brown & Yule, 1983, p.120). He must prove to the assessors that in his test performance he has learned to use, not to repeat, what he has been taught. What we as examiners want to know when testing a students is not whether the students has learned what to have been taught, but whether he is able to produce an extended piece of spoken English appropriate to the communicative situation he encounters (Brown & Yule, 1983, p.120).

Obviously, this popular kind of oral test at TNU is far from being useful in measuring the students’ overall language oral proficiency, and can be said to be lacking in construct validity and reliability (See 2.5, Chapter 2).

Secondly, no specifications of particular test task(s), especially specified components of oral ability to be tested, areas of language knowledge adequate and a marking key, to some extent, results in the teachers’ or test designers’ inadequate and useless tests. It can be said that there is lack of consideration of communicative stress in the oral test construction.

As can be seen in four achievement speaking tests (See Appendices 1 & 2), all the test questions/tasks – topics- are never accompanied with any external prompts helping the students make a structured presentation, and any explicit instructions quantifying language knowledge and ability needed to perform the tasks.

It is extremely necessary for test writers to provide clear instructions helping test takers to organise a spoken presentation for test performance because students are always encouraged to produce effectively organised speech so that the listener finds it easy to catch up with what is being said (Brown & Yule, 1983, p.119).

Also, in order to write test tasks fitting students’ proficiency levels, test writers need really give explicit instructions quantifying language knowledge and ability. The quantification of performance on a particular task much depends on the grading of tasks according to cognitive difficulty (Brown & Yule, 1983, p.121). To put in another way, the same task type can be made easier or more difficult. For example, describing a room with 8 elements is apparently more difficult than a room with 5 elements. Inevitably, test designers or teachers of speaking skill should always bear in mind informed judgements of the degree of this cognitive difficulty or communicative stress (Figure 2.2, Chapter 2) during test operationalization process.

Besides, no official instructions on criteria for marking students’ test performance are presented; thus, the test writers/teachers are unaware of the importance of scoring method(s) for each test task, and they never design a marking key (See 2.4.3, Chapter 2) instructing assessors how to assess students’ performance on test tasks. As discussed in 2.4.3, in a marking key, language and skill categories are identified and awarded separate marks according to test purpose(s). As Underhill (1987, p.94) points out the aim of a marking key is ‘to save time and uncertainty by specifying in advance, as far as possible, how markers should approach the marking of each question or task’. With help of a marking key and a level scale mentioned above, assessors can mark a test more quickly and reliably, for each language or skill category is expected to be separately marked.

Table 4.1 and 4.3 indicate that TNU speaking test administration reveals many a shortcoming. These weak points include (1) lack of test administration standardisation, (2) lack of reliability in marking test takers’ test performance, and (3) lack of supportive testing environment.

First, before test administration there has been no official meeting - named ‘the standardisation meeting ’ by Alderson, Clapham & Wall (1995)- for discussion and agreement on how to mark each question/task among the group of assessors. Perhaps the administrators here tend to think the assessors, as language teachers, must obviously know how to fully elicit the students’ oral proficiency, so they do not need to be informed of what to do during the test. Even when the assessors can be aware of the importance of this meeting, they are unable to hold it. It is partially because the staff’s insufficient knowledge of oral testing cannot help them to design an appropriate marking key and a reasonable description of mark categories with a mark criterion.

Therefore, before test administration, a marking key and a mark criterion for mark categories are first needed from test designers, and then a considerable amount of time must be spent on discussion to reach agreement on the way to mark each question/task. Alderson, Clapham & Wall (1995, p.112) maintain, ‘although this is likely to be expensive, it is the safest way of ensuring that enough discussion will take place for all examiners to understand thoroughly the level scale and the procedures for scoring.’ All these things aim at assuring reliability of an achievement speaking test.

Second, Table 4.3 reveals that, during the students’ test performance, interaction hardly existed between the assessors and the test takers or students apart from 2 students out of 10. These two students were asked 1 or 2 questions. Moreover, the duration of these 10 students’ test performance varies 2 minutes on average. As discussed in 2.1- Chapter 2, spoken language has two functions, interactional and transactional, which are both necessarily incorporated into a speaking test. In fact, in most of the oral tests in use at TNU, namely the achievement test mentioned above, the students are expected to merely produce transactional instances of the language. Can such tests be considered to be able to measure test takers’ or students’ overall oral proficiency? The answer is surely no because they reveals no interactive communication. This also means that the assessors gave scores just on the students’ presentation, which also surely indicates a lack of validity and reliability (See 2.5, Chapter 2).

Last but not least, as regards a supportive testing environment the oral tests were almost administered in noisy rooms. Students should be put at ease before and during their performance, which can increase their confidence. Bachman & Palmer (1996), hence, demonstrate that it is crucial to maintain a supportive environment throughout the test, that is to avoid distractions due to temperature, noise, excessive movement, etc. In order to do this, test administrators and assessors are to be in control of techniques and create an atmosphere which will help each student to feel at ease (Alderson, Clapham & Wall (1995, p.116). For those students waiting for their turn should be sitting in a comfortable room, not standing along the corridor and talking so as not to affect the others’ performance.

The last factor under evaluation involves ways of how test results or students’ final scores in test performance are used. As previously described in 4.1, students’ oral test scores are used to grade them in terms of their progress or achievement after a term/course of study. This is the most popular and common purpose of all achievement tests, that is to make the final decision on students’ proficiency kept in their study record in the form of grades. Furthermore, teachers and students are really interested in receiving feedback on students’ progress which helps students ‘guide their own subsequent learning’, and helps teachers ‘modify their teaching methods and materials so as to make them more appropriate for their students’ needs, interests and capabilities’ (Bachman & Palmer, 1996, p. 98). However, TNU students’ test scores have never been used to either evaluate the effectiveness of instructional programs or make any improvement in teachers’ teaching methods and materials. In other words, oral testing at this institution has no effect on the teaching and learning of speaking skill which is named negative washback or backwash by Hughes (1989), Heaton (1988) and McNamara (2000). Information regarding inferences about students’ proficiency made from test performance can be really useful for assessing the efficiency of a teaching program as well as teachers (Bachman & Palmer, 1996, p. 98).

In conclusion, the analysis of TNU current practices of developing oral language tests reveals a number of weaknesses as follows:

There is no principled basis for oral test operationalization and administration

Oral tests in use lack construct validity and reliability

There is lack of consideration of communicative stress in oral test operationalization

There is lack of test administration standardisation

There is lack of a supportive test taking environment

These current practices are thus far from being consistent with the theoretical framework for test development.

This sub-section first provides the list of the correct answers to the questions and the results collected from the survey.

List of the correct answers to the questions

The correct answers for the questions of the questionnaire are provided in Table 4.4 below.

Table 4.4: Correct Answers for the Questions in the Questionnaire

As far as the two functions of spoken language concerned, criteria for assessing test takers’ interactional and transactional short turns, as discussed in 2.1 (Chapter 2), are more based on their communicative reaction and successfully negotiated ideas rather than on content, size, cohesion or coherence like in taking transactional long turns. Thus, the correct order for Question 1 is d-f-h-e-b-c-a-g, and the correct order for Question 2 is i-h-f-d-g-e-b-c-a. However, every two positions of the last fourth (Question 1) and of the last fifth (Question 2) positions can be exchanged, but the first fourth positions cannot be put in the last fourth (Question 1) or the last fifth (Question 2) positions. That is to say, the order d-f-h-e of question 1, and i-h-f-d of question 2 cannot be changed, and these options must be given the highest priority based on the major features of communicative language testing, that is, interaction efficacy is the assessment prerequisite not accuracy of language forms.

For Question 3, related the number of test tasks/parts included in a test of overall oral proficiency, it is necessary to make use of at least 2 elicitation techniques representing 2 oral test types as demonstrated in 2.3.2 (Chapter 2). The correct answer to this question, hence, is either b, c, or d.

Concerning grading test tasks according to degree of communicative stress reviewed in 2.1 (Chapter 2), elicitation techniques suitable and adequate for each level of proficiency in Question 4 are as follows: Year 1: b, c, d, e, h, i; Year 2: a, b, c, d, f, g, h; and Year 3: a, b, c, f. Similarly, specific tasks in Question 5 used to measure speaking ability for each level are identified as follows: Year 1: b, e, i; Year 2: a, d, g; and Year 3: c, f, h.

As reviewed earlier in 2.4 (Chapter 2), all the steps, in Question 6, must be involved in the oral test design and operationalization process. Therefore, all the options of this question are to be chosen.

The data collected from the questionnaire

Table 4.5 on page 49 presents information regarding the subjects’ own criteria for assessing test takers’ performance on interactional and transactional short turns (Question 1).

Table 4.5: Teachers’ Assessment Priority Perception of Interactional and Transactional Short Turns

Most of the respondents (11/12), as shown in Table 4.5 give option (g) the lowest priority in their assessment criteria for interactional and transactional short turns.

Table 4.6 displays the respondents’ perception of priority in assessing students’ test performance on transactional long turns (Question 2).

Table 4.6: Teachers’ Assessment Priority Perception of Transactional Long Turns

Table 4.6 reveals most of the respondents (11/12) also give option (g) the lowest priority in their assessment of transactional long turns.

Table 4.7 displays all responses the respondents have chosen for Question 3, concerned with the number of test tasks/questions included in an achievement speaking test.

Table 4.7: Teachers’ Choice of Number of Tasks for a Speaking Test

Table 4.7 indicates that 8 out of 12 respondents think a speaking test should make use of two or more tasks/parts.

Table 4.8 reveals the subjects’ choice of elicitation techniques for their tests of speaking (Question 4). All the elicitation techniques in the questionnaire are popularly used in tests of oral ability. The appropriate elicitation techniques for each proficiency level, as previously asserted in Table 4.4, are provided in the brackets right below each option.

Table 4.8: Teachers’ Choice of Elicitation Techniques for Levels of Proficiency

As shown in table 4.8, only 2 out of 12 respondents choose (d) for year 1 and (c) for year 2, and half of them choose (g) for year 1.

Table 4.9 on page 51 displays the respondents’ selection of specific tasks used to measure test takers’ speaking ability at each proficiency level (Question 5). The particular tasks adequate to the three levels, as given in Table 4.4, are presented in the brackets next to each option. For example, a(2) – a refers to one option of the question and (2) to the second level of proficiency (Year 2).

Table 4.9: Teachers’ Choice of Specific Test Tasks for Level of Proficiency

Table 4.9 reveals that only 2 out of 12 respondents use (e) for year 1, 3 use (d) for year 2, and 2 use (h) for year 3. In addition, up to 9 of them use (c) for year 2, and 10 use (d) for year 3.

Table 4.10 presents responses selected for Question 6, associated with the procedure for oral test design and operationalization. The correct answer, as given in Table 4.4 is selection of all the options.

Table 4.10: Teachers’ Choice of Steps to Be Considered in Oral Test Design and Operationalization

Table 4.10 shows that only half of the respondents (6/12) consider option (e) to be essential in the construction process of oral tests.

Table 4.11 reveals information regarding the respondents’ reliability degree within their own inferences about students’ oral test performance (Question 7).

Table 4.11: Teachers’ Confidence in Students’ Test Results

As shown in table 4.11, 10 out of 12 respondents do not believe that their marks given on the students’ oral test performance can reveal the students’ actual interactive ability.

Table 4.12 presents responses selected for Question 8, used to elicit the reason(s) why the respondents are not or not very sure of the marks they have given on the students’ oral test performance.

Table 4.12: Teachers’ Lack of Confidence in Students’ Test Results

A remarkable thing shown in Table 4.12 is that those who are not sure of the reliability of students’ test scores all believe that their uncertainty results from the students’ prior preparation for test tasks.

Personal details about the respondents from questions 9 and 10 provide important facts:

This sub-section analyses the data gathered from the questionnaire survey. The information revealed in Tables 4.5 to 4.12 above can be generalized as follows: (1) most of the teachers are unable to distinguish the two functions of spoken language, (2) the majority of the teachers do not have a sufficient understanding of communicative language testing, (3) quite a number of the teachers fail to recognize the difficulty level of test tasks graded according to communicative stress, (4) the teachers have an inadequate understanding of oral language testing, and (5) the speaking tests in use lack authenticity.

Firstly, most of the teachers fail to distinguish the two different functions of spoken language in their own assessment criteria as discussed in 2.1 (Chapter 2). In particular, for Questions 1 & 2 (Tables 4.5 & 4.6), 11 out of 12 subjects give the lowest priority to option (g), that is the ability to speak at length. When taking transactional long turns, students are required to show their ability to express and convey their ideas at length. Nevertheless, the teachers at TNU hardly expect to see this feature in their students’ oral test performance.

Secondly, the notion of communicative language testing is not fully grasped and soundly engraved on many of the teachers’ mind. In this preferably applied approach to language testing as mentioned in 2.2 (Chapter 2), accuracy of language forms has turned out to be given a lower priority than communicative effectiveness in assessment. As revealed in Tables 4.5 and 4.6, the majority of the teachers, in their assessment criteria, give a high priority to options (a), (b) and (c) of both Questions 1 & 2. In particular, for Question 1, option (a) is given such a high priority by 4 subjects, (b) by 5 and (c) by 7; and for Question 2, option (a) by 5, (b) by 8 and (c) by 8. This implies that on average half of TNU English teaching staff give linguistic accuracy a high priority when evaluating their students’ production of spoken language.

Thirdly, as shown in Table 4.7, for Question 3 option (a) has been selected by only 4 subjects, which means that the majority are able to realize the need to include more than one task or elicitation technique in a test of overall speaking ability as discussed in 2.3.2 (Chapter 2). However, in their selection of proper elicitation techniques or tasks for each level, they reveal their inability to recognize the difficulty level of the given elicitation techniques according to communicative stress (Figure 2.1 – Chapter 2), particularly the difficulty of the tasks themselves. To prove this, as shown in Table 4.8, only 2 out of 12 subjects have chosen (d) (the correct answer) for the first level and (c) (the correct answer) for the second level while 6 subjects have chosen (g) (the wrong answer) for the first level.

Besides, this failure is more strongly confirmed by the information gathered from Question 5. As shown in Table 4.9, only 2 out of 12 subjects have chosen (e) (the correct answer) for the first level, 3 have chosen (d) (the correct answer) for the second level, and 2 have chosen (h) (the correct answer) for the third level whereas up to 9 of them have chosen (c) (the wrong answer) for the second level, and 10 have chosen (d) (the wrong answer) for the third level. This result suggests that most of TNU English teaching staff are unable to recognize the difficulty level based on the relationship within each specified task as mentioned in 2.1 (Chapter 2).

Fourthly, as regards all the steps to be sufficiently considered in the development process of speaking tests as described in 2.4 (Chapter 2), only half of the subjects (6/12) have selected option (e) (Table 4.10), and the other options have not been chosen by all the subjects. This partly reveals the subjects’ incomplete grasp of language test development. As shown in Table 4.7 (Question 3), there is a great agreement on combining more than one kind of elicitation techniques in a test of overall speaking ability (8 subjects/12). However, in relation to the requirements in the development process of a speaking test, half of them have failed to recognize the importance of identifying the number of test tasks or elicitation techniques included in a speaking test.

Lastly, concerning the usefulness of the oral tests in use at TNU, 10 out of 12 subjects, as revealed in Table 4.11 (Question 7), do not entirely believe that their marks given on the students’ oral test performance could reveal the students’ actual speaking ability. These 10 teachers have all chosen option (a), as shown in Table 4.12, which implies that in most of the oral tests the students were informed of the topics or test questions during the class time, and they were well prepared for what they were going to speak before the real tests actually occurred. Such speaking tests can thus be said to lack authenticity, defined as ‘the degree of correspondence of the characteristics of a given language test task to a target language use task’ (Bachman & Palmer, 1996, p. 23), because they do not represent real life language use. Therefore, many of the teachers (7/10) think that the elicitation techniques commonly used in most of the tests are unable to elicit the students’ actual speaking ability (option c). In addition, half of them (5/10) have selected the other two reasons for their low confidence in their own given marks. For the first reason, perhaps they think the students could even guess and prepare what they could be asked about as in many cases the topics or test questions were given beforehand. For the second reason, they have had no specified assessment criteria or guideline to base their marking on as described in 4.1.1. As a result, they have had to ‘design’ their own criteria.

To sum up, the analysis of the data collected from the survey shows that TNU English teaching staff have obtained a limited understanding of oral skill assessment. Obviously, their limited and insufficient grasp of oral language testing may probably lead to their low confidence in their scores given on their students’ oral test ability. Also, their improper practices of developing speaking tests critically evaluated in 4.1.3 must be an inevitable consequence of their incomplete perceptions of speaking skill assessment.

The Pedagogy Department or the English Section should produce an official document including the following suggestions this thesis attempts to make as the first effort to standardise TNU current oral language testing. These recommendations for improving the current development procedure include:

A blueprint which has been discussed in 2.3.2 (Chapter 2) includes the number of test tasks/parts and specifications of each task. The blueprint is suggested as major guidelines for TNU staff to construct their speaking tests.

As discussed in 2.4.2 (Chapter 2), a test of speaking ability that enables assessors or examiners to elicit a test taker’s overall oral proficiency should consist of at least two tasks or elicitation techniques. Undoubtedly, a speaking test making use of two or more tasks or elicitation techniques is said to have construct validity and reliability. Regarding the speaking tests of several popular published exams, a test of overall oral ability always has two or three tasks/parts. For instance, BEC (Business English Certificates) and IELTS (International English Language Tests) have a three-task speaking section, or Let’s Talk presents speaking tests involving two tasks/parts. Thus, it would be advisable to design a blueprint for speaking tests at TNU including two tasks or elicitation techniques.

The following are the suggested components of the blueprint for an achievement oral test.

1. Test structure

1.1 Number of tasks/parts:2 tasks

Language, as discussed in 2.1 (Chapter 2), has two functions, so assessment of students’ ability to use the language orally is to involve these both functions. Furthermore, length of spoken language production is also the basis for this kind of assessment (See Figure 2.1, page 6).

Task 1:Interactional and transactional short turns

The purpose of this task is to evaluate students’ progress in taking interactional and transactional short turns.

Task 2:Transactional long turns

The purpose of this task is to evaluate students’ progress in taking transactional long turns.

1.2 Relative importance of the tasks

This importance ranges on the continuum of spoken language production (Figure 2.1) according to students’ levels of language proficiency. For instance, in a test for first-year students, Task 1 is more important than Task 2.

2. Test task specifications

The purpose of the task

The specified components of oral ability to be tested

The place where the task occurs

Specified and understandable instructions

Expected duration of task performance

Areas of linguistic, pragmatic and topical knowledge adequate

Marking key

Concerning criteria for scoring, based on the discussion on a marking key in 2.4.3 (Chapter 2), the researcher suggests combining the two models of mark categories, i.e. the traditional model and the model of performance criteria, since doing so means that linguistic accuracy is not neglected and objectives of the course or the teaching are not thus neglected either. That is to say, we as language teachers never want to neglect linguistic forms when instructing students. Therefore, this thesis recommends the marking scales adapted from PET Speaking Test by University of Cambridge Local Examinations Syndicate, representing a combination of these two model. These marking scales are used in the marking key of two sample achievement tests introduced in the next sub-section 5.1.2

The following two recommendations hopefully help the test writers to visualize how to write a speaking test for TNU language students in general and for first-year students in particular.