Understand oral English testing in schools better

1. According to research, it is too difficult for highly trained examiners to measure English proficiency spanning more than 9-10 levels. (Covering a scale of no ability to highly proficient.)

2. Even measuring 10 levels professional examiners can be wrong 27% of the time.

3. On such a scale, for classes that meet once or twice a week for 45-minutes, it may take a full year of training or more to improve one level

4. Factors weigh on the whole process which can cause inaccuracies. During some research on IELTS training it was found that after a 3-month intensive training candidates could improve half a level. However, some candidates actually scored at a lower level at the end of the training than they did at the beginning. Reasons for this were the state-of-mind of the candidate on the test day, familiarity and lack of familiarity with the subject matter of the tests, faults in the testing system (IELTS), etc. So it would be possible to test your student at the beginning of the course and then the student could do worse at the test at the end of the course. How would you give a grade in this case?

Alternatively, test the students on the course material at the beginning of the course. If the students are properly placed in the right level classes they should score very low on such a test but after training on the material should score very high at the end of the course.

In that it is too late at this point, the next best thing is to review the material we taught the students and find some distinct and important points that we can test on. However, if the classes consisted of "well, what does everybody want to talk about today?" it may be impossible to test the students.

Finally, the idea of scoring by student attitude or participation in the class or other student behavior focused method, to my mind, but perhaps many would disagree with me, is fraught with the most disadvantages. Such a criteria makes the test exactly one of classroom behavior, not learning or language acquisition. As the focus of most of us teachers is on the student centered classroom we are constantly questioning ourselves if we are meeting the needs and capturing the interests of our students. Some days we do better than others. Some students (especially the very bright and the very dull) are more easily bored than others. We are tempted to punish our troublemakers and reward our so-called 'good' students by the score we would give them. But again, we would be scoring the student on their behavior which is perhaps not the best way to reflect their learning in our class.

The irony is that for many schools it really doesn't matter what score you give the students because the school often does not treat the oral English class taught by a foreigner as a 'real' class. These grades often don't show up as part of their year-end scores.

Nonetheless, I think we should all be careful in how we go about making these kinds of decisions. There are certain 'automatic' impulses that we should be aware of and question. How often do we do things because that's the way everyone else does them? How often do we do things because that's the way it was done when we were students?

These things I have said in the above are with the realization that they may not exactly apply to the certain aspects of the current discussion. Without complete understanding of the situation I know I may be misunderstanding some things. I am just trying to explore various factors and considerations for the purpose of reflection.

We are the teachers and in a position of power with the students and the school. It is a great opportunity to for us to explore and discover the best ways to do things for our students and ourselves.

High-speed correcting

If you find yourself with a large number of tests to correct and little time to do them you could make a cut-out template to help you.

What I like to do though is to make a recording of the answers on my computer, using the 'sound recording' feature under Windows accessories. I then play the recording, sometimes even speeding it up to 200%, and check the papers.

The advantage of recording it on the computer as opposed to a tape recorder is that you don't have to rewind to play it again.

How students learn for tests

When our students take the big exams are the only questions they get right the ones the teacher "taught" them? I don't think so. I would like to know how effective is teacher "teaching" as compared to indirect learning.

I think they are answering some questions on the test correctly for items that they were not "taught". If so, then how did they learn them? I believe Comprehensible Input is playing a bigger role than we realize.

Krashen tells the story of how his French teacher wanted to only speak French to them and was explaining a grammar point, in French. Finally frustrated, she told them in English. However, her effort to explain it in French, all that French speaking to explain something, actually constituted Comprehensible Input for the students and helped their French.

Every time the teacher talks to the students in the L2 is Comprehensible Input. Teachers are naturals for adjusting their English speaking so students can understand them.

So between the teacher's speaking and the student's own study they are getting a lot of CI.

Perhaps the student is reading a business text and it is talking about international finance and the teacher wants the student to learn some language about stocks, bonds, interest rates, prime lending rate, etc. Perhaps the student has some degree of success in learning some of those terms but there are many things in the text that the student was not studying but was learning such as "carry on" when it says "banks cannot carry on making risky loans" or something like that.

The Bob System: Tracking students for formative assessment

I am often called on to teach oral English. Unlike teaching written English where the students will be submitting a lot of writing samples, oral English offers less opportunity to sample the students' English ability.

My primary interest in using the Bob System and some sort of scoring system is in formative assessment.

When I have a clear understanding of how they are doing then I have the ability to try to make my training more effective in two ways.

First, are my students "getting it"? Am I helping them to learn what will be useful for them to know?

Second, I can customize my training more to my students' specific needs. I may not be able to give each student individual training (that ability and technology will be coming in the future) but I could segment the class. I want to know who is doing well and who is doing poorly.

When I know this I can offer extra training to those who need extra help. What about students who are doing very well in the class? Sometimes there is an academic ceiling in the classroom. Bright students cannot go higher because the teacher is teaching to the "middle level" of the class. But if we know which students are doing very well and how many of them there are then we can focus on their needs better by providing extra challenge.

Tracking this sort of information can be very useful in other ways, as of Action Research in the classroom. If you are monitoring many aspects of the student's performance in the classroom and you have a student who always participates correctly, does the pairwork, groupwork, homework, listens and doesn't goof off but does not seem to progress in their English from one term to the next then that would raise some very good questions for the teacher.

Of course, finally, the data that is collected can help in summative assessment. The teacher does not need to simply rely on a final exam for a score. The teacher will have a multidimensional way to look at the students.

[Photo: Some of my 300 college students that I taught weekly last term doing pairwork. Next term I will have 400 college students each week.]

Speaking evaluations made simple

This is a very complicated subject. It is not easy to conduct a speaking test but I will go over just a few things about it and touch on them lightly. There are many ways to do speaking tests. I have studied them, tried some of them and have settled on this way. It is similar to the way I was trained as an IELTS examiner with a few differences.

QUESTIONS

Design three levels of questions.

(1) Easy questions which are answered with straight factual answers. "Where are you from?" "How long have you been here?" "What did you do yesterday?" "What do you like to do on the weekends?" These questions make little demand on the student and only very low-level students will have problems with these.

(2) Moderately difficult questions demand more from the student. These are questions asking a student to describe a city or restaurant, relate the story of a movie recently seen or a book recently read. "Tell me about your last holiday? "Describe your best friend."

(3) Difficult questions are those that require the student to give an opinion and justify their opinion with reasons. "Should students be required to wear school uniforms? Why?" "Should smoking be banned in all buildings? Why?"

Be aware that some questions are not only difficult to discuss in English, sometimes they are just plain difficult to discuss at all. I once designed a question, "If you had two weeks to live, what would you do?" This question was so deep that the students became extremely thoughtful in trying to give their answers to the point that it interfered with any attempts to show fluency. Questions do not need to be so deep.

Although the question may be difficult at times, to understand the question should be simple. Remember, this is a speaking test, not a listening test. For example, "Given the opportunity to go on a round-the-world cruise or participate in a scientific exploration in Africa, which do you think could potentially be more beneficial for your career development?" Many low and mid level students would not be able to understand that question and therefore would not be able to speak on it. Make sure your questions are easily understandable.

I like to let the students ask each other the questions. This way I can focus on listening and evaluating. But I do not allow the students to prepare for the questions except for perhaps just a couple minutes before the interview.

Look me in the eye? In western countries we have no problem looking into people's eyes when speaking to them but this is something that Asians do not do. Therefore, when you conduct the speaking test with Asian students it is best to not try to look deeply into their eyes or to hold their gaze. Look elsewhere, shift your eyes around or even just focus on your band descriptors or rubric.

EVALUATION

You can use a rubric or band descriptor to measure the student's level such as the IELTS band desciptors or the Common European Framework.

You will notice in the IELTS descriptors that at Band 4 it says:

"Is able to talk about familiar topics but can only convey basic meaning on unfamiliar topics and makes frequent errors in word choice. Rarely attempts paraphrase."

and then at Band 5 it says:

"Manages to talk about familiar and unfamiliar topics but uses vocabulary with limited flexibility. Attempts to use paraphrase but with mixed success."

That is why it is important to design your interview questions with easy, moderate and difficult topics so that the student will have to try to produce a full range of English at different challenging levels to respond accurately. The English of many students will begin to break down at the higher levels and this will allow you to see the limit of their English.

I put the band descriptors and all the students names on an Excel spreadsheet. I give the student a score for each rating catagory (Fluency and coherence, Lexical resource, Grammatical range and accuracy)
and the program averages it out into a final band score. Depending on the situation I will add formulas to work that score into a grade, average all the scores to compare one group with another or other things. Click on the picture (above) to see it enlarged.

ACCURACY

The more realistic the task is, talking naturally about a topic the student may actually need to discuss rather than some sort of T/F or multiple choice, the more difficult it is to test. So this sort of test will always be subjective, affected by your personal judgment of the student's performance.

One thing that helps is to be sure to base your judgment as closely as possible on the rubric or band descriptors you are using. You should never compare students to each other. This will lead you off the track. Always compare to your chosen rubric.

I always record my test interviews. A couple days later I will listen to some of the interviews and rescore them without looking at the score I gave the first time. If there is a strong correlation then that is good. If you find that you are scoring much differently the second time then you need to try to understand why and may even need to rescore all your interviews. It happens that you can be in a certain mood that will cause you to score differently. (Another good reason to record is to contribue to a record of the student's progress.)

IELTS research has even shown that male interviewers will sometimes give attractive females a slightly higher score which leads to inaccuracy. If the interviewer is tired, sleepy, hungry or if the interviewer has scored several high level students in a row and suddenly gets a low level student it can affect his accuracy. To run an effective test you need to be aware of all of these things and try to guard against them effecting your judgement.

Organizing speaking tests for large numbers of students

Yesterday, I did a speaking test for 200 students. It's quite a big job and took me all day. But it would have taken much longer if I had never done this before and if I wasn't organized.

Some teachers allow students to choose their partner and choose their subject. Sometimes they can do this days in advance. Consequently, some students will find a dialog and memorize it and then perform it for the test. I don't do it this way. I don't think it's very realistic.

In the workplace, people need to be able to speak English to anyone and they can't always choose the topic. So I tell my students they can choose any partner they want as long as student #1 chooses students #2 and student #3 chooses student #4, etc. I tell them they can choose their own topics to talk about. They come up to my desk and choose from several slips of paper which are facing down. The students cannot look at the paper before they choose. Consequently, they are choosing randomly. I do allow them 3-4 minutes to prepare before the interview. The topics are always things that we practiced discussing in class.

When one pair of students sits down to do their dialog for me another pair of students will come up to the desk, choose a topic and stand aside to prepare. I don't allow them to use dictionaries, notebooks, textbooks or to talk with other students during the preparation period. I sit facing the two students who are talking but behind them I can keep an eye on the next two students who are preparing for their talk. This is important because they many of them can hardly keep themselves from a bit of cheating if it is possible.

The students talk to their partner on the topic. I used to be an IELTS examiner and found it is a bit extra work to have to also be asking the students all the questions. So I like to get the students talking with each other and I listen in. If I think a student can go higher or if their dialog was too short I will jump in with a few extra questions.

While they are talking I am recording them. They hold a cheap $2 clip-on microphone that I found which works really well and is fastened at the end of a ballpoint pen. I use free program called Audacity to record the interview. Although I'm sitting right next to them, I actually listen to the students through a set of earphones. This ensures that everything is being recorded. The interview is recorded for reference in case I want or need to go back and check something or if I need to justify a score I have given.

If their dialog is too short or doesn't reveal their English skills well enough I will ask some questions to make them speak more. Answering"why" questions or questions where they have to explain or justify a viewpoint are some of the toughest questions and are good to push students to the limits.

I have some band descriptors and a list of all the students names and numbers on an Excel sheet. The band descriptors are divided into three areas of speaking (communicative range, overall fluency, accuracy & appropriacy) and four levels (levels 4-7 of a 10 level rating system) While they are talking, I scan the band descriptors and give them an initial score. I continue to listen to them and modify parts of my score as they perform better or worse.

It's very important to have a clear set of standards that the students should speak to. And while they are speaking you should be constantly checking those standards and try to measure the student to those standards as best you can.

After all of the testing is done, I will use Excel to average out the band scores and assign grades on a curve.

Does China have the worst English teachers in the world according to international test results?

Some teachers tried to tell me that China has the worst teachers and students based on the international test results of the Cambridge BEC tests. Is that true? Can we use those tests results and other tests like IELTS and TEOFL to see where the best and worst English teaching is being done?

Here are the statistics for 2004 sorted from lowest pass rate to best pass rate, or you could say from worse to best.

As you can see, China did indeed score the worst of all the other countries. Does this mean China's teachers are the worst? There is an easy test we can apply to find out the answer.

If these statistics show us that China's teachers are the worst in the world then they should also help us to see which teachers are the best in the world.

China, People's Republic of 40%
Indonesia 41%
Vietnam 45%
Brazil 58%
Hong Kong 59%
Italy 67%
France 68%
Spain 70%
India 71%
United Kingdom 74%
Argentina 78%
Bangladesh 79%
Czech Republic 79%
Switzerland 80%
Croatia 81%
Poland 81%
Russian Federation 81%
Germany 84%
Austria 86%
Portugal 89%
Canada 93%
Slovenia 95%

Is it true that Slovenia has the best teachers in the world? They scored a 95% pass rate! Incredible! Even better than Canadian teachers. How did they do that?

And what about British teachers? Is it true that Argentinean teachers are better than British teachers?

I'm afraid we can only use the pass/failure rates of the BEC (and also the IELTS) to show us the rates of those who took the test and passed or failed. It does not reflect the language ability of students in general and the teaching ability of teachers in general.

People take this test for different reasons, different goals at different ages. Only if the BEC was given to ALL High School graduates or at least a true random sampling of students who were ALL at the same level could we use them to try to interpret teaching quality. These statistics from the BEC do not answer the question on the English teaching ability of teachers in China.

But this does not mean Chinese English teachers are good. It only means these particular statistics are not going to be useful in the question.

As I said, it doesn't really matter if China came in last in the BEC tests as this sort of test is not going to tell us anything about the teaching or learning skills.

Two-course strategy to teaching IELTS

My strategy for teaching IELTS is:

1) ENGLISH SKILLS: If there's enough time, work to improve the student's English level through English training.

2) TEST PREP: Help the student gain an understanding of how the IELTS test functions. There are various books that deal specifically with the IELTS, how the speaking part works, the two tasks of the writing part, etc. (I always teach my students how to do a 5-paragraph essay for the 250-word Task 2.)

I think of it as two courses and think it's best taught as two courses. The first one is actually teaching English and the second one is teaching how the functions of how the IELTS test works.

ENGLISH SKILLS TRAINING

I have found that Interchange by Cambridge Universtity Press hits on almost all of the types of content that IELTS does such as the environment, education, work, news, movies, food, people, etc. In addition, it is teaching skills for reading, writing, listening, speaking. There are some examples below from New Interchange level 3.

IELTS TEST PREP SKILLS

The second course can be a short and it's rather easy for the students to get the point of how different parts of the test work. Once they have this test-prep course they should understand it. I believe this one bit of training can help a candidate improve their score by one band level. I suppose some people would question that assertion so let's look at it another way. This training can help a candidate avoid making mistakes that could cost him one band level.

Now if they should fail to get the score they want in IELTS they do not have to take this second course, the test-prep course, again. Not scoring high enough, means that their English level is not high enough and they need to work on their English skills which is a much bigger job.

For this effort I have used various test-prep books for IELTS. All of the ones I have tried have been useful but I guess I wasn't relying on them so completely, using them more as a framework to work from, as I have a lot of things about the test that I have learned and use that as a resource when teaching IELTS. So I have no strong recommendations to make on IELTS test-prep books but am interested to hear other's recommendations on those as well.

SOME TOPICS INTERCHANGE COVERS:

DESCRIBING PEOPLE
Personality types and qualities; relationships; "turn ons and turn offs". Describing personalities; expressing likes and dislikes; expressing agreement and disagreement; complaining Relative descriptions of people; making inferences. Writing about a best friend. "Friends Again – Forever!": Reading a narrative about friendship. "Personality types": Interviewing a classmate to find out about personality.

TALKING ABOUT WORK, JOBS, EMPLOYMENT, CAREERS
Unusual and exceptional jobs; job skills; summer jobs. Giving opinions about jobs; describing and comparing jobs Gerund phrases as subjects and objects; comparisons with -er / Writing about career advantages and disadvantages. "Strategies for Keeping Your Job": Reading advice about behavior in the workplace. "The best and the worst": Finding out about classmates' summer or part-time jobs.

DISCUSSING NEWS AND CURRENT EVENTS
The media; news stories; exceptional events. Describing past events; narrating a story. Listening to news broadcasts; listening to a narrative about a past event; making up stories. Writing a newspaper story. "Strange but True": Reading tabloid news stories. "A double ending": Completing a story with two different endings.

DESCRIBING COUNTRIES AND CULTURES
Cultural comparisons and culture shock; customs; tourism and travel abroad. Expressing emotions; describing expectations; talking about customs; giving advice Noun phrases containing relative living listening to descriptions of and opinions about customs. Writing advice for a visitor to your country. "Culture Check" Reading and completing a questionnaire. "Culture clash": Comparing customs in different countries.

DISCUSSING THE ENVIRONMENT AND GLOBAL PROBLEMS
The environment and world issues. Identifying and describing problems; offering solutions. Listening to people talk about problems, solutions, and accomplishments. Writing about local issues and offering solutions. "The Threat to Kiribati: Reading about an island that is sinking into the sea. "Community planner": Solving some small-scale environmental problems.

TALKING ABOUT THE PAST AND HISTORY
Historic events and people; biography; the future. Talking about historical events; giving opinions about the future. Listening to historical facts; listening for opinions about public figures; listening to predictions. Writing a biography.
"The Global Village": Reading about political and technological changes. "History buff": Taking a history quiz.

DISCUSSING PERSONAL LESSONS AND EVENTS
Milestones and turning points; behavior; regrets. Describing yourself in the past; describing regrets about the past; describing hypothetical situations. Listening to descriptions of important events; listening to regrets and explanations. "If You Could Do It All Again": Reading about three people's life choices. "If only . . . .": Imagining different possibilities for yourself.

Evaluating speaking and validity

A teacher, who has his students do a class presentation as the sole English speaking evaluation task, defended his approach by saying: "I guess this just shows that there is more ways than one to skin a cat - the principles of public speaking are things that we can use every day in our lives and, if it helps to build up the confidence in the students to use their English, then it seems as good a technique to use as many of the others."

Indeed, there is more than one way to accomplish a task. But there are also good ways and bad ways, effective ways and ineffective ways.

I can't say for sure but I'll make a guess that you haven't taken a TESOL course or had any formal training to be an English teacher. I'll admit my formal training is skimpy but I study as much as I can.

Brown & Yule explained some of the challenges of speaking. "...it should also be borne in mind that even native speakers of English find that a straight description is easier, in some sense, than telling a story and, in turn, that telling a story is easier than a justification of an opinion....This is a rather general guide to the level of difficulty. Naturally, a short narrative involving a single character and only two or three events may be easier than a lengthy description covering many details and relationships."

Here is a list of examples of speaking that Brown & Yule point out:


1. Static relationships

a) Describing an object or photograph
b) Instructing someone how to draw a diagram
c) Instructing someone how to assemble a piece of equipment
d) Describing/instructing how a number of objects are to be arranged
e) Giving route directions

2. Dynamic relationships

a) Story telling
b) Giving an eye-witness account

3. Abstract relationships

a) Opinion-expressing
b) Justifying a course of action


If you are familiar with the IELTS speaking test you can see these elements in the test. First the examiner asks some basic information or a description of something. Then to tell a story of something like a holiday you took. Then finally he will ask for some opinions on some topics.

He works from easier tasks to more difficult tasks. Students should begin having difficulty at some point during the test by which the examiner can see the student's level of competence.

Of course, as Karen pointed out, there are different kinds of tests and what we are talking about are not IELTS English competency tests but more like a pass/fail type test.

But my point is that there are a lot of different demands English speaking tasks make and it's important to match the task to the need and to achieve validity.

Brown explains: "The general concept of validity was traditionally defined as 'the degree to which a test measures what it claims, or purports, to be measuring' (Brown, 1996, p. 231).

"Validity was traditionally subdivided into three categories: content, criterion-related, and construct validity (see Brown 1996, pp. 231-249). Content validity includes any validity strategies that focus on the content of the test. To demonstrate content validity, testers investigate the degree to which a test is a representative sample of the content of whatever objectives or specifications the test was originally designed to measure."

So a test of General English would not be valid for Business English students. A Public Speaking test would not be valid for General English students. Would it be good for General English students to be competent in Public Speaking. Probably. Does the ability to speak publicly show competence in a general English speaking situation? No. Could the specific anxiety generated by speaking in public interfere with General English production? Absolutely.

General English requires the ability to negotiate meaning, explaining, understanding when they are not being clear to the listener and using appropriate strategies (circumlocution or defining terms), turn-taking, etc.

To this the teacher questioned: "Why??? What support do you have for this statement. In my years in China, the correlation is pretty strong, good english communicators give good speeches. This seems natural."

I meant invalid as far as content validity is concerned. Are you familiar with content validity?

As explained by Underhill the question is:

"It is relevant? Do the items or tasks in he test match what the test as a whole is supposed to assess? Where the objectives of the programme are set out in detail, for example in a syllabus that lists skills or functions, then the content validity can be assessed by comparing the kind of language generated in the test against the syllabus. The questions then is whether the test produces a good sample of the contest of the syllabus."

Using a public speech as a way to test general English is like giving a driving test on a motorcycle instead of a car.


Brown & Yule (1983) Teaching the Spoken Language, Cambridge University Press. Brown http://www.jalt.org/test/bro_8.htm
There is a lot of good information about testing at: http://www.jalt.org/test/news-title.htm

IELTS examiners - Standarized but not perfect

McDonald's has worked very hard to standardize the look and taste of their hamburgers. Of course, there is some variance. Not every hamburger is exactly the same. The cook may have squirted a bit more sauce on one or sprinkled a bit less lettuce on the other. But due to their standardization process every hamburger is almost the same as any other in any McDonald's around the world. That's the idea.

IELTS, like McDonald's, standardizes its examiners. They all have a set of questions they are to ask. They all have a set procedure they are to follow. They all have a rubric they are to measure by. They all receive training and monitoring and double checking to make sure that they are all as close to standard as humanly possible.

Is it still subjective? Yes. But the degree of variance due to subjectivity has been greatly reduced through their system. I will not contest that.

However, my contention is that this level of control is not sufficient to consistently measure slight changes in English speaking ability from a period like one semester of training. The issue is extremely important, meaning the pass or fail of thousands of students across China.