Showing posts with label assessment. Show all posts
Showing posts with label assessment. Show all posts

08 March 2007

Bob: Deciding on a Participation Index

After teaching some corporate students, and reluctant to hand out school-type "grades" but still needing something, I developed the idea of a "Participation Index" to have a measure of how involved the student was in the training. This experience gave me a different viewpoint on grades.

Attendance
At my college I had one student who came on the first day of class and again for the final exam. I promptly forgot him after seeing him the first time and wondered who he was when I saw him the second time. The class monitors keep track of attendance. Now I have the class monitors also keep track of attendance in my Excel spreadsheet. At the most basic level, if the students are in class they are (hopefully) going to learn something. The idea is to show their "participation".

Class Interaction
But still some students are doing some other homework, reading something else, chatting with their neighbors, etc. Are they learning during this time? No. (Perhaps they already know. In that case it probably should be OK for them to do something else if it doesn't disturb others.) The idea is not to punish students but to show their "participation" in the lesson. Some MBA course instructors give scores up to 40% of the students' grades for "airtime", visibility acquired on basis of class participation. In "Alternative Approaches to Assessing Student Engagement Rates", Elaine Chapman at The University of Western Australia describes several ways of measuring participation and I'm thinking of using the one she describes as "Direct Observation".

Homework
I have 160 students and sometimes hand out homework with each class. So I usually see if the students did the homework without seeing how well they did it. The idea is that applying themselves to the work is beneficial for their learning whether they got it all correct or not. We then go over the questions and answers together and everyone self-corrects.

Quiz
I want to use quizzes more effectively and more frequently to see if the students are "getting it". When I haven't done this I've been surprised how many students really didn't understand (or pay attention).

Exam
To measure the take-away from the training. I'm having lots of other thoughts on this. While the school course teaches things like how to read the corporate year end report, I have found a lot of my students go out after graduation and get jobs as Nokia phone salespeople in a discount department store or other sub-entry level job. It is likely that a lot of what they learned is lost well before they get a chance to use it. I am thinking about not only not testing their comprehension of corporate reports but steering the course away from that sort of thing and focusing more on English that they will have more hope of using soon. If they don't use it they'll lose it. But to help them keep it we should teach what they'll use.

07 March 2007

The disease is fatal. It's called "boredom"!

Troubleshooting a teacher's problem.

A teacher is having some problems with his classes. Despite his efforts he said his classes were too boring. Here is what he said he was doing to get his students interested in English:

1. Play English Videos with Chinese subtitles
2. Role -play
3. Use scores to stress them
4. Ask the students to recite passages
5. Organize some English contests such as speech contest, etc.

1) Beginners and lower intermediate students will always focus on the L1 subtitles. Sometimes they are unable to watch the action as they have to watch the subtitles so closely. They are long and take up a whole class or more. Watching them is a passive event. The students sit and receive information but don't have to interact with it. Finally, watching a movie is something they commonly do outside of class.

For beginners and intermediates, movies are not an easy language learning tool except for listening (and reading). There are a few simple movies that you can watch with your intermediate students in English with English subtitles that they will follow the language enough to laugh when something clever is said and shed a tear or two at the end. Try "Big" with Tom Hanks, "Trading Places" with Eddie Murphy, "Don Juan Demarco" with Johnny Depp and Marlon Brando.

But even with these, they are passive events which only challenge the students' listening and reading ability. They are not as good as something that makes the students interact with the language.

2) Role plays are good if they are interesting for the students. Students seem to especially love anything with negotiation in them. Negotiation is like a game for them.

3) Scores (like death and taxes) always seem to be with us. But (like death and taxes) they should be a motivator of last resort.

4) Reciting passages is an effort of rote memory. It also doesn't require interaction between the student and the language unless the student is studying drama.

5) Speech contests are rote memory efforts with a bit of drama and a score coming up at the end. Debates would be better. Debates are a verbal game or challenge requiring student interaction with the language. For beginners and intermediates it may be best to keep the debates on fun or light-hearted topics to avoid focus on win-lose issues. For example, what is better: KFC or McDonald's? Pizza or ice cream? One million dollars or one million flowers?

27 February 2007

Bob - Effective student monitoring, formative & diagnostic assessments and differentiating instruction


A teacher lamented: "If, as is the case with many teachers, they have 10 or 11 separate classes once a week for one and a half hours - and, an average of 50 students in each class, - they see around 500 students for a total of approximately 26 hours during a full semester. If there is any super teacher that can focus on the performance of every one of their students under these conditions and, realistically, expect to achieve, even, 90% motivation then they have outstanding expectations."

Different teachers have different ways to appeal to and win the hearts and minds of their students. Some teachers do it through the example of their professionalism, sometimes augmented by humor and love, and the confidence that the teacher is able to instill into their students that he will be able to take their English to a higher level.

But in large classes there are students who are slipping through the cracks. As you implied, how can they "focus on the performance of every one of their students under these conditions" rather than just, what I'll call, "broadcast teaching"?

This past semester I was experimenting with a system I like to call "Bob". In a way, "Bob" is a computer system I'm using consisting of an Excel file and Visual Basic program. But it is more an extension or appendage of me as a teacher and many of the things I'm doing with my students. Technology isn't running my classroom but I am using technology to help me do what I need to do easier and better.

I must say it has helped me keep track of my students ten times better than before I began to use it. Of course, I'm a bit absent minded and always have trouble remembering things, people, names, etc.

For people who are interested in this sort of thing you need to know what to do after you gain the ability to closely monitor each student. For example, built into the system you would have some Diagnostic and Formative Assessments.

The next step is grouping the students or segmenting them. (Bob enabled me to develop the name cards or name tents that I spoke of earlier.) Then you can apply what they call "Differentiating Instruction" to better meet the needs of the students and their motivational triggers in a more individual way.[2]

I think most teachers teach to the middle level of the class. We don't teach to the bottom students but we don't teach to the top students either. We teach to the middle students. This is perhaps the normal effort of teachers. Teachers who make a greater effort may try to do something for students who are lagging. But there is also a significant number of students who are doing very well in our classes. If we don't pay attention to their needs they will be stuck at an academic ceiling, possible boredom and lost opportunity to develop. We need to offer them a degree of more challenging work so they can climb higher than the class average to which we are teaching.

By effective use of available technology all of this is possible to the degree that I think you could manage 500 students.

I have a couple hundred students and until using this system I never felt I was really on top of monitoring them. The system is so powerful that recently I set up for a new group of only eight students and the system felt like overkill, too much for such a small group.

From this I can see how all of my students are doing at a glance. Areas that are in pink or dark pink indicate some areas of concern. Areas that are white indicate above average achievement. These colors appear automatically according to the data. If I want detailed information I can drill down into the data but the colored Dashboard provides a good overview.

The Dashboard is a summary consisting of totals or weighted averages of all the other data I'm tracking on other sheets like the daily attendance record (AT), classroom interaction (CIA), homework scores (HW), quiz scores (QUIZ) and bonus work.

It is a system I developed myself and is still a work in progress. It is an Excel file. In addition I've added programming elements that allow Bob to send out SMS messages through my mobile phone to 150 students automatically updating them on their scores.

21 February 2007

IELTS examiners - Standarized but not perfect

McDonald's has worked very hard to standardize the look and taste of their hamburgers. Of course, there is some variance. Not every hamburger is exactly the same. The cook may have squirted a bit more sauce on one or sprinkled a bit less lettuce on the other. But due to their standardization process every hamburger is almost the same as any other in any McDonald's around the world. That's the idea.

IELTS, like McDonald's, standardizes its examiners. They all have a set of questions they are to ask. They all have a set procedure they are to follow. They all have a rubric they are to measure by. They all receive training and monitoring and double checking to make sure that they are all as close to standard as humanly possible.

Is it still subjective? Yes. But the degree of variance due to subjectivity has been greatly reduced through their system. I will not contest that.

However, my contention is that this level of control is not sufficient to consistently measure slight changes in English speaking ability from a period like one semester of training. The issue is extremely important, meaning the pass or fail of thousands of students across China.

Jack Welch and the language classroom

...well, OK, it's not really Jack Welch talking about English teaching but it is adapting a couple of his ideas.

A couple of Jack Welch's ideas that he expounds in his best-selling book, "Winning", are how he uses every opportunity to appraise the people working for him and how he regularly fires the bottom 5% performers.

In my spoken English class I am using almost every exchange with students to help evaluate their English. I'm not waiting for the end of term exam or even quizzes. Especially after a pair or group work activity I like to call on a few students to tell us what they talked about, what decision they made about the situation in the activity, what answer they have to the questions they were to talk about. When they speak I may ask them a couple more questions and then I will give them a score which is recorded in my computer and averaged with other scores and indicators.

In this way I can get a pretty good idea which students are doing well in my class and which ones are not doing so well. When you have classes of 40-50 students there are always a few very naughty students sitting in the back of the class and a few very smiley and bright ones sitting in the front of the class. Both of these types of students distract you from the more unobtrusive majority, many of whom are rather quite although often with good English skills. Only by systematically checking all of the students can we find these students before giving them an exam.

This leads us to the next point...

Another practice of that Welch follows is firing the bottom 5% performers. The employees that are doing the least in the company get fired as a regular practice. Well, as teachers we usually can't fire our students. But we can make a point of identifying exactly who the bottom performers are and trying to develop a plan to help them whether it's extra training, extra homework, extra encouragement or whatever.

In every class there are students who are doing well and students who are doing poorly and many in between. Unlike something like a writing class or four-skills class there may be less evidence of exactly how the students are doing. The speaking class students are not turning in homework and usually no quizzes. What some teachers do is to just teach, teach, teach and give an oral exam at the end of term.

My suggestion is that teachers should use as many opportunities as possible to assess the students, formative evaluation. I have realized that in the past I let many of these assessment opportunities slip by. I now make full use of them.

As far as face goes, we must understand the significance of face in Asian society, respect that and take it into consideration in all that we do. But face saving is not a goal unto itself. There may be instances when we are unable to preserve face and do what our jobs require at the same time. If a student is struggling in the classroom we must try to help them. Of course, we'll do everything we can to help him save face but his grades, his success in school and his success in life are not measured soley by the amount of face he retains. In fact, there are certain situations when it is better to lose face.

Concerning borrowing a page from Jack Welch, a captain of capitalistic-commerce, I believe he knows a lot about organization. I despise some of the snaky things Welch has done. If he is responsible for even half these things he should be tried as a war criminal.[1] I despise his attitude that business is more important than family.[2]

But that does not mean he cannot have good advice that can help us. After all, you or a friend of yours probably enjoy some of Hitler's good ideas, right?[3]

[1] http://www.cleanupge.org/gemisdeeds.html
[2] http://www.msnbc.msn.com/id/7305761/site/newsweek/
[3] http://ask.yahoo.com/20050308.html

Bob: Continous assessment

There are always some students who get more of the teacher's attention, either through their excellent participation in the class or their bad behavior. But these characteristics can mislead the teacher as to how the whole class is doing. About half the students are more unobtrusive and are doing well or poorly. In fact, some people who seem like they are doing poorly are doing pretty well and some who seem like they are doing well are not doing well.

With my class list in my computer (or could be on a sheet of paper) I go down the list or hop around the list calling on every student. It may take from 2-4 days, but I get every student to answer some questions that were the focus of our pair/group work or questions put to the entire body. Sometimes when the students are doing pair/group work, I will take my computer (or list) and walk around to some of the students and interview them personally.

This way I will have a pretty good idea of how each student, even the quiet and unobtrusive ones, are doing in my class even if I have 40, 50 or 100 students.

I will also be very knowledgeable about which students are skipping class because I'm keeping this information on my computer and it keeps accumulative data with alerts to problem areas, although it could also be kept on paper, I will also know who has been missing my class for a long time.

I, too, had a similar problem as you. Two years ago a student came to my first class and then came to my last class for the exam. In the time between I had forgotten all about him. I was determined to not let that happen again and that is why I began keeping track of attendance on my computer.

But in any case, my suggestion is that in the speaking class we can use every opportunity to evaluate our students throughout the term to develop a picture of how each student is doing. Then, using that information we can segment the students into three groups. Doing well, doing OK, doing poorly. With that we can challenge the groups that is doing well to do better so that they are not limited by the "ceiling" of the class. The group that is doing poorly we can offer more attention, perhaps outside the main class or something else.

This is a way to more customize our approach to training to the needs of the class.

Bob: Tracking students' progress

"Bob" is a computerized system to help you manage your class.

A teacher lamented: "I to try to collect the information on how my students were doing, but I still feel that I need a secretary or a volunteer to gather much of the data or at least put it into the computer as while I am doing those things in the class I losing focus on the other students and thinking about my next move."

A secretary or teacher's assistant is great but I don't have one. The data can be kept on a list like the roster. I like to keep mine on my computer using an Excel spreadsheet. It is part of a bigger system I call "Bob".

My spreadsheet keeps a running average so I can see how this student has been doing over a long period of time. Using Excel I can make the data cell or box change colors according to the data I enter. For example, if you give a student a high score then his box can be a certain color for "good", lets say yellow. If you give him a low score then his box can be another color for "warning", lets say red, to alert you that he is not doing very well. Then when you look at your entire sheet of score data if you see lots of "warning" colors and few "good" colors you can get an idea of how the whole class is doing.

In this way you can develop a sort of "dashboard" system just like when you are driving a car. Just as the indicators on your car (speed, engine temperature, RPM's, brake fluid, gas gauge) can tell you how your car is doing and if there is anything that needs your attention, we can have indicators in our teaching that show us how our class is doing.

It doesn't take hardly any time at all to enter in the data. I look on my list of names, choose one that I haven't checked in awhile and call out her name and ask her the question that she should have been talking about during the pair/group work. If she hems-and-haws then I know she wasn't doing the pair/group work.

For this particular class of students, if she gives me a fairly good but not perfect sentence I'll give her a "9". If she has a lot of trouble making a sentence I'll give her a "6". "7" and "8" fall in between. I only do this with 2-4 students at a time, like after pair/group work, so as not to hold up the whole class or bore everyone.

Alternatively, while they are doing the pair/group work I may walk around (with my notebook computer in hand or with a piece of paper and pen) and interview a student here or there. Even if a student comes to ask a question during a break you can make an evaluation and enter a score. To enter the data takes only a second for each student.

Bob: The power of student name tents


"Bob" is a computerized system to help you manage your class.

It is not easy to quickly scan the classroom and know every students' name and know if that particular student is doing well or not in your class. We usually know a few who are doing well and a few who are naughty but to know how all of them are doing without referring to our records is difficult.

I started using name cards, not like a "calling card" or "business card" but actually a sheet of paper with the student's name on it, when I was having trouble remember the students names. I have a few classes with 40-50 students in each one.

Without name cards I found I was pointing at students who were looking at me but not those who were not looking or I was calling on the students whose names I knew but consequently there were a certain group of students that didn't get called.

One way to deal with this is to call students from the roster and another way is to use name cards.

DESIGNING THE NAME CARD AND ADDING HIDDEN DATA TO IT

The name card is an A4 sheet of paper with the student's name and number on it. The name is printed on the lower half of the paper and it is folded over like a tent. Originally I had students write their names on the papers themselves using a marker. Now I print out their names using the computer printer and I am able to add a little data to the name card.

After using these name cards for awhile I wished I could look at my students and not only know their name but know how they were doing in my class. Are they one of the quiet ones who is doing great or are they slowly and quietly sinking in my class? Are they one of the ones who said they have trouble understanding my instructions or are they one of the ones who has asked me to speak faster and more native-like? I was already using "Bob" to help me keep track of how my students were doing.

So I altered the name cards to look like this:

Sue -- 21
Mark 17
John - 36

In the first case, I know his name is Sue and her number is 21. The double dash "--" means she is doing pretty well in my class and his scores on homework, quizzes, attendance and classroom participation all average out pretty high.

In the second case, I know Mark is number 17 and the absence of any dash means he is doing average in my class.

In the third case, I can see that John is number 36 and the single dash "-" means he is not doing so well.

This gives me a deeper understanding of my class. I can quickly throw very challenging questions to my upper level students. I can be more patient with my students who are struggling.

I use this dashing system of marking as it is not very conspicuous. For example, if I used stars as in grading a hotel as in a "five star hotel" it will seem more like a rating system and easy for students to compare each other. With my system the students are unaware that the dashes are indicators of their overall scores.

FURTHER IMPROVEMENTS

Optimizing the name cards a little further, I photocopied a 4-week Learner's Score Card on the other side of the paper. Each week, the students fill in the data about how they used English during the week. I can view this information as I walk around the class by picking up their name card and looking inside the folded area.

I can also check the information at the end of the 4-week period when their Learner's Score Card is completed and totaled. I'm not satisfied with my first effort at a Learner's Score Card as I don't think my questions were crafted carefully enough to give me the most useful data but I like the idea of getting maximum use out of the name card like this. I will certainly use some of the questions like:

In class last week,
Some things I learned:
Some things I didn't understand:
Some things I liked:
Some things I didn't like:
Some things I want to study:
Some things I need help with:
Outside of class last week:
I spoke English (where? to whom?):
I listened to English:
I read English:
I wrote in English:

I don't expect you to copy this name card idea as it won't fit all of your situations but perhaps some aspects of this idea might be helpful to deal with problems you face.

Bob: Effective student monitoring, formative & diagnostic assessments and differentiating instruction

"Bob" is a computerized system to help you manage your class.

Different teachers have different ways to appeal to and win the hearts and minds of their students. Some teachers do it through the example of their professionalism, sometimes augmented by humor and love, and the confidence that the teacher is able to instill into their students that he will be able to take their English to a higher level.

But in large classes there are students who are slipping through the cracks. As you implied, how can they "focus on the performance of every one of their students under these conditions" rather than just, what I'll call, "broadcast teaching"?

This past semester I was experimenting with Bob and have written about it before. [1] In a way, "Bob" is a computer system I'm using consisting of an Excel file and Visual Basic program. But it is more an extension or appendage of me as a teacher and many of the things I'm doing with my students. Technology isn't running my classroom but I am using technology to help me do what I need to do easier and better.

I must say it has helped me keep track of my students ten times better than before I began to use it. Of course, I'm a bit absent minded and always have trouble remembering things, people, names, etc.

For people who are interested in this sort of thing you need to know what to do after you gain the ability to closely monitor each student. For example, built into the system you would have some Diagnostic and Formative Assessments.[1]

The next step is grouping the students or segmenting them. (Bob enabled me to develop the name cards I spoke of earlier.[2]) Then you can apply what they call "Differentiating Instruction" to better meet the needs of the students and their motivational triggers in a more individual way.[3]

I think most teachers teach to the middle level of the class. We don't teach to the bottom students but we don't teach to the top students either. We teach to the middle students. This is perhaps the normal effort of teachers. Teachers who make a greater effort may try to do something for students who are lagging. But there is also a significant number of students who are doing very well in our classes. If we don't pay attention to their needs they will be stuck at an academic ceiling, possible boredom and lost opportunity to develop. We need to offer them a degree of more challenging work so they can climb higher than the class average to which we are teaching.

By effective use of available technology all of this is possible to the degree that I think you could manage 500 students.