The Bob System: Tracking students for formative assessment

I am often called on to teach oral English. Unlike teaching written English where the students will be submitting a lot of writing samples, oral English offers less opportunity to sample the students' English ability.

My primary interest in using the Bob System and some sort of scoring system is in formative assessment.

When I have a clear understanding of how they are doing then I have the ability to try to make my training more effective in two ways.

First, are my students "getting it"? Am I helping them to learn what will be useful for them to know?

Second, I can customize my training more to my students' specific needs. I may not be able to give each student individual training (that ability and technology will be coming in the future) but I could segment the class. I want to know who is doing well and who is doing poorly.

When I know this I can offer extra training to those who need extra help. What about students who are doing very well in the class? Sometimes there is an academic ceiling in the classroom. Bright students cannot go higher because the teacher is teaching to the "middle level" of the class. But if we know which students are doing very well and how many of them there are then we can focus on their needs better by providing extra challenge.

Tracking this sort of information can be very useful in other ways, as of Action Research in the classroom. If you are monitoring many aspects of the student's performance in the classroom and you have a student who always participates correctly, does the pairwork, groupwork, homework, listens and doesn't goof off but does not seem to progress in their English from one term to the next then that would raise some very good questions for the teacher.

Of course, finally, the data that is collected can help in summative assessment. The teacher does not need to simply rely on a final exam for a score. The teacher will have a multidimensional way to look at the students.

[Photo: Some of my 300 college students that I taught weekly last term doing pairwork. Next term I will have 400 college students each week.]

15 ideas for mLearning projects

These are some things that have already been done. I don't post these here as suggestions of what you could do but to give you some ideas that you could adapt:


From: MOBILE LEARNING PROJECTS

1. Students have to carry out scientific enquiry in the context of their discovering and exploring of an environment. Pairs of 11-12 year olds explored a woodland and were presented at certain times with different forms of digital augmentations.
   
2. Students can send SMS that appears on lecturers laptop during the class. They can anonymously ask questions without interrupting the class. The lecturer can choose to respond immediately or wait until a number of questions arise. SMS are available after class.
   
3. Players use GPStracked handheld computers to experience a virtual savannah that appears to be overlaid on a football pitch sized grassy field. Players act collaboratively to carry out a series of lion missions (such as marking their territory, hiding their cubs and hunting). Aim of the game is to encourage players to understand the behaviour of lions though personal experience
   
4. Students in a economics course could acess course mateiral (slides, PDF) and contribute on the discusstion board.
   
5. Composition students were asked to compose and perform music on PDAs. PDAs are equipped with MIDI module and portable keyboard.
   
6. Evaluate to what extent and how a PDA can help students learning. PDAs are issued for the project time with the aim of collectiing application log data. Learn more about how students use PDAs.
   
7. The project aims to test "just-in-time" access to knowledge on mobile terminals for medical students. Especially when they are practicing in medical institutions.
   
8. Medical practitioners on ward are equiped with PDAs to build a portfolio of evidence. A detailed evidence of clinical activities, prior learning comopetencies, course materials, certificates. Furthermore they get acess to learning resources, clinical guidelines and a learning diary.
   
9. Informatics students were required to develop and eavluate their own interactive learning experiences in collaboration with fellow students. PDAs with internet functionality were lent to them to consider as their own.
   
10. Course materials were distriubted in e-book format. Followed by an investigation how students use these learning materials.
    
11. A collaborative treasure hunt game that requires co-ordination between spatially seperated team-members. The teams have to visit locations in an urban area to collect symbols in order to complete tasks.
    
12. A collaborative problem solving application that attempts to support learners in constructing their own understanding of tshare their decisions with fellows.
    
13. A application for percussion composition, allows users to create, manipulate, edit and save original pieces of percussion music through an intuitive interface.
    
14. Children collect with scientific data on spot. Devices can collect data and communicate with sensors that are in the field while also providing instant feedback through on the spot data analysis.
    
15. This project supports the participants in expressing their thoughts through a digital narrative. While the overall process is similar to other digital film projects, the tools used are different. The learners shoot all of their footage and record their soundtrack on smartphones. In addition, the smartphones allow the participants to make their multimedia available to collaborators by sending the images and sound via the multimedia messaging service (MMS) to a blog.

More ideas for mLearning projects

These are some things that have already been done. I don't post these here as suggestions of what you could do but to give you some ideas of how other teachers are using mLearning in the classroom. As technology improves and becomes more efficient we will be able to do more things similar to this:

"Students have to carry out scientific enquiry in the context of their discovering and exploring of an environment. Pairs of 11-12 year olds explored a woodland and were presented at certain times with different forms of digital augmentations."

"Students can send SMS that appears on lecturers laptop during the class. They can anonymously ask questions without interrupting the class. The lecturer can choose to respond immediately or wait until a number of questions arise. SMS are available after class."

"Players use GPStracked handheld computers to experience a virtual savannah that appears to be overlaid on a football pitch sized grassy field. Players act collaboratively to carry out a series of lion missions (such as marking their territory, hiding their cubs and hunting). Aim of the game is to encourage players to understand the behaviour of lions though personal experience."

"Students in a economics course could acess course mateiral (slides, PDF) and contribute on the discusstion board."

"Composition students were asked to compose and perform music on PDAs. PDAs are equipped with MIDI module and portable keyboard."

"Evaluate to what extent and how a PDA can help students learning. PDAs are issued for the project time with the aim of collectiing application log data. Learn more about how students use PDAs."

"The project aims to test 'just-in-time' access to knowledge on mobile terminals for medical students. Especially when they are practicing in medical institutions."

"Medical practitioners on ward are equiped with PDAs to build a portfolio of evidence. A detailed evidence of clinical activities, prior learning comopetencies, course materials, certificates. Furthermore they get acess to learning resources, clinical guidelines and a learning diary."

"Informatics students were required to develop and eavluate their own interactive learning experiences in collaboration with fellow students. PDAs with internet functionality were lent to them to consider as their own."

"Course materials were distriubted in e-book format. Followed by an investigation how students use these learning materials."

"A collaborative treasure hunt game that requires co-ordination between spatially seperated team-members. The teams have to visit locations in an urban area to collect symbols in order to complete tasks."

"A collaborative problem solving application that attempts to support learners in constructing their own understanding of tshare their decisions with fellows."

"A application for percussion composition, allows users to create, manipulate, edit and save original pieces of percussion music through an intuitive interface."

"Children collect with scientific data on spot. Devices can collect data and communicate with sensors that are in the field while also providing instant feedback through on the spot data analysis."

"This project supports the participants in expressing their thoughts through a digital narrative. While the overall process is similar to other digital film projects, the tools used are different. The learners shoot all of their footage and record their soundtrack on smartphones. In addition, the smartphones allow the participants to make their multimedia available to collaborators by sending the images and sound via the multimedia messaging service (MMS) to a blog."

The Bob Project - Are we ready for mLearning?

The idea of students interacting with an automated teaching system is central to the Bob project. Many other people are promoting an idea called mLearning or Mobile Learning:

"Given its definition m-learning could very well be a new form of personal learning that never ends, allowing more and more people to realize how much of our lifetimes on this planet are truly extended adventures in personal learning.

From MasterNewMedia.org
"The advocates of lifelong learning have been advocating this very change in how we conceive, design and deliver education. Individuals are constantly learning, searching, questioning and acknowledging new information from the environment they operate in, no matter what their interest or specialization is. Unless your work assignment is something that a computer or other automated machine could take over from you, an increasing number of work activities depend on your ability to learn and familiarize yourself with a continuosly growing array of new concepts and ideas." [Photo: mLearning at a museum]


Some researchers are testing the use of SMS messaging in the classroom. That would be interesting. Instead of teachers telling the students to put their phones away they'll be saying, "Turn to page 35, take out your mobile phone and send me a message on question #6."

From Using short message service to encourage interactivity in the classroom:
"Interactivity in the classroom is reported to promote a more active learning environment, facilitate the building of learning communities, provide greater feedback for lecturers, and help student motivation. Various definitions of interactivity exist in the literature, alternately focusing on the participants, structure and technology. The PLS TXT UR Thoughts research project builds on existing definitions to define interactivity as a message loop originating from and concluding with the student. The authors chose to introduce mobile phones and short message service (SMS) within the classroom due to the ubiquity of mobile phones among students and the interactive potential of SMS. SMS is a low-threshold application used widely by students to quickly send concise, text-based messages at any time. The research presented involved students sending SMS in real-time, in class, via their personal mobile phones. Using a modem interfacing with customised software to produce SMS files, the lecturer can view the messages and verbally develop the interactive loop with students during class. The SMS are available online after class, allowing interactive loops to further develop via threaded comments."


This is an idea rather close to mine about creating an audio tour for students at a popular student location like a shopping mall. It could even be interactive through SMS or MMS.

From Supporting Mobile Language Learning outside Classrooms:
"The continuous development of wireless and mobile technologies has allowed the creation of an additional platform for supporting learning, one that can be embedded in the same physical space in which the learning is taking place. This paper describes a computer supported ubiquitous learning environment for language learning, called LOCH (Languagelearning Outside the Classroom with Handhelds). In the environment, the teacher assigns field activities to the students, who go around the town to fulfill them and share their individual experiences. The main aim of this project, called One Day Trip with PDA, was to integrate the knowledge acquired in the classroom and the real needs of the students in their daily life."


mLearning can be used by students outside the classroom to make an instant blog of what they see, feel and experience.

From Moblogging for ESOL
"M-learning is a powerful tool for ESOL (English for speakers of other languages). In a recent example, ten adult ESOL learners became ‘photo journalists’ for the college open day. They created a photo diary of events using camera phones and sent their pictures, along with captions, to an e-mail address that automatically published them to a publicly-available web site. To prepare for the event, learners looked at published photo stories on the Internet and analysed the language and content to learn about styles and structures that would be useful in writing their own photo diary. The project proved so successful in engaging learners that even the most hesitant members of the class (e.g. a lady in her 60s, who had never used a mobile phone before, and a visually impaired learner) not only took part, but also found the experience very rewarding."


And of course, mLearning is being used to teach languages.

More ideas


Are these the actual tools that we will use in The Bob Project? No. These are actual wild and crazy ideas that will help us think out of the box and find the most effective way to build Bob. If we don't get out of the box we won't go anywhere.

Vocabulary review schedule - Can it work?

In the past, some teachers recommended a way to systematically review vocabulary based on a schedule where the time intervals of review would become longer and longer. I tried to figure out what that would be like for the student.

If a student 'learned' 20 words on Day 1, reviewed them the next day, then 3 days after that, 1 week after that, 2 weeks after that, 1 month after that, 2 months after that, then here is how many words the student would have to review daily.

20, Day 1
40, Days 2-4
60, Days 5-11
80, Days 12-26
100, Days 27-56
120, Days 57-116
140, Days 117-unlimited

We assume the student starts his project on Day 1 and everyday adds 20 words. If my calculations are correct, if the student stopped reviewing a word because he knew it on Day 118 (after about 4 months), then his vocabulary review load would level off at 140 words a day.

By the time he is supposed to know the word (after 117 days) he will have reviewed the word 7 times.

He will be learning new words at a rate of about 600 a month which would be more than 6000 per year.

We have to acknowledge that this would be only a passive understanding of the word, perhaps useful for reading/listening. But it could provide an 'introduction' to the word that through further contact with the word in extensive English input could turn into an active understanding of the word useful in writing/speaking.

I think such a thing would be doable but only by highly dedicated well-organized students. For most students it would probably be too challenging. Perhaps if the student limited himself to 10 or even 5 words a day it would be easier and he'd only have to review 70 or 35 words a day.

Would reviewing the word 7 times be enough to hold onto it and allow extensive exposure to kick in and help the student keep the word I don't know.

Speaking evaluations made simple

This is a very complicated subject. It is not easy to conduct a speaking test but I will go over just a few things about it and touch on them lightly. There are many ways to do speaking tests. I have studied them, tried some of them and have settled on this way. It is similar to the way I was trained as an IELTS examiner with a few differences.

QUESTIONS

Design three levels of questions.

(1) Easy questions which are answered with straight factual answers. "Where are you from?" "How long have you been here?" "What did you do yesterday?" "What do you like to do on the weekends?" These questions make little demand on the student and only very low-level students will have problems with these.

(2) Moderately difficult questions demand more from the student. These are questions asking a student to describe a city or restaurant, relate the story of a movie recently seen or a book recently read. "Tell me about your last holiday? "Describe your best friend."

(3) Difficult questions are those that require the student to give an opinion and justify their opinion with reasons. "Should students be required to wear school uniforms? Why?" "Should smoking be banned in all buildings? Why?"

Be aware that some questions are not only difficult to discuss in English, sometimes they are just plain difficult to discuss at all. I once designed a question, "If you had two weeks to live, what would you do?" This question was so deep that the students became extremely thoughtful in trying to give their answers to the point that it interfered with any attempts to show fluency. Questions do not need to be so deep.

Although the question may be difficult at times, to understand the question should be simple. Remember, this is a speaking test, not a listening test. For example, "Given the opportunity to go on a round-the-world cruise or participate in a scientific exploration in Africa, which do you think could potentially be more beneficial for your career development?" Many low and mid level students would not be able to understand that question and therefore would not be able to speak on it. Make sure your questions are easily understandable.

I like to let the students ask each other the questions. This way I can focus on listening and evaluating. But I do not allow the students to prepare for the questions except for perhaps just a couple minutes before the interview.

Look me in the eye? In western countries we have no problem looking into people's eyes when speaking to them but this is something that Asians do not do. Therefore, when you conduct the speaking test with Asian students it is best to not try to look deeply into their eyes or to hold their gaze. Look elsewhere, shift your eyes around or even just focus on your band descriptors or rubric.

EVALUATION

You can use a rubric or band descriptor to measure the student's level such as the IELTS band desciptors or the Common European Framework.

You will notice in the IELTS descriptors that at Band 4 it says:

"Is able to talk about familiar topics but can only convey basic meaning on unfamiliar topics and makes frequent errors in word choice. Rarely attempts paraphrase."

and then at Band 5 it says:

"Manages to talk about familiar and unfamiliar topics but uses vocabulary with limited flexibility. Attempts to use paraphrase but with mixed success."

That is why it is important to design your interview questions with easy, moderate and difficult topics so that the student will have to try to produce a full range of English at different challenging levels to respond accurately. The English of many students will begin to break down at the higher levels and this will allow you to see the limit of their English.

I put the band descriptors and all the students names on an Excel spreadsheet. I give the student a score for each rating catagory (Fluency and coherence, Lexical resource, Grammatical range and accuracy)
and the program averages it out into a final band score. Depending on the situation I will add formulas to work that score into a grade, average all the scores to compare one group with another or other things. Click on the picture (above) to see it enlarged.

ACCURACY

The more realistic the task is, talking naturally about a topic the student may actually need to discuss rather than some sort of T/F or multiple choice, the more difficult it is to test. So this sort of test will always be subjective, affected by your personal judgment of the student's performance.

One thing that helps is to be sure to base your judgment as closely as possible on the rubric or band descriptors you are using. You should never compare students to each other. This will lead you off the track. Always compare to your chosen rubric.

I always record my test interviews. A couple days later I will listen to some of the interviews and rescore them without looking at the score I gave the first time. If there is a strong correlation then that is good. If you find that you are scoring much differently the second time then you need to try to understand why and may even need to rescore all your interviews. It happens that you can be in a certain mood that will cause you to score differently. (Another good reason to record is to contribue to a record of the student's progress.)

IELTS research has even shown that male interviewers will sometimes give attractive females a slightly higher score which leads to inaccuracy. If the interviewer is tired, sleepy, hungry or if the interviewer has scored several high level students in a row and suddenly gets a low level student it can affect his accuracy. To run an effective test you need to be aware of all of these things and try to guard against them effecting your judgement.

Total class participation

Sometimes a few sharp students will answer all the questions I put to the class while some students want to space out, read or chat. To force total class participation I ask all the students to stand up. Then when I put a big question to them like "Give me some words about [whatever the subject we're studying]." Each student who replies with a satisfactory word can sit down. In this way everyone has to participate.

Organizing speaking tests for large numbers of students

Yesterday, I did a speaking test for 200 students. It's quite a big job and took me all day. But it would have taken much longer if I had never done this before and if I wasn't organized.

Some teachers allow students to choose their partner and choose their subject. Sometimes they can do this days in advance. Consequently, some students will find a dialog and memorize it and then perform it for the test. I don't do it this way. I don't think it's very realistic.

In the workplace, people need to be able to speak English to anyone and they can't always choose the topic. So I tell my students they can choose any partner they want as long as student #1 chooses students #2 and student #3 chooses student #4, etc. I tell them they can choose their own topics to talk about. They come up to my desk and choose from several slips of paper which are facing down. The students cannot look at the paper before they choose. Consequently, they are choosing randomly. I do allow them 3-4 minutes to prepare before the interview. The topics are always things that we practiced discussing in class.

When one pair of students sits down to do their dialog for me another pair of students will come up to the desk, choose a topic and stand aside to prepare. I don't allow them to use dictionaries, notebooks, textbooks or to talk with other students during the preparation period. I sit facing the two students who are talking but behind them I can keep an eye on the next two students who are preparing for their talk. This is important because they many of them can hardly keep themselves from a bit of cheating if it is possible.

The students talk to their partner on the topic. I used to be an IELTS examiner and found it is a bit extra work to have to also be asking the students all the questions. So I like to get the students talking with each other and I listen in. If I think a student can go higher or if their dialog was too short I will jump in with a few extra questions.

While they are talking I am recording them. They hold a cheap $2 clip-on microphone that I found which works really well and is fastened at the end of a ballpoint pen. I use free program called Audacity to record the interview. Although I'm sitting right next to them, I actually listen to the students through a set of earphones. This ensures that everything is being recorded. The interview is recorded for reference in case I want or need to go back and check something or if I need to justify a score I have given.

If their dialog is too short or doesn't reveal their English skills well enough I will ask some questions to make them speak more. Answering"why" questions or questions where they have to explain or justify a viewpoint are some of the toughest questions and are good to push students to the limits.

I have some band descriptors and a list of all the students names and numbers on an Excel sheet. The band descriptors are divided into three areas of speaking (communicative range, overall fluency, accuracy & appropriacy) and four levels (levels 4-7 of a 10 level rating system) While they are talking, I scan the band descriptors and give them an initial score. I continue to listen to them and modify parts of my score as they perform better or worse.

It's very important to have a clear set of standards that the students should speak to. And while they are speaking you should be constantly checking those standards and try to measure the student to those standards as best you can.

After all of the testing is done, I will use Excel to average out the band scores and assign grades on a curve.