There are always some students who get more of the teacher's attention, either through their excellent participation in the class or their bad behavior. But these characteristics can mislead the teacher as to how the whole class is doing. About half the students are more unobtrusive and are doing well or poorly. In fact, some people who seem like they are doing poorly are doing pretty well and some who seem like they are doing well are not doing well.
With my class list in my computer (or could be on a sheet of paper) I go down the list or hop around the list calling on every student. It may take from 2-4 days, but I get every student to answer some questions that were the focus of our pair/group work or questions put to the entire body. Sometimes when the students are doing pair/group work, I will take my computer (or list) and walk around to some of the students and interview them personally.
This way I will have a pretty good idea of how each student, even the quiet and unobtrusive ones, are doing in my class even if I have 40, 50 or 100 students.
I will also be very knowledgeable about which students are skipping class because I'm keeping this information on my computer and it keeps accumulative data with alerts to problem areas, although it could also be kept on paper, I will also know who has been missing my class for a long time.
I, too, had a similar problem as you. Two years ago a student came to my first class and then came to my last class for the exam. In the time between I had forgotten all about him. I was determined to not let that happen again and that is why I began keeping track of attendance on my computer.
But in any case, my suggestion is that in the speaking class we can use every opportunity to evaluate our students throughout the term to develop a picture of how each student is doing. Then, using that information we can segment the students into three groups. Doing well, doing OK, doing poorly. With that we can challenge the groups that is doing well to do better so that they are not limited by the "ceiling" of the class. The group that is doing poorly we can offer more attention, perhaps outside the main class or something else.
This is a way to more customize our approach to training to the needs of the class.