How to Measure CEFR Progress by Cohort

A school that reports only changes of CEFR level will have little to report for most of the year. The levels are wide and a term is short. That does not mean pupils stood still. This article shows how to follow one group of pupils and how to count progress inside a level. It reports what official sources say about the time a level takes. It explains three traps that make cohort results look better or worse than they are, and it ends with a worked example.

How do you measure CEFR progress by cohort?

To measure CEFR progress by cohort, take a fixed group of pupils and check their level at two points with the same method. Then count how many pupils moved. Count moves between levels, such as A2 to B1, and steps inside a level, such as A2 to A2+. Report both, because most pupils will not change level in one term.

1. Name the cohort. For example, every pupil enrolled in Grade 6 in the first week of September.
2. Check each pupil's level at the start. Use a method you can repeat at the end.
3. Keep a weekly record. Stories completed, comprehension and words retained.
4. Check the level again at the end, the same way.
5. Compare only pupils checked at both points. Report joiners and leavers separately.

The level check itself is a separate subject. How to Assess a Student's English Reading Level covers it.

What is a cohort, and why follow the same pupils?

A cohort is a group of pupils defined once and then followed. 'Every pupil enrolled in Grade 6 in September' is a cohort. 'Whoever is in Grade 6 today' is not, because the names change.

A school can make three kinds of comparison. They are not equally fair.

• The same pupils at two points. The fairest comparison a school can make alone.
• This year's class against last year's. These are different children, who may have started at different levels.
• One class against another. Useful only if both classes started at similar levels.

Even the first kind shows change, not cause. Pupils also learn in their other English lessons and outside school. A cohort table tells you what happened to these pupils. It does not tell you why.

How long does it take to move up a CEFR level?

Cambridge University Press & Assessment publishes an estimate. It states that a learner takes approximately 200 guided learning hours to progress from one CEFR level to the next. Guided learning hours are hours of lessons and supervised study. Cambridge also gives cumulative figures, counted from beginner level.

Cambridge calls these figures 'a guideline only'. It lists four things that change the time: the learner's language learning background, the intensity of study, age, and the amount of study or exposure outside lessons. The page is written for exam candidates, not for school timetables.

The Council of Europe (2001) is more cautious. It warns against reading the levels 'as if they were a linear measurement scale like a ruler'. It reports that many learners take more than twice as long to reach B1 from A2 as they needed to reach A2. It asks for 'extreme caution' in using any scale of levels to calculate the time needed to meet an objective.

Now do the sum for a school. The figures are an example. A class with three 50-minute English lessons a week has 2.5 hours of lessons. Over a 10-week term that makes 25 hours. Against an estimate of 200 hours per level, one term is an eighth of a level. Most pupils will end the term at the level where they began. That is normal.

Report movement inside a level

The Council of Europe (2020) says that language proficiency 'is actually a continuum', like the colours of a rainbow. It calls levels 'a necessary simplification'. It also describes plus levels, such as A2+. A plus level is a very strong competence at a level that does not yet reach the minimum standard for the next level.

Schools are invited to divide the levels further. The Council of Europe (2001) gives the example of a primary to lower secondary school system where 'visible progress at low levels is felt necessary'. Such a system could split the basic levels into about six milestones, with finer steps at A2. For the levels themselves, see CEFR Levels A1–C2 Explained (With Reading Examples).

So report steps as well as levels. Four signs show movement inside a level:

• A step. The pupil moves from A2 to A2+.
• Amount read at the level. Stories completed at A2 this term.
• Comprehension at the level. Quiz results on A2 stories are steady and high.
• Words retained. Words from earlier stories are still known some weeks later.

Vocabulary has been measured against the CEFR levels. Milton (2010) reviewed results from vocabulary size tests. Higher levels went with larger vocabularies. In one set of results he reports, learners who passed A2 exams scored 1,500 to 2,500 on a test of the 5,000 most frequent words. Learners who passed B1 exams scored 2,750 to 3,250.

Milton (2010) also reports individual variation and overlap between the scores at each level. So a word count can show growth inside a level. It cannot place a pupil at a level by itself. How Many Words Do You Need at Each CEFR Level? goes further.

Three traps in cohort results

Trap 1: error in a single test

Every check has error. A pupil near a boundary can land on either side of it on a given day. The Council of Europe (2001) asks the question directly. If an assessment reports results as A2+, B1 or B1+, how accurate are these decisions? Barnett, van der Pols and Dobson (2005) add that natural variation in repeated data can look like real change.

What to do: do not change a pupil's recorded level on one result. Wait until the level check and the term's quiz results agree. This is practice, not a tested rule.

Trap 2: the lowest scorers seem to improve most

This is regression to the mean. Barnett, van der Pols and Dobson (2005) define it: unusually large or small measurements tend to be followed by measurements closer to the mean. They wrote for health researchers, but they describe the effect as ubiquitous in repeated data. It grows with measurement error. It shows most when you follow only a group chosen for its first score.

In a school it looks like this. You pick the ten lowest scorers in September and give them extra sessions. In December their average has risen. Some of that rise would have come with no sessions at all, because some of those low scores were bad days. Campbell (1979) reported rumours of programs that treated only the lowest scorers, so that regression to the mean would provide apparent gains.

What to do: report the whole cohort, not only the lowest group. If you report on a chosen group, compare it with similar pupils who had no extra help.

Trap 3: pupils who join or leave

If four weak readers leave and four strong readers join, the class average rises and nobody has learned anything. Campbell (1979) called attrition, the loss of people between two measurements, a major problem in evaluating programs. He was writing about large social programs. The arithmetic is the same in one class.

What to do: build the table from pupils checked at both points. Give the number of joiners and leavers under it.

What to measure, how often, and what it tells you

Level moves slowly, so check it rarely and check it the same way each time. The weekly measures explain what the level check shows.

A worked example: one class across three terms

The class and the numbers below are invented. Class 6B had 28 pupils in September. Each pupil was checked then, and again at the end of each term.

Read the rows with care. After Term 1 only two pupils had moved, which fits the estimates above. Over the year, 13 of 26 moved one step. Had the school counted only changes of level, it would have reported 6. Had it counted only pupils reaching B1, it would have reported 3.

The cohort sentence belongs in the term report. How to Write a Term Reading Report for Your Board shows where it goes.

How big is the gain?

Report counts of pupils before percentages. In a class of 26, one pupil is almost four percentage points, so a percentage makes a small change look precise.

Say which kind of measure you used. Kraft (2020) reports that effects on tests closely aligned with a program are often two to four times larger than effects on broad standardised tests. His evidence comes from randomised trials, not from school reports, but the lesson carries over. A rise in quiz scores on the stories pupils read is not the same as a rise on a general reading test.

Kraft (2020) also warns that results from correlational studies are not causal effects. A cohort table is no stronger. Present it as a record of change.

How iRead handles this

Every story, quiz and game in iRead is levelled to the CEFR, from A1 to C2. Vocabulary and quizzes are mapped to CEFR bands, so a B1 story means the same thing whichever pack it comes from. A word cleared on two different days is marked Mastered. School leaders see levels reached, words retained and stories completed, by class, grade and term. Attendance at sessions is recorded automatically.

Key takeaway

Define the cohort once, check the same pupils twice, and report steps as well as levels. Say who was left out, and treat one term's movement as an estimate.

Frequently asked questions

Sources

1. Council of Europe (2001). Common European Framework of Reference for Languages: Learning, teaching, assessment. Cambridge University Press, Cambridge. rm.coe.int/1680459f97
2. Council of Europe (2020). Common European Framework of Reference for Languages: Learning, teaching, assessment. Companion volume. Council of Europe Publishing, Strasbourg. rm.coe.int/common-european-framework-of-reference-for-languages-learning-teaching/16809ea0d4
3. Cambridge University Press & Assessment (n.d.). Guided learning hours. Cambridge English Support Site. support.cambridgeenglish.org/hc/en-gb/articles/202838506-Guided-learning-hours
4. Barnett, A. G., van der Pols, J. C., & Dobson, A. J. (2005). Regression to the mean: what it is and how to deal with it. International Journal of Epidemiology, 34(1), 215-220. doi.org/10.1093/ije/dyh299
5. Milton, J. (2010). The development of vocabulary breadth across the CEFR levels. In I. Bartning, M. Martin & I. Vedder (Eds.), Communicative proficiency and linguistic development (Eurosla Monographs Series 1), 211-232. www.eurosla.org/monographs/EM01/211-232Milton.pdf
6. Campbell, D. T. (1979). Assessing the impact of planned social change. Evaluation and Program Planning, 2(1), 67-90. doi.org/10.1016/0149-7189(79)90048-X
7. Kraft, M. A. (2020). Interpreting effect sizes of education interventions. Educational Researcher, 49(4), 241-253. doi.org/10.3102/0013189X20912798