In short
- Judge a program on four things: cost per pupil including staff time, what changed, compared with what, and how big the change was.
- Cost-effectiveness analysis compares programs that pursue the same goal by both their costs and their effects.
- Evaluations run or paid for by developers tend to report larger effects than independent ones, so check every vendor claim.
- A mixed result reported honestly is still evidence, and it supports a better decision than a polished one.
Return on investment is a business term. A school spends money and staff time to get learning, which has no price. That does not mean value cannot be judged. This article gives an honest method: cost per pupil with staff time included, what changed, the comparison and the size of the change. It explains cost-effectiveness analysis, the method economists of education use in place of ROI. It shows why a vendor's ROI claim needs checking, and what to do when results are mixed.
How do you measure the ROI of edtech in a school?
To judge the return on an edtech program, answer four questions. What does it cost per pupil, counting staff time as well as fees? What changed for pupils? Compared with what? And how big was the change? Then set the answer beside the other ways you could spend the same money and time. That last step turns a number into a decision.
- Cost per pupil. Every resource the program uses, divided by the pupils it actually reached.
- What changed. Participation, amount read, comprehension and level steps, measured the same way before and after.
- Compared with what. The same pupils earlier, a similar class without the program, or last year's cohort.
- How big. In plain units first: stories, words, pupils who moved a step.
This four-part check is practice, built on the research below. No study has tested it as a whole.
Why cost-effectiveness is the better question
ROI compares money out with money back. Schools rarely get money back, so economists of education use a different tool. Levin and McEwan (2001) set out cost-effectiveness analysis for education. It compares alternatives that pursue the same goal, such as better reading, by both their costs and their effects.
The point is that the option with the largest effect is not always the best use of a budget. Harris (2009) argued that labels such as small, medium and large do little to inform decisions, because they ignore program costs. In his framework, an effect counts as large only if its ratio to cost is at least as good as that of the alternatives.
Kraft (2020) makes the same point in his guidelines for interpreting effect sizes. Effects from lower-cost programs are more impressive than similar effects from costlier ones. He adds that costlier programs should not be dismissed, since some problems need them.
Step 1: count every cost, including staff time
The fee is only one part of what a program costs a school. Levin and Belfield (2015) describe the ingredients method. List every resource the program needs, put a value on each one, then calculate the total and the average cost per participant. They note that school accounts were designed for other purposes, so they rarely show what one program costs.
Staff time is the ingredient schools most often miss. Kraft (2020) notes that a fuller analysis counts non-monetary costs, such as the opportunity cost of educators' time. Levin and Belfield (2015) value personnel at market prices, in the form of salaries and benefits.
| Ingredient | How to measure it | Often missed? |
|---|---|---|
| Fees or licences | The invoice, divided by the period it covers | Rarely |
| Devices and connection | The share of device time and data the program uses | Often |
| Teacher time beyond normal lessons | A time log kept for two ordinary weeks a term | Often |
| Coordinator time | The same time log | Often |
| Training | Hours of training times the number of staff, counted once | Sometimes |
| Books, printing and rewards | Receipts for the period | Sometimes |
Divide the total by the pupils who actually took part, not by those enrolled. The two figures can differ a lot. For the budgeting side, see What Should a School Reading Program Cost? A Budgeting Guide.
Step 2: what changed, and compared with what?
A change means something only beside a comparison. The strongest comparison a school can run on its own is a similar class that does not use the program, over the same weeks. Before-and-after figures for the same pupils show change, not cause. How to Measure CEFR Progress by Cohort sets out the kinds of comparison and their limits.
Watch the starting point. Campbell (1979) reported rumours from compensatory education of pretests given in a way designed to make scores as low as possible, so that larger gains would appear. Take the baseline the same way as the final check, before anyone has a stake in the result.
Measure what can move in the time you have. Over a few weeks that means participation, amount read, comprehension and staff time. A change of level takes much longer. How to Run a Pilot of a Reading Platform sets out a trial that a poor product would fail.
Step 3: how big, and at what cost per pupil?
Kraft (2020) proposed benchmarks for effect sizes from 747 randomised trials with standardised test outcomes. An effect under 0.05 standard deviations is small, 0.05 to under 0.20 is medium, and 0.20 or more is large. He paired them with per-pupil cost bands in 2016 US dollars. Under $500 is low, $500 to under $4,000 is moderate, and $4,000 or more is high.
Kraft calls the cost bands a rough guide, since they come from only 68 interventions. Both scales are for research studies with a control group. A school's own before-and-after figures are not that kind of evidence, so use the scales to set expectations, not to grade your result.
Worked example (invented figures, in your own currency)
A school compares two ways to raise reading in Grade 6 over one year, for 80 pupils.
Option A, a reading platform: fees 4,000; staff time 2 hours a week for 36 weeks at 30 an hour, 2,160; training 600. Total 6,760, about 85 per pupil.
Option B, a class library and a weekly reading lesson: books 3,000; staff time 3 hours a week for 36 weeks at 30 an hour, 3,240. Total 6,240, or 78 per pupil.
The middle pupil finished 30 stories in Option A and 20 in Option B, with similar comprehension scores.
Cost per story read by the middle pupil: about 2.8 for A and 3.9 for B.
A costs a little more per pupil but less per story read. Before deciding, the school still checks level steps and who read least in each option.
Notice what the example does. It counts staff time in both options, uses the same outcome for both, and puts the result in plain units before any ratio.
Why vendor ROI claims need checking
A vendor's ROI figure is a claim about other schools' pupils, measured by someone with an interest in the answer. Two lines of research say to read such figures with care.
Wolf and colleagues (2020) used study data from the What Works Clearinghouse. Evaluations carried out or commissioned by developers produced average effect sizes substantially larger than those from independent evaluations. The authors found evidence that this was not simply because the developers' programs worked better.
Cheung and Slavin (2016) examined 645 studies of preschool, reading, mathematics and science programs. Effect sizes were roughly twice as large in published articles, small-scale trials and studies using experimenter-made measures. They were also larger in quasi-experiments than in randomised experiments.
None of this means a vendor's figure is wrong. It means you should ask six questions before you use it.
- Who ran the study, and who paid for it?
- How many pupils took part, of what age and level, and for how long?
- Compared with what?
- Was the test made for the program, or was it a broad independent test?
- What did it cost per pupil, with staff time included?
- Can you speak to a school like yours that used it?
What to do when the results are mixed
Most honest evaluations are mixed. Pupils read more, but level steps were modest. Or comprehension rose, but staff time was higher than planned.
Report a mixed result in the same format you would use for a good one. Ask first whether the program was delivered as planned. If pupils read half the planned stories, you have tested a weak version of the program, not the program. Then choose one of three decisions, adopt, adapt or stop, and write down why.
The stage of the decision matters too. For a first approval, ask for a pilot rather than a projection. For a renewal, use your own data from the first year, not the vendor's figures. This advice comes from practice; no study cited here tests ways of reporting to leaders. KPIs & Metrics for a School Reading Program sets out the measures to keep from term to term.
How iRead handles this
A school can run a free pilot of iRead with one class for 4 to 6 weeks, with full dashboards for attendance and retention, before any school-wide decision. Attendance at sessions is recorded automatically. School leaders see levels reached, words retained and stories completed, by class, grade and term. Keep your own staff time log alongside, so the pilot gives you both sides of the ratio.
The core principle: Value for money is the effect set against the full cost, compared with the alternatives. A gain without its comparison or its staff time is not yet a return.
See it in iRead: A school can run a free pilot with one class for 4 to 6 weeks, with full dashboards for attendance and retention, before any school-wide decision.
See the leadership viewKey takeaway
Count every cost per pupil, staff time included, state what changed against a fair comparison, and size it in plain units. Treat vendor figures as claims to test in your own school.
Frequently asked questions
How do you calculate ROI for edtech in a school?
Most school programs do not return money, so judge cost-effectiveness instead. Count every cost per pupil, including staff time. Measure what changed against a fair comparison, and size it in plain units. Then compare that result with the other ways you could spend the same money and time.
What is cost-effectiveness analysis in education?
It is a way to compare programs that pursue the same goal by both their costs and their effects. Levin and McEwan (2001) set out the method for education. Costs are counted with the ingredients method, which values every resource a program uses, including staff time.
Should we trust a vendor's ROI figures?
Treat them as claims to check. Wolf and colleagues (2020) found that evaluations run or commissioned by developers reported substantially larger average effects than independent ones. Ask who ran the study, how many pupils took part, what the comparison was and which test was used.
How long should we trial a program before judging its value?
A pilot of four to six weeks can show participation, amount read, comprehension and staff time. It will rarely show a change of level. For a judgement about learning, plan to compare a full term or a year against a similar class that did not use the program.
Is a cheaper program always better value?
No. Value depends on the effect for each unit of cost, not on cost alone. A cheap program with no effect is poor value. Kraft (2020) notes that the same effect is more impressive from a cheaper program, but a costlier one can still be worth it.
Sources
- Levin, H. M., & McEwan, P. J. (2001). Cost-Effectiveness Analysis: Methods and Applications (2nd ed.). SAGE Publications, Thousand Oaks, CA. collegepublishing.sagepub.com/products/cost-effectiveness-analysis-2-3141
- Levin, H. M., & Belfield, C. (2015). Guiding the development and use of cost-effectiveness analysis in education. Journal of Research on Educational Effectiveness, 8(3), 400-418. doi.org/10.1080/19345747.2014.915604
- Harris, D. N. (2009). Toward policy-relevant benchmarks for interpreting effect sizes: Combining effects with costs. Educational Evaluation and Policy Analysis, 31(1), 3-29. doi.org/10.3102/0162373708327524
- Kraft, M. A. (2020). Interpreting effect sizes of education interventions. Educational Researcher, 49(4), 241-253. doi.org/10.3102/0013189X20912798
- Wolf, R., Morrison, J., Inns, A., Slavin, R., & Risman, K. (2020). Average effect sizes in developer-commissioned and independent evaluations. Journal of Research on Educational Effectiveness, 13(2), 428-447. doi.org/10.1080/19345747.2020.1726537
- Cheung, A. C. K., & Slavin, R. E. (2016). How methodological features affect effect sizes in education. Educational Researcher, 45(5), 283-292. doi.org/10.3102/0013189X16656615
- Campbell, D. T. (1979). Assessing the impact of planned social change. Evaluation and Program Planning, 2(1), 67-90. doi.org/10.1016/0149-7189(79)90048-X