Scores Can Rise After Training for Reasons Beyond Training
Suppose employees with low scores on a product-knowledge test receive extra training, and their average rises on the next test. That result alone cannot tell you how many points the training added. Selecting people who happened to score unusually poorly can itself be followed by an increase in their average score.
When people are selected for low initial scores, their average on a second test can move closer to the overall average of the population they were selected from. This phenomenon is called regression to the mean. The abstract of Barnett and colleagues' methodological paper explains that the effect becomes more noticeable when follow-up examines only a subset selected using the initial measurement. (Regression to the mean: what it is and how to deal with it)
Evaluating training requires asking not only whether participants' scores rose, but how their scores differed from what would have happened without the training.
An Initial Low Score Includes Temporary Variation
A single test score reflects both knowledge and variation specific to that occasion, such as a favorable or unfavorable set of questions or an accidental misreading. Selecting low scorers includes people who lack knowledge alongside people who happened to perform below their usual level.
The temporary reduction need not recur to the same extent. Even without additional learning, the selected group's average may rise on the next test. Conversely, selecting only the highest initial scorers can be followed by a lower average. (The need to control for regression to the mean in social psychology studies)
This does not mean every person's next score must move toward the average. Nor does an observed increase prove that all the improvement was regression to the mean. Learning from training and temporary variation can both contribute.
An Average Can Rise From 60 to 70 Without Additional Knowledge
The following is a hypothetical calculation illustrating the mechanism. Assume four employees have unchanged knowledge and would each average 70 points across repeated tests. On each test, each employee independently has an equal chance of scoring either 60 or 80. Nobody receives additional training.
The table shows one possible set of results under those assumptions. These are invented values, rather than measurements from an actual training program.
| Employee | First test score | Second test score | Selected as an initial low scorer |
|---|
| A | 60 | 60 | Yes |
| B | 60 | 80 | Yes |
| C | 80 | 60 | No |
| D | 80 | 80 | No |
Following only A and B, who initially scored 60, produces an increase from an average of 60 to (60 + 80) ÷ 2 = 70 points. The selected pair improved by 10 points on average despite receiving no training. The average for all four employees remained 70 on both tests.
Under this model, someone who initially scored 60 still has a one-half chance of scoring 60 and a one-half chance of scoring 80 on the second test. Their expected second score is therefore 70. Both selected employees could score 60 again, however; the table need not recur on every occasion.
Real employees differ in knowledge and in how much their test scores fluctuate. You cannot subtract this example's 10 points from the results of an actual training program as a universal adjustment.
Randomly Assign Eligible Employees to Training or a Waiting Group
One proposed application is to select employees using the same score threshold, then randomly assign them either to immediate additional training or to a waiting group that receives it after the next test. This approach applies where postponing that training is operationally acceptable.
Because both groups were selected using the same rule, both can experience regression to the mean after initial low scores. Yu and Chen discuss combining a comparison group with random assignment and comparing changes between groups. Their paper examines inference from repeated measurements; it did not measure the effectiveness of corporate training. (The need to control for regression to the mean in social psychology studies)
For example, assume both groups initially average 50 points. On the next test, the training group averages 65 and the waiting group averages 60. The training group gained 15 points, while the waiting group gained 10. The difference in their gains is 15 − 10 = 5 points.
That five-point difference provides an estimate of the average effect of additional training. Chance differences remain possible, especially with small groups, so the estimate's uncertainty also needs evaluation using group sizes and variation in scores. The difference alone does not establish a definite effect.
Use tests administered during the same period with the same scoring criteria. Employees in the waiting group may learn through their usual work, so the comparison estimates the effect of adding training to learning through usual work. Sharing training materials with the waiting group or having substantially more absences in one group changes the meaning of the planned comparison.
The Whole Workforce Does Not Represent Untrained Low Scorers
Comparing the trained low scorers with the entire workforce, including high scorers, uses groups selected differently. Low scorers can improve while the workforce average stays unchanged even without training, as the hypothetical example shows.
When a comparison group is unavailable, distinguish the observation that participants' average rose from 50 to 65 from the estimate that training added 15 points. Before-and-after scores can document learning outcomes, but those records alone cannot reveal the difference from what would have happened without training.
Distinguish a Score Increase From an Increase Caused by Training
An increase among initial low scorers does not by itself establish training effectiveness. Selecting people for low initial scores can be followed by an increase without training. To evaluate additional training, use a design that estimates the outcome without it, such as a comparison with a waiting group selected using the same criteria.
When presenting participant results on a training business's website, describe the test, participant count, measurement timing, and whether there was a comparison group. These details explain what the published scores mean. Avoid presenting the entire before-and-after increase as an effect attributable solely to training.
SpreadSite is a no-code website builder that lets you create, publish, and update a website from Google Sheets using ready-made templates.
To create a website that clearly presents your training content and participant results, give SpreadSite a try!
Build a website with SpreadSite →
