I appreciate Robert Slavin’s baseball analogy in his HuffPost article ‘Pilot Studies: On the Path to Solid Evidence‘. Dr. Slavin is the Director of the Center for Research and Reform in Education at Johns Hopkins. He puts forward the idea that a new baseball strategy would be “headed towards proof” if 10 teams adopt a strategy and see comparative improvements at season’s end. It’s easy to take for granted that there’s universal data in sports: ranking are reported out in the sports section. So if we knew which teams adopted any new strategy in a given year, we could do an evaluation.
I would label this baseball analogy an unmatched-controls quasi-experiment. It pits teams using the new strategy vs whatever other ‘business-as-usual’ teams. Not a gold standard RCT, yet sensible that 10 teams improving their relative rankings is headed towards proof.
Back to edtech: you can get very close to the baseball analogy via published school proficiency rankings. You can improve it via matching controls. And there exist quantitative digital records of how much/well schools used digital programs. End of school-year summative tests are like a whole ‘season’. And for schoolwide programs, individual teachers and students aren’t opting in or out depending on motivation or capability. This quasi-experimental route, “headed for proof”, enables the game-changing advantages of volume and speed: use it for any new cohort of schools (>10), for any schoolwide program, every year. #edtechguidelines
Coulson on EdTech Evaluation 