Reading education research critically: a teacher's field guide

What this page is: the skills half of our grading methodology — the questions we ask of every source, written up so you can ask them yourself, including of us. Teachers are marketed to relentlessly with "research shows"; this is the twenty-minute self-defense course.

"Research shows" is doing more unsupervised work in education than any phrase in the profession. The remedy isn't cynicism — real findings exist and this library is built on them — it's a short checklist applied consistently. Seven questions:

1. What kind of study is behind the claim?

The hierarchy, roughly: randomized controlled trials (can support "causes"), quasi-experiments and longitudinal studies (weaker but real), correlational studies ("is associated with" — never "causes," however the press release reads), surveys and self-report (what people say, which for skills diverges wildly from what they do). And above singles: meta-analyses — with their own trap: garbage-in-garbage-out, so a meta-analysis of weak studies launders weakness into a confident number (the effect-sizes page continues this thread).

2. Who was studied — and does it stretch to your class?

University undergraduates power an enormous share of psychology; primary-age evidence is chronically thinner, and this library flags the gap wherever it applies. Sample size matters less than sample relevance plus size: twenty eight-year-olds may inform a primary claim better than two thousand undergraduates.

3. What was actually measured?

The outcome-measure question catches the most edtech marketing: "engagement" is not learning, minutes-of-use is not skill, satisfaction is not retention, and completion is not competence. Ask what the improved number was, then ask whether you'd accept that number as evidence in your own classroom. Also: when was it measured? Immediate post-tests flatter everything; retention is where learning lives, and follow-up measurement is rare precisely because it's unflattering.

4. How big is the effect — and is it "significant" or significant?

Statistical significance means "probably not zero," not "large" — with big samples, trivial effects reach p < .05 reliably. Always look for the effect size, then apply realistic benchmarks — the short version: field-trial effects of 0.1–0.2 SD are respectable, and vendor decks quoting lab-derived d = 0.8 for classroom products deserve the raised eyebrow.

5. Who paid, and who benefits?

Funding doesn't falsify findings, but it predicts which findings get run, finished and published. Vendor-funded efficacy studies of the vendor's product warrant the same posture as a vendor's privacy claims: verify, don't trust. That includes us — every "How Wiz Kids applies this" section in this library sits under a disclosed stake, and our position notes exist because this question applies to their author.

6. Has it replicated?

Psychology's replication reckoning hit education's favorites hard — learning styles collapsed entirely, ego depletion wobbled, mindset shrank to honest size. The practical heuristic: trust findings with multiple independent replications (spacing, retrieval, the ones we grade STRONG); hold single-study findings — however charming — as "promising," and be suspicious when a whole product rests on one paper.

7. Is it research at all?

The grey-literature trap: white papers, vendor "studies," conference posters and advocacy reports wear research's costume without its review. The tells: no methods section, no sample description, percentages without denominators, and citations that loop back to the organisation itself. Grey literature can still inform (regulator surveys like Ofcom's are load-bearing here) — the skill is knowing which kind of document you're holding and weighting it accordingly.

The disposition underneath

The checklist compresses to one habit, the same one we teach nine-year-olds about sources: ask who says this, how do they know, and who else agrees — applied to claims about teaching itself. Calibration, not cynicism: the goal is trusting the spacing effect more and the brain-training deck less, which is what evidence-literacy is for.

References

Pashler, H., et al. (2008). PSPI, 9(3) — the learning-styles verdict: the checklist's canonical worked example.
Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science, 349 — the replication reckoning.
Kraft, M. A. (2020). Interpreting effect sizes of education interventions. Educational Researcher, 49(4) — question 4's benchmark source.
Wineburg, S., & McGrew, S. (2019). TCR, 121(11) — the who-says-this discipline, all the way up.

© Glu IO Pty. Ltd. — Wiz Kids (wiz.kids). Link freely; republication requires permission — see terms. Found an error in our reading of the research? We correct fast: tell any teacher piloting Wiz Kids.