Design memo: why our reviews never show their own answers

Design memos document real decisions in Wiz Kids and the research that forced them — including, as here, the ones we got wrong first.

The mistake we shipped

Wiz Kids runs a daily review queue on a spaced schedule: when a skill starts to fade, a short task asks the child to prove they still have it (spaced repetition). Early on, we took a shortcut: many skills reused their teaching task as their review task. It seemed harmless — the task exercises the skill, doesn't it?

Then we looked at what a child actually sees. Our keyboard-shortcut teaching tasks display the chord on an instruction card — "Copy it! Ctrl+C" — because that's what teaching looks like. Which meant the review, weeks later, also displayed Ctrl+C on the card. The child read a keycap and pressed it. Our spreadsheet reviews printed the formula being reviewed in the instructions. The review queue was faithfully scheduling… reading practice.

What the research says we were doing

The testing-effect literature is unambiguous about the mechanism: the memory benefit comes from retrieval — reconstructing the answer from memory. Re-exposure without retrieval (re-reading, copying) is among the weakest study conditions in the entire literature (Roediger & Karpicke, 2006; Dunlosky et al., 2013 rate re-reading "low utility"). When the answer is visible, no retrieval occurs; the event downgrades from the strongest known practice form to the weakest. And the cruelest part: it looks identical in the data. The child "passes the review." Fluency without memory — Bjork's performance/learning distinction wearing our own UI.

The audit and the rule

We audited all 115 skills' review tasks and classified every teaching-task reuse: judgment tasks were fine (a scenario question doesn't contain its answer), practice tasks were fine (typing a passage IS retrieval of motor patterns), but any task that displayed the target action was broken as a review — every shortcut card, every formula-stating spreadsheet instruction. Fifteen skills. Each got a dedicated review variant, and the incident became a standing authoring rule enforced in review:

A review task must never contain its own answer. It states the goal; the method must come from the child.

The design that replaced it — struggle first, help second

For keyboard shortcuts we built a recall mode: the review card states the goal ("Copy your catch!") and shows, where the keycap used to be, "🤫 from memory!". If the child struggles — wrong attempts, visible floundering past a threshold — the chord reveals itself as a hint, after the attempt. That ordering is deliberate and evidence-based: retrieval attempts followed by feedback preserve most of the testing benefit, and even failed retrieval followed by the answer beats being shown the answer up front (Roediger & Butler, 2011; the "pretesting"/errorful-generation literature, e.g., Kornell, Hays & Bjork, 2009). The child who remembers gets a genuine retrieval event; the child who can't gets the answer at the moment it teaches most — and the review reschedules sooner.

Spreadsheet reviews now state questions ("make the sheet find the fair share") and keep formulas out of instructions entirely; the formula appears only in the over-par hint, which by construction arrives after real attempts.

What this costs, and why it's worth it

Recall-mode reviews are harder. Pass rates dipped; some children meet "🤫 from memory!" and stall for ten seconds. That's the point — the ten seconds of reaching is the strengthening (desirable difficulties). A review system optimized to feel smooth optimizes away its own reason to exist. We'd rather our completion metrics look slightly worse and be true.

The general lesson for anyone evaluating practice software (ours included): open a review activity and ask one question — could a child complete this without remembering anything? If yes, whatever it's measuring, it isn't memory.

References

• Roediger, H. L., & Karpicke, J. D. (2006). Test-enhanced learning. Psychological Science, 17(3).
• Dunlosky, J., et al. (2013). Improving students' learning with effective learning techniques. Psychological Science in the Public Interest, 14(1).
• Roediger, H. L., & Butler, A. C. (2011). The critical role of retrieval practice in long-term retention. Trends in Cognitive Sciences, 15(1).
• Kornell, N., Hays, M. J., & Bjork, R. A. (2009). Unsuccessful retrieval attempts enhance subsequent learning. Journal of Experimental Psychology: Learning, Memory, and Cognition, 35(4).
• Bjork, E. L., & Bjork, R. A. (2011). Making things hard on yourself, but in a good way. In Psychology and the Real World.

© Glu IO Pty. Ltd. — Wiz Kids (wiz.kids). Link freely; republication requires permission — see terms. Found an error in our reading of the research? We correct fast: tell any teacher piloting Wiz Kids.