Misplaced confidence

Hey 👋

Hope you’re having a lovely week. Today, we’re wrapping up our series on assessment theory with a cheeky peek at measurement error & confidence…

Big idea 🍉

Reliability refers to the ability of a measure to produce a similar result under similar conditions. If I weigh 70kg and my scales always show 70kg, they are reliable. Lovely.

However, when I use my scales in the garden, they aren't quite as reliable (the grass messes with their mechanics). They tend to fluctuate by about 2kg, and so for me they show a result somewhere between 68-72kg.

How important is this error? Well, it depends on the relative size of the change that we are hoping to measure.

Let's say I wanted to track the weight of a local baby elephant on an annual basis. Setting aside several technical issues, my scales might well be reliable enough to give us confidence that Ellie is gaining weight in an expected manner (approx. 100kg per year) and so is in tip top health.

However, let's say that I also wanted to track the weight of a local middle-aged fox on a daily basis. Would my garden scales be an adequate tool? Probably not, because the change we’d expect to see (a few g’s maybe) would be dwarfed by the error (2kg). It would be super hard for me to make any valid inferences about Foxie’s weight change. Any attempt to do so would represent misplaced confidence.

What's all this got to do with school? Well, learning is hard to measure (especially at the level of the individual). AND it's slow to develop. As a result, if we try to (summatively) assess our students over too short a time interval, we will find ourselves at risk of misplaced confidence. Which can not only lead to us adapting teaching in unhelpful ways, but also to generating unnecessary data-management workload.

Important → We're talking about summative assessment here. High frequency formative assessment is a very different (and positive) thing.

🎓 For more, see this analysis of measurement error, data drops and teacher workload, by big brain Dr Sam Sims.

Summary

• It’s important that we try to consider measurement error when making inferences from assessments.

• If the error is large relative to the change we are hoping to see, there is potential for misplaced confidence.

• This can lead to adapting teaching in unhelpful ways and the generation of unnecessary workload.

Little updates 🥕

• Study of boredom due to over- and under-challenge in math(s) → suggests that many students experience high boredom regardless of achievement, with girls particularly at risk of over-challenge.

• Paper exploring student–teacher gender congruence → finds similarity improves performance for females in STEM and for males in languages, with effects varying based on local contexts.

• Analysis of the impact of later school start times → shows initial increases in adolescent sleep and reductions in substance use, but these diminish after a year.

Support the sharing, sign up to Snacks PRO → join here