Pursuing validity

Yo 👋

Hope life is sunny. This week, we’re kicking off a new series on assessment theory, by unpacking the crucial concept of validity…

Big idea 🍉

Learning is invisible. Assessment is the process by which we attempt to make learning more visible, so we can do things like give students a grade (summative assessment) or guide our subsequent teaching (formative assessment). A key concept to understand within assessment is validity.

Validity refers to the extent that any conclusions (aka 'inferences') we draw from an assessment are an accurate reflection of reality.

For example, if I claim that you’re good at juggling based on how well you describe the process of juggling, then the validity of my inference is low. Whereas if I claim that you are good at juggling based on how well you juggle 3 balls for 1 minute, then the validity of my inference is much higher.

Note that validity isn't a property of a test itself. Instead, it relates to the inferences we draw as the result of a test.

Let's say I asked my class to give me a few examples of osmosis. If they all answered correctly, I could conclude—with reasonable validity—that they understand osmosis. But any conclusions I draw about their wider understanding of science would be much less valid.

Designing assessments that yield valid insights about general understanding becomes trickier as subjects grow in scope and complexity.

For example, I could construct a fairly valid assessment of a 6-year-old’s times-tables understanding by testing them on all their times-tables. But if I want to gauge the mathematical understanding of a 16-year-old, I can't easily test them on everything they should know... this would take days. Instead, we typically use a carefully selected sample of the domain to make as valid an inference as possible with the 2 hours I have available for examination.

Finally, the stakes we attach to any assessment should be in proportion to the validity of our inferences. Allocating students a grade which influences their access to employment or further education should only be done if we have high levels of confidence that our conclusions reflect reality. Otherwise, it just ain’t fair.

(A similar health warning exists for basing pay or promotion on observations of teaching, or job allocation on interview performance).

🎓 For more, check this short article on assessment from Daisy Christodoulou, or better yet: her ace book on the subject.

Summary

• Learning is invisible. Assessment is the process by which we attempt to make it more visible.

• Validity refers to how well any inference we make from an assessment reflects the reality of student understanding.

• We should avoid attaching high stakes consequences to low validity inferences.

Little updates 🥕

• Study comparing storytelling and traditional textbook methods in science → suggests novices learn best with stories throughout, while advanced students benefit from stories followed by textbooks.

• Series of experiments examining how feedback affects mental effort → finds positive feedback boosts confidence, while negative feedback makes tasks seem harder, both boosting motivation!

• Paper comparing student feelings about school based on teacher ratings of their abilities → suggests students who are overestimated feel happier and more confident in school.

Would you like to support the ongoing production of Snacks? Become a benefactor, sign up to PRO today → learn more