Learning centricity vs Learner centricity: some thoughts on Dave Cormier’s human-centred model for discussing AI in assessment

I don’t think I’ve ever read any published article by Dave Cormier that I didn’t like, and I’ve read quite a few. This latest blog post, “A human-centred model for discussing AI in assessment” is no exception. In it, Dave describes his framework for discussing AI in assessment with teachers and other course designers, revolving around 3 questions:

• Did they do it?
• Have they learned?
• Are we helping?

I love the simplicity:  I like how the questions cut through to what matters with absolutely no jargon or equivocal wording. I really like the emphasis on learning and the lack of explicit AI focus. These are good questions for any credentialed work, not just in the context of AI. I think I’d be inclined to add one more question:

• Have we learned?

because this is really the point of it all, and we should be designing our assessments so that we do learn, but maybe that is implicit in the context. There might be other questions to ask like “To what extent can we prove this?” or “Is it equitable?” or (my favourite) “Did it bring joy?” and so on, because there are bigger systems into which this needs to slot, but maybe they are for a different and more focused conversation.

However, though questions are great, the diagram puzzles me:

After poring over it for a while, I think the overlaps are meant to represent where the answer to the question is “yes” (so the sweet spot is where they did it, we helped, and they learned in the process) while the non-overlapping elements are those to which the answer is “no”, and the labels of the intersections are ways of describing the results of aligning them. If so, it’s an odd way of using Venn diagrams. The core problem is that the questions are not sets, so they cannot intersect and, to make it worse, they are questions about different kinds of entity that could never be overlapping sets:

• The work submitted for assessment (artifacts).
• What students learned (cognitive changes).
• What we did to try to help them learn (interventions).

I like the idea of a Venn diagram for this, though, so I started to wonder what should really be in those sets, and what follows is the result.

A learning centred model for discussing AI and assessment

Rather than focusing on the whole learner, I think would make more sense were we to look at the questions from the perspective of the learning (latent or actual) that each question is asking about:

• Intended learning: the knowledge, including subject and pedagogical knowledge and skills, both tacit and explicit, that the teacher (a distributed entity including designated teachers, the learner, other learners, textbook authors, institutional systems and policies, etc) intends to facilitate. Subsumes learning outcomes but also includes ways of thinking, learning, acting, etc.
• Actual learning: the knowledge and skills gained in the process. This speaks not only to what is learned but also to how it is accommodated and integrated with existing knowledge and skills.
• Learning exhibited in assessed work: some being the result of what the student did, perhaps some being from something/someone else, for more or less legitimate reasons.

In real life, this is all deeply intertwingled and learning cannot possibly fall so neatly into measurable sets. Any measurements are almost entirely notional. We might be able to get some clues about the knowledge it represents,  but knowledge is not neatly quantifiable either. It can be extended, embodied, enacted, or embedded, and it’s a collective as well as an individual phenomenon but, even from an individual perspective, it’s not a thing you can count. It’s a thing you do as much as a thing you have. However, this is a model, not the reality that it models, and it is to support a conversation about design and performance, so we don’t need precision or anything like it.  It just needs to be good enough to be able to talk about what we need to talk about.

Here’s my attempt at a quick and dirty diagram to represent this:

Dave’s questions can easily be overlaid on top of this in order to explore how it plays out in any particular course, as can quite a few others. It illustrates that some or all of the work submitted might not contribute to student learning, and that some or all of it might not be the result of our teaching. It correctly shows that we routinely assess work that students did not do, and that we only assess a sample of what was learned.  It highlights the fact that a lot of learning happens that is neither taught nor assessed, that we teach things we do not assess, that teaching behaviours don’t always result in learning, and we assess things we do not teach. It represents the fact that learning can happen even when the student is not the creator of the work that is supposed to lead to it and/or represent it, which can be especially significant in a genAI context. It allows us to overlay plenty of other questions, including the extent to which the tasks we set and the time we allowed for doing them made it more likely that the work submitted was not that of the student.

Possibly wasted efforts?

I’ve labelled learning that is exhibited in the assessed work, or only in the teacher’s intentions, but not in the student, as possibly wasted effort. This is not necessarily a bad thing, as long as the learning that did occur was sufficient and sufficiently worthwhile. In some cases it is absolutely normal and acceptable: for example, if a student provides a website that uses frameworks or libraries, it would be incomplete without them, even though the knowledge they signify is not that of the student (depending on how you understand “knowledge”).  It is also part of the process that it has to be a bit lossy. Learning needs breathing room for scaffolds, connections, and shuffling things around that don’t immediately fit.  It’s not information: it’s a living, breathing, active thing.

All that said, on the whole, I think it would be better if more learning had occurred. Anything that doesn’t fall within the “actual learning” set is stuff that could have been but has not been learned. And, for credentialed assessment, it is quite important that we don’t wind up awarding credentials for things the students have not learned. All that said, the contents of these subsets can indicate failure: if a student uses a generative AI or hires someone to create the work, it may well be a complete waste of time. If a teacher attempts to teach something but the student does not learn it then, though there is a possibility that the student learned other useful things (like to avoid the teacher’s classes), it seems a little wasteful.

Definitely wasted efforts

One of the most interesting subsets is the intersection of work submitted for assessment that complies with the requirements, that we tried to teach, but that is not the result of learning, at least of the sort we seek. Unless it is flagged as such and done intentionally, I think of this as the zone of unreliable assessment. It is the “bad AI” and copypasta zone, but there are plenty of other ways it can occur, including when students formulaically do something that meets the outcomes without understanding why or how it works. Whatever the cause, if we award credentials for knowledge and skills that students lack, we are failing in our credentialing role. This is not a big issue if it is ungraded, of course: it might even be a good thing that assists further learning. It is not unheard of for unlearned stuff to be the only thing we assess, though, and that unequivocally sucks. It’s not just a missed goal but an own goal.

Undiscovered outcomes

Another really interesting subset is the zone of unmeasured outcomes: outcomes for which there is clear evidence in the work provided, but that don’t fit the rubric or intended outcomes of the teacher. I think there is great scope for outcome harvesting here. It is not unusual for the most important outcomes of a course to be unintended and unmeasured, but it is quite unusual for us to provide credentials for them. I think we should try to do so, if we are serious about being human-centred. And, if there are lots of untaught outcomes that consistently appear in the assessed work, it might imply a shortfall in our teaching or description of the course. It’s a lot of work to mine learning outcomes, but GenAIs could be helpful: if we provided them with the work, intended outcomes, and a rubric, then asked what other competences were demonstrated in the work, they might provide a sufficiently close approximation for us to make the judgment calls needed to help our students.

Unmeasured teaching

The zone of unmeasured teaching contains the learning outcomes that we intentionally taught and that were met but that we didn’t assess. I don’t see this as a particularly major issue, apart from the fact that, if we are giving credentials rather than just using assessment for learning purposes, it would be kinder to make it possible for this additional evidence to be considered. This is a good reason for using a portfolio approach to assessment.  If we are not doing that, then this is the zone in which to look for alternative assessments or better constructive alignment.

Mysterious learning

The most interesting zone of all (to me) is the one I have labelled as “Here there be dragons” which, following the lead of ancient cartographers, is my way of describing the significant subset of student knowledge about which we know little or nothing, that occurs while students are learning what we are teaching, that is not reflected (directly) in the assessed work, but that is neither what we taught nor what we intended to teach. This is the frontier territory that few of us ever enter unless we are committed complexivists, but that I think we need to explore most of all. This is especially so in an age of generative AI, where our traditional proxies for learning are breaking all around us and learners have far more supports outside the institution than within it. Reflective portfolios can help reveal a little of it, as can approaches to assessment in which we ask for evidence (any evidence) rather than compliance with our demands. It’s another space where AIs that could observe students  learning might help, if everyone involved happened to be OK with some huge privacy, security, and trust issues. Knowing more about this and celebrating its existence opens up the potential for more learner-controlled outcomes, for the teacher to also be the taught, and for moving more of the unmeasured outcomes into the sweet spot.

The sweet spot

The sweet spot – essentially the happy intersection between teaching, learning, and assessment –  is where constructive alignment occurs, and it should normally be as large as possible. Its relative size is a good proxy for the effectiveness of teaching and assessment, especially if the zone of unreliable assessment is small. There will never and should never be a perfect overlap as long as the knowledge is worth knowing, and there is always knowledge embedded in artifacts that is only borrowed from its creators, but the more we can know of what and how the students know, the more they know of what we know, the more that the evidence of knowledge in what we assess overlaps with what students actually know, the better it will be for everyone. The two-way knowledge flow between learner and teacher (bearing in mind teachers include the students themselves) is particularly important for achieving that.

Is this model in any way usable?

I don’t know if this is any better for this purpose than Dave’s original model: probably not. It’s more cohesive as a Venn diagram but it is more opaque as a prompt for dialogue, and I’m not entirely sure my model captures quite the same phenomena. In caricature, Dave’s is about learners, while mine is about learning. In conjunction with his questions, though, I think it might help to reveal more about both.