Learning centricity vs Learner centricity: some thoughts on Dave Cormier’s human-centred model for discussing AI in assessment

Posted , updated on

I don’t think I’ve ever read any published article by Dave Cormier that I didn’t like, and I’ve read quite a few. This latest blog post, “A human-centred model for discussing AI in assessment” is no exception. In it, Dave describes his framework for discussing AI in assessment with teachers and other course designers, revolving around 3 questions:

  • Did they do it?
  • Have they learned?
  • Are we helping?

I love the simplicity:  I like how the questions cut through to what matters with absolutely no jargon or equivocal wording. I really like the emphasis on learning and the lack of explicit AI focus. These are good questions for any credentialed work, not just in the context of AI. I think I’d be inclined to add one more question:

  • Have we learned?

because this is really the point of it all, and we should be designing our assessments so that we do learn, but maybe that is implicit in the context. There might be other questions to ask like “To what extent can we prove this?” or “Is it equitable?” or (my favourite) “Did it bring joy?” and so on, because there are bigger systems into which this needs to slot, but maybe they are for a different and more focused conversation.

However, though questions are great, the diagram puzzles me:

After poring over it for a while, I think the overlaps are meant to represent where the answer to the question is “yes” (so the sweet spot is where they did it, we helped, and they learned in the process) while the non-overlapping elements are those to which the answer is “no”, and the labels of the intersections are ways of describing the results of aligning them. If so, it’s an odd way of using Venn diagrams. The core problem is that the questions are not sets, so they cannot intersect and, to make it worse, they are questions about different kinds of entity that could never be overlapping sets:

  • The work submitted for assessment (artifacts).
  • What students learned (cognitive changes).
  • What we did to try to help them learn (interventions).

I like the idea of a Venn diagram for this, though, so I started to wonder what should really be in those sets, and what follows is the result.

A learning centred model for discussing AI and assessment

Rather than focusing on the whole learner, I think would make more sense were we to look at the questions from the perspective of the learning (latent or actual) that each question is asking about:

  • Intended learning: the knowledge, including subject and pedagogical knowledge and skills, both tacit and explicit, that the teacher (a distributed entity including designated teachers, the learner, other learners, textbook authors, institutional systems and policies, etc) intends to facilitate. Subsumes learning outcomes but also includes ways of thinking, learning, acting, etc.
  • Actual learning: the knowledge and skills gained in the process. This speaks not only to what is learned but also to how it is accommodated and integrated with existing knowledge and skills.
  • Learning exhibited in assessed work: some being the result of what the student did, perhaps some being from something/someone else, for more or less legitimate reasons.

In real life, this is all deeply intertwingled and learning cannot possibly fall so neatly into measurable sets. Any measurements are almost entirely notional. We might be able to get some clues about the knowledge it represents,  but knowledge is not neatly quantifiable either. It can be extended, embodied, enacted, or embedded, and it’s a collective as well as an individual phenomenon but, even from an individual perspective, it’s not a thing you can count. It’s a thing you do as much as a thing you have. However, this is a model, not the reality that it models, and it is to support a conversation about design and performance, so we don’t need precision or anything like it.  It just needs to be good enough to be able to talk about what we need to talk about.

Here’s my attempt at a quick and dirty diagram to represent this:

Venn diagram focused on learning

Dave’s questions can easily be overlaid on top of this in order to explore how it plays out in any particular course, as can quite a few others. It illustrates that some or all of the work submitted might not contribute to student learning, and that some or all of it might not be the result of our teaching. It correctly shows that we routinely assess work that students did not do, and that we only assess a sample of what was learned.  It highlights the fact that a lot of learning happens that is neither taught nor assessed, that we teach things we do not assess, that teaching behaviours don’t always result in learning, and we assess things we do not teach. It represents the fact that learning can happen even when the student is not the creator of the work that is supposed to lead to it and/or represent it, which can be especially significant in a genAI context. It allows us to overlay plenty of other questions, including the extent to which the tasks we set and the time we allowed for doing them made it more likely that the work submitted was not that of the student.

Possibly wasted efforts?

I’ve labelled learning that is exhibited in the assessed work, or only in the teacher’s intentions, but not in the student, as possibly wasted effort. This is not necessarily a bad thing, as long as the learning that did occur was sufficient and sufficiently worthwhile. In some cases it is absolutely normal and acceptable: for example, if a student provides a website that uses frameworks or libraries, it would be incomplete without them, even though the knowledge they signify is not that of the student (depending on how you understand “knowledge”).  It is also part of the process that it has to be a bit lossy. Learning needs breathing room for scaffolds, connections, and shuffling things around that don’t immediately fit.  It’s not information: it’s a living, breathing, active thing.

All that said, on the whole, I think it would be better if more learning had occurred. Anything that doesn’t fall within the “actual learning” set is stuff that could have been but has not been learned. And, for credentialed assessment, it is quite important that we don’t wind up awarding credentials for things the students have not learned. All that said, the contents of these subsets can indicate failure: if a student uses a generative AI or hires someone to create the work, it may well be a complete waste of time. If a teacher attempts to teach something but the student does not learn it then, though there is a possibility that the student learned other useful things (like to avoid the teacher’s classes), it seems a little wasteful.

Definitely wasted efforts

One of the most interesting subsets is the intersection of work submitted for assessment that complies with the requirements, that we tried to teach, but that is not the result of learning, at least of the sort we seek. Unless it is flagged as such and done intentionally, I think of this as the zone of unreliable assessment. It is the “bad AI” and copypasta zone, but there are plenty of other ways it can occur, including when students formulaically do something that meets the outcomes without understanding why or how it works. Whatever the cause, if we award credentials for knowledge and skills that students lack, we are failing in our credentialing role. This is not a big issue if it is ungraded, of course: it might even be a good thing that assists further learning. It is not unheard of for unlearned stuff to be the only thing we assess, though, and that unequivocally sucks. It’s not just a missed goal but an own goal.

Undiscovered outcomes

Another really interesting subset is the zone of unmeasured outcomes: outcomes for which there is clear evidence in the work provided, but that don’t fit the rubric or intended outcomes of the teacher. I think there is great scope for outcome harvesting here. It is not unusual for the most important outcomes of a course to be unintended and unmeasured, but it is quite unusual for us to provide credentials for them. I think we should try to do so, if we are serious about being human-centred. And, if there are lots of untaught outcomes that consistently appear in the assessed work, it might imply a shortfall in our teaching or description of the course. It’s a lot of work to mine learning outcomes, but GenAIs could be helpful: if we provided them with the work, intended outcomes, and a rubric, then asked what other competences were demonstrated in the work, they might provide a sufficiently close approximation for us to make the judgment calls needed to help our students.

Unmeasured teaching

The zone of unmeasured teaching contains the learning outcomes that we intentionally taught and that were met but that we didn’t assess. I don’t see this as a particularly major issue, apart from the fact that, if we are giving credentials rather than just using assessment for learning purposes, it would be kinder to make it possible for this additional evidence to be considered. This is a good reason for using a portfolio approach to assessment.  If we are not doing that, then this is the zone in which to look for alternative assessments or better constructive alignment.

Mysterious learning

The most interesting zone of all (to me) is the one I have labelled as “Here there be dragons” which, following the lead of ancient cartographers, is my way of describing the significant subset of student knowledge about which we know little or nothing, that occurs while students are learning what we are teaching, that is not reflected (directly) in the assessed work, but that is neither what we taught nor what we intended to teach. This is the frontier territory that few of us ever enter unless we are committed complexivists, but that I think we need to explore most of all. This is especially so in an age of generative AI, where our traditional proxies for learning are breaking all around us and learners have far more supports outside the institution than within it. Reflective portfolios can help reveal a little of it, as can approaches to assessment in which we ask for evidence (any evidence) rather than compliance with our demands. It’s another space where AIs that could observe students  learning might help, if everyone involved happened to be OK with some huge privacy, security, and trust issues. Knowing more about this and celebrating its existence opens up the potential for more learner-controlled outcomes, for the teacher to also be the taught, and for moving more of the unmeasured outcomes into the sweet spot.

The sweet spot

The sweet spot – essentially the happy intersection between teaching, learning, and assessment –  is where constructive alignment occurs, and it should normally be as large as possible. Its relative size is a good proxy for the effectiveness of teaching and assessment, especially if the zone of unreliable assessment is small. There will never and should never be a perfect overlap as long as the knowledge is worth knowing, and there is always knowledge embedded in artifacts that is only borrowed from its creators, but the more we can know of what and how the students know, the more they know of what we know, the more that the evidence of knowledge in what we assess overlaps with what students actually know, the better it will be for everyone. The two-way knowledge flow between learner and teacher (bearing in mind teachers include the students themselves) is particularly important for achieving that.

Is this model in any way usable?

I don’t know if this is any better for this purpose than Dave’s original model: probably not. It’s more cohesive as a Venn diagram but it is more opaque as a prompt for dialogue, and I’m not entirely sure my model captures quite the same phenomena. In caricature, Dave’s is about learners, while mine is about learning. In conjunction with his questions, though, I think it might help to reveal more about both.

Slides from my ICEEL ’24 Keynote: “No Teacher Left Behind: Surviving Transformation”

Posted , updated on

Here are the slides from from my keynote at the 8th International Conference on Education and E-Learning in Tokyo yesterday. Sadly I was not actually in Tokyo for this but the online integration was well done and there was some good audience interaction. I am also the conference chair (an honorary title) so I may be a bit biased, but I think it’s a really good conference, with an increasingly rare blend of both the tech and the pedagogical aspects of the field, and some wonderfully diverse keynotes ranging in subject matter from the hardest computer science to reflections on literature and love (thanks to its collocation with ICLLL, a literature and linguistics conference). My keynote was somewhere in between, and deliberately targeted at the conference theme, “Transformative Learning in the Digital Era: Navigating Innovation and Inclusion.”

the technological connectome, represented in the style of 1950s children's booksAs my starting point for the talk I introduced the concept of the technological connectome, about which I have just written a paper (currently under revision, hopefully due for publication in a forthcoming issue of the new Journal of Open, Distance, and Digital Education), which is essentially a way of talking about extended cognition from a technological rather than a cognitive perspective. From there I moved on to the adjacent possible and the exponential growth in technology that has, over the past century or so, reached such a breakneck rate of change that innovations such as generative AI, the transformation I particularly focused on (because it is topical), can transform vast swathes of culture and practice in months if not in weeks. This is a bit of a problem for traditional educators, who are as unprepared as anyone else for it, but who find themselves in a system that could not be more vulnerable to the consequences. At the very least it disrupts the learning outcomes-driven teacher-centric model of teaching that still massively dominates institutional learning the world over, both in the mockery it makes of traditional assessment practices and in the fact that generative AIs make far better teachers if all you care about are the measurable outcomes.

The solutions I presented and that formed the bulk of the talk, largely informed by the model of education presented in How Education Works, were mostly pretty traditional, emphasizing the value of community, and of passion for learning, along with caring about, respecting, and supporting learners. There were also some slightly less conventional but widely held perspectives on assessment, plus a bit of complexivist thinking about celebrating the many teachers and acknowledging the technological connectome as the means, the object and the subject of learning, but nothing Earth-shatteringly novel. I think this is as it should be. We don’t need new values and attitudes; we just need to emphasize those that are learning-positive rather than the increasingly mainstream learning-negative, outcomes-driven, externally regulated approaches that the cult of measurement imposes on us.

Post-secondary institutions have had to grapple with their learning-antagonistic role of summative assessment since not long after their inception so this is not a new problem but, until recent decades, the two roles have largely maintained an uneasy truce. A great deal of the impetus for the shift has come from expanding access to PSE. This has resulted in students who are less able, less willing, and less well-supported than their forebears who were, on average, far more advantaged in ability, motivation, and unencumbered time simply because fewer were able to get in. In the past, teachers hardly needed to teach. The students were already very capable, and had few other demands on their time (like working to get through college), so they just needed to hang out with smart people, some of whom who knew the subject and could guide them through it in order to know what to learn and whether they had been successful, along with the time and resources to support their learning. Teachers could be confident that, as long as students had the resources (libraries, lecture notes, study time, other students) they would be sufficiently driven by the need to pass the assessments and/or intrinsic interest, that they could largely be left to their own devices (OK, a slight caricature, but not far off the reality).

Unfortunately, though this is no longer even close to the norm,  it is still the model on which most universities are based.  Most of the time professors are still hired because of their research skills, not teaching ability, and it is relatively rare that they are expected to receive more than the most perfunctory training, let alone education, in how to teach. Those with an interest usually have opportunities to develop their skills but, if they do not, there are few consequences. Thanks to the technological connectome, the rewards and punishments of credentials continue to do the job well enough, notwithstanding the vast amounts of cheating, satisficing, student suffering, and lost love of learning that ensues. There are still plenty of teachers: students have textbooks, YouTube tutorials, other students, help sites, and ChatGPT, to name but a few, of which there are more every day. This is probably all that is propping up a fundamentally dysfunctional system. Increasingly, the primary value of post-secondary education comes to lie in its credentialling function.

No one who wants to teach wants this, but virtually all of those who teach in universities are the ones who succeeded in retaining their love of learning for its own sake despite it, so they find it hard to understand students who don’t. Too many (though, I believe, a minority) are positively hostile to their students as a result, believing that most students are lazy, willing to cheat, or to otherwise game the system, and they set up elaborate means of control and gotchas to trap them.  The majority who want the best for their students, however,  are also to blame, seeing their purpose as to improve grades, using “learning science” (which is like using colour theory to paint – useful, not essential) to develop methods that will, on average, do so more effectively. In fairness, though grades are not the purpose, they are not wrong about the need to teach the measurable stuff well: it does matter to achieve the skills and knowledge that students set out to achieve. However, it is only part of the purpose. Mostly, education is a means to less measurable ends; of forming identities, attitudes, values, ways of relating to others, ways of thinking, and ways of being. You don’t need the best teaching methods to achieve that: you just need to care, and to create environments and structures that support stuff like community, diversity, connection, sharing, openness, collaboration, play, and passion.

The keynote was recorded but I am not sure if or when it will be available. If it is released on a public site, I will share it here.

Some meandering thoughts on ‘good’ and ‘bad’ learning

Posted , updated on

There has been an interesting brief discussion on Twitter recently that has hinged around whether and how people are ‘good’ at learning. As Kelly Matthews observes, though, Twitter is not the right place to go into any depth on this, so here is a (still quite brief) summary of my perspective on it, with a view to continuing the conversation.

Humans are nearly all pretty good at learning because that’s pretty much the defining characteristic of our species. We are driven by an insatiable drive to learn at from the moment of our birth (at least). Also, though I’m keeping an open mind about octopuses and crows, we seem to be better at it than at least most other animals. Our big advantage is that we have technologies, from language to the Internet, to share and extend our learning, so we can learn more, individually and collectively, than any other species. It is difficult or impossible to fully separate individual learning from collective learning because our cognition extends into and is intimately a part of the cognition of others, living and dead.

However, though we learn nearly all that we know, directly or indirectly, from and with other people, what we learn may not be helpful, may not be as effectively learned as it should, and may not much resemble what those whose job is to teach us intend. What we learn in schools and universities might include a dislike of a subject, how to conceal our chat from our teacher, how to meet the teacher’s goals without actually learning anything, how to cheat, and so on. Equally, we may learn falsehoods, half-truths, and unproductive ways of doing stuff from the vast collective teacher that surrounds us as well as from those designated as teachers.

For instance, among the many unintended lessons that schools and colleges too often teach is the worst one of all: that (despite our obvious innate love of it) learning is an unpleasant activity, so extrinsic motivation is needed for it to occur. This results from the inherent problem that, in traditional education, everyone is supposed to learn the same stuff in the same place at the same time. Students must therefore:

  1. submit to the authority of the teacher and the institutional rules, and
  2. be made to engage in some activities that are insufficiently challenging, and some that are too challenging.

This undermines two of the three essential requirements for intrinsic motivation, support for autonomy and competence (Ryan & Deci, 2017).  Pedagogical methods are solutions to problems, and the amotivation inherently caused by the system of teaching is (arguably) the biggest problem that they must solve. Thus, what passes as good teaching is largely to do with solving the problems caused by the system of teaching itself. Good teachers enthuse, are responsive, and use approaches such as active learning, problem or inquiry-based learning, ungrading, etc, largely to restore agency and flexibility in a dominative and inflexible system. Unfortunately, such methods rely on the technique and passion of talented, motivated teachers with enough time and attention to spend on supporting their students. Less good and/or time-poor teachers may not achieve great results this way. In fact, as we measure such things, on average, such pedagogies are less effective than harder, dominative approaches like direct instruction (Hattie, 2013) because, by definition, most teachers are average or below average. So, instead of helping students to find their own motivation, many teachers and/or their institutions typically apply extrinsic motivation, such as grades, mandatory attendance, classroom rules, etc to do the job of motivating their students for them. These do work, in the sense of achieving compliance and, on the whole, they do lead to students getting a normal bell-curve of grades that is somewhat better than those using more liberative approaches. However, the cost is huge. The biggest cost is that extrinsic motivation reliably undermines intrinsic motivation and, often, kills it for good (Kohn, 1999). Students are thus taught to dislike or, at best, feel indifferent to learning, and so they learn to be satisficing, ineffective learners, doing what they might otherwise do for the love of it for the credentials and, too often, forgetting what they learned the moment that goal is achieved. But that’s not the only problem.

When we learn from others – not just those labelled as teachers but the vast teaching gestalt of all the people around us and before us who create(d) stuff, communicate(d), share(d), and contribute(d) to what and how we learn – we typically learn, as Paul (2020) puts it, not just the grist (the stuff we remember) but the mill (the ways of thinking, being, and learning that underpin them). When the mill is inherently harmful to motivation, it will not serve us well in our future learning.

Furthermore, in good ways and bad, this is a ratchet at every scale. The more we learn, individually and collectively, the more new stuff we are able to learn. New learning creates new adjacent possible empty niches (Kauffman, 2019) for us to learn more, and to apply that learning to learn still more, to connect stuff (including other stuff we have learned) in new and often unique ways. This is, in principle, very good. However, if what and how we learn is unhelpful, incorrect, inefficient, or counter-productive, the ratchet takes us further away from stuff we have bypassed along the way. The adjacent possibles that might have been available with better guidance remain out of our reach and, sometimes, even harder to get to than if the ratchet hadn’t lifted us high enough in the first place. Not knowing enough is a problem but, if there are gaps, then they can be filled. If we have taken a wrong turn, then we often have to unlearn some or all of what we have learned before we can start filling those gaps. It’s difficult to unlearn a way of learning. Indeed, it is difficult to unlearn anything we have learned. Often, it is more difficult than learning it in the first place.

That said, it’s complex, and entangled. For instance, if you are learning the violin then there are essentially two main ways to angle the wrist of the hand that fingers the notes, and the easiest, most natural way (for beginners) is to bend your hand backwards from the wrist, especially if you don’t hold the violin with your chin, because it supports the neck more easily and, in first position, your fingers quickly learn to hit the right bit of the fingerboard, relative to your hand. Unfortunately, this is a very bad idea if you want a good vibrato, precision, delicacy, or the ability to move further up the fingerboard: the easiest way to do that kind of thing is to to keep your wrist straight or slightly angled in from the wrist, and to support the violin with your chin. It’s more difficult at first, but it takes you further. Once the ‘wrong’ way has been learned, it is usually much more difficult to unlearn than if you were starting from scratch the ‘right’ way. Habits harden. Complexity emerges, though, because many folk violin styles make a positive virtue of holding the violin the ‘wrong’ way, and it contributes materially to the rollicking rhythmic styles that tend to characterize folk fiddle playing around the world. In other words, ‘bad’ learning can lead to good – even sublime – results. There is similarly plenty of space for idiosyncratic technique in many of the most significant things we do, from writing to playing hockey to programming a computer and, of course, to learning itself. The differences in how we do such things are where creativity, originality, and personal style emerge, and you don’t necessarily need objectively great technique (hard technique) to do something amazing. It ain’t what you do, it’s the way that you do it, that’s what gets results. To be fair, it might be a different matter if you were a doctor who had learned the wrong names for the bones of the body or an accountant who didn’t know how to add up numbers. Some hard skills have to be done right: they are foundations for softer skills. This is true of just about every skill, to a greater or lesser extent, from writing letters and spelling to building a nuclear reactor and, indeed, to teaching.

There’s much more to be said on this subject and my forthcoming book includes a lot more about it! I hope this is enough to start a conversation or two, though.

References

Hattie, J. (2013). Visible Learning: A Synthesis of Over 800 Meta-Analyses Relating to Achievement. Taylor & Francis.

Kauffman, S. A. (2019). A World Beyond Physics: The Emergence and Evolution of Life. Oxford University Press.

Kohn, A. (1999). Punished by rewards: The trouble with gold stars, incentive plans, A’s, praise, and other bribes (Kindle). Mariner Books.

Paul, A. M. (2021). The Extended Mind: The Power of Thinking Outside the Brain. HarperCollins.

Ryan, R. M., & Deci, E. L. (2017). Self-determination theory: Basic psychological needs in motivation, development, and wellness. Guilford Publications.

 

Brunel University’s Integrated Programme Assessment – a neat way to decouple learning and credentials

Posted , updated on

I have frequently written about the need to decouple learning and credentials, so I love this approach to doing so from Brunel University. It fully decouples learning and credentials by offering ungraded study blocks (in North America the equivalent of courses, in the UK the equivalent of modules) with no summative assessments, followed by integrative assessment blocks, that provide opportunities for students to pull together what they have learned across their various courses/modules in a variety of (mostly) useful integrative learning activities for which marks are awarded. It’s neat, simple, practical, and effective.

The summative assessment load (for students and their professors) is reduced by more than 60%, the quality of those assessments increases (in every way), students feel better prepared for employment (and employers agree), it improves retention figures, teachers can focus on teaching, assessments are more authentic, more engaging, and it massively reduces cheating. The only significant downside that I can see in this is that it is not quite as flexible as a completely modular program – there are a few dependencies and limits on when and how students learn, albeit that these are no worse than in most in-person universities.

I learned about this from Peter Hartley, who mentioned it in a quite inspiring IFNTF talk on assessment yesterday. Amongst other things, Peter highlighted a wide range of issues with modularization (i.e. the standard approach used in many parts of the world of splitting up a program into a set of self-contained courses) and assessment, including, from his slides:

  1. Not assessing programme outcomes.
  2. Atomisation of assessment.
  3. Students and staff failing to see the links/coherence of the programme.
  4. Modules too short for complex learning.
  5. Surface learning and ‘tick-box’ mentality.
  6. Inappropriate ‘one-size-fits-all’.
  7. Over-standardisation in regulations.
  8. Too much summative assessment and feedback – not enough formative.

While I couldn’t agree more, for the most part, I have mixed feelings about some of Peter’s list of issues. I agree that the traditional 3 or 4 year program(me), in which the course of study is designed to work as a whole, not as a collection of self-contained pieces, is far better for integrating knowledge across a discipline, though I don’t see why it should always take exactly that amount of time to achieve mastery, and I am not even sure whether we should be thinking in terms of disciplines at all. There’s some value in the notion, for sure, and there are some kinds of subject and learning for which it makes sense, but I think a lot of it is down to centuries’ old tradition and post hoc justification rather than careful consideration of fitness for purpose. Also, it seems to me that summative assessment should always be formative, too, so the issue could be partly addressed by simply improving summative assessments, not by scrapping them altogether. However, I think Peter is fundamentally right that, due to modularization, most universities over-assess, that credentials become the reason for learning rather than the measurement of it (with all the very many evils that entails), that the big picture tends to be lost, that there is a ridiculously large administrative burden that results from it, and that learning – the point of the thing after all – consequently suffers. As we and much of the rest of the world start to move towards ever smaller chunks, with associated stackable microcredentials, badges, etc, this is going to be a bigger problem. Brunel’s solution is not the only way, but it is a radically disruptive intervention that that many universities could implement without breaking everything else in the process.

Originally posted at: https://landing.athabascau.ca/bookmarks/view/16012554/brunel-universitys-integrated-programme-assessment-a-neat-way-to-decouple-learning-and-credentials