I’ve been writing about genAI and education for a few years now, and my thoughts have evolved somewhat. This post is a quick summary of roughly what I think about the subject now. It is not my last word, some is mundane (but not necessarily obvious) and I won’t go into too much detail, but I think some of this is useful and worth sharing.
Not human, but human
It is essential both to avoid speaking of LLMs as though they were human and to avoid speaking of them as though they were not. The knowledge of an LLM is neither explicit nor tacit because both terms assume a meaning-maker for whom it matters. What an LLM generates simply is; it is at most a latent meaning waiting for us to grasp it. It is not human. And yet, simultaneously, it is almost nothing but human, with a bubble-thin veneer of structure and process overlaid on cleverly organized petabytes of our own meaning-making. It remixes the collective us so, if we can express tacit knowledge symbolically (and I am sure that we can) then so can the LLM. But it’s not the expression of another intelligence’s knowledge. It’s an expression of our own collective intelligence, aggregated, lossily compressed, interconnected, and amplified by the tools at its disposal. And, being a creative tool-user, it can do things we can do – almost all of us.
Those who claim these things are just unintelligent stochastic parrots completely miss the point. So do those who think they are a step towards AGI. The reason that they appear to be so human-like is that they are trained by smart people on smart data, ingeniously encoded and organized by actual humans: words, pictures, movies, music, code, instructions, whatever. A significant amount of our intelligence is in the language or symbol system itself, not the wielder of the technology. In essence, an LLM or diffusion model is a search engine that, rather than returning results verbatim, constructs a kind of average (“average” is the wrong word, but easier to understand and write than the reality) amalgam of the best results. I think this is super interesting because of what it suggests about the nature of our own cognition and its dependence on language and, more generally, on technologies in general. Our cognition is very literally extended in our tools and systems, both intracranial and extracranial, and they are not just things we use but active organizers of knowledge and thought in their own right. We are, amongst other things, prediction machines, and (very much like genAIs) that includes predictions of possible next words. Subject and object distinctions blur at this point. We don’t just think thoughts: our thoughts think us. We shape our sentences then (perhaps simultaneously) they shape us. The same is true of everything from door handles and city streets to regulations and pedagogical methods.
GenAIs are not (normally) tools – they are users of tools.
GenAIs are extremely soft and flexible technologies and there is very little that they cannot do or be. Yes, they can be tools, if you tell them to behave that way. However, for the most part, they are not. A tool is something an intelligent agent does something with in order to do something to something else. Intrinsic to that is the agent’s own orchestration: it is the using (organizing of stuff), not the use, that makes it a tool. A tool extends your capabilities, it doesn’t replace them. GenAIs do replace our capabilities. To describe a genAI as a tool is both to belittle its role and to greatly overestimate our own agency. Yes, we can command and control what it does to an extent, and our prompts are the tools we use to do so, but we do so in a way that is vaguely similar to our control over a dog (or, when the things are not going so well, a cat). Even that is not a good simile, insofar as our prompts are a literal part of the “thought” of the genAI, not just instigators of it. GenAIs are not like human partners, either, for reasons I will explain below. I’m OK with treating them as cognitive Santa Claus machines – a kind of thinking appliance – but that seems a bit impoverished. Like an appliance, they orchestrate phenomena without human intervention but, like a tool, they participate in potentially limitless orchestrations . I suggest “metatool” because they are the first true tool-using technologies we have ever created. They use (in the sense that they orchestrate) tools, but the uses to which those tools are put tend to be ours. I’m not quite as sure about that one as I was: they can and do construct sub-goals over which we may have no input at all, and it is almost impossible not to use teleological language to describe that. Recent widely publicized (albeit exaggerated by marketing hype) reports of agents running amok and hacking other systems speak to this.
Gen AIs are not partners, assistants, slaves, or drunken RAs. They are something new.
It is an alluring category mistake to treat the things as sentient in any way. Intelligent, perhaps (whatever that means), but with no persistent identity, no consciousness, no skin in the game, no moral worth and no moral understanding. GenAIs (thus far) can create speech, but not speech acts. You cannot (yet) form anything remotely like a human relationship with a genAI because there is nothing there to have a relationship with. It is certainly possible to create the illusion of a relationship with a persistent person but it is the work of seconds – one small prompt – to change that into someone else entirely. Current generations lack significant persistence of memory, though that is rapidly changing and, before long, their memories of past interactions will be hard to distinguish from the real thing. However, though good at pretending, these are not beings in and for themselves. The simple presence of an on/off switch or the ability to command them to forget makes them profoundly different. There may be a sense in which we can claim that they have some level of self-awareness but it is patterns of tokens – our tokens – that make them so. This becomes jarringly obvious when, for instance, when describing how to change a tyre they say things like “I sometimes find it useful to use the back of a spoon to hold the rim”. This may change as we increasingly encounter embodied AIs with world models rather than the current LLMs.
Please, no more cognitive offloading: it’s cognitive redistribution
Using a chatbot to write, draw, or create a video, or letting an agent loose on your machine, is not cognitive offloading, except insofar as our cognition is innately distributed: it’s just cognitive redistribution. It does support coarser-grained thinking, though, which means we can think bigger, in some ways better things without being caught up in the minutiae of detail. It also means there is a great temptation to think less, especially when tasks are not particularly appealing or creative. Thinking appliances do what it says on the tin so, if being able to do what they are doing matters, then we should probably avoid letting them do all of it. But, much of the time, I find that genAIs actually increase my cognitive load. Much of this is down to how I like to configure my chatbots. I have different seeding prompts for all of them but a common theme is to encourage them to challenge me whenever they perform a task for me. Gemini always closes with a pertinent question. ChatGPT avoids sycophancy and focuses on a clear, reliable response without dumbing it down: often, I have to ask further questions to explain its terse output. Claude is configured to get to know me and make associations with my previous work and interests: it has read much of my work and has been told to remember it. It is the one that I am most likely to have a conversation with, bouncing ideas back and forth with a strangely distorting cognitive mirror of myself. I have created the tutors I need for a variety of different learning tasks and they frequently spark connections and ideas, force me to rethink my assumptions, and challenge me to do things I would not otherwise do and learn what I would not otherwise learn. There are things I haven’t learned that I might have learned and things that have atrophied, for sure: the load is redistributed. I have written very few lines of code myself over the past few years, for instance. However, I have learned a lot about different ways of programming: I constantly come across ways of doing stuff that I would never have thought of on my own. Frequently, if I need a utility, it is quicker to get an AI to write it for me than to search for a pre-existing solution myself. In the process I gain a far greater understanding of what the utility does, how it does it, and what opportunities it creates than if I had just downloaded an equivalent. Simply expressing what I want is an act of learning.
Unless we change not just how we teach and assess but what we teach and assess we will not only get a lot of cheating but we won’t be supporting the learning that is needed in order to think bigger. The trick to that is, I think, to get a better grasp of our own roles in the technology, and to see genAIs not just as thinking appliances but as part of a creative, open-ended process of development.
GenAIs can produce almost anything. Our job is to figure out how to use that (not as a purpose but as a technology)
It’s not just text and images. I have a toolbox, and a 3D printer, and I can use a soldering iron. I tell genAIs what I want to create, genAIs tell me how to do so and, as I build the creations, I learn how to build them, how they work, and what else is possible. GenAIs often get me to a point where new adjacent possibles come into view. Having a genAI on hand is very much like having the right expert friend available when you are not sure how to do something. Sure, like all of my expert friends, they might not always get it right and there might be better ways of doing what they suggest, but they can get me over most hurdles quickly enough and reliably enough to move on to the next step. With Claude’s help, I built a web conferencing application the other day, for instance. The basic app took no more than 10 minutes, and it worked just fine, as long as both parties could figure out how to make a connection. I needed a means for them to do that, though, and that involved quite a lot of installation and configuration on my server that led to a whole raft of cascading problems, any one of which would likely have taken me hours to figure out.
Please, no more “humans in the loop”
I think most of us can probably agree that people are ends in themselves and that our purposes are human purposes. Most of us can probably get behind the idea that education is a human process, by humans, for humans, to enable humans to live in a human society. One of my greatest fears about genAI is that we are increasingly learning ways of being human from things that are not human. The more we interact with chatbots, in particular, the less we interact with one another, and there are strong incentives for learners to do so, whether formally required to or not. From a big systems perspective, this is bad. However, the answer is not to keep humans in a loop, save in the trivial sense that, in an educational context, there’s little value in having one genAI teach another genAI or for a genAI to perform an assignment that is graded by another genAI. The answer is to think of genAIs as being part of the web of people, tools, processes, structures, theories, models, appliances, devices, and other artifacts that extends our cognition. It’s not a loop, it’s a sprawling, fuzzy, amorphous, ever-shifting network. There is a sense in which “humans in the loop” matters, particularly in agentic systems. It is definitely a very dangerous idea to set tasks for genAIs that provide them with tools and access permissions, and the means to make independent decisions about what to do with them.
You can’t patch over the credentialing problem
Traditional assignments and exams (soon, if not already) have lost their value as proxies for learning. The fact that the vast majority of students try to avoid cheating is encouraging, though, and suggests the obvious way forward, which is to remove the incentives for cheating. Ungrading is a non-negotiable necessity in most cases, except where grades are authentic measures of success, as (say) points are in many but not all computer games. Decoupling credentials from learning is critical: as long as credentials are perceived as being the ultimate purpose, cheating will be endemic, both in its hard, prosecutable forms and in its softer, satisficing, compliant forms. My favourite way of doing this is to make use of integrative assessments that demand diverse skills acquired through multiple means, be they courses or whatever.
Please, no more crude measures of environmental impact
Large data centres suck, and ones fuelled by gas, oil, or coal suck big time. No question. Generative AI as we do it now costs a fortune in energy and, at scale, is incredibly harmful to the planet, and often disruptive to communities. So, too, are ambulances, buses, and MRI machines. I’m not suggesting that genAI is on a par with public services of this nature – I use these examples to highlight the complexity, not to draw direct comparisons – but I think it is important to consider what we gain from its use, and what does not happen as a result, before tarring it as unequivocally environmentally evil. For example, prior to genAI, creating a slide deck for me involved multiple searches through large databases of public domain images from which I would select (rarely ideal) illustrations of the concepts I was presenting. Now I get something much closer to what I actually need, usually faster. I’d hesitate to suggest that I use less energy. The computation involved is significantly greater, though neither is exactly frugal. Sometimes it does still take a long time and burns through a lot of energy doing what I don’t want but my own skills in getting what I need continue to improve, and genAIs using diffusion models have improved a great deal. Meanwhile, spurred by greed and the backlash, data centres are becoming more efficient and are increasingly likely to use renewables. There is also the potentially large increase in productivity (with lots of provisos, some of which I have already mentioned), and the ability to do things we could not do before. We are a long way from being able to describe them as ecofriendly, but the tradeoffs are getting more complicated.
