Presentation – Generative AIs in Learning & Teaching: the Case Against

Posted

Here are the slides from my presentation at AU’s Lunch ‘n’ Learn session today. The presentation itself took 20 minutes and was followed by a wonderfully lively and thoughtful conversation for another 40 minutes, though it was only scheduled for half an hour. Thanks to all who attended for a very enjoyable discussion! self portrait of chatGPT, showing an androgynous human face overlaid with circuits

The arguments made in this were mostly derived from my recent paper on the subject (Dron, J. (2023). The Human Nature of Generative AIs and the Technological Nature of Humanity: Implications for Education. Digital, 3(4), 319–335. https://doi.org/10.3390/digital3040020) but, despite the title, my point was not to reject the use of generative AIs at all. The central message I was hoping to get across was a simpler and more important one: to encourage attendees to think about what education is for, and what we would like it to be. As the slides suggest, I believe that is only partially to do with the objectives and outcomes we set out to achieve,  that it is nothing much at all to do with the products of the system such as grades and credentials, and that focus on those mechanical aspects of the system often creates obstacles to the achievement of it. Beyond those easily measured things, education is about the values, beliefs, attitudes, relationships, and development of humans and their societies.  It’s about ways of being, not just capacity to do stuff. It’s about developing humans, not (just) developing skills. My hope is that the disruptions caused by generative AIs are encouraging us to think like the Amish, and to place greater value on the things we cannot measure. These are good conversations to have.

Published in Digital – The Human Nature of Generative AIs and the Technological Nature of Humanity: Implications for Education

Posted , updated on

A month or two ago I shared a “warts-and-all” preprint of this paper on the risks of educational uses of generative AIs. The revised, open-access published version, The Human Nature of Generative AIs and the Technological Nature of Humanity: Implications for Education is now available in the Journal Digital.

The process has been a little fraught. Two reviewers really liked the paper and suggested minimal but worthwhile changes. One quite liked it but had a few reasonable suggestions for improvements that mostly helped to make the paper better. The fourth, though, was bothersome in many ways, and clearly wanted me to write a completely different paper altogether. Despite this, I did most of what they asked, even though some of the changes, in my opinion, made the paper a bit worse. However, I drew the line at the point that they demanded (without giving any reason) that I should refer to 8 very mediocre, forgettable, cookie cutter computer science papers which, on closer inspection, had all clearly been written by the reviewer or their team. The big problem I had with this was not so much the poor quality of the papers, nor even the blatant nepotism/self-promotion of the demand, but the fact that none were in any conceivable way relevant to mine, apart from being about AI: they were about algorithm-tweaking, mostly in the context of traffic movements in cities.  It was as ridiculous as a reviewer of a work on Elizabethan literature requiring the author to refer to papers on slightly more efficient manufacturing processes for staples. Though it is normal and acceptable for reviewers to suggest reference to their own papers when it would clearly lead to improvements, this was an utterly shameless abuse of power of a scale and kind that I have never seen before. I politely refused, making it clear that I was on to their game but not directly calling them out on it.

In retrospect, I slightly regret not calling them out. For a grizzly old researcher like me who could probably find another publisher without too much hassle, it doesn’t matter much if I upset a reviewer enough to make them reject my paper. However, for early-career researchers stuck in the publish-or-perish cycle, it would be very much harder to say no. This kind of behaviour is harmful for the author, the publisher, the reader, and the collective intelligence of the human race. The fact that the reviewer was so desperate to get a few more citations for their own team with so little regard for quality or relevance seems to me to be a poor reflection on them and their institution but, more so, a damning indictment of a broken system of academic publishing, and of the reward systems driving academic promotion and recognition. I do blame the reviewer, but I understand the pressures they might have been under to do such a blatantly immoral thing.

As it happens, my paper has more than a thing or two to say about this kind of McNamara phenomenon, whereby the means used to measure success in a system become and warp its purpose, because it is among the main reasons that generative AIs pose such a threat. It is easy to forget that the ways we establish goals and measure success in educational systems are no more than signals of a much more complex phenomenon with far more expansive goals that are concerned with helping humans to be, individually and in their cultures and societies, as much as with helping them to do particular things. Generative AIs are great at both generating and displaying those signals – better than most humans in many cases – but that’s all they do: the signals signify nothing. For well-defined tasks with well-defined goals they provide a lot of opportunities for cost-saving, quality improvement, and efficiency and, in many occupations, that can be really useful. If you want to quickly generate some high quality advertising copy, the intent of which is to sell a product, then it makes good sense to use a generative AI. Not so much in education, though, where it is too easy to forget that learning objectives, learning outcomes, grades, credentials, and so on are not the purposes of learning but just means for and signals of achieving them.

Though there are other big reasons to be very concerned about using generative AIs in education, some of which I explore in the paper, this particular problem is not so much with the AIs themselves as with the technological systems into which they are, piecemeal, inserted. It’s a problem with thinking locally, not globally; of focusing on one part of the technology assembly without acknowledging its role in the whole. Generative AIs could, right now and with little assistance,  perform almost every measurable task in an educational system from (for students) producing essays and exam answers, to (for teachers) writing activities and assignments, or acting as personal tutors. They could do so better than most people. If that is all that matters to us then we might as well therefore remove the teachers and the students from the system because, quite frankly, they only get in the way. This absurd outcome is more or less exactly the end game that will occur though, if we don’t rethink (or double down on existing rethinking of) how education should work and what it is for, beyond the signals that we usually use to evaluate success or intent. Just thinking of ways to use generative AIs to improve our teaching is well-meaning, but it risks destroying the woods by focusing on the trees. We really need to step back a bit and think of why we bother in the first place.

For more on this, and for my tentative partial solutions to these and other related problems, do read the paper!

Abstract and citation

This paper analyzes the ways that the widespread use of generative AIs (GAIs) in education and, more broadly, in contributing to and reflecting the collective intelligence of our species, can and will change us. Methodologically, the paper applies a theoretical model and grounded argument to present a case that GAIs are different in kind from all previous technologies. The model extends Brian Arthur’s insights into the nature of technologies as the orchestration of phenomena to our use by explaining the nature of humans’ participation in their enactment, whether as part of the orchestration (hard technique, where our roles must be performed correctly) or as orchestrators of phenomena (soft technique, performed creatively or idiosyncratically). Education may be seen as a technological process for developing these soft and hard techniques in humans to participate in the technologies, and thus the collective intelligence, of our cultures. Unlike all earlier technologies, by embodying that collective intelligence themselves, GAIs can closely emulate and implement not only the hard technique but also the soft that, until now, was humanity’s sole domain; the very things that technologies enabled us to do can now be done by the technologies themselves. Because they replace things that learners have to do in order to learn and that teachers must do in order to teach, the consequences for what, how, and even whether learning occurs are profound. The paper explores some of these consequences and concludes with theoretically informed approaches that may help us to avert some dangers while benefiting from the strengths of generative AIs. Its distinctive contributions include a novel means of understanding the distinctive differences between GAIs and all other technologies, a characterization of the nature of generative AIs as collectives (forms of collective intelligence), reasons to avoid the use of GAIs to replace teachers, and a theoretically grounded framework to guide adoption of generative AIs in education.

Dron, J. (2023). The Human Nature of Generative AIs and the Technological Nature of Humanity: Implications for Education. Digital, 3(4), 319–335. https://doi.org/10.3390/digital3040020

Originally posted at: https://landing.athabascau.ca/bookmarks/view/21104429/published-in-digital-the-human-nature-of-generative-ais-and-the-technological-nature-of-humanity-implications-for-education

We shape our buildings and, afterwards, our buildings shape us: some lessons in how not to build an online university, and some ideas for doing it better

Posted , updated on

My heart briefly leapt to my throat when I saw Thursday’s Globe & Mail headline that the Albertan government had (allegedly) dropped its insane plan to force Athabasca University to move 65% of its workforce to the town of Athabasca. It seemed that way, given that the minister for post secondary education was referring to his demands and accompanying threat as only a ‘suggestion’ (broadly along the lines of Putin’s ‘suggestion’ that Ukraine should be part of Russia, perhaps). However, other reports, have said that he has denied any change in his requirements, albeit that he now claims it is open to negotiation. A ham-fisted negotiation tactic or just plain confused? I hope so, but I doubt it. I think that this is just a ploy to push the real agenda through with little resistance, and largely unnoticed. In the Globe & Mail article, the minister goes on to say “I would indeed like to see, at a bare minimum, senior executives and administrative staff be based in the town, as they have been for the past several decades.” A majority of what might be described as administrative staff do probably live in Athabasca anyway, and there is no reason for any of them to leave, so that’s just gaining a few easy election points from town voters. If the government actually wanted to help the town it would invest in the infrastructure and support needed to allow it to thrive, which it has signally failed to do for several decades, at least. No, his main target is clearly the senior executives: basically, he and the UCP want to put a team of executive lackeys in charge so that they can push their agenda through unopposed by anyone they care about. They have already sacked the incumbent and installed a chair of the board of governors who will do their bidding, and they have increased representation on the board from the town of Athabasca so this is the obvious next step. The execs won’t have to be fired. If they are required to move to Athabasca, most of what is probably the best executive team ever assembled in this or any other Albertan university will resign. Whoever replaces them will do the UCP’s dirty work, largely free from media oversight. Job done, bad press averted.

The UCP will, I am very sad to say, appear to have support from our own professional and faculty union (AUFA), even though most of us will, whether weakly or strongly, oppose it. This is because AUFA has a small but disproportionately powerful caucus in Athabasca, members of which have been deeply involved with an activist group called KAAU (Keep Athabasca in Athabasca University), who actually paid an insider lobbyist to start this fracas in the first place. Seriously. A casual observer might perceive at least a portion of the union’s leadership as putting the interests of the town ahead of the interests of the university. At best, their loyalties appear to be divided. The evidence for this is all too apparent in press statements and blog posts on the subject. Though most of us (including me) support the continuing presence of AU in Athabasca, these posts do not represent the views of most of those in the union, only those in charge of it. Only around 20% or thereabouts of AUFA members actually live in Athabasca, a percentage that has steadily fallen over the course of the last two decades, and almost all of those are professional members, not academics. Most members who had the chance to leave over the past 20 years did so. This is a point worth dwelling on.

We shape our buildings…

Athabasca High Street
Athabasca High Street at peak season

Athabasca is a tiny, inclement (-40 in Winter, bugs in summer) Northern town over 180km away from the nearest International airport. There is one (private) bus from Edmonton leaving late at night that arrives in town at 2:46am after a 3+ hour journey on a small, treacherous road. When it got too big for its Edmonton home, the university was (disastrously) moved there by a conservative government in 1984, ostensively to fill a gap left by the closure of the town’s main employer, but more likely due to the property interests held there by those behind the plan. About half the faculty resigned rather than work there. Ironically, the first president of AU deliberately named the university after a geographical feature of Alberta (the Athabasca River) precisely to avoid associating it with any city or region, so that local politics wouldn’t interfere with its mission. We might have been named after a mountain were it not that the University of Alberta happened to be demolishing Athabasca Hall (a students’ residence) at the time, so the name was free for us to use. It had nothing whatsoever to do with the town. It is possible that the president who named it was even unaware of the town’s existence or, at least, considered it to be too insignificant to be an issue.

Whatever charms the town may have (and it has a few), Athabasca has been a hobble for AU from the very start. I wrote about this at some length 5 years ago, just as we were on the cusp of making the massive changes we have been implementing ever since, but I would like to focus on two particularly relevant aspects in this post: the effects on the hiring pool, and the short-circuiting of communication with the rest of the university.

Firstly, it is really difficult to attract good employees to the town. Some residents of Athabasca will say that they feel insulted by this, believing that it implies that they are not the best and brightest. This is either disingenuous or a confirmation that they are, in fact, not the best and brightest, because all it means is that we have fewer good people to choose from. There are, of course, some incredibly smart, talented, creative people who live in Athabasca. But, equally, some are not: we have too often had to pick the best of a not-too-great bunch. The more people we expect to live in Athabasca, the bigger the problem of those who are not the best and brightest becomes. The undesirability of the place is confirmed by the KAAU itself, whose biggest complaint – the one that (at least on the face of it) drove their lobbying and union discontent in the first place – is that people have been leaving the town in droves since they were no longer required to stay, which pretty much says all that needs to be said. It is also notable that faculty and tutors are not and have never successfully been required to work in the town in all the university’s history, because it would be impossible to recruit sufficient numbers of sufficient quality, a fact that all parties involved in this (including the minister) acknowledge. We should get the best possible staff for almost every role – we all play some role in our distributed teaching model – but it is true in spades, plus some, for our executive team who, more than anyone else, have to be the most excellent that we can get. Right now, we have the best executive team that has ever been assembled at AU, bar none, and that is only possible because – for the first time ever – none of them have had to live in Athabasca.

…and our buildings shape us

Athabasca has, overwhelmingly, been home for staff that support but that do not directly implement its mission. Historically, these staff (predominantly administrators) have had extremely privileged access to the the leaders of the university compared with the rest of us. Even if they didn’t bump into them socially or in the canteens and halls, they would talk to people that did. And they would be the ones attending meetings in person while the rest of us phoned in or, in latter years, struggled with webmeeting systems that never really worked properly for in-person attendees, despite absurdly expensive equipment designed to support it. Fixing this was never a particularly high priority because those with the power to do so were the ones attending in-person, and it was just fine for them. Inevitably, Athabasca residents had a much better idea of what was going on and who was doing what than anyone else. More problematically, they had far greater influence over it: they didn’t ask for this, but they certainly got it. It is no wonder that they are now peeved, because most of their power, influence, and control over everything has been massively diminished since most of the execs left town. Their perception – voiced on many occasions by the Athabasca-dominated union –  that too much has recently been happening without consultation and that there is not enough communication from our leaders is, objectively speaking, completely false: in fact, it is far better than it has ever been, for those of us (the majority of staff) living remotely. They just no longer have a direct line themselves. I think this is the root of most of the union troubles of the last few years, whether consciously or not, and of the current troubles with the Albertan government.

In-person communities short-circuit online communities. I’ve seen it in teaching contexts a thousand times over: it just takes one group to branch off in person to severely damage or destroy a previously successful online community. Without fail, online communication becomes instrumental and intermittent. Tacit knowledge, in particular, disappears (apart from for the in-person group). Researchers like me (and many others at AU, including our president, in some of his former roles) have spent a great deal of time trying to make native online tools, systems, and working/teaching approaches that reduce these effects, but with only limited success. Combining fully online and in-person communities invariably wrecks the online community. Only when it is fully online, or when the online community is just an extension of the in-person community, can it thrive. Without the best of research-driven online tools and processes (most of which are not implemented at AU), hybrids are a disaster, and they are not much improved with even the best we have to offer.

In the past, the problem was partially offset by the fact that we had a few smaller learning centres elsewhere, in St Albert, Edmonton and Calgary (and, formerly, Fort McMurray), that were visited by the execs with varying frequency. However, this created what were, in many ways, bigger problems. It was incredibly inefficient, environmentally damaging, and expensive, wasting a lot of time and energy for all concerned. More significantly, although it helped to keep the exec team to be a little more in touch with others around the university and it helped to fill gaps in online communication for those living near them, it actually exacerbated the problem for our online community, because it created yet more in-person enclaves and cliques that developed independently of one another, sharing very little with the rest. Our business school, for instance, lived an almost entirely separate life from the rest of the university, in its own campus in St Albert (a satellite city attached to Edmonton), running its own largely independent communications and IT infrastructure but frequently meeting in person. As a result, we never developed the kind of unified online culture needed to sustain us.

Even more importantly, few of those with the power to change it ever learned what remote working was like for our students, so we didn’t create that online culture or community for them, either. Because of the inequalities that ensued, those of us who did know what it was like were not able to adequately influence the rest (especially the executive team) to get something done about it, because we were crowded out by the clamour of local communities. It’s not that the problem was unrecognized: it’s just that immediate operational concerns of in-person employees always came first. This was – and remains – a huge mistake. Too few of our students feel they belong, too few barely if ever interact with another student, too few see anything of the university beyond the materials provided for the courses they take. We have some excellent teaching processes, but processes (even the best) are only a part of what makes for a rewarding education. Yes, we do have plentiful support of all kinds, teaching approaches that should (for some but not all faculties) provide opportunities to develop relationships with human tutors, and the occasional opportunity to engage more broadly (mainly through the Landing), but many students completely bypass all of that. The need for it is beyond obvious, as evidenced by large number of Discords, Facebook Groups, Subreddits, and so on that they set up themselves to support one another. However, these are just more isolated enclaves, more subcultures, more virtual islands, without a single unifying culture to knit them together.

Online communication at AU has, as a direct result of its physical campuses, always tended to be extremely instrumental and terse, if it happened at all. When I arrived 15 years ago, most of my colleagues hardly ever communicated online with colleagues outside of a formal, intentional context. Those of us who did were yet another little clique. Emails (which were and remain the most commonly used tech) were only sent if there were a purpose, and most of the tacit knowledge, that more than anything else makes a traditional institution work despite its typically dire organization, was absent. In its place the university developed a very rigid, unforgiving, impersonal set of procedures for pretty much everything, including our teaching. If there was no procedure then it didn’t happen. There were gigantic gaps. The teaching staff – especially tutors but also most of the faculty – were largely unable to share in a culture and the admin-focused tacit knowledge that resided largely in one remote location. This was the largest part of what drove Terry Anderson and I to create the Landing: it was precisely to support the tacit, the informal, the in-between, the ad-hoc, the cultural, the connective aspects of a university that were missing. We touted it as a space between the formal spaces, actively trying to cultivate and nurture a diverse set of reasons to be there, to make others visible. Treating it as a space was, though, a mistake. Though it did (and does) help a little, the Landing was just another place to visit: it therefore has not (or has not yet) fulfilled our vision for it to seep into the cracks and to make humans visible in all of our systems. And we were not able to support the vital soft, human processes that had to accompany the software because we were just academics and researchers, not bosses: technologies are the tools, structures, and systems and what we do with them, but what we do with them is what matters most.  We need much more, and much better, and we need to embed it everywhere, in order to get rid of the short circuits of in-person cliques and online islands. A further death-knell to our online community was instigated by the (Athabasca-dominated) union that one day chose – without consultation – to kill off the only significant way for AUFA members to communicate more informally, its mailing list, only reluctantly bringing it back (after about 2 years of complaints), in a diluted, moderated, half-assed format that did not challenge their power. From an informal means of binding us, it became another instrumental tool.

Moving on

Despite the problems, it would be a senseless waste to pull out of Athabasca. We need a place for the library, for archives, for outreach into communities in the region, for labs, for astronomy, and to support research based in the region, of which there is already a growing amount. Virtually no one at the university thinks for a moment that we should leave the town. We are just doubling down on things to which it is best suited, rather than making it a centre of all our operations.  If people want to live there, they can. We can make a difference to an under-served region in our research, our outreach, and our facilities, and we are constantly doing more to make that happen, as a critical part of our reinvention of the university. It has symbolic value, too, as the only physical space that represents the university, albeit that few people ever see it.

However…

Athabasca should never become the seat of power, whether due to numbers of collocated workers or because it is where the exec team are forced to live. I am not singling the town out for special treatment in this: nowhere should play this role. We are and must be an online community, first and foremost. This is especially the case for our exec team. In fact, the more distributed they are the better. They will not walk the talk and fix what is broken unless they live with the consequences, and they are the last people who should be clustered together, especially with a particular employee demographic. This brings benefits to the university and to the communities to which we belong, including to Athabasca.

By far the greatest threat from the Albertan government’s intrusions and our own union’s efforts to restore their personal power is to the identity and culture – the very soul –  of the institution itself. Slowly (too slowly) and a bit intermittently we have, in recent years, been staggering towards creating a unified, online-native culture that embraces the whole institution. It has not been easy, especially thanks to the Athabascan resistance. But, regardless of their interference, we have made other mistakes. Our near-virtual implementation was the result of a large group representing the whole university, but one that lacked well-defined leadership or a clear mandate, that rushed development due to the pandemic, and that ignored most of what it found in its investigations of needs in its report to the university, leading to a hasty and incomplete implementation that has caused some unrest, most notably among those at Athabasca who are used to the comforts and conveniences of in-person working. For the majority of us who were already working online before the pandemic, things have got better, for the most part, but the benefits are very uneven. Too often we have poorly replicated in-person processes and methods to accommodate the newcomers, leading to (for instance) endless ineffectual meetings and yet more procedures. The near-virtual strategy remains a work in progress, and things will improve, but it got off to a stumbling, over-hasty start.

With limited funds, and contributing to the multiple failings of the near-virtual plan, we have signally failed to put enough effort into developing the technical infrastructure needed to support our nascent online community (one of the main needs identified by the near-virtual committee but not appearing in any meaningful way in the plan). I think we really should have focused on creating workable technologies to support our own community before working on teaching and administrative systems (or at least at the same time) but, after a decade of neglect while we were on the verge of bankruptcy, I guess we did need to fix those pretty urgently because they are what our students depend on. It’s just a bit tricky to pull yourself up by your own bootstraps if you are still using off-the-shelf tools designed to support in-person organizations (and commercial ones at that) rather than those designed for a virtual institution, especially when the more important human and organizational aspects are still rooted firmly in place-based thinking. I wrote about one aspect of that the other day. This won’t be a problem for long, I hope.  The fruits of the reinvention of our student-facing systems – that is taking up the bulk of our development resources right now – should start to appear around the end of this year, if the Albertan government or our own union doesn’t destroy it first. I hope that we can then get round to fixing our own house because, if we don’t, we will be easy prey for the next politician seeking easy votes and/or a sly buck from their investments.

Shaping our lives

The title of this post is a quote from Churchill. In fact, he liked it so much that he used variants on the phrase (sometimes preferring ‘dwellings’ to ‘buildings’) a number of times over a course of decades. I could equally have used Culkin’s (usually misattributed to Mcluhan) ‘we shape our tools and then our tools shape us’ because, as the first president of the university recognized many decades ago, we exist as a university within our communications network, not in a physical nor even a virtual space.

The recursive dynamic implied by Churchill’s and Culkin’s aphorisms applies to any complex adaptive system. In most systems – natural ecosystems, money markets, ant-trails, cities, and so on – this leads to metastability and adaptation, as agents adapt to their environments and, in the process, change those environments, in an endless emergent cycle of evolution. However, the large and slow moving elements of any complex system influence the small and fast moving far more than vice versa and humans are the only creatures that we know of who can deliberately mess with this dynamic by making radical and rapid changes to the large and slow moving parts of the spaces in which they dwell. In the past it has happened to Athabasca University due to the machinations of a small number of self-serving politicians and geographically located cliques, not due to educators. If we can prevent government interference and diminish the significance of those cliques then we can change that, and we have been doing so, rebuilding our systems to serve the needs of staff and students, not of a few land developers or groups of local residents.  This is not the time to stop. We are on the verge of creating a viable community and infrastructure for learning that could scale more or less indefinitely, where everyone – especially the students – can feel a part of something wonderful. Not cogs in machines, not products, but parts of an organic, evolving whole to which we all belong, and to which we all contribute. This matters: to our staff, to our students, to the people of Alberta, to the people of Canada, to the world. We should not be condemned to merely serve a small part of the economic needs of a small community, nor even of a province or country. If we follow that path then we will whimperingly shrink into a minor anachronistic irrelevance that appears as no more than a footnote in the annals of history, out-competed by countless others. Athabasca University matters most because it (not quite alone, but as part of a small, select pack of open and distance institutions) is beating a path that others can follow; an open, expansive, human-centred path towards a better future for us all. Let’s not let this die.

A few thoughts on learning management systems, and on integrated learning environments and their implementation

Posted , updated on

Why do we build digital learning systems to mimic classrooms?

It is understandable that, when we teach in person, we have to occupy and make different uses of the same or similar environments like classrooms, labs, workshops, lecture theatres, and offices. There are huge financial, physical, and organizational constraints on making the environment fit the task, so it would be madness to build a whole new classroom every time we wished to run a different class.

Online, we could build anything we like

But why do we do the same when we teach online? There are countless tools available and, if none are suitable, it is not too hard to build them or modify them to suit our needs. Once they are built, moving between them just takes a tap of a screen or the click of a mouse. Heck, you can even occupy several of them at once if you have a decent monitor or more than one device.

So why don’t we do this?

Here are a few of the more obvious reasons that using the perfect app for the context of study rarely happens:

  • Teachers’ lack of knowledge of the options (it takes time and effort to discover what’s available).
  • Teachers’ lack of skill in using them (most interesting tools have a learning curve, and that gets steeper in inverse proportion to the softness and diversity of the toolset, so most teachers don’t even know how to make the most of what they already have).
  • Lack of time and/or money for development (a real-life application is what it contains, not just the shell that contains it, and it is not always as easy to take existing stuff and put it in a new tool as it might be in a physical space).
  • Costs and difficulties in management (each tool adds costs in managing faults, configuration, accounting for use, performance, and security).
  • Cognitive load involved for learners in adapting to the metaphors, signposts, and methods needed to use the tool itself.

All of these are a direct consequence of the very diversity that would make us want to use different apps in the first place. This is a classic Faustian bargain in which the technology does what we want, and in the process creates new problems to solve.  Every virtual system invents at least some of the dynamics of how people and things interact with it and within it. In effect, every app has its own physics. That makes them harder to find out about, harder to learn, harder to develop, costlier to manage, and more difficult to navigate than the static, fixed facilities found in particular physical locations. They are all different, there are few if any universals, and any universal today may become a conditional tomorrow. Gravity doesn’t necessarily work the same way in virtual systems.

image of a pile of containersAnd so we get learning management systems

The learning management system (LMS) kind of deals with all of these problems: poorly, harmfully, boringly, and painfully, but it does deal with them. Currently, most of the teaching at Athabasca University is through the open source Moodle LMS, lightly modified by us because our needs are not quite like others (self-pacing and all that). But Moodle is not special: in terms of what it does and how it does it, it is not significantly different from any other mainstream LMS – Blackboard, Brightspace, Canvas, Sakai, whatever.

Almost every LMS essentially automates the functions, though not exactly the form, of traditional classrooms. In other parts of the world people prefer to use the term ‘managed learning environment’ (MLE) for such things, and it is the most dominant representative of a larger category of systems usually described as virtual learning environments (VLEs) that also includes things like MOOs (multi-user dungeons, object oriented), immersive learning environments, and simpler web-based teaching systems that replicate aspects of classrooms such as Google Classroom or Microsoft’s gnarly bundle of hastily repurposed rubbish for teaching that I’m not sure even has a name yet. Notice the spatial metaphors in many of these names.

Little boxes made of ticky tacky

The people who originally designed LMSs back in the 90s (I did so myself) based their designs on the functions and entities found in a traditional university because that was their context, and that was where they had to fit. Metaphorically, an LMS or MLE is a big university building with rather uniform classrooms, with perhaps a yard where you can camp out with a few other systems (plugins, LTI hooks, etc) that conform to its requirements and that are allowed in to classrooms when invited, and a few doors and gateways (mainly hyperlinks) linking it circuitously or in jury-rigged fashion to other similarly weakly connected buildings (e.g. places to register, places to seek support, places to talk to an advisor, places to complain, places to find books, and so on). It doesn’t have metaphorical corridors, halls, common rooms, canteens, yards, libraries or any of the other things that normally make up a physical university. You rarely get to even be aware of other classrooms beyond those you are in. Some people (me in a past life) might give classrooms cute names like ‘the learning cafe’ but it’s still just another classroom. You teleport from one classroom to the next because what happens in corridors (really a big lot of incredibly important pedagogically useful stuff, as it happens) is not perceived by the designers as a useful classroom function to be automated or perhaps, more charitably, they just couldn’t figure out how to automate that.

Reified roles

It’s a very controlled environment where everyone has a programmatically enforced role (mostly reflecting traditional educational roles), that may vary according to the room, but that are far less fluid than those in physical spaces. There are strong hierarchies, and limited opportunities for moving between them. Some of those hierarchies are new: the system administrator, for instance, has way more power than anyone in a physical university to determine how learning happens, like an architect with the power to move walls, change the decor, add extensions, and so on, at will. The programmers of the system are almost god-like in their command of its physics. But the ways that they give teachers (or learning designers, or administrators) control, as designers, directors, and regulators of the classroom, are perhaps the most pernicious. In a classroom a teacher may lead (and, by default, usually does). In an LMS, a teacher (or someone playing that role) must lead. The teacher sees things that students cannot, and controls things that the students may not. A teacher configures the space, and determines with some precision how it will be used. With a lot of effort and risk, it can be made to behave differently, but it almost never is.

Functions are everything

An LMS is typically built along functional lines, and those functions are mostly based on loose, superficial observations of what teachers and students seem to do in physical classrooms. The metaphorical classrooms are weird, because they are structured by teaching (seldom learning) function rather than along pedagogical lines: for instance, if you want to talk with someone, you normally need to go to a separate enclosed area inside the classroom or leave a note on the teacher’s desk. Same if you want to take a test, or share your work with others. Another function, another space. Some have many little rooms for different things. Lectures are either literally that (video recordings) or (more usefully, from a learning perspective), text and images to be read on screen, based on the assumption that the only function of lectures is information transmission (it is so very, very much not – that’s its least useful and least effective role). There’s seldom a chance to put even put up your hand to question something. Notices can usually only be pinned on the wall by teachers. Classroom timetables are embodied in software because of course you need a rigid and unforgiving timetable in a medium that sells itself on enabling learning anywhere, any time. Some, including Moodle, will allow you to break up the content differently, but it’s still another timetable; just a timetable without dates. It’s still the teacher who sets the order, pacing and content.

Robot overlords

It’s a high-tech classroom. There are often robots there that are programmed to make you behave in ways determined by those higher in the hierarchy (sometimes teachers, sometimes administrators, sometimes the programmers of the software). For instance, they might act as gatekeepers that prevent you from moving on to the next section before completing the current one, or they might prevent you submitting work before or after a specified date. They might mark your work. There are surveillance cameras everywhere, recording your every move, often only accessible to those with more powerful roles (though sometimes a robot or two might give you a filtered view of it).

Beginnings and ends

You can’t usually go back and visit when your course is over because someone decided it would be a good idea to set opening and closing enrolment dates and assumed that, when they were done, the learning was done (which of course it never is – it keeps on evolving long after explicit teaching and testing occurred). Again, it’s because physical classes are scheduled and terms come to an end because they must be, not because it makes pedagogical sense. And, like almost everything, you can override this default, but hardly anyone ever does, because it brings back those Faustian bargains, especially in manageability.

Dull caricatures of physical spaces

Basically, the LMS is an automated set of metaphorical classrooms that hardens many of the undesirable by-products of educational systems in software in brain-dead ways that have little to do with how best to teach, and that stretch the spatial metaphors that inform it beyond breaking point. Each bit of automation and each navigational decision hardens pedagogical choices. For all the cozy metaphors, programmers invent rather than replicate physics, in the process warping reality in ways that do no good and much harm. Classrooms solved problems of physics for in-person teaching and form part of a much larger structure that has evolved to teach reasonably well (including corridors, common rooms, canteens, and libraries, as it happens). Their more visible functions are only a part of that and, arguably, not the main part. There is much pedagogy embedded in the ways that physical universities, whether by accident or design, have evolved over centuries to support learning in every quadrangle and nook of a coffee shop. LMSs just focus on a limited subset of teaching roles, and empower the teacher in ways that caricature their already excessive dominance in the classroom (which only occurred because it had to, thanks to physics and the constraints it imposed).

LMSs are crap, but they contain recognizable semblances of their physical counterparts and just enough configurability and flexibility to more or less work as teaching tools, a bit, for everyone, almost no matter what their level of digital proficiency might be. They more or less solve the Faustian bargains listed earlier, but they do so by stifling what we wanted and should have been able to do in the first place with online tools, in the process creating new and quite horrific problems, as well as demolishing most of what makes physical universities work in the first place. It never has been true that virtual learning environments are learning environments – they are only ever parts of them – and there are places to escape from them, such as the Landing, other virtual systems, or even just plain old email, but then all those Faustian bargains come back to haunt us again. There has to be a better way.

Beyond the LMS

Cognisant of the issues, Athabasca University is now some way down the path to developing its own distinctive solutions to these problems, in a multi-year multi-million-dollar initiative known as (following the spatial metaphor) the Integrated Learning Environment (ILE). The ILE is not an application. It is an umbrella term for a lot of different, usually independent systems working together as one. Though some of the most interesting opportunities are still only loosely imagined, perhaps because they cause problems that are fiendishly hard to solve (e.g. how can we integrate systems that we build ourselves without creating risks for the rest of the ILE, and what happens when they need to be maintained?) a lot of progress is being made on the non-teaching foundations on which the rest depends (student admin systems, support tools, procedures, etc), as well as on the most visible and perhaps the biggest of its parts, BrightSpace, a proprietary commercial LMS that is meant to replace Moodle, for no obvious pedagogical or technical reasons (it’s no better). It might make economic sense. I don’t know, but I do know that open source software typically costs a fair bit to own, albeit because of the things that make it a much better idea (freedom, flexibility, ownership, etc). There is probably a fair bit of time and money being spent with Desire2Learn (makers of Brightspace) on the things that we spent a fair bit of time and money on many years ago to make Moodle a bit less classroom-like. The choice no doubt has something to do with how reliably and easily it can be made to work with some of the other proprietary commercial systems that someone has decided will make up the ILE. It bothers me greatly that we are not trying hard to choose open source solutions, for reasons that will become clearer in the rest of this post. However, (pedagogically speaking) all the mainstream LMSs are much of a muchness, making the same mistakes as one another in very similar ways, so it probably won’t wreck too much of what we already do within Moodle. But, on its own, it won’t move us much further forward and we could do it better. That’s what the ILE is supposed to do – to make the LMS just a part of a much larger teaching environment, intimately connected with the rest of what the university does for or with students, and extensible with new and better ways of learning, teaching, and assessing learning.

picture of lego bricksLego bricks make poor metaphors

When we were first imagining the ILE, though the approach was admirably participative, engaging much of the university community, I was very worried by the things we were encouraged to focus on. It was all about the functionality, the usability, the design, the tools, the pedagogies, the business systems that supported them. Those things matter, for sure, and should be not be ignored, but they should and will change and grow all the time: in fact, part of the point of building this thing is to do just that. Using the city metaphor, pretty much all that we (collectively) considered were the spaces (the rooms, mainly), and the stuff that goes on inside them, much like LMS designers thought of universities as just collections of classrooms in which teaching functions were performed. Space and stuff are, not uncoincidentally, exactly what Stewart Brand identified long ago as inevitably being the fastest-changing, most volatile parts of any town or city (after site, structure, skin, and services). I’ve written a fair bit on the universality of this principle across all systems. It’s a solid structural principle that applies as much to ecosystems and educational systems as to cities. As Brand observes himself, drawing from O’Neill et al (1986), the larger, slower-changing elements of any system affect the smaller, faster-changing more than vice versa. This is for much the same reasons that path dependencies set in. It’s about the prior providing the context for what follows. Flexible things have to fit into the gaps left by less flexible, older, pre-existing things. In physical spaces, of course these tend to be bigger and/or slower, but the same is true in virtual spaces, where size seldom matters that much, but hardness (inflexibility, brittleness) really does. Though lip service was paid to the word ‘integrated’ in our discussions,  I had the strong feeling that the kind of integration we had in mind was that of a Lego set. In fact, I think we were aiming to find a ‘Lego Athabasca University’ set, with assembly instructions and a picture on the box. The vendors who came to talk with us made much of how effectively they could do that, rather than how effectively they could make it possible for others to do that.

Metaphors matter. Lego bricks have to fit together tightly, in pre-specified ways, especially if you are following a plan. If you want to move them around, you have to dismantle a bit of the structure to fit them in. It’s difficult to integrate things that are not bricks, or that are made by different toy companies to work in different ways. At best you get what Brand calls ‘magazine architecture’, or ‘no road’ architecture, beautiful, fit for purpose, intricate and solid, but slow to learn. Lego is not a terrible way to build, compared with buying everything pre-assembled, but it could be improved.

Signals and boundaries

Drawing inspiration from John Holland’s brilliant last work, Signals & Boundaries, I tried to make the case that, instead, we should be focusing on the boundaries (the interfaces between the buildings and the rest of the city), and the signals that pass between them (the people, the messages, etc, the forms they take and how they move around). In Brand’s terms, I wanted us to be thinking about skin and services, and perhaps even structure, though site – Athabasca University – was a given. Though a few people nodded in agreement, I think it mainly fell on deaf ears. We wanted oven-ready solutions, not the infrastructure to enable those solutions. Though the city metaphor works well, because we are talking about human constructions, others would result in similar ways of thinking: cells in bodies, organisms in ecosystems, brains, termite mounds, and so on. All are organized by boundaries (at many levels of hierarchy) and the signals that pass between them.

The Lego set metaphor – whether deliberately or not – seems to have prevailed for now. A lot of old buildings are being slated for demolition and a lot of new virtual buildings are now being erected as part of this development, many of them chosen not because of problems with existing buildings but so that they can more easily connect together and live in the same cloud. This will very likely work, for now, but it is not cheap and it is not flexible, especially given the fact that most of it is not open so, like a rental property, we are not allowed to fix things, add utilities, change the walls, etc, and we are wholly dependent on the landlords being nice to us and each other (knowing that some – ahem, Microsoft – have a long history of abusing their tenants). Those buildings will age. We will find them cramped. Some will age faster than others, and will have to be modified to keep up, perhaps at high cost. Companies renting them might go out of business or change their terms so we might have to demolish the buildings and rent/make new ones. We will be annoyed at how they do things, usually without asking us. We will hate the landlords who dictate what we can do and how we can do it, and who will keep upping the rent while not doing what we ask. We will want more, and the only way to get it will be to build extensions, buy new brick sets, if it is not enough to pay someone to remodel the interiors (and it won’t be). Of course, because most of the big structural elements will not be open source, we will not be able to do that ourselves.

What the ILE really should be

The ILE is, I think, poorly named, because it should not be an environment at all. Following the building metaphor, the ILE is (or should be) more like the system that connects a lot of buildings, bringing them together into a coherent, safe, livable community. It’s infrastructure and services; it is the roads, the traffic signals, the doors, the sidewalks, the water pipes, the waste pipes, the electricity, the network cables; it is the services – fire, police, schools, traffic control, etc; it is all the many rules, standards, norms and regulations that make them work together to help make an environment in which people can live, work, play, and grow. It’s part of the environment – the part that makes it work – but it is not the environment itself. The environment itself is Athabasca University, not just the tools, processes, and systems that support its functions. That includes, most importantly, the people who are part of the university, or who are visitors to it, who are not just users of the environment or dwellers in its walls, but who are or should be the most significant and visible parts of it, just as trees are part of the environment of forests, not users of the forest. Those people live in physical as well as other virtual environments (social media, Word documents, websites, etc) that the ILE can connect together too, to make them a part of it, so the spatial metaphor gets weird at this point. The ILE makes environmental boundaries fuzzy, permeable, and shifting. It’s not an ILE, it’s an ILI – an integrated learning infrastructure.

If we focused on the connections and interfaces, and on how information and processes need to pass across them, and if we thought hard about the nature of those signals, then we could build a system that is resilient, that adapts, that lasts, that grows, that evolves, with parts that we can seamless replace or improve because the interfaces – the building facades, the mains pipes, the junction boxes, etc – will mostly stay the same, evolving slowly as they should. This is about strategy, not planning,  a way of thinking about systems rather than a sequence of things to do.

Some of the key people involved in the process realize this. They are talking about standards, protocols, and projects to build interfaces between systems, and imagining future needs, though they are inevitably distracted by the process of renting Lego bricks, so I am not sure how much they will be able to stay focused on that. I hope they prevail over those who think they are building a set of classrooms and tightly connected admin offices out of self-contained interlocking bricks because our future depends on getting it right. We are aiming to grow. It just takes one critical piece in the Lego building to fail to support that, and the rest falls apart like a… well, like a pile of bricks.

References

Brand, S. (1997). How buildings learn. Phoenix Illustrated. https://www.penguinrandomhouse.ca/books/320919/how-buildings-learn-by-stewart-brand/9780140139969

Holland, J. H. (2012). Signals and Boundaries: Building Blocks for Complex Adaptive Systems. MIT Press.  https://mitpress.mit.edu/books/signals-and-boundaries

O’Neill, R.V., DeAngelis, D.L, Waide, J. B., & Allen, T. F. H. (1986). A Hierarchical Concept of Ecosystems. Princeton University Press. http://www.gbv.de/dms/bs/toc/025157787.pdf

Postman, N. (1998). Five things we need to know about technological change. Denver, Colorado, 28.  https://student.cs.uwaterloo.ca/~cs492/papers/neil-postman–five-things.html

Skills lost due to COVID-19 school closures will hit economic output for generations (hmmm)

Posted , updated on

Snippet from OECD report on covid-19 and education This CBC report is one of many dozens of articles in the world’s press highlighting one rather small but startling assertion in a recent OECD report on the effects of Covid-19 on education – that the ‘lost’ third of a year of schooling in many countries will lead to an overall lasting drop in GDP of 1.5% across the world. Though it contains many more fascinating and useful insights that are far more significant and helpful, the report itself does make this assertion quite early on and repeats it for good measure, so it is not surprising that journalists have jumped on it. It is important to observe, though, that the reasoning behind it is based on a model developed by Hanushek and Woessman over several years, and an unpublished article by the authors that tries to explain variations in global productivity according to amount and  – far more importantly – the quality of education: that long-run productivity is a direct consequence of the cognitive skills (or knowledge capital) of a nation, that can be mapped directly to how well and how much the population is educated.

As an educator I find this model, at a glance, to be reassuring and confirmatory because it suggests that we do actually have a positive effect on our students. However, there may be a few grounds on which it might be challenged (disclaimer: this is speculation). The first and most obvious is that correlation does not equal causation. The fact that countries that do invest in improving education consistently see productivity gains to match in years to come is interesting, but it raises the question of what led to that investment in the first place and whether that might be the ultimate cause, not the education itself.  A country that has invested in increasing the quality of education would, normally, be doing so as a result of values and circumstances that may lead to other consequences and/or be enabled by other things (such as rising prosperity, competition from elsewhere, a shift to more liberal values, and so on).  The second objection might be that, sure, increased quality of education does lead to greater productivity, but that it is not the educational process that is causing it, as such. Perhaps, for instance, an increased focus on attainment raises aspirations. A further objection might be that the definition of ‘quality’ does not measure what they think it measures. A brief skim of the model used suggests that it makes extensive use of scores from the likes of TIMSS, PIRLS and PISA, standardized test approaches used to compare educational ‘effectiveness’ in different regions that embody quite a lot of biases, are often manipulated at a governmental level, and that, as I have mentioned once or twice before, are extremely dubious indicators of learning: in fact, even when they are not manipulated, they may indicate willingness to comply with the demands of the powerful more than learning (does that improve GDP? Probably).  Another objection might be that absence of time spent in school does not equate to absence of education. Indeed, Hanushek and Woessman’s central thesis is that it is not the amount but the quality of schooling that matters, so it seems bizarre that they might fall back on quantifying learning by time spent in school. We know for sure that, though students may not have been conforming to curricula at the rate desired by schools and colleges, they have not stopped learning. In fact, in many ways and in many places, there are grounds to believe that there have been positive learning benefits: better family learning, more autonomy, more thoughtful pedagogies, more intentional learning community forming, and so on.  Out of this may spring a renewed focus on how people learn and how best to support them, rather than maintaining a system that evolved in mediaeval times to support very different learning needs, and that is so solidly packed with counter technologies and so embedded in so many other systems that have nothing to do with learning that we have lost sight of the ones that actually matter. If education improves as a result, then (if it is true that better and more education improves the bottom line) we may even see gains in GDP. I expect that there are other reasons for doubt: I have only skimmed the surface of the possible concerns.

I may be wrong to be sceptical –  in fairness, I have not read the many papers and books produced by Hanushek and Woessman on the subject, I am not an economist, nor do I have sufficient expertise (or interest) to analyze the regression model that they use. Perhaps they have fully addressed such concerns in that unpublished paper and the simplistic cause-effect prediction distorts their claims. But, knowing a little about complex adaptive systems, my main objection is that this is an entirely new context to which models that have worked before may no longer apply and that, even if they do, there are countless other factors that will affect the outcome in both positive and negative ways, so this is not so much a prediction as an observation about one small part of a small part of a much bigger emergent change that is quite unpredictable. I am extremely cautious at the best of times whenever I see people attempting to find simple causal linear relationships of this nature, especially when they are so precisely quantified, especially when past indicators are applied to something wholly novel that we have never seen before with such widespread effects, especially given the complex relationships at every level, from individual to national.  I’m glad they are telling the story – it is an interesting one that no doubt contains grains of important truths – but it is just an informative story, not predictive science.  The OECD has a bit of track record on this kind of misinterpretation, especially in education. This is the same organization that (laughably, if it weren’t so influential) claimed that educational technology in the classroom is bad for learning. There’s not a problem with the data collection or analysis, as such. The problem is with the predictions and recommendations drawn from it.

Beyond methodological worries, though, and even if their predictions about GDP are correct (I am pretty sure they are not – there are too many other factors at play, including huge ones like the destruction of the environment that makes the odd 1.5% seem like a drop in the barrel) then it might be a good thing. It might be that we are moving – rather reluctantly – into a world in which GDP serves as an even less effective measure of success than it already is. There are already plentiful reasons to find it wanting, from its poor consideration of ecological consequences to its wilful blindness to (and causal effect upon) inequalities, to its simple inadequacy to capture the complexity and richness of human culture and wealth. I am a huge fan of the state of Bhutan’s rejection of the GDP, that it has replaced with the GNH happiness index. The GNH makes far more sense, and is what has led Bhutan to be one of the only countries in the world to be carbon positive, as well as being (arguably but provably) one of the happiest countries in the world. What would you rather have, money (at least for a few, probably not you), or happiness and a sustainable future? For Bhutan, education is not for economic prosperity: it is about improving happiness, which includes good governance, sustainability, and preservation of (but not ossification of) culture.

Many educators – and I am very definitely one of them – share Bhutan’s perspective on education. I think that my customer is not the student, or a government, or companies, but society as a whole, and that education makes (or should make) for happier, safer, more inventive, more tolerant, more stable, more adaptive societies, as well as many other good things. It supports dynamic meta-stability and thus the evolution of culture. It is very easy to lose sight of that goal when we have to account to companies, governments, other institutions, and to so many more deeply entangled sets of people with very different agendas and values, not to mention our inevitable focus on the hard methods and tools of whatever it is that we are teaching, as well as the norms and regulations of wherever we teach it. But we should not ever forget why we are here. It is to make the world a better place, not just for our students but for everyone. Why else would we bother?

Originally posted at: https://landing.athabascau.ca/bookmarks/view/6578662/skills-lost-due-to-covid-19-school-closures-will-hit-economic-output-for-generations-hmmm

Black holes are simpler than forests and science has its limits

Posted , updated on

Mandelbrot set (Wikipedia, https://en.wikipedia.org/wiki/Mandelbrot_set)Martin Rees (UK Astronomer Royal) takes on complexity and emergence. This is essentially a primer on why complex systems – as he says, accounting for 99% of what’s interesting about the world – are not susceptible to reductionist science despite being, at some level, reducible to physics. As he rightly puts it, “reductionism is true in a sense. But it’s seldom true in a useful sense.” Rees’s explanations are a bit clumsy in places – for instance, he confuses ‘complicated’ with ‘complex’ once or twice, which is a rooky mistake, and his example of the Mandelbrot Set as ‘incomprehensible’ is not convincing and rather misses the point about why emergent systems cannot be usefully explained by reductionism (it’s about different kinds of causality, not about complicated patterns) – but he generally provides a good introduction to the issues.

These are well-trodden themes that most complexity theorists have addressed in far more depth and detail, and that usually appear in the first chapter of any introductory book in the field, but it is good to see someone who, from his job title, might seem to be an archetypal reductive scientist (he’s an astrophysicist) challenging some of the basic tenets of his discipline.

Perhaps my favourite works on the subject are John Holland’s Signals and Boundaries, which is a brilliant, if incomplete, attempt to develop a rigorous theory to explain and describe complex adaptive systems, and Stuart Kauffman’s flawed but stunning Reinventing the Sacred, which (with very patchy success) attempts to bridge science and religious belief but that, in the process, brilliantly and repeatedly proves, from many different angles, the impossibility of reductive science explaining or predicting more than an infinitesimal fraction of what actually matters in the universe. Both books are very heavy reading, but very rewarding.

Address of the bookmark: https://aeon.co/ideas/black-holes-are-simpler-than-forests-and-science-has-its-limits

Originally posted at: https://landing.athabascau.ca/bookmarks/view/2874665/black-holes-are-simpler-than-forests-and-science-has-its-limits

Cocktails and educational research

Posted , updated on

A lot of progress has been made in medicine in recent years through the application of cocktails of drugs. Those used to combat AIDS are perhaps the most well-known, but there are many other applications of the technique to everything from lung cancer to Hodgkin’s lymphoma. The logic is simple. Different drugs attack different vulnerabilities in the pathogens etc they seek to kill. Though evolution means that some bacteria, viruses or cancers are likely to be adapted to escape one attack, the more different attacks you make, the less likely it will be that any will survive.

Simulated learningUnfortunately, combinatorial complexity means this is not a simply a question of throwing a bunch of the best drugs of each type together and gaining their benefits additively. I have recently been reading John H. Miller’s ‘A crude look at the whole: the science of complex systems in business, life and society‘ which is, so far, excellent, and that addresses this and many other problems in complexity science. Miller uses the nice analogy of fashion to help explain the problem: if you simply choose the most fashionable belt, the trendiest shoes, the latest greatest shirt, the snappiest hat, etc, the chances of walking out with the most fashionable outfit by combining them together are virtually zero. In fact, there’s a very strong chance that you will wind up looking pretty awful. It is not easily susceptible to reductive science because the variables all affect one another deeply. If your shirt doesn’t go with your shoes, it doesn’t matter how good either are separately. The same is true of drugs. You can’t simply pick those that are best on their own without understanding how they all work together. Not only may they not additively combine, they may often have highly negative effects, or may prevent one another being effective, or may behave differently in a different sequence, or in different relative concentrations. To make matters worse, side effects multiply as well as therapeutic benefits so, at the very least, you want to aim for the smallest number of compounds in the cocktail that you can get away with. Even were the effects of combining drugs positive, it would be premature to believe that it is the best possible solution unless you have actually tried them all. And therein lies the rub, because there are really a great many ways to combine them.

Miller and colleagues have been using the ideas behind simulated annealing to create faster, better ways to discover working cocktails of drugs. They started with 19 drugs which, a small bit of math shows, could be combined in 2 to the power of 19 different ways – about half a million possible combinations (not counting sequencing or relative strength issues). As only 20 such combinations could be tested each week, the chances of finding an effective, let alone the best combination, were slim within any reasonable timeframe. Simplifying a bit, rather than attempting to cover the entire range of possibilities, their approach finds a local optimum within one locale by picking a point and iterating variations from there until the best combination is found for that patch of the fitness landscape. It then checks another locale and repeats the process, and iterates until they have covered a large enough portion of the fitness landscape to be confident of having found at least a good solution: they have at least several peaks to compare. This also lets them follow up on hunches and to use educated guesses to speed up the search. It seems pretty effective, at least when compared with alternatives that attempt a theory-driven intentional design (too many non-independent variables), and is certainly vastly superior to methodically trying every alternative, inasmuch as it is actually possible to do this within acceptable timescales.

The central trick is to deliberately go downhill on the fitness landscape, rather than following an uphill route of continuous improvement all the time, which may simply get you to the top of an anthill rather than the peak of Everest in the fitness landscape. Miller very effectively shows that this is the fundamental error committed by followers of the Six-Sigma approach to management, an iterative method of process improvement originally invented to reduce errors in the manufacturing process: it may work well in a manufacturing context with a small number of variables to play with in a fixed and well-known landscape, but it is much worse than useless when applied in a creative industry like, say, education, because the chances that we are climbing a mountain and not an anthill are slim to negligible. In fact, the same is true even in manufacturing: if you are just making something inherently weak as good as it can be, it is still weak. There are lessons here for those that work hard to make our educational systems work better. For instance, attempts to make examination processes more reliable are doomed to fail because it’s exams that are the problem, not the processes used to run them. As I finish this while listening to a talk on learning analytics, I see dozens of such examples: most of the analytics tools described are designed to make the various parts of the educational machine work ‘ better’, ie. (for the most part) to help ensure that students’ behaviour complies with teachers’ intent. Of course, the only reason such compliance was ever needed was for efficient use of teaching resources, not because it is good for learning. Anthills.

This way of thinking seems to me to have potentially interesting applications in educational research. We who work in the area are faced with an irreducibly large number of recombinable and mutually affective variables that make any ethical attempt to do experimental research on effectiveness (however we choose to measure that – so many anthills here) impossible. It doesn’t stop a lot of people doing it, and telling us about p-values that prove their point in more or less scupulous studies, but they are – not to put too fine a point on it – almost always completely pointless.  At best, they might be telling us something useful about a single, non-replicable anthill, from which we might draw a lesson or two for our own context. But even a single omitted word in a lecture, a small change in inflection, let alone an impossibly vast range of design, contextual, historical and human factors, can have a substantial effect on learning outcomes and effectiveness for any given individual at any given time. We are always dealing with a lot more than 2 to the power of 19 possible mutually interacting combinations in real educational contexts. For even the simplest of research designs in a realistic educational context, the number of possible combinations of relevant variables is more likely closer to 2 to the power of 100 (in base 10 that’s  1,267,650,600,228,229,401,496,703,205,376). To make matters worse, the effects we are looking for may sometimes not be apparent for decades (having recombined and interacted with countless others along the way) and, for anything beyond trivial reductive experiments that would tell us nothing really useful, could seldom be done at a rate of more than a handful per semester, let alone 20 per week. This is a very good reason to do a lot more qualitative research, seeking meanings, connections, values and stories rather than trying to prove our approaches using experimental results. Education is more comparable to psychology than medicine and suffers the same central problem, that the general does not transfer to the specific, as well as a whole bunch of related problems that Smedslund recently coherently summarized. The article is paywalled, but Smedlund’s abstract states his main points succinctly:

“The current empirical paradigm for psychological research is criticized because it ignores the irreversibility of psychological processes, the infinite number of influential factors, the pseudo-empirical nature of many hypotheses, and the methodological implications of social interactivity. An additional point is that the differences and correlations usually found are much too small to be useful in psychological practice and in daily life. Together, these criticisms imply that an objective, accumulative, empirical and theoretical science of psychology is an impossible project.”

You could simply substitute ‘education’ for ‘psychology’ in this, and it would read the same. But it gets worse, because education is as much about technology and design as it is about states of mind and behaviour, so it is orders of magnitude more complex than psychology. The potential for invention of new ways of teaching and new states of learning is essentially infinite. Reductive science thus has a very limited role in educational research, at least as it has hitherto been done.

But what if we took the lessons of simulated annealing to heart? I recently bookmarked an approach to more reliable research suggested by the Christensen Institute that might provide a relevant methodology. The idea behind this is (again, simplifying a bit) to do the experimental stuff, then to sweep the normal results to one side and concentrate on the outliers, performing iterations of conjectures and experiments on an ever more diverse and precise range of samples until a richer, fuller picture results. Although it would be painstaking and longwinded, it is a good idea. But one cycle of this is a bit like a single iteration of Miller’s simulated annealing approach, a means to reach the top of one peak in the fitness landscape, that may still be a low-lying peak. However if, having done that, we jumbled up the variables again and repeated it starting in a different place, we might stand a chance of climbing some higher anthills and, perhaps, over time we might even hit a mountain and begin to have something that looks like a true science of education, in which we might make some reasonable predictions that do not rely on vague generalizations. It would either take a terribly long time (which itself might preclude it because, by the time we had finished researching, the discipline will have moved somewhere else) or would hit some notable ethical boundaries (you can’t deliberately mis-teach someone), but it seems more plausible than most existing techniques, if a reductive science of education is what we seek.

To be frank, I am not convinced it is worth the trouble. It seems to me that education is far closer as a discipline to art and design than it is to psychology, let alone to physics. Sure, there is a lot of important and useful stuff to be learned about how we learn: no doubt about that at all, and a simulated annealing approach might speed up that kind of research. Painters need to know what paints do too. But from there to prescribing how we should therefore teach spans a big chasm that reductive science cannot, in principle or practice, cross. This doesn’t mean that we cannot know anything: it just means it’s a different kind of knowledge than reductive science can provide. We are dealing with emergent phenomena in complex systems that are ontologically and epistemologically different from the parts of which they consist. So, yes, knowledge of the parts is valuable, but we can no more predict how best to teach or learn from those parts than we can predict the shape and function of the heart from knowledge of cellular organelles in its constituent cells. But knowledge of the cocktails that result – that might be useful.