The ‘paradigms’ chapter more or less wrote itself – we’ve churned those ideas around for long enough now that we both know the topic rather well – but this one caused us a lot more trouble. Our difficulties were largely due to the fact that we started out with roughly as much idea about what the term ‘informal learning’ means as anyone else. In other words, we kind of recognized it when we saw it, but could come up with no plausible definition that was not either simply wrong, incomplete, or vaguely defined as ‘not formal’ (sometimes adding the utterly circular cop-out notion of ‘non-formal’). As we later figured, ‘formal’ is no better defined than ‘informal’, so that didn’t help. Faced with the need to cover a fairly representative sample of work in the area, we therefore made a mess of it. Our initial draft consisted mainly of a set of examples culled mainly from Terry’s encyclopaedic knowledge of the literature in the field, bound together in loosely connected themes. Because the literature we were citing was based on a large, vague, and often mutually contradictory variety of understandings of ‘informal learning’ the chapter reflected this too: the parts were fine, but the whole was quite incoherent. We needed a better framework.
So we started to brainstorm a few different ways of thinking about the problem, looking at as many ways the term was used as we could find, identifying common patterns and frequently associated concepts, trying to distinguish necessary from sufficient conditions, and consequently finding a much bigger mess than the one we had started with. The amount of fuzzy thinking and loose, almost arbitrary terminology found in the field of informal learning turns out to be quite staggering. It’s not a field: it’s a jungle.
Not for the first time, though, I found Michael Erault’s work in the area to be an inspiration and source of clarity. Erault doesn’t try to come up with a single defining characteristic, instead recognizing that there is a richly variegated continuum of informal-to-formal ways that people learn from and with one another (at least in the workplace settings he has studied). Although (as far as I know) he didn’t explicitly use the term, the sets of characteristics that Erault uses to identify relative degrees of informality seemed to me to imply that he was thinking in terms of what Wittgenstein described as Familienähnlichkeit (family resemblances). No single cluster of characteristics define learning as informal (or formal, for that matter) but, if enough are present, we can usually recognize it as one or the other, or somewhere in between.
This gave us a useful starting point, but it still left a lot of vagueness, and Erault’s focus on informal workplace learning did not fully address all of the meanings and instantiations of informal learning that are particularly significant when examining digital contexts – all the stuff that happens in exchanges through social media, for instance, from Quora to YouTube tutorials and back through email, Reddit, and Twitter. Also, it seemed to gloss over the formal stuff which (as we noted) is as poorly defined as ‘informal’, and that almost never occurs in anything resembling a ‘pure’ form: there is hardly ever any formal learning without informal learning lurking close by. It would be a lot easier if we just talked about formal teaching, because that does refer to a much clearer set of better-defined activities, but teaching is not at all the same thing as learning. Indeed, sometimes the relationship is very oblique indeed, notwithstanding Frere’s claims that you cannot call it teaching unless learning occurs. And then there’s the complex role of credentials of various kinds in both assessing and influencing learning. We wanted to find a way to capture the richness of that, but could find no existing work that worked well enough for us.
We went through a lot of different concepts and representations (yes, there were Venn diagrams!) before finally hitting on the notion that it is not so much a two-dimensional continuum between formal and informal, but a multi-dimensional spectrum defined in terms of relative degrees of dependence/independence and intentionality/non-intentionality.
We (tentatively) reckon that we can situate at least most existing work in the field within this framework, and that it provides a helpful way of thinking about whatever is happening in a particular moment of a learning trajectory (another concept from Erault that I’ve found very useful in the past, especially when talking about transactional control in my first book). An individual’s learning trajectory will constantly wind around this space and, when other individuals are involved (not just formal teachers), their paths will affect one another in interesting ways. After we’d worked this out, the rest of the chapter fell more or less into place. You can read the result here.
Here’s the chapter abstract:
Governments, business leaders, educators, students, and parents realize the need to inculcate a culture of lifelong learning – learning that spans geography, time, and lifespan. This learning has both formal and informal components. In this chapter, we examine the conceptual basis upon which informal learning is defined and some of the tools and techniques used to support informal learning. We overview the rapid development in information and communications technologies that not only creates opportunities for learners, teachers, and researchers but also challenges us to create equitable and culturally appropriate tools and contexts in which high-quality, continuous learning is available to all.
Dron J., Anderson T. (2022) Informal Learning in Digital Contexts. In: Zawacki-Richter O., Jung I. (eds) Handbook of Open, Distance and Digital Education. Springer, Singapore. https://doi.org/10.1007/978-981-19-0351-9_84-1
This is a chapter by me and Terry Anderson for Springer’s new Handbook of Open, Distance, and Digital Education that updates and refines our popular (1658 citations, and still rising, for the original paper alone) but now long-in-the-tooth ‘three generations’ model of distance learning pedagogy. We have changed the labels for the pedagogical families this time round to ones that I think are more coherent, divided according to their epistemological underpinnings: the objectivist, the subjectivist, and the complexivist. and we have added some speculations about whether further paradigms might have started to emerge in the 11 years since our original paper was published. Our main conclusion, though, is that no single pedagogical paradigm will dominate in the foreseeable future: that we are in an era of great pedagogical diversity, and that this diversity will only increase as time goes by.
The three major paradigms
Objectivist: previously known as ‘behaviourist/cognitivist’, what characterizes objectivist pedagogies is that they are both defined by assumptions of an objective external reality, and driven by (usually teacher-defined) objectives. It’s a paradigm of teaching, where teachers are typically sages on the stage using methods intended to achieve effective learning of defined facts and skills. Examples include behaviourism, learning styles theories, brain-based approaches, multiple intelligence models, media theories, and similar approaches where the focus is on efficient transmission and replication of received knowledge.
Subjectivist: formerly known as ‘social constructivist’, subjectivist pedagogies are concerned with – well – subjects: they are concerned with the personal and social co-construction of knowledge, recognizing its situated and always unique nature, saying little about methods but a lot about meaning-making. It’s a paradigm of learning, where teachers are typically guides on the side, supporting individuals and groups to learn in complex, situated contexts. Examples include constructivist, social constructivist, constructionist, and similar families of theory where the emphasis is as much on the learners’ growth and development in a human society as it is on what is being learned.
Complexivist: originally described as ‘connectivist’ (which was confusing and inaccurate), complexivist pedagogies acknowledge and exploit the complex nature of our massively distributed cognition, including its richly recursive self-organizing and emergent properties, its reification through shared tools and artefacts, and its many social layers. It’s a paradigm of knowledge, where teachers are fellow learners, co-travellers and role models, and knowledge exists not just in individual minds but in our minds’ extensions, in both other people and what we collectively create. Examples include connectivism, rhizomatic learning, distributed cognition, cognitive apprenticeship, networks of practice, and similar theories (including my own co-participation model, as it happens). We borrow the term ‘complexivist’ from Davis and Sumara, whose 2006 book on the subject is well worth reading, albeit grounded mainly in in-person learning.
No one paradigm dominates: all typically play a role at some point of a learning journey, all build upon and assemble ideas that are contained in the others (theories are technologies too), and all have been around as ways of learning for as long as humans have existed.
Beyond these broad families, we speculate on whether any new pedagogical paradigms are emerging or have emerged within the 12 years since we first developed these ideas. We come up with the following possible candidates:
Theory-free: this is a digitally native paradigm that typically employs variations of AI technologies to extract patterns from large amounts of data on how people learn, and that provides support accordingly. This is the realm of adaptive hypermedia, learning analytics, and data mining. While the vast majority of such methods are very firmly in the objectivist tradition (the models are trained or designed by identifying what leads to ‘successful’ achievement of outcomes) a few look beyond defined learning products into social engagement or other measures of the learning process, or seek open-ended patterns in emergent collective behaviours. We see the former as a dystopic trend, but find promise in the latter, notwithstanding the risks of filter bubbles and systemic bias.
Hologogic: this is a nascent paradigm that treats learning as a process of enculturation. It’s about how we come to find our places in our many overlapping cultures, where belonging to and adopting the values and norms of the sets to which we belong (be it our colleagues, our ancestors, our subject-matter peers, or whatever) is the primary focus. There are few theories that apply to this paradigm, as yet, but it is visible in many online and in-person communities, and is/has been of particular significance in collectivist cultures where the learning of one is meaningless unless it is also the learning of all (sometimes including the ancestors). We see this as a potentially healthy trend that takes us beyond the individualist assumptions underpinning much of the field, though there are risks of divisions and echo chambers that pit one culture against others. We borrow the term from Cumbie and Wolverton.
Bricolagogic: this is a free-for-all paradigm, a kind of meta-pedagogy in which any pedagogical method, model, or theory may be used, chosen for pragmatic or personal reasons, but in which the primary focus of learning is in choosing how (in any given context) we should learn. Concepts of charting and wayfinding play a strong role here. This resembles what we originally identified as an emerging ‘holistic’ model, but we now see it not as a simple mish-mash of pedagogical paradigms but rather as a pedagogic paradigm in its own right.
Another emerging paradigm?
I have recently been involved in a lengthy Twitter thread, started by Tim Fawns on the topic of his recent paper on entangled pedagogy, which presents a view very similar indeed to my own (e.g. here and here), albeit expressed rather differently (and more eloquently). There are others in the same thread who express similar views. I suggested in this thread that we might be witnessing the birth of a new ‘entanglist’ paradigm that draws very heavily on complexivism (and that could certainly be seen as part of the same family) but that views the problem from a rather different perspective. It is still very much about complexity, emergence, extended minds, recursion, and networks, and it negates none of that, but it draws its boundaries around the networked nodes at a higher level than theories like Connectivism, yet with more precision than those focused on human learning interactions such as networks of practice or rhizomatic learning. Notably, it leaves room for design (and designed objects), for meaning, and for passion as part of the deeply entangled complex system of learning in which we all participate, willingly or not. It’s not specifically a pedagogical model – it’s broader than that – though it does imply many things about how we should and should not teach, and about how we should understand pedagogies as part of a massively distributed system in which designated teachers account for only a fraction of the learning and teaching process. The title of my book on the subject (that has been under review for 16 months – grrr) sums this up quite well, I think: “How Education Works”. The book has now (as of a few days ago) received a very positive response from reviewers and is due to be discussed by the editorial committee at the end of this month, so I’m hoping that it may be published in the not-too-distant future. Watch this space!
Here’s the chapter abstract:
Building on earlier work that identified historical paradigm shifts in open and distance learning, this chapter is concerned with analyzing the three broad pedagogical paradigms – objectivist, subjectivist, and complexivist – that have characterized learning and teaching in the field over the past half century. It goes on to discuss new paradigms that are starting to emerge, most notably in “theory-free” models enabled by developments in artificial intelligence and analytics, hologogic methods that recognize the many cultures to which we belong, and a “bricolagogic,” theory-agnostic paradigm that reflects the field’s growing maturity and depth.
Dron J., Anderson T. (2022) Pedagogical Paradigms in Open and Distance Education. In: Zawacki-Richter O., Jung I. (eds) Handbook of Open, Distance and Digital Education. Springer, Singapore. https://doi.org/10.1007/978-981-19-0351-9_9-1
This is the (near enough final) English version of my journal paper, translated into Chinese by Junhong Xiao and published last year (with a CC licence) in Distance Education in China. (Reference: Dron, Jon (2021). Technology, technique, and culture in educational systems: breaking the iron triangle (translated by Junhong Xiao). Distance Education in China, 1, 37-49. DOI:10.13541/j.cnki.chinade.2021.01.005).
The underlying theory is the same as that in my paper Educational technology: what it is and how it works (Reference: Dron, J. Educational technology: what it is and how it works. AI & Soc 37, 155–166 (2022). https://doi.org/10.1007/s00146-021-01195-z direct link for reading, link to downloadable preprint) but this one focuses more on what it means for ways we go about distance learning. It’s essentially about ways to solve problems that we created for ourselves by solving problems in the context of in-person learning that we inappropriately transferred to a distance context.
Here’s the abstract: This paper presents arguments for a different way of thinking about how distance education should be designed. The paper begins by explaining education as a technological process, in which we are not just users of technologies for learning but coparticipants in their instantiation and design, implying that education is a fundamentally distributed technology. However, technological and physical constraints have led to processes (including pedagogies) and path dependencies in In-person education that have tended to massively over-emphasize the designated teacher as the primary controller of the process. This has resulted in the development of many counter technologies to address the problems this causes, from classrooms to grades to timetables, most of which have unnecessarily been inherited by distance education. By examining the different strengths and weaknesses of distance education, the paper suggests an alternative model of distance education that is more personal, more situated in communities and cultures, and more appropriate to the needs of learners and society.
I started working on a revised version of this (with a snappier title) to submit to an English language journal last year but got waylaid. If anyone is interested in publishing this, I’m open to submitting it!
This Scientific American article tells the tale of one of the genesis stories of complexity science, this one from 1952, describing what, until relatively recently, was known as the Fermi-Pasta-Ulam (FPU) problem (or ‘paradox’, though it is not in fact a paradox). It is now more commonly known as the Fermi-Pasta-Ulam-Tsingdou (FPUT) problem, in recognition of the fact that it was only discovered thanks to the extraordinary work of Mary Tsingou, who wrote the programs that revealed what, to Fermi, Pasta, and Ulam, was a very unexpected result.
The team was attempting to simulate what happens to energy as it moves around atoms connected by chemical bonds. This is a classic non-linear problem that cannot be observed directly, and that cannot be solved by conventional reductive means (notwithstanding recent work that reveals statistical patterns in complex systems like urban travel patterns). It has to be implemented as a simulation in order to see what happens. Fermi, Pasta, and Ulam thought that, with enough iterations, it would reveal itself to be ergodic: that, given long enough, every state of a given energy of the system would be visited an equal number of times. Instead, thanks to Mary Tsingou’s work, they found that it was non-ergodic. Weird stuff happened, that could not be predicted. It was chaotic.
The discovery was, in fact, accidental. Initial results had shown the expected regularities then, one day, they left the program running for longer than usual and, instead of the recurring periodic patterns seen initially, it suddenly went haywire. It wasn’t a bug in the code. It was a phase transition, perhaps the first unequivocal demonstration of deterministic chaos. Though Fermi died and the paper was not actually published until nearly a decade later, it is hard to understate the importance of this ‘accidental’ discovery that deterministic systems are not necessarily ergodic. As Stuart Kauffman puts it, ‘non-ergodicity gives us history‘. Weather is non-ergodic. Evolution is non-ergodic. Learning is non-ergodic. We are non-ergodic. The universe is non-ergodic. Though there are other strands to the story that predate this work, more than anything else this marks the birth of a whole new kind of science – the science of complexity – that seeks to deal with the 90% or more of phenomena that matter to us, and that reductive science cannot begin to handle.
Here’s a bit of Tsingou’s work on the program, written for the MANIAC computer:
It was not until 2008 that Tsingou’s contribution was fully recognized. In the original paper she was thanked in a footnote but not acknowledged as a co-author. It is possible that, had it been published right away she might have received proper credit. However, it is at least as possible that she might not. The reasons for this are a mix of endemic sexism, and (relatedly) the low esteem accorded to computation at the time.
The relationship between these two factors runs deep. Historically, the word ‘computer’ originally referred to a job title. As scientists in the 19th Century amassed vast amounts of data that needed processing, there was far too much for an individual to handle. They figured out that tasks could be broken up into smaller pieces and farmed out in parallel to humans who could do the necessary rote arithmetic. Because women were much cheaper to hire, and computing was seen as a relatively unskilled (albeit very gruelling and cognitively demanding) role, computing therefore became a predominantly female occupation. From the 19th Century onwards into the mid 20th Century, all-women teams worked on astronomical data, artillery trajectories, and similar tasks, often performing extremely complex mathematical calculations requiring great precision and endurance, always for far less pay than they deserved or that a man would receive. Computers were victims of systematic gender discrimination from the very beginning.
The FPUT problem, however, is one that doesn’t lend itself to chunking and parallel computation: the output of one iteration of the computation is needed before you can calculate the next. Farming it out to human computers simply wouldn’t work. For work of this kind, you have to have a machine or it would take decades to come up with a solution.
In the first decade or so after digital computers were invented significant mathematical skill was needed to operate them. Because of their existing exploitation as human computers, there was, luckily enough, a large workforce of women with advanced math skills whose manual work was being obsoleted at the same time, so women played a significant role in the dawn of the industry. Mary Tsingou was not alone in making great contributions to the field.
By the 1970s that had changed a lot, not in a good way, but numbers slowly grew again until around the mid-1980s (a terrible decade in so many ways) when things abruptly changed for the worse.
Whether this was due to armies of parents buying PCs for their (male) children thanks to aggressive marketing to that sector, or highly selective media coverage, or the increasing recognition of the value of computing skills in the job market reinforcing traditional gender disparities, or something else entirely (it is in fact complex, with vast self-reinforcing feedback loops all the way down the line), the end result was a massive fall in women in the field. Today, less than 17% of students of computer science are women, while the representation of women in most other scientific and technical fields has grown considerably.
There’s a weirder problem at work here, though, because (roughly – this is an educated guess) less than 1% of computer science graduates ever wind up doing any computer science, unless they choose a career in academia (in which case the figure rises to very low single figures), and very few of them ever do more mathematics than an average greengrocer. What we teach in universities has wildly diverged from the skills that are actually needed in most computing occupations at an even sharper rate than the decline of women in the trade. We continue to teach it in ways that would have made sense in the 1950s, when it could not be done without a deep understanding of mathematics and the science behind digital computation, even though neither of these skills has much if any use at all for more than a minute fraction of our students when they get out into the real world. Sure, we have broadened our curriculum to include many other aspects of the field, but we don’t let students study them unless they also learn the (largely unnecessary in most occupations) science and math (a subject that suffers even lower rates of non-male participation than computing). Thinking of modern computing as a branch of mathematics is a bit like treating poetry as a branch of linguistics or grammar, and thinking of modern computing as a science is a bit like treating painting as a branch of chemistry. It’s not so much that women have left computing but that computing – as a taught subject – has left women.
Computing professionals are creative problem solvers, designers, architects, managers, musicians, writers, networkers, business people, artists, social organizers, builders, makers, teachers, or dreamers. The main thing that they share in common is that they work with computers. Some of them are programmers. A few (mostly those involved in designing machines and compilers) do real computer science. A few more do math, though rarely at more than middle school level, unless they are working on the cutting edge of a few areas like graphics, AI, or data science (in which case the libraries etc that would render it unnecessary have not yet been invented). The vast majority of computing professionals are using the outputs of this small elite’s work, not reinventing it. It it not surprising that there is enormous diversity in the field of computing because computers are universal machines, universal media, and universal environments, so they encompass the bulk of human endeavour. That’s what makes them so much fun. If you are a computing professional you can work with anyone, and you can get involved in anything that involves computers, which is to say almost everything. And they are quite interesting in and of themselves, partly because they straddle so many boundaries, and ideas and tools from one area can spark ideas and spawn tools in another.
If you consider the uses of computer applications in many fields, from architecture or design to medicine or media to art or music, there is a far more equal gender distribution. Computing is embedded almost everywhere, and it mostly demands very different skills in each of its uses. There are some consistent gaps that computing students could fill or, better, that computing profs could teach in the context they are used. Better use could be made of computers across the board with just a little programming or other technical skills. Unfortunately, those who create, maintain, and manage computers and their applications tend to mainly come out of computer science programs (at least in North America and some other parts of the world) so many are ill prepared for participating in all that richness, and computing profs tend to stick with teaching in computer science programs so the rest of the world has to figure out things they could help with for themselves.
I think it is about time that we relegated computer science to a minor (not unimportant) stream and got back into the real world – the one with women in it. There’s still a pressing need to bring more women into that minor stream: we need inspirations like Mary Tsingou, we could do worse than preferentially hiring more non-male professors, and we desperately need to shift the discriminatory culture surrounding (especially) mathematics but, if we can at least teach in a way that better represents the richness and diversity of the computing profession itself, it would be a good start.
Originally posted at: https://landing.athabascau.ca/bookmarks/view/10624709/some-thoughts-for-ada-lovelace-day
With a possible audience of thousands, and without a clue they were there because the Zoom output was streamed to a different site (weird), I talked very fast about my experience of higher education for about 20 minutes at the THE UK Student Festival yesterday. The talk was recorded and will be used again for the THE Canada Student Festival later in the month. It’s a huge event – over 8000 enrolled (though not all attending every session) – with quite a lot of other keynotes and a great many other talks, panels, and discussions, aimed at helping students starting out in higher education. The audience was very different from those I normally talk to, and the (very sensible) strict 20-minute format gave me a lot less leeway than the usual hour allowed, so I found it interestingly challenging. These are the slides I used.
The brief I was given was not to preach, but to share my experience of higher education, as a student and as an educator. My personal agenda was to talk about why and how online learning is worth doing (especially at AU), so I tried (a bit clunkily) to aim my story in that direction. I failed to remember to mention some key things, I spent too long on others, and I suspect that the most memorable message that came through was, for in-person students living on campus, to get a kettle (it’s a great way to make lots of new friends fast) but, hopefully, my bigger message got through to some.
The essential point of my wild ramble was not that kettles are the solution to success in higher education, but that students and the rest of us should take ownership of our own education: it should be done by us, not to us. We should learn the way we want to learn, and we should learn what we want to learn; we should seek adventure and challenge rather than easy pickings; and we should hang out with interesting people and/or those we care about, because that’s how most learning happens, as well as being a large part of what makes it meaningful. I noted that we should focus on learning and should to try to ignore grades as much as possible, because grades destroy the love of doing something simply because we enjoy it. Essentially, my advice was about finding the things that intrinsically motivate us and reducing the effects of things that demotivate us. My own educational journey, and (I think) that of most committed educators, has largely followed that path: that’s how we thrived in a system not conducive to intrinsic motivation, and it’s the path we try to encourage our own students to take. I observed that it is much easier to own your own education when it is done online, at least if it is done in ways that take advantage of the medium, and not through a pale simulacrum of in-person teaching, because the teacher cannot be in control, the level of challenge is much more controllable, and there are way more people (online and in your own environment) who can support you.
There has been a lot of negative reaction of late to virtual proctors of online exams. Perhaps students miss the cheery camaraderie of traditional proctored exams, sitting silently in a sweaty room with pen and paper, doing one of the highest stakes, highest stress tasks of their lives, with someone scrutinizing their every nervous tic whose adverse judgment may destroy their hopes and careers, for the benefit of an invisible examiner whose motives and wishes are unclear but whose approval they dearly seek. Lovely. Traditional. Reassuring. A ritual for us all to cherish. It’s enough to bring a tear to the eye.
But exams cost a huge amount of money to host and to invigilate. It is even worse when one of the outcomes might, for the student or the invigilator, be death or disability due to an inconvenient virus.
I have a better solution.
Instead of costly invigilators and invigilation centres, all we need to do is to send out small (returnable, postage-paid) robots to students’ homes. A little robot sitting on the student’s desk or kitchen table as they sit their written exam (on paper, of course – tradition matters), recording every blink, watching their fingers writing on the paper, with 360 degree panoramic camera and the ability to zoom in on anything suspicious or interesting. Perhaps it could include microphones, infrared and microwave sensors, and maybe sensors to monitor skin resistance, pulse, etc, in order to look for nefarious activities or to call the ambulance if the student seems to be having a heart attack or stroke due to the stress. It could be made to talk, too. Perhaps it could offer spoken advice on the process, and alerts about the time left at carefully selected intervals. Students could choose the voice. It would also allow students to sit exams wherever and whenever they please: we are all in favour of student choice. With a bit of ingenuity it could scan what the students have written or drawn, and send it back to an examiner. Or, with a bit more ingenuity and careful use of AI, it could mark the paper on the spot, saving yet more money. Everyone wins.
It would be important to be student-centric in its design. It could, for instance, be made to look like a cute little furry animal with googly eyes to put students more at ease. Maybe it could make soothing cooing noises like a tribble, or like a cat purring. Conversely, it could be made to scuttle ominously around the desk and to appear like a spider with venomous-looking fangs, making gentle hissing noises, to remind students of the much lamented presence of in-person invigilators. Indeed, maybe it could be made to look like a caricature of a professor. More advanced models could emit bad smells to replicate invigilator farts or secret smoking habits. It could be made small and mobile, so that students could take it with them if they needed a bathroom break, during which it might play soothing muzak to put the student at ease, while recording everything they do. It would have to be tough, waterproof, and sterilizable, in order to cope with the odd frustrated student throwing or dunking it.
Perhaps it could offer stern spoken warnings if anomalies or abuses are found, and maybe connect itself to a human invigilator (I hear that they are cheaper in developing nations) who could control it and watch more closely. Perhaps it could be equipped with non-lethal weaponry to punish inappropriate behaviour if the warnings fail, and/or register students on an offenders database. It could be built to self-destruct if tampered with.
Though this is clearly something every university, school, and college would want, and the long-term savings would be immense, such technologies don’t come cheap. Quite apart from the hardware and software development costs, there would be a need for oodles of bandwidth and storage of the masses of data the robot would generate.
I have a solution to that, too: commercial sponsorship.
We could partner with, say, Amazon, who would be keen to mine useful information about the students’ surroundings and needs identified using the robot’s many sensors. A worn curtain? Stubborn stains? A shirt revealing personal interests? Send them to Amazon! Maybe Alexa could provide the voice for interactions and offer shopping advice when students stop to sharpen their pencils (need a better pencil? We have that in stock and can deliver it today!). And, of course, AWS would provide much of the infrastructure needed to support it, at fair educational prices. I expect early adopters would be described as ‘partners’ and offered slightly better (though still profitable) deals.
And there might be other things that could be done with the content. Perhaps the written answers could be analyzed to identify potential Amazon staffers. Maybe students expressing extremist views could be reported to the appropriate government agency, or at least added to a watch-list for the institution’s own use.
Naysayers might worry about hackers breaking into it or subverting its transmissions, or the data being sent to a country with laughable privacy laws, or the robot breaking down at a critical moment, or errors in handwriting recognition, but I’m sure that could be dealt with, the same as we deal with every other privacy, security, and reliability issue in IT in education. No problem. No sir. We have lawyers.
The details still need to be ironed out here and there, but the opportunities are endless. What could possibly go wrong? I think we should take this seriously. Seriously.
I am slowly getting used to the ugly abbreviation WFH that has emerged during the pandemic, though I don’t much like it because it’s not always accurate. Even in pandemic times I often work from my boat (WFB). In non-pandemic times I’ve worked from a tent (WFT), a library (WFL), a hotel room (WFHR), a park bench (WFPB), a conference (WFC), a plane (WFP), a bus (WF… OK, you get the picture), and much, much more. I have even worked at Athabasca University’s own buildings (Working from Work?) on rare occasions. But why do most of us in the trade so rarely use terms like learning from home when working from home (WFH) is so ubiquitous?
Terms like e-learning, online learning, distance learning, remote learning, and so on, are weird. Learning is never remote, electronic, online, or at a distance. There is more sense to terms like distance education, online education, remote teaching, and so on, because education and teaching describe relationships between people, and there are different ways that those relationships can be mediated, that do (or should) deeply affect the process. There is also a whole slew of intentional and implicit structures, systems, methods, and toolsets that are assumed when we prefix education with terms like distance or online. But why online or distance learning?
As teachers we are (rightly) taught that it’s not about the teaching, it’s about the learning. For at least the last 30 years or more we have, for instance, therefore been strongly encouraged to use the term ‘learning & teaching’ instead of ‘teaching & learning’ because learning must come first. I’ve corrected people myself for getting the order wrong, many times. Charitably, therefore, it might be that we are trying to draw attention to the fact that it’s about learning. But, if so, why distance or online?
I think something nasty has happened to the term ‘learning’ when it is used this way, because I think that what we actually mean by it is ‘teaching’. Some British English dialects take that dubious elision fully on board. When something nasty happens to someone as a consequence of something they have done that is perceived to be wrong, or even when some punishment is inflicted on them by someone else, it is common in some circles to say ‘that’ll learn yer’ (the ‘yer’ is important – don’t imagine the Queen saying in received pronunciation ‘that will learn you’ because it would be wrong). When I hear the phrase I imagine it being said with a snarl. It’s a cruel thing to say, though it can be used kind-of humorously, at least if, as many of my compatriots do, you appreciate a particularly crude form of Benny-Hillish shadenfreude (‘Ha ha, you fell flat on your face and hurt yourself. That’ll learn yer’).
Outside a subset of British and perhaps some other minor English vernaculars, learning is never something that we do to people. It’s something done by people, with what and with whom is around them (and that might include a teaching website, textbook, or course pack). So let’s stop calling people distance or online learners because it devalues and obscures what they are actually doing. They are not being learned at. They are being taught at a distance, and learning from home (or wherever they happen to be).
Why do we build digital learning systems to mimic classrooms?
It is understandable that, when we teach in person, we have to occupy and make different uses of the same or similar environments like classrooms, labs, workshops, lecture theatres, and offices. There are huge financial, physical, and organizational constraints on making the environment fit the task, so it would be madness to build a whole new classroom every time we wished to run a different class.
Online, we could build anything we like
But why do we do the same when we teach online? There are countless tools available and, if none are suitable, it is not too hard to build them or modify them to suit our needs. Once they are built, moving between them just takes a tap of a screen or the click of a mouse. Heck, you can even occupy several of them at once if you have a decent monitor or more than one device.
So why don’t we do this?
Here are a few of the more obvious reasons that using the perfect app for the context of study rarely happens:
Teachers’ lack of knowledge of the options (it takes time and effort to discover what’s available).
Teachers’ lack of skill in using them (most interesting tools have a learning curve, and that gets steeper in inverse proportion to the softness and diversity of the toolset, so most teachers don’t even know how to make the most of what they already have).
Lack of time and/or money for development (a real-life application is what it contains, not just the shell that contains it, and it is not always as easy to take existing stuff and put it in a new tool as it might be in a physical space).
Costs and difficulties in management (each tool adds costs in managing faults, configuration, accounting for use, performance, and security).
Cognitive load involved for learners in adapting to the metaphors, signposts, and methods needed to use the tool itself.
All of these are a direct consequence of the very diversity that would make us want to use different apps in the first place. This is a classic Faustian bargain in which the technology does what we want, and in the process creates new problems to solve. Every virtual system invents at least some of the dynamics of how people and things interact with it and within it. In effect, every app has its own physics. That makes them harder to find out about, harder to learn, harder to develop, costlier to manage, and more difficult to navigate than the static, fixed facilities found in particular physical locations. They are all different, there are few if any universals, and any universal today may become a conditional tomorrow. Gravity doesn’t necessarily work the same way in virtual systems.
And so we get learning management systems
The learning management system (LMS) kind of deals with all of these problems: poorly, harmfully, boringly, and painfully, but it does deal with them. Currently, most of the teaching at Athabasca University is through the open source Moodle LMS, lightly modified by us because our needs are not quite like others (self-pacing and all that). But Moodle is not special: in terms of what it does and how it does it, it is not significantly different from any other mainstream LMS – Blackboard, Brightspace, Canvas, Sakai, whatever.
Almost every LMS essentially automates the functions, though not exactly the form, of traditional classrooms. In other parts of the world people prefer to use the term ‘managed learning environment’ (MLE) for such things, and it is the most dominant representative of a larger category of systems usually described as virtual learning environments (VLEs) that also includes things like MOOs (multi-user dungeons, object oriented), immersive learning environments, and simpler web-based teaching systems that replicate aspects of classrooms such as Google Classroom or Microsoft’s gnarly bundle of hastily repurposed rubbish for teaching that I’m not sure even has a name yet. Notice the spatial metaphors in many of these names.
Little boxes made of ticky tacky
The people who originally designed LMSs back in the 90s (I did so myself) based their designs on the functions and entities found in a traditional university because that was their context, and that was where they had to fit. Metaphorically, an LMS or MLE is a big university building with rather uniform classrooms, with perhaps a yard where you can camp out with a few other systems (plugins, LTI hooks, etc) that conform to its requirements and that are allowed in to classrooms when invited, and a few doors and gateways (mainly hyperlinks) linking it circuitously or in jury-rigged fashion to other similarly weakly connected buildings (e.g. places to register, places to seek support, places to talk to an advisor, places to complain, places to find books, and so on). It doesn’t have metaphorical corridors, halls, common rooms, canteens, yards, libraries or any of the other things that normally make up a physical university. You rarely get to even be aware of other classrooms beyond those you are in. Some people (me in a past life) might give classrooms cute names like ‘the learning cafe’ but it’s still just another classroom. You teleport from one classroom to the next because what happens in corridors (really a big lot of incredibly important pedagogically useful stuff, as it happens) is not perceived by the designers as a useful classroom function to be automated or perhaps, more charitably, they just couldn’t figure out how to automate that.
It’s a very controlled environment where everyone has a programmatically enforced role (mostly reflecting traditional educational roles), that may vary according to the room, but that are far less fluid than those in physical spaces. There are strong hierarchies, and limited opportunities for moving between them. Some of those hierarchies are new: the system administrator, for instance, has way more power than anyone in a physical university to determine how learning happens, like an architect with the power to move walls, change the decor, add extensions, and so on, at will. The programmers of the system are almost god-like in their command of its physics. But the ways that they give teachers (or learning designers, or administrators) control, as designers, directors, and regulators of the classroom, are perhaps the most pernicious. In a classroom a teacher may lead (and, by default, usually does). In an LMS, a teacher (or someone playing that role) must lead. The teacher sees things that students cannot, and controls things that the students may not. A teacher configures the space, and determines with some precision how it will be used. With a lot of effort and risk, it can be made to behave differently, but it almost never is.
Functions are everything
An LMS is typically built along functional lines, and those functions are mostly based on loose, superficial observations of what teachers and students seem to do in physical classrooms. The metaphorical classrooms are weird, because they are structured by teaching (seldom learning) function rather than along pedagogical lines: for instance, if you want to talk with someone, you normally need to go to a separate enclosed area inside the classroom or leave a note on the teacher’s desk. Same if you want to take a test, or share your work with others. Another function, another space. Some have many little rooms for different things. Lectures are either literally that (video recordings) or (more usefully, from a learning perspective), text and images to be read on screen, based on the assumption that the only function of lectures is information transmission (it is so very, very much not – that’s its least useful and least effective role). There’s seldom a chance to put even put up your hand to question something. Notices can usually only be pinned on the wall by teachers. Classroom timetables are embodied in software because of course you need a rigid and unforgiving timetable in a medium that sells itself on enabling learning anywhere, any time. Some, including Moodle, will allow you to break up the content differently, but it’s still another timetable; just a timetable without dates. It’s still the teacher who sets the order, pacing and content.
It’s a high-tech classroom. There are often robots there that are programmed to make you behave in ways determined by those higher in the hierarchy (sometimes teachers, sometimes administrators, sometimes the programmers of the software). For instance, they might act as gatekeepers that prevent you from moving on to the next section before completing the current one, or they might prevent you submitting work before or after a specified date. They might mark your work. There are surveillance cameras everywhere, recording your every move, often only accessible to those with more powerful roles (though sometimes a robot or two might give you a filtered view of it).
Beginnings and ends
You can’t usually go back and visit when your course is over because someone decided it would be a good idea to set opening and closing enrolment dates and assumed that, when they were done, the learning was done (which of course it never is – it keeps on evolving long after explicit teaching and testing occurred). Again, it’s because physical classes are scheduled and terms come to an end because they must be, not because it makes pedagogical sense. And, like almost everything, you can override this default, but hardly anyone ever does, because it brings back those Faustian bargains, especially in manageability.
Dull caricatures of physical spaces
Basically, the LMS is an automated set of metaphorical classrooms that hardens many of the undesirable by-products of educational systems in software in brain-dead ways that have little to do with how best to teach, and that stretch the spatial metaphors that inform it beyond breaking point. Each bit of automation and each navigational decision hardens pedagogical choices. For all the cozy metaphors, programmers invent rather than replicate physics, in the process warping reality in ways that do no good and much harm. Classrooms solved problems of physics for in-person teaching and form part of a much larger structure that has evolved to teach reasonably well (including corridors, common rooms, canteens, and libraries, as it happens). Their more visible functions are only a part of that and, arguably, not the main part. There is much pedagogy embedded in the ways that physical universities, whether by accident or design, have evolved over centuries to support learning in every quadrangle and nook of a coffee shop. LMSs just focus on a limited subset of teaching roles, and empower the teacher in ways that caricature their already excessive dominance in the classroom (which only occurred because it had to, thanks to physics and the constraints it imposed).
LMSs are crap, but they contain recognizable semblances of their physical counterparts and just enough configurability and flexibility to more or less work as teaching tools, a bit, for everyone, almost no matter what their level of digital proficiency might be. They more or less solve the Faustian bargains listed earlier, but they do so by stifling what we wanted and should have been able to do in the first place with online tools, in the process creating new and quite horrific problems, as well as demolishing most of what makes physical universities work in the first place. It never has been true that virtual learning environments are learning environments – they are only ever parts of them – and there are places to escape from them, such as the Landing, other virtual systems, or even just plain old email, but then all those Faustian bargains come back to haunt us again. There has to be a better way.
Beyond the LMS
Cognisant of the issues, Athabasca University is now some way down the path to developing its own distinctive solutions to these problems, in a multi-year multi-million-dollar initiative known as (following the spatial metaphor) the Integrated Learning Environment (ILE). The ILE is not an application. It is an umbrella term for a lot of different, usually independent systems working together as one. Though some of the most interesting opportunities are still only loosely imagined, perhaps because they cause problems that are fiendishly hard to solve (e.g. how can we integrate systems that we build ourselves without creating risks for the rest of the ILE, and what happens when they need to be maintained?) a lot of progress is being made on the non-teaching foundations on which the rest depends (student admin systems, support tools, procedures, etc), as well as on the most visible and perhaps the biggest of its parts, BrightSpace, a proprietary commercial LMS that is meant to replace Moodle, for no obvious pedagogical or technical reasons (it’s no better). It might make economic sense. I don’t know, but I do know that open source software typically costs a fair bit to own, albeit because of the things that make it a much better idea (freedom, flexibility, ownership, etc). There is probably a fair bit of time and money being spent with Desire2Learn (makers of Brightspace) on the things that we spent a fair bit of time and money on many years ago to make Moodle a bit less classroom-like. The choice no doubt has something to do with how reliably and easily it can be made to work with some of the other proprietary commercial systems that someone has decided will make up the ILE. It bothers me greatly that we are not trying hard to choose open source solutions, for reasons that will become clearer in the rest of this post. However, (pedagogically speaking) all the mainstream LMSs are much of a muchness, making the same mistakes as one another in very similar ways, so it probably won’t wreck too much of what we already do within Moodle. But, on its own, it won’t move us much further forward and we could do it better. That’s what the ILE is supposed to do – to make the LMS just a part of a much larger teaching environment, intimately connected with the rest of what the university does for or with students, and extensible with new and better ways of learning, teaching, and assessing learning.
Lego bricks make poor metaphors
When we were first imagining the ILE, though the approach was admirably participative, engaging much of the university community, I was very worried by the things we were encouraged to focus on. It was all about the functionality, the usability, the design, the tools, the pedagogies, the business systems that supported them. Those things matter, for sure, and should be not be ignored, but they should and will change and grow all the time: in fact, part of the point of building this thing is to do just that. Using the city metaphor, pretty much all that we (collectively) considered were the spaces (the rooms, mainly), and the stuff that goes on inside them, much like LMS designers thought of universities as just collections of classrooms in which teaching functions were performed. Space and stuff are, not uncoincidentally, exactly what Stewart Brand identified long ago as inevitably being the fastest-changing, most volatile parts of any town or city (after site, structure, skin, and services). I’ve written a fair bit on the universality of this principle across all systems. It’s a solid structural principle that applies as much to ecosystems and educational systems as to cities. As Brand observes himself, drawing from O’Neill et al (1986), the larger, slower-changing elements of any system affect the smaller, faster-changing more than vice versa. This is for much the same reasons that path dependencies set in. It’s about the prior providing the context for what follows. Flexible things have to fit into the gaps left by less flexible, older, pre-existing things. In physical spaces, of course these tend to be bigger and/or slower, but the same is true in virtual spaces, where size seldom matters that much, but hardness (inflexibility, brittleness) really does. Though lip service was paid to the word ‘integrated’ in our discussions, I had the strong feeling that the kind of integration we had in mind was that of a Lego set. In fact, I think we were aiming to find a ‘Lego Athabasca University’ set, with assembly instructions and a picture on the box. The vendors who came to talk with us made much of how effectively they could do that, rather than how effectively they could make it possible for others to do that.
Metaphors matter. Lego bricks have to fit together tightly, in pre-specified ways, especially if you are following a plan. If you want to move them around, you have to dismantle a bit of the structure to fit them in. It’s difficult to integrate things that are not bricks, or that are made by different toy companies to work in different ways. At best you get what Brand calls ‘magazine architecture’, or ‘no road’ architecture, beautiful, fit for purpose, intricate and solid, but slow to learn. Lego is not a terrible way to build, compared with buying everything pre-assembled, but it could be improved.
Signals and boundaries
Drawing inspiration from John Holland’s brilliant last work, Signals & Boundaries, I tried to make the case that, instead, we should be focusing on the boundaries (the interfaces between the buildings and the rest of the city), and the signals that pass between them (the people, the messages, etc, the forms they take and how they move around). In Brand’s terms, I wanted us to be thinking about skin and services, and perhaps even structure, though site – Athabasca University – was a given. Though a few people nodded in agreement, I think it mainly fell on deaf ears. We wanted oven-ready solutions, not the infrastructure to enable those solutions. Though the city metaphor works well, because we are talking about human constructions, others would result in similar ways of thinking: cells in bodies, organisms in ecosystems, brains, termite mounds, and so on. All are organized by boundaries (at many levels of hierarchy) and the signals that pass between them.
The Lego set metaphor – whether deliberately or not – seems to have prevailed for now. A lot of old buildings are being slated for demolition and a lot of new virtual buildings are now being erected as part of this development, many of them chosen not because of problems with existing buildings but so that they can more easily connect together and live in the same cloud. This will very likely work, for now, but it is not cheap and it is not flexible, especially given the fact that most of it is not open so, like a rental property, we are not allowed to fix things, add utilities, change the walls, etc, and we are wholly dependent on the landlords being nice to us and each other (knowing that some – ahem, Microsoft – have a long history of abusing their tenants). Those buildings will age. We will find them cramped. Some will age faster than others, and will have to be modified to keep up, perhaps at high cost. Companies renting them might go out of business or change their terms so we might have to demolish the buildings and rent/make new ones. We will be annoyed at how they do things, usually without asking us. We will hate the landlords who dictate what we can do and how we can do it, and who will keep upping the rent while not doing what we ask. We will want more, and the only way to get it will be to build extensions, buy new brick sets, if it is not enough to pay someone to remodel the interiors (and it won’t be). Of course, because most of the big structural elements will not be open source, we will not be able to do that ourselves.
What the ILE really should be
The ILE is, I think, poorly named, because it should not be an environment at all. Following the building metaphor, the ILE is (or should be) more like the system that connects a lot of buildings, bringing them together into a coherent, safe, livable community. It’s infrastructure and services; it is the roads, the traffic signals, the doors, the sidewalks, the water pipes, the waste pipes, the electricity, the network cables; it is the services – fire, police, schools, traffic control, etc; it is all the many rules, standards, norms and regulations that make them work together to help make an environment in which people can live, work, play, and grow. It’s part of the environment – the part that makes it work – but it is not the environment itself. The environment itself is Athabasca University, not just the tools, processes, and systems that support its functions. That includes, most importantly, the people who are part of the university, or who are visitors to it, who are not just users of the environment or dwellers in its walls, but who are or should be the most significant and visible parts of it, just as trees are part of the environment of forests, not users of the forest. Those people live in physical as well as other virtual environments (social media, Word documents, websites, etc) that the ILE can connect together too, to make them a part of it, so the spatial metaphor gets weird at this point. The ILE makes environmental boundaries fuzzy, permeable, and shifting. It’s not an ILE, it’s an ILI – an integrated learning infrastructure.
If we focused on the connections and interfaces, and on how information and processes need to pass across them, and if we thought hard about the nature of those signals, then we could build a system that is resilient, that adapts, that lasts, that grows, that evolves, with parts that we can seamless replace or improve because the interfaces – the building facades, the mains pipes, the junction boxes, etc – will mostly stay the same, evolving slowly as they should. This is about strategy, not planning, a way of thinking about systems rather than a sequence of things to do.
Some of the key people involved in the process realize this. They are talking about standards, protocols, and projects to build interfaces between systems, and imagining future needs, though they are inevitably distracted by the process of renting Lego bricks, so I am not sure how much they will be able to stay focused on that. I hope they prevail over those who think they are building a set of classrooms and tightly connected admin offices out of self-contained interlocking bricks because our future depends on getting it right. We are aiming to grow. It just takes one critical piece in the Lego building to fail to support that, and the rest falls apart like a… well, like a pile of bricks.
Brand, S. (1997). How buildings learn. Phoenix Illustrated. https://www.penguinrandomhouse.ca/books/320919/how-buildings-learn-by-stewart-brand/9780140139969
Holland, J. H. (2012). Signals and Boundaries: Building Blocks for Complex Adaptive Systems. MIT Press. https://mitpress.mit.edu/books/signals-and-boundaries
O’Neill, R.V., DeAngelis, D.L, Waide, J. B., & Allen, T. F. H. (1986). A Hierarchical Concept of Ecosystems. Princeton University Press. http://www.gbv.de/dms/bs/toc/025157787.pdf
Postman, N. (1998). Five things we need to know about technological change. Denver, Colorado, 28. https://student.cs.uwaterloo.ca/~cs492/papers/neil-postman–five-things.html
These are the slides from my keynote today at the Oxford Brookes “Theorizing the Virtual” School of Education Research Conference. As theorizing the virtual is pretty much my thing, I was keen to be a part of this! It was an ungodly hour of the day for me (2am kickoff) but it was worth staying up for. It was a great bunch of attendees who really got into the spirit of the thing and kept me wide awake. I wish I could hang around for the rest of it but, on the bright side, at least I’m up at the right time to see the Super Flower Blood Moon (though it’s looking cloudy, darn it). In this talk I dwelt on a few of the notable differences between online and in-person teaching. This is the abstract…
Pedagogical methods (ways of teaching) are solutions to problems of helping people to learn, in a context filled with economic, physical, temporal, legal, moral, social, political, technological, and organizational constraints. In mediaeval times books were rare and unaffordable, and experts’ time was precious and limited, so lectures were a pragmatic solution, but they in turn created more problems. Counter-technologies such as classes, classrooms, behavioural rules and norms, courses, terms, curricula, timetables and assignment deadlines were were devised to solve those problems, then methods of teaching (pedagogies) were in turn invented to solve problems these counter-technologies caused, notably including:
· people who might not want (or be able) to be there at that time,
· people who were bored and
· people who were confused.
Better pedagogies supported learner needs for autonomy and competence, or helped learners find relevance to their own goals, values, and interests. They exploited physical closeness for support, role-modelling, inspiration, belongingness and so on. However, increasingly many relied on extrinsic motivators, like classroom discipline, grades and credentials to coerce students to learn. Extrinsic motivation achieves compliance, but it makes the reward or avoidance of the punishment the goal, persistently and often permanently crowding out intrinsic motivation. Intelligent students respond with instrumental approaches, satisficing, or cheating. Learning seldom persists; love of the subject is subdued; learners learn to learn in ineffective ways. More layers of counter-technologies are needed to limit the damage, and so it goes on.
Online, the constraints are very different, and its native forms are the motivational inverse of in-person learning. An online teacher cannot control every moment of a learner’s time, and learners can use the freedoms they gain to take the time they need, when they need it, to learn and to reflect, without the constraints of scheduled classroom hours and deadlines. However, more effort is usually needed to support their needs for relatedness. Unfortunately, many online teachers try (or are required) to re-establish the control they had in the classroom through grading or the promise of credentials, recreating the mediaeval problems that would otherwise not exist, using tools like learning management systems that were designed (poorly) to replicate in-person teaching functions. These are solutions to the problems caused by counter-technologies, not to problems of learning.
There are better ways, and that’s what this session is about.
This is a long paper (about 10,000 words), that summarizes some of the central elements of the theoretical model of learning, teaching and technology developed in my recently submitted book (still awaiting review) and that gives a few examples of its application. For instance, it explains:
why, on average researchers find no significant difference between learning with and without tech.
why learning styles theories are a) inherently unprovable, b) not important even if they were, and c) a really bad idea in any case.
why bad teaching sometimes works (and, conversely, why good teaching sometimes fails)
why replication studies cannot be done for most educational interventions (and, for the small subset that are susceptible to reductive study, all you can prove is that your technology works as intended, not whether it does anything useful).
This theoretical paper elucidates the nature of educational technology and, in the process, sheds light on a number of phenomena in educational systems, from the no-significant-difference phenomenon to the singular lack of replication in studies of educational technologies. Its central thesis is that we are not just users of technologies but coparticipants in them. Our participant roles may range from pressing power switches to designing digital learning systems to performing calculations in our heads. Some technologies may demand our participation only in order to enact fixed, predesigned orchestrations correctly. Other technologies leave gaps that we can or must fill with novel orchestrations, that we may perform more or less well. Most are a mix of the two, and the mix varies according to context, participant, and use. This participative orchestration is highly distributed: in educational systems, coparticipants include the learner, the teacher, and many others, from textbook authors to LMS programmers, as well as the tools and methods they use and create. From this perspective, all learners and teachers are educational technologists. The technologies of education are seen to be deeply, fundamentally, and irreducibly human, complex, situated and social in their constitution, their form, and their purpose, and as ungeneralizable in their effects as the choice of paintbrush is to the production of great art.
Originally posted at: https://landing.athabascau.ca/bookmarks/view/8692242/my-latest-paper-educational-technology-what-it-is-and-how-it-works