Here are my slides from my presentation at the Innovate Learning Summit yesterday. It’s not world-shattering stuff – just a brutal attack on proctored, unseen written exams (PUWEs, pronounced ‘pooies’), followed by a description of the rationale, process, benefits, and unwanted consequences behind the particular portfolio-based approach to assessment employed in most of my teaching. It includes a set of constraints that I think are important to consider in any assessment process, grouped into pedagogical, motivational, and housekeeping (mainly relating to credentials) clusters. I list 13 benefits of my approach relating to each of those clusters, which I think make a pretty resounding case for using it instead of traditional assignments and tests. However, I also discuss outstanding issues, most of which relate to the external context and expectations of students or the institution, but a couple of which are fairly fundamental flaws (notably the extreme importance of prompt, caring, helpful instructor/tutor engagement in making it all work, which can be highly problematic when it doesn’t happen) that I am still struggling with.
A group of us at AU have begun discussions about how we might transform our assessment practices, in the light of the far-reaching AU Imagine plan and principles. This is a rare and exciting opportunity to bring about radical and positive change in how learning happens at the institution. Hard technologies influence soft more than vice versa, and assessments (particularly when tied to credentials) tend to be among the hardest of all technologies in any pedagogical intervention. They are therefore a powerful lever for change. Equally, and for the same reasons, they are too often the large, slow, structural elements that infest systems to stunt progress and innovation.
Almost all learning must involve assessment, whether it be of one’s own learning, or provided by other people or machines. Even babies constantly assess their own learning. Reflection is assessment. It is completely natural and it only gets weird when we treat it as a summative judgment, especially when we add grades or credentials to the process, thus normally changing the purpose of learning from achieving competence to achieving a reward. At best it distorts learning, making it seem like a chore rather than a delight, at worst it destroys it, even (and perhaps especially) when learners successfully comply with the demands of assessors and get a good grade. Unfortunately, that’s how most educational systems are structured, so the big challenge to all teachers must be to eliminate or at least to massively reduce this deeply pernicious effect. A large number of the pedagogies that we most value are designed to solve problems that are directly caused by credentials. These pedagogies include assessment practices themselves.
With that in mind, before the group’s first meeting I compiled a list of some of the main principles that I adhere to when designing assessments, most of which are designed to reduce or eliminate the structural failings of educational systems. The meeting caused me to reflect a bit more. This is the result:
Principles applying to all assessments
- The primary purpose of assessment is to help the learner to improve their learning. All assessment should be formative.
- Assessment without feedback (teacher, peer, machine, self) is
judgement, not assessment, pointless.
- Ideally, feedback should be direct and immediate or, at least, as prompt as possible.
- Feedback should only ever relate to what has been done, never the doer.
- No criticism should ever be made without also at least outlining steps that might be taken to improve on it.
- Grades (with some very rare minor exceptions where the grade is intrinsic to the activity, such as some gaming scenarios or, arguably, objective single-answer quizzes with T/F answers) are not feedback.
- Assessment should never ever be used to reward or punish particular prior learning behaviours (e.g. use of exams to encourage revision, grades as goals, marks for participation, etc) .
- Students should be able to choose how, when and on what they are assessed.
- Where possible, students should participate in the assessment of themselves and others.
- Assessment should help the teacher to understand the needs, interests, skills, and gaps in knowledge of their students, and should be used to help to improve teaching.
- Assessment is a way to show learners that we care about their learning.
Specific principles for summative assessments
A secondary (and always secondary) purpose of assessment is to provide evidence for credentials. This is normally described as summative assessment, implying that it assesses a state of accomplishment when learning has ended. That is a completely ridiculous idea. Learning doesn’t end. Human learning is not in any meaningful way like programming a computer or storing stuff in a database. Knowledge and skills are active, ever-transforming, forever actively renewed, reframed, modified, and extended. They are things we do, not things we have.
With that in mind, here are my principles for assessment for credentials (none of which supersede or override any of the above core principles for assessment, which always apply):
- There should be no assessment task that is not in itself a positive learning activity. Anything else is at best inefficient, at worst punitive/extrinsically rewarding.
- Assessment for credentials must be fairly applied to all students.
- Credentials should never be based on comparisons between students (norm-referenced assessment is always, unequivocally, and unredeemably wrong).
- The criteria for achieving a credential should be clear to the learner and other interested parties (such as employers or other institutions), ideally before it happens, though this should not forestall the achievement and consideration of other valuable outcomes.
- There is no such thing as failure, only unfinished learning. Credentials should only celebrate success, not punish current inability to succeed.
- Students should be able to choose when they are ready to be assessed, and should be able to keep trying until they succeed.
- Credentials should be based on evidence of competence and nothing else.
- It should be impossible to compromise an assessment by revealing either the assessment or solutions to it.
- There should be at least two ways to demonstrate competence, ideally more. Students should only have to prove it once (though may do so in many ways and many times, if they wish).
- More than one person should be involved in judging competence (at least as an option, and/or on a regularly taken sample).
- Students should have at least some say in how, when, and where they are assessed.
- Where possible (accepting potential issues with professional accreditation, credit transfer, etc) they should have some say over the competencies that are assessed, in weighting and/or outcome.
- Grades and marks should be avoided except where mandated elsewhere. Even then, all passes should be treated as an ‘A’ because students should be able to keep trying until they excel.
- Great success may sometimes be worthy of an award – e.g. a distinction – but such an award should never be treated as a reward.
- Assessment for credentials should demonstrate the ability to apply learning in an authentic context. There may be many such contexts.
- Ideally, assessment for credentials should be decoupled from the main teaching process, because of risks of bias, the potential issues of teaching to the test (regardless of individual needs, interests and capabilities) and the dangers to motivation of the assessment crowding out the learning. However, these risks are much lower if all the above principles are taken on board.
I have most likely missed a few important issues, and there is a bit of redundancy in all this, but this is a work in progress. I think it covers the main points.
Further random reflections
There are some overriding principles and implied specifics in all of this. For instance, respect for diversity, accessibility, respect for individuals, and recognition of student control all fall out of or underpin these principles. It implies that we should recognize success, even when it is not the success we expected, so outcome harvesting makes far more sense than measurement of planned outcomes. It implies that failure should only ever be seen as unfinished learning, not as a summative judgment of terminal competence, so appreciative inquiry is far better than negative critique. It implies flexibility in all aspects of the activity. It implies, above and beyond any other purpose, that the focus should always be on learning. If assessment for credentials adversely affects learning then it should be changed at once.
In terms of implementation, while objective quizzes and their cousins can play a useful formative role in helping students to self-assess and to build confidence, machines (whether implemented by computers or rule-following humans) should normally be kept out of credentialling. There’s a place for AI but only when it augments and informs human intelligence, never when it behaves autonomously. Written exams and their ilk should be avoided, unless they conform to or do not conflict with all the above principles: I have found very few examples like this in the real world, though some practical demonstrations of competence in an authentic setting (e.g. lab work and reporting) and some reflective exercises on prior work can be effective.
A portfolio of evidence, including a reflective commentary, is usually going to be the backbone of any fair, humane, effective assessment: something that lets students highlight successes (whether planned or not), that helps them to consolidate what they have learned, and that is flexible enough to demonstrate competence shown in any number of ways. Outputs or observations of authentic activities are going to be important contributors to that. My personal preference in summative assessments is to only use the intended (including student-generated) and/or harvested outcomes for judging success, not for mandated assignments. This gives flexibility, it works for every subject, and it provides unquivocal and precise evidence of success. It’s also often good to talk with students, perhaps formally (e.g. a presentation or oral exam), in order to tease out what they really know and to give instant feedback. It is worth noting that, unlike written exams and their ilk, such methods are actually fun for all concerned, albeit that the pleasure comes from solving problems and overcoming challenges, so it is seldom easy.
Interestingly, there are occasions in traditional academia where these principles are, for the most part, already widely applied. A typical doctoral thesis/dissertation, for example, is often quite close to it (especially in more modern professional forms that put more emphasis on recording the process), as are some student projects. We know that such things are a really good idea, and lead to far richer, more persistent, more fulfilling learning for everyone. We do not do them ubiquitously for reasons of cost and time. It does take a long time to assess something like this well, and it can take more time during the rest of the teaching process thanks to the personalization (real personalization, not the teacher-imposed form popularized by learning analytics aficionados) and extra care that it implies. It is an efficient use of our time, though, because of its active contribution to learning, unlike a great many traditional assessment methods like teacher-set assignments (minimal contribution) and exams (negative contribution). A lot of the reason for our reticence, though, is the typical university’s schedule and class timetabling, which makes everything pile on at once in an intolerable avalanche of submissions. If we really take autonomy and flexibility on board, it doesn’t have to be that way. If students submit work when it is ready to be submitted, if they are not all working in lock-step, and if it is a work of love rather than compliance, then assessment is often a positively pleasurable task and is naturally staggered. Yes, it probably costs a bit more time in the end (though there are plenty of ways to mitigate that, from peer groups to pedagogical design) but every part of it is dedicated to learning, and the results are much better for everyone.
Some useful further reading
This is a fairly random selection of sources that relate to the principles above in one way or another. I have definitely missed a lot. Sorry for any missing URLs or paywalled articles: you may be able to find downloadable online versions somewhere.
Boud, D., & Falchikov, N. (2006). Aligning assessment with long-term learning. Assessment & Evaluation in Higher Education, 31(4), 399-413. Retrieved from https://www.jhsph.edu/departments/population-family-and-reproductive-health/_docs/teaching-resources/cla-01-aligning-assessment-with-long-term-learning.pdf
Boud, D. (2007). Reframing assessment as if learning were important. Retrieved from https://www.researchgate.net/publication/305060897_Reframing_assessment_as_if_learning_were_important
Cooperrider, D. L., & Srivastva, S. (1987). Appreciative inquiry in organizational life. Research in organizational change and development, 1, 129-169.
Deci, E. L., Vallerand, R. J., Pelletier, L. G., & Ryan, R. M. (1991). Motivation and education: The self-determination perspective. Educational Psychologist, 26(3/4), 325-346.
Hussey, T., & Smith, P. (2002). The trouble with learning outcomes. Active Learning in Higher Education, 3(3), 220-233.
Kohn, A. (1999). Punished by rewards: The trouble with gold stars, incentive plans, A’s, praise, and other bribes (Kindle ed.). Mariner Books. (this one is worth forking out money for).
Kohn, A. (2011). The case against grades. Educational Leadership, 69(3), 28-33.
Kohn, A. (2015). Four Reasons to Worry About “Personalized Learning”. Retrieved from http://www.alfiekohn.org/blogs/personalized/ (check out Alfie Kohn’s whole site for plentiful other papers and articles – consistently excellent).
Reeve, J. (2002). Self-determination theory applied to educational settings. In E. L. Deci & R. M. Ryan (Eds.), Handbook of Self-Determination research (pp. 183-203). Rochester, NY: The University of Rochester Press.
Ryan, R. M., & Deci, E. L. (2017). Self-determination theory: Basic psychological needs in motivation, development, and wellness. Guilford Publications. (may be worth paying for if such things interest you).
Wilson-Grau, R., & Britt, H. (2012). Outcome harvesting. Cairo: Ford Foundation. http://www.managingforimpact.org/sites/default/files/resource/outome_harvesting_brief_final_2012-05-2-1.pdf.
This is a slide deck for a talk I’m giving today, at a faculty workshop, on the subject of learning outcomes.
I think that well-considered learning outcomes can be really helpful when planning and designing learning activities, especially where there is a need to assess learning. They can help keep a learning designer focused, and to remember to ensure that assessment activities actually make a positive contribution to learning. They can also be helpful to teachers while teaching, as a framework to keep them on track (if they wish to remain on track). However, that’s about it. Learning outcomes are not useful when applied to bureaucratic ends, they are very poor descriptors of what learning actually happens, as a rule, and they are of very little (if any) use to students under most circumstances (there are exceptions – it’s a design issue, not a logical flaw).
The big point of my talk, though, is that we should be measuring what students have actually learned, not whether they have learned what we think we have taught, and that the purpose of everything we do should be to support learning, not to support bureaucracy.
I frame this in terms of the relationships between:
- what we teach (what we actually teach, not just what we think we are teaching, including stuff like attitudes, beliefs, methods of teaching, etc),
- what a student learns in the process (an individual student, not students as a whole), and
- what we assess (formally and summatively, not necessarily as part of the learning process).
There are many things that we teach that any given student will not learn, albeit that (arguably) we wouldn’t be teaching at all if learning were not happening for someone. Most students get a small subset of that. There are also many things that we teach without intentionally teaching, not all of them good or useful.
There are also very many things that students learn that we do not teach, intentionally or otherwise. In fact, it is normal for us to mandate this as part of a learning design: any mildly creative or problem-solving/inquiry-oriented activity will lead to different learning outcomes for every learner. Even in the most horribly regimented teaching contexts, students are the ones that connect everything together, and that’s always going to include a lot more than what their teachers teach.
Similarly, there are lots of things that we assess that we do not teach, even with great constructive alignment. For example, the students’ ability to string a sentence together tends to be not just a prerequisite but something that is actively graded in typical assessments.
My main points are that, though it is good to have a teaching plan (albeit that it should be flexible, reponsive to student needs, and should accommodate serendipity)learning :
- students should be participants in planning outcomes and
- we should assess what students actually learn, not what we think we are teaching.
From a learning perspective, there’s less than no point in summatively judging what learners have not learned. However, that’s exactly what most institutions actually do. Assessment should be about how learners have positively changed, not whether they have met our demands.
This also implies that students should be participants in the planning and use of learning outcomes: they should be able to personalize their learning, and we should recognize their needs and interests. I use andragogy to frame this, because it is relatively uncontroversial, is easily understood, and doesn’t require people to change everything in their world view to become better teachers, but I could have equally used quite a large number of other models. Connectivism, Communities of Practice, and most constructivist theories, for instance, force us to similar conclusions.
I suggest that appreciative inquiry may be useful as an approach to assessment, inasmuch as the research methodology is purpose-built to bring about positive change, and its focus on success rather than failure makes sense in a learning context.
I also suggest the use of outcome mapping (and its close cousin, outcome harvesting) as a means of capturing unplanned as well as planned outcomes. I like these methods because they only look at changes, and then try to find out what led to those changes. Again, it’s about evaluation rather than judgment.
The teaching gestalt presentation slides (PDF, 9MB)
This is my Spotlight Session from the 34th Distance Teaching & Learning Conference, at Wisconsin Madison, August 8th, 2018. Appropriately enough, I did this online and at a distance thanks to my ineptitude at dealing with the bureaucracy of immigration. Unfortunately my audio died as we moved to the Q&A session so, if anyone who was there (or anyone else) has any questions or observations, do please post them here! Comments are moderated.
The talk was concerned with how online learning is fundamentally different from in-person learning, and what that means for how (or even whether) we teach, in the traditional formal sense of the word.
Teaching is always a gestalt process, an emergent consequence of the actions of many teachers, including most notably the learners themselves, which is always greater than (and notably different from) the sum of its parts. This deeply distributed process is often masked by the inevitable (thanks to physics in traditional classrooms) dominance of an individual teacher in the process. Online, the mask falls off. Learners invariably have both far greater control and far more connection with the distributed gestalt. This is great, unless institutional teachers fight against it with rewards and punishments, in a pointless and counter-productive effort to try to sustain the level of control that is almost effortlessly attained by traditional in-person teachers, and that is purely a consequence of solving problems caused by physical classroom needs, not of the needs of learners. I describe some of the ways that we deal with the inherent weaknesses of in-person teaching especially relating to autonomy and competence support, and observe how such pedagogical methods are a solution to problems caused by the contingent side effects of in person teaching, not to learning in general.
The talk concludes with some broad characterization of what is different when teachers choose to let go of that control. I observe that what might have been Leonardo da Vinci’s greatest creation was his effective learning process, without which none of the rest of his creations could have happened. I am hopeful that now, thanks to the connected world that we live in, we can all learn like Leonardo, if and only if teachers can learn to let go.
Figure 1: Skinner’s teaching machine
It is not much of a surprise that many apps are designed to be addictive, nor that there is a whole discipline behind making them so, but I was particularly interested in the delightfully named Dopamine Labs‘ use of behaviourist techniques (operant conditioning with variable ratio scheduling, I think), and the reasoning behind it. As the article puts it:
One of the most popular techniques … is called variable reinforcement or variable rewards.
It involves three steps: a trigger, an action and a reward.
A push notification, such as a message that someone has commented on your Facebook photo, is a trigger; opening the app is the action; and the reward could be a “like” or a “share” of a message you posted.
These rewards trigger the release of dopamine in the brain, making the user feel happy, possibly even euphoric, Brown says.
“Just by controlling when and how you give people that little burst of dopamine, you can get them to go from using [the app] a couple times a week to using it dozens of times a week.”
For well-designed social media and games, the reward is intrinsic to the activity, and perfectly aligned with its function. If the intent is to create addicts – which, in both kinds of system, it probably is – the trick is to design an environment that builds rewards into the algorithms (the rules) of the system, and to keep them coming, ideally making it possible for the rewards to increase in intensity as the user gains greater expertise or experience, but varying ratios or intervals between rewards to keep things interesting. Though this particular example falls out from behaviourist theory, it is also well supported by cognitivist and brain-based understandings of how we think. Drug dealers know this too, as it happens. If you want to keep people using your product, this is how to make your product particularly addictive.
Lovers of learning experience addiction too. The more we learn, the more there is to learn, the greater the depth and pleasure there is to be found in doing so, and the sporadic ups and downs, especially when faced with challenges we eventually solve, are part of the joy of it. Increasing mastery of anything is a reward in itself that seems quite intrinsic to our make-up, and to that of many other animals. Doing it in a social context is even better, as we share in the learning of others and gain value (social capital, different perspectives, help overcoming problems, etc) in the process. We gain greater control, greater autonomy, greater capability to live our lives as we want to live them, which is very motivating. As long as the reward comes from the activity itself, and the activity is not harmful, this is good news. It makes sense from an evolutionary perspective. We are innately motivated to learn, because learning is an extremely valuable survival characteristic. Learning generally makes dopamine positively drip from our eyeballs.
So what’s the problem with applying the principle in education?
None at all, until you hit something that you do not wish to learn, that is too difficult to master right now, that is too boring, that has no obvious rewards in and of itself. The correct response to this problem is, ideally, to find what there is to love in it. Good teachers can help with that a lot, inspiring, revealing, supporting, demonstrating, and discussing. Other learners can make a huge difference too, supporting, modelling behaviours, filling gaps, and so on. We very often learn things for other people, with other people, or because of other people. Educational systems offer a good substrate for that.
If intrinsic motivation fails to move us, then at least the motivation should be self-determined. Figure 2 shows a very successful and well-validated model of motivation (from Ryan and Deci) that, amongst other things, usefully describes differing degrees of extrinsic motivation (external, introjected, identified, and integrated) that, as they approach the right of the diagram, increasingly approach intrinsic motivation in value, though ‘external regulation’ is rather different, of which more soon. When intrinsic motivation fails, what we need is some kind of internal regulation to push us onwards. It is not a bad idea to find some internally regulated reason that aligns with your beliefs about yourself and your goals, or that at least fits with some purpose or goal that you find valuable. It’s sometimes useful to develop a bit of ‘grit‘ – to be able to do something that you don’t love doing in order to be able to do things that you do love doing, to find reasons for learning stuff that are meaningful and fit with your personal values, even if the immediately presenting activity is not fun in itself. Again, teachers and other people can help a lot with that, by showing ways that they are doing so themselves, by providing support, by engaging, or by being the reason that we do something in the first place. It’s all very social, at its heart.
Figure 2: Forms of motivation
That social element is important, and not clearly represented in the diagram, despite being a critical aspect of intrinsic motivation and mattering a lot for the ‘higher’ identified forms of extrinsic motivation. From an evolutionary perspective, I suspect this ability to learn because of the presence of others accounts for our species’ apparent dominance in our ecosystems. We are not particularly clever as independent individuals but, collectively, we are mighty smart. This could not be the case without having an innate inclination to value, and to gain value from, other people, and for this to have the consequence that others very materially contribute towards our motivation to do something. I guess I should mention that ‘innate’ does not mean ‘pre-programmed’ – this is almost certainly an emergent phenomenon. But it is a big part of who we are.
So far so good. Educational systems are, at least in principle, very effective ways of bringing people together. It all goes horribly wrong, however, when the educators’ response to amotivation (or worse, to motivation to avoid) is to change the rules by throwing in extrinsic rewards and punishments, like grades, say, or applying other controls to the process like forced attendance. Externally regulated extrinsic motivation is extremely dangerous.
Extrinsic rewards and punishments do work, in the sense that they coerce people and other animals into behaving as the giver of the rewards or punishments wishes them to behave. And yes, dopamine is implicated. This immediate effectiveness is what makes them so alluring. But it’s like giving an athlete performance-enhancing but ultimately harmful drugs. Rewards and punishments are also highly addictive and, like other addictions, you need more and more to sustain your addiction because you become inured to the effects, and withdrawal gets more painful the longer you are addicted. This works two ways. Those that get the rewards (the good grades, gold stars, praise, whatever) go on to want more of them, and will do what they need to get them, whether or not there are any further benefits (like, say, learning). Cheating is one popular way to do this. Tactical study, where the student tries to do what will get good grades rather than learn for the love of it, is another. But grading, though extrinsically motivating for the most part, is not always effective: bad grades can achieve the opposite effect, like drugs spiked with something horrible. Those that get grades as punishments often try to avoid them by whatever means they can: dropping out and cheating (a way to bypass the system to get hold of the good stuff) are popular solutions.
The biggest problems, however, come when you take the rewards/punishments away. As a vast body of research has shown and continues to show, this diminishes intrinsic motivation and often eliminates it altogether. If people are not very inclined to do something then you can temporarily boost interest by adding extrinsic rewards or punishments but, when you take them away, people are considerably less inclined to do the thing than they were before your started even when they originally liked to do it. At a high level this can be explained by the fact that, in giving a reward or punishment, you are drawing attention away from (crowding out) the thing itself and, at the same time, sending a strong signal that the activity itself is not rewarding enough in itself to be worth doing. But I am not sure that this fully explains the very strong negative effects on motivation that we actually see when rewards or punishments are withdrawn. I idly speculate that part of the reason for this effect might be the dopamine crash. We come to associate an activity with a dopamine boost and, when that boost is no longer forthcoming, it can be very disappointing, like smoking a nicotine-free cigarette (trust me – that’s awful). Cold turkey is not the best state to be in, especially when you associate it with an activity like learning something. It could really put you off a subject. This is just a thought: I know of no evidence that it is true, but it seems a plausible hypothesis that would be worth testing.
Whatever the cause, the effects are terrible. By extrinsically driving our students, we kill the love of the activity itself for those that might have loved it, and permanently prevent those that might have later found it valuable from ever wanting to do it again. Remarkably few survive unscathed, and a disproportionate number of those that do go on to become teachers, and so the cycle continues. I don’t think this is how education should be, and I don’t think it is what most of us in the system intend from it.
Getting out of the loop
The only really effective way to ensure lifelong interest and ongoing love of learning is to find the reward in the activity itself, not in an extrinsic reward. The games and social applications described in this article do that very well but it is important to remember that the intent of the designers of the applications is to increase addiction to them in order to sell or promote the product, and that there is perfect alignment between the reward and the activity itself. This is built into the rule system. In an education system that is driven by marks, we are making grades (not learning) the product, and making those the source of the addiction. This is very different. It has nothing to do with the activity of learning itself: it is extrinsic to the process. It might be even more effective give our students addictive drugs (higher concentrations equate to higher grades) to increase the incentive. I’m surprised no one has tried this.
But, seriously, what we really need to be doing is to make learning the addiction.
We can reduce the harm to an extent by removing grades from the teaching process and focusing on useful feedback and encouragement instead. If forced to judge, we can use pass/fail grades that are still harmful but not quite as controlling. If we are inexplicably drawn to grading, then we can build systems similar to those of ‘likes’ and badges of social media where, instead of rewards we give awards – in other words, we remove the expectation of a grade but, where merit is found, sometimes show our approval – and we can make that a social process, so that it is not dominated by a teacher and therefore does not involve exercise of arbitrary power. We can use pedagogies that give teachers and students the chance to model and demonstrate their passion and interest. We can encourage students to reflect on why they are doing it, ideally shared so they can gain inspiration from others. We can help students to integrate work with other things that matter to them. We can help them personalize their own learning so that it is appropriately challenging, not too dull, not to hard, and so that it matches the goals they set for themselves. We can help them to set those goals, and help them to figure out how to attain them. We can make them participants in the grading process, picking outcomes and assessments that match their interests and needs. We can build communities that support and nourish learning through sharing and mutual support. This is just a small sample of ways – there are really quite a few things that we can do, even within a broken system, to make learning addictive, to find ways to make it rewarding in and of itself, even when there is little initial interest to build upon. But we are still stuck in a system that treats grades as rewards, so we are still faced with a furious current pushing against all of our efforts.
Really, we need to change the system, but just a bit: our current educational systems have evolved for pragmatic reasons, mainly because alternatives are too expensive or inconvenient for teachers to manage, not because they are any good for learners. One of the consequences of that is that it is almost impossible to run an institutional course or program without at least some form of grading, even if only at pass/fail level, even if only at the end.
An obvious big part of the solution is to decouple learning and grading. Some more advanced competency-based approaches already do that, as do things like challenge assessments and assessment of prior experience and learning, to some extent project/essay/thesis paths, outcomes-based programs, and even some kinds of professional exams (the latter not in a good way, for the most part, because they tend to drive the process). However, there are risks that universities might turn into an up-market version of driving schools, teaching how to pass the tests and doing just as they are doing now, rather than enabling more expansive learning as they should. To avoid that, it is critical that learners are involved in helping to determine their own personalized outcomes, and very much not to have those learning outcomes ‘personalized’ for them – personal, not personalized, as Alfie Kohn puts it and as Stephen Downes agrees. Grades that learners control, for activities that they choose to undertake, are many times better than grades that someone else imposes. It would also be a good idea either to split teaching activities into assemblable chunks, or into open narratives, without alignment with specific awards or qualifications. Students might build competences from smaller pieces – often from different sources – in order to seek a specific award, or might gain more than one award from a single learning narrative (or perhaps from a couple that overlap). It would be a very good idea to provide ways to mentor and help learners to seek appropriate paths, perhaps through personal tuition, and/or through automated help, and/or through membership of supportive communities (I am a fan of action learning sets for this kind of thing). Such mechanisms might also assist in the preparation of portfolios of evidence that would be an obvious way to manage the formal assessment process. I’m not in any way suggesting that we educators (especially for adult learners) should get rid of our accreditation role, merely that we should stop using it to drive our teaching and to enforce compliance in our students.
I think that such relatively small tweaks to how we teach and assess could have massive benefits further upstream. In one fell swoop it would change the focus of educational systems from grades to learning, and change the reward structure from extrinsic to intrinsic. Instead of building fixed-length courses with measurable outcomes that we the teachers control, we could create ecosystems for learning, where cooperation and collaboration would have greater value than competition, where learners are really part of a club, not a cohort, where teachers are perceived as enablers of learning, not as causes, and certainly not as judges. The words ‘learner-centred’ have been much over-used, often being a shorthand for ‘a friendlier way of making students comply with our demands’ or ‘helping students to get better grades’, but I think they fairly accurately denote what this sort of system would entail when taken seriously. Some of my friends and colleagues prefer ‘learning-centred’ and that works for me too. But really this is about being more human and more humane. It’s about breaking the machines that determine what we do and how we do it, and focusing instead on what we – collectively and individually – want to be. We can do this by thinking carefully about what motivates people, as opposed to attempting to motivate them. As soon as our attitude is one of ‘how can we make our students to this?’ rather than ‘how can we help our students to do this?’ we have failed. It’s easy to create addicts of extrinsic motivation. It is hard to make addicts of learning. But, sometimes, the hard way is the right way.
Address of the bookmark: http://www.cbc.ca/news/technology/marketplace-phones-1.4384876
Originally posted at: https://landing.athabascau.ca/bookmarks/view/2816324/addicted-to-learning-or-addicted-to-grades
We already know that extrinsically motivated students (mainly those driven by grades and testing) are far more likely to cheat than those who are more intrinsically motivated. I bookmarked yet another example of this effect just the other day but there are hundreds if not thousands of research papers that confirm this in many different ways. And, as this article reaffirms, we already know that mastery learning approaches (that focus on supporting control, appropriate levels of challenge, and, ideally, social engagement) tend to make cheating far less likely, because they tend to better support intrinsic motivation. Hardly anyone cheats if they are doing stuff they love to do, unless some strong extrinsic force overrides it (like grades, rewards, punishments, hard-to-meet deadlines, etc).
This research reveals another interesting facet of the problem that exactly accords with what self-determination theory would predict: that, whether or not the pedagogy is sensible (supportive of intrinsic motivation) or dumb (extrinsically driven), a student’s dislike of a course appears to predict an increased likelihood of cheating. This is pretty obvious when you think about it. If someone does not like a course then, by definition, they are not intrinsically motivated and, if they are still taking it despite that, the only motivation they can possibly have left is extrinsic.
The increased chances of cheating on disliked courses, whether or not mastery learning techniques are used, is completely unsurprising because it ain’t what you do, it’s the way that you do it. If mastery learning techniques are not working then it probably means that we are simply not using them very well. Most likely there is not enough support, or not enough learner control, or insufficient social engagement, or not enough/too much challenge, or there’s too much pressure, or something along those lines. It is actually much more difficult and usually far more time consuming to teach well using techniques that respect learner autonomy and individual needs than it is to follow the objectivist instructivist path, at least in an institutional environment that deeply embeds extrinsic motivation at its very core, so it is not surprising that it quite often fails. It is also very possible that the problem is almost entirely due to the surrounding educational ecosystem. For instance if it is one that forces students down institutionally-determined paths whether or not they are ready, whether or not it matters to them, or if not enough time is allowed for it, or if the stakes for failure are high, then even well-designed courses with enthusiastic, supportive, skilled, well-informed, compassionate, unpressured teachers are not likely to help that much.
Some people will take a pragmatic lesson from this to look more carefully for cheating on courses that they know to be disliked. That’s not the solution. Others will look at those courses and try to find ways to make them more likeable. That’s much better. But really, once we have done that, we need to be wondering about why anyone would be taking a course that they dislike in the first place. And that points to a central problem with our educational systems and the tightly coupled teaching and accreditation that they embed deep in their bones. Given enough time, support, and skilled tuition, almost anyone can learn almost anything, and love doing so. We live in a time of plenty, where there are usually countless resources, people, and methods to learn almost anything, in almost any practical way, so it makes no sense that people should still be forced to learn in ways that they dislike, at inappropriate times, and at an inappropriate pace. If they do, it is because (one way or another) we make them do so, and that’s the root of the problem. We – the educators and, above all, the educational system – are the cause of cheating, as much as we are the victims of it. And we are the ones that should fix it.
Originally posted at: https://landing.athabascau.ca/bookmarks/view/2762299/study-links-student-cheating-to-whether-a-course-is-popular-or-disliked
A straightforward and briefly reported study that supports the rather obvious hypothesis that quite young (15-month-old) children can and do learn from observing adults, at least in the short term. The twist here is that adults in the study were deliberately trying to model an attitude (grit) more than a distinct behaviour, in an attempt to teach the kids to do the same.
It is fair to say that the researchers demonstrated to the kids that persevering with problems after initially failure can lead to desirable results, and that the kids appeared to be more inclined to do the same after watching adults doing so: this accords well with the title of the paper. I’m not sure that the adults adequately demonstrated grit, though. I don’t know about you, but I actually enjoy solving problems and positively relish the failures that teach me how to succeed. In fact, in many situations (programming, for example) I deliberately make things fail in order to understand how they do so, and that’s part of the fun, even though (and partly because) I may curse and fume when the process fails to enlighten me. Same for many commercially available puzzles, from Rubik’s Cubes to letter-sliding games. Seems to me that grit involves more than doing something enjoyable on the way to achieving some anticipated goal that matters to us. It’s often about doing unenjoyable things, sometimes for goals we don’t even find particularly interesting or worthwhile, often over a prolonged period. That’s not what was happening here. This is interesting, though, if only to confirm that really quite young kids are able to see others as beings like themselves, and to transfer the lessons of stories that they construct about what they perceive others to be doing into actions they then take themselves.
The brief timeframe of the study means that it doesn’t show whether this is how grit is actually learned over time. The extent to which lessons persist depends on a great many things, including prior experience, repetition, who is repeating it, success in the short term, effectiveness of the attitude in overcoming meaningful challenges in the long term, social value of the attitude, current context, and counter-examples over time. Outside an experimental context we pick up attitudes and sentiments from kids as much as they do from us, from one another, and from the world at large. There are usually very many others around us who are all engaged in a rich reciprocal dance with us through which we collectively construct our various intersecting cultures and subcultures, including our attitudes and values. Also, life is seldom so neatly structured and categorized that a lesson can be so directly transferred from one context to another. At least, such cases are not the interesting ones. Though the experimenters tried to make the tasks a bit different, the study was really set up to highlight the similarities, and to lead to results that would please the children. In real life, we usually need to connect one situation with another that is quite different, separated by time, and to choose between competing strategies to deal with it, often with others around us that are adopting different approaches, all of which will influence us. Often, we are not even particularly interested in the outcomes. It’s much harder to do experiments that reflect that reality. In fact, it’s probably impossible, at least without adopting the ethical precepts of Josef Mengele. The researchers laudably note a range of other limitations, including cultural differences, beliefs of children about adults, task specific issues, and so on, and make no extravagant claims that it can be generalized further. Indeed it cannot.
That said, this is good evidence for something that I believe is not a bad idea: that teachers (formal or otherwise) should act as they hope their students will act. A very large part of the role of a teacher is to model how people in their field (or society at large, in the case of younger kids) think and behave, to enact and demonstrate their approaches and attitudes, perhaps more than to pass on the facts, skills, and technologies of their discipline, or to provide support for gaining such knowledge.
Bearing that in mind, while there is value in ‘grit’ and I don’t want to knock it too much, I think there are other attitudes that might matter a whole lot more, especially those that enable us to not just stick with stuff we don’t enjoy but to find pleasure and meaning in it. Passion is way more useful than grit, in the long run. Caring, too. Teachers that light fires in students’ hearts achieve way more than those that simply show them how to stick at things they hate.
Persistence, above and beyond IQ, is associated with long-term academic outcomes. To look at the effect of adult models on infants’ persistence, we conducted an experiment in which 15-month-olds were assigned to one of three conditions: an Effort condition in which they saw an adult try repeatedly, using various methods, to achieve each of two different goals; a No Effort condition in which the adult achieved the goals effortlessly; or a Baseline condition. Infants were then given a difficult, novel task. Across an initial study and two preregistered experiments (N = 262), infants in the Effort condition made more attempts to achieve the goal than did infants in the other conditions. Pedagogical cues modulated the effect. The results suggest that adult models causally affect infants’ persistence and that infants can generalize the value of persistence to novel tasks.
Address of the bookmark: http://science.sciencemag.org/content/357/6357/1290
The title of this Alphr article is a little misleading because the point the article rightly makes is that it all depends on the type of praise given. It reports on research from the University of Toronto that confirms (yet again) what should be obvious: praising learners for who they are (‘you’re so smart’) is a really bad idea, while praising what they do (‘you did that well’) is not normally a bad idea. The issue, though, is essentially one of intrinsic vs extrinsic motivation. By praising the person for being a particular way you are positioning that as the purpose, rather than a side-effect, of the activity, and positioning yourself as the arbiter, so disempowering the learner. By praising the behaviour, you are offering useful feedback on performance that empowers the recipient to choose whether and how to do such things again, as well as supporting needs for relatedness (it shows you care) and competence (it helps them improve). Both forms of praise contribute to feelings of self-esteem, but only one supports intrinsic motivation.
The nice twist in these particular studies (here and here) is that the researchers were looking at effects on morality. They found that ability praise (teling them they are smart) is very strongly correlated with a propensity to cheat. Exactly as theory would predict, kids who have been told that they are smart are significantly more likely to respond to the extrinsic motivation (the need to live up to expectations when given ability praise) by cheating, when given the opportunity. Interestingly, praising the behaviour (performance praise) has little or no effect on likelihood of cheating when compared with those given no praise at all: it is only when an expectation is set that the children are perceived as smart that cheating behaviour increases. It is also interesting, if tangential, that boys appeared to be way more likely to cheat than girls under all the conditions though, once primed by ability praise, girls were more likely to cheat than boys that had received no praise or performance praise.
The lesson is nothing like as simple as remembering to just praise the action, not the person. Praising behaviours can, when used badly, be just as disempowering as praising the person. For instance, while in some senses it might be possible to view grades as a kind of abbreviated praise (or punishment, which amounts to much the same thing) for a behaviour, there’s a critical difference: the fact that it will be graded is known in advance by the learner. This is compounded by the fact that the grade matters to them, often more than the performance of the activity itself. Thus, achieving the grade becomes the goal, not the consequence of the behaviour, and it reinforces the power of the grader to determine the behaviour of the learner, with a consequent loss of learner autonomy. That shift from intrinsic to extrinsic motivation is the big issue here, not the praise itself. There are lots of ways to give both performance praise and ability praise that are not coercive. They are only harmful when used to manipulate behaviour.
Address of the bookmark: http://www.alphr.com/science/1007043/highly-praised-children-are-more-inclined-to-cheat
A Quartz article that claims (accurately) that p-learning bootcamps dominate for those learning programming and other technical skills and (inaccurately) that the reason for that is that e-learning is much less engaging. In fact, there’s a sneaky and almost unnoticeable sleight of hand here, because what is actually claimed is that online learning can be less engaging and, based on that indubitable fact, extrapolates from the particular to the general, asserting that all online learning suffers the same way. Nonsense. Yes, there is a lot of rubbish online learning and, in fairness, even a well-evolved establishment like Athabasca University has some of it, at least for some people some of the time. But that’s not at all surprising because every university (online or not) presents the same problem and, if you are trying to make a single learning design work for everyone, you are sure to make it too complex for some and too boring for others (personalization technologies and intelligent personal learning designs notwithstanding). Athabasca has a lot less of it thanks to its extremely rigorous quality assurance processes, but it would be crazy to imagine that everything it does is perfect for every learner at every time, as much as it would be crazy to imagine that everyone at a bricks and mortar institution gets a wonderful learning experience every time. Crazier, in fact. The thing is, it ain’t what you do, it’s the way that you do it, that’s what gets results. It’s not that online learning is less effective (countless studies prove otherwise), it’s simply that it tends to be done in a way that gives control and flexibility to learners. The immersion of physical bootcamps does have one major and very distinctive benefit: that it pulls people out of everything else and forces them to engage for a lot of hours in the day. The ‘bootcamp’ part of it ensures that they are well and truly immersed, with no way to back out apart from backing out completely It’s not that the learning experience is any better – very far from it in most cases. Most bootcamps I have seen use inane pedagogies that would not pass muster even in a conventional university, let alone somewhere like Athabasca University that actually pays attention to such things. It’s just that people are there and they have to do stuff (a lot of social pressure is involved, as well as loss-aversion) so they wind up learning a lot simply because they put in the hours, and that happens simply because they are enrolled on a bootcamp and cannot get away.Not dissimilar to the ways traditional universities work, as it happens, just a lot more intense. Online learning gives people more choice and greater control so, if they are not innately fascinated or they have not very single-mindedly put aside enough time then, of course, they wind up learning less quickly because they put in less time over a longer period. Duh. It’s not rocket science. This is not about teaching effectiveness or smart learning designs, it is simply about stopping people from being distracted and doing other things. The solution, if a solution is needed, is for online learners to block out the time and drop the distractions. I can imagine plenty of learning designs for online learning that would make that happen – simply making it real-time and using smart tools for desktop sharing, real-time interaction, and monitoring of progress would achieve much the same results, as long as the ground rules are fully understood by all concerned. I was at a conference the other day that did pretty much that. Such an approach doesn’t happen much, of course, for all the reasons people go for online learning in the first place, inasmuch as such methods take away the control and flexibility that make it so appealing. On the other hand, perhaps there is a place for such techniques. It seems there is a market and, as long as expectations are carefully managed (you don’t distract yourself with reading emails and engaging in social media, you pledge to be available, you block out your calendar) it might work pretty well. But why bother? Seems to me that online learning is better precisely because of the control it gives people. If they need extrinsic motivation to force them to learn then that’s the problem that should be solved before enrolling on any courses, and it will do them a world of good in many other situations too. Address of the bookmark: https://qz.com/1064814/the-awkward-irony-of-not-being-able-to-take-a-good-coding-bootcamp-online/ Earlier today I responded to a prospective student who was, amongst other things, seeking advice on strategies for success on a couple of our self-paced programming courses. My response was just a stream of consciousness off the top of my head but I think it might be useful to others. Here, then, with some very light editing to remove references to specific courses, are a few fairly random thoughts on how to succeed on a self-paced online programming course (and, for the most part, other courses) at Athabasca University. In no particular order: Online learning can be great fun as long as you are aware of the big differences, primarily relating to control and personal agency. Our role is to provide a bit of structure and a supportive environment to enable you to learn, rather than to tell you stuff and make you do things, which can be disconcerting at first if you are used to traditional classroom learning. This puts more pressure on you, and more onus on you to organize and manage your own learning, but don’t ever forget that you are not ever really alone – we are here to help. In summary, I think it really comes down to three big things, all of which are really about motivation, and all of which are quite different when learning online compared to face-to-face: This advice is by no means comprehensive! If you have other ideas or advice, or things that have worked for you, or things that you disagree with, do feel free to share them in the comments.
A Quartz article that claims (accurately) that p-learning bootcamps dominate for those learning programming and other technical skills and (inaccurately) that the reason for that is that e-learning is much less engaging. In fact, there’s a sneaky and almost unnoticeable sleight of hand here, because what is actually claimed is that online learning can be less engaging and, based on that indubitable fact, extrapolates from the particular to the general, asserting that all online learning suffers the same way.
Yes, there is a lot of rubbish online learning and, in fairness, even a well-evolved establishment like Athabasca University has some of it, at least for some people some of the time. But that’s not at all surprising because every university (online or not) presents the same problem and, if you are trying to make a single learning design work for everyone, you are sure to make it too complex for some and too boring for others (personalization technologies and intelligent personal learning designs notwithstanding). Athabasca has a lot less of it thanks to its extremely rigorous quality assurance processes, but it would be crazy to imagine that everything it does is perfect for every learner at every time, as much as it would be crazy to imagine that everyone at a bricks and mortar institution gets a wonderful learning experience every time. Crazier, in fact.
The thing is, it ain’t what you do, it’s the way that you do it, that’s what gets results. It’s not that online learning is less effective (countless studies prove otherwise), it’s simply that it tends to be done in a way that gives control and flexibility to learners. The immersion of physical bootcamps does have one major and very distinctive benefit: that it pulls people out of everything else and forces them to engage for a lot of hours in the day. The ‘bootcamp’ part of it ensures that they are well and truly immersed, with no way to back out apart from backing out completely It’s not that the learning experience is any better – very far from it in most cases. Most bootcamps I have seen use inane pedagogies that would not pass muster even in a conventional university, let alone somewhere like Athabasca University that actually pays attention to such things. It’s just that people are there and they have to do stuff (a lot of social pressure is involved, as well as loss-aversion) so they wind up learning a lot simply because they put in the hours, and that happens simply because they are enrolled on a bootcamp and cannot get away.Not dissimilar to the ways traditional universities work, as it happens, just a lot more intense.
Online learning gives people more choice and greater control so, if they are not innately fascinated or they have not very single-mindedly put aside enough time then, of course, they wind up learning less quickly because they put in less time over a longer period. Duh. It’s not rocket science. This is not about teaching effectiveness or smart learning designs, it is simply about stopping people from being distracted and doing other things. The solution, if a solution is needed, is for online learners to block out the time and drop the distractions. I can imagine plenty of learning designs for online learning that would make that happen – simply making it real-time and using smart tools for desktop sharing, real-time interaction, and monitoring of progress would achieve much the same results, as long as the ground rules are fully understood by all concerned. I was at a conference the other day that did pretty much that. Such an approach doesn’t happen much, of course, for all the reasons people go for online learning in the first place, inasmuch as such methods take away the control and flexibility that make it so appealing. On the other hand, perhaps there is a place for such techniques. It seems there is a market and, as long as expectations are carefully managed (you don’t distract yourself with reading emails and engaging in social media, you pledge to be available, you block out your calendar) it might work pretty well. But why bother? Seems to me that online learning is better precisely because of the control it gives people. If they need extrinsic motivation to force them to learn then that’s the problem that should be solved before enrolling on any courses, and it will do them a world of good in many other situations too.
Address of the bookmark: https://qz.com/1064814/the-awkward-irony-of-not-being-able-to-take-a-good-coding-bootcamp-online/
Earlier today I responded to a prospective student who was, amongst other things, seeking advice on strategies for success on a couple of our self-paced programming courses. My response was just a stream of consciousness off the top of my head but I think it might be useful to others. Here, then, with some very light editing to remove references to specific courses, are a few fairly random thoughts on how to succeed on a self-paced online programming course (and, for the most part, other courses) at Athabasca University. In no particular order:
Online learning can be great fun as long as you are aware of the big differences, primarily relating to control and personal agency. Our role is to provide a bit of structure and a supportive environment to enable you to learn, rather than to tell you stuff and make you do things, which can be disconcerting at first if you are used to traditional classroom learning. This puts more pressure on you, and more onus on you to organize and manage your own learning, but don’t ever forget that you are not ever really alone – we are here to help.
In summary, I think it really comes down to three big things, all of which are really about motivation, and all of which are quite different when learning online compared to face-to-face:
This advice is by no means comprehensive! If you have other ideas or advice, or things that have worked for you, or things that you disagree with, do feel free to share them in the comments.