Understanding the response to financial and non-financial incentives in education: Field experimental evidence using high-stakes assessments

Posted , updated on

What they did

This is a report by, Simon Burgess, Robert Metcalfe, and Sally Sadoff on a large scale study conducted in the UK on the effects of financial and non-financial incentives on GCSE scores (GCSEs are UK qualifications usually taken around age 16 and usually involving exams), involving over 10,000 students in 63 schools being given cash or ‘non-financial incentives’. ‘Non-financial incentives’ did not stretch as far as a pat on the back or encouragement given by caring teachers – this was about giving tickets for appealing events. The rewards were given not for getting good results but for particular behaviours the researchers felt should be useful proxies for effective study: specifically, attendance, conduct, homework, and classwork. None of the incentives were huge rewards to those already possessing plenty of creature comforts but, for poorer students, they might have seemed substantial. Effectiveness of the intervention was measured in terminal grades. The researchers were very thorough and were very careful to observe limitations and concerns. It is as close to an experimental design as you can get in a messy real-world educational intervention, with numbers that are sufficient and diverse enough to make justifiable empirical claims about the generalizability of the results.

What they found

Rewards had little effect on average marks overall, and it made little difference whether rewards were financial or not. However, in high risk groups (poor, immigrants, etc) there was a substantial improvement in GCSE results for those given rewards, compared with the control groups.

My thoughts

The only thing that does surprise me a little is that so little effect was seen overall, but I hypothesize that the reward/punishment conditions are so extreme already among GCSE students that it made little difference to add any more to the mix.  The only ones that might be affected would be those for whom the extrinsic motivation is not already strong enough. There is also a possibility that the demotivating effects for some were balanced out by the compliance effects for others: averages are incredibly dangerous things, and this study is big on averages.

What makes me sad is that there appears to be no sense of surprise or moral outrage about this basic premise in this report.

dogs being whipped, from Jack London's 'Call of the Wild' It appears reasonable at first glance: who would not want kids to be more successful in their exams? When my own kids had to do this sort of thing I would have been very keen on something that would improve their chances of success, and would be especially keen on something that appears to help to reduce systemic inequalities. But this is not about helping students to learn or improving education: this is completely and utterly about enforcing compliance and improving exam results. The fact that there might be a perceived benefit to the victims is a red herring: it’s like saying that hitting dogs harder is good for the dogs because it makes them behave better than hitting them gently. The point is that we should not be hitting them at all. It’s not just morally wrong, it doesn’t even work very well, and only continues to work at all if you keep hitting them. It teaches students that the end matters more than the process, that learning is inherently undesirable and should only done when there is a promise of a reward or threat of punishment, and that they are not in charge of it.

The inevitable result of increasing rewards (or punishments – they are functionally equivalent) is to further quench any love of learning that might be left at this point in their school careers, to reinforce harmful beliefs about how to learn, and to further put students off the subjects they might have loved under other circumstances for life.  In years to come people will look back on barbaric practices like this much as we now look back at the slave trade or pre-emancipation rights for women.

Studies like this make me feel a bit sick.

 

Address of the bookmark: http://www.efm.bris.ac.uk/economics/working_papers/pdffiles/dp16678.pdf

Smart learning – a new approach or simply a new name? | Smart Learning

Posted , updated on

Kinshuk has begun a blog on smart learning and, in this post, defines what that means. I particularly like:

“ I have come to realize that while technology can help us in improving learning, a fundamental change is needed in the overall perception of educators and learners to see any real effect. Simply trying to create adaptive systems, intelligent systems, or any sort of mobile/ubiquitous environments is going to have only superficial impact, if we do not change the way we teach, and more importantly, the way we think of learning process (and assessment process).“ 

This very much echoes my own view. At least that fundamental change is needed in the context of formal education. Outside our ivory towers that fundamental change has already happened and continues to accelerate. Google Search, Wikipedia, Twitter, Reddit, StackExchange, Facebook and countless others of their net-enabled ilk are amongst the most successful learning technologies (more accurately, components of learning technologies) ever created, arguably up there with language and writing, ultimately way beyond printing or schools. 

Kinshuk goes on to talk of an ecosystem of technology and pedagogy, which I think is a useful way of looking at it. Terry Anderson, too, talks of the dance between technology and pedagogy with much the same intent. I agree that we have to take a total systems view of this. My own take on it is that pedagogies are technologies – learning technologies are simply those with pedagogies in the assembly, whether human-instantiated or embedded in tools. Technologies and pedagogies are not separate categories. Within the ecosystem there are many other technologies involved in the assembly apart from those we traditionally label as ‘learning technologies’ such as timetables, organizational structures, regulations, departmental roles, accreditation frameworks, curricula, organizational methods, processes and rituals, not to mention pieces like routers, protocols, software programs and whiteboards. But, though important, technologies are not the only objects in this ecology. We need to think of the entire ecosystem and consider things that are not technologies at all like friendship, caring, learning, creativity, belief, environment, ethics, and, of course, people. As soon as you get past the ‘if intervention x, then result y’ mindset that plagues much learning technology (and education) research, and start to see it as a complex adaptive system that is ultimately about what it means to be human, you enter a world of rich complexity that I think is far more productive territory. It’s an ecosystem that is filled not just with process but with meaning and value. 

On a more mundane and pragmatic note, I think it is worth observing that learning and accreditation of competence must be entirely separated – accreditation is an invasive parasite in this ecosystem that feeds on and consumes learning. Or maybe it is more like the effluent that poisons it. Either way, I’d prefer that accreditation should not be lumped under the ‘smart learning’ banner at all. ‘Smart accreditation’ is fine – I have no particular concerns about that, as a separate field of study. In some ways it is worthy of study in smart learning because of its effects. That is somewhat along the lines of studying oil spills when considering natural ecosystems.  Assessment (feedback, critical reflection, judgement, etc), on the other hand, is a totally different matter. Assessment is a critical part of almost any pedagogy worthy of the name and so of course must be part of a smart learning ecology. I’m not sure that it warrants a separate category of its own but it is certainly important. It is, however, highly dangerous to take the ‘easy’ next step of using it to assert competence, especially when that assertion becomes the reason for learning in the first place, or is used as a tool to manipulate learners. That is what predominantly drives education now, to the point that it threatens the entire ecosystem. 

That said, I’d like to think that it is possible that the paths of accreditation and assessment might one day rejoin because they do share copious commonalities. It would be great to find ways that the smart stuff we are doing to support learning might, as a byproduct, also be useful evidence in accreditation, without clogging up the whole ecosystem. Technologies like Caliper, TinCan, and portfolios offer much promise for that. 

Address of the bookmark: http://www.kinshuk.info/2015/05/smart-learning/

Assessment in historical perspective

Posted , updated on

Fascinating article by Ben Wilbrink (1997) that traces the evolution of assessment approaches, mainly in higher education, from mediaeval times. In the process this offers some intriguing insights into how universities themselves, and the pedagogies with which we are familiar, evolved.

Address of the bookmark: http://benwilbrink.nl/publicaties/97AssessmentStEE.htm

What exams have taught me

Posted , updated on

http://community.brighton.ac.uk/jd29/weblog/45251.html

I have argued at some length on numerous occasions that exams, especially in their traditional unseen, time-limited, paper-based form, without access to books or Internet or friends, are the work of the devil and fundamentally wrong in almost every way that I can think of. They are unfair, resource-intensive, inauthentic, counter-productive, anti-educational, disspiriting, soulless products of a mechanistic age that represent an ethos that we should condemn as evil.

And yet they persist.

I have been wondering why something so manifestly wrong should maintain such a hold on our educational system even though it is demonstrably anti-educational. Surely it must be more than a mean-spirited small-minded attempt to ensure that people are who they say they are?

I think I have the answer.

Exams are so much a part of our educational system that pervade almost every subject area that they teach a deeper, more profound set of lessons than any of the subjects that they relate to. Clearly, from their ubiquity, they must relate to more important and basic things to learn than, say, maths, languages, or history. Subjects may come and subjects may go but the forms of assessment remain startlingly constant. So, I have been thinking about what exams taught me:

  • that slow, steady, careful work is not worth the hassle – a bit of cramming (typically one-three days seemed to work for me) in a mad rush just before the event works much more effectively and saves a lot of time
  • the corollary – adrenalin is necessary to achieve anything worth achieving
  • that the most important things in life generally take around three hours to complete
  • that extrinsic motivation, the threat of punishment and the lure of reward, is more important than making what we do fun, enjoyable and intrinsically rewarding
  • that we are judged not on what we achieve or how we grow but on how well we can display our skills in an intense, improbably weird and disconcerting setting

I learnt to do exams early in life better than I learnt most of the subjects I was examined on and have typically done far better than I deserve in such circumstances. I have learnt my lessons well in real life. I (mostly) hit deadlines with minutes to spare and seldom think about them more than a day or two in advance. I perform fairly well in adrenalin-producing circumstances. I summarise and display knowledge that I don’t really have to any great extent. I extemporise. I do things because I fear punishment or crave reward. I play to the rules even when the rules are insane. A bit of high blood pressure comes with the territory. Sometimes this is really useful but I am really trying hard to get out of the habit of always working this way and tp adopt some other approaches sometimes.

There are many other lessons that our educational systems teach us beyond the subject matter – I won’t even begin to explore what we learn from sitting in rows, staying quiet and listening to an authority figure tell us things but, suffice it to say, I haven’t retained much knowledge of grammar, calculus, geography or technical drawing, but I am still unlearning attitudes and beliefs that such bizarre practices instilled in me.

Assessment is good. Assessment tells us how we are doing, where we need to try new things, different approaches, as well as what we are doing right. Assessment is a vital part of the learning process, whether we do it ourselves or get feedback from others (both is best). But assessment should not be the goal. Assessment is part of the process.

Accreditation is good too. Accreditation tells the world that we can do what we claim we can do. it is important that there are ways to verify to others that we are capable (most obviously in the case of people on whom others depend greatly such as surgeons, bus drivers and university professors). Except in cases where the need to work under enormous pressure in unnatural conditions is a prerequisite (there are some occasions) I would just prefer that we relied on authentic evidence rather than this frighteningly artificial process that tells us very little about how people actually perform in the task domain that they are learning in.

The biggest problem comes when we combine and systematise assessment and accreditation into an industrialised, production-line approach to education, losing sight of the real goals. There are many other ways to do this that are less harmful or even positively useful (e.g. portfolios, evidence-based assessment, even vivas when done with care and genuine dialogue) and many are actually used in higher education. We just need more of them to redress the balance a bit.