Strategies for successful learning at AU

Posted , updated on

Earlier today I responded to a prospective student who was, amongst other things, seeking advice on strategies for success on a couple of our self-paced programming courses. My response was just a stream of consciousness off the top of my head but I think it might be useful to others. Here, then, with some very light editing to remove references to specific courses, are a few fairly random thoughts on how to succeed on a self-paced online programming course (and, for the most part, other courses) at Athabasca University. In no particular order:

  • Try to make sure that people close to you know what you are doing and, ideally, are supportive. Other people can really help, not just for the mechanical stuff but for the emotional support. Online learning, especially the self-paced form we use, can feel a bit isolating at times, but there are lots of ways to close the gap and they aren’t all found in the course materials and processes. Find support wherever you can.
  • Make a schedule and try to keep to it, but don’t blame yourself if your deadlines slip a bit here and there – just adjust the plan. The really important thing is that you should feel in control of the process. Having such control is one of the huge benefits of our way of teaching, but you need to take ownership of the process yourself in order to experience the benefits.
  • If the course provides forums or other social engagement try to proactively engage in them. Again, other people really help.
  • You will have way more freedom than those in traditional classrooms, who have to follow a teacher simply because of the nature of physics. However, that freedom is a two-edged sword as you can sometimes be swamped with choices and not know which way to go. If you are unsure, don’t be afraid to ask for help. But do take advantage of the freedom. Set your own goals. Look for the things that excite you and explore further. Take breaks if you are getting tired. Play. Take control of the learning process and enjoy the ride.
  • Enjoy the challenges. Sometimes it will be hard, and you should expect that, especially in programming courses like these. Programming can be very frustrating at times – after 35 years of programming I can still spend days on a problem that turns out to involve a misplaced semi-colon! Accept that, and accept that even the most intractable problems will eventually be solved (and it is a wonderful feeling when you do finally get it to work). Make time to sleep on it. If you’re stuck, ask for help.
  • Get your work/life/learning balance right. Be realistic in your aspirations and expect to spend many hours a week on this, but make sure you make time to get away from it.
  • Keep a learning journal, a reflective diary of what you have done and how you have addressed the struggles, even if the course itself doesn’t ask for one. There are few more effective ways to consolidate and connect your learning than to reflect on it, and it can help to mark your progress: good to read when your motivation is flagging.
  • Get used to waiting for responses and find other things to learn in the meantime. Don’t stop learning because you are waiting – move on to something else, practice something you have already done, or reflect on what you have been doing so far.
  • Programming is a performance skill that demands constant and repeated practice. You just need to do it, get it wrong, do it again, and again, and again, until it feels like second nature. In many ways it is like learning a musical instrument or maybe even driving. It’s not something you can learn simply by reading or by being told, you really have to immerse yourself in doing it. Make up your own challenges if you run out of things to do.
  • Don’t just limit yourself to what we provide. Find forums and communities with appropriate interests. I am a big fan of StackOverflow.com for help and inspiration from others, though relevant subreddits can be useful and there are many other sites and systems dedicated to programming. Find one or two that make sense to you. Again, other people can really help.

Online learning can be great fun as long as you are aware of the big differences, primarily relating to control and personal agency. Our role is to provide a bit of structure and a supportive environment to enable you to learn, rather than to tell you stuff and make you do things, which can be disconcerting at first if you are used to traditional classroom learning. This puts more pressure on you, and more onus on you to organize and manage your own learning, but don’t ever forget that you are not ever really alone – we are here to help.

In summary, I think it really comes down to three big things, all of which are really about motivation, and all of which are quite different when learning online compared to face-to-face:

  1. Autonomy – you are in control, but you must take responsibility for your own learning. You can always delegate control to us (or others) when the going gets hard or choices are hard to make, but you are always free to take it back again, and there will be no one standing over you making you do stuff apart from yourself.
  2. Competence – there are few things more satisfying than being able to do more today than you could do yesterday. We provide some challenges and we try to keep them difficult-but-achievable at every stage along the way, but it is a great idea for you to also seek your own challenges, to play, to explore, to discover, especially if the challenges we offer are too difficult or too boring. Reflection can help a lot with this, as a means to recognize what, how, and why you have learned.
  3. Relatedness – never forget the importance of other people. You don’t have to interact with them if you don’t want to do so (that’s another freedom we offer), but it is at the very least helpful to think about how you belong in our community, your own community, and the broader community of learners and programmers, and how what and how you are learning can affect others (directly or indirectly).

This advice is by no means comprehensive! If you have other ideas or advice, or things that have worked for you, or things that you disagree with, do feel free to share them in the comments.

Our educational assessment systems are designed to create losers

Posted , updated on

The always wonderful Alfie Kohn describes an airline survey that sought to find out how it compared with others, which he chose not to answer because the airline was thus signalling no interest in providing the best quality experience possible, just aiming to do enough to beat the competition. The thrust of his article is that much the same is true of standardized tests in schools. As Kohn rightly observes, the central purpose of testing as it tends to be used in schools and beyond is not to evaluate successful learning but to compare students (or teachers, or institutions, or regions) with one another in order to identify winners and losers.

‘When you think about it, all standardized tests — not just those that are norm-referenced — are based on this compulsion to compare. If we were interested in educational excellence, we could use authentic forms of assessment that are based on students’ performance at a variety of classroom projects over time. The only reason to standardize the process, to give all kids the same questions under the same conditions on a contrived, one-shot, high-stakes test, is if what we wanted to know wasn’t “How well are they learning?” but “Who’s beating whom?”‘

It’s a good point, but I think it is not just an issue with standardized tests. The problem occurs with all the summative assessments (the judgments) we use. Our educational assessment systems are designed to create losers as much as they a made to find winners. Whether they follow the heinous practice of norm-referencing or not, they are sorting machines, built to discover competent people, and to discard the incompetent. In fact, as Kohn notes, when there are too many winners we are accused of grade inflation or a dropping of standards.

Wrong Way sign This makes no sense if you believe, as I do, that the purpose of education is to educate. In a system that demands grading, unless 100% of students that want to succeed get the best possible grades, then we have failed to meet the grade ourselves. The problem, though, is not so much the judgments themselves as it is the intimate, inextricable binding of judgmental with learning processes. Given enough time, effort, and effective teaching, almost anyone can achieve pretty much any skill or competence, as long as they stick at it. We have very deliberately built a system that does not aim for that at all. Instead, it aims to sort wheat from chaff. That’s not why I do the job I do, and I hope it is not why you do it either, but that’s exactly what the system is made to do. And yet we (at least I) think of ourselves as educators, not judges. These two roles are utterly separate and inconsolably inconsistent.

Who needs 100%?

It might be argued that some students don’t actually want to get the best possible grades. True. And sure, we don’t always want or need to learn everything we could learn. If I am learning how to use a new device or musical instrument I sometimes read/watch enough to get me started and do not go any further, or skim through to get the general gist. Going for a less-than-perfect understanding is absolutely fine if that’s all you need right now. But that’s not quite how it works in formal education, in part because we punish those that make such choices (by giving lower grades) and in part because we systematically force students to learn stuff they neither want nor need to learn, at a time that we choose, using the lure of the big prizes at the end to coax them. Even those that actually do want or need to learn a topic must stick with it to the bitter end regardless of whether it is useful to do the whole thing, regardless of whether they need more or less of it, regardless of whether it is the right time to learn it, regardless of whether it is the right way for them to learn it. They must do all that we say they must do, or we won’t give them the gold star. That’s not even a good way to train a dog.

It gets worse. At least dogs normally get a second chance. Having set the bar, we normally give just a single chance at winning or, at best, an option to be re-tested (often at a price and usually only once), rather than doing the human thing of allowing people to take the time they need and learn from their mistakes until they get as good as they want or need to get. We could learn a thing or two from computer games –  the ability to repeat over and over, achieving small wins all along the way without huge penalties for losing, is a powerful way to gain competence and sustain motivation. It is better if students have some control over the pacing but, even at Athabasca, an aggressively open university that does its best to give everyone all the opportunity they need to succeed, where self-paced learners can choose the point at which they are ready to take the assessments, we still have strict cut-offs for contract periods and, like all the rest, we still tend to allow just a single stab at each assessment. In most of my own self-paced courses (and in some others) we try to soften that by allowing students to iterate without penalty until the end but, when that end comes, that’s still it. This is not for the benefit of the students: this is for our convenience. Yes, there is a cost to giving greater freedom – it takes time, effort, and compassion – but that’s a business problem to solve, not an insuperable barrier. WGU’s subscription model, for instance, in which students pay for an all-you-can-eat smorgasbord, appears to work pretty well.

Meta lessons

It might be argued that there are other important lessons that we teach when we competitively grade. Some might suggest that competition is a good thing to learn in and of itself, because it is one of the things that drives society and everyone has to do it at least sometimes. Sure, but cooperation and mutual support is usually better, or at least an essential counterpart, so embedding competition as the one and only modality seems a bit limiting. And, if we are serious about teaching people about how to compete, then that is what we should do, and not actively put them in jeopardy to achieve that: as Jerome Bruner succinctly put it, ‘Learning something with the aid of an instructor should, if instruction is effective, be less dangerous or risky or painful than learning on one’s own’ (Bruner 1966, p.44).

Others might claim that sticking with something you don’t like doing is a necessary lesson if people are to play a suitably humble/productive role in society. Such lessons have a place, I kind-of agree. Just not a central place, just not a pervasive place that underpins or, worse, displaces everything else. Yes, grit can be really useful, if you are pursuing your goals or helping others to reach theirs. By all means, let’s teach that, let’s nurture that, and by all means let’s do what we can to help students see how learning something we are teaching can help them to reach their goals, even though it might be difficult or unpleasant right now. But there’s a big difference between doing something for self or others, and subservient compliance with someone else’s demands. ‘Grit’ does not have to be synonymous with ‘taking orders’. Doing something distasteful because we feel we must, because it aligns with our sense of self-worth, because it will help those we care about, because it will lead us where we want to be, is all good. Doing something because someone else is making us do it (with the threat/reward of grades) might turn us into good soldiers, might generate a subservient workforce in a factory or coal face, might keep an unruly subjugated populace in check, but it’s not the kind of attitude that is going to be helpful if we want to nurture creative, caring, useful members of 21st Century society.

Societal roles

It might be argued that accreditation serves a powerful societal function, ranking and categorizing people in ways that (at least for the winners and for consumers of graduates) have some value. It’s a broken and heartless system, but our societies do tend to be organized around it and it would be quite disruptive if we got rid of it without finding some replacement. Without it, employers might actually need to look at evidence of what people have done, for instance, rather than speedily weeding out those with insufficient grades. Moreover, circularly enough, most of our students currently want and expect it because it’s how things are done in our culture. Even I, a critic of the system, proudly wear the label ‘Doctor’, because it confers status and signals particular kinds of achievement, and there is no doubt that it and other qualifications have been really quite useful in my career. If that were all accreditation did then I could quite happily live with it, even though the fact that I spent a few years researching something interesting about 15 years ago probably has relatively little bearing on what I do or can do now.  The problem is not accreditation in itself, but that it is inextricably bound to the learning process. Under such conditions, educational assessment systems are positively harmful to learning. They are anti-educative. Of necessity, due to the fact that they tend to determine precisely what students should do and how they should do it, they sap intrinsic motivation and undermine love of learning. Even the staunchest of defenders of tightly integrated learning and judgment would presumably accept that learning is at least as important as grading so, if grading undermines learning (and it quite unequivocally does), something is badly broken.

A simple solution?

It does not have to be this way. I’ve said it before but it bears repeating: at least a large part of the solution is to decouple learning and accreditation altogether. There is a need for some means to indicate prowess, sure. But the crude certificates we currently use may not be the best way to do that in all cases, and it doesn’t have to dominate the learning process to the point of killing love of learning. If we could drop the accreditation role during the teaching process we could focus much more on providing useful feedback, on valorizing failures as useful steps towards success, on making interesting diversions, on tailoring the learning experience to the learner’s interests and capabilities rather than to credential requirements, on providing learning experiences that are long enough and detailed enough for the students’ needs, rather than a uniform set of fixed lengths to suit our bureaucracies.

Equally, we could improve our ability to provide credentials. For those that need it, we could still offer plenty of accreditation opportunities, for example through a portfolio-based approach and/or collecting records of learning or badges along the way. We could even allow for some kind of testing like oral, written, or practical exams for those that must, where it is appropriate to the competence (not, as now, as a matter of course) and we could actually do it right, rather than in ways that positively enable and reward cheating. None of this has to bound to specific courses. This decoupling would also give students the freedom to choose other ways of learning apart from our own courses, which would be quite a strong incentive for us to concentrate on teaching well. It might challenge us to come up with authentic forms of assessment that allow students to demonstrate competence through practice, or to use evidence from multiple sources, or to show their particular and unique skillset. It would almost certainly let us do both accreditation and teaching better. And it’s not as though we have no models to work from: from driving tests to diving tests to uses of portfolios in job interviews, there are plenty of examples of ways this can work already.

Apart from some increased complexities of managing such a system (which is where online tools can come in handy and where opportunities exist for online institutions that conventional face-to-face institutions cannot compete with) this is not a million miles removed from what we do now: it doesn’t require a revolution, just a simple shift in emphasis, and a separation of two unnecessarily and mutually inconsistent intertwined roles. Especially when processes and tools already exist for that, as they do at Athabasca University, it would not even be particularly costly. Inertia would be a bigger problem than anything else, but even big ships can eventually be steered in other directions. We just have to choose to make it so.

 

Reference

Bruner, J. S. (1966). Toward a Theory of Instruction. Cambridge MA: The Belknap Press of Harvard University Press.

Higher Education Whisperer: MIT For Credit edX Course Shows How to Market e-Learning

Posted , updated on

Great critique by Tom Worthington of an alleged for-credit MOOC from MIT that was anything but a MOOC. As Tom rightly points out, two instructors, 31 students, and online materials from EdX do not a MOOC make. As he notes, this kind of instructional process has been working pretty well for decades, including at Athabasca, as it happens. What is relatively novel, perhaps, is that the fact that the course itself was supplied at no (extra) cost to the institution. Effectively (though not quite in this case as it was an MIT course in the first place) this was a typical use of an OER course with accreditation and tuition wrapped around it, following a practice that has been common in many places – especially in developing countries – since the earliest MOOCs in 2008. Tom himself has created a great OER course on green computing that we use here at AU, which follows much the same pattern (though we have lightly adapted the Australian course for local use).

Less stress in online learning?

Tom observes that in this intervention, as in his own teaching, students tend to take the online option due to scheduling difficulties, not by preference, but that they are less stressed by the process than their face-to-face counterparts. This makes sense because there’s a lot more teaching presence in a course that is a) designed for online delivery (usually with great care and attention to detail) and b) supported by live teachers. Online learners in this kind of set-up are getting a huge amount of support for their learning, both from course designers/developers and from their own professors. Technically speaking, some of that exuberance of teaching will cancel out due to the inevitable tension between structure and dialogue implied by transactional distance theory, but the opportunities for feedback on coursework, at least, more than compensate for the high transactional distance caused by the industrial teaching approach of a pre-prepared online course. At least, I hope so, because (though mainly with courses we have developed ourselves and only rarely with OERs) this is exactly what Athabasca University has been doing for nearly 50 years, apparently with some success.

More stress in online teaching?

Personally, I have to admit, I normally hate teaching other people’s courses, although it is something I have often done. However well-developed they might be, there are always things I disagree with, factually and pedagogically, and I deeply dislike the strait jackets such structured courses create. This is perhaps a little hypocritical of me because I expect tutors on my courses to do exactly that, and routinely allocate my own faculty to teach courses that others have written, putting them in exactly that position. Whatever. Few seem to suffer my aversion to the same degree and many seem to positively relish it. I guess it makes it easier, with fewer choices to make. To each their own. But even I am very happy to take an existing OER (like Tom’s) and alter it to my own purposes, and am even happier to offer alternative OERs for my students to use within a pedagogical framework I have created. I think this is just common sense, giving both me and my students plenty of freedom to do what suits us best. Either way, re-use of existing well-designed courses is at least as great an idea as it was when Otto Peters came up with his industrial model of distance learning some decades ago.

The reputation of online learning

Tom notes that online and distance education has a bad reputation: to some extent, yes, sure, some people feel that way. Yes, there have been some bad examples of the modality that have resulted in bad press (ahem…Phoenix) and naive folk that have never experienced online learning do tend to believe that there is some magic that happens face to face that cannot be replicated online. They are right, as it happens: some things are difficult or impossible to replicate and it is a kind of magic. But the converse is also true – great things happen online that cannot be replicated face to face, and that’s a kind of magic too. And, just as not everyone gets a great online experience,  for many ‘face to face’ learners the experience is uniformly dire, with large impersonal lectures, ill-conceived pedagogies delivered by untrained teachers, and considerably less human interaction than what would typically be found online. On balance, while it is not quite correct to say that there is no significant difference, because there really are some basic differences in the need for self-management and control, there is no significant difference in the outcomes we choose to measure.

But, to return to the point, although some look upon online degrees less kindly, there are many employers who actively prefer those that have learned online because it is strong proof of their self-determination, will-power, and desire to succeed. I can confirm this positive perception: our students at AU are, on average, streets ahead of their traditionally taught counterparts, especially when you consider that a great many do not have the traditional qualifications needed to get into a conventional institution. I am constantly amazed by the skill and perseverance of our amazing students. On my own courses, especially in graduate teaching, I do everything I can to enable them to teach one another, because they tend to come to us with an incredible wealth of knowledge that just needs to be tapped and channelled.

A workable model

Though Tom is a little critical, I see value in what MIT is doing here. For some years now I have been trying to make the case at AU that we should be offering support for, and the means to credential, MOOCs offered elsewhere. This would give freedom to students to pick ways of learning that suit them best, to gain the benefits of diversity, and allow us to provide the kind of tutorial support and accreditation that we are pretty good at, at only a fraction of the (roughly) $100K cost of developing a typical course. It would give us the freedom to extend our offerings quite considerably, and avoid the need to keep developing the same curricula that are found everywhere else, so that we could differentiate ourselves by not just the style of teaching but also the subjects that we offer. This can in principle be done to some extent already through our challenge process (if you can find an equivalent course, take it, then take our challenge paper for a lot less than the price of a full course) and we do have independent study courses at graduate level that can be used much this way, with tutor support. But we could make a lot more of it if we did it just a bit more mindfully.

Address of the bookmark: http://blog.highereducationwhisperer.com/2017/06/mit-for-credit-edx-course-shows-how-to.html

Original page

Posts by Matthew Prineas – Athabasca University’s new Provost and Vice-president, Academic

Posted , updated on

I suspect everyone on Athabasca University’s staff will be very interested in these posts by Matthew Prineas, who we will welcome on September 5th as our new provost and VPA, that show a great understanding of at least some of the benefits and challenges of distance learning. Amongst other things, he has done some really good work on embedding OERs at UMUC, and has strong credentials (!) in the field of competency based methods of learning and accreditation. These things matter a great deal to our future. It also seems that he has a subtle appreciation of our distributed teaching approach, though I should note that there are more ways to skin this cat than the industrial model – we need to aim for post-industrial, where we achieve economies of scale not (just) by write-once-deliver-many teaching but by leveraging the value of human interaction on a large scale that distributed network technologies enable. It is great, though, that we’re getting a VPA who seems aligned with our mission and who reaches out to the world through social media. See, too, his Twitter posts at https://twitter.com/mprineas?lang=en

These are exciting times at AU!

Address of the bookmark: https://evolllution.com/author/matthew-prineas/

Original page

Athabasca’s bright future

Posted , updated on

Tony BatesThe always excellent Tony Bates provides a very clear summary of Ken Coates’s Independent Third-Party Review of Athabasca University released a week or two ago and, as usual, provides a great critical commentary as well as some useful advice on next steps.

Tony rightly points out that our problems are more internal than external, and that the solutions have to come from us, not from outside. To a large extent he hits the nail right on the head when he notes:

“Major changes in course design, educational technology, student support and administration, marketing and PR are urgently needed to bring AU into advanced 21st century practice in online and distance learning. I fear that while there are visionary faculty and staff at AU who understand this, there is still too much resistance from traditionalists and those who see change as undermining academic excellence or threatening their comfort zone.”

It is hard to disagree. But, though there are too many ostriches among our staff and we do have some major cultural impediments to overcome, it is far less people that impede our progress than it is our design itself, and the technologies – especially the management technologies – of which it consists. That must change, as a corequisite to changing the culture that goes along with it. With some very important exceptions (more on that below) our culture is almost entirely mediated through our organizational and digital technologies, most notably in the form of very rigid processes, procedures and rules, but also through our IT. Our IT should, but increasingly does not, embody those processes. The processes still exist, of course – it’s just that people have to perform them instead of machines. Increasingly often, to make matters worse, we shape our processes to our ill-fitting IT rather than vice versa, because the ‘technological debt’ of adapting them to our needs and therefore having to maintain them ourselves is considered too great (a rookie systems error caused by splitting IT into a semi-autonomous unit that has to slash its own costs without considering the far greater price paid by the university at large). Communication, when it occurs, is almost all explicit and instrumental. We do not yet have enough of the tacit flows of knowledge and easy communication that patch over or fix the (almost always far greater) flaws that exist in such processes in traditional bricks and mortar institutions. The continual partial attention and focused channels of communication resulting from working online mean that we struggle with tacit knowledge and the flexibility of embedded dialogue in ways old fashioned universities never have to even think about. One of the big problems with being so process-driven is that, especially in the absence of richer tacit communication, it is really hard to change those processes, especially because they have evolved to be deeply entangled with one another – changing one process almost always means changing many, often in structurally separate parts of the institutional machine, and involves processes of its own that are often entangled with those we set out to change. As a result for much of its operation, our university does what it does despite us, not because of us. Unlike traditional universities, we have nothing else to fall back on when it fails, or when things fall between cracks. And, though we likely have far fewer than most traditional universities, there are still very many cracks to fall through.

This, not coincidentally, is exactly true of our teaching too. We are pretty darn good at doing what we explicitly intend to do: our students achieve learning outcomes very well, according to the measures we use. AU is a machine that teaches, which is fine until we want the machine to do more than what it is built to do or when other, faster, lighter, cheaper machines begin to compete with it.  As well as making it really hard to make even small changes to teaching, what gets lost – and what matters about as much as what we intentionally teach – is the stuff we do not intend to teach, the stuff that makes up the bulk of the learning experience in traditional universities, the stuff where students learn to be, not just to do. It’s whole-person learning. In distance and online learning, we tend to just concentrate on parts we can measure and we are seldom even aware of the rest. There is a hard and rigid boundary between the directed, instrumental processes and the soft, invisible patterns of culture and belonging, beyond which we rarely cross. This absence is largely what gives distance learning a bad reputation, though it can be a strength if focused teaching of something well-defined is exactly what is needed, or if students are able to make the bigger connections in other ways (true of many of our successful students), when the control that the teaching method provides is worth all the losses and where a more immersive experience might actually get in the way. But it’s a boundary that alienates a majority of current and prospective students. A large percentage of even those we manage to enrol and keep with us would like to feel more connected, more a part of a community, more engaged, more belonging. A great many more don’t even join us in the first place because of that perceived lack, and a very large number drop out before submitting a single piece of work as a direct result.

This is precisely the boundary that the Landing is intended to be a step towards breaking down.

https://landing.athabascau.ca/file/view/410777/video-decreasing-the-distance

If we cannot figure out how to recover that tacit dimension, there is little chance that we can figure out how to teach at a distance in a way that differentiates us from the crowd and that draws people to us for the experience, rather than for the qualification. Not quite fair. Some of us will. If you get the right (deeply engaged) tutor, or join the right (social and/or open) course, or join the Landing, or participate in local meet-ups, or join other social media groups, you may get a fair bit of the tacit, serendipitous, incidental learning and knowledge construction that typifies a traditional education. Plenty of students do have wonderful experiences learning with others at AU, be it with their tutors or with other students. We often see those ones at convocation – ones for whom the experience has been deep, meaningful, and connected. But, for many of our students and especially the ones that don’t make it to graduation (or even to the first assignment), the chances of feeling that you belong to something bigger, to learn from others around you, to be part of a richer university experience, are fairly low. Every one of our students needs to be very self-directed, compared with those in traditional institutions – that’s a sina qua non of working online – but too many get insufficient support and too little inspiration from those around them to rise beyond that or to get through the difficult parts. This is not too surprising, given that we cannot do it for ourselves either. When faced with complicated things demanding close engagement, too many of our staff fall back on the comfortable, easy solution of meeting face to face in one of our various centres rather than taking the hard way, and so the system remains broken. This can and will change.

Moving on

I am much heartened by the Coates report which, amongst other things but most prominently and as our central value proposition, puts our leadership in online and distance education at the centre of everything. This is what I have unceasingly believed we should do since the moment I arrived. The call to action of Coates’s report is fundamentally to change our rigid dynamic, to be bold, to innovate without barriers, to evolve, to make use of the astonishingly good resources – primarily our people – to (again) lead the online learning world. As a virtual institution this should be easier than it would be for others but, perversely, it is exactly the opposite. This is for aforesaid reasons, and also because the boundaries of our IT systems create the boundaries of our thinking, and embed processes more deeply and more inflexibly than almost any bricks and mortar establishment could hope to do. We need soft systems, fuzzy systems, adaptable systems, agile systems for our teaching, research, and learning community development, and we need hard systems, automated systems, custom tailored, rock solid systems for our business processes, including the administrational and assessment recording outputs of the teaching process. This is precisely the antithesis of what we have now. As Coates puts it:

“AU should rebrand itself as the leading Canadian centre for online learning and twenty- first century educational technology. AU has a distinct and potentially insurmountable advantage. The university has the education technology professionals needed to provide leadership, the global reputation needed to attract and hold attention, and the faculty and staff ready to experiment with and test new ideas in an area of emerging national priority. There is a critical challenge, however. AU currently lacks the ICT model and facilities to rise to this opportunity.”

We live in our IT…

We have long been challenged with our IT systems, but things were not always so bad. Our ICT model has made a 180 degree turnaround in the past few years in the exact opposite direction to one that will support continuing evolution and innovation, driven by people that know little about our core mission and that have failed to understand what makes us special as a university. The best defence offered for these poor decisions is usually that ‘most other universities are doing it,’ but we are not most other universities.  ICTs are not just support tools or performance enhancers for us. We are our IT. It is our one and only face to our students and the world. Without IT, we are literally nothing. We have massively underinvested in developing our IT, and what we have done in recent years has destroyed our lead, our agility, and our morale. Increasingly, we have rented generic, closed, off-the-shelf cloud-based applications that would be pretty awful in a factory, that force us into behaviours that make no sense, that sap our time and will, and that are so deeply inappropriate for our very unique distributed community that they stifle all progress, and cut off almost all avenues of innovation in the one area that we are best placed to innovate and lead. We have automated things that should not be automated and let fall into disrepair the things that actually give us an edge. For instance, we rent an absurdly poor CRM system to manage student interactions, building a call centre for customers when we should be building relationships with students, embedding our least savoury practices of content delivery still further, making tweaks to a method of teaching that should have died when we stopped using the postal service for course packs. Yes, when it works, it incrementally improves a broken system, so it looks OK (not great) on reports, but the system it enhances is still irrevocably broken and, by further tying it into a hard embodiment in an ill-fitting application, the chances of fixing it properly diminish further. And, of course, it doesn’t work, because we have rented an ill-fitting system designed for other things with little or no consideration of whether it meets more than coarse functional needs. This can and must change.

Meanwhile, we have methodically starved the environments that are designed for us and through which we have innovated in the past, and that could allow us to evolve. Astonishingly, we have had no (as in zero) central IT support for research for years now, getting by on a wing and a prayer, grabbing for bits of overtime where we can, or using scarce, poorly integrated departmental resources. Even very well-funded and well-staffed projects are stifled by it because almost all of our learning technology innovations are completely reliant on access, not only to central services (class lists, user logins, LMS integration, etc), but also to the staff that are able to perform integrations, manage servers, install software, configure firewalls, etc, etc.  We have had a 95% complete upgrade for the Landing sitting in the wings for nearly 2 years, unable to progress due to lack of central IT personnel to implement it, even though we have sufficient funds to pay for them and then some, and the Landing is actively used by thousands of people. Even our mainstream teaching tools have been woefully underfunded and undermined: we run a version of Moodle that is past even its security update period, for instance, and that creaks along only thanks to a very small but excellent team supporting it. Tools supporting more innovative teaching with more tenuous uptake, such as Mahara and OpenSIM servers, are virtual orphans, riskily trundling along with considerably less support than even the Landing.

This can and will change.

… but we are based in Athabasca

There are other things in Coates’s report that are given a very large emphasis, notably advice to increase our open access, particularly through forming more partnerships with Northern Albertan colleges serving indigenous populations (good – and we will need smarter, more human, more flexible, more inclusive systems for that, too), but mainly a lot of detailed recommendations about staying in Athabasca itself. This latter recommendation seems to have been forced upon Coates, and it comes with many provisos. Coates is very cognizant of the fact that being based in the remote, run-down town of Athabasca is, has been, and will remain a huge and expensive hobble. He mostly skims over sensitive issues like the difficulty of recruiting good people to the town (a major problem that is only slightly offset by the fact that, once we have got them there, they are quite unlikely to leave), but makes it clear that it costs us very dearly in myriad other ways.

“… the university significantly underestimates the total cost of maintaining the Athabasca location. References to the costs of the distributed operation, including commitments in the Town of Athabasca, typically focus on direct transportation and facility costs and do not incorporate staff and faculty time. The university does not have a full accounting of the costs associated with their chosen administrative and structural arrangements.”

His suggestions, though making much of the value of staying in Athabasca and heavily emphasizing the importance of its continuing role in the institution, involve moving a lot of people and infrastructure out of it and doing a lot of stuff through web conferencing. He walks a tricky political tightrope, trying to avoid the hot potato of moving away while suggesting ways that we should leave. He is right on both counts.

Short circuits in our communications infrastructure

Though cost, lack of decent ICT infrastructure, and difficulties recruiting good people are factors in making Athabasca a hobble for us, the biggest problem is, again, structural. Unlike those working online, among those living and working in the town of Athabasca itself, all the traditional knowledge flows occur without impediment, almost always to the detriment of more inclusive ways of online communication. Face to face dialogue inevitably short-circuits online engagement – always has, always will. People in Athabasca, as any humans would and should, tend to talk among themselves, and tend to only communicate with others online, as the rest of us do, in directed, intentional ways. This might not be so bad were it not for the fact that Athabasca is very unrepresentative of the university population as a whole, containing the bulk of our administrators, managers, and technical staff, with less than 10 actual faculty in the region. This is a separate subculture, it is not the university, but it has enormous sway over how we evolve. It is not too surprising that our most critical learning systems account for only about 5% of our IT budget because that side of things is barely heard of among decision-makers and implementors that live there and they only indirectly have to face the consequences of its failings (a matter made much worse by the way we disempower the tutors that have to deal with them most of all, and filter their channels of communication through just a handful of obligated committee members). It is no surprise that channels of communication are weak because those that design and maintain them can easily bypass the problems they cause. In fact, if there were more faculty there, it would be even worse, because then we would never face any of the problems encountered by our students. Further concentrations of staff in Edmonton (where most faculty reside), St Albert (mainly our business faculty) and Calgary do not help one bit, simply building further enclaves, which again lead to short circuits in communication and isolated self-reinforcing clusters that distort our perspectives and reduce online communication. Ideas, innovations, and concerns do not spread because of hierarchies that isolate them, filter them as they move up through the hierarchy, and dissipate them in Athabasca. Such clustering could be a good part of the engine that drives adaptation: natural ecosystems diversify thanks to parcellation. However, that’s not how it works here, thanks to the aforementioned excess in structure and process and the fact that those clusters are far from independently evolving. They are subject to the same rules and the same selection pressures as one another, unable to independently evolve because they are rigidly, structurally, and technologically bound to the centre. This is not evolution – it is barely even design, though every part of it has been designed and top-down structures overlay the whole thing. It’s a side effect of many small decisions that, taken as a whole, result in a very flawed system.

This can and must change.

The town of Athabasca and what it means to us

Athabasca high street

Though I have made quite a few day trips to Athabasca over the years, I had never stayed overnight until around convocation time this year. Though it was a busy few days so I only had a little chance to explore, I found it to be a fascinating place that parallels AU in many ways. The impression it gives is of a raw, rather broken-down and depressed little frontier town of around 4,000 souls (a village by some reckonings) and almost as many churches. It was once a thriving staging post on the way to the Klondike gold rush, when it was filled with the rollicking clamour of around 20,000 prospectors dreaming of fortunes. Many just passed through, but quite a few stayed, helping to define some of its current character but, when the gold rush died down, there was little left to sustain a population. Much of the town still feels a bit temporary, still a bit of a campground waiting to turn into a real town. Like much of Northern Alberta, its fortunes in more recent years have been significantly bound to the oil business, feeding an industry that has no viable future and the morals of an errant crow, tied to its roller coaster fortunes. There are signs that money has been around, from time to time: a few nice buildings, a bit of landscaping here and there, a memorial podium at Athabasca Landing.  But there are bigger signs that it has left.

Athabasca Landing

Today, Athabasca’s bleak main street is filled with condemned buildings, closed businesses, discount stores, and shops with ‘sale’ signs in their windows. There are two somewhat empty town centre pubs, where a karaoke night in one will denude the other of almost all its customers.

There are virtually no transit links to the outside world: one Greyhound bus from Edmonton (2 hours away) comes through it, in the dead of night, and passenger trains stopped running decades ago. The roads leading in and out are dangerous: people die way too often getting there, including one of our most valued colleagues in my own school. It is never too far from being reclaimed by the forces of nature that surround it. Moose, bear, deer, and coyotes wander fairly freely. Minus forty temperatures don’t help, nor does a river that is pushed too hard by meltwaters from the rapidly receding Athabasca Glacier and that is increasingly polluted by the side-effects of oil production.

Athabasca

So far so bleak. But there are some notable upsides too. The town is full of delightfully kind, helpful, down-to-earth people infused with that wonderful Canadian spirit of caring for their neighbours, grittily facing the elements with good cheer, getting up early, eating dinner in the late afternoon, gathering for potlucks in one another’s houses, and organizing community get-togethers. The bulk of housing is well cared-for, set in well-tended gardens, in quiet, neat little streets. I bet most people there know their neighbours and their kids play together. Though tainted by its ties with the oil industry, the town comes across as, fundamentally, a wholesome centre for homesteaders in the region, self-reliant and obstinately surviving against great odds by helping one another and helping themselves. The businesses that thrive are those selling tools, materials, and services to build and maintain your farm and house, along with stores for loading your provisions into your truck to get you through the grim winters. It certainly helps that a large number of residents are employees of the university, providing greater diversity than is typically found in such settlements, but they are frontier folk like the rest. They have to be.

It would be unthinkable to pull the university out at this point – it would utterly destroy an already threatened town and, I think, it would cause great damage to the university. This was clearly at the forefront of Coates’s mind, too. The solution is not to withdraw from this strange place, but to dilute and divert the damage it causes and perhaps, even, to find ways to use its strengths. Greater engagement with Northern communities might be one way to save it – we have some big largely empty buildings up there that will be getting emptier, and that might not be a bad place for some face-to-face branching out, perhaps semi-autonomously, perhaps in partnership with colleges in the region. It also has potential as a place for a research retreat though it is not exactly a Mecca that would draw people to it, especially without transit links to sustain it. A well-designed research centre cost a fortune to build, though, so it would be nice to get some use out of it.

Perhaps more importantly, we should not pull out because Athabasca is a part of the soul of the institution. It is a little fitting that Athabasca University has – not without resistance – had its fortunes tied to this town. Athabasca is kind-of who we are and, to a large extent, defines who we should aspire to be. As an institution we are, right now, a decaying frontier town on the edge of civilization that was once a thriving metropolis, forced to help ourselves and one another battle with the elements, a caring bunch of individuals bound by a common purpose but stuck in a wilderness that cares little for us and whose ties with the outside world are fickle, costly, and tenuous. Athabasca is certainly a hobble but it is our hobble and, if we want to move on, we need to find ways to make the best of it – to find value in it, to move people and things away from it that it impedes the most, at least where we can, but to build upon it as a mythic hub that helps to define our identity, a symbolic centre for our thinking. We can and will help ourselves and one another to make it great again. And we have a big advantage that our home town lacks: a renewable and sustainable resource and product. Very much unlike Athabasca the town, the source of our wealth is entirely in our people, and the means we have for connecting them. We have the people already: we just need to refocus on the connection.

Computer science students should learn to cheat, not be punished for it

Posted , updated on

This is a well thought-through response to a recent alarmist NYT article about cheating among programming students.

The original NYT article is full of holy pronouncements about the evils of plagiarism, horrified statistics about its extent, and discussions of the arms wars, typically involving sleuthing by markers and evermore ornate technological fixes that are always one step behind the most effective cheats (and one step ahead of the dumber ones). This is a lose-lose system. No one benefits. But that’s not the biggest issue with the article. Nowhere does the NYT article mention that it is largely caused by the fact that we in academia typically tell programming students to behave in ways that no programmer in their right mind would ever behave (disclaimer: the one programming course that I currently teach, very deliberately, does not do that, so I am speaking here as an atypical outlier).

As this article rightly notes, the essence of programming is re-use of code. Although there are certainly egregiously immoral and illegal ways to do that (even open source coders normally need to religiously cite their sources for significant uses of code written by others), applications are built on layer upon layer upon layer of re-used code, common subroutines and algorithms, snippets, chunks, libraries, classes, components, and a thousand different ways to assemble (in some cases literally) the code of others. We could not do programming at all without 99% of the code that does what we want it to do being written by others. Programmers knit such things together, often sharing their discoveries and improvements so that the whole profession benefits and the cycle continues. The solution to most problems is, more often than not, to be found in StackExchange forums, Reddit, or similar sites, or in open source repositories like Github, and it would be an idiotic programmer that chose not to (very critically and very carefully) use snippets provided there. That’s pretty much how programmers learn, a large part of how they solve problems, and certainly how they build stuff. The art of it is in choosing the right snippet, understanding it, fitting it into one’s own code, selecting between alternative solutions and knowing why one is better (in a given context) than another. In many cases, we have memorized ways of doing things so that, even if we don’t literally copy and paste, we repeat patterns (whole lines and blocks) that are often identical to those that we learned from others. It would likely be impossible to even remember where we learned such things, let alone to cite them.  We should not penalize that – we should celebrate it. Sure, if the chunks we use are particulary ingenious, or particularly original, or particularly long, or protected by a licence, we should definitely credit their authors. That’s just common sense and decency, as well as (typically) a legal requirement. But a program made using the code of others is no less plagiarism than Kurt Schwitters was a plagiarist of the myriad found objects that made up his collages, or a house builder is a plagiarist of its bricks.

And, as an aside, please stop calling it ‘Computer Science’. Programming is no more computer science than carpentry is woodworking science. It bugs me that ‘computer science’ is used so often as a drop-in synonym for programming in the popular press, reinforced by an increasing number of academics with science-envy, especially in North America. There are sciences used in computing, and a tiny percentage of those are quite unique to the discipline, but that’s a miniscule percentage of what is taught in universities and colleges, and a vanishingly small percentage of what nearly all programmers actually do. It’s also worth noting that computer science programs are not just about programming: there’s a whole bunch of stuff we teach (and that computing professionals do) about things like databases, networks, hardware, ethics, etc that has nothing whatsoever to do with programming (and little to do with science). Programming, though, especially in its design aspects, is a fundamentally human activity that is creative, situated, and inextricably entangled with its social and organizational context. Apart from in some research labs and esoteric applications, it is normally closer to fine art than it is to science, though it is an incredibly flexible activity that spans a gamut of creative pursuits analogous to a broad range of arts and crafts from poetry to music to interior design to engineering. Perhaps it is most akin to architecture in the ways it can (depending on context) blend art, craft, engineering, and (some) science but it can be analogous to pretty much any creative pursuit (universal machines and all that).

Address of the bookmark: https://thenextweb.com/dd/2017/05/30/lets-teach-computer-science-students-to-cheat/#.tnw_FTOVyGc4

Original page

Learnium

Posted , updated on

Learnium is yet another attempt to overlay a cloud-based social medium on institutional learning, in the same family as systems like Edmodo, Wikispaces Classroom, Lore, GoingOn, etc, etc. I deliberately exclude from this list the far more excellent, theoretically grounded, and innovative Curatr, as well as dumb bandwagoners like – of all things – Blackboard (not deserving of a link but you could look up their atrocious social media management tools if you want to see how not to do this).

Learnium has a UK focus and it includes mobile apps as well as institutional integration tools. It looks slick, has a good range of tools, and seems to be gaining a little traction. This is trying to do something a little like what we tried to do with the Landing, but it should not be confused with the Landing in intent or design philosophy, notwithstanding some superficial similarities. Although the Landing is often used for teaching purposes, it deliberately avoids things like institutional roles, and deliberately blurs such distinctions when its users make use of them (eg. when they create course groups). It can be quite confusing for students expecting a guided space and top-down structure, and annoying if you are a teacher trying to control the learning space to behave that way, but that’s simply not how it is designed to work. The Landing is a learning space, where everyone is a teacher, not an institutional teaching space where the role is reserved for a few.

Learnium has a far more institutionally managed, teacher/course-oriented perspective. From what I can tell, it’s basically an LMS, cut down in some places, enhanced in its social aspects. It’s closer to Canvas than Moodle in that regard. It might have some value for teachers that like the social media tools but that dislike the lack of teacher-control, lack of privacy, deeply problematic ethics, and ugly intrusions of things like Facebook, and who do not want the cost or hassle of managing their own environments.  It is probably a more congenial environment for social pedagogies than most institutional LMSs, allowing learning to spread beyond class groups and supporting some kinds of social networking. There is a lot of scope and potential for vertical social networks like this that serve a particular kind of community in a tailored fashion. This is very much not Facebook, and that’s a very good thing.

But Learnium is an answer to the question ‘how can I use social media in my courses?’ rather than ‘how can social media help to change how people learn?’ It is also an answer to the question of ‘how can Learnium make money?’ rather than ‘how can Learnium help its users?’ And, like any cloud-based service of this nature (sadly including Curatr), it is not a safe place to entrust your learning community: things like changes to terms of service, changes to tools, bankcruptcy ,and takeovers are an ever-present threat. With the exception of open systems that allow you to move everything, lock stock and barrel, to somewhere else with no significant loss of data or functionality, an institution (and its students) can never own a cloud-based system like this. It might be a small difference from an end user perspective, at least until it blows up, but it’s all the difference in the world.

Address of the bookmark: https://www.learnium.com/about/institutions/

Original page

Every attempt to manage academia makes it worse

Posted , updated on

Excellent post from Mike Taylor on the inevitable consequences of the use of incentives to shape a system (in this case, an educational system). As Mike notes, the problem is well-known and well understood, yet  otherwise intelligent people continue to rely on extrinsic incentives to attempt to shape behaviour. It’s a classic Monkey’s Paw problem – you get what you wish for but something very bad will inevitably happen, often worse than the problem you are trying to solve. We can make people do things with extrinsic incentives (reward and punishment), sure, but in doing so we change the focus from what we want to achieve to the reward itself, which invariably destroys intrinsic motivation to do what we want done, reinforces our power (and thus the weakness of those we ‘incentivize’), and ultimately backfires on us in tragically predictable ways, because what we actually want done is almost never the thing we choose to measure.

some consequences of incentives, Edwards and Roy (2017)

Our educational systems (and many others) are built around extrinsic incentives, from grades through to performance-related pay through to misguided research assessment exercises, evaluations based on publication records, etc. The consequences are uniformly dire.

Mike quotes Tim Harford (from http://timharford.com/2016/09/4035/) as providing what seems to me to be the only sensible solution:

“The basic principle for any incentive scheme is this: can you measure everything that matters? If you can’t, then high-powered financial incentives will simply produce short-sightedness, narrow-mindedness or outright fraud. If a job is complex, multifaceted and involves subtle trade-offs, the best approach is to hire good people, pay them the going rate and tell them to do the job to the best of their ability.”

Well said. Except that I would add that the effects on motivation of any incentive scheme are always awful, and that’s the biggest reason not to do it. It’s not just that it doesn’t achieve the results we hope for: it’s that it is unkind and dehumanizing. With that in mind, I wouldn’t tell them to do the job to the best of their ability. I might ask them. I might help to structure a system so that they and everyone else can see the positive and negative consequences of actions they take. I might try to nurture a community where people value one another and are mutually supportive. I might talk to them about what they are doing and offer my support in helping them to do it better. I might try to structure the system around what people want to do rather than trying to make them fit in the system I want to build. At least, that’s what I would do on a good day. On a bad day, under pressure from multiple quarters, overworked and overstressed, I might fall back on a three line whip or a plea to do their bit. I might make trades (‘do this and I will take away that’) or appeal to a higher authority (‘the Dean says we must…’) or to my own authority (‘this has to be done and you are the best one to do it..’), or to duty (‘it is in our contract that we have to do performance assessments…’).  And that’s where the problems begin.

Mike recommends Tim Harford’s ‘The Undercover Economist’ as a way out of this loop. I will read this, as I have read many books offering similar insights. It seems at first glance to fit very well with the findings of self-determination theory as well as behavioural economics. However, though the causes described here are the result of a failure to understand human motivation, this is, at heart, a systems problem of a broader nature: I recommend The Systems Bible (formerly Systemantics) by John Gall Systemantics by John Gall (formerly the Systems Bible) for a comprehensive set of explanations of the kinds of phenomena that give rise to stupid behaviour by groups of intelligent people. The book is deliberately funny, but the underlying theory on which it is based is extremely sound.

Address of the bookmark: https://svpow.com/2017/03/17/every-attempt-to-manage-academia-makes-it-worse/

Babies in the learning-style bathwater

Posted , updated on

A recent Guardian article reports on a letter sent to the paper by 30 eminent academics from neuroscience, education, and psychology disciplines, voicing concerns about the absurd popularity of learning styles among teachers.

They are, of course, correct to be concerned. There is no good evidence that being taught according to your learning style has any positive value, despite decades of spurious attempts to show a correlation. Moreover, even if there were such a correlation, it would behoove teachers to help learners to learn using different styles because real-life learning doesn’t come neatly packaged in forms that fit with how we want/are constituted to learn, and teaching should primarily be concerned with supporting learners’ capacity to learn. The fact that there are scores if not hundreds of incompatible learning style theories, most of which have similarly (un)compelling evidence to support them, should be a clue that there is something seriously wrong with the whole idea. And it’s not a harmless foible. Not only is it a massive waste of time and money, not to mention a terrible example to set in truthiness acceptance, it can be actively harmful to learners, teaching them to believe that they can only learn properly if things are packaged to suit their style.

What’s shocking in the article is the report on the number of teachers who, despite a total lack of evidence and copious amounts of debunking, continue to use and believe in the things. To our shame, I have even seen examples of it at AU (our own Math Site mentions them) where we really ought to know better. But we are not unusual in this. Not at all. In the UK and Netherlands in 2012, 80% of teachers apparently believed that individuals learned better when doing so in a manner according with their preferred learning style. This is like discovering that 80% of the world’s scientists believe that their horoscopes determine the results of their experiments.

That said, there’s a baby in this very dirty bathwater that should not be thrown out.

If a belief in learning styles means that teachers feel challenged to design learning experiences in different ways to suit more diverse needs, that’s not a bad thing, apart from that it increases the costs of learning development. In fairness, it would work at least as well if they used astrological star sign personality characteristics as a basis but, whatever the reasons, giving students choices is a worthwhile outcome. And, just like horoscopes, there is value to learners themselves in providing an opportunity and a framework for reflection, even if the framework itself is erroneous and based on fallacies.

I’m a sceptic, but even I use variants on the theme. For example, I often try to provide versions of learning content that are meant to cater for serialist and holist ways of learning (Gordon Pask’s approach to categorizing learning strategies). Notwithstanding the extra effort and cost of designing at least two ways to approach a topic, it’s a good creative catalyst for me, and it gives students greater choice and control over their own learning.

And, in fairness, not all learning-style types of theory are equally awful. Slightly less harmful variants talk of learning preferences rather than styles, which does not necessarily imply that those preferences are a good idea nor that they even need to be catered for, though it still perpetuates the myth that there are relatively fixed characteristics in such things. Much better ones, including Pask’s, talk of selectable learning strategies rather than stable characteristics or preferences of learners, which seems eminently sensible to me: it’s just about general pedagogical patterns. It’s not about labelling learners, though (sadly) some do try to apply the labels to learners, and even Pask himself (arguably) sometimes seems to present it in that way. The best of breed models recognize that learning strategies can and should change in different learning contexts as well as over time, and make no attempt to label or pigeon hole learners themselves at all. I think it is really useful to find regularities and patterns in learning designs, and that’s the baby we should not throw out when we (rightly) reject learning style theories.

Address of the bookmark: https://www.theguardian.com/education/2017/mar/13/teachers-neuromyth-learning-styles-scientists-neuroscience-education

Original page

Intrinsic and Extrinsic Motivation

Posted

A short article from Lisa Legault that summarizes self-determination theory (SDT) and its findings very succinctly and clearly. It’s especially effective at highlighting the way the spectrum of extrinsic-to-intrinsic motivation works (including the integrated/identified/introjected continuum), and in describing the relationships between autonomy, competence, and relatedness. Nothing new here, nothing inspirational, just a useful resource to point people at so they can learn about the central tenets of SDT

Address of the bookmark: https://www.researchgate.net/profile/Lisa_Legault/publication/311692691_Intrinsic_and_Extrinsic_Motivation/links/5856e60d08ae77ec37094289.pdf