Can Behavioral Tools Improve Online Student Outcomes? Experimental Evidence from a Massive Open Online Course

Posted , updated on

Well-written and intelligently argued paper from Richard W. Patterson, using an experimental (well, nearly) approach to discover the effects of a commitment device, reminder and focus tool to improve course completion and performance in a MOOC.  It seems that providing tools to support students to  pre-commit to limiting ‘distracting Internet time’ (and that both measures and controls this) has a striking positive effect, though largely on those that appear to be extrinsically motivated: they want to successfully complete the course, rather than to enjoy the process of learning. Reminders are pretty useless for anyone (I concur – personally I find them irritating and, after a while, guilt-inducing and thus more liable to cause procrastination) and blocking distracting websites has very little if any effect – unsurprising really, because they don’t really block distractions at all: if you want to be distracted, you will simply find another way. This is good information.

It seems to me that those who have learned to be extrinsically motivated might benefit from this, though it will reinforce their dangerous predeliction, encourage bad habits, and benefit most those that have already figured out how to work within a traditional university system and that are focused on the end point rather than the journey. While I can see some superficially attractive merit in providing tools that help you to achieve your goals by managing the process, it reminds me a little of diet plans and techniques that, though sometimes successful in the short term, are positively harmful in the long term. This is the pattern that underlies all behaviourist models – it sort-of works up to a point (the course-setter’s goals are complied with), but the long-term impact on the learner is generally counter-productive. This approach will lead to more people completing the course, not more people learning to love the subject and hungry to apply that knowledge and learn more. In fact, it opposes such a goal. This is not about inculcating habits of mind but of making people do things that, though they want to reach some further end as a result, they do not actually want to do and, once the stimulus is taken away, will likely never want to do again. It is far better to concentrate on supporting intrinsic motivation and to build learning activities that people will actually want to do – challenges that they feel impelled to solve, supporting social needs, over which they feel some control. For that, the instructivist course format is ill suited to the needs of most. 

Abstract

Online education is an increasingly popular alternative to traditional classroom- based courses. However, completion rates in online courses are often very low. One explanation for poor performance in online courses is that aspects of the online environ- ment lead students to procrastinate, forget about, or be distracted from coursework. To address student time-management issues, I leverage insights from behavioral economics to design three software tools including (1) a commitment device that allows students to pre-commit to time limits on distracting Internet activities, (2) a reminder tool that is triggered by time spent on distracting websites, and (3) a focusing tool that allows students to block distracting sites when they go to the course website. I test the impact of these tools in a large-scale randomized experiment (n=657) conducted in a massive open online course (MOOC) hosted by Stanford University. Relative to students in the control group, students in the commitment device treatment spend 24% more time working on the course, receive course grades that are 0.29 standard deviations higher, and are 40% more likely to complete the course. In contrast, outcomes for students in the reminder and focusing treatments are not statistically distinguishable from the control. These results suggest that tools designed to address procrastination can have a significant impact on online student performance. 

Address of the bookmark: http://www.human.cornell.edu/pam/academics/phd/upload/PattersonJMP11_18.pdf

Thomas Frey: By 2030 over 50% of Colleges will Collapse

Posted , updated on

Thomas Frey provides an analysis of current trends in education (and, more broadly, learning) and predicts a grim future for colleges and, by extension, schools and universities. This is not a uniformly well-informed article – Frey is clearly an outsider with a somewhat caricatured or at least highly situated US-centric view of the educational system – but, though repeating arguments that have been made for decades and offering no novel insights, the issues are well summarized, well expressed, and the overall thrust of the article is hard to argue with.

His main points are summarized in a list:

  1. Overhead costs too high – Even if the buildings are paid for and all money-losing athletic programs are dropped, the costs associated with maintaining a college campus are very high. Everything from utilities, to insurance, to phone systems, to security, to maintenance and repair are all expenses that online courses do not have. Some of the less visible expenses involve the bonds and financing instruments used to cover new construction, campus projects, and revenue inconsistencies. The cost of money itself will be a huge factor.
  2. Substandard classes and teachers – Many of the exact same classes are taught in thousands of classroom simultaneously every semester. The law of averages tells us that 49.9% of these will be below average. Yet any college that is able to electronically pipe in a top 1% teacher will suddenly have a better class than 99% of all other colleges.
  3. Increasingly visible rating systems – Online rating systems will begin to torpedo tens of thousands of classes and teachers over the coming years. Bad ratings of one teacher and one class will directly affect the overall rating of the institution.
  4. Inconvenience of time and place – Yes, classrooms help focus our attention and the world runs on deadlines. But our willingness to flex schedules to meet someone else’s time and place requirements is shrinking. Especially when we have a more convenient option.
  5. Pricing competition – Students today have many options for taking free courses without credits vs. expensive classes with credits and very little in between. That, however, is about to change. Colleges focused primarily on course delivery will be facing an increasingly price sensitive consumer base.
  6. Credentialing system competition – Much like a doctor’s ability to write prescriptions, a college’s ability to grant credits has given them an unusual competitive advantage, something every startup entrepreneur is searching for. However, traditional systems for granting credits only work as long as people still have faith in the system. This “faith in the system” is about to be eroded with competing systems. Companies like Coursera, Udacity, and iTunesU are well positioned to start offering an entirely new credentialing system.
  7. Relationships formed in colleges will be replaced with other relationship-building systems – Social structures are changing and the value of relationships built in college, while often quite valuable, are equally often overrated. Just as a dating relationship today is far more likely to begin online, business and social relationships in the future will also happen in far different ways.
  8. Sudden realization that “the emperor has no clothes!” – Education, much like our money supply, is a system built on trust. We are trusting colleges to instill valuable knowledge in our students, and in doing so, create a more valuable workforce and society. But when those who find no tangible value begin to openly proclaim, “the emperor has no clothes!” colleges will find themselves in a hard-to-defend downward spiral.

It is notable that many of the issues raised are fully addressed by online universities like AU, and have been for decades. We have moved on to bigger and more intractible problems! In particular, the idea that classes and teachers are a fixture that cannot be changed is a bit quaint. It is also fair to say that Frey has only a rough idea of how education works: the notion that high quality lectures has anything much to do with learning or the university experience shows a failure to understand the beast – but then, the same is true of potential students and more than a few professors. But pricing competition, credentialling competition, relationship-building and, above all, the ’emporer has no clothes’ arguments hit home, and I think will have the effects he anticipates much sooner than 2050. Nothing new here, and a bit coarse, but it clearly expresses the stark reality of the consequences.

Address of the bookmark: http://www.futuristspeaker.com/2013/07/by-2030-over-50-of-colleges-will-collapse/

» Assessing teachers’ digital competencies Virtual Canuck

Posted , updated on

Terry Anderson on an Estonian approach to assessing teacher competences (and other projects) using Elgg – the same framework that underpins the Landing. I’ve downloaded the tool they have developed, Digimina, and will be trying it out, not just for exactly the purposes it was developed, but as the foundation for a more generalized toolset for sharing the process of assessment. May spark some ideas, I think.

A nice approach to methodology: Terry prefers the development of design principles as the ‘ultimate’ aim of design-based research (DBR), but I like the notion of software as a hypothesis that is used here. It’s essentially a ‘sciency’ way of describing the notion of trying out an idea to see whether it works that makes no particular claims to generality, but that both derives from and feeds a model of what can be done, what needs to be done, and why it should be done. The generalizable part is not the final stage, but the penultimate stage of design in this DBR model. In this sense, it formalizes the very informal notion of bricolage, capturing some of its iterative nature. It’s not quite enough, I think, any more than other models of DBR quite capture the process in all its richness. This is because the activity of formulating that hypothesis itself follows a very similar pattern at a much finer-grained scale to that of the bigger model. When building code, you try out ideas, see where it takes you, and that inspires new ideas through the process of writing as much as of designing and specifying. Shovelling that into a large-scale process model hides where at least an important amount of the innovation actually happens, perhaps over-emphasizing the importance of explicit evaluation phases and underplaying the role of construction itself.

Address of the bookmark: http://terrya.edublogs.org/2015/04/24/assessing-teachers-digital-competencies/

Wait for It: Delayed Feedback Can Enhance Learning – Scientific American

Posted , updated on

Report on a nice bit of cognitivist research – as the title suggests, delayed feedback (ie don’t give the answer right away) assists retention, and is best done after an unpredictable delay of a few seconds. What’s most interesting about it is the hypothesized reason: it’s curiosity. It only works if it piques your interest enough to want to know the answer, and your level of attention is raised when the timing of an upcoming anticipated event is uncertain. Like so many things in learning, motivation plays a big role here. 

Address of the bookmark: http://www.scientificamerican.com/article/wait-for-it-delayed-feedback-can-enhance-learning/

Government to close two in every five universities – University World News

Posted , updated on

This is pretty bad news for universities in Russia, coming on top of existing major cuts. It is notable that this mostly affects certificate mills with dubious credentials and very shady practices that have sprung up since the 90s, but will also affect some state-funded institutions. While there is clearly a long-overdue crackdown in progress on unethical companies pretending to offer education but really just selling certification, this doesn’t seem to tell the whole story and the article does not explain the underlying problems that this is a solution for. It makes me wonder whether this is just a local problem in Russia, or whether it is a part of a more general trend. I presume there may be some places where universities are gaining ground but, for the most part, the news I read suggests that most, the world over, are in more or less worsening straits. Is there any research out there on this as a global phenomenon? 

Address of the bookmark: http://www.universityworldnews.com/article.php?story=20150417043945585

Open University’s numbers dive 28% as pool of part-timers dries up

Posted , updated on

Quite a slide in just five years!

It is blamed by the incoming VC on general drops in student numbers in the UK, that most notably affect part-time students (like AU, all of the OU’s students are part-time). A recent article suggested a 37% drop in recent years in the UK HE sector overall, so 28% is perhaps not that awful, in relative terms.

The drop is not too surprising, given that UK fees have risen precipitously in recent years thanks to decades of attack on higher education by both labour and conservative governments. In the OU’s case, according to one of the commenters, this equates to an increase from £500 (a bit over $900 Canadian) to £1600 (about $3000) for a course. When course costs rise, part-timers (many of whom are self-funding and were, in the past, doing it for love as often as for career change or advancement) are inevitably going to be the first casualties.

Address of the bookmark: http://www.timeshighereducation.co.uk/news/open-universitys-numbers-dive-28-as-pool-of-part-timers-dries-up/2018593.article

Arizona State University is doing something very interesting that we should not ignore

Posted , updated on

This is an obvious development and one that we will see a great deal more of in the coming months and years. The real ‘innovation’ (though many of us have been arguing for this for a long time and ASU is not the first – it’s just a big one with a big partner) is that it makes it possible to truly disaggregate learning and credit. That’s not quite what ASU is offering, though, just yet. Instead, it is providing learners with the opportunity to take as long as they need, at a much lower cost than a traditional course, without any credential barriers, to gain a ‘real’ university qualification of at least as high a quality as one achieved by traditional means. It that sounds familiar, it should – Athabasca tries to do much the same thing. However, the Athabasca course still comes with a (distant) time limit and it costs roughly ten times a bit more (corrected: in fact, ASU charges $200 per credit hour for the accreditation, so it’s not a brilliant deal even though it is relatively cheap and highly competitive with AU).

In doing this, ASU (with the positively gleeful support of EdX) is contributing to the revolutionization of higher education.  I suspect it, as an institution, doesn’t see far enough ahead yet to fully take on the full implications yet, though some of those leading this probably do get it. The clues are that it seems to think it will contain it by making this an on-ramp to a future degree, it only half-heartedly recognizes that this has to become a full degree in itself, it apparently believes that what it is doing is a MOOC (if the inventor of the term, Dave Cormier, were not very much alive, kicking and doing truly wonderful things he would turn in his grave), and it still requires students to take the course to gain the credit. That’s short term thinking that cannot last very long. It’s straddling the 21st Century but it still has one leg firmly held in the 11th Century. The really interesting next step is that it is pretty much inevitable that ASU and those following this path (and there will be many more) will need to recognize learning achieved elsewhere and by other means too. At that point, assuming this is not an isolated initiative, traditional approaches to higher education are mostly either dead in the water or will have to recognize and nurture the greater value of a higher education, which has little or nothing to do with accreditation. More interestingly, perhaps, they (and the rest of us) will have to actually provide education that actually achieves what it sets out to do, without the crutch of rewards and punishments bound up with the process to keep it going.

We at Athabasca University already have challenge and PLAR processes that can, already, trivially easily surpass what ASU is offering, and leapfrog the coming crowd of competitors on the credit front. I don’t understand why we have not been jumping on that bandwagon with arms outstretched, while mentioning in passing that we have done something better than this for decades (though the idea stretches back to the 19th Century, so it’s not that amazing). AU also has the skills and experience to provide useful online education that could actually help someone achieve credit, whether with us or elsewhere. But, though it could keep us going, that’s likely not where our true sustainability lies. With things like the Landing (and, especially, what it might become) we also have the means to provide the really valuable stuff too – to provide a rich, vibrant learning community that offers both transient and ongoing learning networks for lifelong learning and development that can go far beyond the weird anachronism of courses – and to provide it online. I really hope that we, as an institution rather than a few of us individuals, realize this. If not, things look bleak. Universities are too deeply entangled in society and the economy to go away any time soon so they are not threatened as an institution yet, but they can and must change, if only to compete with one another and the coming wave of commercial competitors. ASU is not providing the answer yet – this is a local solution that makes economic sense for all concerned but that doesn’t even begin to see where it fits in with the big picture – but it is driving in the thin end of a big wedge that will eventually force the bigger changes to come about. 

Address of the bookmark: https://www.edx.org/blog/reimagine-freshman-year-global-freshman

PISA and irony: The 2015 Brown Center Report on American Education

Posted , updated on

It probably comes as no surprise that I have an extremely low opinion of PISA, the well-intentioned but operationally horrific international testing framework used to compare schooling (I use the word advisedly) in different countries. PISA matters to governments because it gives an apparently objective measure of the ‘effectiveness’ of education and it matters to the rest of us because governments’ desire to score highly in PISA league tables has a massive (and catastophic) effect on systems of education. This is ‘teaching to the test’ at a gargantuan scale, with all the awful consequences that entails. The laudable desire to improve literacy and basic knowledge leads to the consequence that, internationally, education becomes primarily concerned with compliance, standardization and the ability to perform to someone else’s criteria on command. I’d like to think that there is a bit more to it than that. The cost of literacy does not have to be dehumanization or an extrinsically driven populace, and I am quite sure that is not what OECD intends, but that is the systemic effect of these interventions. And so to this report…

This report is interesting on many levels but I would like to draw your attention to section 3, in which it is shown that there is quite a strong negative correlation between intrinsic motivation and the ability to perform well on PISA-oriented math tests, at an inter-country level. In other words, countries reporting the lower levels of intrinsic motivation tend to report higher levels of attainment (ie. test compliance).  Within a given country there is a very modest positive correlation – for instance, American kids who like math tend to do slightly better on the tests than those that do not, but it is not enough of a difference to make a difference.

The authors seem puzzled by this! I leave you to draw your own conclusions about standardized tests, grades, schools, education and government interventions. Paolo Freire and Ivan Illich would have had a field day.

Address of the bookmark: http://www.ewa.org/sites/main/files/file-attachments/brown_ctr_2015_v2.pdf

RIP community@brighton

Posted , updated on

Community@brighton was once the biggest social media site within an educational institution in the world, with tens of thousands of users and a very active participatory community in its first two or three years. Sadly, though it continued for a good 8 years (I think it began around 2005 or early 2006) and eventually grew to over 100,000 users, it didn’t last. It was put out of its misery last year, after many years of neglect and slow decline, and frozen in time as a set of PDF files, that I have just discovered. The link provided here is to the last post that I wrote on the site, examining some of the reasons for its decline. I am glad that it has been archived, even if only as static PDFs.

Community@brighton was a fairly direct parent of the Landing, which shares much of its genetic material and vision, so it does kind-of live on. As well as much of the technology, we took a lot of the lessons learned at Brighton, both good and painful, and have been learning more ever since. Because of those lessons, I think the Landing has much more going for it than its ancestor and it has already thrived for much longer. By this time in community@brighton’s life it was almost dead, doing virtually nothing but support a few students finding accommodation, while the Landing continues to blossom and shows no signs of slowing.

The Landing’s other and even more direct parent was another Elgg-based site, Me2U, that Terry Anderson instigated here at AU around the same time we started community@brighton in the UK. Me2U was a smaller site and that was a very good thing because, like Facebook in its early days, it cultivated a passionate core following. Me2U has had a much happier future (if being swallowed by the Landing in a single gulp one day in January 2010 could be considered happy!) and you are looking at it now. Its posts can still be found if you wander back through the Landing archives, and they make fascinating reading.

Address of the bookmark: http://blogs.brighton.ac.uk/elearning/files/2014/08/Community@Brighton_-s-blog_-The-last-post_-183u815.pdf

Top 20 in Educational Technology to Connect with through Social Media – AACE

Posted , updated on

Well this is nice – I (only just) made the top 20! Nice to be counted among the notably more luminous folk even if my feeble contributions are orders of magnitude smaller. Of course, this is just about people who write about educational technology using a particular subset of social media, who have some connection with AACE, whether as committee members or keynotes, and it is an informally garnered list so it is, at best, only partially representative of the broader field.  Even so, though they would knock me and a few others off the list, I think the following would mostly qualify too (and I avidly read most of what they write):

Donald Clark

Eric Duval 

David Wiley

Dave Cormier

Audrey Watters

Johannes Cronje

There are many others who have not yet done the AACE conference circuit or who have slipped off it, but who are well worth following on social media. I did start writing a list of some but realized early on that, even if I listed 100 or more, I would still miss people that really matter. You know who you are. The interesting thing though, I think, is that if you followed and interacted with just a few of these people on social media (not just what they write but what they curate and share) you would likely learn a great deal more about learning technologies, e-learning and education as a whole than you would by wading through a dozen courses or hand-curated textbooks. Crowds teach.

Still – thanks for the boost, AACE folk!

Address of the bookmark: http://blog.aace.org/2015/03/22/top-20-in-educational-technology-to-connect-with-through-social-media/