For Sale: “Your Name Here” in a Prestigious Science Journal

Posted , updated on

A Scientific American article on the prevalence of plagiarism and contract cheating in journal articles.  The tl;dr version lies near the end of the article:

“Now that a number of companies have figured out how to make money off of scientific misconduct, that presumption of honesty is in danger of becoming an anachronism. ‘The whole system of peer review works on the basis of trust,’ Pattinson says. ‘Once that is damaged, it is very difficult for the peer review system to deal with.'” 

Very sad. The only heartening thing about all this is that there are now thousands of scam journals (I think I now get at least half a dozen solicitations from these every day that I have learned to junk immediately) who would be more than willing to publish such articles. I rather like the idea that worse than useless fraudulent articles might get published in worse than useless scam journals. A nice little self-contained economy. Unfortunately, some of the cheats target real journals with real reputations and, worse, may be believed by genuine researchers who are taken in by the lies they purvey, endangering the whole academic research endeavour. Apparently the going price for that in China is around 93,000RMB, or $15,000.

This is very much like the issue we face in course assessment too. In some of my own courses I have designed what I reckon to be virtually foolproof methods of preventing most forms of cheating. They mostly work pretty well, but they don’t cope much better with contract cheating than more traditional assignment/exam based courses. My only partial solution to that problem is to try to price cheats out of the market: most of my courses have to be done from start to finish in order to pass, which is a lot more time consuming than writing a few boilerplate essays, exams or exercises. For assignments and exams on most courses you can get a passing grade for as little as $5, if you are willing to take the risk. The risk of discovery is very high because the essay mills tend to plagiarize or self-plagiarize (well, they are cheats – caveat emptor!) and, due to the semi-public nature of cheating sites, it is just as easy for us to discover students seeking ghost writers as it is for them to seek a ghost writer. In fact, when we find such sites, we tend to pass on our findings to colleagues in other institutions, a nice example of informal crowd-sourcing. However, I am absolutely sure some do get away with it, and it makes little or no difference whether teaching is online or face to face. There’s an example of contract cheating in exams in today’s news, but it is hardly newsworthy, apart from that it is endemic. Beyond contract cheating, I also know that some students have family members or friends who are motivated to ‘help’, sometimes quite considerably. There was a charmingly improbable example of a mother sitting her daughter’s exam a while back, for instance. 

I suspect that the ultimate solution to this in the case of courses is structural, not technological nor even directly pedagogical. We are in an un-winnable arms war in which everyone loses as long as the purpose of courses is seen to be to get accreditation, rather than to enable learning. As long as a grade sits enticingly at the end of it, that will inevitably cause some students to seek shortcuts to getting it. Cheats destroy the credibility not just of their own qualifications but those of every other student who has honestly run the course. If we got rid of grades altogether, cheating during the learning process would dry up to the merest trickle (though, bizarrely, might not go away altogether). Making accreditation a separate issue, completely disassociated from learning and teaching, would allow us to concentrate our firepower on preventing cheating at the point of accreditation rather than distracting us during a course, so we could make our courses far more engaging, enjoyable and useful: we could simply concentrate on pedagogy rather than trying to design cheating out of them. For the (entirely separate) accreditation, we could let rip with all the weaponry at our disposal, of course: biometrics, Faraday cages, style detectors, plagiarism detection tools and all the multifarious technologies and techniques we have developed to attempt to thwart cheats could be employed with relative ease by specialists trained to spot miscreants. Better still, we could use other means of proving authenticity such as social network analysis combined with public facing posts, or employer reports, or authentic portfolios created over long periods with multiple sources of authentication. This would also have the enormous benefit of largely solving what is perhaps the biggest challenge in all of education, that of motivation, getting rid of the extrinsic driver that eats at the soul of learning in our educational systems. It would also allow learners to control how, when, with whom and what they learn, rather than having to take a course that might bore them or confuse them. They could easily take a course elsewhere – even a MOOC – and prove their knowledge separately. It would make it easier for us to design courses that are apt for the learning need, rather than having to fit everything into one uniform size and shape. It would also overcome the insane contradiction of teachers telling students they have failed to learn when, quite clearly, it is the teachers that have failed to teach. Athabasca does, of course, have the mechanisms for this, in its PLAR and challenge processes. It could easily be done.

A similar solution might work, at least a little, for journal cheaters. There are different cultural norms around cheating in China, as I have observed previously, that perhaps play a role in the preponderance of Chinese culprits mentioned in the article, but a lot of the problem might be put down to the over-valuation of publication for career progression, prestige and reward in that country. If the rewards and reputation were less tightly bound to publication and more intrinsic to the process, we might see some improvement. This could be done in many ways: for instance, greater value could be given to internal dissemination of results,  open publication (inherently less liable to fraud thanks to many eyes), team work, blogging, supervisor reports, peer review (of people, not papers) and citations (though that is inevitably going to be the next easy target for fraud, if it is not already, so should not be treated too seriously). There are lots of ways to measure academic value apart from through numbers of publications, many of which relate to hard-to-spoof process rather than an easily forged product. The worrisome trend of journals charging authors for publication is an extremely bad idea that can only exacerbate the problem: publication becomes a commodity that is bought and sold, of value in and of itself (like grades) rather than as a medium to disseminate research.

These are sad times for academia, eaten from the inside and out, but they also present an opportunity for us to rethink the process. The standards and values that have evolved over many centuries and that once stood us in good stead when adult education was an elite affair just don’t apply any more. What our forebears sought in opening up academia was to expand the reach of education to all. Instead, we turned it into a system to deliver accreditation. That system is on a self-destruct course as long as we continue to act as though nothing has really changed. 

Address of the bookmark: http://www.scientificamerican.com/article/for-sale-your-name-here-in-a-prestigious-science-journal/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%253A+ScientificAmerican-News+%2528Content%253A+News%2529

Defaults matter

Posted , updated on

I have often written about the subtle and not-so-subtle constraints of learning management systems (LMSs) that channel teaching down a limited number of paths, and so impose implicit pedagogies on us that may be highly counter productive and dissuade us from teaching well – this paper is an early expression of my thoughts on the matter. I came across another example today.

When a teacher enters comments on assignments in Moodle (and in most LMSs), it is a one-time, one-way publication event. The student gets a notification and that’s it. While it is perfectly possible for a dialogue to continue via email or internal messaging, or to avoid having to use such a system altogether, or to overlay processes on top of it to soften the hard structure of the tool, the design of the software makes it quite clear this is not expected or normal. At best, it is treated as a separate process. The design of such an assignment submission system is entirely about delivering a final judgement. It is a tacit assertion of teacher power. The most we can do to subvert that in Moodle is to return an assignment for resubmission, but that carries its own meanings and, on resubmission, still returns us to the same single feedback box.

Defaults are very powerful things that profoundly shape how we behave (e.g. see here, here and here). Imagine how different the process would be if the comment box were, by default, part of a dialogue, inviting response from the student. Imagine how different it would be if the student could respond by submitting a new version (not replacing the old) or by posting amendments in a further submission, to keep going until it is just right, not as a process of replacement but of evolution and augmentation. You might think of this as being something like a journal submission system, where revisions are made in response to reviewers until the article is acceptable. But we could go further. What if it were treated as a debugging process, using approaches like those in Bugzilla or Github to track down issues and refine solutions until they were as good as they could be, incorporating feedback and help from students and others on or beyond the course? It seems to me that, if we are serious about assignments as a formative means of helping someone to learn (and we should be), that’s what we should be doing. There is really no excuse, ever, for a committed student to get less than 100% in the end. If students are committed and willing to persist until they have learned what they come here to learn, it is not ever the students’ failure when they achieve less than the best: it is the teachers’.

This is, of course, one of the motivations behind the Landing. In part we built this site to enable pedagogies like this that do not fit the moulds that LMSs ever-so-subtly press us into. The Landing has its own set of constraints and assumptions, but it is an alternative and complementary set, albeit one that is designed to be soft and malleable in many more ways than a standard LMS. The point, though, is not that any one system is better than any other but that all of them embed pedagogical and process assumptions, some of which are inherently incompatible.

The solution is, I think, not to build a one-size-fits-all system. Yes, we could easily enough modify Moodle to behave the way I suggest and in myriad other ways (e.g. I’d love to see dialogue available in every component, to allow student-controlled spaces wherever we need them, to allow students to add to their own courses, etc) but that doesn’t work either. The more we pack in, the softer the system becomes, and so the harder it is to operate it effectively. Greater flexibility always comes at a high price, in cognitive load, technical difficulty and combinatorial complexity. Moreover, the more we make it suit one group of people, the less well it suits others. This is the nature of monolithic systems.

There are a few existing ways to greatly reduce this problem, without massive reinvention and disruption. One is to disaggregate the pieces. We could build the LMS out of interoperable blocks so that we could, for instance, replace the standard submission system with a different one, without impacting other parts of the system. That was the goal of OKI and the now-defunct E-Framework although, in both cases, assembly was almost always a centralized IT management function and not available to those who most needed it – students and teachers. Neither have really made it to the mainstream. Sakai (an also-ran LMS that still persists) continues to use OKI technologies under the hood but the e-framework (a far better idea) seems dead in the water. These were both great ideas. There just wasn’t the will or the money, and competition from incumbents like Moodle and Blackboard was too strong. Other widget-based methods (e.g. using Wookie) offer more hope, because they do not demand significant retooling of existing systems, but they are currently far from on the ascendent and the promising EU TENCompetence project that was a leader behind this seems moribund, its site offline.

Another approach is to use modules/plugins/building blocks within an existing system. However, this can be difficult or impossible to manage in a manner that delivers control to the end user without at the same time making it difficult for those that do not want or need such control, because LMSs are monoliths that have to address the needs of many people. Not everyone needs a big toolkit and, for many, it would actively make things worse if they had one. Judicious use of templates can help with that, but the real problem is that one size does not fit all. Also, it locks you in to a particular platform, making evolution dependent on designers whose goals may not align with how you want to teach.

Bearing that in mind, another way to cope with the problem is to use multiple independent systems bound by interoperability standards – LTI, OpenBadges or TinCan, for example. With such standards, different learning platforms can become part of the same federated environment, sharing data, processing, learning paths and so on, allowing records to be kept centrally while enabling incompatible pedagogies to run independently within each system. That seems to me to be the most sensible option right now. It’s still more complex for all concerned than taking the easy path, and it increases management burden as well as replicating too much functionality for no particularly good reason. But sometimes the easy path is the wrong one, and diversity drives growth and improvement.

Great Firewall of China

Posted

Terry Anderson on great form discussing the problems of accessing scholarly and other content in China, with some nice insights into the environment in which Chinese scholars must conduct their research. I had not considered these particular issues of embedded Google services before, though have long been uncomfortable with the potential privacy concerns of using Google Analytics. Great stuff, well worth a read.

Address of the bookmark: http://terrya.edublogs.org/2014/12/13/great-firewall-of-china/

Workflow Automates Any Task on iOS

Posted , updated on

This is a very cool app that greatly extends the capacity of an iOS device to do many different things. I used a workflow on my iPad to add this link to the Landing from an item saved in Pocket, for instance, simply by selecting the workflow I had created from Pocket’s Share menu. Now, if only I could bundle that up and share it as an app, we could make Landing bookmarking A lot easier. 

This app sorely lacks help so far, though it is early days and clearly this will be coming soon. Though the app is pretty intuitive and has helpful hints, and there are some nice examples to play with, having existing programming skills is definitely valuable. It took me about an hour of trial and error to figure this simple workflow out.

Address of the bookmark: http://pocket.co/sBfTW

Investigating student motivation in the context of a learning analytics intervention during a summer bridge program

Posted , updated on

Very interesting, carefully performed and well articulated study that seems to suggest that showing students their data from early warning systems (learning analytics systems designed to identify at-risk student behaviours, usually through their interactions, or lack of interactions, in a learning management system) generally has a negative impact on their intrinsic motivation.

This is pretty much what one might expect because, as the researchers suggest, it inevitably shifts the focus from mastery to performance, and away from doing something for its own sake. This is probably among the worst things you could do to a learner, so it is not a trivial problem. It doesn’t negate the value of an EWS when used as intended, to help identify at-risk students and to focus tutor attention where it is most needed. I believe that an EWS can be very useful, as long as it is used with care (in every sense) and the results are treated critically. But it does raise a few alarm bells about the need to educate educators not just on the effective use of EWSs but on the nature of motivation in general. 

Address of the bookmark: http://www.sciencedirect.com/science/article/pii/S0747563214003793#

Automated Collaborative Filtering and Semantic Transports – draft 0.72

Posted , updated on

I had to look up this article by the late Sasha Chislenko for a paper I was reviewing today, and I am delighted that it is still available at its original URL, though Chislenko himself died in 2000. I’ve bookmarked the page on systems dating back to 1997 but I don’t think I’ve ever done so on this site, so here it is, still open to the world. Chislenko was writing in public way before it was fashionable and, I think, probably before the first blogs – this is still and, sadly, will always be a work in progress.

This particular page was one of a handful of articles that deeply influenced my early research and set me on a course I’m still pursuing to this day. Back in 1997, as I started my PhD, I had conceived of and started to build a web-based tagging and bookmark sharing system to gather learner-generated recommendations of resources and people so that the crowd could teach itself. It seemed like a common sense idea but I was not aware of anything else like it (this was long before del.icio.us and Slashdot was just a babe in arms), so I was looking for related work and then I found this. It depressed me a little that my idea was not quite as novel as I had hoped, but this article knocked me for six then and it continues to impress me now. It’s still great reading, though many of the suggestions and hopes/fears expressed in it are so commonplace that we seldom give them a second thought any more.

This, along with a special issue of ACM Communications released the same year, was my first introduction to collaborative filtering, the technology that would soon sit behind Amazon and, later, everything from Google Search to Netflix and eBay. It gave a name to what I was doing and to the system I was building, which was consequently christened ‘CoFIND’  (Collaborative Filter in N-Dimensions). 

Chislenko was a visionary who foresaw many of the developments over the past couple of decades and, as importantly, understood many of their potential consequences.  More of his work is available at http://www.lucifer.com/~sasha/articles/ – just a small sample of his astonishing range, most of it incomplete notes and random ideas, but packed with inspiration and surprisingly accurate prediction. He died far too young.

Address of the bookmark: http://www.lucifer.com/~sasha/articles/ACF.html

Constructivism versus objectivism: Implications for interaction, course design, and evaluation in distance education.

Posted , updated on

I’d not come across this (2000) article from Vrasidas till now, more’s the pity, because it is one of the clearest papers I have read on the distinction between objectivist (behaviourist/cognitivist)  and constructivist/social-constructivist approaches to teaching. It wasn’t new by any means even 15 years ago, but it provides an excellent overview of the schism (both real and perceived) between objectivism and constructivism and, in many ways, presages a lot of the debate that has gone on since surrounding the strengths, weaknesses and novelty of connectivist approaches. Also contains some good practical hints about how to design learning activities.

Address of the bookmark: http://vrasidas.intercol.edu/continuum.pdf

Instructional quality of Massive Open Online Courses (MOOCs)

Posted , updated on
This is a very interesting, if (I will argue) flawed, paper by Margaryan, Bianco and Littlejohn using a Course Scan instrument to examine the instructional design qualities of 76 randomly selected MOOCs (26 cMOOCs and 50 xMOOCs – the imbalance was caused by difficulties finding suitable cMOOCs). The conclusions drawn are that very few MOOCs, if any, show much evidence of sound instructional design strategies. In fact they are, according to the authors, almost all an instructional designer’s worst nightmare, on at least some dimensions.  
I like this paper but I have some fairly serious concerns with the way this study was conducted, which means a very large pinch of salt is needed when considering its conclusions. The central problem lies in the use of prescriptive criteria to identify ‘good’ instructional design practice, and then using them as quantitative measures of things that are deemed essential to any completed course design. 

Doubtful criteria 

It starts reasonably well. Margaryan et al use David Merrill’s well-accepted abstracted principles for instructional design to identify kinds of activities that should be there in any course and that, being somewhat derived from a variety of models and theories, are pretty reasonable: problem centricity, activation of prior learning, expert demonstration, application and integration. However, the chinks begin to show even here, as it is not always essential that all of these are explicitly contained within a course itself, even though consideration of them may be needed in the design process – for example, in an apprenticeship model, integration might be a natural part of learners’ lives, while in an open ‘by negotiated outcome’ course (e.g. a typical European PhD) the problems may be inherent in the context. But, as a fair approximation of what activities should be in most conventional taught courses, it’s not bad at all, even though it might show some courses as ‘bad’ when they are in fact ‘good’. 
The authors also add five more criteria abstracted from literature relating rather loosely to ‘resources’, including: expert feedback; differentiation (i.e. personalization); collaboration; authentic resources; and use of collective knowledge (i.e. cooperative sharing). These are far more contentious, with the exception of feedback, which almost all would agree should be considered in some form in any learning design (and which is a process thing anyway, not a resource issue). However, even this does not always need to be the expert feedback that the authors demand: automated feedback (which is, to be fair, a kind of ossified expert feedback, at least when done right), peer feedback or, best of all, intrinsic feedback can often be at least as good in most learning contexts. Intrinsic feedback (e.g. when learning to ride a bike, falling off it or succeeding to stay upright) is almost always better than any expert feedback, albeit that it can be enhanced by expert advice. None of the rest of these ‘resources’ criteria are essential to an effective learning design. They can be very useful, for sure, although it depends a great deal on context and how it is done, and there are often many other things that may matter as much or more in a design, like including support for reflection, for example, or scope for caring or passion to be displayed, or design to ensure personal relevance. It is worth noting that Merrill observes that, beyond the areas of broad agreement (which I reckon are somewhat shoehorned to fit), there is much more in other instructional design models that demands further research and that may be equally if not more important than those identified as common.

It ain’t what you do…

Like all things in education, it ain’t what you do but how you do it that makes all the difference, and it is all massively dependent on subject, context, learners and many other things. Prescriptive measures of instructional design quality like these make no sense when applied post-hoc because they ignore all this. They are very reasonable starting frameworks for a designer that encourage focus on things that matter and can make a big difference in the design process, but real life learning designs have to take the entire context into account and can (and often should) be done differently. Learning design (I shudder at the word ‘instructional’ because it implies so many unhealthy assumptions and attitudes) is a creative and situated activity. It makes no more sense to prescribe what kinds of activities and resources should be in a course than it does to prescribe how paintings should be composed. Yes, a few basics like golden ratios, rules of thirds, colour theory, etc can help the novice painter produce something acceptable, but the fact that a painting disobeys these ‘rules’ does not make it a bad painting: sometimes, quite the opposite. Some of the finest teaching I have ever seen or partaken of has used the most appalling instructional design techniques, by any theoretical measure.

Over-rigid assumptions and requirements

One of the biggest troubles with such general-purpose abstractions is that they make some very strong prior assumptions about what a course is going to be like and the context of delivery. Thanks to their closer resemblance to traditional courses (from which it should be clearly noted that the design criteria are derived) this is, to an extent, fair-ish for xMOOCs. But, even in the case of xMOOCs, the demand that collaboration, say, must occur is a step too far: as decades of distance learning research has shown (and Athabasca University proved for decades), great learning can happen without it and, while cooperative sharing is pragmatic and cost-effective, it is not essential in every course. Yes, these things are often a very good idea. No, they are not essential. Terry Anderson’s well-verified (and possibly self-confirming, though none the worse for it) theorem of interaction equivalency  makes this pretty clear.

cMOOCs are not xMOOCs

Prescriptive criteria as a tool for evaluation make no sense whatsoever in a cMOOC context. This is made worse because the traditional model is carried to extremes in this paper, to the extent that the authors bemoan the lack of clear learning outcomes. This doesn’t naturally fall out from the design principles at all, so I don’t understand why they are even mentioned, and it seems an abitrary criterion that has no validity or justification beyond the fact that they are typically used in university teaching. As teacher-prescribed learning outcomes are anathema to Connectivism it is very surprising indeed that the cMOOCs actually scored higher than the xMOOCs on this metric, which makes me wonder whether the means of differentiation were sufficiently rigorous. A MOOC that genuinely followed Connectivist principles would not provide learning outcomes at all: foci and themes, for sure, but not ‘at the end of this course you will be able to x’. And, anyway, as a lot of research and debate has shown, learning outcomes are of far greater value to teachers and instructional designers than they are to learners, for whom they may, if not handled with great care, actually get in the way of effective learning. It’s a process thing – helpful for creating courses, almost useless for taking them. The same problem occurs in the use of course organization in the criteria – cMOOC content is organized bottom-up by learners, so it is not very surprising that they lack careful top-down planning, and that is part of the point.

Apparently, some cMOOCs are not cMOOCs either

As well as concerns about the means of differentiating courses and the metrics used, I am also concerned with how they were applied. It is surprising that there was even a single cMOOC that didn’t incorporate use of ‘collective knowledge’ (the authors’ term for cooperative sharing and knowledge construction) because, without that, it simply isn’t a cMOOC: it’s there in the definition of Connectivism . As for differentiation, part of the point of cMOOCs is that learning happens through the network which, by definition, means people are getting different options or paths, and choosing those that suit their needs. The big point in both cases is that the teacher-designed course does not contain the content in a cMOOC: beyond the process support needed to build and sustain a network, any content that may be provided by the facilitators of such a course is just a catalyst for network formation and a centre around which activity flows and learner-generated content and activity is created. With that in mind it is worth pointing out that problem-centricity in learning design is an expression of teacher control which, again, is anathema to how cMOOCs work. Assuming that a cMOOC succeeds in connecting and mobilizing a network, it is all but certain that a great deal of problem-based and inquiry-based learning will be going on as people post, others respond, and issues become problematized. Moreover, the problems and issues will be relevant and meaningful to learners in ways that no pre-designed course can ever be. The content of a cMOOC is largely learner-generated so of course a problem focus is often simply not there in static materials supplied by people running it. cMOOCs do not tell learners what to do or how to do it, beyond very broad process support which is needed to help those networks to accrete. It would therefore be more than a little weird if they adhered to instructional design principles derived from teacher-led face-to-face courses in their designed content because, if they did, they would not be cMOOCs. Of course, it is perfectly reasonable to criticize cMOOCs as a matter of principle on these grounds: given that (depending on the network) few will know much about learning and how to support it, one of the big problems with connectivist methods is that of getting lost in social space, with insufficient structure or guidance to suit all learning needs, insufficient feedback, inefficient paths and so on. I’d have some sympathy with such an argument, but it is not fair to judge cMOOCs on criteria that their instigators would reject in the first place and that they are actively avoiding. It’s like criticizing cheese for not being chalky enough.

It’s still a good paper though

For all that I find the conclusions of this paper very arguable and the methods highly criticizable, it does provide an interesting portrait of MOOCs using an unconventional lens. We need more research along these lines because, though the conclusions are mostly arguable, what is revealed in the process is a much richer picture of the kinds of things that are and are not happening in MOOCs. These are fine researchers who have told an old story in a new way, and this is enlightening stuff that is worth reading.
 
As an aside, we also need better editors and reviewers for papers like this: little tell-tales like the fact that ‘cMOOC’ gets to be defined as ‘constructivist MOOC’ at one point (I’m sure it’s just a slip of the keyboard as the authors are well aware of what they are writing about) and more typos than you might expect in a published paper suggest that not quite enough effort went into quality control at the editorial end. I note too that this is a closed journal: you’d think that they might offer better value for the money that they cream off for their services.

Address of the bookmark: http://www.sciencedirect.com/science/article/pii/S036013151400178X