Fascinating article from 2013 on an experiment on a live website in which the experimenters manipulated rating behaviour by giving an early upvote or downvote. An early upvote had a very large influence on future voting, increasing the chances by nearly a third that a randomly chosen piece of content would gain more upvotes in future, with final ratings increased by 25% on average. Interestingly, downvotes did not have the same effect, making very little overall difference. Topics and prior relationships made some difference.
This accords closely with many similar studies and experiments, including a social navigation study I performed about a decade ago, involving clicking on a treasure map, the twist being that participants had to try to guess where, on average, most other people would click. About half the subjects could see where others had already clicked, the about half could not. The participants were aware that the average was taken from those that could not see where others had clicked. The click patterns of each set were radically different…
On closer analysis, of those that could see where others had clicked, around a third of the subjects followed what others had done (as this recent experiment suggests), around a third followed a similar pattern to the ‘blind’ partipants, and around a third actively chose an option because others had not done so – on the face of it this latter behaviour was a bit bizarre, given the conditions of the contest, though it is quite likely that they were assuming just such a bias would occur and acting accordingly.
One thing that might be useful, though very difficult, would be to try to weed out the herd followers and downgrade their ratings. StackExchange tries to do something like this by giving more weight to those that have shown expertise in the past, but it has not fully sorted out the problem of the super-influential that have a lot of good karma as a result of gaming the system, as well as the networks that form within it leading to bias (a problem shared by the less-sophisticated but also quite effective Reddit). At the very least, it might be helpful to introduce a delay to feedback being shown until a certain amount of time has passed or a threshold has been reached.
One thing is certain, though: simple aggregated ratings that are fed back to prospective raters (including those voting in elections) are almost purpose-built to make stupid mobs. As several people have shown, including Surowiecki and Page, crowds are normally only wise when they do not know what the rest of the crowd is thinking.
ABSTRACT
Our society is increasingly relying on the digitized, aggregated opinions of others to make decisions. We therefore designed and analyzed a large-scale randomized experiment on a social news aggregation Web site to investigate whether knowledge of such aggregates distorts decision-making. Prior ratings created significant bias in individual rating behavior, and positive and negative social influences created asymmetric herding effects. Whereas negative social influence inspired users to correct manipulated ratings, positive social influence increased the likelihood of positive ratings by 32% and created accumulating positive herding that increased final ratings by 25% on average. This positive herding was topic-dependent and affected by whether individuals were viewing the opinions of friends or enemies. A mixture of changing opinion and greater turnout under both manipulations together with a natural tendency to up-vote on the site combined to create the herding effects. Such findings will help interpret collective judgment accurately and avoid social influence bias in collective intelligence in the future.
Address of the bookmark: http://www.sciencemag.org/content/341/6146/647.full
