Three Minds On Digital Marketing @ Organic

A couple of articles that came out recently - one from TechCrunch and another from MIT's Technology Review - sparked a lot of healthy debate here at Organic.

How useful and reliable are ratings and reviews? Do super users bunk up the system? Or are these systems simply set up to fail?

According to TechCrunch, five-star ratings systems (YouTube's was called out) are unreliable and inferior to other, better-defined/fewer-choices models like thumbs up/thumbs down or favoriting. The weakness of the five-star system was pegged to user subjectivity (But how can any ratings system get around that? Aren't we looking for others to share their own personal experiences and point of view in reviews?) and the tendency to vote only when one really loves (five stars!) or really hates (one star!) something.

And this is where the heated e-mail debate begins...

Fang-Yu Lin: "Yes, only people who love or hate a video enough would bother to rate it, hence the U shaped rating curve. However, one cannot simply extrapolate that to other 5-star rating systems. Amazon's customer reviews, for instance, seem more evenly distributed. My guess is that by requiring people to write a review along with the rating, herd behaviors are greatly reduced... The issue here is not the 5-star scale itself, but the entire system one designs around it."

Bridget McKinley: "But are Amazon's ratings and reviews really all that much more reliable? The MIT Technology Review article on recent Carnegie Mellon research indicates even the Amazon system has serious issues. The culpruit? A handful of superusers and their bias."

Fang-Yu Lin: "Curiously, many seem to accept the thesis that if some users rate more items than the majority of users, reviews would be skewed as a result. Is it really so? For every item, each user has only one vote. It doesn't matter if certain reviewers are way more active; on a given item he or she can cast one vote, just like everyone else."

Bridget McKinley: "Or course, you can argue that even though each user only votes once, the likelihood of encountering reviews from power users are higher on any given item... Could this be a key issue hindering the reliability of reviews? Well, maybe. Conventional wisdom (along with hard data from a variety of sources) indicates that fewer than 10% of users end up generating more than 90% of the content on sites with social media functionality (functionality like ratings and reviews). That can certainly lead to volatility and sway that may undermine the very point of providing that type of service on a given site in the first place."

Fang-Yu Lin: "There is no proof that these [Amazon] frequent reviewers effectively acted as a voting bloc and submitted unvaried reviews, thus making the point moot."

Bridget McKinley: "But even if they aren't acting in a bloc, they are significantly affecting the curve. Recent studies by Bazaarvoice, Keller Fay Group. and JupiterKagan have all concluded that positive reviews tend to outweigh negative ones by an overwhelming margin (the Bazaarvoice analysis finds a 8:1 disparity)."

Fang-Yu Lin: "Looking at the average Amazon rating by item [per the Carnegie Mellon research], yes, there is a tendency toward higher scores. However, this is to be expected for a shopping site: People buy things that they perceive of a higher value. Of course more reviews are on the positive end (Sometimes things don't live up to the expectation, hence the higher standard deviations)."

Fang-Yu Lin: "Now look at the average rating by user: There's still a bias toward higher scores for the same reason, but the bias is much less pronounced. The standard deviations are lower here too. This seems to suggest that many Amazon reviewers are rather evenhanded."

But what is really the point of all this, what is the crux of the matter...

Is the fundamental question really about the benefit or detriment of power reviewers on the user experience? After all, without them, most sites (including Amazon) would be left with a much shallower pool of user participation and feedback. Perhaps... but perhaps the question is more about how sites can best maintain usefulness and credibility in star-rated and other voting systems with power users in the mix. With around 70% of digital consumers relying on and trusting in other's opinions online, this is a problem that needs fixing fast.

Bridget McKinley
Fang-Yu Lin