Study: Amazon Reviews Are Garbage

Five stars don't mean all that much, it turns out

Illustration: R. A. Di Ieso
Apr 29, 2016 at 3:43 PM ET

When you’re looking for a product on Amazon, the reviews are everything. How else are you supposed to decide between dozens of different bulk supplies of hair ties or six-packs of cotton underwear? It’s all in the stars, man. See five of them and your decision is practically made for you. But it turns out those reviews don’t tell you much about the quality of the item, according to a new study.

This might not matter for purchases of undies, where the risk of an uninformed purchase is panty-lines—which, OK, is scary—but what about things like bike helmets or smoke alarms?

Researchers looked at Amazon user ratings for over 1,000 products in 120 different categories—everything from car seats to sunblock. They found that the average number of stars a product received in reviews was inconsistent with the scores it received on Consumer Reports, a site that tests and evaluates the quality of items on a more objective level. For any two products chosen at random, they found only a 57 percent chance that the higher-rated product was higher-rated by Consumer Reports.

“A correspondence of 50 percent would be random, so user ratings provide very little insight about objective product performance,” explained Bart de Langhe, author of the study and professor of marketing at the University of Colorado Boulder, in a press release.

Amazon scores were generally higher for products that were more expensive or came from a brand with a “premium” reputation—in other words, it appears reviewers might be influenced by superficial signals of quality. “User ratings do not shield consumers from the influence of good old-fashioned advertising and price-signaling,” said de Langhe.

What’s more, the researchers discovered that ratings were often based on very low sample sizes and didn’t predict the likelihood of a product’s resale value. They also conducted a series of studies in which participants were surveyed about how they use online ratings and asked to search for and evaluate items on Amazon. It turns out that shoppers fail to consider the number of reviews behind a product’s star-rating. “This is a mistake,” said de Langhe. “Oftentimes, there are just not enough ratings for a product or there is too much disagreement among reviewers. In this case, consumers should not trust the average very much, but they do nonetheless.”

This might not be reason to stop counting stars, though. The researchers point out that there was some (albeit a weak) correlation between Amazon reviews and Consumer Reports ratings, and that the link was much stronger with star-ratings that were based on a greater number of reviews. So, the researchers write, “When sample sizes are sufficient, consumers can learn something about objective quality, but they should realize the information is far from perfect and base their quality judgment on additional sources of evidence.”