Why demoted Yelp reviewers still matter

By Kevin Manne

BUFFALO, N.Y. — When a reviewer loses an ‘elite’ badge on the restaurant review site Yelp, the quality of their reviews declines, but the perceived value of those reviews remains surprisingly high because of how the site displays their status, according to new research from the University at Buffalo School of Management.

Available online ahead of publication in Production and Operations Management, the study investigated the impact of non-financial incentives like badges and status on Yelp, and found that when consumers see a badge indicating that a reviewer once was elite, they place more trust in those reviews, regardless of their actual depth or accuracy.  

“Demoted reviewers feel they’ve been treated unfairly, which leads them to decrease their effort and produce lower-quality reviews,” says study co-author Wreetabrata Kar, PhD, assistant professor of marketing in the UB School of Management. “But Yelp users can see a record of the years when a reviewer held ‘elite’ status, which serves as a dominating cue and causes readers to place more trust in the reviewer, even after they’ve been downgraded. This disconnect between actual and perceived quality can undermine the accuracy of reviews and potentially impact the platform’s long-term sustainability.”

Using the Yelp academic dataset, the researchers analyzed more than 6 million reviews of nearly 200,000 businesses written by more than a million reviewers from 2006-2018, specifically focusing on nearly 9,000 reviewers who were downgraded from the elite status. They used propensity-score matching and difference-in-differences techniques — methods that compare similar groups and measure changes over time — to establish a causal link between losing elite status and changes in the quality of reviews, while controlling for factors that could influence these outcomes.

The researchers say these findings have broad implications for the design of online platforms that rely on user-generated content, highlighting the importance of balancing recognition and quality control to maintain user trust and platform integrity.

According to Kar, designing recognition systems is always a challenging task, as it requires balancing exclusivity (how many users receive recognition) with the risk of alienation (how reviewers feel when their status is downgraded).

“Platform managers should consider how recognition systems are displayed, because subtle design choices can significantly impact user behavior and perception — and may even undermine their goal of promoting high-quality content,” he says. 

Kar collaborated on the study with Vandith Pamuru, PhD, assistant professor of information systems at the Indian School of Business, and Warut Khern-am-nuai, PhD, associate professor of information systems at the McGill University Desautels Faculty of Management.