Sorting rated content

Anything with user-submitted ratings can be sorted by those ratings. But what is the best way? The naive approach is to sort by average rating. Unfortunately, this would rate 5-stars from a single user as higher than 4.5-stars from 150 users.

This guy has an answer. In short, his recommendation is to take the lower bound of the 95% confidence interval, given by:

[latex size=”3”]\frac{\hat{p}+\frac{z^2{\alpha/2}}{2n} \pm z{\alpha/2}\sqrt{(\hat{p}(1-\hat{p}) + z^2{\alpha/2}/{4n})/n}}{1+z^2{\alpha/2}/n}[/latex]

where

  • [latex]\hat{p}[/latex] is the observed fraction of positive ratings
  • [latex]z_{\alpha/2}[/latex] is the [latex]1-\alpha/2[/latex] quantile of the standard normal distribution
  • [latex]n[/latex] is the total number of ratings

Or, in Ruby:

require 'statistics2'



def cilowerbound(pos, n, power)
    if n == 0
        return 0
    end
    z = Statistics2.pnormaldist(1-power/2)
    phat = 1.0pos/n
    (phat + zz/(2n) - z * Math.sqrt((phat(1-phat)+zz/(4n))/n))/(1+z*z/n)
end

where

  • pos is the number of positive rating
  • n is the total number of ratings
  • power refers to the statistical power (0.05 recommended)

3 Comments

  1. This is kind of sucky though, because (at present) it's just a binary system--every rating is either "positive" or not. If your threshold is say 3/5 to be considered positive, then somehow 100 people giving a 3 would actually be worth less than 90 people giving a 5 and 10 people giving a 2. That just sounds nutty.

    I just checked the site and he acknowledges this. But still, that seems like a huge problem given your pitch that this is a solution.

    The IMDb's method takes a weighted average of the actual rating with the "mean" rating across all films (6.7, apparently). So a movie's rating (for the top 250, or the bottom 100) asymptotically approaches 6.7 as the number of votes goes to zero, and approaches the actual rating as the number of votes goes to infinity. Less statistically correct but very simple and effective.

  2. Sorry, 100 people giving a 3 would be worth MORE than 90 people giving a 5 and 10 people giving a 2. Anyway.

  3. It's binary, but it's very possible to represent the 0-5 star data accurately. Simply count each person as a potential of 5 positive votes, and then add a positive vote for each star they give the item.

    Case A: 100 people rating an item 3 becomes 300 positive votes out of 500 possible.
    Case B: 90 people rating an item 5 and 10 rating it 2 becomes 470 positive votes out of 500 possible.

    Using this formula, A rates at 0.556 and B rates at 0.916 -- a nice reflection of the actual ratings.

    Btw, because I had such a hard time finding it originally, I've packaged the statistics2 library as a gem.

Leave your thoughts
  • You can use some HTML in your comment.
  • Your comment may not display immediately due to spam filtering. Please wait for moderation.