Gizmodo
I've always teased friends at CNet about their rating system, which appears to always rate products between 7 and 8, meaning everything is more or less "very good" in score. Ecoustics has done the job of analyzing 1,325 reviews from 2007 (all of them?) and figured out the exact math: 96% of all ratings from last year fell between 5.0 and 8.9; about 70% fall between 6 and 7.9. Maybe CNet should make anything a "CNet 6 or below" a "1", and anything that scores a "CNet 10" a "5". After all, anything below a 6 means DO NOT BUY to me.
Other interesting points:
8.0 to 8.9 Excellent A product that receives a rating in this range is superior in so many ways that its relatively few drawbacks are not very important. 18.6%7.0 to 7.9 Very good While the strengths of a product scoring in this range certainly outweigh its weaknesses, it has some minor faults that certain users should be aware of. 41.8%
6.0 to 6.9 Good This range represents a product that is above average. Its strengths slightly outweigh its weaknesses, making it good for most uses but not a standout. 27.8%
So most products rate as very good. One might argue that "Very good" means average, and if 40% of all gadgets are rated so, they should be called that. ("Average" is a 5.0-5.9 on the CNet scale.)
It is, of course, complicated. Average implies, at the bottom line, that you probably won't be thrilled to own a device with such a CNet score. While "Very Good" implies you will be. Regular people will be happy with a lot of this gear, while the best gets an "Editor's Choice."
It is interesting that the video game reviews, which are much more subjective, made up the head and tails of the reviews.
The highest and lowest rated products both happened to be video games:
* 9.5 - The Elder Scrolls IV: Oblivion (PlayStation 3)
* 1.7 - Pimp My Ride (PSP)
CNet's reviews strive to be objective and reasonable, and in this, they've succeeded across the board. But I do wonder if a tighter review scale might serve the public better, along with more opinionated takes on what the best piece of gear in every category is. I mean, how often do your friends ask you what the second best set is on the market?
On top of telling them what I think is the best deal or best overall, I've long made a habit of recommending brands to friends, instead of particular models, so it's also interesting to see that ecoustics did a rundown of which brands did best, with Casio scoring lowest on average and RIM being highest (yes, over Apple.)
So, check out the article. I still use CNet for research and buying advice, and I'm sure many of you do, too, so it's good to understand that rating system. [ecoustics]
Comments
its like figure skating. maybe the scores are high because most products that make it to the market are actually reasonably capable of accomplishing the necessary tasks.
Do you change the scale to represent the high quality of products out there today, or do you simply allow products to all score in the same general range?
The reason to change the scale is that, if everything scores between 5 and 8.9, that means that a seemingly small difference (7 versus 7.5) is actually over twice as large as it seems in the metrics.
The reason to not change the scale is that CNet relies on advertising to pay the bills, and this rating system makes everyone happy (more or less), even if it is less useful to their target audience.
Yes, I have over-analyzed something that nearly everyone concluded immediately will never be changed.
Sadly, this is the kind of chart that makes CNet's advertisers jiz their pants.
Ignore the score.
It isn't going to take that much more time to read the text below the score, and you'll get a much better idea the pluses and minuses of the product.
I've never trusted reviews from Cnet, ESPECIALLY after the Gamespot's firing of Jeff Gerstmann because of his "truly honest" review. Their advertising team is intimately connected to their reviewers.
I chuckle every time a DirectTV commercial is on, touting their DVR as "better than TiVo according to Cnet dot com." Money talks, and can buy a good review (or supress a bad one) at Cnet and Gamespot.
There is also selection bias in that CNET probably declines to review products that are pure junk, just like the Times Book Reviews don't review crappy books unless they are released by "big names." I'd be curious to know if the products between 5.0 and 8.9 fall into a natural curve. If so, I think you've got no legitimate complaint here. <4 =fail, 5=bad, 7=good, 8.5=great. Now go play.
I don't think thats any kind of bias, or reflective of a broken review system, but rather a reflection of the quality (or lack thereof) of consumer goods.
This is pretty typical of most review sites. Regardless of what they say they rate on a grade scale. 60-70% is a D 70-80% a B and so on. In absolute terms most products do more right than wrong so this makes sense. However we really care how they rank in relation to each other, no absolute. I think a "5 star system" works better in gerneal. Sure its less granular, but I think reviews are more likely to give crap products 1 or two stars while the average is more around 3.
Not all that surprising. As someone who has an unhealthy gadget fetish (as just about everyone who surfs giz)i still think that there site is invaluable. Sure there reviews aren't always spot on but they're still relatively objective. Moreover, and most importantly, the 'reader reviews' are typically where you find the goods. I spend at least 20mins going through them for all my big purchases. Usually, the readers are even more objective. Or, just out of there minds. In any case, I've yet to be disappointed.
I rate this article at 7.9. Please send my cheque to the usual address.
Well, if you're expecting a wider gaussian distribution, I doubt that's going to happen. It simply means that most companies have decent QA, some with superior QA, and some with shitty QA.
@bandit: Great point.
@Don Is: Cheap shot. I don't think any one has any grounds accusing CNet gadget reviewers of being anything but honest.
As for CNET and there absolutely, for sure, guaranteed virus and trojan free downloads, I rate them a smoking hole in the ground where their credibility crashed some time ago.
I'd also like to take this time to thank them for all the trojans and viri they tried to install on my system every time I was desperate or stupid enough to try and download something there.
That makes sense - most products big enough to be reviewed range from decent to pretty good as far as most people who look at them will say. Rarely is any product the ultimate everyone-pleaser, and most products that are total crap are rare, and obvious enough you won't find them for sale at most places.
I really think they should adopt a similar rating system to what the big brokerages use for recommending (or not) stocks. They usually have one of three ratings, Buy, Sell or Hold (or some variation thereof). Cnet could do something similar. How about Buy, Don't Buy or Wait for V2.0?
Comment on CNet Scoring System Analyzed; 70% of all gadgets between 6.0 and 7.9 This also makes comparing reviews between sites very difficult. But I think that the number itself shouldn't be taken all that seriously, its the relative score that matters. How does a product score compared to others like it is what really matters. Unless of course you just want to buy something and you don't care what.
Where's the bell curve, useless statistics without it...
Some of their reviews dont make sense, especially in their cell phone section, it think the problem is that they dont have a standard, except maybe in their prizefights. Plus kent German depresses the hell out of me. Its like he hates cell phones but he got stuck with that job.
@BensAngel:
If you widen the distribution you get a more normalised bell curve.
The point's already been made: CNET don't review absolute shite so there is a weighting towards "average to good".
I have never found CNet to be useful past price comparisons. Their reviews are barely informative and the product comparison feature on the site ultimately sucks.
How are user scores distributed?
For products that receive similar scoring from users and editors, it'd be hard to argue with the results.
If there's no correlation, CNET needs a new scoring system.
I think it might have to do with multiple releases of very similar products, look at computers most are the same and take a look as digital cameras most of the updated cameras aren't much better than before and many of the previous were fine.
Why would anyone trust a review from CNET? We have all read about the reviewer who resigned after management told him to give the product a better rating.
CNET and JD Powers - two places where it's well known you can *buy* a good review.
Comment on this post
Reply by EmailLogin with your username and password below. Or comment on this post via email.
Forgot your username or password? New User?