If you are looking to buy a refrigerator or other boxy, new stainless steel-clad appliance, there is likely no better—or at least better respected—reference manual than Consumer Reports. It is theoretically as unbiased as a publication could be. It takes no ads, buys all its own review units*, and subjects them to rigorous, super-standardized testing.
The core belief of the Consumer Reports system is that everything can be objectively evaluated, reviewed and assigned a definitive numerical score denoting its worth (or lack thereof). Objectivity is the religion of Consumer Reports. And that works well for a lot of things! Refrigerators and washing machines, for instance. Or specific attributes of products, like a computer's battery life.
There are lot of things it doesn't work so well for, though. Like where software is heavily involved. But even for the things that a purely "objective" review system doesn't work so well for evaluating, the bias of this kind of rigid system is apparent and its limits easily accounted for. For the system to exist, it cannot acknowledge that there are things it cannot evaluate. The foundational belief of the system is that it can evaluate everything, and what it determines in the end is the objective truth.
So, if you go back to June, it's pretty fascinating that Consumer Reports—rigorous and objective Consumer Reports!—hastily recommended the iPhone 4 and then just as quickly retracted its own recommendation. At best, it means Consumer Reports recommended the phone without fully testing it. The reason for the retraction: the iPhone 4 antenna's weakspot. It was very likely the tipping point for Apple, leading to a hasty press conference where it tried to assure the world how truly awesome the iPhone 4's antenna was.
Fast forward to today. In Consumer Reports' Spring electronics buyers guide, the iPhone 4 is its highest-rated smartphone on AT&T, tying with a handful of other phones—Samsung Captivate, Vibrant and myTouch 4G—for the highest score across any carrier, with a score of 76. Every phone with that score—and several phones with lower scores—carries a Consumer Reports recommended rating. Except for the iPhone 4. Consumer Reports even recommends the older, virtually obsolete iPhone 3GS, despite its worse screen, slower guts and fatter form factor. (The Verizon iPhone 4, despite being announced 2 months ago, is missing entirely from the ratings.)
Obviously, the normally mild-mannered Consumer Reports put itself in an interesting position by staking out such a bold position on the iPhone 4. It's hard to back down from that. But is that position really in the best interest of consumers? The iPhone 4 is better than many of the phones it recommends, deathspot or no. (Android fans, let's at least concede it's much better than the iPhone 3GS. The $100 price difference is negligible, considering the thousand-dollar-level expenses of a 2-year contract.) Is the occasional annoyance of the deathspot a true dealbreaker in a phone that is objectively better than most phones in screen quality, battery life, apps and speed? Maybe so. But even Consumer Reports own scores say at face value that it's one of the best phones, period. Yet they will not recommend it. (And nowhere in the buyers guide have we found a reference to the iPhone 4's antenna problem, or an explanation as to why they do not. Maybe I missed it! Let me know if you spot it.)
Why does it score so highly if it is so broken? Why can't the scoring system account for such a show-stopping flaw? If it can't, what does that say about the value of the scores themselves? If Consumer Reports will not recommend its objectively highest-scoring phone, what does that say about objective reviews?
*A tiny update: I've been told that some consumer electronics do in fact give review units to Consumer Reports for testing. And indeed, Consumer Reports' wording does allow some looseness in the policy that they buy every review unit they test (emphasis mine):
As with all test programs at Consumer Reports, the Electronics program buys virtually all products at retail, like any consumer.
P.S. I'm not actually trying to shit on Consumer Reviews in particular, but I do want interrogate, what exactly, is an objective review?