The Basic Differences in Chips

First off, I should note that the Core i7 chip has what Intel calls a "turbo mode." That is, when it's not utilizing all of its cores, it can dynamically overclock itself up to 3.4GHz on whatever single core is in use. It can, as shown in this video, work in steps. So you get the turbo benefit when using some of the four cores in this iMac's chip, but you also get it when all cores are being partially used. For example, if four cores are running but only at a fraction of their total capacity (less then 100%), the cores can use that electrical/thermal overhead to overclock to varying degrees. This should theoretically make up for the difference between the two-core 3.06GHz chip and the hyperthreaded quad core chip at a base of 2.8GHz.

The other thing to realize about these newer Core i7 chips are that they have no northbridge—or bus—between the memory and CPU. The memory controller is built right into the processor, and there's a new tech called QuickPath interconnect which connects the cores in a point-to-point architecture. Core i7 supports triple-channel memory (which would use three banks at once), but this iMac only came loaded with two banks of RAM filled. Like our other iMac, that's a 2GB + 2GB arrangement.

Matt explains more about i7 here. (And yes, there are differences between i7 and i5, besides clock speed.)

*Note that this machine also had a faster ATI Radeon 4850 video card with 512MB of RAM (versus the 4670 card in the other iMac) which may have impacted performance in several apps. I have no idea which of these apps uses the GPU to accelerate its tasks under Snow Leopard. (For example, Preview may use it to help render JPGs faster, or it may not. Apple could not tell me. In Adobe After Effects, the Radeon series of cards apparently is not supported for OpenCL acceleration. )

Performance with Multithreaded Apps

In short, any task we tried that expressly was written to either a) take advantage of multiple cores, or, b) take advantage of multiple cores through Snow Leopard's multicore middleware, Grand Central Dispatch, were 2 to 3 times faster. (More on that here.) These results include:

• 64-bit versions of Geekbench, which focus on CPU and memory tests.
• Adobe After Effects benchmarks
• Opening 20 images of Tokyo Tower that are 2000x2000 pixels and 35MB each.

Impressive stuff, but honestly, those tests were kind of uninteresting to me. I mean, those tests don't really have any correlation to my daily computing use. So on a whim, after benchmarking, I tested Handbrake, the DVD ripping software I love. It, too, was freaking fast.

I know the app is multithreaded, but I did not know what level of optimization it was written for. I was blown away by a 3x speed multiplier with the i7. On the Core i7 iMac, it took 43 minutes to rip a DVD, Storm Riders, a surfing film from the '70s featuring Gerry Lopez (my favorite) and others. On the Core 2 Duo machine, it took 147 minutes! I know this is basically a DVD read test coupled with decoding and video conversion, but the results have me excited because this is a real task that takes my computer a long time to do, performed by a program that hasn't been revised in a year.

Performance With Single-Core Optimized Apps (Otherwise Known as Reality)

Unfortunately, there are still very few applications that take advantage of multiple cores directly or via Snow Leopard's GCD, not even video-based, let alone general purpose computing.

Photoshop CS4 on the Mac, which is not set up to handle multicore processors, showed almost less than a 3% improvement using the Driver Heaven benchmark. Basic tasks, like booting and shutdown, saw virtually none. Playing the 1080p Quicktime trailer of Avatar consistently showed that the i7 was using 3% less of its total CPU than the Core2Duo, but I wonder if that's a result of the faster graphics card kicking in using CoreCL. Xbench, the old program that does a more comprehensive job of benchmarking a system from disks to processors, showed almost no difference.

I think Xbench, which hasn't been updated in years, is a solid benchmark for that old program that you depend on but has been long abandoned or at least ignored by its developer.

These scores, again, are in relation to the top line 3.06GHz Core 2 Duo iMac we tested. Some benchmarks have come in from the web comparing the i7 to the i5. Here's one that claims a 30% jump using Geekbench. Now we know Geekbench likes and does well with more cores and is a synthetic CPU test. But if the i5 is 30% slower, and the i7 pulls even with the 3.06 GHz Core 2 Duo chip in single threaded activity—most day to day activity—does that mean the i5 is slower than the cheaper Core 2 Duo? Maybe. Probably not 30%, since Geekbench is strictly CPU/memory and likes more cores, and this stuff does not translate so literally in the real world. But we can assume the i5 will have 30% less jump from the top tier Core 2 Duos, translating into a mere 1.3X to 2X speed increase from last gen chips on programs that like cores.

Value

For the most part, in our review, I said that you should stick to the preconfigured options, upgrading to Apple's next recommended config before considering upgrades to the lower tier models. How does that advice change now that we've seen the i7? I don't know! I guess it depends if you're a betting man. If you think programs for Snow Leopard using GCD are coming, paying $200 to $500 bucks more from the top line Core 2 Duo chip for an i5 or i7 might make sense. The probability of you getting programs that can use those extra cores goes up if you are a graphics or video professional who expects to see support from Adobe, Apple, etc. (Apple already claims big jumps in Aperture that we weren't able to test.) Or if you rip a lot of DVDs! The rest of you? The Core 2 Duo stuff could be fine for today and fine for tomorrow. But the Core i7 is not worse for today and will definitely be faster tomorrow. It just costs more.

Me personally? I'd opt for the Core i7. I just might wait til the new iMacs refresh a bump and the i7 is cheaper and part of a standard build. But I'm patient like that.

[iMac Review]

Specifically, using his charts and mine, it wasn't hard to recognize the jump in the multithreaded, 64 bit results from geek bench in the categories of integer, floating point and memory streaming tests, as well as the threaded tests. (Memory tests were slightly faster, the others were drastically so.) Interesting, as the Core i5 chip is clocked at 2.66GHz and the Core2Duo iMac I tested runs at 3.06GHz.

(The turbo boost function, which overclocks the Core i5 chip to up to 3.2GHz when running non-multithreaded apps, should be kicking in performance here, too.)

Interesting, but two things to remember: Core i7 chips are coming out for the iMac shortly and will run at 2.8GHz and have hyperthreading so the 4 cores emulate 8. And there are still not many (if any at all) major OS X apps that can take advantage of Snow Leopard's multicore support. [Electronista's tests, Gizmodo's iMac Review]

The 27-inch iMac we're testing is a nice machine, but the specs—3.06GHz Core2Duo CPU with 4GB of 1066MHz RAM and an ATI Radeon HD 4670 graphics set up—are the stock low end parts for that size. And that chip is the higher end (built to order) CPU of the previous generation. (The graphics situation is weirder. The ATI card here is better than the stock 9400M NVidia setup of the old base 24-incher but not better than the built to order ATI 4850 option; the 4850 remains the top end choice for this generation's iMac, too.) These quick test results show a few changes, but, well, there are plenty of issues that nullify any meaning that can be interpreted beyond "duh". (Longer bars are better in both charts.)





First of all, XBench is just falling apart as a test these days, having being revised a long time ago and taking advantage of zero of the latest OSX technologies. Also, from what we've seen, XBench scores take a hit in the openGL rendering on Snow Leopard, compared to Leopard. The new machine seems slower than the old 2.66GHz iMac we tested last year (With OS X 10.5) in XBench in OpenGL and overall because of Snow Leopard, not the new computer. Plus, Xbench is just really, really old. I don't really trust these results, especially running between two operating systems.





Secondly, we used Geekbench. Geekbench runs in 32 and 64-bit mode in Snow Leopard and has been updated to take advantage of varying number of cores. (It's multithreaded better than most software and not surprisingly given the simple nature of a synthetic bench). As you can see, though, Geekbench only tests core system tests focusing around CPU/Math/Memory performance. And here, the faster chip has the advantage, apparently in 64-bit mode, too. This isn't surprising either and the numbers don't jump off the page.

So, you'll have to wait for us to test Core i5/i7 machines with ATI 4850 graphics and—Apple willing— 16GB of RAM before we can comment beyond the fact that this machine is prettier by 45% than the last generation of iMac.

But, even if we had that machine, the quad core CPUs don't have enough software beyond native Snow Leopard apps to really take advantage of the extra cores. There's always turbo mode, which bumps utilized cores up in speed when software isn't running across all four channels, but you're talking about chips that run slightly slower clocks than Core2Duos, so its up in the air how turbo that turbo can get.

Point being: I have nothing for you. More soon.

Of course, you can argue that Windows is not as optimized as Mac OS X in that machine. On the other side, Apple's Intel-based hardware is really not that special. This shows in the gaming test, where Call of Duty 4 squeezes 5 more frames per second in Windows 7:

In other tests, however, Snow Leopard consistently beats Windows 7 running on this machine. Especially painful is the battery life test:

This one, however, can really be attributed to bad drivers, since the author of the tests says that he "was able to get just around an hour and a half with Windows 7 with general usage on the same machine" running Boot Camp 2.1 instead of the 3.0 version he used for the test. [CNET]

Why Do Benchmarks Matter?

Benchmarks typically measure the performance of the bottlenecks in your system. Benchmarks of your car measure its speed, braking and cornering. Benchmarks of your mechanical toothbrush measure the percentage of plaque it can remove from your teeth. As you attempt to test more complex systems, it becomes increasingly more difficult to create accurate benchmarks. These days, computers can be very difficult to test accurately.

On paper, making a great benchmark seems simple—it should be a quantitative test that measures something meaningful, delivers correct results and produces similar results when repeated in similar circumstances. However, in the real world, it can be difficult to find a test that fits all three criteria. Worse, it's relatively easy for anyone with an agenda to change the starting variables enough to manipulate a benchmark's results. It's more important than ever for you to know the difference between good and bad benchmarks—especially if you want to avoid being hoodwinked.

There are dozens of examples of benchmark shenaniganry over the last decade, but I'm going to pick on Nvidia. In 2008 Nvidia famously claimed that high-end quad-core CPUs were overkill, and that the GPU could do everything the CPU could do better and faster. As is frequently the case, there was a demo to sell the point. Nvidia was showing a video transcoding app that used the power of Nvidia GPUs to convert video 19x faster than a quad-core CPU. However, the application used for the CPU part of the comparison was only able to utilize a single core on the CPU, an unusual situation for video conversion apps even then. When the exact same test was run using an industry-standard software that could use all four CPU cores, the performance difference was much less dramatic. So, while Nvidia created a benchmark that really did work, the results weren't indicative of the actual performance that people in the real world would get.

The Lab vs. The Real World

There are two basic types of benchmarks: synthetic and real world. Even though we tend to favor real-world benchmarks at Maximum PC (where I am editor-in-chief), both types of tests have their place. Real-world benchmarks are fairly straightforward—they're tests that mimic a real-world workflow, typically using common applications (or games) in a setting common to the typical user. On the other hand, synthetic benchmarks are artifices typically used to measure specific parts of a system. For example, synthetic benchmarks let you measure the pixel refresh speed of a display or the floating-point computational chutzpah of a CPU. However, the danger of relying on synthetic benchmarks is they may not measure differences that a user would actually experience.

Let's look at hard drive interface speeds, for instance. Synthetic benchmarks of the first generation SATA interface showed a speedy pipe between SATA hard drives and the rest of the system—the connection benchmarked in the vicinity of 150MB/sec. When the second generation SATA 3Gbps spec was introduced, tests showed it was twice as fast, delivering around 300MB/sec of bandwidth to each drive. However, it wasn't correct to say that SATA 3Gbps-equipped drives were twice as fast as their first-gen SATA kin. Why not? In the real world, that extra speed didn't matter. If you tested two identical drives, and enabled SATA 3Gbps on one and disabled it on the other, you'd notice minimal—if any—performance differences. The mechanical hard drives of the era weren't capable of filling either pipe to capacity—a higher ceiling means nothing when nobody's bumping their head. (Today, SSD drives and even the large mechanical disks can saturate even a SATA 3Gbps pipe, but that's a topic for another day.)

So, real-world benchmarks are perfect, right? Not necessarily. Let's look at the Photoshop script we run at Maximum PC to measure system performance. We built a lengthy Photoshop script using dozens of the most common actions and filters, then we measure the time it takes to execute the script on a certain photo using a stopwatch. It's a relatively simple test, but there's still plenty of opportunity for us to muck it up. We could use an image file that's much smaller or larger than what you currently get from a digital camera. If we ran the script on a 128KB JPEG or a 2GB TIFF, it would measure something different than it does using the 15MB RAW file we actually use for the test.

So, how do we know that our Photoshop benchmark is delivering correct results? We test it. First, we run the benchmark many times on several different hardware configurations, tweaking every relevant variable on each configuration. Depending on the benchmark, we test different memory speeds, amounts of memory, CPU architectures, CPU speeds, GPU architectures, GPU memory configurations, different speed hard drives and a whole lot more; then we analyze the results to see which changes affected the benchmark, and by how much.

But by comparing our results to the changes we made as well as other known-good tests, we can determine precisely what a particular benchmark measures. In the case of our Photoshop script, both CPU-intensive math and hard disk reads can change the results. With two variables affecting outcome, we know that while the test result is very valuable, it is not, all by itself, definitive. That's an important concept: No one benchmark will tell you everything you need to know about the performance of a complex system.

Making Your Own Photoshop Benchmark

Once you get the hang of it, it's never a bad idea to run your own benchmarks on a fairly regular basis. It will help you monitor your machine to make sure its performance isn't degrading over time, and if you do add any upgrades, it will help you see if they're actually doing anything. Just don't forget to run a few tests when your computer is new (and theoretically performing at its peak), or before you swap in new RAM or a new HDD or other parts. If you forget, you won't have a starting data point to compare to future results.

If you don't own an expensive testing suite like MobileMark or 3DMark, don't sweat it. If you have an application that you use regularly and can record and play back macros or scripts, like Photoshop, you can build a script that includes the activities you frequently use. We run a 10MP photograph through a series of filters, rotations and resizes that we frequently use as one of our regular system testing benchmarks at Maximum PC.

To make your own, launch Photoshop and open your image. Then go to Windows —> Action, click the down arrow in that palette to select New Action. Name it and click Record, then proceed to put your file through your assorted mutations. Always remember to revert to the original file between each step, and make the final action a file close, so you can easily tell when the benchmark is done. Pile in a lot of actions: As a general rule, you want the total script to take at least two minutes to run—the longer it takes, the less important small inaccuracies on your stopwatch work matter. When you're finished assigning actions and have closed the file, click the little Stop button in the action palette to finish your script.

Once finished, make sure your new action is highlighted, then click the menu down arrow in the Action palette again and select Action Options. Assign a function key, which will let you start your benchmark by pressing a keyboard shortcut. (We use F2.) Then, open the Action palette menu again, and select Playback Options. Set it to Step-by-Step and uncheck Pause for Audio Annotation. Once that's done, ready your stopwatch. (Most cell phones include one, in case you aren't a track coach.) Load your image, then simultaneously start the stopwatch and press the keyboard shortcut you just selected. Stop the stopwatch when the file closes. We typically run this type of test three times, to minimize any human error we introduce by manually timing the test. If you want to try the same script we use at Maximum PC, you can download it here.

Gaming Benchmarks

Additionally, if you're a gamer, there are tons of games with built-in benchmarks. These help you know what settings to run in games to maximize image quality without sacrificing framerate as well as measure the impact of use on your computer's overall speed.

Check out Resident Evil 5 benchmark, which includes both DirectX 9 and DirectX 10 modes. Running this test is easy—simply install it and select DirectX 9 or DirectX 10 mode. (Remember, you'll need a Radeon 4800 series card or newer or a GeForce 8800 series card or newer and be running on Vista or Windows 7 to use DirectX 10 mode.) If you want to compare performance over a period of time, we recommend the fixed run, it's simply more repeatable. If you're trying to tell what settings to use, the variable mode isn't as consistent, but it shows actual gameplay, which will be more representative of your in-game experience. Once you're in the game, you'll want to change to your flat panel's native resolution and do a test run of your benchmark. For a single-player game, we like to choose settings that will minimize the framerate drops below 30fps. For multiplayer, we sacrifice image quality for speed and target 60fps. After all, dropped frames in a deathmatch will get you killed.

The Practical Upshot

Like everything else, there are good benchmarks and bad benchmarks. However, there's absolutely nothing mysterious about the way a benchmarking should work. In order to know whether you can trust benchmarks you read online, you need to know exactly what's being tested—how the scenario starts, what variables are changed and exactly what's being measured. If you can't tell that a test is being run in a fair, apples-to-apples manner, ask questions or try duplicating the tests yourself. And when someone doesn't want to share their testing methodology? That's always a little suspicious to me.

Will Smith is the Editor-in-Chief of Maximum PC, not the famous actor/rapper. His work has appeared in many publications, including Maximum PC, Wired, Mac|Life, and T3, and on the web at Maximum PC and Ars Technica. He's the author of The Maximum PC Guide to Building a Dream PC.

Booting the old touch took 31 seconds. The new touch takes 19. Loading a web page dropped from 34 seconds to 15. And most games teetered between loading 33% and 50% faster.

Despite Apple not reaching that 50% benchmark across the board, Macworld is still impressed because the "the new iPod touch feels much faster at any task you throw at it: applications launch (and quit) faster, Web pages load more quickly, processor-intensive games and programs perform better-you name it."

And the new touch should be faster. TUAW confirmed that this latest ipod to have a very similar ARM Cortex A8 processor as the quick iPhone 3GS, which is a surprise to absolutely no one. [Macworld and TUAW via The iPhone blog]

Quicktime tests show the amount of CPU being used while playing back the bad ass trailer from James Cameron's Avatar at 1080p. that Quicktime 10, on Snow Leopard, is easily more efficient than old versions of Quicktime. Even on non supported H.264 hardware, like the Macbook Pro's 15-inch 8600GT card, it just works better in QT10. Note: "15-Inch PowerBook" is an error; should read 15-Inch MacBook Pro—all test machines are Intel-based


Geekbench is a synthetic benchmark testing cpu and memory performance. Snow Leopard, Apple claims, runs math much faster and geek bench tells that same story here when we use the 32 and 64-bit versions under Snow. Note: "15-Inch PowerBook" is an error; should read 15-Inch MacBook Pro—all test machines are Intel-based


Xbench is an older piece of software that tests an entire system, from cpu to disks to graphics. Oddly, Open GL performance on the 13inch Macbook was half that of what we saw in Leopard, which caused the score to drop a bit in Snow Leopard. Math processing was faster, however. Note: "15-Inch PowerBook" is an error; should read 15-Inch MacBook Pro—all test machines are Intel-based



Installation times were about 30% less in Snow, but the size of the install also dropped from 16GB to 10GB, so that makes a lot of sense to me. Common sense.


Zipping the same file up we used in our Quicktime tests showed Snow Leopard as faster, perhaps due to the improved math processing performance.
Using Google's v8 suite for javascript speed testing, Snow Leopard was faster using Safari 4 than Leopard. Firefox, for the record, isn't even close. Chrome might be, though.
Initial Time Machine Backups to a USB Drive were faster under Snow, too. Note: "15-Inch PowerBook" is an error; should read 15-Inch MacBook Pro—all test machines are Intel-based

Yes, it's smaller, which I believe is a first for an OS upgrade. How? They lost about 6GB (Apple claims 7GB) by ditching printer drivers and installing them on demand over the internet and all the binaries are now intel-only, ditching the Power PC support for good. Power PC apps still can run, though, by downloading or installing Rosetta virtual support for PowerPC apps from the install disc.

Notice there's no difference in these results between Snow and old Leopard. The reason? Handbrake hasn't been optimized for Snow yet.

These numbers are based on Javascript benchmarks, which don't give a total view of a browser's speed but do tell us how adept a browser is at dealing with intensive code. Chromium scored 657ms on the test to Safari's 886ms. Firefox scored 1,508ms and Opera 10 Beta 3 (my personal browser of choice) lagged way behind with 5,958ms. Keeping in mind that Chromium is pre-alpha and thus better seen as a fun dev project than an actual candidate for a primary browser, we're pretty excited. Once Google irons out the bugs and gets some damn extensions, Chrome on Mac is going to be a stiff challenger to Firefox. [CNET]

Having run similarly obsessive benchmark roundups before, Testfreaks is no stranger to flash drive testing. They're quick to show us, then, that these drives are generally pretty terrible, and that if you're looking for performance, you'll probably be better served by a decent mainstream drive over a plasticized shrimp. One gripe throws a damper on this wonderful test: I kinda wish there were a few more of the flash drive legends, like the beer drive, the humping dog and for good measure, Sylvester Stallone's genitalia. [Testfreaks]

* The CPU performance is Faster by 40-70%
* The fillrate* is 3x to 4x higher
* Texture effects and filters are about 10x faster

Keep in mind that it's also running OpenGL1.x on the benchmark test, and iPhone 3GS can run OpenGL 2.x. The upside is that when games do get optimized or developed for OpenGL 2.x, that's when your old phone will no longer be able to play them, or at least play them with better graphics. It's pretty obvious once you play with the phone that it's rendering your old games much faster, but it's good to get it in quantitative form. [glbenchmark via Extremetech via Ubergizmo]

In their review of June's upclocked, marked-down MacBook Air, MacWorld noticed some odd benchmark results. Compared to the Late 2008 MacBook Air, clocked at 1.86GHz, the new top-end model, clocked at 2.13GHz, couldn't quite keep up. This is odd, since the rest of the specs have remained basically static, and the only difference between the processors is clock speed.

Confirming their suspicions, they found the June 2009 1.83GHz Air to be markedly slower than the Late 2008 1.83GHz model, despite nearly identical specs. So, what the hell?

The obvious answer here would be some kind of firmware change, since it doesn't make sense from a hardware perspective. MacWorld speculates that the newer Airs could have more aggressive thermal management features, which throttle the processor when it gets too hot—a theory somewhat supported by the fact that high-stress benchmarks showed a proportionally greater performance decrease than easier ones.

Another possibility in the same vein: Underutilizing faster processors simply gives Apple better power consumption and heat results. Problem is, that doesn't explain why the matched processors perform differently. Also, Apple's whole pretense that the new Airs are faster than the old ones would have been a intentional, egregious lie.

Until these benchmarks are replicated and examined, we'll have to remain a little bit skeptical. But if they can be, then Apple's got some splainin' to do, I think. [MacWorld via MacRumors]

Their conclusion:

Platter based hard drives and high-end solid state drives, all run faster on Windows 7. Solid state drives see the largest performance boost, which showed up to a 35% improvement in read performance and up to a 23% boost in write performance.

They also found serious jumps in burst read performance, which explains why, given a general speed difference of about a third, Windows 7 feels so much quicker than Vista or XP. Obviously, they weren't testing the netbook edition, but I doubt this particular metric will differ between versions. Convinced yet? [HotHardware via Slashdot]

These results from DirectX 9 testing speak well for Windows 7, topping Vista on the lower and higher spec systems while, much of the time, just about keeping pace with XP. In fact, with Intel's i7 quad core processor, we actually see a brief moment of performance gains over XP. Blasphemy!

Crysis DirectX 10 testing did not go over as well for 7, which got pretty trounced by Vista (though 7 won a DirectX 10 round later with Far Cry 2). We chalk these inconsistencies not only to different games but to the fact that both Windows 7 and its graphics drivers are still in beta. Overall, early performance testing of Windows 7 gaming leaves us optimistic, even if there's still plenty of room to grow. [Firing Squad via Kotaku]

The new maximum component score is a 7.9. Microsoft has (and will continue to) raise the cap as new hardware components come out. If a current component score is a 5.6 under Vista, it will continue to be around a 5.6 (give or take) unless it was subject to various tweaks and feedback changes Microsoft took when adjusting the index, such as new disk tests.

For gaming...

...scores in the 6.0 to 6.9 range to support DX10 graphics and deliver good frames rates at typical screen resolutions (like 40-50 frames per second at 1280x1024). In the range of 7.0 to 7.9, we would expect higher frame rates at even higher screen resolutions.

But still, it's a bit of a clunky way to determine how well your graphics card will perform in newer games. Ballpark, sure, but nothing specific. For that, you should use more dedicated benchmarks like 3DMark.

The test is by no means exhaustive, as there are many hundreds—if not thousands—of USB drives on the market today. The test did include popular sticks from recognizable brands, as well as some budget and novelty pieces that you might be likely to pick up. The results were somewhat predictable: sticks from reputable companies like OCZ, Lexar and Sandisk offer greater read/write speeds, and sticks in the 4GB range perform consistently well.

There were some surprises, namely the standout performance from OCZ units and the plodding, about-as-fast-as-it-would-be-if-it-stored-data-on-actual-wood Brando Wood drive. And results aside, I've derived some comfort from the fact that out there somewhere, there exists a man named Kristofer Brozio who is willing to actually run a dozen time-consuming benchmarks on over 20 USB keys. Check out the full results at the source link. [Test Freaks via Engadget]

Early on, we showed you completely non-scientific evidence of Win 7's pleasingly fast boot time. (Shutting down is another matter—my build (6801) sometimes takes forever.) I was glad that Kingsley-Hughes—using a Phenom 9700 quad-core system with ATI Radeon 3850, 2GB of Corsair Dominator RAM and WD's 10K RPM Raptor as primary drive—managed to demonstrate that the fast boot isn't a fluke. By the way, Vista SP1 had the slowest boot.

In two other tests, PassMark Performance and PCMark Vantage, Win 7 pre-beta beat the Vista builds, though it failed to trounce them in the CineBench R10 test.

Remember, this is a pre-beta, so nothing is guaranteed, but what makes this newsworthy is that Kingsley-Hughes—who incidentally is in no way a Bond villain—ran similar tests with Vista a few years back, and early Win 7 makes a mockery of that noise. Check the ZDNet article for the full system specs and benchmark scores—I'm sure at least some of you will want the nitty gritty. [ZDnet via Lifehacker]

You probably only read Consumer Reports if a) you are at your grandparents house or b) you are a grandparent yourself. But that's too bad, because tucked quietly away in the NYC suburb of Yonkers lies one of the biggest and best electronics testing labs money can buy. And what goes on here at Consumer Reports main test facility probably puts most other tech pubs to shame.

We got a chance to look at all of the top dollar gear used to put everything found in CR's electronics pages in a complete vacuum of testing, basically removing every possible outside variable to test the pure hardware performance. That means anechoic chambers built on their own foundation (at a cost of $2.5 million in 1980) for total sound isolation; industrial-quality cell tower base station generators inside fully RF-shielded rooms that can crank out every possible mobile phone frequency at any strength; a "head and torso simulator" named Pedro, able to be calibrated down to the millimeter for testing every aspect of cellphone call quality possible, and a nameless human finger simulator composed of, well, meat (in action below as well). See our captioned gallery for a closer look:

Unfortunately, what makes CR so exemplary as a reliable testing lab also contribute to its fate to be found mostly on grandmother's end table next to the bowl of fossilized peppermints. As a non-profit organization, CR doesn't sell any advertising to anyone, anywhere, nor do they accept any review units or advance loaners from the company—everything they test, from a new BMW to an electric toothbrush, they buy.

While that means employees get pretty sick re-sale discounts on new cars every year, it also means CR is fighting an eternally uphill battle vs. the other tech pubs that don't keep such high standards, and that CR must keep all of its online content walled within a pay site for subscribers only. The subscribers it has are among the most loyal of any magazine, but the vast majority of them are older.

(The aforementioned human finger simulator gets put to the test on a mower that CR's resident high-RPM blade expert refers to as "the most dangerous thing i've ever tested." - video edited by BBG)

And due to the natural constraints of a magazine with no ads, the mountains of test data gathered for any particular product end up truncated and distilled into CR's famous comparative charts, where their scores are rendered in linearly receding bars and crimson doughnut dots. CR's benchmarks are designed to place all new products on a relative continuum, rating them "fair" to "excellent" in comparison to how products over the last several years have fared with the same rigorously standardized tests. But a problem there, obviously, is that often it looks like CR loves just about everything—this year's television are naturally going to present marked improvements over what's been available over the last few years, which tends to stretch the data toward the good end. Kind of like how you have to search forever find a review on CNET with a score of less than 7.0.

(Inside the soundproof womb of the anechoic chamber - video edited by BBG).

Such are the dilemmas of serious hardware testing that makes any type of claims towards ultimate authoritativeness. But it's also the reason why the old bound volumes of Consumer Reports are the most well-worn volumes in the periodicals room of the public library where I used to work. The data is there, and it's rock solid. Taking a tour of their labs and meeting the engineers that do the work, it's immediately apparent that what goes on in Yonkers is among the more vigorous and pure analysis of technology being done by anyone, anywhere.

After all, don't you just have to trust folks who keep this poster hanging above their main laptop test bench?

[Consumer Reports, video courtesy Consumer Reports, edited by the good gentlemen of Boing Boing Gadgets - see them for more]

According to Primate Labs, the overall performance in Geekbench is lower on the January 2008's MacBook Pro (with Intel Core 2 Duo T8300 running at 2.4GHz) than the June 2007 model (with Core 2 Duo T7700 at 2.4GHz) because the new machine "has less L2 cache than processor in the old MacBook Pro." However, they say that in theory the new Penryn-based notebook will give you more battery mileage as well as reduced heat at a lower price than the previous generation. [Primate Labs]

We know the MacBook and MacBook Air traded off on processor speed and memory, which is why we also threw the MacBook Pro in the mix. And we know that the MacBook and MacBook Pro aren't the fastest models available. But the computers used, as well as the tests, are ones that will be more applicable to the average Mac user.

We ran 4 different tests to measure the speed differences between the computers. MP3 encoding, video conversion, transferring a .Zip file from a thumbdrive to computer, and file duplication test on the thumbdrive. No other applications were running during the test and the computer was set up for better performance.

A few notes on testing...

The Computers:

• The MacBook Air has a 1.6 GHz custom Intel processor, 2 GB 667 MHz DDR2 RAM, and an 80 GB, 1.8", 4200 RPM HDD.

• The MacBook (a generation old) has a 2 GHz Intel Core Duo processor, 1 GB 667 MHz DDR2 RAM, and a 120 GB, 2.5", 5400 HDD.

• The MacBook Pro (also a generation old) has a 2.2 GHz Intel Core Duo processor, 2GB 667 MHz DDR2 RAM, and a custom 160 GB, 2.5", 5400 RPM, Seagate Momentus HDD.

The Tests:

• For the MP3 Encoding, we used iTunes and Seu Jorge's album Cru, which is 46 minutes long. We set up custom import settings, which were 192 kbps VBR, set at high quality.

• The Video Conversion test was done using a trailer for 300 that was 1:46 in length and 73 MB (.mov). We converted using the export option in Quicktime 7.4 that used the iPhone export preset.

• The Thumbdrive to MacBook file transfer test was done using an 803 MB .zip file and a 2GB Lexar Lightning Thumbdrive (30 MB/s read, 21 MB/s write).

• The File Duplication test was done on the Lexar thumbdrive using the same zip file used in the previous test.

And if you're curious, here are the Xbench results for the MacBook Air. I'll let you do the honor of comparing it to your own computer.