Crunching the Numbers: Firefox 4.0 Pulls Ahead

After last week’s benchmarking project, I combed through the comments and ran the numbers myself using the latest Firefox 4.0 beta, Chrome 10, and Opera 11. The results? It looks like Firefox 4.0 has the speed crown on Linux, for now.

If you look through the comments on the Weekend Project, you’ll see a lot of results are standalone browsers or one or two browsers — interesting, but not apples-to-apples comparisons since you can’t compare Opera 11 on one machine to Firefox 4.0 on another machine. But a few of the folks who did the “homework,” did tally up Firefox 4.0, Chrome 10, and Opera 11. And I did as well, taking a few hours on Monday of this week to run each browser through the tests on a system running Linux Mint 10.

Here’s the setup I used: Linux Mint 10 64-bit, Core i7 Q720 @1.60GHz, and 8GB of RAM. I tested Firefox 4.0b13, Chrome 10.0.648.134, and Opera 11. Here’s the results I got, with the winners in bold:

SunSpider

• Firefox 4.0b13: 296.3 ms
• Chrome 10: 362.4 ms
• Opera 11: 358.6 ms

V8

• Firefox 4.0b13: 3928
• Chrome 10: 4637
• Opera 11: 3496

Kraken

• Firefox 4.0b13: 6040.9 ms
• Chrome 10: 14514.0 ms
• Opera 11: 12244.5 ms

Not surprisingly, Chrome comes out ahead on the benchmark the Chrome team uses to test Chrome, and Firefox comes out well ahead on the benchmark that the Firefox team is using. (I’m not suggesting that the benchmarks are biased — I’m saying that the teams are running different tests and tuning slightly different things, which means they perform better on the things they test.)

To run the tests, I shut down all other applications (not counting things like NetworkManager, obviously, or background applications like Tomboy). Then I started each browser, ran each test three times, and then exited the browser. The scores shown are the best of the three. Also, I used the Firefox 4.0b13 from the Ubuntu PPA rather than the 64-bit build from Mozilla. I did test that one using SunSpider and found that it was in the same neighborhood as the PPA-provided Firefox 4.0b13.

Bottom line? It looks like Firefox is nosing ahead just slightly. But really? There isn’t a lot of daylight between the major browsers.

Thanks much to all the folks who submitted “homework,” too. It was interesting to see the scores for some of the browsers I haven’t tested, like Midori and Konqueror. Please be sure to skim through the comments and add your own results if you have time.

With three more Firefox releases slated for 2011, I’ll be keeping a close eye on the benchmarks as well as looking for more tests to throw at the browsers we use on Linux.