It looks like you're new here. If you want to get involved, click one of these buttons!
A common question that people building or buying a gaming rig have is of what processor to get. You know that the processor is important; the real question is which processors will tend to perform how well in various games. With video cards, there are benchmarks all over the place that make it pretty clear how they'll tend to perform. With processors, good gaming benchmarks are scarce to the degree that I'm not aware of a single site that I can point to and say, they've got good data that won't lead you to a wildly wrong conclusion.
The real occasion for this post is that AnandTech tried to do such an article today. I won't link to it, as it's basically worthless. But it's worth examining how they botched the testing to make the results worthless outside of some esoteric cases.
Some games tend to put more load on a video card, while others put more load on a processor. Video card reviews want games that put a heavy load on the video card, so that you won't immediately have a processor bottleneck. Otherwise, they'd say, we tested these ten cards and they all got 80 frames per second, because that's all that the CPU can handle.
There's a general principle that if you're benchmarking a type of component, you want a test in which results will depend mostly on that component. If you're benchmarking video cards, you want the video card to be the limiting factor. If you're benchmarking solid state drives, you want the SSD to be the limiting factor. If you're benchmarking USB controllers, you want the USB controller to be the limiting factor. And if you're benchmarking processors, you want the processor to be the limiting factor.
But tech sites tend to put a lot of focus on benchmarking video cards, and while they do benchmark processors, they don't put much emphasis on games when benchmarking processors. When they do, sometimes they reuse the same games that they use to benchmark video cards--that is, games specifically chosen because the CPU probably won't be the limiting factor. Oops.
And then there's the choice of what graphical settings to use. Turning up adjustable graphical settings tends to put far more extra load on the GPU than the CPU, but it does vary by setting and by game. Some settings are almost pure GPU load and the difference in CPU load is negligible--or in some cases, literally zero added CPU load over the course of a benchmarking run. Anything that you can set in video drivers and any post-processing effects are almost invariably in this category. Turning these on can be useful if you're trying to benchmark video cards. Even so, anti-aliasing and anisotropic filtering can be dangerous to turn on, as different video cards may use different algorithms that give different image quality, and thus a different relative performance hit because they're not doing the same things.
But if you're trying to benchmark anything other than GPU performance, all such settings should be turned off or to the minimum. Different sites take different approaches to CPU benchmarks. Some go to a preset low or medium settings. Anandtech went with max settings, which means that all of the pure GPU load settings were set to the maximum. Oops.
The result was that in every game they benchmarked, adding a second Radeon HD 7970 or GeForce GTX 580 greatly increased performance for many of the CPUs--which pretty conclusively proves that the game had largely been GPU-limited before. The CrossFire or SLI results are not a clean comparison to get rid of a GPU bottleneck, either, as at that point, you may well be measuring CrossFire or SLI weirdness as much as CPU performance. For analogous reasons, some sites have taken to using SSDs when benchmarking other components just so that the hard drive doing something weird won't interfere with results. That's how you end up with worthless results.
Obviously, AnandTech isn't the only tech review site out there. Tom's Hardware publishes monthly lists of the best gaming CPUs for the money, purporting to offer an answer to the question at the start. Unfortunately, their lists tend to be about as good as assuming that all current generation processors have prices in line with their performance and not looking up benchmarks anywhere. There are obvious problems with that last approach, and it's very different from the problems with Tom's Hardware's.
So it is instructive to look at where Tom's Hardware goes wrong in their CPU benchmarks. A game engine doesn't scale perfectly to n threads and no more, outside of single-threaded games. Different portions of an engine may scale to different numbers of threads. Within the rendering loop, a lot of work will scale well to many threads, but some won't--and in particular, at the start and end of a frame, you may have a substantial single-threaded component. Some work such as loading data off of the hard drive or processing network activity isn't part of the normal rendering loop, and can also affect results. Which components are waiting on which others can bounce back and forth a lot, and you could easily have the CPU waiting on the GPU one millisecond and the other way around the next.
If a game is mostly GPU bound, you'll often see results where processors with higher single-threaded performance fare a little better but not a lot better. You have to get to a situation where you're almost purely GPU bound in order for a bunch of different processors to give almost exactly the same frame rate.
So what does Tom's Hardware do? They benchmark games that don't put that much load on the CPU (remember, video card reviews get a lot more attention), note that better single-threaded performance gives you slightly higher frame rates when you're mostly GPU bound, and then conclude that Intel processors are better because they have higher single-threaded performance. So they recommend almost entirely Intel processors all up and down the lineup--and in particular, they recommend a Core i3-3220 over an FX-6300, which I think is sheer lunacy if you want a gaming rig that will last you several years.
Now, you could say, if the Core i3-3220 slightly beats an FX-6300 in situations where you're mostly GPU-bound and they're the same price, then why not get it? You'll be GPU limited in a lot of games, after all. Even if it's 100 frames per second for the former and 90 for the latter, 100 is more than 90, so even if it the difference doesn't matter much, why not get the 100? Of course, I'd argue that if you want to increase performance in situations where you're mostly GPU limited, the place to spend more is the video card, not the processor.
The problem is, what happens if you pick up a game that puts a lot of load on the CPU? For the computations that games tend to do, most things can be readily scaled to as many CPU cores as you care to. In that case, you could easily end up largely CPU-limited in a game. In such a game, you're all but guaranteed that the FX-6300 will beat the Core i3-3220, and probably by a lot--likely in the ballpark of a 50%-100% performance advantage.
So if the Core i3-3220 wins slightly in situations where the difference doesn't matter, but the FX-6300 will predictably win by a huge margin in situations where the difference does matter, which do you prefer? It's sort of a philosophical question, but even if you don't think that the situations where the difference matters will be very common, I still say that preferring the FX-6300 is an easy call.
But Tom's Hardware hasn't come across any such games in their benchmarks (and remember, games to get benchmarked largely get chosen on the basis of putting a heavy load on the GPU, not the CPU), so they recommend the Core i3-3220. In fairness, such games aren't common and won't be for quite some time; a game that struggles to run on a Core i3-3220 today wouldn't have a very big market. But it used to be that games that required a dual core processor wouldn't have much of a market; that's hardly the case today.
Now, I didn't come here to bury AnandTech and Tom's Hardware. AnandTech has some excellent articles explaining what various components do. Both often have useful hardware reviews and benchmarks. It's not like they're sites run by fanboy idiots who haven't a clue what they're talking about.
But that just points back to the title: if even some of the better tech sites have tried to come up with benchmarking methods that predict how well various processors will run games and they've failed miserably, and some sites that review gaming hardware have declined to even try, then it surely isn't easy to do.