Benchmarking: Facts and Fiction
Deceptions, benchmarks, and statistics... what they have in common is their ability to seem trustworthy even when they actually misrepresent the truth. In this article, I will discuss real world vs. sythentic benchmarks and what this means for you as you decide what or even whether to buy. First, when discussing benchmarks, we must first establish that a good benchmark measures as few changes as possible. If computer A has X CPU, X Graphics Card, and X memory, and computer B has Y CPU, Y Graphics Card, and Y memory, then the measurements are comparing the system as a whole and do not really help you to determine if upgrading from X to Y graphics card is worth it. A benchmark might depend more on the CPU or memory performance, which skews the value of the GPU. The ideal scenario is that the benchmarked computer system is 100% identical except for the part being measured. Secondly, the benchmarked system will probably not be identical or even close to your computer. A better graphics card will make a greater difference on a system with the fastest CPU than it will on a system with the slowest CPU. The reason is that on the slower system, the slower CPU creates a bottleneck that the graphics card must wait on. It can be difficult to estimate these things accurately, and most reviews/benchmarks are done on top-of-the-line or near top-of-the-line systems to create the greatest possible margin for the part being tested. Consider benchmarks in reviews to be a "best case" difference between X part and Y part, and be aware that if your system specs don't match up to the benchmark system specs, then you probably won't see quite that amount of difference. Third, every benchmark should be judged by percentage of difference. This is very important to understand. If a given part is 10% faster, it's not going to make a noticeable difference. Let's use a best case scenario. If a given CPU is 10% faster, and you have a calculation that takes 3:30 min to complete, that calculation will instead complete in 3:09. An example of this sort of process would be mp3 encoding, not gaming or business applications. If you really think you're going to notice 20 seconds over a 3.5 minute process, and you run that process all the time, then by all means, go for the performance increase ... more>>>