To me the DxOMark sensor rating seems a bit bizarre. They recently rated the sensor of Nikon D3300 to the same score as the Canon 1Dx got which I find unlikely to reflect their performances, and which also made me really skeptical of their scoring system.
I do value their measurements of the sensor, which results in a benchmark tool for comparing different camera sensors, but are they really relevant? As far as I know there is no documentation of how the score is computed from Dynamic Range, Color Sensitivity and Low-light ISO scores. Also DxO Labs' interest lies in selling their software, not directly in giving scientific benchmarking tools and could very well be biased towards a certain manufacturer.
Just how useful is the DxOMark scores and how can you use it to compare different cameras?
Answer
DXOMark primary "scores" are utterly useless. IGNORE THEM. It is a futile effort to try and reduce a complex entity such as a DSLR to a single, scalar number that tells you everything about it. It's a fallacy. There are too many factors to consider, and which factors are most important for a given photographer differ. A single score entirely defeats the purpose of running measurements in the first place.
When it comes to DXO's other scores, such as low light and landscape and whatnot, take them with a hefty dose of salt. Their general scores are heavily weighted, and often based on derivations of measurements rather than actual measurements. For example, the Landscape score is based on the Print DR "measure". The problem is that DXO does not actually MEASURE Print DR, in that it is not based on samples taken from actual downsampled images. Print DR is a simple mathematical extrapolation from the TRUE measured dynamic range of the sensor.
Therefor, Print DR does not really tell you anything about the sensor. When DXO says the D800 and D600 have 14.4 stops of DR, that is Print DR, which is extrapolated from the actual hardware DR, which is 13.2 stops. Same thing goes for Canon sensors. When DXO tells you the sensor has 12 stops of DR, that really isn't the case. In reality, most Canon sensors, at a hardware level, have around 10.95 stops of DR.
The problem is worse than this, however. Much of the color depth and color sensitivity scoring information is weighted, as are many of the ISO-based scores. Weights are based on cameras achieving certain thresholds, such as the SNR at a certain ISO being higher than a certain level. This grants a certain "bonus" to the score for that camera. The moment any kind of weight-based bonus scoring enters into the game, your ability to directly compare anything by any score completely goes out the window. Your now on a non-linear playing field where you honestly don't know if that Nikon camera over there with a score of 95 has been heavily weighted relative to this Canon camera over here with a score of 80.
When it comes to actual measurements, DXO information is some of the best available. Their measures of SNR, Screen DR, color sensitivity, etc. are quite sound, as it's all taken directly from multiple RAW images samples for each camera tested. Their testing methodology is fairly rigorous, and there is nothing to indicate that part of their methodology should be doubted. Scientifically, as far as how they test and what they measure, DXO has solid practices and solid information.
DXO is really a mixed bag. They may have solid testing practices, but their scoring, given that it is often based on mathematically derived, weighted information and the fact that a couple of their scores are often given "bonus" points simply for meeting certain thresholds, completely debases the entire point of what DXO does: To produce a linear score for each camera that allows cameras to be easily compared. It was a flawed concept to start with, single-number scoring, but they made it so much worse by how they handle the actual scoring process.
No comments:
Post a Comment