12/31/2019 DxOMark camera sensor testing protocol and scores - DXOMARK
DxOMark camera sensor testing
protocol and scores
[Link] 1/8
12/31/2019 DxOMark camera sensor testing protocol and scores - DXOMARK
Posted on September 11, 2017 by DxOMark
Reading Time: 10 min read
For our DxOMark camera sensor reviews, we measure the image quality performance only of camera
sensors that are capable of capturing images in RAW format, and we do this before demosaicing or
any JPG processing has taken place. You can read more about the DxOMark approach to image quality
measurement and why we base our testing on RAW image les here. In this article, we explain how we
test for different criteria in the DxOMark image quality test lab and how the test results translate into
sub-scores and the nal product scores.
We measure camera sensors for the following criteria:
Color sensitivity
Noise (standard deviation, signal-to-noise ratio, dynamic range, and tonal range)
ISO sensitivity (speed)
The DxOMark use case scores and overall score for camera
sensors
We have de ned three use case scores on which to base our camera sensor reviews — Portrait,
Landscape, and Sports — which all report on different aspects of sensor performance. Each use case
score is associated with one of the image quality metrics mentioned above, and de ned as follows:
Portrait score: Color Depth
Flash studio photography involves controlled lighting, and even when shooting hand-held, studio
photographers rarely move from the lowest ISO setting of their cameras. What matters most when
shooting products or portraits is a rich color rendition and color depth. The best image quality metric
that correlates with color depth is color sensitivity, which indicates to what degree of subtlety color
nuances can be distinguished from one another (and often means a hit or a miss on a pantone
[Link] 2/8
12/31/2019 DxOMark camera sensor testing protocol and scores - DXOMARK
palette). Maximum color sensitivity reports in bits the number of colors that the sensor is able to
distinguish.
The higher the color sensitivity, the more color nuances can be distinguished. As with dynamic range,
color sensitivity is greatest when ISO speed is minimal, and tends to decrease rapidly with rising ISO
settings. In DxOMark testing we measure only the maximum color sensitivity. A color sensitivity of
22bits is excellent, and differences below 1 bit are barely noticeable.
Landscape score: Maximum Dynamic Range
Landscape photographers carefully compose their images and choose the time of the day for shooting
in the best light. This type of photography commonly involves mounting the camera on a tripod and
using the lowest possible ISO setting to minimize noise. Unless there is motion in a scene, relatively
long shutter speeds are not an issue with a tripod. On the other hand, dynamic range is paramount.
Photographers will often aim for detail in high-contrast settings, juxtaposing a bright sky with shadowy
foliage or mountain ranges. Ideally, the dynamic range of the camera should be greater than the
dynamic range of the scene, otherwise details in shadows are lost or highlights are burned.
Dynamic range falls rapidly with higher ISO settings, as any analog or digital ampli cation performed
will increase the noise in the darker areas, making it harder to distinguish among ne levels of
contrast. Maximum dynamic range is the greatest possible amplitude between light and dark details a
given sensor can record, and is expressed in EVs (exposure values) or f-stops, with each increase of 1
EV (or one stop) corresponding to twice the amount of light.
Dynamic range corresponds to the ratio between the highest brightness a camera can capture
(saturation) and the lowest brightness it can capture (which is typically when noise becomes more
important than the signal — that is, a signal-to-noise ratio below 0 dB). A value of 12 EV is excellent,
with differences below 0.5 EV usually not noticeable. Dynamic range is an open scale.
Sports & action score: Low-Light ISO
Unlike the two previous scenarios in which light is either generous (studio) or the camera can be
mounted on a tripod (landscape), photojournalists and action photographers often struggle with low
available light and fast motion in the scene. When shooting sports or action events, the photographer’s
primary objective is to freeze motion, giving priority to short exposure times. To compensate for the
lack of exposure, photographers have to increase the ISO setting, which results in a decreased signal-
to-noise ratio (SNR). How far can ISO be increased while maintaining decent image quality? The
DxOMark low-light ISO metric will give you this information.
The SNR indicates how much noise is present in an image compared to the actual image information
[Link] 3/8
12/31/2019 DxOMark camera sensor testing protocol and scores - DXOMARK
(signal). The higher the SNR value, the better the image quality, as detail is not drowned out by noise.
The SNR is given in dB, which is a logarithmic scale: an increase of 6 dB corresponds to doubling the
SNR, which in turn equates to half the noise for the same signal. A SNR value of 30dB means excellent
image quality. We have therefore de ned low-light ISO as the highest ISO setting for a camera that
allows it to achieve a SNR of 30dB while keeping a good dynamic range of 9 EVs and a color depth of
18bits. A difference in low-light ISO of 25% equals 1/3 EV and is only slightly noticeable. Low-light ISO
is an open scale.
Sensor Overall Score
The Sensor Overall Score is an average of the three use case scores above: the Portrait Score based
on color depth, the Landscape Score based on dynamic range, and the Sports Score based on low-light
ISO. The Sensor Overall Score shows a camera’s:
Sensor quality in terms of noise
Ability to render high contrast
Threshold for the formation of colored noise
Ability to shoot in low light
However, the Sensor Overall Score does not show a camera’s resolution and ability to render ne detail,
nor does it take into account optical aberrations, as those criteria depend on the lens used with the
camera (see our article about lens scoring. It is worth keeping a few additional points in mind when
reading DxOMark camera sensor reviews:
The Sensor Overall Score is logarithmic: A 5-point difference on the scale corresponds to a gain or loss of
sensitivity of 1/3 of a stop.
The Sensor Overall Score is normalized for a de ned printing scenario of 8 Megapixels printed on a 8×12-
inch (20cm x 30cm) size at 300 dpi resolution. Any other normalization, even at a higher resolution, would
lead to the same ranking, given that any camera that cannot deliver the chosen resolution is eliminated
from the comparison.
The Sensor Overall Score is open and not a percentage: The score is computed so that current camera
models, from low-end DSCs up to professional DSLRs and medium-format cameras, score within a range
from 0 to 100. However, new technologies may well lead to scores higher than 100 in the future.
Let’s take a closer look at the setups and methodologies used in the DxOMark test lab to measure the
individual test criteria that feed into out Overall Sensor Score.
Noise and dynamic range
We measure Noise using a transmission target placed on top of a uniform light box. The transmission
target is an in-house design made of a thick black plastic plate with precision-drilled holes. These
[Link] 4/8
12/31/2019 DxOMark camera sensor testing protocol and scores - DXOMARK
holes (or “patches”) are equipped with range of neutral density lters designed to absorb light in the
same way for all wavelengths. The lters are made of pure optical glass with no structures that could
be measured as noise. Other image quality measurement solutions make use of printed targets, but
we believe such targets are inappropriate for noise measurements, as the intrinsic noise pattern of the
paper can be recorded by the tested camera and then confused with a noise pattern. We place high-
density lters on neighboring positions to limit reciprocal illumination of the patches.
Noise target exposure setup
Noise target exposure setup
The light box behind the target features two uorescent daylight spectrum tubes with a diffuser sheet on
top to achieve perfect uniformity on each lter. The luminance is approximately 1500cd/m². The light
absorption levels of the lters range from 0% to 99.99% in order to test across a dynamic range of 4 density
steps (= 13.3 f-stops — a dynamic range much greater than today’s digital cameras can achieve. When
shooting such a chart, the sensor of the test camera is exposed to a wide range of light levels, with a
1/10,000 ratio from minimum to maximum. For comparison, the dynamic range of a printed target is
typically 2 density steps (6.65 f-stops), which is not enough to simulate high dynamic range or back-lit
scenes.
We measure the luminance of each uniform area (patch) on the chart using a certi ed luminance-
meter, and then input the values into DxO Analyzer software. Once we have measured the target and
calibrated the DxO Analyzer software, we shoot test images of the noise target at different ISO
[Link] 5/8
12/31/2019 DxOMark camera sensor testing protocol and scores - DXOMARK
settings, and measure the Noise for each color channel of the target image (R, Gr, Gb, B). We compute
the mean gray level and noise values for each patch and for all images shot at different ISO settings,
and nally interpolate these numerical values for all gray levels to calculate and plot signal-to-noise
ratio (SNR) curves, from which DxO Analyzer extracts the SNR 18%, the dynamic range, and the tonal
range.
ISO sensitivity
The purpose of the saturation-based ISO sensitivity measurement is to measure the exposure
necessary to reach a given sensor’s saturation point. To measure the camera sensor’s ISO sensitivity,
we set up the camera body without a lens on a stand to receive light from a controlled source. The
light source is positioned far enough away from the camera sensor to ensure good light uniformity on
the sensor plane. We then use a certi ed lux meter to precisely measure the illuminance that the
sensor receives.
ISO sensitivity measurement: Before
shooting a picture, the lux meter is placed
exactly at the position of the camera sensor
to measure the exposure.
For cameras with non-removable optics — for example most compact and bridge cameras, we modify the
measurement protocol. To minimize the in uence of the optics on ISO sensitivity and to avoid vignetting, we
perform the measurements at the image center. We set up the camera on a tripod to receive light from the
noise target with the center patch open. We then calculate ISO sensitivity as follows:
[Link] 6/8
12/31/2019 DxOMark camera sensor testing protocol and scores - DXOMARK
ISO sensitivity measurement: Before
shooting a picture, the luminance is
measured on the center patch.
We obtain the sensor exposure by multiplying illuminance by exposure time. To correct for any differences
between actual and manufacturer-indicated exposure, we measure sensor integration time with a custom-
designed device that features a set of LEDs that are lit in sequence at a known, precisely-timed frequency.
To perform the measurement, we count the number of lit LEDs on an exposure taken with the test camera.
Measurement of exposure time. The device
contains two rows of LED lit in sequence at a
precisely known frequency. The number of lit
LEDs determines the exposure time with an
accuracy of 1%.
Color sensitivity
[Link] 7/8
12/31/2019 DxOMark camera sensor testing protocol and scores - DXOMARK
Color sensitivity measurement requires knowing the RAW response of the sensor to a selection of
known colors. For this purpose, we illuminate an image of a color target with perfectly-known
re ectance spectra, using a controlled and spectrally-measured illuminant. We perform all color
measurements using a dedicated Kyoritsu color light box, which provides uniform lighting over a
surface about 50cm wide. The light box allows us to switch between several normalized illuminants
(e.g., A, D65, CW, etc.).The test target is a Gretag Macbeth ColorChecker composed of 18 uniform
color patches and 6 gray level patches. For each illuminant, we measure the re ection spectra of the
patches with a spectrometer. In a next step, we measure RAW image values. Uniformity of the
illuminance is below 7% (checked with a lux-meter).
Color measurement setup with Kyoritsu color lightbox.
DXOMARK invites our readership (you) to post comments on the articles on this website. Read more
about our Comment Policy.
[Link] 8/8
