Nikon D800/E outer AF sensor accuracy

– Whitepaper / Technical Study –

Falk Lumo [LumoLabs (fl)]
Dieter Lukas [panobilder.de]

This technical study evaluates the performance of the outer (towards the left and right edges) autofocus sensors (AF detectors aka AF focus points), esp. with respect to accuracy and with a wide angle lens. Applicable to the Nikon D800 and Nikon D800E full frame SLR cameras.

1. The case

Advance notice: In this study, we will make no distinction between the Nikon D800 and Nikon D800E cameras, both referred to as Nikon D800, or D800 or D800/E. We evaluated two specimen, referred to as sample 1 and sample 2. One was a D800, the other a D800E. But the differences found are due to sample variation, not model variation (based on our current knowledge). – end of advance notice –

The D800 is the current holy grail in camera land. Waiting lists are long enough to make Apple fanboys wonder ;) It will be long months before Nikon will be able to satisfy all preorders. At the same time, the camera at $ 3,000 is not cheap. So, it better be great indeed.

According to current experience (and including our own here at LumoLabs being happy to have access to three D800s), the D800 produces wonderful images, both in terms of image rendition (DxO places its sensor 1/3 stop ahead of any other camera ever produced, any price, for a reason) and in terms of resolution (where it approaches medium format territory and actually delivers on the promise, particularly the E model).

But like with all new technically complex items, it isn't without flaws. There are some reports floating around the web that the performance of the outer AF detectors is subpar. Especially with a fast and/or wide angle lens like 24mm f/1.4 or 14mm f/2.8 where it may be considered broken. Especially with the leftmost AF point. We decided to evaluate the issue.

2. Our test procedure

We prepared a special test target, depicted in Fig. 1.

Fig. 1 AF detector test target. The image shows the center 2/3 crop of the full frame image, both horizontally and vertically. The image is overlaid with an AF points illustration from the Nikon homepage.

Note that like Canon and Pentax, Nikon uses an APS-C AF module (DX crop) for their larger frame digital cameras. It therefore covers only the inner 2/3 of the frame (slightly less actually, cf. above).

Measurements have been performed at the 5 AF points marked red, without moving the camera. The red focus indicators in the viewfinder aligned to the center circles of the test chart respectively.

Each of the five AF points at positions (-5,0), (-3,0), (0,0), (3,0), (5,0) have been evaluated using the same procedure and without moving the camera. Each AF detector "sees" the same local focus target. Except AF point (0,0), the AF points are of the linear type (not cross type). Cross types are marked orange in the Nikon AF points illustration.

The procedure involved two samples of the D800, an identical (single) Nikkor 24-70/2.8G lens used at 24mm f/2.8. The distance to the target was 1.20 m (50x focal length, a distance commonly proposed by vendors for focus calibration work). The light was tungsten (2850 K according to Lightroom's white balance tool) at light value LV 10 (shot at 1/160s and ISO 100).

Factors notstudied:

1. Color temperature (expected to influence absolute values but not the outer AF point issue).

2. Focal length (reports suggest that shorter focal lengths have the bigger problem).

3. Subject distance (may have a significant impact; something we were too lazy to study – however, a 50x focal length distance is a relevant range anyway).

4. Aperture (expected to have no influence except of course to make the issue more visible in images – AF detectors behave almost independently from the aperture setting).

5. Lens sample variation (not expected to have an influence).

We did notsimply take auto-focused images using different focus points. The result would both fluctuate too much and be too dependent on proper AF fine calibration. Moreover, we refrained to judge focus quality by visual inspection or using a tool like lens align. It's subjective and imprecise.

Our procedure was as follows: At each (of five) focus point, we did a fullAF fine calibration. The AF fine calibration involves a series of test shots, followed by numerical evaluation of sharpness atthe corresponding position (which eliminates lens field curvature errors from the equation), followed by an algorithmic fitting procedure to find the optimum AF fine calibration adjustment value. Normally, we would have used our tool set (as described in our "Image Understanding" whitepaper published at LumoLabs). However, we rather decided to give the "Focal" tool from Reikan a chance (cf. www.fo-cal.co.uk). The tool has its own issues and cannot compete with our tools in precision. However, it still proved sufficient at the task and with some guidance, provided the required results as presented below. The focus target is the official Focaltarget. It is recognized and also used to verify that the image plane is parallel enough to the sensor. So, while not perfect in our setup, it was verified to be good enough for our task.

We then determined the AFfine calibration micro adjustment parameter (called AFMA by Reikan, a term we opt to use as well here), as a function of the focus point position (ranging from x= -5 to +5). Ideally, the AFMA(x)function would be a constant, meaning that AF fine calibration is effective for all focus points. If not, the AFMA(x)would at least be independent from the sample, so it can be corrected by a firmware patch. Both possibilities have been evaluated for.

3. Results

First, let's have a look at individual test shots, as shown in Fig. 2for sample 2.

Focus point (-5,0) at AFMA value 0; sample 2.

Focus point (-5,0) at AFMA value -15; sample 2.

Focus point (0,0) at AFMA value 0; sample 2.

Focus point (+5,0) at AFMA value 0; sample 2.

Fig. 2 Crop from focused area at various focus points. The HTML version provides access to JPGs which then are 100% crops from the original. Developed in Lightroom PV2010 all sliders flat (no brightness, no contrast, no sharpening).
It is obvious that the left AF point has poor focus. It is much improved after setting AFMA to -15.

From these test shots, the optimal value for AFMA is computed, cf. Fig. 3.

Focus point (-5,0) computes to an optimum AFMA value of -17; sample 2.

Focus point (0,0) computes to an optimum AFMA value of -1; sample 2.

Focus point (+5,0) computes to an optimum AFMA value of -8; sample 2.

Fig. 3 AF fine calibration micro adjustment parameter AFMA as determined by optimization. Shown for three AF points.

Now, we can summarize the accuracy of all evaluated AF points by a plot of AFMA value vs. focus point position, as shown in Fig. 4. (Note: we approximated AFMA values below -20 and above +20 from the known function of "image quality as function of AFMA deviation from the optimum.)

Fig. 4 -AFMA(x) Focus calibration (negative AFMA value) as a function of AF point position. 2 samples.

The function AFMA(x)can be approximately described by:

AFMA (x) = AFMA₀+ b x + 1/5 a x²

such that

AFMA (-5) - AFMA₀= 5 (a - b)

We'll call a the "parallax error" and b the "tilt error" and they are normalized such that for the outermost AF point, they have equal weight.

Fig. 5 Characterization of AF focus inaccuracy, in terms of tilt and parallax errors.

While the two samples we evaluated have quite similar inaccuracy of the leftmost focus point (requiring about ~ -13 AF extra fine tuning adjustment additional to the center point), their characteristics is quite different: Sample 1 is more due to tilt while the other is more due to parallax. The consequence is that the rightmost focus point cannot be corrected when correcting for the leftmost and center ones.

To put things into perspective: fine tuning adjustment values between -10 and +10 are fully acceptable and more often than not, adjustment values vary by as much depending on the test.

4. Conclusion

The issue with the accuracy of the outer AF focus points of the D800 is real and probably affects all units out there to some (varying) extent.

It is said to be pronounced at ultra wide angles. However, we found it to be (just) unacceptable at 24 mm as well. We guess that every camera with the "Nikon Advanced Multi-CAM 3500 FX" auto focus module is affected, i.e., D800, D800E and D4.

We evaluated the characteristics of the sample variation of the issue and found that the cause may not be consistent across units. There may be a mixture of a tilt problem (such as caused by a misaligned AF auxiliary mirror, possibly independent of the focal length) and a parallax problem (such as caused by a decentered AF relay lens, possibly dependent on the focal length) where both components contribute with variable strength from sample to sample (our statements about possble causes are speculative, our statements about tilt and parallax errors and sample variation are not).

Therefore, we express our concern if the issue can be cured by a simple firmware patch. As far as we can see, such a patch would require at least two additional calibration parameters. We rather see many units undergo a thorough re-calibration job at Nikon.

Breaking news: According to www.nikonians.org/forums/dcboard.php?az=show_topic&forum=430&topic_id=7054&mesg_id=7054&page=#7303 , Nikon has set up a repair / calibration procedure which enables Nikon service centers to fix the issue, much in line with our findings which suggest that each camera has to be cared for on an individual basis.

Meanwhile, we recommend to refrain from using all but the center 15 cross type AF points for any work relying on critical focus, at least on the wider end of focal lengths.