Page 89 -

68 2 Image formation

Sensor noise. Throughout the whole sensing process, noise is added from various sources,
which may include fixed pattern noise, dark current noise, shot noise, amplifier noise and
quantization noise (Healey and Kondepudy 1994; Tsin, Ramesh, and Kanade 2001). The
final amount of noise present in a sampled image depends on all of these quantities, as well
as the incoming light (controlled by the scene radiance and aperture), the exposure time, and
the sensor gain. Also, for low light conditions where the noise is due to low photon counts, a
Poisson model of noise may be more appropriate than a Gaussian model.
As discussed in more detail in Section 10.1.1, Liu, Szeliski, Kang et al. (2008) use this
model, along with an empirical database of camera response functions (CRFs) obtained by
Grossberg and Nayar (2004), to estimate the noise level function (NLF) for a given image,
which predicts the overall noise variance at a given pixel as a function of its brightness (a
separate NLF is estimated for each color channel). An alternative approach, when you have
access to the camera before taking pictures, is to pre-calibrate the NLF by taking repeated
shots of a scene containing a variety of colors and luminances, such as the Macbeth Color
Chart shown in Figure 10.3b(McCamy, Marcus, and Davidson 1976). (When estimating
the variance, be sure to throw away or downweight pixels with large gradients, as small
shifts between exposures will affect the sensed values at such pixels.) Unfortunately, the pre-
calibration process may have to be repeated for different exposure times and gain settings
because of the complex interactions occurring within the sensing system.
In practice, most computer vision algorithms, such as image denoising, edge detection,
and stereo matching, all benefit from at least a rudimentary estimate of the noise level. Barring
the ability to pre-calibrate the camera or to take repeated shots of the same scene, the simplest
approach is to look for regions of near-constant value and to estimate the noise variance in
such regions (Liu, Szeliski, Kang et al. 2008).

ADC resolution. The final step in the analog processing chain occurring within an imag-
ing sensor is the analog to digital conversion (ADC). While a variety of techniques can be
used to implement this process, the two quantities of interest are the resolution of this process
(how many bits it yields) and its noise level (how many of these bits are useful in practice).
For most cameras, the number of bits quoted (eight bits for compressed JPEG images and a
nominal 16 bits for the RAW formats provided by some DSLRs) exceeds the actual number
of usable bits. The best way to tell is to simply calibrate the noise of a given sensor, e.g.,
by taking repeated shots of the same scene and plotting the estimated noise as a function of
brightness (Exercise 2.6).

Digital post-processing. Once the irradiance values arriving at the sensor have been
converted to digital bits, most cameras perform a variety of digital signal processing (DSP)
operations to enhance the image before compressing and storing the pixel values. These in-
clude color filter array (CFA) demosaicing, white point setting, and mapping of the luminance
values through a gamma function to increase the perceived dynamic range of the signal. We
cover these topics in Section 2.3.2 but, before we do, we return to the topic of aliasing, which
was mentioned in connection with sensor array fill factors.