Implementing a CMOS Image Sensor Noise Performance Model Workshops/2007 Workshop/2007 Papers/… ·...
4
Implementing a CMOS Image Sensor Noise Performance Model Keith Findlater 1 , Ryan Gow 2 , David Renshaw 3 , Lindsay Grant 4 , John Hart 4 , Stuart McLeod 4 , Robert Nicol 4 1 Gigle Semiconductor, Edinburgh, UK. Email: [email protected] 2 University of Oxford, Oxford, UK 3 University of Edinburgh, Edinburgh, UK 4 STMicroelectronics Imaging Division, Edinburgh, UK An accurate model of CMOS image sensor noise performance is useful in optimising sensor design and setting targets, ultimately leading to improvements in image quality. Many attempts at modelling CMOS image sensor noise are based on over-simplistic assumptions and may not incorporate a sufficient variety of noise sources to yield a realistic simulator [1]–[3]. These issues inspire the creation of a unified and accurate sensor noise model capable of displaying noisy end-images. In [4], we describe a noise model that incorporates plausible distributions to create realistic images. This paper presents further details of the implementation of such a tool. Our model has been implemented in MATLAB. The use of this high level programming tool has aided the speed of model development, as many routines are already available for image manipulation and statistical analysis. Figure 1 shows the well known CMOS image sensor noise chain that has been modelled [5]. Optical effects other than diffusion crosstalk in the silicon have not been considered. Realistic modelling of non-Gaussian effects is required to model shot noise, dark current noise, pixel 1/f and RTS noise, and other readout related noises. Generation of such distributions for megapixel images and for video sequences can be computationally expensive and acceleration methods are needed for acceptable performance and usability from the simulator. For example, the generation of Poisson statistics to model photon and dark current shot noise involves the generation of a Poisson random variable for each pixel with the variance equal to the mean arrival rate. It was found that repeatedly generating such variables was very slow. Fig 2 shows the method used to reduce the computation time, in which pixels of the same value are grouped together and the same Poisson variable used for those pixels. In typical images, the number of unique values is much less than the total number of pixels, so this results in a considerable reduction in computation time. This technique can equally be applied to photon shot and dark current shot noise. Fixed pattern noise (FPN) such as photo-response non-uniformity (PRNU) also impacts image quality. Our method of modelling this is shown in Fig 3. In this case, a normal distribution is used to model the PRNU. A single frame of PRNU gain values is generated, and this is used over multiple frames to simulate video sequences, resulting in a fixed pattern noise. This method of generating a single frame of noise values is used to model all the fixed pattern effects. Of course the distribution used must be adjusted depending on the type of FPN [6]. Fixed pattern dark current noise is a key example of this. It is important to account for the non-Gaussian nature of this noise, especially the extreme outliers that appear as white pixels [7]. The exponential temperature dependence of this noise is also very important. Figure 4 shows how the dark current is simulated in our model. Using measured characterisation data, three different distributions have been fitted, one for each temperature. These distributions are then scaled by the gain and exposure time values to produce an FPN frame. Following this the shot noise of the dark current is added. As can be seen in Fig 5, realistic dark current effects are produced using this method. Furthermore, we have accounted for non-Gaussian random noise, such as in the pixel source follower [8] and other readout electronics. As with the dark current modelling, we did not succeed in finding a single distribution which could account for the majority of the population while also reflecting the extreme outliers. Therefore we combined the Fischer-Tippet-Gumbel [9] distribution with a uniform distribution as is shown in Fig 6. Fig 7 shows in detail how we have modelled the RTS and 1/f noise of the pixel source follower using the combination of these two distributions. Many noise sources in our model required the use of non-Gaussian distributions. However we were able to model the thermal noise of the readout chain, the vertical fixed pattern noise, and the time varying line noise, using Gaussian random variables with appropriate variance. Figure 8 shows the method of modelling time varying line noise while Figure 9 shows an example of measured values versus the model; they are in good agreement. Figure 10 shows the complete noise model chain, and a simulated image with all noise sources enabled is given in Figure 11. Exaggerated noise parameters have been used for illustration purposes. The model has proved useful for evaluating trade-offs between various noise parameters. An example of this is given in Figure 12. For two different values of vertical FPN, the simulated light level is swept. It can be seen that at low light, even a low value of vertical FPN (in this case 5 electrons input referred) is still visible. Many other similar investigations are possible using this model and the results provide a means of setting and justifying design and process targets for image sensors development. 47
Transcript of Implementing a CMOS Image Sensor Noise Performance Model Workshops/2007 Workshop/2007 Papers/… ·...
Implementing a CMOS Image Sensor Noise Performance Model Keith Findlater1, Ryan Gow2, David Renshaw3, Lindsay Grant4, John Hart4, Stuart McLeod4, Robert Nicol4 1Gigle Semiconductor, Edinburgh, UK. Email: [email protected] 2University of Oxford, Oxford, UK 3University of Edinburgh, Edinburgh, UK 4STMicroelectronics Imaging Division, Edinburgh, UK An accurate model of CMOS image sensor noise performance is useful in optimising sensor design and setting targets, ultimately leading to improvements in image quality. Many attempts at modelling CMOS image sensor noise are based on over-simplistic assumptions and may not incorporate a sufficient variety of noise sources to yield a realistic simulator [1]–[3]. These issues inspire the creation of a unified and accurate sensor noise model capable of displaying noisy end-images. In [4], we describe a noise model that incorporates plausible distributions to create realistic images. This paper presents further details of the implementation of such a tool. Our model has been implemented in MATLAB. The use of this high level programming tool has aided the speed of model development, as many routines are already available for image manipulation and statistical analysis. Figure 1 shows the well known CMOS image sensor noise chain that has been modelled [5]. Optical effects other than diffusion crosstalk in the silicon have not been considered. Realistic modelling of non-Gaussian effects is required to model shot noise, dark current noise, pixel 1/f and RTS noise, and other readout related noises. Generation of such distributions for megapixel images and for video sequences can be computationally expensive and acceleration methods are needed for acceptable performance and usability from the simulator. For example, the generation of Poisson statistics to model photon and dark current shot noise involves the generation of a Poisson random variable for each pixel with the variance equal to the mean arrival rate. It was found that repeatedly generating such variables was very slow. Fig 2 shows the method used to reduce the computation time, in which pixels of the same value are grouped together and the same Poisson variable used for those pixels. In typical images, the number of unique values is much less than the total number of pixels, so this results in a considerable reduction in computation time. This technique can equally be applied to photon shot and dark current shot noise. Fixed pattern noise (FPN) such as photo-response non-uniformity (PRNU) also impacts image quality. Our method of modelling this is shown in Fig 3. In this case, a normal distribution is used to model the PRNU. A single frame of PRNU gain values is generated, and this is used over multiple frames to simulate video sequences, resulting in a fixed pattern noise. This method of generating a single frame of noise values is used to model all the fixed pattern effects. Of course the distribution used must be adjusted depending on the type of FPN [6]. Fixed pattern dark current noise is a key example of this. It is important to account for the non-Gaussian nature of this noise, especially the extreme outliers that appear as white pixels [7]. The exponential temperature dependence of this noise is also very important. Figure 4 shows how the dark current is simulated in our model. Using measured characterisation data, three different distributions have been fitted, one for each temperature. These distributions are then scaled by the gain and exposure time values to produce an FPN frame. Following this the shot noise of the dark current is added. As can be seen in Fig 5, realistic dark current effects are produced using this method. Furthermore, we have accounted for non-Gaussian random noise, such as in the pixel source follower [8] and other readout electronics. As with the dark current modelling, we did not succeed in finding a single distribution which could account for the majority of the population while also reflecting the extreme outliers. Therefore we combined the Fischer-Tippet-Gumbel [9] distribution with a uniform distribution as is shown in Fig 6. Fig 7 shows in detail how we have modelled the RTS and 1/f noise of the pixel source follower using the combination of these two distributions. Many noise sources in our model required the use of non-Gaussian distributions. However we were able to model the thermal noise of the readout chain, the vertical fixed pattern noise, and the time varying line noise, using Gaussian random variables with appropriate variance. Figure 8 shows the method of modelling time varying line noise while Figure 9 shows an example of measured values versus the model; they are in good agreement. Figure 10 shows the complete noise model chain, and a simulated image with all noise sources enabled is given in Figure 11. Exaggerated noise parameters have been used for illustration purposes. The model has proved useful for evaluating trade-offs between various noise parameters. An example of this is given in Figure 12. For two different values of vertical FPN, the simulated light level is swept. It can be seen that at low light, even a low value of vertical FPN (in this case 5 electrons input referred) is still visible. Many other similar investigations are possible using this model and the results provide a means of setting and justifying design and process targets for image sensors development.
47
[1] J. Farrell et al., “A simulation tool for evaluating digital camera image quality”, Proc. of SPIE, v. 5294, pp. 124–131, 2004.
[2] H. Wach and E. Dowski, “Noise modeling for design and simulation of computational imaging systems”, in Proceedings of the Color Imaging Conference: Color Science, Systems, and Application, pp. 211–216, 2004.
[3] R. Costantini and S. Süsstrunk, “Virtual Sensor Design”, Proceedings of SPIE, v. 5301, pp. 408–419, 2004. [4] R. Gow et al., “A Comprehensive Tool for Modeling CMOS Image Sensor Noise Performance”, submitted to
IEEE Transactions on Electron Devices, accepted for publication 2007 [5] A. Theuwissen, “The Effect of Shrinking Pixels in Existing CMOS Technologies”, Fraunhofer IMS Workshop
CMOS Imaging, 2002. [6] L. Grant, “Characterisation of noise sources in CMOS image sensors”, presented at IEEE Solid-State Circuits
Conference Image Sensor Design Forum, Feb. 2005. [7] J. Janesick, Scientific Charge-Coupled Devices, SPIE Press, 2001. [8] K. Findlater et al., “Source follower noise limitations in CMOS active pixel sensors”, Proc. of SPIE, v5251, pp
187-195, Feb 2004 [9] J. Beirlant, et al, Statistics of Extremes: Theory and Applications, Hoboken, NJ: Wiley, 2004.
Pinned photodiode
Q to V conversion
PRNU Photon shot noise
Dark current shot noise Dark current FPN
kTC noise Line noise
Source follower
1/f noiseRTS noiseThermal
noise
Sample & hold Amplification
kTC noise
Thermal noise Column FPN
Quantisation noise
A-to-D conversion
Fig. 1. Main noise sources in the CMOS imager pipeline (adapted from [5]).
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
cbbaaacba
[ ]cba
3 x 3 Input Image
Find &count
uniquevalues
[ ]234frequency
value
Generate vectorsof r.v.'s
A[] = number_generator(a, 4)
B[] = number_generator(b, 3)
C[] = number_generator(c, 2)
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
232
432
111
CBBAAACBA
Map to originalmatrix locations
3 x 3 Output Image
number_generator(value, frequency)
if value >= 1000Draw 'frequency' valuesfrom normal distribution withMEAN and VARIANCE = 'value'
elseDraw 'frequency' valuesfrom Poisson distribution withLAMBDA = 'value'
Fig. 2. Simple example of how to exploit recurring pixel values.
Specify PRNU %per channel(or mono) 100
channelchannel
prnu=σ
[ ]channel1,N σ
Generate 'gain map'using normal distribution
(accouting for Bayer pattern)
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
OMM
L
L
GBB
RGR
PPPP
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
OMM
L
L
)2,2()1,2(
)2,1()1,1(
XXXX
Element-by-Element
Multiplication
FixedGainMap
Inputframe i
m
n
m
n
Gain map mean 1 sooutput image mean =input image mean, i.e.
zero mean PRNU
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
OMM
L
L
)2,2()1,2(
)2,1()1,1(
ZZZZ
Outputframe i
m
n
Fig. 3. Technique of modeling PRNU.
48
Fig. 4. Process of modeling dark current. Underlined text indicates input parameters specified by the user.
Mean
ppm > 30e
Max. Tail
FTG
Uniform
Fig. 6. Sketch of a synthetic flicker noise distribution, indicating the two underlying pdf’s used and which parameters are controllable by the user.
Noiseless Dark current at 25C
Dark current at 45C Dark current at 65C Fig. 5. Impact of dark current at 25˚C, 45˚C and 65˚C on a noiseless image at 0.25s exposure, using exaggerated dark current parameters for illustration.
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
OMM
L
L
)2,2()1,2(
)2,1()1,1(
''''
DCBA
m
nFlickernoise i],0[NA' )1,1((1,1) A=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
OMM
L
L
)2,2()1,2(
)2,1()1,1(
XXXX
Input frame i
m
n
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
OMM
L
L
)2,2()1,2(
)2,1()1,1(
ZZZZ
Output frame i
m
n
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
OMM
L
L
)2,2()1,2(
)2,1()1,1(
FFFF
FTG
m
n
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
OMM
L
L
)2,2()1,2(
)2,1()1,1(
UUUU
Uniform
m
n
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
OMM
L
L
)2,2()1,2(
)2,1()1,1(
DCBA
m
n
Array Variation(1000 frame average)
],0[NB' )2,1((1,2) B=
Mean Max
Mean Max
Generate Normal rv for each pixel based on array distribution
Temporal
Fixed
Generate suitable FTG distribution to model main body
Generate suitable uniform distribution
to model outliers
Fig. 7. Overview of the technique used to model flicker/RTS noise.
Determine temperature
25C 45C 65C
Logistic Inverse Gaussian Log-Normal
Generate m x n array from appropriate distribution with best-fit parameters
Calculate proportion of non-zero elements in uniform distribution
Generate m x n array from uniform distribution
Uniform
ppm > threshold level Max. Tail
Add distributions and offset to desired mean
Mean
DCNU distribution (e/s = 'rate')
DCNU(e) = DCNU(e/s) x exposure time
DCSN = Shot noise algorithm applied to DCNU
Dark Calibration
if dark_cal == 1Subtract mean from DCSN array
else if dark_cal == 0Do not subtract mean from DCSN
Output frame = input frame + DCSN
49
Specify sigma of row noise (e)
Generate rv 's from Normal distribution [ ]lines0,N σ
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
OMM
L
L
)2,2()1,2(
)2,1()1,1(
XXXX
Input frame i
m
n
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
OMM
L
L
)2,2()1,2(
)2,1()1,1(
ZZZZ
Output frame i
m
n
Rownoise i
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
M
B
A
NN
m
1
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
OMM
L
L
BB
AA
NNNN
m
n
Replicate along
n columns
Values constant for given row
Fig. 8. Technique used to model temporal row noise.
Fig. 9. Comparison of an actual temporal row noise histogram and a normal distribution with the same mean and standard deviation.
Fig. 10. Block diagram of the overall sensor noise model.
Fig. 11. Simulation of the entire sensor noise chain, based on exaggerated noise values for illustration purposes.
Fig. 12. Images showing the impact of various VFPN levels at low and medium light.
Pixel Cross-Talk
Dark Current Thermal Noise
1/f & RTS
Row Noise
Column Noise
ADCQuantisation Colourisation
Summary Report
Photo Response
Non-Uniformity
PhotonShot Noise
Peak signal (e/s)Exposure (s)Full well (e)Scene choice
Number of frames
Temperature (C)Mean (e/s)Tail (e/s)
ppm > 250aADark cal. setting
PRNU for each colour channel (%)
Cross-talk for the 8 neighbours of each colour channel (%)
Std. deviation (e)
Mean (e)ppm > 30e
Max. tail value (e)
Std. deviation (e)
Temp: Mean, tail value (e)VFPN: Std. deviation (e)
Architecture-specific ADC parameters
Parameters required by external algorithms
FPNFPNTEMP
FPN TEMPTEMP
FPNTEMP TEMP FPN
TEMP
Noisy Image(s)
Noiseless Image
Pixel Cross-Talk
Dark Current Thermal Noise
1/f & RTS
Row Noise
Column Noise
ADCQuantisation Colourisation
Summary Report
Photo Response
Non-Uniformity
PhotonShot Noise
Peak signal (e/s)Exposure (s)Full well (e)Scene choice
Number of frames
Temperature (C)Mean (e/s)Tail (e/s)
ppm > 250aADark cal. setting
PRNU for each colour channel (%)
Cross-talk for the 8 neighbours of each colour channel (%)
Std. deviation (e)
Mean (e)ppm > 30e
Max. tail value (e)
Std. deviation (e)
Temp: Mean, tail value (e)VFPN: Std. deviation (e)
Architecture-specific ADC parameters
Parameters required by external algorithms
FPNFPNTEMP
FPN TEMPTEMP
FPNTEMP TEMP FPN
TEMP
Noisy Image(s)
Noiseless Image
50
![Power Efficient CMOS Full Adders with Reduced Transistor Count · The TGA full adder [5] using 20 transistors is based on CMOS transmission gates and CMOS inverters. It ... In TFA](https://static.fdocument.org/doc/165x107/5c01a10709d3f20f068d0c17/power-efficient-cmos-full-adders-with-reduced-transistor-count-the-tga-full.jpg)
