An Energy-efficient Adaptive CMOS Image Sensor

by

Jaehyuk Choi

A dissertation submitted in partial fulfillment

of the requirements for the degree of
Doctor of Philosophy
(Electrical Engineering)
in The University of Michigan
2013

Doctroral Committee:

Professor Euisik Yoon, Chair

Professor Kensall D. Wise
Assistant Professor David D. Wentzloff
Assistant Professor James W. Cutler
 © Jaehyuk Choi
All Rights Reserved 2013
 ACKNOWLEGEMENTS

My tenure as a graduate student at the University of Michigan has been a great time

in my life. I experienced many things and learned a lot, and importantly I had an exciting

time with my colleagues and friends.

First I would like to thanks my advisor Professor Euisik Yoon. He has supported my

study in the graduate school and always provided a motivation and trust. I learned

innovative thinking, communication skills and the ability to drive research.

I also would like to thank my dissertation committee Professor Ken Wise, David D.

Wentzloff, and James W. Cutler. I greatly appreciate their feedback and encouragement.

I would like to thank current image group members Seok-Jun Park and Jihyun Cho.

We spent considerable time together especially during tapeout. They helped me a lot of

things in both design and testing. I appreciate their contribution to this work. I would also

like to thank previous image group members, Kwang-Hyun Lee, Sang-Wook Han and

Seong-Jin Kim. Due to their advice and instruction, my research about the multi-

resolution image sensor was very successful.

I give thanks to current and past lab members Xia Lou, Khaled AlAshmouny, Fan Wu,

Sung-Yun Park, Kyunghwan Na, Yu-Chih Chen, Patrick Ingram, Roberto Miguez, Mike

McCormick, Maesoon Im, Il-Joo Cho, Hyung-Kew Lee, Jaehoon Chung, Young-Ji Kim,

Hwa Sung Shin, Dr. Songjun Lee, Dr. Bo Yang and Dr. Ararta Nakajima. I had

wonderful time with you every day.

ii
 Finally, I want to give my thanks to my mother. Thank you for constant

encouragement, support and love.

iii
 TABLE OF CONTENTS

Acknowlegements.............................................................................................................. ii

List of figures .................................................................................................................. viii

List of Tables .................................................................................................................. xiii

Abstract………………………………………………………………………………xiii

Chapter 1 Introduction and Background ....................................................................... 1

1.1 Evolution of digital imaging systems ........................................................................ 1

1.2 Solid-state imaging devices ....................................................................................... 3

1.3 Challenges in low-power image sensors ................................................................... 6

1.4 Research goals ........................................................................................................... 8

1.5 Thesis organizations ................................................................................................ 11

Chapter 2 Introduction to CMOS Image Sensors ....................................................... 13

2.1 Image signal chain and overall architecture ............................................................ 13

2.2 Photogeneration ....................................................................................................... 15

2.3 Photodetectors ......................................................................................................... 18

2.3.1 P-N photodiode .............................................................................................. 18

2.3.2 Pinned photodiode .......................................................................................... 25

2.4 Pixel circuit ............................................................................................................. 26

2.4.1 3-T Pixel with p-n photodiode ....................................................................... 27

iv
 2.4.2 4-T Pixel with pinned photodiode .................................................................. 29
2.4.3 Shared-pixel architecture................................................................................ 30

2.5 Correlated double sampling (CDS) ......................................................................... 32

2.6 Analog-to-digital converter (ADC) ......................................................................... 35

2.6.1 Serial ADC ..................................................................................................... 35

2.6.2 Column-parallel ADC .................................................................................... 35
Chapter 3 Low Power CMOS Image Sensors, Dynamic Range Enhancement and
Embedded Image Signal Processing............................................................. 39

3.1 Power consumption in CMOS image sensors ......................................................... 40

3.1.1 Power consumption: static, dynamic and leakage .......................................... 41

3.1.2 Power Consumption in the Image Signal Chains ........................................... 44
3.1.3 Overall Power Consumption .......................................................................... 58
3.1.4 Low-power design strategy ............................................................................ 61
3.1.5 Power consumption figure-of-merit (FOM) ................................................... 62

3.2 Low Power CMOS Image Sensors.......................................................................... 63

3.2.1 Limitation of Voltage Scaling ........................................................................ 64

3.2.2 Reset Signal Boosting .................................................................................... 67
3.2.3 In-Pixel Pulse Width Modulation (PWM) ..................................................... 70

3.3 Wide Dynamic Range Image Sensors ..................................................................... 74

3.3.1 Sensitivity Enhancement ................................................................................ 75

3.3.2 Dynamic Range Enhancement Schemes ........................................................ 78
3.3.2.1 Nonlinear Scheme: Logarithmic Pixel ..................................................... 80
3.3.2.2 Linear Scheme: Sensitivity Adjustment ................................................... 82
3.3.2.3 Linear Scheme: Multiple Exposures ........................................................ 84
3.3.2.4 Summary of Integrated Dynamic Range Enhancement Schemes ............ 85
3.4 CMOS Image Sensors with Integrated Image Signal Processing for Bandwidth
Reduction ……………………………………………………………………………..86

3.4.1 Feature extraction ........................................................................................... 87

v
 3.4.2 Region-of-Interest (ROI) readout ................................................................... 89
3.4.3 Variable-resolution image Readout ................................................................ 91

3.5 Motion-adaptive multi-resolution image sensor ..................................................... 92

3.6 Summary…………………………………………………………………………..95

Chapter 4 Energy/Illumination-Adaptive Image Sensor with Reconfigurable Modes

of Operation .................................................................................................. 102

4.1 Adaptive imaging operation and overall architecture ........................................... 103

4.2 Pixel and column circuits for reconfigurable modes ............................................. 104

4.3 In-equality readout ................................................................................................ 112

4.4 Dual exposure with pixel merging ........................................................................ 115

4.5 Implementation and experimental results ............................................................. 117

4.6 Summary …………………………………………………………………………121

Chapter 5 Object-Adaptive Image Sensor with Embedded Feature Extraction for

Motion-triggered Object-of-Interest Imaging ........................................... 123

5.1 Introduction ........................................................................................................... 123

5.2 Histogram of Oriented Gradients (HOG) feature .................................................. 125

5.3 CMOS imager with motion-triggered object-of-interest imaging ......................... 127

5.3.1 Object-of-interest imaging ........................................................................... 127

5.3.2 Overall architecture ...................................................................................... 128
5.3.3 Pixel circuit .................................................................................................. 129
5.3.4 Column circuit .............................................................................................. 132
5.3.5 Gradient-to-Angle Converter (GAC) ........................................................... 133

5.4 Chip characteristics and object detection simulation ............................................ 136

5.4.1 Testing of the object detection ..................................................................... 139

vi
 5.4.2 Power figure of merit (FOM) comparison with previous low-power imagers
……………………………………………………………………...140

5.5 Summary …………………………………………………………………………141

Chapter 6 Conclusion and Future Work .................................................................... 144

6.1 Conclusion and summary of contributions............................................................ 144

6.2 Future adaptive image sensor ................................................................................ 146

6.2.1 Adaptive imaging system-on-a-chip ............................................................ 146

6.2.2 Integration in a sensor platform ................................................................... 147
Appendix ……………………………………………………………………………….149

vii
 LIST OF FIGURES

Figure 1.1 CCD image sensor. .................................................................................... 3

Figure 1.2 Passive pixel and active pixel sensor architecture..................................... 6

Figure 1.3 Four main challenges in image sensors. .................................................... 7

Figure 1.4 Concept of adaptive image sensing. .......................................................... 9

Figure 1.5 Operation of adaptive image sensing. ..................................................... 10

Figure 2.1 Image signal chain. .................................................................................. 13

Figure 2.2 Overall architecture of CMOS image sensor........................................... 14

Figure 2.3 Image signal chain. .................................................................................. 16

Figure 2.4 Absorption of the photon. ........................................................................ 17

Figure 2.5 Charge collection. .................................................................................... 18

Figure 2.6 P-N photodiode........................................................................................ 18

Figure 2.7 Photogeneration in P-N photodiode. ....................................................... 19

Figure 2.8 Photocurrent. ........................................................................................... 20

Figure 2.9 Dark current mechanism.......................................................................... 22

Figure 2.10 Charge integration. ................................................................................ 23

Figure 2.11 Pinned photodiode. ................................................................................ 26

Figure 2.12 3-T pixel with p-n photodiode. .............................................................. 27

Figure 2.13 Operation of 3-T pixel. .......................................................................... 28

Figure 2.14 4-T pixel with pinned photodiode. ........................................................ 29

Figure 2.15 Operation of 4-T pixel. .......................................................................... 30

viii
Figure 2.16 Shared pixel circuit. ............................................................................... 31

Figure 2.17 Correlated double sampling circuit. ...................................................... 32

Figure 2.18 ADC schemes in CMOS image sensor.................................................. 35

Figure 2.19 Single-slope ADC.................................................................................. 36

Figure 3.1 Block diagram of CMOS imager............................................................. 40

Figure 3.2 Signal chain of CMOS imager. ............................................................... 45

Figure 3.3 Signal chain of CMOS imager ................................................................ 46

Figure 3.4 Pixel circuit with load capacitance of in-pixel source follower .............. 47

Figure 3.5 Correlated double sampling circuit with programmable gain amplification

........................................................................................................................... 48

Figure 3.6 Column-parallel single-slope ADC ......................................................... 49

Figure 3.7 Preamplifier in the single-slope ADC ..................................................... 51

Figure 3.8 Dynamic comparator ............................................................................... 52

Figure 3.9 D flip-flop ................................................................................................ 53

Figure 3.10 Digital signal readout circuit and the operation .................................... 54

Figure 3.11 Row scanner .......................................................................................... 56

Figure 3.12 Overall power consumption in the signal chain according to the spatial

resolution (a) 320 × 240 (b) 1280 × 960 ........................................................... 58

Figure 3.13 Analog power consumption in the signal chain according to the spatial

resolution (a) 320 × 240 (b) 1280 × 960 ........................................................... 59

Figure 3.14 Digital power consumption in the signal chain according to the spatial

resolution (a) 320 × 240 (b) 1280 × 960 ........................................................... 60

ix
Figure 3.15 Digital power consumption according to the power supply voltage (320

× 240) ................................................................................................................ 60

Figure 3.16 Overall power consumption according to the pixel pitch (a) 320 × 240

(b) 1280 × 960................................................................................................... 61

Figure 3.17 Overall power consumption according to the resolutions ..................... 63

Figure 3.18 Signal range in the 4-T pixel ................................................................. 65

Figure 3.19 Charge injection and clock feedthrough in the reset transistor ............. 66

Figure 3.20 Hard reset: reset gate boosting .............................................................. 68

Figure 3.21 Reset level boosting with capacitive coupling (1) ................................. 69

Figure 3.22 Reset level boosting with capacitive coupling (2) ................................. 69

Figure 3.23 In-pixel pulse width modulation scheme ............................................... 70

Figure 3.24 In-pixel comparator without static bias current ..................................... 72

Figure 3.25 In-pixel PWM with sub-VT comparator ................................................ 73

Figure 3.26 In-pixel sub-VT comparator with current-controlled threshold ............. 73

Figure 3.27 In-pixel nMOS common source amplifier with p-n photodiode ........... 76

Figure 3.28 In-pixel pMOS common source amplifier with pinned photodiode ...... 77

Figure 3.29 Dynamic range enhancement schemes .................................................. 79

Figure 3.30 Logarithmic pixel .................................................................................. 80

Figure 3.31 Linear-logarithmic pixel ........................................................................ 81

Figure 3.32 Sensitivity adjustment using in-pixel capacitor..................................... 82

Figure 3.33 Lateral overflow scheme (a) Pixel circuit (b) Operation procedure ...... 83

Figure 3.34 Triple exposure scheme ......................................................................... 85

Figure 3.35 Pixel with direct frame difference output .............................................. 88

x
Figure 3.36 In-pixel comparator for the frame difference ........................................ 88

Figure 3.37 Pixel structure with two additional photodiodes ................................... 90

Figure 3.38 Images from motion-adaptive multi-resolution image sensor ............... 92

Figure 3.39 Pixel and column architecture of the multi-resolution image sensor .... 93

Figure 3.40 Multi-resolution readout: (a) Block diagram (b) Timing diagram ........ 94

Figure 4.1 Operation of adaptive imaging .............................................................. 103

Figure 4.2 Operation and block diagram of the adaptive image sensor with

reconfigurable modes ...................................................................................... 104

Figure 4.3 Pixel circuit of reconfigurable pixel ...................................................... 105

Figure 4.4 Pixel architecture and equivalent circuits in each mode ....................... 106

Figure 4.5 Timing diagram of the pixel control in each mode ............................... 107

Figure 4.6 In-pixel SAR ADC in monitoring mode ............................................... 108

Figure 4.7 Capacitive DAC with reduced capacitance ........................................... 108

Figure 4.8 Column-parallel ADC for the reconfigurable modes ............................ 109

Figure 4.9 Timing diagram of column-parallel ADC (a) Single ended input (normal,

WDR mode), (b) Differential input (high-sensitivity mode) .......................... 110

Figure 4.10 Hybrid latch circuit .............................................................................. 111

Figure 4.11 Comparison with conventional latch in single-slope ADC ................. 112

Figure 4.12 Conventional digital image signal readout circuit and operation ........ 112

Figure 4.13 Image locality ...................................................................................... 113

Figure 4.14 Digital image signal readout circuit with in-equality readout ............. 114

Figure 4.15 Averaged probability of equality ......................................................... 115

xi
Figure 4.16 Dual exposure scheme: pixel output response and conventional

implementation ............................................................................................... 115

Figure 4.17 Dual exposure with pixel merging ...................................................... 116

Figure 4.18 Chip microphotograph ......................................................................... 117

Figure 4.19 Sample images ..................................................................................... 118

Figure 4.20 SNR vs. Power-FOM Plot ................................................................... 120

Figure 5.1 Motion-triggered object-triggered object-of-interest (OOI) imaging.... 125

Figure 5.2 HOG feature extraction procedure ........................................................ 127

Figure 5.3 Operation of the motion-triggered object-of-interest imaging .............. 128

Figure 5.4 Overall architecture ............................................................................... 129

Figure 5.5 Pixel circuit ............................................................................................ 130

Figure 5.6 Equivalent pixel circuit for motion sensing and low-power imaging ... 131

Figure 5.7 Column circuit ....................................................................................... 132

Figure 5.8 Voting process ....................................................................................... 133

Figure 5.9 Gradient-to-angle converter (GAC) ...................................................... 134

Figure 5.10 Timing diagram of gradient-to-angle converter (GAC) ...................... 134

Figure 5.11 Chip micrograph .................................................................................. 137

Figure 5.12 Sample images ..................................................................................... 138

Figure 5.13 Object detection procedure .................................................................. 139

Figure 6.1 Overall architecture of adaptive imaging SoC ...................................... 147

Figure 6.2 Illustration of imaging Cubic (iCube) sensor platform. ........................ 148

xii
 LIST OF TABLES

Table 1.1 Main features of digital imaging systems ................................................... 2

Table 3.1 Parameters for the estimation of power consumption .............................. 46

Table 3.2 Power consumption of conventional CMOS image sensors..................... 63

Table 3.3 Solar energy harvesting example .............................................................. 65

Table 3.4 Summary of low-power CMOS image sensors ........................................ 74

Table 3.5 Summary of integrated dynamic range enhancement schemes ................ 86

Table 4.1 Chip characteristics ................................................................................. 118

Table 4.2 FOM Comparison with Previous Works ................................................ 119

Table 5.1 Capacitance ratio in the GAC ................................................................. 135

Table 5.2 Encoding of 9 bin numbers ..................................................................... 135

Table 5.3 Chip characteristics ................................................................................. 138

Table 5.4 Comparison with previous low-power sensors ....................................... 141

xiii
 ABSTRACT

In this thesis, an energy-efficient CMOS image sensor has been studied to adaptively

provide low-power consumption, high dynamic range, and reduced spatial-temporal

bandwidth for applications in distributed sensor networks and wireless biomedical

imaging systems.

In wireless sensor nodes, image sensors typically have four main challenges to

address: low power operation from limited energy sources such as batteries or energy

harvesting units, high dynamic range imaging to cover a wide range of illumination, high

spatial resolution for high-quality imaging, and high temporal resolution for video

streaming output. However, it is difficult to optimize all of these as there are trade-offs

among these four parameters and additionally the environmental conditions continuously

change, requiring optimization of sensor operation on-the-fly.

One of the solutions to this trade-off is implementing an image sensor with situation-

aware adaptability. In this work, three on-chip adaptive functions have been investigated

including (1) energy-adaptive imaging for low-power operation, (2) illumination-adaptive

imaging for high-dynamic range, and (3) object-adaptive imaging for temporal-spatial

bandwidth suppression.

For proofs of concepts of this adaptive imaging, we designed two prototype chips.

Both prototype sensors have been fabricated and fully characterized. In the first chip, we

implemented the energy/illumination adaptation by reconfiguring the mode of operation.

xiv
Most of the time, the sensor operates in a monitoring mode at an extremely low power

(less than 1.36 μW/frame) with the power supplied from energy harvesting units. It

switches to either a high-sensitivity or a wide-dynamic range mode of operation

depending on illumination conditions. This illumination adaptation provides

reconfigurability for sensitivity and dynamic range. The sensor can provide a high

sensitivity of 23.9 V/lx·s at low illumination, but it can be reconfigured to generate a high

dynamic range of up to 99 dB in high illumination on-the-fly.

In the second chip, we implemented motion-triggered, object-adaptive imaging to

suppress the redundancies in spatial and temporal bandwidths in image signals. The

sensor will wake up when triggered by motion and extracts features from the captured

image for the detection of objects-of-interest. Full image capture operation is performed

only when the objects-of-interest are found, resulting in a significant reduction of total

power consumption at the sensor node. Object detection has been performed using the

classification algorithm and has achieved a 94.5 % success rate for human recognition

from 200 test images.

Adaptability enables energy-efficient image sensing with continuously varying

environmental conditions such as illumination, energy availability, and various objects

changing in the focal plane. Using the three adaptation schemes, the implemented image

sensors provide high dynamic range images (> 99 dB) at low power consumption (< 1.36

μW) while maintaining low bandwidth (< 3.5 %) for image signal transmission.

xv
 CHAPTER 1 INTRODUCTION AND BACKGROUND

1.1 Evolution of digital imaging systems

The evolution of digital imaging systems can be categorized as three generations

according to their applications and technologies. In the 1st generation, the emergence of

digital imaging systems enabled the development of a variety of devices such as digital

still cameras, camcorders, webcams and so on. Industry also benefitted from the digital

imaging solutions, for example, broadcasting systems, machine visions, and inspection

systems. In the 1st generation of imaging systems, Charge Coupled Devices (CCD) had

been widely used since developed in the 1970s. In the 2nd generation, imaging systems

were integrated with handheld devices such as cell phones, laptops, and tablet PCs during

the 1990s. This evolution was mainly enabled by the development of a low-cost CMOS

process and also by improvements in communication technology. The main differences

from the 1st generation imaging systems which were mainly used for consumer

electronics and industry are that the 2nd generation imaging systems are operated by

battery and are linked with the networks. As such the image sensor consumes less power,

has a small form factor, and has a low cost for the application of portable devices. Due to

these demands, CMOS image sensors replaced CCDs in most imaging products [1.1].

Currently, imaging systems are expected to be distributed in a wide area due to the

emergence of sensor networks and ubiquitous computing. Moreover, wireless biomedical

sensors also integrate image sensors for applications in endoscopy, microscopy and

retinal implants. This 3rd generation of imaging systems are distributed, fully networked,

1
and operated with extremely low-power consumption from energy harvesting in order to

monitor wide or unreachable areas. The networked distributed imaging systems have

several advantages over conventional imaging system [1.2]:

1. Sensing visibility: multiple low-resolution imagers cover wider range than a

single high-resolution imager

2. Pose diversity: multiple imagers are in different poses (position, orientation)

3. Statistical convergence: we can get the information based on the gross statistics of

the environment

4. Multiple perspectives: we can get more information from the view-dependent

properties of the object from different perspectives

Generation 1 2 3
Appliance, Distributed
Handheld devices
industry network
Applications Digital still camera, Cell phone,
Surveillance networks,
Micro vehicles,
camcorder, laptop,
Wireless biomedical
Industrial automations tablet
imaging

Technology CCD CMOS CMOS SoC

Topology Standalone Linked Fully networked
Power delivery Wired or battery Battery Energy harvesting

Table 1.1 Main features of digital imaging systems

The main applications of distributed image sensor network are military surveillance,

environmental monitoring, traffic management, surveillance over distributed crops, and

healthcare / patient monitoring. The other main applications of 3rd generation imaging

systems are biomedical sensors. One such commercialized sensor is the capsule

endoscope [1.3] which integrates an image sensor, a controller, and a wireless transmitter.

These image sensors can also be integrated with a wireless microscope [1.4] or with a

2
retinal stimulator [1.5, 1.6], and so on. The main features of each generation are

summarized in table 1.1.

1.2 Solid-state imaging devices

A charge-coupled-device (CCD) is a device that transfers charges integrated in photo-

sites serially by applying clock signals. Incident photons are converted to charge by a

photo-detector. Generated charge is transferred vertically through vertical CCD, and

transferred horizontally through horizontal CCD using the clock signal. In order to

enhance the charge transfer efficiency, a high voltage clock signal is used. In the output

stage, transferred charge is converted to voltage through the amplifier. The basic structure

of a CCD image sensor is shown in figure 1.1

Figure 1.1 CCD image sensor.

CCDs have been used for high-end imaging systems due to low noise and high

quantum efficiency. However, it consumes large power due to a charge transfer operation

with high voltage. Power consumption is critical factor for mobile devices or any other

3
power-limited applications which require extremely low power consumption. Scalability

is another important issue because it is difficult to scale the pixel array larger due to

limited speed of charge transfer. Moreover, a CCD image sensor is difficult to integrate

CMOS circuitry such as an image processor or an analog-to-digital converter, because a

fabrication of CCD requires optimized fabrication process.

CMOS image sensors have not been widely used although it has longer history than a

CCD image sensor. Since MOS image sensors have been developed in 1960’s, low SNR

of a CMOS image sensor made it behind from the market, and main focus of research in

imaging devices has been concentrated on a CCD.

In 1980’s, a MOS image sensor was resurfaced due to the development of CMOS

fabrication technology. CMOS image sensor technology was rapidly developed due to the

integration ability, scalability, low-cost, and low power consumption [1.7].

An imaging system with CMOS image sensor can be implemented with a single chip,

or so called “Camera-on-a-chip”, because both CMOS imagers and integrated circuits

can be fabricated using standard CMOS technology with low cost. By integrating ADCs,

periphery circuits, and image signal processors into one chip, the size of imaging system

can be reduced. On the other hand, fabrication of CCD is specialized and optimized for

effective photo-carrier generation and charge transfer. Imaging systems using CCD

require off-chip components such as timing generators, analog-to-digital converters, and

image processors. Accordingly, the cost for imaging systems with CCD imager is higher

than one chip solution with CMOS image sensor.

CMOS image sensors can be categorized into two types: a passive pixel sensor or an

active pixel sensor. A passive pixel sensor (PPS) consists of a photodiode and a selection

4
transistor as shown in figure 1.2 (a). The photocurrent which is generated from incident

light is converted to voltage in the column-parallel charge amplifier [1.8]. A PPS has a

high fill factor which is defined as the ratio of photodiode area over pixel area. In general,

high fill factor provides higher sensitivity in a given form factor because a photodiode

can collect more electrons. However, a PPS has low SNR for several reasons. First, small

photo-current is difficult to read out due to large capacitance of column line. Second,

considerable leakage current through the selection transistor corrupts the signal.

Moreover, scalability of a pixel array is limited because the column-parallel charge

amplifier has to drive larger capacitance.

Due to the low SNR of a PPS, most CMOS imagers have an active pixel sensor (APS)

architecture. As shown in figure 1.2 (b), an active pixel has an in-pixel amplifier in order

to drive the column line, which has large parasitic capacitance. Fill factor is decreased

compared with passive pixel because of the area from in-pixel circuits. However, the

isolation from the column line through an in-pixel buffer provides higher SNR compared

with the PPS architecture.

CMOS image sensors output the voltage using an in-pixel amplifier, whereas the

charge is transferred serially in CCD image sensors. Therefore, readout with high frame

rate and random access is possible. Moreover, photo-detectors such as photo-gates and

pinned photodiodes provide enhanced sensitivity and better noise immunity compared

with conventional p-n junction photodiodes. The architecture and the operation

principles of CMOS image sensor will be explained in chapter 2.

5
 Figure 1.2 Passive pixel and active pixel sensor architecture.

1.3 Challenges in low-power image sensors

In applications for distributed image sensors and wireless biomedical sensors, we

have four main challenges in image sensors as shown in figure 1.3. The first is power

consumption because imaging systems use limited energy sources such as a battery or

energy harvesting from the environment. State-of-art image sensors have large power

consumption > 30 mW which is not applicable to be operated with energy harvesting.

The power consumption of CMOS image sensors is difficult to be scaled down because

the voltage scaling directly suppresses the SNR which brings loss of image quality. The

second challenge is wide dynamic range. In outdoor applications, the illumination level

varies from extremely dark to extremely bright. In biomedical sensors, illumination level

is typically low, especially when monitoring inside the body or monitoring cells.

Accordingly, the dynamic range and the sensitivity of image sensors should be high in

order to guarantee reliable monitoring.

6
 Power Consumption Illumination Range
(Energy harvesting, battery or
(Saturated or dark image)
wireless power delivery)

Spatial Resolution Temporal Resolution

(High frame rate
(Large pixel array > 10,000 pixels)
> 15 frames per seconds)

Figure 1.3 Four main challenges in image sensors.

Conventional CMOS imagers without integrated dynamic range enhancement

schemes do not have enough dynamic range and sensitivity. The third challenge is large

bandwidth (spatial) for large pixel arrays. The image signal has 8-bit (gray-scale) or > 10-

bit (color) bit-depth. The number of pixels is typically larger than 10,000 (100×100

pixels). Therefore we need to transmit 80,000-bit for the transmission of one image frame,

which is too large in bandwidth-limited applications. The image compression schemes

such as JPEG and DPCM can reduce the bandwidth. However, the image compression

entails the loss of information and requires power-consuming processing. The last

challenge is high temporal resolution (or frame rate). Conventional image sensors have

fixed frame rate from 15 to 60 frames per second (fps). Some high-speed sensors have

high frame rate (> 1,000 fps) in order to suppress the motion-blur from moving objects

[1.9-1.11]. As mentioned, transmitting 80k-bit with 15 fps requires huge bandwidth and

power consumption. However, it is rare that the event we want to monitor occurs 15

times in a second. Therefore, imaging with constant frame rate is redundant. If sensors

can monitor specific event and provide event-driven image transmission, temporal

redundancy will be dramatically suppressed.

7
 It is difficult to optimize for the four main challenges because these parameters have

trade-off relationships each other. For example, the reduction of power consumption

entails the reduction of dynamic range and bandwidth. Bandwidth reduction also entails

the reduction of dynamic range or the increase of power consumption when we use

complex image processing algorithms. Therefore, we need to find optimization points

that consider these four parameters: power consumption, dynamic range & sensitivity,

spatial bandwidth and temporal bandwidth. However, this optimization point varies

according to the applications and time.

1.4 Research goals

The main goal of this work is implementing an energy-efficient image sensor that is

applicable to the 3rd generation imaging system including distributed image sensor

networks and biomedical sensors, i.e., implementing an image sensor with low-power

consumption, wide dynamic range, and low spatial-temporal bandwidth. Considering four

main parameters (power consumption, dynamic range & sensitivity, spatial bandwidth,

and temporal bandwidth), an efficient way to implement an image sensor is providing the

adaptability, because these four parameters always have trade-off. Figure 1.4 shows the

proposed concept of an adaptive image sensing scheme. Three adaptation schemes

suppress the redundancy of data and reduce the power consumption by adapting to the

environment. In the adaptive imaging, the first procedure starts from motion sensing to

reduce temporal bandwidth. The sensor is in sleep mode, while only the integrated

motion sensor is turned on, and monitors with extremely low-power consumption. The

sensor is woken up and is triggered to operation mode from the movement of the object.

By motion-driven activation, we can eliminate constant imaging and constant image

8
transmission with fixed frame rate. Therefore, we can save both the power consumption

and the bandwidth especially in a stationary environment. Then, the motion-triggered

sensor monitors the object in the focal plane. As mentioned, not all of the objects in the

scene are of interests. For example, a swaying tree in the background is not interesting.

Power consumption Illumination range

Energy source Illumination
adaptation adaptation
Adaptive power supply Adaptive dynamic range &
from the energy harvesting & sensitivity
illuminations conditions from the illumination conditions

Spatial / temporal resolution

Object
adaptation
Object-of-interest imaging
from the target-object detection

Figure 1.4 Concept of adaptive image sensing.

The sensor detects target objects in the 2nd procedure, i.e., it classifies whether the

object in the focal plane is of our interest or not. In the 2nd procedure, instead of

generating an image signal which has a large amount of data and redundancy, we

generate a low-bandwidth feature for the object detection. The generated feature can be

processed for the object detection in the host or in the local node. The processor classifies

the object and feeds back the request signal to the sensor. If a target object is detected,

imaging operation is initiated. This object-of-interest imaging for the object adaptation

saves power consumption and bandwidth further.

9
 After thhe object deetection, thee sensor staarts the imaaging operaation with loow power

c
consumptio n. The mainn goal of thhe imaging is
i to providde a quality image signal in wide

r
ranges of iilluminationn with limitted energy source. Foor low pow
wer consum
mption, the

p
proposed seensor has ann energy soource adaptaation using adaptive power supplyy voltage.

F wide dynamic
For d rannge, the seensor also hhas an illuumination aadaptation which
w has

a
adaptive dynnamic rangee and sensittivity for varrying illumiination condditions.

Figure 1.5 Operation of aadaptive im

mage sensin
ng.

I order to iimplement tthe adaptivee image sensor, we impplemented tw

In wo prototyppe chips as

f
follows:

1. An adaptive C
CMOS imagge sensor w
with reconfigurable m
modes of ooperations:

monstrates thhe operationn of the eneergy-sourcee adaptationn and the illuumination

Dem

adapptation

10
 2. A CMOS image sensor with embedded feature extraction for motion-triggered

object-of-interest imaging: Demonstrates the operation of the motion adaptation

and the object adaptation

1.5 Thesis organizations

Chapter 2 introduces a CMOS image sensor including basic operation principles. In

chapter 3, the power consumption of CMOS image sensor is estimated first. From this

estimation, the challenge and strategy of low-power CMOS imager will be discussed.

Chapter 3 also illustrates previous works in three categories that are essential in the

wireless image sensing systems: low-power CMOS image sensors, wide dynamic range

& high-sensitivity image sensors, and image sensors with integrated image processing for

the bandwidth suppression.

Chapter 4 describes an energy / illumination-adaptive image sensor. A proposed

adaptive CMOS image sensor with reconfigurable modes of operations will be explained.

Chapter 5 illustrates an object-adaptive image sensor. A motion-triggered object-of-

interest imaging based on the integrated feature extraction will be explained.

Chapter 6 summarizes this work and discuss about future works.

11
Chapter 1 References

[1.1] E. R. Fossum, “CMOS image sensors: electronic camera-on-a-chip,” IEEE

Transactions on Electron Devices, vol. 44, no. 10, pp. 1689-1698, 1997.
[1.2] M. Rahimi, S. Ahmadian, D. Zats, R. Laufer, and D. Estrin, “Magic of Numbers
in Networks of Wireless Image Sensors,” in Workshop on Distributed Smart
Cameras (DSC), 2006.
[1.3] M. Zhang, A. Bermak, X. Li, and Z. Wang, “A low power CMOS image sensor
design for wireless endoscopy capsule,” in 2008 IEEE Biomedical Circuits and
Systems Conference, 2008, pp. 397-400.
[1.4] Cui X, Lee LM, Heng X, Zhong W, Sternberg PW, Psaltis D, Yang C. “Lensless
high-resolution on-chip optofluidic microscopes for Caenorhabditis elegans and
cell imaging,” Proc Natl Acad Sci USA, 2008
[1.5] J. Ohta, T. Tokuda, K. Hiyama, and S. Sawamura, “Retinal stimulator embedded
with light-sensing function in distributed microchip architecture for subretinal
implantation Backside Topside Stimulus Electrodes,” in International Image
Sensor Workshop, 2009, pp. 2-5.
[1.6] H.-G. Graf et al., “High Dynamic Range CMOS Imager Technologies for
Biomedical Applications,” IEEE Journal of Solid-State Circuits, vol. 44, no. 1, pp.
281-289, Jan. 2009.
[1.7] A. El Gamal, H. Eltoukhy, "CMOS image sensors," Circuits and Devices
Magazine, IEEE , vol.21, no.3, pp. 6-20, May-June 2005
[1.8] I. L. Fujimori and C. G. Sodini, “A 256×256 CMOS differential passive pixel
imager with FPN reduction techniques,” IEEE Journal of Solid-State Circuits, vol.
35, no. 12, pp. 2031-2037, 2000.
[1.9] S. Lauxtermann, P. Schwider, P. Seitz, H. Bloss, J. Ernst and H. Firla, “A high-
speed CMOS imager acquiring 5000 frames/sec,” IEDM Technical Digest, pp.
875-878, Dec. 1999.
[1.10] A. I. Krymski, Nianrong Tu, “A 9-V/Lux-s 5000-Frames/s 512×512 CMOS
Sensor,” IEEE Transactions on Electron Devices, vol. 50, no. 1, pp. 136-143,
2003.
[1.11] S. Kleinfelder, S. –H. Lim, and A. El Gamal, “A 10000 Frames/s CMOS Digital
Pixel Sensor,” J. Solid State Circuits, vol. 36, no. 12, pp. 2049-2058, Dec. 2001.

12
 C
CHAPTER
R 2 INTROD
DUCTION TO CMOS
S IMAGE SENSORS
S

2 Image ssignal chain

2.1 n and overaall architectture
An imagee sensing iss the processs that meaasures the number
n of inncident phootons. The

f
final outputt from imagge sensors iss the digitall image signnal that quaantizes the nnumber of

i
incident phootons. An im
mage signal chain is shoown in figuure 2.1. Inciddent photonns generate

e
electrons inn photo-deteectors. The photo-geneerated electrrons are sw
wept by elecctric field,

w
which is ffrom reversse-biased voltage appllied to phooto-detectorrs. In this way, the

p
photocurren
nt (iph) flow
ws in photoo-detectors. The photoocurrent is integrated in photo-

d
detectors annd is converrted into thee voltage VPD. This vooltage is readd out thouggh in-pixel

a
amplifier cirrcuits. Finallly, output voltage
v VOUUT is convertted to the digital signal..

Figurre 2.1 Imagge signal chain.

13
 Figure 2.2 shows one example of the CMOS image sensor architectures. CMOS

image sensors include pixel array, column-parallel correlated double sampling (CDS)

circuits & ADCs, latches, row scanners and column scanners.

Row scanner
10-b image

Figure 2.2 Overall architecture of CMOS image sensor.

The operation is as follows: (1) incident photons are converted into electrons, and

electrons are accumulated in the photodiode in each pixel during the integration time

(TINT, also called exposure time) (2) in-pixel circuits convert integrated electrons into

voltage. (3) The row scanner scans and selects one row out of pixel arrays. One row of

pixels selected by the row scanner is read out through the column line. (4) CDS circuits

cancel the fixed pattern noise that is inherent in pixel arrays. CDS circuits also amplify

the signal and suppress the input-referred noise. (5) Noise-cancelled signals from CDS

circuits are converted into the digital signals through ADCs. Converted digital signals are

temporarily stored in latches before the readout. (6) Column scanners scan latches

14
column by column, read out digital signals. (7) The procedure repeats for entire rows, and

one image frame is generated.

2.2 Photogeneration
A photon is the carrier of electromagnetic radiation of all wavelengths. It is massless

and is traveling at the speed of light. The energy of photon can be expressed as Eq. 2.1.

hc
E = hν = (2.1)
λ

,where h is Planck’s constant, c is speed of light λ is wavelength, and ν is frequency.

Silicon is the most widely used material for photo-detectors as well as for

contemporary VLSI circuits. With incident photons, electron-hole pairs (e-h pairs) are

generated. This process is called photogeneration. Separated e-h pairs can be combined

and this process is called recombination. The photogeneration and recombination process

is shown in figure 2.3. Silicon is in the equilibrium state without any incident photons. In

the equilibrium state, the number of electrons in the conduction band (n0) is equal to the

number of holes in the valence band (p0). Electron-hole (e-h) pairs are continuously

generated from the thermal energy. However, generation rate (G) is equal to

recombination rate (R) in the equilibrium.

When there are incident photons, the equilibrium is broken. When the photon energy

(E=hν) is higher than the bandgap energy (Eg), photons collide with electrons in the

lattice, and electrons absorb the energy of photons to become free electrons. Therefore,

excess e-h pairs are generated as Eq.2.2:

n = n + ∆n, p = p + ∆p (2.2)

15
,where n0 is electron concentrations in the conduction band (@ equilibrium) and p0: hole

concentrations in the valence band (@ equilibrium). Generated e-h pairs are diffused and

recombined within minority carrier lifetime (τ). Recombination rate (R) can be expressed

as Eq. 2.3.

∆n
R= (2.3)
τ

Incident photons into silicon surface travel in silicon. After traveling some distance, they

are finally absorbed. Figure 2.4 shows the photonflux. The photon flux F can be

expressed as Eq. 2.4.

Figure 2.3 Image signal chain.

dF(x)
= −αF(x) → F(x) = F e (2.4)
dx

16
,where F0 is the photon-flux of incident photons (F(0)), α is the absorption coefficient

[cm-1] that is material and wavelength dependent. The inverse of absorption coefficient

α-1 is the penetration depth at which the photon-flux is reduced to e-1 (=0.37). Note that

photons are absorbed after traveling the distance of α-1 (penetration depth). The

penetration depth is dependent on both the material and the wavelength. As shown in

figure 2.4, photons that have the wavelength of blue travel short distance, whereas

photons that have the wavelength of red travel long distance because the penetration

depth is proportional to the wavelength.

Green
Blue

Red

Figure 2.4 Absorption of the photon.

The generation rate of e-h pairs at the distance of x is proportional to the photon flux

density dF(x)/dx as shown in Eq. 2.5.

d(F − F(x))
G(x) = = αF(x) = αF e (2.5)
dx

17
2 Photodeetectors
2.3
Since phooto-generateed electronss are lost bby the recoombination,, we have to collect

e
electrons inn order to m
measure thee number oof photons. In order too prevent thhe loss of

e
electrons, w
we can just m
move electroons by electtric field andd collect in the collectioon sites as

s
shown in figgure 2.5. Inn order to reaach a detecttable signal level, electtrons are acccumulated

i the capaccitor during the integrattion time. Thhe device thhat providess the electricc field and
in

a the storrage capacittor is the phhotodiode, w

also which is madde of p-n junnction.

Figu
ure 2.5 Charrge collection.

2
2.3.1 P-N photodiode
p
A p-n juunction photodiode is the photodeetector thatt is availablle from thee standard

C
CMOS proccess. Figure 2.6 shows tthe p-n juncction photoddiode.

Figu
ure 2.6 P-N
N photodiod
de.

18
 The p-n junction is typically formed with n+/pwell or with nwell/p-sub. Photodiodes

operate in the reverse biased condition. In the reverse biased condition, incident photons

induce photocurrents in photodiodes.

p - type n - type

generation

generation
generation
recombination recombination

E-field
Reverse bias

photocurrent
potential
Diffusion
length

Depletion
region
x
LP W LN
d1 d2

Figure 2.7 Photogeneration in P-N photodiode.

Photocurrent

As shown in figure 2.7, photo-generated electrons are swept by electric field in

depletion regions. Note that photo-generated electrons in neutral regions are mostly

recombined. In neutral regions, only a small portion of electrons that are generated within

the diffusion length (Lp, Ln) from depletion regions diffuse into depletion regions and

contributes to the photocurrent (Jph(n), Jph(p), diffusion currents). Electrons from depletion

regions mostly contribute to the photo-current (Jdep).

19
 Figure 2.8 shows the components of photocurrent. The total photocurrent density can

be expressed as Eq. 2.6.

J =J ( ) +J +J ( )

(2.6)
≈J = −q G(x)dx = qF e −e

,where d2-d1 is the depletion width W. In order to maximize the photocurrent,

photodiodes should be optimized in order to have large depletion region. This can be

achieved by (1) lowering the doping concentration of the bulk (2) shallow junction and (3)

large reverse bias.

Figure 2.8 Photocurrent.

Dark current

Without any illumination, there is a leakage current in the photodiode. This is called

“dark current”. It is typically process dependent variable [2.1]. The dark current cause

shot noise and fixed pattern noise, therefore, it decrease the SNR. The mechanism of the

dark current can be categorized by two: one is the reverse-bias leakage current and the

other is the surface generation current [2.2, 2.3]. Figure 2.9 shows the dark current

20
mechanism. The reverse bias leakage current mostly consists of two components: (1)

generation current from the depletion current (thermal generation), (2) minority carrier

diffusion current in neutral regions. The source of surface generation current is two: (1)

the surface of n and (2) the interface with STI. Carriers are generated in the near vicinity

of a surface via the interaction with interfacial taps due to the discontinuity of the lattice

structure. The dark current equations can be expressed as follows:

n
Generation current: J = q W (2.7)
τ

D n
Diffusion current: J =q (2.8)
τ N

qG
Surface generation current: J = (2.9)
2

Total dark current: J =J +J +J (2.10)

,where ni is the intrinsic concentration of silicon, τ is the generation lifetime (τ = τ +

τ ), W is the depletion width, GS is the surface generation rate (GS = sgni), sg is the

surface generation velocity. The intrinsic concentration ni is shown in Eq. 2.11 and the

depletion width W is shown in Eq. 2.12.

/
T
n = N N e /
=A e / (2.11)
300

2ϵ(N + N )(V − V)
W= (2.12)
qN N

21
,where NC, NV are carrier densities, Eg is the bandgap energy, NA, ND are doping

concentrations, Vbi is the built-in potential of the p-n junction.

Figure 2.9 Dark current mechanism.

According to Eq. 2.7 ~ 2.10, we can notice the factors affecting the dark current: (1)

Temperature: if temperature increases, dark current also increases because of ni term in

the generation current and ni2 term in the diffusion current. (2) Doping concentration: if

doping concentration increases (accordingly more defects), dark current increases due to

the decrease of τG. (3) Reverse bias voltage: more reverse-biased voltage increases dark

current due to increased W.

The dark current in p-n photodiodes decreases the SNR. This dark current is mostly

from the surface generation, which is generated from the interface defects in the surface

and in the STI region. Moreover, electrons that are generated near the surface can be

easily trapped in the surface states and recombined, which do not contribute to the

photocurrent. Therefore, p-n photodiodes have poor blue response.

By process variations, the surface can be isolated from the interface. Therefore, dark

current can be suppressed. The photodiode which has isolated surface is called “pinned

photodiode”. The pinned photodiode will be explained in section 2.3.2.

22
C
Charge inteegration an
nd detection
n

The phooto-current (I
( ph) is integgrated in thee junction capacitance
c of p-n juncctions, and

l
lowers the potential
p off cathode (inn electron-ppotential, electrons are accumulateed and the

p
potential inncreases) ass shown in figure 2.10. Note that the darkk current (IId) is also

i
integrated. Operation procedurees are as follows: ((1) Before the integgration of

p
photocurren
nt, the cathoode of phootodiode is reset (VR) (2) photocuurrent dischharges the

j
junction caapacitance ((CD) duringg the integrration time (TINT). Noote that thee parasitic

c
capacitance (CP) also contributes
c to the sensse node volttage VPD. (33) After TINNT, VPD is

r
read out using in-pixel buffers. (4) Reset againn, next fram
me starts.

Figuree 2.10 Charrge integrattion.

F well caapacity
Full

Like waater is spilleed out of botttle when w

we pour morre than the m
maximum capacity of

t bottle, same phenomena happens in phhotodiodes. When illum

the mination is high and

e
electrons arre full of thee capacitancce, electronss are spilledd over. The definition
d o full well
of

c
capacity is tthe maximuum capacityy of charge sstorage of pphotodiodes. A full welll capacity

c be expreessed as Eqq. 2.13.

can

223
 1
N = C (V)dV [electrons] (2.13)
q

C (V) ≈ C = A ( )( )
for p-n junction

/
≈A for p-n+ junction
( )

,where CPD is the photodiode capacitance, Cj is the junction capacitance, VRESET is the

reset voltage and VSWING is maximum voltage swing of the photodiode. Assuming CPD is

constant, full well capacity can be expressed as Eq. 2.14.

C V
N = [electrons]
q
(2.14)

Q =C V [C]

Conversion gain

The conversion gain is defined as the amount of voltage variation from one electron.
q
A =
Capacitance of detection node
(2.15)
q
= [μV/electron]
C

Higher conversion gain means that small amount of photons can induce high voltage

swing. Therefore, higher conversion gain increases the sensitivity, which will be

explained in the next subsection. In p-n photodiodes, the capacitance of detection node is

CPD. It means that higher the conversion gain, the full well capacity is lowered. If we can

separate the detection node from photodiodes, we can achieve both high full well

capacitance and high conversion gain. This will be revisited in the section 2.3.2.

24
Sensitivity

The sensitivity is defined as the ratio of output signal [V] to an incident illumination

level [lx] in a second. The unit of sensitivity is [V/lx·s]. Note that high conversion gain

increase the sensitivity, however, high sensitivity does not guarantee that the pixel has

high conversion gain. The sensitivity can be increased from process optimization,

microlens optimization, and so on.

Fill factor

Pixels consist of photodiodes and in-pixel readout circuits (typically source follower).

The circuit area is shielded by metal layers in order to prevent from the photogeneration.

Only the area of photodiode is exposed. Fill factor is defined as the portion of the pixel

area that contributes to photo-sensitivity, i.e., the ratio of photodiode area to the pixel

area. It is expressed as Eq. 2.16.

A
FF = [%] (2.16)
A

Larger fill factor increases the sensitivity because photodiodes collect more photons.

One way to increase fill factor is shared pixel architecture. In shared pixel architecture,

neighboring pixels share one readout circuit, and improves fill factor. Another way to

achieve high fill factor is the backside illumination (BSI) technology. Using the BSI

technology, photodiodes are exposed to the backside and the metal lines are located in the

front side.

2.3.2 Pinned photodiode

The structure of a pinned photodiode is shown in figure 2.11. On top of the p-n

junction, shallow p+ region is formed. This additional p+ implant isolates the n-region

25
from the surface and also from the STI region in order to suppress the dark current.

Moreover, it offers better blue response compared with p-n photodiodes since the surface

is isolated [2.2]. Note that n-region is fully depleted in order to collect more photons.

Figure 2.11 Pinned photodiode.

In this structure, the profile has a potential well due the p+-n-p structure. Therefore,

photogenerated electrons can be collected in the potential well. The minimum potential of

the well is called “pinning potential” which is generated from the profile of p+-n-p

structure. Another advantage from pinned photodiodes is that we can completely transfer

the collected electrons into the sense node without residual charges. Since the pinning

potential is higher than the potential of sense node (which is reset to the power supply

voltage), electrons in the well are completely transferred to the sense node with the aid of

potential difference (electric field).

2.4 Pixel circuit

Pixel circuits consist of a photodiode and an in-pixel readout circuit. The photodiode

has two types: p-n photodiodes and pinned photodiodes. In-pixel circuit drives the

column line that has large parasitic capacitance. A source follower is typically used as an

26
in-pixel readout circuit. According to the number of transistors in pixel, the pixel is

categorized as 3-T or 4-T pixel.

2.4.1 3-T Pixel with p-n photodiode

Figure 2.12 shows the 3-T pixel with p-n photodiode. Note that the bias current (not

shown in the figure) is connected with column line (SIG) and is shared by all rows. The

control signals (RST, SEL) is driven by the row scanner.

The operation procedure is described with electron potential diagrams in figure 2.13.

The operation procedure is as follows: (1) Reset: the PD node is reset to VDD – VTHR due

to the VT drop in the reset transistor. In the reset operation, kTC reset noise (VkTC1) that is

thermal noise generated in the R-C circuit is added. (2) Integration: VPD is decreased by

integration of photocurrents. ‘Q’ is the integrated charge. (3) Signal readout: after

integration time (TINT), VPD is read out through source followers. (4) Reset: reset PD

node again for the next frame. (5) Reset level readout: this phase is for the correlated

double sampling, which cancels the fixed-pattern-noise (FPN) in pixel.

SIG

Figure 2.12 3-T pixel with p-n photodiode.

27
 Figure 2.13 Operation of 3-T pixel.

The FPN is mainly from the VT variation (in reset transistors and in source follower

transistors) which is different from each pixel. By subtracting the reset level from the

signal level, VTHR term is removed. In this phase, another kTC noise is added (VkTC2). (6)

Double sampling: VTHR term is removed. However, VkTC is not cancelled from the double

sampling because two kTC noise are not correlated each other, i.e., these two kTC noise

are added in different timing. Since the kTC noise power can be expressed as kT/C, 3-T

pixel has low SNR due to small capacitance of CPD.

Moreover, as explained in the previous section, full well capacity and conversion gain

are in trade-off relationships, i.e., larger well capacity induces lower conversion gain.

Therefore, it is desirable to separate the detection node from photodiodes. The 4-T pixel

28
separates the detection node by using an additional transistor between the photodiode and

the source follower.

Figure 2.14 4-T pixel with pinned photodiode.

2.4.2 4-T Pixel with pinned photodiode

Figure 2.14 shows the 4-T pixel with pinned photodiode. As shown in the figure, the

detection node which is called floating diffusion ‘FD’ is separated from photodiodes. The

operation procedure is shown in figure 2.15. Assuming that FD node is reset before the

integration, the operation is as follows: (1) Integration (2) Reset (FD): before the signal

readout, reset FD node. The kTC noise is added (VkTC1) (3) Reset level readout: read out

the reset level for noise cancelling (4) Charge transfer: charges in photodiodes are

transferred into the floating diffusion (TX is on). As explained before, pinning potential

is higher than FD node, charges are fully transferred. (5) Signal readout: note that the

signal has same kTC noise as in the reset phase (VkTC1) since charges are dumped into the

floating diffusion which already has kTC noise VkTC1. (6) Correlated double sampling:

the CDS operation cancels both the FPN and the kTC noise.

29
 Figure 2.15 Operation of 4-T pixel.

Since kTC noise is cancelled in 4-T pixel, a 4-T pixel architecture offers better SNR

compared with a 3-T pixel. Moreover, conversion gain of 4-T pixel (AC(4T) = q/CFD) is

higher than 3-T pixel (AC(3T) = q/CPD). Note that the capacitance of FD (CFD) consists of

the gate capacitance of one transistor and the junction capacitance of two transistors. The

capacitance of FD is small (typically < 5 fF) for the high conversion gain.

2.4.3 Shared-pixel architecture

The in-pixel circuits including the switch and the source follower takes some portion

of the pixel area and decreases the sensitivity in a given pixel size. In order to increase

the fill factor, neighboring pixels can share one readout circuit. This is called shared-pixel

architecture [2.13-2.16]. Figure 2.16 shows 4-shared pixel circuit which has one readout

circuit shared by four vertically neighboring pixels. In the shared pixel architecture, two

30
or four neighboring pixels share one floating diffusion (FD) node, one reset transistor and

one source follower circuit.

Figure 2.16 Shared pixel circuit.

In addition to small pixel size, we can easily perform the pixel merging in the shared

pixel architecture. The pixel merging is the process to add signals from multiple pixels

and provides better sensitivity and SNR (especially at low illumination) while reducing

the spatial resolution. In the shared pixel architecture, integrated charges from

photodiodes in a shared group are simultaneously transferred into a floating diffusion

node and summed.

Contemporary CMOS imager typically uses two or four shared architecture. If more

pixels are shared, we can achieve smaller pixel size with high fill factor. However, the

31
capacitance of FD node induces decreased conversion gain. Moreover, the leakage

current from the other photodiodes in a shared group can corrupt the signal especially at

high illumination. Asymmetry of the layout is another issue in the shared pixel

architecture because only one readout path exists in multiple pixels. Due to the

differences in adjacently located circuit components, photodiodes are no longer identical

and the fixed pattern noise (FPN) can be induced [2.17].

Figure 2.17 Correlated double sampling circuit.

2.5 Correlated double sampling (CDS)

A correlated double sampling (CDS) is an essential operation in order to suppress the

fixed pattern noise (VFPN) and the kTC noise (VkTC). The fixed pattern noise is mainly

from the variation of threshold voltage in in-pixel readout circuits. The kTC noise is the

thermal noise in the RC circuit that is generated from the reset transistor and the sense

node capacitance. In order to cancel the noise (VN = VFPN + VkTC), CDS circuits sample

reset voltage (VRST + VN) and signal voltage (VSIG + VN) successively, and subtract each

other. One simple CDS circuit is shown in figure 2.17. The circuit consists of an

operational amplifier, two capacitors (C1 and C2) and switches.

32
 The operation is two-step procedure. The first step is the sampling phase. Unity

feedback is formed by turning on the switch ‘F’. While unity feedback is formed, the

reset voltage (VRST + VN) is sampled on C1. In this phase, charge in each capacitor can be

expressed as follows:

Q = C1{V – (V + V )}
(2.17)
Q = 0

The second step is the amplification phase. Unity feedback is removed by turning off

‘F’. After the unity feedback is removed, the signal voltage from the pixel is input to C1.

Since the ‘-‘ input of the operational amplifier tries to keep the voltage VREF due to the

feedback through C2, the charge variation from the new input VSIG + VN is totally

delivered into C2. Now the charge in each capacitor can be expressed as follows:

Q = C {V – (V + V )}
(2.18)
Q = C (V −V )

The output of the CDS circuit can be expressed as follows:

C
V =V + (V −V ) (2.19)
C

CDS circuits generate noise cancelled signal (VRST - VSIG). CDS circuits can be

implemented to amplify the signal by making the capacitance of C2 programmable (PGA,

programmable gain amplification) [2.3]. Using the amplification by the gain A, the input-

referred noise will be suppressed as V /A where VN2 is the noise in the signal chain

after the CDS circuit (mostly from the ADC). The variable capacitance can be

33
implemented with configurable capacitance of C2. For example, C2 can be implemented

with

C2 = C1/8 + S0C1/8 + S1C1/4 + S2C1/2

A = 1: C2 = C1 (S2S1S0 = “111”)

A = 2: C2 = C1/2 (S2S1S0 = “110”)

A = 4: C2 = C1/4 (S2S1S0 = “100”)

A = 8: C2 = C1/8 (S2S1S0 = “000”)

The gain A is controlled globally, i.e., same gain is applied to entire pixel array. The

amplified signals are input to the ADC, and this amplification suppresses the input

referred noise.

The CDS circuit shown in figure 2.17 induces another FPN. The operational amplifier

has the offset voltage from the mismatch of differential pair, i.e., there is voltage

difference (VOFFSET) to make each current in the differential pair same. When unity

feedback is formed, the ‘-‘ input voltage will be VREF + VOFFSET. Since the CDS circuit is

typically implemented with column parallel, each column generates different VOFFSET and

the output image will have vertical stripe pattern. This FPN can be cancelled by 2nd CDS

in the ADC, i.e., ADC samples both VREF + VOFFSET (when CDS is in sampling phase)

and VREF + VOFFSET + VSIG and cancels each other. The result of 2nd CDS is desired signal

VSIG.

34
2.6 Analog-to-digital converter (ADC)
CMOS image sensors have integrated ADCs for the digital signal output. As shown in

figure 2.18, there are two schemes to implement ADCs: serial ADCs and column-parallel

ADCs.

2.6.1 Serial ADC

Serial ADCs convert the signal of entire pixels serially. One row of signals is read out

to the line buffer, which is implemented with sample and hold circuit. The signals stored

in the line buffer are accessed serially by the column scanner. Therefore, high-speed

ADCs such as pipeline ADCs are required [2.12]. As the pixel array is scaled to the large

format, the speed bottleneck in the ADC is a critical factor.

10-b image

10-b image

Figure 2.18 ADC schemes in CMOS image sensor.

2.6.2 Column-parallel ADC

Column-parallel ADCs convert the signal of one row in parallel. Cyclic ADCs,

successive approximation (SAR) ADCs and single-slope (SS) ADCs are widely used for

column-parallel ADCs [2.4]-[2.9]. Among them, SS ADCs are mostly widely used in

CMOS image sensors.

35
 Figure 2.19 shows the architecture and the operation principle of the SS ADC. Input

voltage (VIN) is compared with the ramp signal (RAMP). When ramp signal crosses the

input voltage, the comparator output toggles and latches the counter value into the latch.

A single-slope ADC takes relatively small area because it requires only one

comparator and latches. Moreover, it provides better linearity than cyclic or SAR ADC

because the DAC output (ramp signal) is monotonously increased. The matching between

columns is the most critical factor in column parallel ADCs. The SS ADC provides good

matching because the ramp signal is globally applied to entire columns and the gain

variation of the comparator is not so critical as long as the comparator has enough gain.

The main drawback of SS ADCs is long A/D conversion time. For n-bit conversion, SS

ADCs have 2n clock cycles whereas SAR and cyclic ADC have n clock cycles.

Figure 2.19 Single-slope ADC.

36
Chapter 2 References

[2.1] H. I. Kwon, I. M. Kang, B.-G. Park, J. D. Lee, and S. S. Park, “The Analysis of
Dark Signals in the CMOS APS Imagers From the Characterization of Test
Structures,” IEEE Transactions on Electron Devices, vol. 51, no. 2, pp. 178-184,
Feb. 2004.
[2.2] N. V. Loukianova, H. O. Folkerts, J. P. V. Maas, D. W. E. Verbugt, A. J. Mierop,
W. Hoekstra, E. Roks, A. J. P. Theuwissen, “Leakage current modeling of test
structures for characterization of dark current in CMOS image sensors. IEEE
Transactions on Electron Devices, 50(1), pp. 77-83. 2003
[2.3] J. Nakamura, Image Sensors and Signal Processing for Digital Still Cameras,
Taylor & Francis
[2.4] K. Mabuchi et al., “CMOS image sensors comprised of floating diffusion driving
pixels with buried photodiode,” IEEE Journal of Solid-State Circuits, vol. 39, no.
12, pp. 2408-2416, Dec. 2004.
[2.5] N. Kawai and S. Kawahito, “Noise Analysis of High-Gain, Low-Noise Column
Readout Circuits for CMOS Image Sensors,” IEEE Transactions on Electron
Devices, vol. 51, no. 2, pp. 185-194, Feb. 2004.
[2.6] S. Kawahito, J.-H. Park, K. Isobe, S. Shafie, T. Iida, and T. Mizota, “A CMOS
Image Sensor Integrating Column-Parallel Cyclic ADCs with On-Chip Digital
Error Correction Circuits,” IEEE International Solid-State Circuits Conference
Digest of Technical Papers, pp. 56-595, Feb. 2008.
[2.7] M.-W. Seo et al., “An 80μV<inf>rms</inf>-temporal-noise 82dB-dynamic-range
CMOS Image Sensor with a 13-to-19b variable-resolution column-parallel
folding-integration/cyclic ADC,” IEEE International Solid-State Circuits
Conference Digest of Technical Papers, pp. 400-402, Feb. 2011.
[2.8] I. Takayanagi et al., “A 1.25-inch 60-frames/s 8.3-M-pixel digital-output CMOS
image sensor,” IEEE Journal of Solid-State Circuits, vol. 40, no. 11, pp. 2305-
2314, Nov. 2005.
[2.9] B. Pain and E. R. Fossum, “CMOS active pixel sensor with on-chip successive
approximation analog-to-digital converter,” IEEE Transactions on Electron
Devices, vol. 44, no. 10, pp. 1759-1763, 1997.

37
[2.10] M. F. Snoeij, P. Donegan, A. J. P. Theuwissen, K. A. A. Makinwa, and J. H.
Huijsing, “A CMOS Image Sensor with a Column-Level Multiple-Ramp Single-
Slope ADC,” IEEE International Solid-State Circuits Conference. Digest of
Technical Papers, pp. 506-618, Feb. 2007.
[2.11] S. Yoshihara et al., “A 1/1.8-inch 6.4MPixel 60 frames/s CMOS Image Sensor
with Seamless Mode Change,” IEEE International Solid State Circuits
Conference - Digest of Technical Papers, pp. 1984-1993, Feb. 2006..
[2.12] K. –B. Cho, C. Lee, S. Eikedal, A. Baum, J. Jiang, C. Xu, X. Fan, “A 1/2.5 inch
8.1Mpixel CMOS Image Sensor for Digital Cameras,” IEEE International Solid-
State Circuits Conference Digest of Technical Papers, pp. 508-618, Feb. 2007.
[2.13] M. Mori, M. Katsuno, S. Kasuga, T. Murata, and T. Yamaguchi, “A 1/4in 2M
pixel CMOS image sensor with 1.75 transistor/pixel,” IEEE International Solid-
State Circuits Conference Digest of Technical Papers, pp. 110-111, Feb. 2004.
[2.14] Y. C. Kim, Y. T. Kim, S. H. Choi, H. K. Kong, S. I. Hwang, J. H. Ko, B. S. Kim,
T. Asaba, S. H. Lim, J. S. Hahn, J. H. Im, T. S. Oh, D. M. Yi, J. M. Lee, W. P.
Yang, J. C. Ahn, E. S. Jung, Y. H. Lee, “1/2-inch 7.2MPixel CMOS Image
Sensor with 2.25/spl mu/m Pixels Using 4-Shared Pixel Structure for Pixel-Level
Summation,” IEEE International Solid State Circuits Conference Digest of
Technical Papers, pp. 1994-2003, Feb. 2006.
[2.15] N. Tanaka, J. Naruse, A. Mori, R. Okamoto, H. Yamashita, and M. Monoi, “A
1/2.5-inch 8Mpixel CMOS image sensor with a staggered shared-pixel
architecture and an FD-boost operation,” IEEE International Solid-State Circuits
Conference Digest of Technical Papers, pp. 44-45, Feb. 2009.
[2.16] H. Takahashi, T. Noda, T. Matsuda, T. Watanabe, M. Shinohara, T. Endo, S.
Takimoto, R. Mishima, S. Nishimura, K. Sakurai, H. Yuzurihara, S. Inoue, “A
1/2.7-in 2.96 MPixel CMOS Image Sensor With Double CDS Architecture for
Full High-Definition Camcorders,” IEEE Journal of Solid-State Circuits, vol. 42,
no. 12, pp. 2960-2967, Dec. 2007.
[2.17] R. D. McGrath, H. Fujita, R. M. Guidash, T. J. Kenney, W. Xu, “Shared pixels
for CMOS image sensor arrays,” IEEE International Image Sensor Workshop, pp.
9-12, Jun. 2005.

38
 CHAPTER 3 LOW POWER CMOS IMAGE SENSORS, DYNAMIC RANGE
ENHANCEMENT AND EMBEDDED IMAGE SIGNAL PROCESSING

In this chapter, power consumption of CMOS image sensors will be estimated and

power reduction strategy will be discussed. Then three categories of CMOS image

sensors, which are essential for distributed imaging systems and biomedical sensors: (1)

low-power CMOS image sensors, (2) wide-dynamic range & high-sensitivity CMOS

image sensors and (3) CMOS image sensors with integrated image signal processing for

bandwidth reduction. This chapter will introduce principles, challenges and previous

works.

Since CMOS image sensors can be fabricated with CMOS process, power

consumption can be scaled down compared with CCD image sensors. However,

conventional CMOS image sensors still have large power consumption and they are

difficult to be used with battery or with energy harvesting because applications of

conventional image sensors are mostly 2nd generation imaging systems such as mobile

devices. The SNR and the image quality have been a critical factors rather than power

consumption. Moreover, voltage scaling has a limitation due to the signal range of the

pixel.

In CMOS image sensors, a variety of circuits for image enhancement and processing

can be integrated on a chip in order to implement camera-on-chip solutions. A dynamic

range of conventional CMOS image sensors is around 60 dB, whereas the natural scene

has more than 100 dB dynamic range. Therefore, high-sensitivity readout circuits and

39
dynamic range enhancement schemes should be integrated in order to provide a reliable

monitoring.

A bandwidth is another critical factor because the image signal transmission with

constant frame rate requires huge bandwidth. For bandwidth saving, lightweight image

processing schemes such as the motion sensing or the region-of-interest (ROI) readout

has been integrated in CMOS image sensors. These schemes process the signal from

pixel arrays, and outputs events or features with low bandwidth.

3.1 Power consumption in CMOS image sensors

The block diagram of a CMOS imager for power estimation is shown in figure 3.1. In

the typical approach, separate power supply between analog and digital block are used in

order to achieve higher noise immunity.

VDD

Pixel array

Correlated double sampling

VDDIO
Column parallel ADCs

PAD Latch (10b)

Column scanner

Figure 3.1 Block diagram of CMOS imager.

40
 For the estimation, the analog power supply is further divided by two parts: pixel

(VDDP) and CDS/ADC circuit (VDDA). The digital power supply can be further divided by

two parts: the core including timing controllers and ADC digital circuits (VDD) and I/O

circuits (VDDIO). In this section, power consumption in each block of CMOS imager will

be estimated.

3.1.1 Power consumption: static, dynamic and leakage

The power consumption can be categorized into three: static, dynamic and the

leakage power consumption [3.1].

Static power consumption

The static power consumption can be expressed as follows:

P =I ∙V (3.1)

,where IAVG is the average current and VDD is the power supply voltage.

In imaging operation, most of static power consumption comes from the analog

circuits including pixel arrays, CDS circuits, and also some part of ADCs (mostly from

the comparators). These circuits repeat the operation row by row in rolling-shutter

operation. The average current IAVG can be calculated with the bias current and the duty

cycle as follows:

# ∙T T
I =I ∙ ≅I ∙ (3.2)
T T

,where TON is the time that the circuit is activated. The row access time TROW can be

defined as the time of one-row operation including the pixel readout, CDS (TCDS), ADC

(TADC), digital signal readout (TRD) and blanking time before the next row operation.

Assuming zero blanking time, the TROW can be expressed as:

41
 T =T +T +T (3.3)

Using Eq.3.3, the static power consumption of each circuit can be expressed as

follows:

T
P =I V (3.4)
T

Dynamic Power Consumption

The dynamic power consumption in digital circuits consists of two components: one

is from the load capacitor driving and the other is from the short circuit current during the

transition. In a single gate, power consumption can be expressed as:

P =C V f+ t V I f

= C V f+ C V f (3.5)

= (C + C )V f

The first term shows the dynamic power from capacitor driving. CL is the load

capacitance of the gate and f is the frequency of transitions from ‘0’ to ‘1’. The second

term shows the dynamic power consumption from the short circuit current. tsc represents

the time of short circuit current generation, i.e., both pMOS and nMOS are turned on

during the transition. IPEAK is the peak short circuit current. In CMOS imagers, many

digital circuits have to drive long metal line connected with each column circuits. In this

case, total load capacitance (CL) consists of load capacitance of each column circuit (cl(col))

and also the parasitic capacitance of metal line (cm) as follows:

42
 Eq. 3.6 C = # c( ) + Lc
(3.6)
≈# c( )+W c

,where #COL is the number of columns, L is the length of metal line, and cm is the parasitic

capacitance per unit length. The length L can be expressed as #COLWPIX where WPIX is the

pixel pitch. Some of digital circuits such as the counter in single slope ADCs and the

clock generator operate with high frequency. Therefore, dynamic power consumption in

CMOS imagers is dominated by the power consumption from the capacitor driving.

Leakage Power Consumption

When circuits are not operating, there is power consumption from the leakage current.

The leakage current mainly comes from three components: sub-threshold (sub-VT)

current, junction leakage, and gate oxide tunneling [3.2]. In CMOS imagers, the leakage

power consumption is more severe in digital circuits compared with analog circuits

because digital circuits use thin-oxide transistor with low VT for voltage scaling whereas

analog circuits including pixels use thick-oxide transistor which allows higher voltage

without breakdown in order to enhance the signal swing.

The sub-VT current can be expressed as follows:

I = Ae 1−e (3.7)

,where η is the drain-induced-barrier-lowering (DIBL) coefficient and γ’ is linearized

body effect coefficient. As long as drain-source voltage (VDS) is not zero, the sub-VT

current flows even when the gate-source voltage VGS is zero. Note that low VT

43
significantly increases the sub-VT leakage in the exponential form. In short channel

devices in digital circuits, the DIBL which lowers VT increase the sub-VT current.

The junction leakage is from the reverse-biased p-n junctions. The most significant

factor of junction leakage is from the band-to-band tunneling (BTBT). Due to high

electric field from revere biased p-n junctions, electrons tunnel from the valence band (p

region) to the conduction band (n region).

The gate leakage current is from the gate oxide tunneling. High electric field across

the SiO2 makes electrons in the inverted silicon surface tunnel directly to the gate. This

gate tunneling is more noticeable in the thin-oxide transistors.

Typically, leakage power consumption is much lower and is not comparable to static

and dynamic power consumptions in conventional CMOS imagers. However, some of

low-power imagers operate in the sub-VT region with scaled VDD. In this case, leakage

power is an important factor in order to achieve minimum energy [3.3]. The low-power

CMOS imagers using sub-VT readout circuits will be introduced in section 3.2.

3.1.2 Power Consumption in the Image Signal Chains

In this section, power consumption in main blocks of image signal chains will be

estimated. A signal chain of CMOS imagers is shown in figure 3.2. The operation timing

diagram is shown in figure 3.3. The main parameters for power estimation are shown in

table 3.1. In the estimation of dynamic power consumption, we consider the dynamic

power consumption only from the capacitive driving (CLVDD2f) for simplicity. Moreover,

the leakage power consumption will be ignored. This estimation causes error compared

with actual dynamic power consumption. However, main purpose of power estimation in

this section is to get power reduction strategy by inspecting the tendency of power

44
consumption according to critical parameters such as the number of pixel array, power

supply voltage and pixel pitch. In this section, we will show only estimated equation

without detailed derivation for digital circuit blocks. Detailed estimation of dynamic

power consumption in digital circuit blocks will be shown in Appendix A.

Pixel Array

The circuit schematic of pixels in one column is shown in figure 3.4. The load

capacitance of in-pixel source follower consists of junction capacitance of selection

transistor, column line (SIG) parasitic capacitance and sample and hold capacitance (CS).

The sample and hold capacitor is a part of CDS circuits. The load capacitance of in-pixel

source follower can be calculated as follows:

C =# c +# W c +C (3.8)

The gain-bandwidth product (GB) of the amplifier is:

g 2βI
GB = f = = (3.9)
2πC 2πC

Figure 3.2 Signal chain of CMOS imager.

45
RST 0 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

TX 0 0 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

SEL 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

F 0 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

SS 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

SR 0 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Counter CLK 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

PCH 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 0 0 0

TCDS TADC TREAD

Figure 3.3 Signal chain of CMOS imager

Pixel array #COL × #ROW 320 × 240 (QVGA)

Pixel pitch WPIX 5 um
Frame rate FR = 1/TINT 30 fps
Integration time TINT 33.3 ms
Row access time TROW = 1/(FR·#ROW) 139 μs
CDS time TCDS 8 μs
ADC time TADC = TCLK·2N 99.7 μs
Digital signal readout time TREAD = TCLK ·#COL 31.2 μs
ADC resolution N 10 bit
Pixel power supply VDDP 2.8 V
Analog power supply VDDA 2.8 V
Digital power supply VDD 1.8 V
I/O power supply VDDIO 1.8 V
Clock frequency (period) fCLK (TCLK) 10 MHz (100 ns)
Process transconductance parameter K 50 uA/V2
Capacitance of capacitors
CS, CF 1 pF
in CDS, ADC circuit
Gate capacitance of transistor (minimum W/L) cg 0. 5 fF
Input capacitance of inverter (×1) 3cg 1.5 fF
Parasitic capacitance of metal line (per unit length) cm 0.5 fF

Table 3.1 Parameters for the estimation of power consumption

46
 SIG
Figure 3.4 Pixel circuit with load capacitance of in-pixel source follower

For 1% settling, the 3-dB bandwidth should be 4.6 times larger than the pole

frequency. Therefore, we assume that the time constant is 5 times smaller than the

required time slot (T). Then the 3-dB bandwidth of in-pixel source follower can be shown

as Eq.3.10.

5 2βI
f = = (3.10)
0.5 ∙ T 2πC

In Eq. 3.10, the sampling time is set as 0.5TCDS because two signals have to be

sampled for the correlated double sampling. The source follower needs to be turned on

during the CDS time. Considering the duty cycle (TCDS/TROW), the power consumption of

pixel arrays can be calculated as Eq.3.11.

T
Eq. 3.11 P =# I V
T
(3.11)
(20πC ) T
=# V
2β ∙ T T

Column-parallel CDS Circuit

47
 The CDS circuit consists of the programmable gain amplifier (PGA) with capacitive

feedback as shown in figure 3.5. The PGA consists of one OTA, one input sampling

capacitor (CS) and programmable feedback capacitors (CF). In figure 3.5, the gain of

PGA (APGA) can be set up to 8 as an example. The 3-dB frequency of the CDS circuit can

be described as:

5 1 2β(0.5I )
f = = ∙ (3.12)
0.5 ∙ T A 2π(2C )

The power consumption of the CDS circuit can be calculated as Eq.3.13.

T
P =# I V
T
(3.13)
(40A πC ) T
=# V
β∙T T

Figure 3.5 Correlated double sampling circuit with programmable gain

amplification

Column-parallel Single-slope ADC

The single slope ADC consists of one comparator and latches. The latch can be

implemented with SRAMs, D flip-flops or sometimes counter circuits. The power

48
consumption of the single-slope ADC is dependent on the topology of the comparator

and the latch. In this estimation, one common topology that consists of a static

preamplifier, a dynamic comparator and a 10-b D flip-flop will be analyzed as shown in

figure 3.6. The clock frequency of counter in the single-slope ADC is determined by the

number of pixels and also by the frame rate. The A/D conversion time (TADC) can be

determined as Eq.3.14.

T =T −T −T (3.14)

Note that there is an additional time budget for the digital signal readout (TREAD)

because no additional line memories to store one row of image signals. With line

memories, the ADC output will be stored in the line memory and the signal readout can

be done while the next ADC cycle is performed.

Preamplifier Comparator
VCOM
VRAMP
VDDA VDD
F
SS CS CC
10-b
VO(CDS)
latch
SR CS CC
F
IADC
VCOM

Figure 3.6 Column-parallel single-slope ADC

The A/D conversion time can be expressed with required clock period (TCLK) and the

frequency (fCLK) as Eq.3.15.

49
 1 2 +#
f = = (3.15)
T 1
−T
FR ∙ #

A. Comparator: Preamplifier

The main role of a preamplifier is to suppress the offset voltage of dynamic

comparators by providing a high gain. The offset voltage of the preamplifier can be

suppressed by another CDS operation in the ADC. Therefore, the gain of the preamplifier

is determined by the offset voltage of dynamic comparator. Assuming one LSB is 1 mV

and the offset voltage of the dynamic comparator is 10 mV, the gain of the preamplifier

(APA) over 10 guarantees the offset error under 1 LSB. In this estimation, the differential

amplifier with diode-connected loads will be used as shown in figure 3.7.

The bandwidth of a preamplifier is determined by the clock frequency fCLK. It is

desirable to design the delay of a preamplifier under TCLK in order to latch the counter

signals within 1 LSB range, i.e., keep the bandwidth of the preamplifier over fCLK. The 3-

dB bandwidth of the preamplifier is:

1 g
f =f = ∙
A 2πC
(3.16)
1 2β(0.5I )
= ∙
A 2πC

The load capacitance (CL) of the preamplifier includes the gate capacitance of load

transistors and the gate capacitance of the dynamic comparator. Assuming total load

capacitance (CL) is 16cg, the power consumption of the preamplifier can be calculated as:

50
 32A πf c T +T
P =# V (3.17)
β T

12cg

4cg

Vinp Vinn

IADC

Figure 3.7 Preamplifier in the single-slope ADC

B. Comparator: Dynamic Comparator

The dynamic comparator performs the level decision from pre-amplified signals. It

operates with two phases: the precharge and the comparison. Figure 3.8 shows the

comparator circuit. The first stage that consists of a latch circuit, switches (pMOS for

precharge), and inverters contributes to the power consumption in each clock cycle. The

2nd stage that consists of a RS latch and buffers has only one time switching during the

ADC time. For simplicity, we consider only the gate capacitance for total load

capacitances. The power consumption per one comparator (per column) can be expressed

as Eq.3.18.

2 1
P = 6c V + 12c V (3.18)
T T

51
 Figure 3.8 Dynamic comparator

C. Latches & counter

The latch for the N-bit single-slope ADC consists of N D flip-flops. The circuit

schematic of the D flip-flop is shown in figure 3.9. Note that the clock signal (CK) is

from the comparator output (Q) and the input signal (D) is from the counter. The

comparator output has the frequency 1/TROW and the counter output has different

frequency according to the bit position. The power consumption of one latch in the

single-slope ADC can be represented as follows:

1 f ( )
P _ = 13c +c V + 5c V (3.19)
T T

,where fCNT(n) is the counter output frequency of n-th bit position. As shown in Eq.3.19,

the power consumption of two inverters in the master stage of the latch is dependent on

the input frequency, i.e., counter output frequency fCNT(n). In the single-slope ADC, when

the clock signal is low, the master stage continuously changes its state from the counter

signals. After the clock signal goes high and the counter signal is latched to the slave

stage, no more switching occurs. The power consumption is dependent on the output

52
toggling from the comparator. Therefore, the power consumption is largely dependent on

the signal level (i.e., light intensity) of single slope ADCs. In this estimation, we define

the illumination level parameter ‘I’ which varies 1/2N (dark) to 1 (bright). For The power

consumption of the N-bit latch in single slope ADCs which uses a Gray counter can be

expressed as follows:

P =# N 13c +c + 5c I(1 + 1 + 2

1 (3.20)
+ ⋯+ 2 )V
T

Note that the 2nd term includes the illumination level parameter ‘I’. If the signal level is

high from high illumination, the power consumption increases because the latch has more

switching before the comparator output is toggled.

Figure 3.9 D flip-flop

Compared with a binary counter, a Gray counter provides better noise immunity

because the output change occurs only in one bit position, i.e., timing jitter induces only 1

LSB error. Another advantage is low power consumption because total number of

53
switching is half of the binary counter. Detailed power estimation will be shown in

appendix A. The overall power consumption from the counter can be expressed as:

P = (39c + 0.5# W c )

(3.21)
1 + 2 + ⋯+ 2
×V
T

Digital signal readout

Figure 3.10 shows the digital signal readout circuit and the operation. The output of

the 10-b latch drives the readout buffer. The readout buffer of all columns share one

sense line (SL). The sense line is precharged with VDD first. The column selection signal

CS from the column scanner is enabled column by column. When stored digital signal is

‘1’, the readout buffer draws the current from the precharged sense line and the voltage of

the sense line is dropped. The sense amplifier senses the voltage drop of the sense line.

Sense amp
PCH
Sense line (SL)

Q
CS CPAR

qCOL(n) VREF

(a) Digital signal readout circuit

(b) Operation

Figure 3.10 Digital signal readout circuit and the operation

54
 The power consumption occurs from two parts: precharge switches and sense

amplifiers. The power consumption from the precharge switches is shown in Eq. 3.22.

#
P = N# W c (3.22)
T

In Eq. 3.22, we assumed VDD swing of the sense line. We also assumed the worst case

and all sense line voltage drops. Using the same dynamic comparator as in the single-

slope ADC, the power consumption from sense amplifiers is:

#
P = 18c V (3.23)
T

The total power consumption from the digital signal readout is:

#
P = {N# W c + 18c }V (3.24)
T

Periphery circuits

A. Column scanner

In this estimation, a simple shift register instead of a decoder will be used for both

the column and the row scanner. The column scanner requires #COL shifting operation per

one row time. The detailed power estimation will be shown in appendix A. The total

power consumption from the column scanner is shown in Eq. 3.25:

#
P = 12.5c + 0.5Nc V (3.25)
T

B. Row scanner

55
 The row scanner consists of a shift register and drivers as shown in figure 3.11. The

row scanner requires #ROW shifting operation per one frame. In each row, the row scanner

consists of one flip flop, three NAND gates, three level-up converters and three inverters.

The NAND gate enables the pixel control signals (TX, R, S) according to the control

signals (TXG, RG, SG). Since the pixel power supply is higher than digital power supply

(VDD), level-up converter is required to control the pixel circuit. The total power

consumption of the #ROW-bit shift register is as follows:

1
P = {7c # + 20c }V
T
(3.26)
1
+ 6c +# W c +c V
T

VDD VDDH VDDH

q(n-1) D Q TX
TXG
Q Q
CLK

R
RG
thin-ox

S
SG

Figure 3.11 Row scanner

C. Timing generator

Simple timing generator consists of two counters: one is the horizontal counter (H-

counter, assume N+1 bit counter) for column-wise operations and the other is the vertical

counter (V-counter) for row-wise operations. Additional logic gates and registers are

included for the signal generation. For simplicity, only the power consumption of two

56
counters will be estimated. The power consumption of binary ripple counter has shown in

appendix A. The power consumption of two counters is as follows:

1
P = 22c (1 + 2 + ⋯ + 2 ) V (3.27)
T

1
P = 22c (1 + 2 + ⋯ + 2 ) V (3.28)
T

,where X is log # .

From Eq.3.27 and 3.28, the total power consumption from the timing generator is

roughly estimated as shown in Eq. 3.29.

1
P ≈ 22c (1 + 2 + ⋯ + 2 ) V
T
(3.29)
1
+22c (1 + 2 + ⋯ + 2 ) V
T

I/O circuit

IO circuit includes the ESD protection circuit and drivers. Typically the strength of

driving can be controlled according to the register values. Accordingly, the power

consumption varies from the driving capability. In this estimation, the driver that has

simple two-stage inverter chain will be estimate. The most power-consuming I/O pad is

the clock signal pad, and the next is digital output pad. The power consumption from the

clock signal and digital output pad are shown in Eq. 3.30 and Eq. 3.31 respectively.

P/ ( ) = (16c +C )f V (3.30)

57
 P/ ( ) = (16c +C )(1 + 2 + ⋯

1 (33.31)
+2 ) V
T

From Eqq.3.30 and E

Eq.3.31, wee roughly esttimate the I//O power coonsumptionn as shown

i Eq. 3.32.
in

P ≈ (16c +C )f V

(33.32)
1
+(16c +C )(1 + 2 + ⋯ + 2 ) V
T

3
3.1.3 Overaall Power C
Consumptioon
Using tthe power consumptioon equationns derived iin the previious sectionn, we can

e
estimate thee overall pow
wer consum
mption. Figuure 3.12 shoows the pow
wer consumpption from

e
each power supply volttage.

Figure 3.112 Overall power

p consumption in n the signal chain accoording to th
he spatial
resolution (a) 3320 × 240 (b
b) 1280 × 9660

558
 wer consum
The pow mption from
m the I/O cirrcuits takes significantt portion in the small

n
number of ppixel arrayss (320 × 2440) due to thhe large cappacitance of I/O circuiits. As the

n
number of pixel increeases, the pparasitic cappacitance of
o metal linnes and thee column-

p
parallel circuits increaase, therefoore, relativee portion oof the I/O power connsumption

d
decreases. F
Figure 3.133 and 3.14 shows detaailed powerr consumpttion from thhe analog

p
power suppply (VDDA). In the analoog power coonsumptionn, the portioon of the AD
DC power

i
increases inn the large pixel arrays. This is becaause the CD
DS time (TCDDS) is fixed according

t the chargge transfer time in the ppixel. Moreover, the looad capacitaance of CDS
to S circuit is

f
fixed with ssampling caapacitance. T
The only facctor related with the nuumber of pixxels is the

r
row TROW) as shhown in Eqq. 3.13. In the ADC ppower, the cclock frequuency fCLK
time (T

i
increases acccording to the
t number of pixels ass shown in E
Eq.3.15.

Figure 3.113 Analog p

power consu
umption in n the signal chain according to th
he spatial
resolution (a) 3320 × 240 (b
b) 1280 × 9660

559
 Figure 3.114 Digital p
power consuumption in the signal chain accorrding to thee spatial
resolution (a) 3320 × 240 (b
b) 1280 × 9660

Figure 33.14 shows detailed pow

wer consum m the digital power suppply (VDD).
mption from

A the num
As mber of pixeel increases,, the portionn of power consumptioon from diggital signal

r
readout incrreases. Thiss is because the power consumptioon from the digital signnal readout

h three faactors that arre increasedd according to the num

has mber of pixells: number of circuits

(
(number of columns), load
l capacittance, and frrequency.

F
Figure mption according to th
3.155 Digital poower consum he power su
upply voltaage (320 ×
240)

660
 Figgure 3.16 O
Overall pow
wer consumption accorrding to thee pixel pitch
h
(a) 320 × 240
2 (b) 12880 × 960

Figure 33.15 shows the

t effect off voltage scaaling (VDD = 0.8~1.8 V
V) in the diggital power

c
consumptio n. As expeccted, 1/2 scaling induces around 11/4 power reeduction. Fiigure 3.16

s
shows the ppower consuumption acccording to the
t pixel pittch. As pixeel pitch incrreases, the

p
pixel powerr consumption increases due to thee increase off signal line parasitic caapacitance.

3
3.1.4 Low-p
power desiggn strategyy
From thhe power estimation in previous seections, we can get sevveral concluusions and

p
power reducction strateggies for low-power CM
MOS image sensors.
s

( Voltagee scaling: obbviously wee have to usse voltage scaling in booth analog aand digital
(1)

b
blocks. In the
t case of ddigital circuuits, the volltage scalingg is effectivve because tthe power

c
consumptio n is dependdent on VDDD2. Since thhe operationn frequency is below 10 MHz in

l
low resoluttion CMOS < 320 × 2440), which is applicabble to the low-power
S imager (<

w
wireless sennsor node, we
w can easilly achieve ssub-1V pow
wer supply. However,
H thhe voltage

s
scaling of thhe pixel andd the analogg power suppply directlyy affects thee SNR due tto reduced

s
signal mplement altternative pixel circuits and CDS
rangees. Thereforre, it is desiirable to im

c
circuits whiich are less vulnerable
v tto the voltagge scaling. IIn chapter 4,
4 a low-pow
wer imager

661
employing the reconfigurable pixel architecture with reduced power supply will be

discussed.

(2) Suppression of unnecessary switching: we can reduce the power consumption by

reducing unnecessary switching activities. For example, the latch circuit in the single-

slope ADC continuously changes the state before the comparator output is toggled.

Another example is the precharging circuit. Whenever the digital signal has ‘0’ value, the

sense line voltage drops and dynamic current is drawn in the precharge operation. In

chapter 4, we proposed several techniques in order to suppress redundant switching.

(3) Bandwidth reduction: The image sensor has high spatial-temporal resolutions and

the I/O power takes more than 50% of total power consumption as shown in figure 3.12.

In chapter 5, the bandwidth reduction scheme using embedded image processing

algorithm will be explained.

(4) Small pixel size: As shown in figure 3.16, the power consumption increases

according to the pixel size due to increased parasitic capacitance. In order to achieve low-

power readout and an additional functionality, many low-power image sensors use

additional in-pixel circuits, which increase the pixel size. In chapter 4 and chapter 5, a

proposed low-power readout scheme with small pixel size will be explained.

3.1.5 Power consumption figure-of-merit (FOM)

Figure 3.17 shows the power consumption according to the spatial resolutions.

The power consumption is not increased linearly according to the resolution because

of periphery circuits. However, the tendency is almost linear because the portion of

power consumption from the periphery circuit is not so high. One of way to show the

overall power consumption regardless of spatial-temporal resolution is the power FOM:

62
 Total power consu umption pW
FOM
M= [ ]
# of pixxels × Fram
me rate p pixels fram
me

Figgure 3.17 Overall

O pow
wer consump
ption accorrding to thee resolution
ns

In this tthesis, we w FOM for thhe power coomparison. Since this

will utilize tthis power F

p
power FOM
M does not reflect
r any SNR degraadation, we will use addditional FO
OM, which

r
reflects SNR
R in chapterr 4.

Resolution Frame ratte Supply Voltage

Vo Power consumption FOM
STMicroeleectronics 2.8V (analog) 75 mW
m (analog)
1280 x 1024 15 fps 4043
[3.4
4] gital)
1.8V (dig 4.5 mW
m (digital)
Cannon
n [3.5] 1944 x 1452 60 fps Not sho
own 2 mW
220 1299

Aptina [3.6] 640 x 480 30 fps Not sho

own 17 mW 1845
2.8V (analog)
Samsung
g [3.7] 4400 x 3300 30 fps 5 mW
512 1175
gital)
1.2V (dig

Taable 3.2 Pow

wer consum
mption of coonventionall CMOS im
mage sensorrs

3 Low Poower CMOS

3.2 S Image Seensors
Table 33.2 shows thhe power consumption aand the pow
wer FOM off conventionnal CMOS

i
image sensoors. As shoown in the ttable, the ppower consuumption of conventionnal CMOS

663
imagers is over 30 mW in most cases. The applications of these sensors are mostly digital

still cameras, cell phones, and automobiles. For the application of distributes sensors

which are operated by batteries or energy harvestings, the power consumption of

conventional sensors is too large to guarantee long lifetime of sensor nodes. Table 3.3

shows the power delivery from the energy harvesting. According to this table, in an

overcast day (1,000 lx illumination), available power from the 3 × 3 mm2 solar cell (150

pW/mm2·lx) is only 1.35 μW. This power delivery requires 135 pW/pixel·frame FOM

with 100 × 100 image sensor (@ 1 fps). This power budget is definitely small for

conventional CMOS image sensors. Therefore, we need alternative approaches in order

to implement a low power CMOS image sensor.

3.2.1 Limitation of Voltage Scaling

The most efficient way to scale the power consumption is the voltage scaling.

However, the voltage scaling has limitations due to the SNR degradation. Figure 3.18

shows the signal ranges in the 4-T pixel architecture. The maximum and the minimum

signal in the source follower output and in the FD can be expressed as follows:

(3.33)

(a) VOUT(MAX) = (VDD – VTHR) – VTH : reset transistor’s and source follower’s VT drop

(b) VOUT(MIN) = VDSsat: bias current transistor in saturation

(c) VFD(MAX) = VDD – VTHR: reset transistor’s VT drop

(d) VFD(MIN) = VDSsat + VTHR: source follower’s VT drop

From the equation 3.33(c) and 3.33(d), the maximum signal swing in the FD is

ΔVFD < (VDD – VTHR) – (VDSsat + VTH) (3.34)

64
 Vendor Illuminance Output Power Size Power / mm2·lx
3000 mm2
200 lx 80 uW 133.3 pW
(estimated)
Cymbet [3.8]
3000 mm2
1000 lx 350 uW 116.7 pW
(estimated)
Cymbet [3.8] Sunny 0.08 uW 0.07 mm2 5.7 nW
6000 lx (direct 3.25mm2
Clare [3.9] 200 uW 10.3 nW
sunlight) (estimated)
Enocean [3.10] 200 lx 13.5 uW 448 mm2 150.6 pW

Illumination Full moon on a

Dark overcast day Overcast day Full daylight
condition clear night
Illuminance [3.11] 1 100 1,000 10,000

Table 3.3 Solar energy harvesting example

Then, the minimum power supply can be calculated from Eq. 3.35.

(a) VDD > ΔVFD + VTHR + VDSsat + VTH

(3.35)
(b) VDD > ΔVFD + VDSsat + 2VT (VT = VTHR = VTH)

TX RST

∆VPPD
VPIN
PPD
∆VFD
VDD
VDD VTHR

FD
SIG

VTH
SEL
VOUT

VB VDSsat

Figure 3.18 Signal range in the 4-T pixel

65
 As shown in Eq. 3.35 (b), the minimum power supply voltage is “signal swing +

VDSsat + two VT drop”. This means that the voltage scaling reduces the signal swing,

accordingly reduces the SNR. Assuming VT = 0.8 V, VDSsat = 0.2 V, and signal swing

ΔVFD is 1V (for 10-b resolution, 1 mV LSB), then minimum power supply voltage is 2.8

V. Note that contemporary CMOS image sensors use 2.8 V for the pixel & analog power

supply voltage.

Another factor to limit the voltage scaling is the charge injection and the clock feed-

through in the reset transistor as shown in figure 3.19. When the reset transistor is turned

off, the voltage of FD is less than VDD – VTHR due to the charge injection and the clock

feedthrough. The VFD after the reset operation is VDD - VTHR - ΔVCH. Since the

capacitance of FD is very small (< 5 fF), the resultant error ΔVCH will be large (> 100

mV). Considering this error, Eq. 3.35 (b) can be modified as

VDD > ΔVFD + VDSsat + 2VT + ΔVCH (3.36)

RST

TX
SIG

FD
SEL

VDD-VTHR
VDD-VTHR-∆VCH
VFD
Figure 3.19 Charge injection and clock feedthrough in the reset transistor

66
 Another side effect occurs in the source follower. Since the nMOS transistor in the

pixel is not formed on the separate p-well, the body of nMOS transistor is connected with

ground. This induces a body effect and the gain of the source follower becomes less than

1 as follows:

g 1
A ( ) = = <1 (3.37)
g +g 1+η

Typically the source follower gain is around 0.8 ~ 0.9. Therefore, we can modify the

Eq. 3.36 as follows:

ΔV
V > +V + 2V + ΔV (3.38)
A ( )

Moreover, the source follower has nonlinearity due to the channel length modulation

of the bias transistor. The output voltage VOUT and the current bias IBIAS has nonlinear

relationship as follows:

(a) VOUT = VFD - VGS

(3.39)
2
(b) IBIAS = K(W/2L)(VGS-VT) (1+λVOUT)

This non-linearity distorts the image signal, which is not desirable for the post image

processing. If the power supply voltage is scaled down and enough signal swing is not

guaranteed, the bias transistor falls into the linear region especially when the illumination

is high (VOUT is low).

3.2.2 Reset Signal Boosting

Hard reset

67
 One VT drop out of two VT drop in the pixel circuit occurs in the reset transistor. In

order to remove the VT drop in the reset transistor, the signal ‘RST’ is boosted from VDD

to VDD + VT using the charge pump circuit as shown in figure 3.20 [3.12]. This operation

makes the reset transistor operates in the linear region, whereas the reset transistor

operates in the sub-VT region without reset signal boosting.

The reset operation with boosting is called hard reset and the reset operation without

boosting is called soft reset. It is known that the hard reset doubles the kTC noise [3.13].

However, the kTC noise can be suppressed in the 4-T structure. By using the reset signal

boosting, the minimum power supply voltage can be expressed as Eq.3.40.

ΔV
V > +V + V + ΔV (3.40)
A ( )

Figure 3.20 Hard reset: reset gate boosting

Capacitive coupling

Hard reset is an effective way to increase the signal swing with lower power supply

voltage. However, additional boosting circuits that consist of charge pump are required.

The charge pump circuits need large capacitors in order to boost the reset gate (RST) that

68
has large parasitic capacitance. This is not desirable in terms of the area and also the

power consumption.

Figure 3.21 Reset level boosting with capacitive coupling (1)

Another method to increase the reset level is using a capacitive coupling [3.14].

Figure 3.21 shows the pixel circuit with the capacitive boosting. Note that the selection

transistor is not used. For the row selection, the FD node is reset and the source follower

is turned on. Using the soft reset, the FD node is reset with VDD – VT. The source

follower output (SIG) increase slower than the FD node due to the large parasitic

capacitance. Through the gate-source capacitance of the source follower, the increase of

source follower output couples with the FD node, therefore, the reset level is boosted.

Figure 3.22 Reset level boosting with capacitive coupling (2)

69
 Another scheme to boost reset level is shown in figure 3.22 [3.7]. In this scheme,

additional metal line (FDB) has been placed. The parasitic capacitance between the FDB

and the FD node operates as a coupling capacitor. When the FDB line is driven high, the

FD node voltage increases.

Figure 3.23 In-pixel pulse width modulation scheme

3.2.3 In-Pixel Pulse Width Modulation (PWM)

As explained in the previous section, conventional source follower readout has

limitations of voltage scaling. Moreover, non-linearity and gain loss further limit the

voltage scaling. Instead of using the source follower, an in-pixel comparator can be used

for the readout by performing the single-slope ADC operation, which is one kind of pulse

width modulation (PWM). Figure 3.23 shows the concept of in-pixel PWM. One input of

the comparator is connected with the detection node. The other input is driven by the

ramp signal. The comparator is toggled when the ramp signal crosses the voltage of

detection node, then the counter value is latched. The advantage of the in-pixel PWM

scheme is that there is no gain loss from the body effect. Moreover, it is less vulnerable

from the current bias variation, which induces non-linearity. However, this scheme

70
generally requires larger pixel size due to the in-pixel comparator, which requires pMOS

transistors. Another disadvantage is the input voltage variation during the ADC time

because it requires 2N cycles of operation for the N-bit conversion. In high illuminations,

the detection node can be saturated during the ADC time.

In-pixel comparator without static bias current

Figure 3.24 shows one in-pixel PWM scheme [3.15]. In this work, nMOS-only

comparator that has common source amplifier topology is used and no fill factor loss

takes place. The operation is as follows: (1) Reset: the photodiode is reset by forming the

unity feedback. (2) Integration (3) Precharge: before the readout, the column line is

precharged (4) PWM: the ramp signal is applied to the source of the transistor. When the

ramp signal crosses the voltage VPD – VT, the charged column line starts to be discharged.

By regenerating the voltage variation of the column line, counter value is latched.

In this scheme, no static bias current is required except in the reset operation.

Therefore, power consumption can be reduced. However, there is a limitation in the

voltage scaling. The signal swing in the photodiode can be expressed as follows:

VT < VPD < VRAMP(RST) + VT

(3.41)
(VRAMP(RST) = ΔVPD(max))

By adjusting the voltage of VRAMP, the maximum signal swing can be controlled.

Since the comparator has the common source topology, there is a current source

connected with VDD. In order to guarantee that current sources operate in the saturation

region, the minimum power supply voltage can be calculated:

VDD > ΔVPD(max) + VT + VDsat (3.42)

71
 VDD

IPIX

PCH
RR

PCH

PCH

discharge
R S
SIG

VRAMP

(a) Pixel architecture (b) Operation

Figure 3.24 In-pixel comparator without static bias current

Therefore, it is difficult to achieve sub-1V power supply. In this scheme, ramp signal

driving is another critical factor. Since the discharge current flows into the ramp signal

line, it corrupts the ramp signal if it is not buffered. Driving the ramp signal will require

more power consumption.

In-pixel Sub-VT comparator

Figure 3.25 shows the in-pixel PWM scheme with a sub-VT comparator [3.16]. By

operating the comparator in sub-VT region, voltage can be scaled down to < 0.5 V and the

power consumption is 1.19 μW for 128 × 128 pixels. However, the comparator uses

pMOS transistors that make the pixel size larger. The comparator requires 3 additional

transistors compared with the 4-T pixel architecture. In order to guarantee the input range

of the comparator, there is an additional diode-connected transistor in the reset path. This

reduces the reset level and suppresses the SNR. Moreover, correlated double sampling is

not applied in this scheme, therefore, FPN is large.

72
 S

VRAMP
R

ENB

Figure 3.25 In-pixel PWM with sub-VT comparator

MCS

Figure 3.26 In-pixel sub-VT comparator with current-controlled threshold

73
 Figure 3.26 shows another in-pixel PWM scheme with sub-VT comparator [3.17].

Instead of applying voltage ramp signal, this scheme uses current DAC and changes the

threshold voltage of the transistor MCS for the PWM. The transistor MCS is biased by

the current DAC and operates in sub-VT region. The pixel includes only 5 transistors,

however, it includes pMOS transistors that make the pixel size larger.

Source follower
Scheme In-pixel pulse-width-modulation (PWM)
with voltage scaling
Pros Small pixel No gain loss, non-linearity

Cons Gain loss, non-linearity Large pixel, input voltage change

Reference [3.12] [3.16] [3.15]

Sub-VTH in-pixel
Readout circuit Source follower
comparator
nMOS-only comparator

Power supply 1.5 V 0.5 V 1.35 V

0.42 uW (pixel)
Power consumption 550 uW (total) 1.19 uW (total)
55.2 uW (total)
Power FOM
723.4 pW 8.6 pW 468 pW
[pW/pixel·frame]
SNR 49.63 dB 23.1 dB 52.9 dB

Pixel array 176 x 144 128 x 128 128 x 96

5x5 um2 5x5 um2 3.2 x 3.2 um2

Pixel size (0.35 um CMOS) (0.13 um CMOS) (0.35 um CMOS)

Pros Small pixel VDD scaling Small pixel

Large pixel Limited VDD

Cons Limited VDD
Low SNR Ramp generation

Table 3.4 Summary of low-power CMOS image sensors

Table 3.4 summarizes the previous low-power CMOS image sensors.

3.3 Wide Dynamic Range Image Sensors

A dynamic range (DR) is the parameter that determines the illumination range, which

can be incorporated in one image. The DR of natural scenes is more than 100 dB,

whereas the DR of conventional CMOS image sensors is only 60 ~ 70 dB. The definition

of the DR in image sensors is as follows:

74
 Saturation level
DR = 20 ∙ log [dB] (3.43)
Minimum detectable level

The minimum detectable signal is limited by noise floor. As shown in Eq. 3.43, in

order to increase the dynamic range, we can increase the saturation level or reduce the

noise floor. When we refer to “dynamic range extension”, it typically means the dynamic

range enhancement schemes, which increase the saturation level in order to prevent the

saturation over wide ranges of illumination.

Reducing the noise floor is common issue for both conventional CMOS image

sensors and WDR CMOS image sensors. In low illuminations, SNR is degraded due to

low signal level. If CMOS image sensors have high sensitivity, i.e., generate higher

signal level with less incident photons, SNR increases due to the increase of signal level.

However, increasing only the sensitivity reduces the dynamic range because the signal

will be saturated from smaller photons. Therefore, it is important to optimize both

schemes according to the illumination. The sensitivity can be enhanced by several ways.

One is the process optimization such as photodiodes, color filter arrays and microlens.

The other is by the readout circuits. The CDS circuit and the programmable amplifier

(PGA) is one example.

This section will explain about the sensitivity enhancement schemes. The integrated

dynamic range enhancement schemes will be illustrated in the next section.

3.3.1 Sensitivity Enhancement

As explained in chapter 2, the sensitivity is defined as the ratio of the output signal

[V] to the incident illumination level [lx] in a second. The unit of sensitivity is [V/lx·s].

Sensitivity can be increased by process optimization, microlens optimization, and so on.

75
One of way to increase the sensitivity is applying high gain in order to amplify the signal

and suppress the noise. The programmable amplification in the CDS circuit is a good

choice to increase the sensitivity. However, the PGA requires significant power

consumption because it is configured with capacitive feedback. The capacitance is large

enough to guarantee the matching between columns (typically > 1 pF). Moreover, column

parallel PGAs only suppress the noise of ADCs, not the readout circuit noise from in-

pixel source followers. Therefore, there are previous works that employ in-pixel

amplification instead of column parallel amplification.

SIG1
SIG2

Figure 3.27 In-pixel nMOS common source amplifier with p-n photodiode

One previous work used the nMOS common source amplifier with p-n photodiode

[3.18] as shown in figure 3.27. As mentioned, in the 3-T pixel structure, kTC reset noise

is not cancelled. In order to suppress the kTC noise in the 3-T structure, this work used

unity feedback using in-pixel commons amplifiers. In simple one pole amplifiers, the

gain increment by AV induces the bandwidth reduction by AV. Therefore, the noise power

76
from kTC noise is decreased by AV. Assuming the amplifier gain is infinite, the gain of

the amplifier can be defined as follows:

C
A = (3.44)
C

Figure 3.28 In-pixel pMOS common source amplifier with pinned photodiode

In Eq.3.43, CS is the capacitance of the detection node and CI is the gate-drain

capacitance of the transistor M2. Since CS has the capacitance over 10 fF, and CI is less

than 0.5 fF, high gain > 20 is achievable. However, the gain of common source amplifiers

cannot be an infinite and it is typically around 10, which brings out the reduction of the

gain compared with Eq. 3.44.

Another previous work used pMOS common source amplifiers with pinned

photodiode [3.19]. This work uses similar topology as a conventional 4-T pixel structure

except that they used pMOS common source amplifiers as shown in figure 3.28. In

pinned photodiodes, the reset level should be high enough in the floating diffusion in

order to guarantee complete charge transfer. In order to increase the reset level that is set

77
by the unity feedback, a pMOS amplifier is chosen. In the selection transistor, the voltage

bias VCASC is applied for the cascade configuration. It suppresses the Miller effect and

higher gain is achievable. The input referred noise is suppressed by the gain of amplifier

(gmRi) as follows:

kT 1
v = γ ∙ (3.45)
C g R

,where CS is the sampling capacitor in the column (not shown in the figure) This work

achieved 0.86 e- readout noise whereas the conventional CMOS imagers have > 2 e-

readout noise. However, pMOS transistors increase the pixel size (11 × 11 μm2 in this

work).

In summary, it is desirable to implement the in-pixel amplification for two reasons: (1)

Avoid using power-consuming column-parallel PGAs while increasing the sensitivity (2)

The in-pixel amplification suppresses the noise from pixel readout circuits.

3.3.2 Dynamic Range Enhancement Schemes

The integrated DR enhancement scheme can be categorized as two: non-linear

scheme and linear scheme.

The basic principle of nonlinear scheme is to compress the pixel output response and

cover wider ranges of illumination before the pixel output is saturated. Figure 3.29 (a)

shows basic concept of nonlinear scheme.

The linear scheme can be categorized as two: sensitivity adjustment and multiple

exposure schemes. In the sensitivity adjustment, the sensitivity is intentionally decreased

by reducing the conversion gain. As explained before, the conversion gain is q/CD where

CD is the capacitance of the detection node. By connecting additional capacitance COV

78
with the CD, the conversion gain will be reduced to q/(CD + COV). The additional

capacitance lowers the sensitivity. However, it increases the illumination range that is

able to be sensed. Figure 3.29 (b) shows the basic concept of sensitivity adjustment

scheme.

(a) Compressive (b) Linear: sensitivity adjustment

VSIG

VA2

VA1
VB2
VB1
t
TINT(SHORT) TINT(LONG)
(c) Linear: Multiple exposure

Figure 3.29 Dynamic range enhancement schemes

In the multiple exposure scheme, multiple signals from multiple integration time

(from short to long) are synthesized. Figure 3.29 (c) shows the example of dual exposure.

In the figure, two cases of pixel output response are shown: one is in high illumination

(output is saturated) and the other is in low illumination. By applying the dual exposure,

VA1 and VA2 will be synthesized. Although VA2 is saturated signal in high illumination,

we can get the information from VA1. Likewise, VB1 and VB2 will be synthesized in low

illumination case. In this case, VB1 can be low and therefore has low SNR. However, VB2

has enough signal level from long integration time.

79
 In the next three subsections, each scheme will be explained with previous works.

3.3.2.1 Nonlinear Scheme: Logarithmic Pixel

For a nonlinear response, a logarithmic response can be used because it is easy to

implement by the transistor which operates in the sub-VT region. The logarithmic pixel is

shown in figure 3.30. Since the current is exponentially proportional to the gate-source

voltage VGS, VGS is logarithmically dependent on the current. However, since the

transistor operates in the sub-VT region, it suffers high FPN due to the variation. In order

to achieve both high SNR from the dark to moderate illumination and high DR in high

illumination, linear-logarithmic pixel can be implemented

Figure 3.30 Logarithmic pixel

Figure 3.31 shows the linear-logarithmic pixel [3.20]. If the detection node voltage

VPD is smaller than VLOG-2VT, VPD can be derived as follows:

(a) i ( )/
=i =i e =i e

( )/
=i =i e (3.46)

V +V
(b) V =
2

80
 kT i
(c) V =V −2 ln
q i

By series connection of two diode-connected transistors M2 and M3, the logarithmic

response term in Eq. 3.46 (c) has doubled gain. When VPD is larger than VLOG-2VT, the

transistor M2 and M3 is turned off and the pixel operates in the linear region as

conventional 3-T pixel. In this structure, by adjusting VLOG, the trip point between the

linear region and the logarithmic region can be controlled.

VLOG

VRST

VX SH
R

VPD
S

Figure 3.31 Linear-logarithmic pixel

The advantage of the logarithmic pixel is that it does not have much fill factor loss

and the operation simple, i.e., no additional timing control is required. However, the

logarithmic response from the sub VT region has higher FPN that is cannot be cancelled

from the CDS operation. The previous work shows 112 dB DR, 9.4 μm pixel pitch, 30 %

FF using 0.35 μm CMOS process. It shows 0.83 % FPN in the linear region and 1.37 %

FPN in the logarithmic region.

81
3.3.2.2 Linear Scheme: Sensitivity Adjustment
The sensitivity is dependent on the conversion gain q/CFD. With large CFD, we can

transfer and store more charge in the FD node. However, the sensitivity is decreased due

to larger capacitance. As shown in figure 3.32, the sensitivity can be controlled using an

additional capacitor CS and a selection switch S2. In this scheme, the pixel keeps high

sensitivity in low illumination (S2 off). The pixel decreases the sensitivity in high

illumination in order to prevent the saturation from high illumination (S2 on). However,

the dynamic range extension is limited according to the capacitance of CS. Larger CS

induces lower sensitivity due to reduced fill factor in low illumination.

CS
S2 R

TX

FD
S1

Figure 3.32 Sensitivity adjustment using in-pixel capacitor

Another scheme using the sensitivity adjustment is shown in figure 3.33. When

illumination is high and the accumulation of photogenerated electrons exceeds the well

capacity, a photodiode is saturated and electrons are spilled over. If we collect these

spilled electrons in the additional capacitor with larger capacitance, we can read out

signals with reduced sensitivity. This lateral overflow scheme [3.21] is shown in figure

82
3.33. In figure 3.33 (a), the pixel circuit is shown. An additional overflow capacitor COV

and the floating diffusion (FD) is connected through the transistor S2.

Figure 3.33 Lateral overflow scheme (a) Pixel circuit (b) Operation procedure

The operation procedure is described in figure 3.33 (b) with electron potential

diagrams. The operation procedure is as follows: (1) Reset FD and COV (S: on) (2)

Integration, if charges are spilled over, they are collected both in the FD and COV. (3)

Reset FD: S is turned off, and FD is reset. (4) 1st signal readout: transfer charges into FD

83
and then readout signal (5) 2nd signal readout: S is turned on again and charge stored in

COV is read out together with charges in the FD. Two signals from (4) and (5) are

synthesized in external circuits.

The advantage of this scheme is that it has linear response. Linear response is

preferable in a post image processing. It also has better SNR than nonlinear methods.

However, due to the additional capacitor and transistor, fill factor will be decreased. In

this work, the poly capacitor is implemented for COV and it is placed below the metal line.

Therefore, fill factor loss is decreased. The previous work achieved 93 dB DR, 5.6 μm

pixel pitch, 45 % fill factor using 0.18 μm CIS process. The limitation is that shared pixel

structure cannot be applicable with this scheme. Therefore, it is difficult to scale down

the pixel size.

3.3.2.3 Linear Scheme: Multiple Exposures

Another linear scheme for the dynamic range enhancement is a multiple exposure

scheme. Image signals with multiple integration time are synthesized. Figure 3.34 shows

the architecture using triple exposure [3.22]. In the case of triple exposure, three image

signals with three different integration times from short to long have to be synthesized. In

order to synthesize three image signals, two image signals should be temporarily stored in

line memories before final image signals are read out and are synthesized together.

Therefore, multiple exposure scheme requires additional memories.

Assuming three integration times are defined as follows:

TROW: one row access time, T1 = L·TROW, T2 = M·TROW, T3=N·TROW (L > M > N)

The image signal from T1 is temporarily stored during (M+N)·TROW. After M·TROW,

the image signal from T2 is temporarily stored during N·TROW. After N·TROW, the final

84
image signal from T3 is read out and three image signals are synthesized into one image

signal with wide dynamic range. Therefore, total M+2N rows of line memories are

required. This takes significant portion of total die area. In the previous work, memory

occupies 12 % of total area.

Moreover, this scheme requires multiple readouts, which require additional power

consumption.

Figure 3.34 Triple exposure scheme

3.3.2.4 Summary of Integrated Dynamic Range Enhancement Schemes

Table 3.5 shows the summary of integrated dynamic range enhancement schemes. As

mentioned in chapter 1, the image sensor has to generate a high dynamic range image

(around 100 dB) at low-power consumption. Non-linear scheme using the logarithmic

pixel has no fill-factor loss and simple operations. However, it has low SNR and

temperature variation, which does not satisfy the requirement of reliability. Therefore, the

linear scheme is better choice in applications for distributes sensors. The sensitivity

adjustment scheme is one good solution. However, it is not applicable to the shared pixel

85
architecture, which is critical to achieve small form factor and low cost The multiple-

exposure scheme also has larger form factor due to the additional line memory. Moreover,

it has higher power consumption than other two schemes.

In the proposed adaptive image sensor, we proposed the low-power dual exposure

scheme that does not require additional circuits or memories. It will be discussed in

chapter 4.

Dynamic Range Linear

Non-linear
Enhancement Sensitivity adjustment Multiple exposure
Multiple signals from
Store overflown charge
Compress the pixel multiple integration time
Principle output response
to the large capacitor
(short to long) are
(reduce the sensitivity)
synthesized
Linear response
Linear response
No fill-factor loss Better SNR than non-
Pros Simple operation
Better SNR than non-
linear
linear
Small pixel size
Low SNR (logarithmic Larger pixel size Large power
Cons response from Sub-VT (additional in-pixel consumption,
transistor) capacitor) Line memory

Table 3.5 Summary of integrated dynamic range enhancement schemes

3.4 CMOS Image Sensors with Integrated Image Signal Processing for Bandwidth
Reduction
If an image sensor generates M × N resolution 8-bit images with 30 frames per

second (fps), total bandwidth is 240·M·N, which requires huge bandwidth and power

consumption for an image signal transmission. However, the imaging with constant

frame rate and the maximum resolution has redundancy in terms of both the spatial and

temporal domain.

An image compression is one way to reduce the bandwidth [3.23, 3.24]. Many image

compression techniques such as differential pulse code modulation (DPCM) or JPEG

86
format for still images or MPEG for videos are applied to the commercial digital imaging

systems.

Even though the image compression offers the bandwidth savings, it still requires

large bandwidth if imaging systems operate with fixed resolution and fixed frame rate.

Moreover, typically these image compression algorithms require complex processing.

Therefore, it is not adequate for distributed imaging systems especially for applications of

surveillance systems that continuously monitor specific target objects or specific events.

Instead of generating images with maximum frame rate and resolution, several

CMOS imagers generate only low-bandwidth features from motion events [3.25-3.29] or

low resolution images that contains only target objects in order to save bandwidth [3.31,

3.32]. Another scheme generates variable resolution images for the bandwidth reduction

[3.33-3.35]. In the next three subsections, previous works with integrated image

processing for the bandwidth saving will be introduced.

3.4.1 Feature extraction

The motivation of a feature extraction is suppressing the bandwidth by generating low

bandwidth features from specific events. Instead of transmitting full resolution images,

the sensor delivers features only when specific events have been monitored.

One of simple feature is the temporal change from the motion sensing. The basic

operation of the motion sensing in CMOS image sensors is frame difference that has

simple operation. The difference between current frame signals and stored previous frame

signals are subtracted. For the motion detection, storage elements are required to store the

signals of previous frame. These storage elements can be placed in pixels as depicted in

figure 3.35 [3.29]. There are two sample and hold circuits and two signal readout paths

87
for direct frame difference output. However, fill factor is decreased and pixel size

becomes large due to in-pixel capacitors and additional readout paths. Pixel size is

32.2μm × 32.2μm in 0.5μm technology, and fill factor is 33%.

Figure 3.35 Pixel with direct frame difference output

ICOMP

R F

S
VO VC

VCOMP

ISF

Figure 3.36 In-pixel comparator for the frame difference

88
 Another previous work for the motion event generation contains two capacitors and a

comparator in pixel as shown in figure 3.36 [3.25]. It generates normal image through the

source follower as conventional 3T-pixel sensors. In addition, the comparator for the

motion detection is integrated and generates motion images (frame difference). The

analog output of frame difference is quantized into 2-bit digital signal in order to encode

three events: no-motion, darker and brighter changes. This pixel contains 6 transistors

and two capacitors. Fill factor is 17% and pixel size is 25.2 × 25.2 μm2 with 0.5 μm

technology.

Another example of simple feature extraction is the address event representation

(AER). Instead of frame-based imaging, the sensor generates the output asynchronously

based on the events [3.30]. The pixel responds to relative changes of light intensity and

asynchronously generates 1-b signals. The sensor outputs the address of pixels that

generated 1-b signals, i.e., have enough contrast change. Since the pixel has to monitor

the temporal contrast change, it has additional capacitors and circuits, which increase the

pixel size (8.1% fill factor, 40×40 μm2 pixel size using 0.35μm technology).

By transmitting the output only when the temporal change is detected, data

redundancy can be suppressed. However, the data rate of AER increases according to the

number of pixels. Moreover, it responds to environmental conditions such as changes of

illuminations or movements of background in addition to movements of actual target

objects.

3.4.2 Region-of-Interest (ROI) readout

The region-of-Interest (ROI) image means the partial image containing an interested

object out of entire image. The ROI readout scheme also suppresses the redundancy

89
because the background image is redundant. We don’t need to generate entire image

including redundant backgrounds. By generating only some parts of an entire image,

required bandwidth and power consumption are saved. Accordingly, fast and simple post

image processing is applicable.

The ROI can be simply determined by external logics [3.32]. A pixel array is divided

with fixed-sized block, and several blocks for the ROI are selected by external logics.

Basic block size is 32 × 32 pixels. The ROI size can be variable by the external control.

However, this work does not include integrated ROI decision that detects target objects.

This work just selects a group of blocks and outputs ROI images.

SIG

Figure 3.37 Pixel structure with two additional photodiodes

More advanced ROI readout capability has been published [3.31]. In this work, the

sensor generates the ROI image containing brighter objects compared with the

background image. This approach is targeted to the application for the car plate (that is

bright from the headlights) monitoring at night. The pixel circuit is shown in figure 3.37.

90
In the pixel, two additional photodiodes are integrated. Each additional photodiode

detects brightness using column / row parallel charge amplifiers. Each output of charge

amplifiers generates profiles in X-Y directions. Using these profiles, ROI is decided by

external circuit. The size of additional photodiode is small and fill factor is not severely

reduced. However, this scheme is focused on the limited application and cannot detect

dark object.

3.4.3 Variable-resolution image Readout

Variable-resolution readout means that the image sensor generates images with

variable-resolutions from the maximum to the reduced resolutions. In order to reduce the

resolution, a subsampling is a simple method to be implemented. However, the

subsampling induces the aliasing effect. Therefore, a low resolution image is generally

obtained by averaging or summing the image signals rather than skipping.

Low resolution images can be generated in various ways. By integrating passive

capacitive networks, averaged value can be calculated [3.33, 3.34]. However, additional

capacitors take large area.

Another method is charge-binning technique. The charge binning is originated from

CCD image sensors, and is applied to CMOS image sensors by using the shared pixel

architecture [3.36, 3.37]. In this scheme, charges integrated in several photodiodes are

transferred into a floating diffusion with small capacitance, and sensitivity is enhanced.

The image signal can also be merged in digital domain [3.35]. After the analog-to-

digital conversion, signals are averaged using digital circuits. However, it has worse SNR

compared with charge domain merging because the readout noise and quantization noise

are added..

91
3 Motion
3.5 n-adaptive m
multi-resollution imagge sensor
Constannt frame-ratee images with
w maximuum spatial rresolution eentail high bbandwidth

a also large power coonsumptionn [3.38, 3.399]. This mayy limit its appplication foor portable
and

d
devices or wireless
w sennsor networrks in whichh both banddwidth and power conssumptions

a limited. Reducing the spatial resolution iis one way to reduce tthe amountt of image
are

d
data. Howevver, details in the imagee may be lost from the reduced ressolution.

One effficient apprroach to reduce the bandwidth

b is to providde multiple temporal

r
resolutions in one focaal-plane: i.ee., to providde a high frrame rate oonly in the region-of-

i
interest (RO
OI) where thhe moving objects are located andd allow a loow frame raate for the

r
rest of stattionery bacckgrounds. The imagees of the sstationary oobjects are typically

r
redundant aat a high frrame rate annd it only ddegrades thhe SNR whhile wasting the large

b
bandwidth.

Figure 3.38 Imaages from m

motion-adaptive multii-resolution
n image sen
nsor

Moreovver, the spattial resolutioon in the ROI

R can be further opttimized to rreduce the

a
amount of ddata. Combiining the muultiple tempporal and sppatial resoluutions, we caan acquire

t image w
the with detailss for stationnery objectts and withh negligible motion-bluur for the

m
moving objeects at low-ppower consumption and low-bandw
width.

992
 In orderr to suppresss the redunddancy of banndwidth, a m
motion-adapptive multi--resolution

C
CMOS imagger that is aable to proviide an adapttive spatial-ttemporal m
multi-resolutiion for the

s
specific reggion of interrest which is autonomoously determ
mined from motion deteection has

b
been propossed [3.40]. F
Figure 3.388 shows outpput images from the m
multi-resoluttion image

s
sensor.

Figure 33.39 Pixel and column architecture of the m

multi-resoluttion image sensor

The sennsor can sim

multaneouslly generate spatial-tem
mporal multti-resolutionn readouts

f
from the tw
wo channels: one at low frame rate with the maaximum spaatial resolutiion for the

s
stationery bbackgroundss delivering the details of scenery; and the othher at high frame
f rate

993
w the redduced spatiial resolutioon for the m
with moving objeects in the ROI suppreessing the

m
motion blurr. Since the high framee readout is performed only in the ROI, banddwidth and

p
power conssumption caan be signifficantly redduced. In thhe ROI, thee spatial ressolution is

r
reduced by four times. The chargges from foour shared ppixels are ssummed in a floating

d
diffusion noode in orderr to enhance the SNR duuring the shhort integratiion time.

Figure 3.40 Multi--resolution readout: (aa) Block diagram (b) T

Timing diaagram

The RO
OI is determiined in real--time by thee integrated on-chip mootion detection circuit.

T
This circuitt has been implementted in a sm
mall area by
b sequentiial inter-pixxel switch

994
operations and does not require any additional in-pixel memory. Figure 3.39 shows the

pixel and column architecture of the multi-resolution sensor. The pixel has 4-shared pixel

architecture. The floating diffusions (FD) of two 4-shared groups are connected with

inter-pixel switch (IS).

Instead of using additional in-pixel capacitor or frame memory, a floating diffusion

node in the pixel has been used as an analog memory. The signal from the lower FD is

transferred to upper FD through IS and the half of previous frame signal is stored in

upper FD (because the capacitance of two FDs are same). After the half of integration

time, the half of current frame signal is generated in the lower FD. The frame difference

can be generated by comparing two FDs.

Figure 3.40 shows the block diagram of multi-resolution sensor. In the multi-

resolution readout, the row decoder and the column selector provide independent pixel

controls for the ROI as well as the background region. The dual channel CDS/ADC

blocks will generate the normal and ROI image signals respectively. The motion

detection is performed only within the ROI. From continuous motion detection, the ROI

will be redefined and updated from the FPGA. The ROI signals are accessed at a high

frame rate during the blanking time in the one row time that is assigned for normal

readout.

3.6 Summary
In this chapter, we estimated and analyzed power consumption in CMOS image

sensor. From this estimation, we can draw power reduction strategies: voltage scaling

with modified architecture, suppression of redundant switching, reduction of output

bandwidth, and small pixel size.

95
 This chapter introduced three categories of previous works about low-power CMOS

image sensors, integrated dynamic range enhancement / high-sensitivity readout, and the

integrated image processing for the bandwidth savings. As mentioned in chapter 1, for

applications of distributed sensors, four requirements (low-power consumption, reliability,

spatial and temporal bandwidth) should be satisfied together. However, these four

requirements are generally in trade-off and this trade-off is clearly shown from the

previous works.

For the low-power consumption, a voltage scaling is limited due to the SNR

degradation. The in-pixel PWM is an alternative method to achieve low-power

consumption. However, it also has SNR degradation and the limitation of voltage scaling.

Moreover, the input voltage of in-pixel comparator is corrupted from the illumination

change due to the long conversion time.

In order to achieve reliable sensing in wide range of illumination, we have to

increase the dynamic range. Two methods for wide dynamic range are integrated

dynamic range enhancement scheme and high-sensitivity readout. Three dynamic range

enhancement schemes are introduced in this chapter. Non-linear scheme using the

logarithmic pixel offers no fill-factor loss and simple operation. However, it has low SNR

and temperature variations, which do not satisfy the requirement of reliability. The

sensitivity adjustment scheme is not applicable to the shared pixel architecture and

increases the form factor. The multiple-exposure scheme also makes large form factor

due to the additional line memory. Moreover, it requires higher power consumption than

other two schemes.

96
 For the spatial and temporal bandwidth savings, integrated lightweight image

processing is essential. The motion sensing and the ROI image readout suppress both the

spatial and temporal redundancy because they generate low bandwidth output only when

events or interested objects are detected. The variable-resolution readout also provides a

spatial bandwidth saving by merging neighboring pixels. However, these schemes have

limitations for applications of wireless imaging systems due to large pixel size and high

sensitivity for environmental conditions (motion from the background, illumination).

Since it is difficult to satisfy four requirements at the same time, it is desirable to

implement an image sensor adaptable, i.e., the image sensor changes its operation

adaptively according to environmental conditions and applications. It is also important to

implement with small pixel size and small form factor in order to reduce both power

consumption and cost. In this chapter, a motion-adaptive imager that is able to provide an

adaptive spatial-temporal multi-resolution for the specific region of interest that is

autonomously determined from the motion detection has been introduced. The next two

chapters will explain about the three adaptive imaging methods: energy, illumination, and

objective adaptive imaging.

97
Chapter 3 References

[3.1] J. M. Rabaey, A. Chandrakasan, B. Nikolic, Digital Integrated Circuits, Prentice-

Hall
[3.2] S. Nukhopadhyay, H. M-. Meimand, C. Neau, K. Roy, “Leakage in Nanometer
Scale CMOS Circuits,” Symp. on VLSI Technology, Systems, and Applications,
pp. 510-618, 2003
[3.3] B. Zhai, L. Nazhandali, J. Olson, A. Reeves, M. Minuth, R. Helfand, S. Pant,
D.Blaauw, T. Austin, "A 2.60pJ/Inst Subthreshold Sensor Processor for Optimal
Energy Efficiency," Symp. on VLSI Circuits, pp. 154-155, 2006.
[3.4] K. Findlater et al., “SXGA pinned photodiode CMOS image sensor in 0.35 μm
technology,” Digest of Technical Papers. ISSCC., 2003, vol. 1, pp. 218-489.
[3.5] H. Takahashi et al., “A 1/2.7 inch Low-Noise CMOS Image Sensor for Full HD
Camcorders,” in 2007 IEEE International Solid-State Circuits Conference. Digest
of Technical Papers, 2007, pp. 510-618.
[3.6] R. Johansson et al., “A 1/13-inch 30fps VGA SoC CMOS image sensor with
shared reset and transfer-gate pixel control,” in 2011 IEEE International Solid-
State Circuits Conference, 2011, pp. 414-415.
[3.7] S. Lee et al., “A 1/2.33-inch 14.6M 1.4μm-pixel backside-illuminated CMOS
image sensor with floating diffusion boosting,” in 2011 IEEE International Solid-
State Circuits Conference, 2011, pp. 416-418.
[3.8] Cymbet Corp., Dec 2012, <www.cymbet.com>
[3.9] Clare, http://www.clare.com/Products/SolarCell.htm
[3.10] Enocean, Dec 2012, <http://www.enocean.com/en/solar-energy-harvesting/>
[3.11] Wikepedia, Dec 2012, <http://en.wikipedia.org/wiki/Lux>
[3.12] K. -B. Cho, I. Krymski and E. R. Fossum, “A 1.5-V 550-μW 176 × 144
autonomous CMOS active pixel image sensor,” IEEE Transactions on Electron
Devices, vol. 50, no. 1, pp. 96-105, Jan. 2003.
[3.13] B. Pain, G. Yang, M. Ortiz, C. Wrigley, B. Hancock and T. Cunningham,
“Analysis and enhancement of low-light-level performance of photodiode-type
CMOS active pixel imagers operated with sub-threshold reset,” Int. Image Sensor
Workshop, 1999

98
[3.14] S. Lee, K. Lee, J. Park, H. Han, Y. Park, T. Jung, Y. Jang, “A 1/2.33-inch 14.6M
1.4μm-pixel backside-illuminated CMOS image sensor with floating diffusion
boosting,” ISSCC Dig Tech. Papers, pp. 416-418, Feb. 2011.
[3.15] K. Kagawa, S. Shishido, M. Nunoshita, and J. Ohta, “A 3.6pW/frame·pixel
1.35V PWM CMOS Imager with Dynamic Pixel Readout and no Static Bias
Current,” in 2008 IEEE International Solid-State Circuits Conference - Digest of
Technical Papers, 2008, pp. 54-595.
[3.16] S. Hanson, Z. Foo, D. Blaauw, and D. Sylvester, “A 0.5 V Sub-Microwatt CMOS
Image Sensor With Pulse-Width Modulation Read-Out,” IEEE Journal of Solid-
State Circuits, vol. 45, no. 4, pp. 759-767, Apr. 2010.
[3.17] M. –T. Chung, C. –C. Hsieh, “A 0.5V 4.95μW 11.8fps PWM CMOS imager with
82dB dynamic range and 0.055% fixed-pattern noise,” ISSCC Dig Tech. Papers,
pp. 114-116, Feb. 2012.
[3.18] J. Yang, K. G. Fife, L. Brooks, C. G. Sodini, A. Betts, and P. Mudunuru, “A
3MPixel Low-Noise Flexible Architecture CMOS Image Sensor,” in 2006 IEEE
International Solid State Circuits Conference - Digest of Technical Papers, 2006,
pp. 2004-2013.
[3.19] C. Lotto, P. Seitz, and T. Baechler, “A sub-electron readout noise CMOS image
sensor with pixel-level open-loop voltage amplification,” in 2011 IEEE
International Solid-State Circuits Conference, 2011, pp. 402-404.
[3.20] M. Vatteroni, P. Valdastri, A. Sartori, A. Menciassi, and P. Dario, “Linear–
Logarithmic CMOS Pixel With Tunable Dynamic Range,” IEEE Transactions on
Electron Devices, vol. 58, no. 4, pp. 1108-1115, Apr. 2011.
[3.21] N. Akahane, S. Sugawa, S. Adachi, K. Mori, T. Ishiuchi, and K. Mizobuchi, “A
Sensitivity and Linearity Improvement of a 100-dB Dynamic Range CMOS
Image Sensor Using a Lateral Overflow Integration Capacitor,” IEEE Journal of
Solid-State Circuits, vol. 41, no. 4, pp. 851-858, Apr. 2006.
[3.22] J. Solhusvik et al., “A 1280×960 3.75 μm pixel CMOS imager with Triple
Exposure HDR,” in International Image Sensor Workshop, 2009.
[3.23] S. Kawahito et al., “A CMOS image sensor with analog two-dimensional DCT-
based compression circuits for one-chip cameras,” IEEE Journal of Solid-State

99
 Circuits, vol. 32, no. 12, pp. 2030-2041, 1997.
[3.24] W. D. León-Salas, S. Balkir, K. Sayood, N. Schemm, and M. Hoffman, “A
CMOS imager with focal plane compression using predictive coding,” IEEE
Journal of Solid-State Circuits, vol. 42, no. 11, pp. 2555–2572, 2007.\
[3.25] U. Mallik, M. Clapp, G. Cauwenberghs, and R. Etienne-Cummings, “Temporal
change threshold detection imager,” in ISSCC. 2005 IEEE International Digest of
Technical Papers. Solid-State Circuits Conference, 2005., 2005, pp. 362-364.
[3.26] V. Gruev and R. Etienne-Cummings, “A Pipelined Temporal Difference Imager,”
IEEE Journal of Solid-State Circuits, vol. 39, no. 3, pp. 538-543, Mar. 2004.
[3.27] S. Mizuno, K. Fujita, H. Yamamoto, N. Mukozaka, and H. Toyoda, “A 256 × 256
compact CMOS image sensor with on-chip motion detection function,” IEEE
Journal of Solid-State Circuits, vol. 38, no. 6, pp. 1072-1075, Jun. 2003.
[3.28] Y. Muramatsu, S. Kurosawa, M. Furumiya, H. Ohkubo, and Y. Nakashiba, “A
signal-processing CMOS image sensor using a simple analog operation,” IEEE
Journal of Solid-State Circuits, vol. 38, no. 1, pp. 101-106, Jan. 2003.
[3.29] L.-gee Chen, “A single-chip CMOS APS camera with direct frame difference
output,” IEEE Journal of Solid-State Circuits, vol. 34, no. 10, pp. 1415-1418,
1999.
[3.30] P. Lichtsteiner, C. Posch, T. Delbruck, "A 128× 128 120 dB 15 μs Latency
Asynchronous Temporal Contrast Vision Sensor," Solid-State Circuits, IEEE
Journal of , vol.43, no.2, pp.566-576, Feb. 2008
[3.31] Y. Sugiyama et al., “A high-speed CMOS image sensor with profile data
acquiring function,” IEEE Journal of Solid-State Circuits, vol. 40, no. 12, pp.
2816-2823, Dec. 2005.
[3.32] O. Schrey, J. Huppertz, G. Filimonovic, A. Bubmann, W. Brockherde, and B. J.
Hosticka, “A 1K×1K High Dynamic Range CMOS Image Sensor With On-Chip
Programmable Region-of-Interest Readout,” IEEE Journal of Solid-State Circuits,
vol. 37 no. 7 pp. 911-915, Jul. 2002
[3.33] Z. Zhou, B. Pain and E. Fossum, “A CMOS Imager with On-Chip Variable
Resolution for Light-Adaptive Imaging,” ISSCC Dig Tech. Papers, pp. 174-175,
Feb., 1998.

100
[3.34] S. E. Kemeny, R. Panicacci, B. Pain, L. Matthies, and E. R. Fossum,
“Multiresolution Image Sensor,” IEEE Transactions on Circuits and Systems for
Video Technology, vol. 7, no. 4, pp. 575-583, Aug. 1997
[3.35] E. Artyomov et al., “Morton (Z) Scan Based Real-Time Variable Resolution
CMOS Image Sensor,” IEEE T. CAS Video Tech., vol. 15, pp. 947-952, Jul.,
2005.
[3.36] K. Young Chan et al., “1/2-inch 7.2MPixel CMOS Image Sensor with 2.25/spl
mu/m Pixels Using 4-Shared Pixel Structure for Pixel-Level Summation,” in
2006 IEEE International Solid State Circuits Conference - Digest of Technical
Papers, 2006, pp. 1994-2003.
[3.37] S. Yoshihara et al., “A 1/1.8-inch 6.4 MPixel 60 frames/s CMOS Image Sensor
With Seamless Mode Change,” IEEE Journal of Solid-State Circuits, vol. 41, no.
12, pp. 2998-3006, Dec. 2006.
[3.38] S. Kleinfelder and A. El Gamal, “A 10000 frames/s CMOS digital pixel sensor,”
IEEE Journal of Solid-State Circuits, vol. 36, no. 12, pp. 2049-2059, 2001
[3.39] A I. Krymski, N. E. Bock, D. Van Blerkom, and E. R. Fossum, “A high-speed,
240-frames/s, 4.1-mpixel CMOS sensor,” IEEE Transactions on Electron Devices, vol.
50, no. 1, pp. 130-135, Jan. 2003.
[3.40] J. Choi, S. –W. Han, S. –J. Kim, S. –I. Chang, E. Yoon; , "A Spatial-Temporal
Multiresolution CMOS Image Sensor With Adaptive Frame Rates for Tracking the
Moving Objects in Region-of-Interest and Suppressing Motion Blur," Solid-State
Circuits, IEEE Journal of , vol.42, no.12, pp.2978-2989, Dec. 2007

101
 CHAPTER 4 ENERGY/ILLUMINATION-ADAPTIVE IMAGE SENSOR WITH
RECONFIGURABLE MODES OF OPERATION

For outdoor surveillance or biomedical applications, sensitivity and dynamic range

are important to deliver reliable information over widely changing illumination. However,

constant monitoring with maximum awareness requires large power consumption and is

not feasible for energy-limited applications such as battery-operated and/or energy-

scavenging wireless sensor nodes. One of ways to reduce the power is voltage scaling

[4.1, 4.2]. However, it significantly reduces the SNR and results in poor image quality

[4.2]. The signal can be easily corrupted from the noise in dark conditions or be saturated

in bright conditions. Dual power supply has been applied but still the conventional source

follower readout scheme limited the signal range [4.3]. Most imagers with high-

sensitivity and wide dynamic range reported [4.4, 4.5] consume large power > 50 mW,

unsuitable for wireless imager node applications. Therefore, it is imperative to implement

an imager adaptable to an environment, i.e., the sensor keeps monitoring at an extremely

low power, turns into the high sensitivity or wide dynamic range imaging when

illumination varies, or when detailed image transmission is requested from the network.

The sensor changes its operation back to monitoring mode when the energy harvesting is

poor or it is needed to save more charges in the battery. In this chapter, we report an

adaptive CMOS image sensor which employs four different imaging modes: monitoring,

normal, high-sensitivity and wide-dynamic-range (WDR) modes. This adaptable feature

102
e
enables thee reliable monitoring
m w
while signiificantly ennhancing thee battery liifetime of

w
wireless imaage sensor nnodes.

4 Adaptivve imaging operation and overalll architectu

4.1 ure
Figure 4.1
4 shows tthe operatioon of the aadaptive imaaging and ffigure 4.2 shows
s the

b
block diagraam of the pproposed sennsor chip. IIn the monittoring modee, the sensoor operates

a 0.8V suppply. In-pixxel comparaators and 8--b successivve approxim

at mation (SA
AR) ADCs

e
enable pow
wer saving. For an in-ppixel ADC,, a SAR ADC is chossen becausee its short

c
conversion time (8 cyccles) saves power.
p Otheer three moodes operatee at 1.8V suupply with

10-b single--slope (SS) ADCs. In hhigh-sensitivvity mode, each

e pixel ggenerates an amplified

s
signal from fiers, which suppresses the input
m in-pixel diifferential ccommon-souurce amplifi

r
referred noiise in dark cconditions. In normal mode,
m the siignal is readd out thoughh a source

f
follower as in conventional sensors.

Figure 4.1 Operation of adaptivee imaging

F

1103
 Figuree 4.2 Operaation and block diagraam of the ad
daptive imaage sensor with
reconfiigurable modes

In WDR
R mode, duual exposuree prevents thhe signal frrom saturatiion by provviding two

i
integration times, shorrt and long, in the sam
me frame. T
The sensor has colum
mn parallel

A
ADCs and 110-b latchess. The ADC
C is designedd to be recoonfigured too operate as either 8-b

S
SAR or 10--b SS ADCs accordingg to the moddes. The coolumn scannner accessess the latch

a image signals
and s are rread out throough sense aamplifiers.

4 Pixel and column circuits forr reconfigu

4.2 urable modees
Figure 44.3 shows tthe pixel arrchitecture aand figure 44.4 shows eequivalent ccircuits in

e
each mode. Two pixelss are verticaally shared aand form onne group. Tw
wo groups can
c form a

d
differential pair for in-pixel differential aamplifiers. This differrential connfiguration

e
eliminates a gain loss ffrom the boody effect inn the sourcee follower. IIn order to rreduce the

1104
VGS drop aand increasee the signall swing, low
w-VTH transsistors are uused for ‘T
TAMP.’ The

c
control signnals for transsmission gaates, ‘TX0’ and
a ‘TX1,’ are boostedd by bootstraap circuits

o
outside the pixel. For differential readout, eaach group uuses separatte signal linnes ‘SIG0’

a ‘SIG1.’’ ‘COM’ linne, which is common to two grouups, is connnected to eiither pixel
and

p
power suppply (for souurce followeer operationn) or currennt bias (for differential amplifier

o
operation). ‘RSTC’ is connected to the capaacitive DAC
C for in-pixxel SAR AD
DC in the

m
monitoring mode or is connected tto pixel pow
wer supply ffor the reset operation. Figure
F 4.5

s
shows the tiiming diagraam of the piixel control in each mode.

Figgure 4.3 Pixxel circuit oof reconfigu

urable pixeel

1105
 COM

Figure 4.4 Pixel architecture and equivalent circuits in each mode

In monitoring mode, the pixel circuit operates as a preamplifier of SAR ADC at 0.8V

as shown in figure 4.6. The signal from the capacitive DAC, which is located in the

column circuits, is provided through ‘RSTC’ and the in-pixel preamplifier generates the

amplified comparison signal in each conversion cycle. The pixel power supply (0.8 V) is

used for the reference voltage of SAR ADC.

106
 SAR ADC Photodiode reset

RST0

RST1

TX0

TX1

Reset Charge sharing

Monitoring mode

RST0 RST0

RST1 RST1

TX0 TX0

TX1 TX1

Reset Charge transfer Reset Charge transfer

Normal / WDR mode High-sensitivity mode

Figure 4.5 Timing diagram of the pixel control in each mode

Figure 4.7 shows the capacitive DAC for the SAR ADC. Since most power

consumption comes from the driving of capacitances, we used a modified DAC with

reduced capacitance [4.6]. Conventional DAC has total 28C capacitances for 8-bit

conversion. In addition to large power consumption, placing capacitors in the small

column pitch is also a problem. However, the modified DAC has only 47C total

capacitances.

107
 In order to adjust the signal swing, we used parasitic capacitance of ‘RSTC’ instead

of employing additional reference capacitors or voltage generators.

Figure 4.6 In-pixel SAR ADC in monitoring mode

RSTC

Figure 4.7 Capacitive DAC with reduced capacitance

The reference voltage of SAR ADCs in the monitoring mode is as follows:

108
 C
V = V (4.1)
C +C

,where CDAC is DAC capacitance, CPAR is the parasitic capacitance of RSTC line. In order

to suppress the FPN from the parasitic capacitance variation, we added calibration

capacitors CCAL which are configured by 4-b digital signals. However, we did not have to

use the calibration capacitors in the testing because we did not see any noticeable FPN

from signal swing variation. The measured signal swing is 0.41 V out of 0.8 V power

supply. Since the pixel operates at 0.8V, it is difficult to achieve a complete charge

transfer from pinned photodiodes due to low reset level in the floating diffusion.

Therefore, we chose to apply charge sharing. After integration, the floating diffusion

node is reset and then the integrated charge is shared with the floating diffusion by

opening the transfer gate. This operation disables true CDS like a 3-T pixel; however, we

can achieve significant power savings.

Figure 4.8 Column-parallel ADC for the reconfigurable modes

109
 In normal mode, the pixel circuit is configured as a source follower that is exactly

same as in conventional 4-T pixels. In high-sensitivity mode, the pixel operates as a

differential common-source amplifier and achieves a higher gain (> 6). The differential

common source amplifier generates the output signal from the two floating diffusion

nodes: one gives a signal voltage and the other gives the reset voltage. In WDR mode, the

same circuit is used as in the normal mode. In order to enhance the dynamic range, pixel

merging with dual exposure is applied to the pixel. This will be explained in the next

subsection.

Figure 4.9 Timing diagram of column-parallel ADC (a) Single ended input (normal,
WDR mode), (b) Differential input (high-sensitivity mode)

Figure 4.8 shows the column parallel ADC and figure 4.9 shows its operation. The

column parallel ADC consists of a preamplifier operating at 1.8 V, a dynamic comparator

at 0.8V, and switches for mode configuration. In monitoring mode, only 0.8 V supply is

110
used and column-level preamplifiers are turned off for power saving. The amplified

signal from the pixel (‘SIG0’, ‘SIG1’) is directly delivered to the dynamic comparator

through ‘SM’ for SAR ADC. In the other three modes, the column parallel ADC operates

as a single-slope ADC. The coupling capacitor ‘CC’ stores the offset of the preamplifier

and suppresses the column FPN. In the normal and WDR modes, reset voltage is sampled

in CDAC, signal voltage is sampled in CS, and then VRAMP is swept. In case of high-

sensitivity mode, the operation is fully differential, i.e., the differential signals from the

pixel is sampled in CS and CDAC simultaneously.

CLKSAR
T
clk

Figure 4.10 Hybrid latch circuit

111
 Comp

CN
NT
CNT C
CNT
QCO
ONV

QDIFF
D

Figgure 4.11 Comparison with conveentional lattch in singlee-slope ADC

C

Figure 44.10 shows the hybrid latch circuiit. The latchh operates ffor both SA
AR and SS

A
ADC by m
multiplexing. Differentiial latching reduces sw
witching poower becausse it only

s
switches onnce during thhe whole SS ADC cyccles, while thhe SRAM oor transfer-ggate based

f
flip-flop alw
ways changges its state whenever the counterr signal chaanges. The switching

c
characteristi
ics of differrential and cconventionaal latches aree shown in ffigure 4.11.

4 In-equaality readou
4.3 ut
Figure 44.12 shows conventiona
c al digital siggnal readoutt scheme.

Figuree 4.12 Convventional diigital imagee signal reaadout circuiit and operation

1112
 The preecharged seense line (S
SL, sense aamplifier innput) is disscharged thrrough the

r
readout celll according to
t the storedd value in thhe latch.

Figgure 4.13 Im
mage localitty

When thhe digital siggnals from the column latch are acccessed, thee prechargedd line may

b frequenttly discharged by the readout

be r cell according to the storeed value in the latch.

W
When the sstored signaal is ‘1’, thee prechargedd line is discharged annd the line should be

c
charged agaain for the next colum
mn readout. Precharginng the sensse line that has large

c
capacitance entails hugge dynamic power
p consuumption.

Fortunattely, the im
mage signal has
h strong llocality as shown
s in figgure 4.13. IIt has high

p
probability that neighbboring pixels have sim
milar valuess. In order to reduce the
t power

c
consumptio n in the precharge
p mplement a scheme too read an inequality
node, we im

b
between neeighboring ccolumns sinnce most im
mage signalls typically have locallly similar

v
values (low spatial bandwidth).

Figure 4.14
4 shows tthe digital ssignal readoout circuit. W
We put XOR
R gates in thhe readout

c for upper 4 MSBs. One of the two XOR ggate inputs ccomes from
cell m the left collumn. The

1113
l
line keeps tthe pre-charrged voltagee if the two columns haave the sam
me value. In this case,

t
there is neggligible dynaamic currennt. We testedd with 100 images from
m the imagee database

[
[4.7]. The probability of equalityy in each bbit positionn is shown in Figure 4.15. The

p
probability of equality is defined aas Eq. 4.2. T
This plot is from the avveraged ressult of 100

i
images. Figgure 4.15 ((a) shows thhe probabillity that 1-bb digital siignals in sppecific bit

p
position aree same in thhe neighboriing columnss. Figure 4.115 (b) show
ws the probaability that

1-b digital ssignal in speecific bit poosition is ‘1’. The probaability of ‘1’’ is defined as Eq. 4.3.

# of pixelss with equaality

P = ((4.2)
Total # of pixels

# of pixxels with ′1′′

P = ((4.3)
Total # of pixels

Figurre 4.14 Diggital image signal

s read
dout circuit with in-equ
uality readout

The proobability of equality meeans the proobability off negligible dynamic cuurrent. As

s
shown in thhe figure, thhe probabiliity of equality is 0.73 (@ 4th bit)) and 0.94 ((@ MSB),

w
whereas thee probabilityy of ‘1’ is aaround 0.5. The power consumptioon overheadd from the

t
transmission
n gate basedd XOR is negligible.
n A
According too this analysis, we can achieve a

p
power reducction of 68%
% (only from
m the pre-ccharge) and 5% from thhe total chipp power in

1114
t monitorring mode. The
the T area ovverhead from
m XOR gattes is only 00.36 % from
m the core

s
size.

Fiigure 4.15 A
Averaged p
probability of equalityy

Figuree 4.16 Duall exposure sscheme: pixxel output rresponse an

nd conventiional
impplementatioon

4 Dual exxposure witth pixel merrging

4.4
As menntioned in chhapter 3, dyynamic rangge extensionn using the llinear schem
me such as

t sensitivvity adjustm
the ment and thee multiple eexposure scchemes provvide better SNR than

n
nonlinear scchemes. In the linear scheme,
s muultiple expossure is betteer choice beecause the

s
sensitivity a
adjustment rrequires in-ppixel capaciitors and thiis scheme iss not applicaable to the

1115
shared pixel architecture. However, multiple exposure scheme requires additional line

memories. It also entails large power consumption due to the multiple readouts.

In this work, we implemented a dual exposure scheme with pixel merging. The

operation principle is shown in figure 4.16. The signal from short integration time (TINT2)

and the signal from long integration time (TINT) are synthesized. As shown in the pixel

output response, we can get the signal without saturation from the short integration time

(TINT2) in high illumination. In conventional dual exposure scheme, line memories are

required for the synthesis as shown in figure 4.16. Moreover, two times readout doubles

the power consumption from the readout.

Figure 4.17 Dual exposure with pixel merging

In order to avoid additional line memory and additional power consumption, we

proposed a dual exposure scheme based on the pixel merging. As shown in figure 4.17,

the even group has a regular integration time (TINT) and the odd group has a short

integration time (TINT2). In the readout, one row from the even group and one row from

the odd group are merged, and generate a signal with enhanced dynamic range. This

116
s
scheme sacrifices the sspatial resoolution by hhalf; howeveer, it is adeequate for low-power

a
applications
s because no
n additionaal in-pixel eelement or iimage reconnstruction is required

a the opeeration is sim

and me readout, no additionnal power
mple. Sincee we need oonly one tim

c
consumptio n is requiredd as the connventional m
multiple expposure schem
mes.

Figure 44.18 Chip m

microphotoograph

4 Implem
4.5 mentation an
nd experim
mental resullts
h been fabbricated usinng 0.18μm CIS processs. A chip m
A prototype chip has micrograph

i shown in figure 4.18. The perforrmance of thhe sensor is summarizeed in table 4.1.
is

Figure 44.19 shows the captureed images ffrom the faabricated deevice. Figure 4.19 (a)

s
shows the im
mage from tthe monitorring mode (@
@ 9 lx) andd figure 4.199 (b) shows the image

f
from the higgh-sensitivitty mode in a dark conddition (@ 0.44 lx).

1117
Tablee 4.1 Chip ccharacterisstics

Figu
ure 4.19 Saample imagges

1118
 Figure 44.19 (c) shoows the image from thee normal moode when illlumination high. The

s
saturation frrom high illlumination iis eliminatedd in WDR m
mode as shoown in figurre 4.19 (d).

The pow
wer figure of
o merit (Poower-FOM) can be defi
fined as the power norm
malized to

f
frame rate aand the num
mber of pixeels, given bby power/fraame·pixel. T
This does not
n exactly

r
reflect a goood indicattion for thee performannce of imaage sensorss because tthe power

c
consumptio n of certainn circuit bloocks does not
n linearly increase ass the pixel array size

b
becomes laarge. More importantlly, signal-too-noise rattio (SNR) can be siggnificantly

c ed with poower consuumption to the level that the aacquired im

compromise mages are

e
extremely ppoor. Anothher figure oof merit, SN
NR-power FOM, can bee defined coonsidering

b
both SNR aand power coonsumptionn in its figurees:

SNR-Powerr FOM = ×1
10 [dB] ((4.4)

We com
mpared the SNR-Powerr FOM of oour work w
with previouus low-pow
wer CMOS

i
imagers in ttable 4.2. W FOMs of preevious workks, we used the whole
When we calculate the F

c
chip power consumptioon and the reeported SNR
R.

Tablle 4.2 FOM

M Comparisson with Previous Worrks

We alsoo plot the F

FOM in twoo axes: one is SNR annd the otherr is Power F
FOM. We

i
indicate a diagonal line that shoows an equuivalent SN
NR/Power-F
FOM line w
where the

1119
p
projection of achievabble SNR is estimatedd when additional poower consum
mption is

a
allowed.

Figure 4..20 SNR vs.. Power-FO

OM Plot

In the m
monitoring mode, the ssensor shouuld be able to operate solely depeending on

h
harvested ennergy. Thenn, the powerr FOM shouuld be less tthan 100 [ppW/frame/piixel] to be

s
suitable for the operatioon of 1 μW @ 100×1000 pixels, 1 ffps. At the ssame time, iin order to

m
monitor thhe environm
ment withoout severe performannce degraddation, the effective

r
resolution oof images shhould be oveer 6 bits (SN
NR = 36.1 ddB). Furtherr image proocessing in

t sensor node or inn the base station cann enhance the SNR and providde reliable
the

m
monitoring. Our sensorr working inn the monitooring mode satisfies this condition,, while the

o
other previoous sensors cannot.

In orderr to get moore accuratee environm

mental data (high-sensittivity) or inn extreme

i
illumination
n conditionss (wide-dynaamic range)), the sensorr will adaptiively changee its mode

o operationn and the poower source can be swittched from eenergy harvvester to a baattery. For
of

1120
these accurate image capturing modes, at least 8-b resolution (SNR = 48.2 dB) is required.

As shown in figure 4.20, our sensor can provide adequate SNR in other modes of

operation at the minimum power consumption. The sensor, however, will be mostly

working in monitoring mode by default unless the special needs happen. During the

monitoring mode, the extra energy harvested will recharge the battery. Considering both

SNR and power FOM, this work shows the best performance as shown in the figure.

4.6 Summary
In this chapter, we explained the energy/illumination adaptive imaging. As a

prototype, an adaptively reconfigurable CMOS imager has been designed and

characterized. It operates at 1.36 μW in the monitoring mode from harvested energy and

reconfigures to capture high-sensitive (24V/lx·sec) and wide dynamic range images

(100dB) at 867 μW in battery operation upon request. The chip achieved power FOM of

15 pW/pixel·frame in 0.18 µm technology.

121
Chapter 4 References

[4.1] K. -B. Cho, A. I. Krymski and E. R. Fossum, “A 1.5-V 550-μW 176 × 144
autonomous CMOS active pixel image sensor,” IEEE Transactions on Electron
Devices, vol. 50, no. 1, pp. 96-105, Jan. 2003
[4.2] S. Hanson, Z. Foo, D. Blaauw, and D. Sylvester, “A 0.5 V Sub-Microwatt CMOS
Image Sensor With Pulse-Width Modulation Read-Out,” IEEE Journal of Solid-
State Circuits, vol. 45, no. 4, pp. 759-767, Apr. 2010
[4.3] Q. Gao, S. Member, and O. Yadid-pecht, “A Low Power Block Based CMOS
Image Sensor with Dual VDD,” IEEE Sensor Journal, pp. 1-9, 2011
[4.4] J. Yang, K. G. Fife, L. Brooks, C. G. Sodini, A. Betts, P. Mudunuru and H. –S.
Lee, “A 3MPixel Low-Noise Flexible Architecture CMOS,” ISSCC Dig Tech.
Papers, pp. 496-497, Feb. 2006
[4.5] S.-W. Han, S.-J. Kim, J. Choi, C.-K. Kim, and E. Yoon, “A High Dynamic Range
CMOS Image Sensor with In-Pixel Floating-Node Analog Memory for Pixel
Level Integration Time Control,” Dig. Symp. VLSI Circuits, pp. 31-32, June 2006
[4.6] U.S. Patent, #US6,686,865 B2, Feb. 2004
[4.7] Caltech Image Database, http://www.vision.caltech.edu/html-files/archive.html
[4.8] K. Kagawa, S. Shishido, M. Nunoshita, and J. Ohta, “A 3.6pW/frame·pixel
1.35V PWM CMOS Imager with Dynamic Pixel Readout and no Static Bias
Current,” ISSCC Dig Tech. Papers, pp. 54-55, 2008

122
 CHAPTER 5 OBJECT-ADAPTIVE IMAGE SENSOR WITH EMBEDDED
FEATURE EXTRACTION FOR MOTION-TRIGGERED OBJECT-OF-
INTEREST IMAGING

5.1 Introduction
The image signal transmission requires large bandwidth to transmit the 8-b M × N

signals with 15 fps. The wireless transmission with high bandwidth requires high power

consumption > 10 mW which is not feasible in wireless sensor nodes. Using the low-

power transmission such as UWB still requires > 1 mW power consumption [5.1].

Therefore, it is important to reduce the bandwidth as well as reduce the power

consumption of the image sensor. One way to reduce the bandwidth is to generate signals

only when event happens by monitoring temporal changes [5.2-5.4]. However, this event-

based imaging has extraneous redundancy because the sensor may also respond to

environmental conditions, such as change of illumination or background movement in

addition to actual target objects.

In this chapter, a motion-triggered object-of-interest (OOI) imaging to suppress the

redundancy in imaging as well as transmission of signals will be explained. Figure 5.1

shows the operation procedure of the motion-triggered OOI imaging. In most of time the

sensor is in sleeping mode until it is triggered by motion. Then it wakes up, it generates

and transmits 8-b features for object detection. The signal processing unit which resides

either in the host or in the sensor node, performs detection of objects and feeds back 1-b

123
request signal to initiate further imaging operation if the target object-of-interest is

identified.

The feature extraction is the process that provides transformed representation sets

with less bandwidth. There is a variety of image processing for the feature extraction:

from simple low-level feature extraction that does not need shape information (for

example: edge detection, curvature detection, optical flow estimation) to the complex

high-level feature extraction that finds the shapes in the images (for example: Hough

transform, template matching).

The object detection is the task of finding presence and position of the object. The

object detection requires descriptors, which is a set of feature data describing the

properties of image. By comparing and matching the descriptors with the descriptors of

known objects, we can detect the object. The descriptor should be congruent, compact

and invariant (for example, invariant to the orientation of the object or invariant to the

illumination) [5.5]. A variety of algorithms for generating descriptor has been researched

such as Haar wavlet, sacale-invariant feature transform (SIFT), shape context and so on.

Among many feature extraction algorithms, we incorporated the histogram-of-oriented-

gradients (HOG) because it gives a high detection rate of objects with simple operation

[5.6]. The HOG feature is used in the human detection for applications of pedestrian

detectors. Humans have been proven to be difficult object to detect due to the variability

of appearance, cloth and articulation. While HOG feature provides powerful performance

(i.e., low miss rate of detection) compared with other features including wavelets [5.7,

5.8], it has simple operation to be implemented in wireless sensor nodes. In this work, we

implemented the HOG feature extraction algorithm using mixed-signal circuitry in order

124
t save bothh power andd area. The HOG featuure output rrequires onlyy 3.5% of bbandwidth
to

w
when compared with coonventionall 8-b image capturing.

Figure 55.1 Motion--triggered oobject-trigggered objecct-of-interesst (OOI) im

maging

For the object adapptive imaginng, we impplemented a prototype CMOS imaage sensor

w an inteegrated histtogram of oriented

with o graadient featurre extractorr. The protootype chip

h 256 × 2256 pixel arrrays and gennerates L × K × 9 8-bitt features (@

has @ 256/L × 2256/K cell

s
size) where the numberr of blocks L × K is conntrollable. IIn the case oof 32 × 32 blocks,
b we

c achievee ×113 banddwidth savinng comparedd with geneerating imagge signals. IIn the next
can

s
section, thee operationn principles of the HO
OG featuree will be eexplained. T
Then, the

p
proposed CM
MOS imagee sensor witth OOI imagging will be described.

5 Histogrram of Orieented Grad

5.2 dients (HOG
G) feature
OG feature extraction procedure is shown iin figure 5.2. The HO
The HO OG feature

e
extraction cchain is as foollows:

( Spatial gradient ggeneration: apply two 1-D masks GX and GY for thee x and y
(1)

d
direction reespectively. The maskss are GX = [-1 0 1] aand GY = [-1 0 1]T. T
The result

g
generates tw
wo M × N gradient im
mages from
m the M × N image ffor x and y direction

r
respectively
y.
1125
(2) Magnitude and angle calculation: from two gradient images, calculate the magnitude

|H| and angle θ as follows:

|H| = GX + GY (5.1)

GY
θ = tan (5.2)
GX

(3) Voting: each pixel is voted to 9 bin numbers (0 ~ 8), which decode the angle from 0°

to 180° as shown in figure 5.2. For example, if the angle θ is 50°, then the bin number is

2.

(4) Histogram generation: M × N image is divided by L × K blocks. Each block has M/L

× N/K pixels. Typically the block size M/L × N/K is over 8 × 8. Each block generates

one histogram. Each histogram has 9 bin numbers as an index. According to the voting,

the magnitude |H| is accumulated in each bin number. The accumulated magnitude in a

cell will be finally normalized.

The histogram of oriented gradients shows the magnitude distributions in each angle.

For example, if there is a horizontal pillar in the scene, the histogram will have the

biggest magnitude accumulation in the bin number 4 (80° ~ 100°).

The block size (M/L × N/K) induces different detection result. Large cell size can

generate the information over wide area instead of losing the details. Small cell size can

generate detailed information but requires more processing. Some of object detectors

have the classification algorithms that use multiple cell sizes in one detection procedure

[5.9]. For example, the histogram from four different cell sizes 8 × 8, 16 × 16, 32 × 32

and 64 × 64 are concatenated as a feature set and the concatenated feature set is used for

the classification.

126
 As an integrated feature exttractor in CMOS imaage sensorss, it is desirable to

i
implement t cell sizee programm
the mable in ordeer to be appplicable to thhe different situations

s
such as the object or thhe classificattion algorithhm. Moreovver, after thee sensor gennerates the

h
histogram w
with M/L × N/K cell size, histograams with muultiple cell ssize can be generated

b the post image proceessing, whicch can be appplicable to the algorithhm mentioneed above.
by

Figure 5.2 HO
OG featuree extraction
n proceduree

5 CMOS imager witth motion-ttriggered ob

5.3 bject-of-intterest imagiing

5
5.3.1 Objecct-of-interest imaging
Figure 55.3 shows thhe simplifieed block diaagram of thrree differennt modes off operation

i the propoosed sensor chip. In mootion sensinng mode, thee sensor gennerates 128 × 128 1-b
in

m
motion mapp for motionn triggeringg. The colum
mn-parallel ADCs opeerate as a 1-b motion

c
comparator.
. Once the motion
m is ddetected, thee sensor wakkes up and turns into the
t feature

e
extraction m
mode. In thiis mode, thee sensor gennerates and ttransmits 8--b feature siignals that

1127
are extracted from full 8-b 256 × 256 images. We have SRAM blocks, which temporarily

store intermediate signals during HOG feature calculation. One part of SRAM blocks are

allocated for line buffers to store 3 rows of images and the rest are allocated for feature

signal accumulation. In imaging & storing mode, the sensor generates and transmits 8-b

images upon the request from the host. The SRAM operates as a frame memory, which

can store region-of-interest images containing the object-of-interest.

Figure 5.3 Operation of the motion-triggered object-of-interest imaging

5.3.2 Overall architecture

The overall architecture of the sensor is shown in figure 5.4. The column-parallel 8-b

single slope ADCs operate as a 1-b motion comparator in the motion sensing mode and

generates a 1-b motion map with frame-difference signals from the pixel. For HOG

feature extraction, we need to calculate the magnitude and angle of spatial gradients. The

pixel array is divided into m × m sub-blocks where the size of sub-blocks is fully

programmable. A set of histogram that maps the magnitude of spatial gradients in each

angle is generated for each block. The processing of feature signal extraction is as

follows: (1) Three rows of images are read out and are temporarily stored in SRAMs; (2)

Spatial gradients (GX and GY) are calculated with masking [-1 0 1] and [-1 0 1]T,

128
respectively; (3) Gradient-to-angle converter (GAC) generates 9 bin numbers, which map

9 different angles from 0° to 180°; and (4) Histogram generator accumulates the

magnitude of gradients (|GX+GY|) corresponding to its angle (θ = tan-1(GY/GX))

calculated from GAC, and store them back to SRAMs. The accumulated magnitudes are

normalized by K in order to deliver full-scale 8-b feature signals.

Figure 5.4 Overall architecture

5.3.3 Pixel circuit

Figure 5.5 shows the pixel architecture. The pixel employs a reconfigurable

differential topology, which enables both source follower readout and differential

common source amplification without any additional transistors, as reported in our

previous work [5.10]. Two vertical neighboring pixels share the COM line and connect

signals through SIG0 and SIG1. In this sensor, we used an additional in-pixel capacitor

CE/O as frame buffer memory for motion sensing as well as for low-power imaging. The

in-pixel capacitor is implemented using an MIM capacitor and it is placed on top of pixel

129
circuits. Therefore, the capacitor does not induce any fill-factor loss. Two vertically

neighboring pixels are connected with a merging switch (M) for pixel merging.

COM
Even
TXE
RE
FDE

SE

CSE CE

VCOMP(E)
M

TXO
RO
FDO

SO

CSO CO
Odd
VCOMP(O)

Figure 5.5 Pixel circuit

The equivalent circuit of the pixel is shown in figure 5.6. In the motion sensing mode,

operation sequence is as follows: (1) Two pixels are merged and one in-pixel capacitor

samples the previous frame signal (V1); (2) The other in-pixel capacitor samples the

current frame signal (V2); (3) A short pulse is applied to VCOMP(E) in the even pixel and

130
V1 is compared with V2; (4) Likewise, another pulse is applied to VCOMP(O) in the odd

pixel and two voltages are compared again. When the frame difference |V1 - V2| is higher

than the amplitude of VCOMP pulse swing, 1-b motion output is generated.

VDDP

(1) TXE TXO (2)

COM

V1 V2
(3) (4)
VCOMP(E) VCOMP(O)

M
Motion sensing
VDDP

TXE TXO

COM

VSIG VRST
VCOMP(E) VCOMP(O)

Low-power imaging

Figure 5.6 Equivalent pixel circuit for motion sensing and low-power imaging

For low-power imaging in the feature extraction mode, the pixel circuit operates as a

pre-amplifier for an 8-b single slope ADC. The integrated charges in a photodiode are

transferred to an in-pixel capacitor in the even pixel whereas the odd pixel keeps the reset

voltage. A ramp signal is applied to VCOMP(E) and the potential is dropped at SIG0 node

131
due to charge transfer. The latching occurs around the reset voltage. In this scheme, VGS

drop in the amplification transistors does not induce the loss of signal swing as in the

conventional source follower readout; i.e., the power supply can be scaled down by more

than 0.5 V (nominal voltage drop in VGS). In this sensor, we used 1.3V for pixel power

supply which significantly saves power consumption in the pixel circuits.

5.3.4 Column circuit

Figure 5.7 shows the column circuit. The column circuit consists of the column-

parallel 8-b ADC and SRAMs.

Figure 5.7 Column circuit

The ADC consists of sample & hold circuit, preamplifier, multiplexer, comparator

and 8-b latch. Note that the sample & hold circuit and the preamplifier are used for the

readout from the in-pixel source follower only for the testing purpose. The preamplifier is

turned off in actual operation. In actual operation, the differential signal line SIG0,1 are

directly connected with comparator inputs through the multiplexer. The offset voltage of

the in-pixel differential comparator is stored in the capacitor CC in order to suppress the

FPN from the offset voltage. When F is enabled, the output from the reset voltage is

sampled on CC. After F is disabled, integrated charges are transferred and ramp signal is

132
s
swept for thhe AD convversion. Thee power suppply voltage of the collumn circuiit (VDD) is

0 V.
0.8

5
5.3.5 Gradiient-to-Anggle Convertter (GAC)
Figure 5.8 shows the votingg process. Since the angle θ ccan be exppressed as

ttan-1(GY/GX
X), we can ddecode the angle into 9 bin numbeers using thee following equation:

tanθ = ((5.3)

Figgure 5.8 Vooting processs

In orderr to decodee with 9 bin numbers using the above equaation, tanθ should be

c
compared w
with the resuult from a divider. Thiss operation rrequires a ddivider, also a look-up

t
table for thee trigonomeetric functionn. Instead of
o using the complex opperation, wee proposed

a simple m
mixed-signaal approachh in order to avoid digital im
mplementatioon, which

c
consumes hhuge area/poower. The gradient-to-a
g angle conveerter (GAC)) circuit bassed on the

m al approachh is shown iin figure 5.99. The timinng diagram of GAC is shown in
mixed-signa

f
figure m 0° to 180° in 4-b binn numbers.
5.10. The GAC ggenerates 9 angle inforrmation from

T 8-b graadient signaals (GX andd GY) are coonverted to analog signnals using tw
The wo binary

c
capacitive D
DACs. Wheen the switcch S1 is off,
f, the binaryy DAC outpput is divideed by two.

O
One capaciitor (C0,1) operates aas the part of DAC, which gennerates trigonometric

f
function, while the otther (C0s,1s) operatess as a holdd capacitor, which conntains the
w

g
gradient.

1133
 R

S2 S3

C0 C20 C40
S1
(i)
C0S
|GX|

C1S
|GY|
S1 (ii)
C1 C60 C80

S2 S3
DAC
R

Figure 5.9 Gradient-to-angle converter (GAC)

Figure 5.10 Timing diagram of gradient-to-angle converter (GAC)

134
 The graddient is mulltiplied by tanθ
t and 1/taanθ in each channel, reespectively, according

t capacitannce values of
to o C20, C40,, C60, and C80.
C

(i) <1) (ii) >1

1)

t3

t4

Table 5.1 Capacitancce ratio in tthe GAC

The propposed GAC

C performs two compariisons in parrallel using ttwo comparrators. The

c
comparison
n is as follow
ws:

|GX| ∙ tanθ = |G
GY| (tanθ < 1) ((5.4)

|GY|
|G
GX| = (tanθ > 1)) ((5.5)
tanθ

Table 5.22 Encodingg of 9 bin nu

umbers

1135
 In the first comparison, |GY| is compared with |GX| ∙ tanθ. In the second comparison,

| |
|GX| is compared with . In each comparison, two-step comparisons are performed. In

the first comparison, θ is 40° in the first step and is 20° in the second step. This two-step

comparison divides the range of the angle as follows: θ > 40° or 20° < θ < 40° or θ < 20°.

Likewise, in the second comparison, θ is 60° in the first step and is 80° in the second step.

This two-step comparison divides the range of the angle as follows: θ < 60° or 60° < θ <

80° or θ > 80°. Each step generates 1-bit comparison result. Therefore, total 4-bit output

is generated from two two-step comparison. Since tanθ = - tan(180° − θ), we don’t need

to do the operation for the angle higher than 90°. In other words, the comparison only

cares about the magnitude of tanθ and detects the angle range from 0° to 90°. The sign bit

(which we already have generated from the gradient calculation) shows whether the

decoded angle is in 0° < θ < 90° or in 90° < θ < 180°. After two-step comparisons during

t3 and t4, the 4-bit output from the comparison is encoded according to the sign bit (S) in

order to generate a final bin number. Table 5.2 shows the encoding table. The sign bit

determines whether the encoded angle is in 0° < θ < 90° or in 90° < θ < 180°.

5.4 Chip characteristics and object detection simulation

The prototype chip has been fabricated using 0.18 μm 1P4M CMOS process. Figure

5.11 shows the chip microphotograph. The core size is 2.35 × 3.18 mm2. Figure 5.12

shows the captured images from the fabricated device including a 128 × 128 1-b motion

map and a 256×256 8-b image from the in-pixel single-slope ADC. A 256 × 256 angle

map shows the angle calculated from the GAC. A 8-b feature from 8 × 8 blocks are

136
s
shown withh the vector. In this figuure, only twoo angles thaat have the biggest
b maggnitude are

s
shown for simplicity.

Figurre 5.11 Chiip micrograaph

The perrformance oof the sensorr is summarrized in tablle 5.3. We aachieved a normalized
n

p
power of 133.46 pW/fraame•pixel inn motion seensing and 51.94
5 pW/fframe•pixel in feature

e
extraction. IIn order to vverify the performance of the integgrated featuure extraction unit, we

t
tested the oobject detecttion from thhe extractedd features ussing 200 pedestrian im
mages from

D
DaimlerChr
rysler dataset [6.10]. F
For testing, we input ttest images serially intto the 8-b

l
latch in thee column paarallel ADC
C and generrated featuree. The test result show
ws 94.5 %

d
detection raate. The detaailed proceddure of the object
o detecction will bee described in section

5
5.4.1.

1137
 Figu
ure 5.12 Saample imagges

Tablee 5.3 Chip ccharacterisstics

1138
5
5.4.1 Testin
ng of the ob
bject detecttion
Figure 5.13 showss the objecct detection procedure. The objecct detectionn requires

c
classifier w
with embeddded classificcation algoriithm. The cclassifier ideentifies the object-of-

i
interest andd generates 1-b output of the detection resultt. In order tto classify tthe object,

p
pre-trained model hass to be loaaded on thee classifier.. The trainning requirees another

a
algorithm such as suppport vector machine (S
SVM), whicch is supervvised learniing model

w associaated learningg algorithm

with ms that analyyze the data and recogniize the patteerns.

Figure 5.113 Object detection

d prrocedure

For the training, usuually lots off images (trraining imagge set) moree than 1,0000 are used.

T
This traininng set incluudes both the
t positivee images (w
with the taarget objectt) and the

n
negative im
mages (withoout target obbject). For thhe object deetection testting, a differrent set of

i
images (tesst image sett) are used. The featurre is input to the trainned classifieer and the

d
detection raate can be m
measured. Inn this work, we used ssimple SVM
M for the traaining and

S
SVM classifier withouut any advvanced algoorithms in the
t MATLA
AB. We used 1,000

1139
training images (500 positive, 500 negative) for the training [6]. In order to measure the

detection rate, we used 200 test image sets (100 positive, 100 negative) [6.10]. The test

images are serially input to the latch in the single-slope ADC and generated HOG feature

from the fabricated chip are input to the classifier. According to the measurement, it

shows 95% detection rate for the positive image set and 94% detection rate for the

negative image set. The overall detection rate is 94.5 %. We expect improved detection

rate by using advanced training and classification algorithms [6.9].

5.4.2 Power figure of merit (FOM) comparison with previous low-power imagers
The proposed CMOS imager with object-of-interest imaging consumes low power

applicable to wireless sensor network applications. In low-power sensors, a small pixel

pitch is an important factor because the parasitic capacitance of metal line (Cm) is linearly

increased according to the pixel pitch (WPIXEL) as shown in Eq.5.6

C =N ∙W ∙c (5.6)

In Eq.5.6, NPIXEL is the number of pixels and cm is the parasitic capacitance per μm.

Since the power consumption is proportional to the capacitance, low-power sensors

should have small pixel pitch. However, small pixel pitch decreases the sensitivity of the

pixel due to reduced fill factor. Therefore, it is important to keep high fill factor while

reducing the pixel pitch. Many low-power sensors use additional in-pixel circuits [6.1-6.3]

or pMOS transistors [6.4] for the in-pixel event generation or in-pixel ADC.

In this work, we use the differential pixel topology by grouping two vertically

neighboring pixels in order to achieve in-pixel ADC without additional transistors. The

additional in-pixel memory is implemented with MIM capacitor and it is located on the

top of pixel circuits in order to keep small pixel pitch. In this way, we achieved 5.9 μm

140
p
pixel pitch with 30% fill
f factor. F
For the com
mparison off power connsumption, tthe power

f
figure of m
merit (FOM
M) is typicaally used. T
The power FOM is ddefined as tthe power

n
normalized frame·pixel. Table 5.4
to frame ratte and the nnumber of piixels, given by power/fr

s
shows the ppower FOM
M comparisoon. The propposed sensoor has 13.466 pW poweer FOM in

t motion sensing, 51.94 pW in tthe feature eextraction inncluding thee imaging power,
the p and

i shows thee lowest pow

it wer consump
mption as shoown in the taable.

Table 55.4 Comparrison with previous

p low
w-power seensors

5 Summaary
5.5
w embeddded feature extraction
In this chapter, thee object-adaaptive imagge sensor with

f motion-ttriggered obbject-of-inteerest imaginng has been described. The sensor wakes up

for

t
triggered by motion ssensing andd extracts features
f froom the capptured imagge for the

d
detection off object-of-iinterest. Full image cappturing operaation is perfformed onlyy when the

i
interested obbjects are foound, whichh significanttly reduces ppower consuumption at the sensor

n
node. The cchip operatees at 0.22 
W/frame inn motion seensing mode and operaates at 3.4


W/frame f feature eextraction, reespectively..
for

1141
Chapter 5 References

[5.1] S. Chen; W. Tang; X. Zhang & E. Culurciello, "A 64×64 Pixels UWB Wireless
Temporal-Difference Digital Image Sensor," IEEE Transactions on Very Large
Scale Integration (VLSI) Systems, vol.20, no.12, pp.2232-2240, Dec. 2012.
[5.2] L.-gee Chen, “A single-chip CMOS APS camera with direct frame difference
output,” IEEE Journal of Solid-State Circuits, vol. 34, no. 10, pp. 1415-1418,
1999.
[5.3] U. Mallik, M. Clapp, G. Cauwenberghs, & R. Etienne-Cummings, “Temporal
change threshold detection imager,” ISSCC Dig Tech. Papers, pp. 362-364, Feb.
2005.
[5.4] N. Massari, M. Gottardi, S. Jawed, "A 100μW 64×128-Pixel Contrast-Based
Asynchronous Binary Vision Sensor for Wireless Sensor Networks," ISSCC Dig
Tech. Papers, pp. 588-638, Feb. 2008.
[5.5] Mark S. Nixon and Albert S. Aguado, “Feature Extraction and Image Processing,”
Book Chapter 7, Academic Press, 2008
[5.6] N. Dalal, B. Triggs,”Histogram of oriented gradients for human detection,”
CVPR Dig Tech. Papers, pp. 886-893, Jun. 2005.
[5.7] A. Mohan, C. Papageorgiou, and T. Poggio. Example-based object detection in
images by components. PAMI, 23(4):349-361, April 2001
[5.8] P. Viola, M. J. Jones, and D. Snow. Detecting pedestrians using patterns of
motion and appearance. The 9th ICCV, Nice, France, volume 1, pages 734–741,
2003
[5.9] S. Maji, A. C. Berg, and J. Malik, “Classification using intersection kernel
support vector machines is efficient,” in 2008 IEEE Conference on Computer
Vision and Pattern Recognition, 2008, pp. 1-8.
[5.10] J. Choi, S. Park, J. Cho & E. Yoon, “A 1.36μW adaptive CMOS image sensor
with reconfigurable modes of operation from available energy/illumination for
distributed wireless sensor network,” ISSCC Dig Tech. Papers, pp. 112-114, Feb.
2012.
[5.11] M. T. Chung & C.-C. Hsieh, “A 0.5V 4.95μW 11.8fps PWM CMOS imager with
82dB dynamic range and 0.055% fixed-pattern noise, “ISSCC Dig Tech. Papers,

142
 pp. 114-116, Feb. 2012
[5.12] Daimler Pedestrian Benchmark Datasets
(http://www.gavrila.net/Research/Pedestrian_Detection/)

143
 CHAPTER 6 CONCLUSION AND FUTURE WORK

This thesis presents an environment-adaptive CMOS image sensor for energy-

efficient operation. This chapter summarizes the main contributions and future directions

of adaptive sensors.

6.1 Conclusion and summary of contributions

Main contributions of the completed work are summarized as follows:

(1) Implementation of adaptive imaging at low-power consumption (< 1.36 μW) with

high dynamic range (99 dB) and low data rate (< 3.5 %);

(2) Low power imaging at the best SNR-power FOM of 25.22 dB among the work

reported up to date; and

(3) Low power imaging with feature extraction at the best power FOM (51.94 pW)

among the work reported up to date.

We proposed an adaptive image sensor that is applicable to wireless distributed

image sensor networks and wireless biomedical imaging systems. The adaptive imaging

includes: (1) the energy-adaptive imaging for the low-power consumption, (2) the

illumination-adaptive imaging for high dynamic range, and (3) the object-adaptive

imaging for spatial-temporal bandwidth savings. In order to prove the feasibility of the

proposed adaptive imaging schemes, we designed two prototype chips: one for the

energy/illumination adaptation, and the other for the object adaptation. Both prototype

sensors have been fabricated and fully characterized.

144
 For the energy adaptation, we implemented the low-power monitoring operation in

the adaptive image sensor. The sensor keeps monitoring at an extremely low power <

1.36 μW/frame, and only turns into high sensitivity or wide dynamic range imaging

operation when the power delivery is enough from energy harvesting units. In-pixel SAR

ADCs enable the voltage scaling down to < 0.8 V without any significant loss of signal

swing and non-linearity. The differential latches in the single slope ADC further reduce

the switching power. The in-equality readout scheme is also introduced to reduce the

number of switching at the output.

The illumination adaptation extends the dynamic range for reliable imaging in wide

range of illumination. At low illumination, the pixel circuit is reconfigured to become a

high-gain amplifier, and provides high sensitivity of 23.9 V/lx·s. At high illumination, in-

pixel dynamic range extension is performed using pixel merging and dual exposure

readout. This scheme significantly saves the chip size compared with conventional

schemes, and consumes less power compared with the multiple exposure schemes. The

sensor provides high dynamic range over 99 dB.

The object adaptation has been implemented by identifying the object-of-interest

(OOI) from the features extracted from image signals. The sensor wakes up triggered by

motion sensing and extracts features from the captured image for detection of the object-

of-interest. Full image capturing operation is performed only when interesting objects are

found, which significantly reduces bandwidth and power consumption of the sensor node.

The fabricated chip operates at 0.22 W/frame in the motion sensing and operates at 3.4

W/frame in the imaging with feature extraction.

145
 In summary, the adaptability enables an energy-efficient image sensing in

continuously varying environmental conditions such as illumination, energy harvesting,

and a variety of objects captured in the focal plane. By accommodating the three

adaptation schemes, the fabricated image sensor could provide high dynamic range

images at low power consumption, while maintaining low bandwidth in image signal

transmission.

6.2 Future adaptive image sensor

For the complete system, the adaptive image sensor should incorporate other

processing units including power management units and control blocks for self-decision

making, and also has to be mounted on a platform suitable for the deployment in a

distributed image sensor networks and for portable biomedical imaging systems.

6.2.1 Adaptive imaging system-on-a-chip

In the previous works, the energy/illumination adaptation and the object adaptation

capability have been implemented separately in the two prototype chips. In the future

sensor platform, all adaptive imaging capabilities should be integrated in a single chip in

order to achieve low-cost and small form-factor as well as low power consumption. In the

two prototype chips that we designed, some parts of control signals has been generated

from the off-chip FPGA that is mounted on the testing board for flexibility in

characterization. In the future design, all control blocks as well as other essential circuit

blocks have to be integrated on a chip, especially power management units (PMU) and

wireless transmitters. The power management unit (PMU) selects the energy source

either from energy harvesting units or from batteries, and it generates multiple power

supply voltages for the entire sensor unit.

146
 The controller is essential in order to manage the adaptive operation. The controller

should include: energy source monitoring, illumination-level analysis from previous

frame signals, motion triggering from the motion sensor output, and object detection

algorithm. The overall architecture of the adaptive imaging SoC is illustrated in figure 6.1.

Figure 6.1 Overall architecture of adaptive imaging SoC

6.2.2 Integration in a sensor platform

After the design and the characterization of the adaptive imaging SoC, the final step

would be the integration of the sensor platform including lens, solar cells and batteries.

One of implementation is a three-dimensional cubic module that enhances the range of

monitoring. Figure 6.3 shows one possible implementation of imaging-cubic (iCube)

sensor platforms. The cubic sensor includes four adaptive imaging SoCs, two solar cell

panels, and batteries in each plane. The lens is mounted on top of each imager. The

foldable architecture using a flexible PCB enables 360° field of view. Therefore, we can

147
a
achieve higgh sensing visibility in multiple perspectiives for efffective surrveillance.

M
Moreover, tthe sensor pplatform cann be implem mall size less than 5 × 5 × 5 mm3
mented in sm

a low costt. The low-cost manuffacturing ennables multiiple placem

at ments of sennsors in a

g
given area. From the iimaging cubbe systems,, we are able to collecct more accurate data

b
based on gross
g statisstics for ennvironmental monitoring, surveiillance, andd security

a
applications
s.

Figure 6.2 Illlustration of imagingg Cubic (iCu

F ube) sensorr platform.

1148
APPENDIX

149
 Appendix A

Estimation of Power Consumption in CMOS Image Sensor

150
 In this section, we describe detailed estimation of dynamic power consumption in

digital circuit blocks. As mentioned in chapter 3, we consider the dynamic power

consumption only from the capacitive driving (CLVDD2f) for simplicity. Moreover, the

leakage power consumption will be ignored. This estimation cause error compared with

actual dynamic power consumption. However, basic purpose of power estimation in this

section is to get power reduction strategy by inspecting the tendency of power

consumption according to important parameters such as the number of pixel array, power

supply voltage and pixel pitch.

Column-parallel Single-slope ADC

A. Comparator: Dynamic Comparator

The dynamic comparator performs the level decision from the pre-amplified signals.

It operates with two phases: the precharge and the comparison. Figure A.1 shows the

comparator circuit.

Figure A.1 Dynamic comparator

151
 The first stage which consists of latch, switch (pMOS for precharge) and inverter

induces the power consumption in each clock cycle. The frequency is 2N/TROW. The 2nd

stage which consists of RS latch and buffer has only one time switching during the ADC.

Therefore the frequency is 1/TROW.

The power consumption per one comparator (per column) can be expressed as:

P = (P +P +P ) + (P +P )

== {2c V + 2c V + 2c V }+

1 (A.1)
{2c V + 10c V }
T

2 1
= (6c )V + (12c )V
T T

,where PSW is the power consumption from the precharge switches, PLATCH is from the

latch, PINV is from the inverter, PSR is from the RS latch, and PBUFFER is from the buffer..

B. Latches

The latch for N-bit single-slope ADC consists of N D flip-flops. The circuit schematic

of D flip-flop is shown in figure A.2. The output Q is connected with read buffer (one

nMOS transistor, not shown in the figure) for the digital signal readout.

The power consumption of one latch in the single-slope ADC can be represented as

follows:

(A.2)
P _ = 2c V f + 3c V f : from 2 transmission gates & 2 inverters, master
+2c V f + 4c V f : from 2 transmission gates & 2 inverters, slave
+7c V f +c V f : from clock buffer and output buffers

152
 = 13c +c V f + 5c V f
1 f ( )
= 13c +c V + 5c V
T T

fD = 1/TROW, input (D) frequency, the input signal is the counter output.
fck = fCNT(n)/TROW, clock (clk) frequency, the clock signal is the comparator output.
fCNT(n): counter output frequency of n-th bit position

Figure A.2 D flip-flop

The total number of switching of N-b Gray code is (1+1+2+…2N-2). Therefore, the

counter output frequency fCNT(n) is (1+1+2+…2N-2)/TROW.

As shown in the equations, the power consumption of two inverters in the master

stage is dependent on the input frequency. In the single-slope ADC, when clock signal is

low, the master stage continuously changes its state from the counter signals. After clock

goes high and the counter signal is latched to the slave stage, no more switching occurs.

Therefore, the power consumption is largely dependent on the signal level (i.e., light

intensity) of single slope ADC. In this estimation, we define the illumination level

parameter ‘I’ which varies 1/2N (dark) to 1 (bright). For the power consumption of N-bit

latch in the single slope ADC which uses Gray counter can be expressed as follows:

153
 1
P =N 13c +c V
T
(A.3)
1
+5c V I(1 + 1 + 2 + ⋯ + 2 )
T

Note that the 2nd term includes the illumination level parameter ‘I’. If the signal level

is high from high illumination, the power consumption increases because the latch has

more switching before the comparator output is toggled.

C. Counter

Figure A.3 shows the Gray-code counter circuit. The 1st stage is the ripple counter

which generates the binary code. The Gray code from the XOR gates are buffered in the

2nd stage flip-flops in order to suppress the glitches.

D Q D Q Q<N-1>
Q Q

D Q D Q Q<1>
Q Q

D Q D Q Q<0>

CLK Q CLK Q
Figure A.3 Gray-code counter

The power consumption of the N-bit ripple counter can be expressed as:
(A.4)

154
 P = 2c V F + 3c V F : from 2 transmission gates & 2 inverters, master
+2c V F + 4c V F : from transmission gates & 2 inverters, slave
+7c V F + 4c V F : from clock buffer and output buffers
= 22c V F

1 + 2 + ⋯+ 2
F=
T

The exclusive-or (XOR) gate for the Gray code conversion consists of 3 gates. The
power consumption of XOR gate is:

P = 2c V F+c V F (A.5)

The 2nd stage flip-flops for the glitch suppression has the power consumption as
follows:
(A.6)
P = 2c V F + 3c V F : from 2 transmission gates & 2 inverters, master
+2c V F + 4c V F : from transmission gates & 2 inverters, slave
+7c V F + 4c V F : from clock buffer and output buffers
= 15c V F + 7c V F

1 + 1 + ⋯+ 2 F
F = ≈
T 2

The inverters drive the metal lines with high capacitance. The power consumption
from the driver is as follows:

P =# W c V F (A.7)

The overall power consumption from the counter can be expressed as:

P =P +P +P +P

(A.8)
1 + 2 + ⋯+ 2
= (39c + 0.5# W c )×V
T

Periphery circuits

155
A. Column scanner
In this estimation, a simple shift register instead of decoder will be used. In each
stage of shift register, the output buffer drives N nMOS transistor (for N-b digital signal
readout, the column selection) in one column. The clock frequency fck is #COL/TROW, and
the input frequency fD is half of the clock frequency. The power consumption of one flip
flop is shown as:
(A.9)
P _ = 2c V f + 3c V f : from 2 transmission gates & 2 inverters, master
+2c V f + 4c V f : from 2 transmission gates & 2 inverters, slave
+7c V f + Nc V f : from clock buffer and output buffers

= 7c V f + 11c + Nc V f

f
= 7c V f + 11c + Nc V
2

#
= 12.5c + 0.5Nc V
T

The total power consumption from the column scanner is:

#
P = 12.5c + 0.5Nc V (A.10)
T

B. Row scanner
The row scanner consists of a shift register and drivers as shown in figure A.4. In

each row, the row scanner consists of one flip flop, three NAND gates, three level-up

converters and three inverters. In the shift register, the input frequency fD is 1/TROW and

the clock frequency fck is #ROW/TROW. The power consumption of one D flip-flop in the

shift register can be expressed:

(A.11)
P _ = 2c V f + 3c V f : from 2 transmission gates & 2 inverters, master
+2c V f + 4c V f : from 2 transmission gates & 2 inverters, slave

156
 +7c V f + 4c V f : from clock buffer and output buffers
#
= 7c V + 15c V

VDD VDDH VDDH

q(n-1) D Q TX
TXG
Q Q
CLK

R
RG
thin-ox

S
SG

Figure A.4 Row scanner

The total power consumption of the #ROW-bit shift register is as follows:

# 1
P = 7c V + 15c V (A.12)
T T

The NAND gate enables the pixel control signals (TX, R, S) according to the external

control signals (TXG, RG, SG). The power consumption of three NAND gates in each

row is as follows:

1
P = 3c V (A.13)
T

Since the pixel power supply is higher than digital power supply (VDD), level-up

converter is required to control the pixel circuit. In one level-up converter,

(A.14)
P = 2c V (1st stage)

P = 2c V (2nd stage)

157
 P = 4c V +# (W c + c )V (driver)

P =P +P +P

The power consumption from level-up converters from all rows can be expressed:

1
P = 3P = 2c V
T
(A.15)
1
+ 6c +# W c +c V
T

The total power consumption from the row scanner is:

P =P +P +P

1
= {7c # + 20c }V
T (A.16)

1
+ 6c +# W c +c V
T

158