Design and FPGA Implementation of Real-Time Automatic Image

Enhancement Algorithm
DONG GuoWei*a, HOU ZuoXuna, TANG Qia, PAN Zhenga, LI Xina
a
Beijing Institute of Space Mechanics & Electricity, Beijing, China 100190

ABSTRACT

In order to improve image processing quality and boost processing rate, this paper proposes an real-time automatic
image enhancement algorithm. It is based on the histogram equalization algorithm and the piecewise linear enhancement
algorithm, and it calculate the relationship of the histogram and the piecewise linear function by analyzing the histogram
distribution for adaptive image enhancement. Furthermore, the corresponding FPGA processing modules are designed to
implement the methods. Especially, the high-performance parallel pipelined technology and inner potential parallel
processing ability of the modules are paid more attention to ensure the real-time processing ability of the complete
system. The simulations and the experimentations show that the algorithm is based on the design and implementation of
FPGA hardware circuit less cost on hardware, high real-time performance, the good processing performance in different
sceneries. The algorithm can effectively improve the image quality, and would have wide prospect on imaging
processing field.
Key words: histogram equalization, piecewise linear, FPGA, parallel structures, pipeline technology

1. INTRODUCTION
In recent years, image enhancement technology is widely used in biomedical, machine vision, aerospace remote
sensing[1]. Obtaining high-quality images is a difficulty among these areas due to several reasons that image quality can
be affected by environment, light, movement and exposure. At the same time, processing the image in real time is more
challenge, because the image resolution and the frame rate are higher. That means the traditional way which enhancing
the image in frequency domain cannot satisfy the requirement of real-time image because of the transform between
different domains.
In real-time automatic image enhancement application, histogram equalization[2-3] and piecewise linear transformation[4-5]
are the common methods. The advantage of histogram equalization is that it can automatically enhance contrast ratio of
the image. However, the effects of enhancement cannot be controlled. This method enhances the targets and also the
background noise at the same time; sometimes gray-scale merger phenomenon may happen. While piecewise linear
transformation is comparatively easier than previous method that it has higher adaptability because it can be any form of
combination. More user inputs required such as determining the segment region and setting scale coefficient to obtain a
better enhancement effect is the defect.
Thus, this paper aims to propose a modified piecewise linear transformation algorithm which utilizes the advantages of
the histogram equalization method and the piecewise linear transform method, and adaptively adjusts the scale
coefficient of the piecewise linear function by analyzing the histogram information. The algorithm is adapted to be used
for the color images, and can be easily implemented by the FPGA. It can effectively improve the image quality and can
be used in real-time systems.

2. ENHANCEMENT ALGORITHM PRINCIPLE

The traditional piecewise linear enhancement algorithms expands certain gray areas by linear transformation function.
The general idea is to expand the visual sensitive areas; while to compress the visual insensitive areas, such as the
background of the image. This method effectively increases the contrast between the extended area, and the compressed
areas for the whole images finally. Its applications include the image contrast adjustment, the foreground objects
enhancement, the background suppression[6] and etc. But there is an important issue in practical applications is that the
coefficient cannot be determined by the algorithm automatically, but it need to be specified manually still.

Selected Papers of the Chinese Society for Optical Engineering Conferences held July 2016,
edited by Yueguang Lv, Weimin Bao, Guangjun Zhang, Proc. of SPIE Vol. 10141, 1014113
© 2016 SPIE · CCC code: 0277-786X/16/$18 · doi: 10.1117/12.2251105

Proc. of SPIE Vol. 10141 1014113-1

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 11/18/2016 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx

 As shown in Figure 1, gray scale (a interval in the histogram) distributions are not equal in the image. Similar, there are
obvious problems that almost all of the pixels locate in the narrow gray scale in many pictures taken in the daily life. The
image qualities of these photos cannot be guaranteed because taking the photos in improper light environments or using
inappropriate exposure times. Therefore, expanding the gray scales that contain more pixels to the wider gray scales and
compressing other gray scales automatically is a valuable ideal. As a result, it can make the pixels uniformly distributing
in the whole image, and can enhance the contrast ratio of the core image area.
104
45
4

36

25
2

IS

05
0

Figure 1. The example image that the pixels locates in a narrow interval.
Based on the above considerations, we consider how to divide the interval, and transform each interval in reasonable
piecewise linear function, its core is to automatically determine the he coefficient of each interval and to find the
relationship between different coefficients.
First, determine the number and pattern of dividing the interval. Simplicity, using the average range method to evenly
dividing the image into a plurality of gray value range. After analysis, more than five ranges can better describe a
picture. Below, we describe the image by five intervals.
The image gray level is from 0 to 255, evenly divided into five sections, respectively [0,51], (51,102], (102,153],
(153,204], (204,255]. If it is a color image, we should make the YUV transform, use the Y component to the make image
enhancement processing, and remain the U and V components. Just use the linear functions to make the transform, and
the formula is:
K1*x x∈ 0,51
K2* x-51 +y1 x∈ 51,102
y= K3* x-102 +y2 x∈(102,153] (1)
K4* x-153 +y3 x∈(153,204]
K5* x-204 +y4 x∈(204,255]
Where y1 = 51 * K1, y2 = 51 * (K1 + K2), y3 = 51 * (K1 + K2 + K3), y4 = 51 * (K1 + K2 + K3 + K4).
Second, determine the coefficients for each interval linear function. There are some coefficients affects the enhancement
algorithm, including the scale factors K1~K5. If we want to make the enhancement automatically, the proportionality
coefficient of the 5 functions ought to be generated automatically.
This paper proposes a method that can generate the K1~K5 automatically. The enhanced images should be in the
character of relatively uniform distribution for the histogram equalization method. As a reference, the scale factor of five
sections of the interval histogram can be determined through calculating the pixel number R1~R5 in the 5 interval of the
histogram. And generating the scale factor K1~K5 by constructing the linear relationship between K and R.
Mathematically, it is assumed between them satisfies the following formula:
Kn=k*Rn+b k＞0, n=1,2…5 (2)
Where the variable k is linear factor, b is the threshold factor. From this formula, if the number of pixels is bigger in a
interval, suppose a larger amount of image information contained in this interval, and should set a larger scale factor
consequentially. If automatic extending the interval according to the computed scale factor, the contrast of this interval is
increased as a result.

Proc. of SPIE Vol. 10141 1014113-2

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 11/18/2016 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx

 Second, it needs to determine the relationship between coefficients. Analysis, there are constraint for the extension, for
example, when the number of pixels in the interval (204,255] is large, stretching only to the low gray value direction;
while stretching only to the high gray value direction for the interval [0,51]. The Compression directions are opposite the
stretching direction. In order to meet the above requirements, this paper presents a constraint, such as (3) shown in
formulas:
K1+ K2+ K3+K4+K5=5 (3)
In addition, the formula (1) shows that the minimum gray value 0 of the image remains 0 after linear transformation.
Integrate function (1) and (3), and ensure the maximum grayscale value 255 is still 255 after the linear transform.
At the same time, the number of the pixels in the 5 interval should satisfy the following constraint:
R1+R2+R3+R4+R5=N=H*V (4)
Where N is the total number of the pixels in the image, H, V is the number of pixels in the image horizontal and vertical
directions respectively. For example, if the resolution of a image is 1920 * 1080, H = 1920, V = 1080, N = 2073600.
The formula (2) into equation (3) obtained:
k*(R1+R2+R3+R4+R5)+5b=5 (5)
The formula (4) into equation (5) obtained:
k=5(1-b)/N=5(1-b)/(H*V) 0≤b<1 (6)
The formula (6) into equation (2) obtained:
Kn=5(1-b)Rn/(H*V)+b 0≤b<1, n=1,2…5 (7)
The formula (7) shows that the threshold factor b is essentially as a parameter to control the enhanced strength, the
smaller the b value is, the greater the intensity enhancement should be. H * V = 1920 * 1080, for example, if b = 0.5,
when Rn = 1920 * 1080, Kn = 3; when Rn = 1920 * 1080/5, Kn = 1; when Rn = 0, Kn = 0.5.
After repeated testing and certification, a reasonable range of the value of b is [0.3, 0.75], and the default value is 0.6.

3. IMPLEMENTATION IN FPGA
The modularization programming method[7] is applied in the design of FPGA to improve the timing and reduce the delay
effectively. And the parallel structures[8] and pipelined technology[9] are used in the realization of these modules.
Figure 2 shows the block diagram of image processing system based on FPGA of a CMOS[10] camera. The automatic
enhancement module which is marked in the grey box is the core of the system.
2DKYU

EbOV
co Ihl2 2DKVIl TmsE Acoso
2CU OL DxinsL DxinsL
yloqns yloqns wognlE

Figure 2. Overall block diagram of the FPGA image processing system.

Figure 3 shows the block diagram of auto enhancement module. The module is divided into three sub-modules based on
their functions. The three sub-modules are information statistic sub-module, piecewise linear transformation sub-module
and delay synchronization sub-module.
The information statistic sub-module realizes the histogram statistics of the image luminance data and calculation of the
ratio coefficient. The piecewise linear transformation sub-module obtains the piecewise function according to the above
ratio coefficient, and then adjusts the gray value of the input image pixels linearly. The delay synchronization
sub-module delays the image chrome data and synchronizes it with the output image data.

Proc. of SPIE Vol. 10141 1014113-3

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 11/18/2016 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx

 CFLozu
moqnv

moqnif

HP-PRxs

Figure 3. Block diagram of the automatic image enhancement module.

Because the data of whole image needs to be analyzed and processed when the information statistic sub-module and the
piecewise linear transformation sub-module work, there is hysteresis which means the ratio coefficient used in this image
is obtained based on the last image. To solve this problem, the information statistic sub-module is placed close to the
CMOS drive module in the front-end data stream. After this adjustment, the ratio coefficient are computed with the
image data is stored into the SDRAM. And the ratio coefficient is obtained when the image data is read out from the
SDRAM. Figure 4 shows the practical structure of the automatically image enhancement module.
ZDKY]

bbC
wogn1'
moqnjE
moqn¡(

CIAIO2
Dxrnsz.
woqns CgIcnP CgIcnP
bI6C6N
Trusn

Figure 4. Practical structure of the automatically image enhancement module.

In addition, the function of Y component transformation is required to convert the Bayer image into luminance image
because the output format of CMOS drive module is in Bayer format not YUV format.
The synchronization design is applied in the implementation of circuit[11]. The system clock is 80Mhz and the data is
sampled on the rising edge of the clock.
In the information statistic sub-module, parallel structure and pipeline technology are used in the design of circuit and
the Y component transformation circuit converts the Bayer image into Y component data. The three color components
which are R, G and B are separated and computed in parallel. Because Y component transformation involves the real
time multiplication, the shift operation is used to simplify the design and improve the accuracy. A dedicated four stage
pipeline multiplier is designed in this paper to realize the multiplication by shift and addition operation. This design can
save hardware resources. The function of the histogram statistics circuit is to count the distribution number of the gray
value of the Y component in such five sections [0,51], (51,102], (102,153], (153,204] and (204,255]. The circuit with
parallel structure calculates the statistical histogram in each section and the number of pixels of each section R1~R5. The
implementation in FPGA can generate big combinational logic blocks and have a bad effect on system timing[12] because
the high bit wide and real-time accumulation and other operation. To reduce latency, the pipeline technology is used to
divide the high bit wide accumulator into many low bit wide ones. The ratio coefficient calculation circuit realizes the
operation in equation (7) and gets five coefficients K1~K5 based on R1~R5. To avoid floating-point operation caused by
the decimal threshold factor, the factor 1-b is left shift 10 bits and converts into 1024~1024×b. It means that the
magnification of the calculated K1~K5 is 1023 times. The threshold factor is the parameter of enhanced strength and can
be adjusted according to the user demand. The default value is 0.6.

Proc. of SPIE Vol. 10141 1014113-4

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 11/18/2016 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx

 The piecewise linear transformation sub-module realizes the five segment linear function in equation (1). The image data
is divided into five sections and each section corresponds to a linear function. The transformation of image data in each
section is parallel. To improve the character of real-time of the system, multi-level pipeline structure is applied. Finally
the linear-transformed data of five sections merge into one output.
The delay synchronization sub-module caches and delays the image chrome data and implements the synchronous output
of the delayed chrome data and the piecewise linear-transformed intensity data.

4. EXPERIMENTAL RESULT
The automatic enhancement algorithm is applied to a certain type of visible light CMOS camera in the laboratory, and
the test is carried out. PC software is developed by VC. The video display is achieved by calling the API function to
operate the image acquisition card .In order to verify the robustness of the algorithm, multiple tests were performed on
different types of images. Below is an example of one of the examples for analysis.
Figure 5 shows the contrast enhancement effect under different threshold factors (b=0.75, b=0.6 and b=0.33). Figure
5.(a) shows the original image and enhanced image. Figure 5.(b) is the histogram of each sub graph in Figure 5.(a). It can
be seen that the original image is dark. A lot of detail information is concentrated in the gray value of [20,80] interval
which is not suitable for human eye to identify. The enhanced image becomes bright and the details are clear. The
enhancement intensity becomes larger with the decrease of the b value of the enhancement coefficient.
It can be seen from the histogram, the area with large amount of information is stretched, the area with less information
is compressed. For example, the gray value of the detail information in Figure 5.(b)(4) is in [50,160] interval. It is 2
times larger than the original image whose gray value is in [20,80] interval. In Figure 5.(b)(4), the gray value range of
[100,220] which has little information in the original image is compressed to [170,220] range. The area is compressed by
1/2.

(a)

x 104

2 15000
15000 15000
1.5

JIL
10000
10000 10000
(b) 1

5000 5000 5000

0.5

o L o 0 0
0 100 200 0 100 200 0 100 200 100 200
(1) (2) (3) (4)
Figure 5. Processing effect of the proposed algorithm with different threshold factor b: (a) image, (b) histogram, (1)original
image, (2) b=0.75, (3) b=0.6, (4) b=0.33.
Table 1 is the histogram statistical tables for the five segments of the original image. R1~R5 represents the number of Y
component pixel in five segments. Table 2 gives the K value of the corresponding interval when the b value is different.
Table 1. The pixel numbers in five intervals of the histogram.
R1 R2 R3 R4 R5
Original image 1459960 483298 28678 12708 88956

Table 2. The values of the 5 scale coefficient with different threshold factor b.
K1 K2 K3 K4 K5
b=0.75 1.630 1.041 0.768 0.758 0.804
b=0.6 2.008 1.066 0.628 0.612 0.686
b=0.33 2.688 1.111 0.377 0.350 0.474

Proc. of SPIE Vol. 10141 1014113-5

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 11/18/2016 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx

 It can be seen from the table above, the ratio coefficient K1, K2 value increases when the R1, R2 value increases. The
scale coefficient K3, K4, K5 value decrease when the R3, R4 value decreases. When the ratio coefficient K1 and K2 are
larger than 1, the range of the image is stretched. When the scale coefficient K1 and K2 are smaller than 1, the range of
the image is compressed.
It shows enhanced image contrast before and after in Figure 6, when b is fixed in 0.6. For the original image, the image
in Figure 6.(a)(1) is too dark. Figure 6.(b)(1) is slightly dark, and Figure 6.(c)(1) is a little bright, Figure 6.(d)(1) is too
bright. After the image enhancement, the image is moderate, the regional contrast of the area with rich details is
increased significantly, the algorithm in this paper can not only enhance the partial dark image processing, but also can
reduce the brightness of bright image.

Ir

A
..

A
s

(1) (2) (3) (4)

Figure 6. Processing effects for different images (b = 0.6): (1) original image, (2) enhanced image, (3) histogram of original
image, (4) histogram of enhanced image.

5. CONCLUSION
In this paper, a piecewise linear enhancement algorithm is proposed, which can generate the ratio coefficient of five
sections automatically. It can enhance the color image automatically. The idea of histogram equalization and the
flexibility of piecewise linear transformation are comprehensively utilized. And then the above algorithm is implemented
by FPGA. In order to meet the real-time requirement of the system, the high efficiency parallel pipeline technique is
considered and the potential parallelism of the hardware modules is considered. The algorithm is easy to implement and
has strong adaptability. It can effectively improve the image information and image quality, and make the information
rich area fully displayed.

Proc. of SPIE Vol. 10141 1014113-6

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 11/18/2016 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx

 REFERENCES

[1] Wang, J. and Li, G. H., "Application of CMOS Image Sensor in Spacescraft Remote Sensing," Spacecraft
Recovery & Remote sensing. Papers 29(2): 42-47 (2008).
[2] Cao, J. L., Lv, H. B. and Li, G. Z., "Histogram equalization algorithm based on adaptive local gray level
modification, " Infrared and Laser Engineering. Papers 33(5): 513-515 (2004).
[3] Cheng, H. D. and Shi, X. J., "A simple and effective histogram equalization approach to image enhancement,"
Digital Signal Processing. Papers 14(2): 158-170 (2004).
[4] Chen, M. L., Chen, C. X. and Gu, J. P., "A piecewise linear transform based adaptive histogram method,"
Foreign Electronic Measurement Technology. Papers (2): 36-38 (2015).
[5] Zhang, J. H., Yang, G. and Xu, Q., "Based on piecewise linear transformation of image enhancement," National
Conference on Image and Graphics. (2008).
[6] Li, X. B., "An infrared measurement image adaptive piecewise linear gray transform method," Optoelectronic
Technology. Papers 31(4): 236-239 (2011).
[7] Wang, J. H., Liu, C. L., Chen, D. C., et al, "Design of Real-time Video Processing System Based on
DSP+FPGA," Foreign Electronic Measurement Technology. Papers 26(9): 42-44 (2007).
[8] Diaz, J., Ros, E., Pelayo, F., et al, "FPGA Based Real-time Optical-flow System," IEEE Transactions on
Circuits & Systems for Video Technology. Papers 16(2): 274-279 (2006).
[9] Genovese, M. and Napoli, E., "ASIC and FPGA Implementation of the Gaussian Mixture Model Algorithm for
Real-time Segmentation of High Definition Video," IEEE Transactions on Very Large Scale Integration
Systems. Papers 22(3): 537-547 (2013).
[10] Yang, H. L., Wang, J. and Yang, H. W., "Design of High Frame Frequency CMOS Real-time Image
Acquisition Systerm," Microcomputer Information. Papers 24(30): 309-311 (2008).
[11] Chen, B. W., Liang, Z. Y., Wang, Y. X., et a1, "Design of a High Speed CMOS Imaging System Based on
FPGA," Computer Measurement & Control. Papers 20(5): 1397-1400 (2012).
[12] Chen, G. Z., Liu, T. and Li, Y. Z., "Design of Image Sampling and Storage System Based on FPGA,"
Mechanical Engineering & Automation. Papers (4): 44-46 (2007).

Proc. of SPIE Vol. 10141 1014113-7

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 11/18/2016 Terms of Use: http://spiedigitallibrary.org/ss/termsofuse.aspx