Laplacian based image fusion

Jesse Scott and Michael A. Pusateri

Integrated Design Services, College of Engineering
The Pennsylvania State University
University Park, PA, USA
[email protected], [email protected]

Abstract — A fundamental goal in multispectral image fusion is A. Data versus Information

to combine relevant information from multiple spectral ranges
When thinking about the fusion process, one must consider
while displaying a constant amount of data as a single channel.
how to evaluate the results of any fusion. When discussing
Because we expect synergy between the views afforded by
different parts of the spectrum, producing output imagery with
image fusion terminology, the word “data” refers to the actual
increased information beyond any of the individual imagery pixels of a single frame while the word “information” refers to
sounds simple. the value of the data of that frame. The goal, when fusing
imagery either indirectly or directly, is to reduce the total
While fusion algorithms achieve synergy under specific scenarios, amount of data, increase the information contained in the data,
it is often the case that they produce imagery with less or both simultaneously. This fusion efficiency can be
information than any single band of imagery. Losses can arise represented as a ratio of information divided by data, meaning
from any number of problems including poor imagery in one that successful fusion is intended to increase the ratio. In the
band degrading the fusion result, loss of details from intrinsic specific case of real-time image fusion, the amount of data per
smoothing, artifacts or discontinuities from discrete mixing, and image is inflexible since the image resolution is normally fixed
distracting colors from unnatural color mapping. by the resolution of the display.
We have been developing and testing fusion algorithms with the B. Problem Statement
goal of achieving synergy under a wider range of scenarios. This While increasing the information in an output image sounds
technique has been very successful in the worlds of image simple at first, many issues arise from mixing multiple
blending, mosaics, and image compositing for visible band channels. The simple explanation is that if two channels of
imagery. The algorithm presented in this paper is based on information are worth x and y respectively, then a two input
direct pixel-wise fusion that merges the directional discrete fusion should create an output with total information between
laplacian content of individual imagery bands rather than the
max(x, y) and sum(x, y). However, the outcome of many
intensities directly. The laplacian captures the local difference in
the four-connected neighborhood. The laplacian of each image is
fusion techniques results in less than max(x, y) and fusion will
then mixed based on the premise that image edges contain the likely never reach the sum(x, y) bound. Less information after
most pertinent information from each input image. This fusion derives from losing information in each channel or
information is then reformed into an image by solving the two- creating a distracting output as the result of the mixing process.
dimensional Poisson equation. These losses can be caused by loss of details from intrinsic
smoothing, artifacts/discontinuities from discrete mixing, or
The preliminary results are promising and consistent. When even distracting colors from unnatural color mapping.
fusing multiple continuous visible channels, the resulting image is
similar to grayscale imaging over all of the visible channels. C. Target System Limitations
When fusing discontinuous and/or non-visible channels, the In our research, the fusion of multiple sensor channels has
resulting image is subtly mixed and intuitive to understand. restrictions that limit the viable methods. Our system is a true
real-time system and can only have latencies equal to rows of
Keywords - Laplacian; image fusion; multi-spectral; non- the image. The fusion must not only result in information
linear; Poisson greater than or equal to max(x, y), but must also accommodate
input imagery that is improperly exposed. The final restriction
I. INTRODUCTION is the system must not distract the user since the system is a
The fundamental goal in multisensor image fusion is to multispectral night vision goggle used as vision replacement
combine relevant information from multiple sources while for military operators. With these restrictions, the resulting
displaying the same amount of data as a single channel. The fusion method must mix multiple wavelengths to a single
fusion result should contain more information than any single image in real-time while always maintaining or increasing the
input channel. Image fusion operates at a pixel or local region situational awareness available in any sensor independently.
level and is broadly categorized into two domains: direct data
fusion and indirect metadata fusion [1]. Both categories have II. BACKGROUND
unique merits which can be applied independently and A. Existing Fusion Techniques
simultaneously. The technique of data fusion has a huge
variety of approaches that vary from simple channel Many image fusion techniques that currently exist primarily
assignments to spatial frequency transforms. attempt to mix the intensities of the input imagery in some
fashion and generate an output image with information from directly. By comparison, the Laplacian captures the local
the input sources. This fusion can be as simple as assigning the difference in the four-connected neighborhood where this
input channels to color channels to less simple alpha blending method captures the same information, but separated for each
techniques generating an output grayscale image to complex of the connected neighbors illustrated in Figure 2 and Figure 3.
methods that use neural network processing [2]. The directional Laplacian components of each image are then
mixed based on the premise that maximum image edge
There is a category of image fusion methods that does not intensity contains the most pertinent information from any of
manipulate the pixel intensities directly, but manipulates a the input images. This information is then reformed into an
transformed version of the image intensities. There is a large image by solving the two-dimensional Poisson equation that
volume of literature that investigates transform-based fusion basically solves a properly constrained two-dimensional
techniques [3][4][5]. There has also been some literature integration. The pseudo code for the method is shown in
working at evaluation techniques to compare fusion methods of Figure 1.
any sort [6][7][8].
function [imNew] = Laplacian_Fusion(imStack)
B. Translation-based Image Editing [height width] = size(imStack);
Gradients and derivatives are used in many applications n = height*width;
including High Dynamic Range (HDR) compression [9], image %% Construct the sparse matrix
stitching [10], image editing [11], and noise reduction [12]. All R = n long array of ones
of these areas of application have a common thread of image G = cat(2, R*-1, R*-1, R*4, R*-1, R*-1);
quality as the utmost important aspect of the process. On the for each width step
opposite side, each of these applications does not have the G(i, 2) = 0;
extreme time constraints that a real-time system requires. The G(i+1, 4) = 0;
basic premise in each of the applications is to convert the d = [-(width) -1 0 1 (width)];
image(s) into derivatives, manipulate the derivative image(s), M = create a diagonal nxn matrix from G and d
%% Construct the gradient field matrix
and convert back to an intensity image.
% calculate differences of each pixel
C. Current State-of-the-Art for each of imStack layers
for each of 4-neighbors directions
Using the current sophistication of the devices fielded by differences = imfilter(imStack,
the armed forces as a measure, very few have multiple difference_kernel);
wavelength imagers integrated. Of these multi-sensor systems, % select maximum magnitude of gradient
most are analog systems that have only two methods of fusion: for each 4-neighbor directions
different color phosphors referred to as color assignment fusion max = max(differences);
(CAF), and voltage averaging referred to as A+B fusion. CAF % sum 4-neighbor directions and accumulate with initial conditions
typically uses amber phosphor for longer wavelengths and B = max_left + max_right + max_top + max_bottom;
green phosphor for shorter wavelengths while keeping the %% Solve linear system of equations and format result
analog signals separate. A+B fusion adds the voltages and vecNew = M\B';
scales the result by half then displays the result as a imNew = shape vecNew into an image based on height and width
monochrome image.
Figure 1 - Pseudo code for proposed fusion method
The coming generation of digital fusion goggles (DFGs)
has a completely different outlook on mixing the multiple data The process can be summarized into a three-step process of
streams. These streams now can be mixed by mathematical constructing the sparse matrix, constructing the gradient field
manipulation via pixel-by-pixel, local region, and/or global matrix with initial conditions, and solving the matrix equation.
analysis. However, the results generally are related back to the The construction of the sparse matrix is reasonably straight
original A+B and CAF techniques. The mathematical equation forward given a known edge detection kernel; it is not
of extracting all the information from both channels and dependent on the image content, but dependent on image
summing them together for display is a long term goal, but dimensions that are constant in most implementations including
does not account for the overwhelming influence of meeting this design.
the user’s expectations. Since most users have been exposed to The construction of the gradient field is more difficult, but
the typical analog techniques, we must provide them with these still relatively straight forward. The gradient field is composed
fusion methods in addition to any other methods that are of initial conditions and gradient content. The gradient content
developed. This addition facilitates transition to new methods, can be determined with simple kernel-based filters of both
but also causes the problem of choosing the method based on images, but the initial conditions are a bit more complicated
familiarity and not tactical advantage. because of the source of the information. The initial conditions
are a single-wide pixel border where each pixel is used as the
III. PROPOSED METHOD starting point of the integration. If the initial condition border
The primary technique of derivative-based editing has been is selected from the image area of any of the inputs, then the
very successful in the worlds of image blending, mosaics, and selection of the data has a significant impact on the solution
image compositing. In this research, we focus on a direct since the initial conditions work as the basis for the result
pixel-wise fusion method that merges the local directional image. The initial condition source image gives the result
discrete difference content of images rather than the intensities image a basis to which the other image information is added.
A. Constructing the Sparse Matrix used as a positively correlated representation of image
Construction the sparse matrix is a simple task that is only information. The results of each of the four max operations are
related to the kernel used for filtering and the width of the input then summed to provide a composite derivative value. This
images. The layout is a simple diagonal matrix with the center value is directly related in a non-linear fashion to the sum of
value of the kernel on the primary diagonal axis and the four- the maximum of the Laplacian of the image.
connected neighbor value on the four adjacent diagonal axes.
Since image border behavior causes kernel truncation, these 0 -1 0
missing values must be mirrored in the diagonal matrix
structure as well.
-1 4 -1
0 0 0 0 -1 0
0 -1 0
0 1 -1 0 1 0
Figure 3 – Laplacian composite of directional kernels

0 0 0 0 0 0 C. Solving the Matrix

The process of solving the properly constrained matrix
equation is shown in Figure 1. We use the direct matrix solver
0 0 0 0 0 0 to get an exact solution. There are multiple iterative methods
that attempt to minimize the total error at the initial condition
locations. The exact method is fast, but consumes significant
-1 1 0 0 1 0 PC resources while the approximation is significantly slower
yet uses substantially less resources. As a reminder, the matrix
is the square of the pixel count of the images being fused.
0 0 0 0 -1 0
IV. RESULTS
Figure 2 – Kernels for local directional differences A. Example 1
Figure 5 is an example of Laplacian fusion of a standard
B. Gradient Field Formation
color image of flowers. The picture in Figure 4 was taken with
The gradient field is formed by combining the directional a standard point-and-shoot still camera and therefore has
gradients from each of the input images. Figure 2 shows the reasonably good exposure in all three bands. All three images
kernels used to extract the directional local differences. For have an ideal spatial alignment because of the inherent nature
each image from top-left to bottom-right are the right, top, left, of the acquisition system.
and bottom differences. If all kernels are summed the result is
the Laplacian kernel shown in Figure 3. The processing for the Laplacian fusion image in Figure 5
is actually a two-step process: first fusion of red and green and
The images are fused by mixing the results of the kernel then the fusion of the first result with the blue channel. The
filtering for each of the four directions. The mixing is done image on the right in Figure 5 represents the standard fusion of
using the max operator, giving the technique a non-linear RGB imagery using the MATLAB color to grayscale
behavior. The selection of the maximum gradient is intended conversion.
to capture the maximum information since the gradient is being

Figure 4 - Example 1 Originals - Color, Red, Green, Blue (Left to right)

 Figure 5 - Example 1 Results - Laplacian Fusion (Left) and MATLAB’s rgb2gray (Right)
Upon inspection of the output from both the standard fusion Figure 6 is intended to illustrate the fusion of imagery that
and Laplacian fusion, the basic similarity between the two is properly exposed in wavelengths outside the standard visible
should be the first and most obvious characteristic. The images bands. In the fusion image of Figure 6, note that the computer
are very similar with the same textures and general content. A monitor and desk lamps contain increased information from the
more detailed visual analysis will bring up many minor LWIR content than the rest of the scene. The NIR imagery
differences. The oversaturated part of the front flower bundle dominates the content of the Laplacian since the NIR source
has more detail in the Laplacian results than the standard image has significantly more patterns and textures than the
results. However, the Laplacian results are lacking the details LWIR source image. The only location where the LWIR input
on the inside of the vase that the standard results retain. dominates is the PC in the middle of the scene. Having human
Overall, the general similarities are a strong indicator of the vision their entire lives, the NIR channel dominance is
capability of this technique to maintain or improve image generally preferred by users based on their familiarity with
fusion without causing degraded results. visible imagery. The emissive, rather than reflective, nature of
the LWIR waveband results in imagery that is mostly uniform
B. Example 2 and has minimal, but strong gradient content.
The bottom of Figure 6 is an example of Laplacian fusion
of a registered multispectral image pair. The top left image in C. Example 3
Figure 6 is a near-infrared (NIR) frame and the top right image The bottom of Figure 7 is an example of Laplacian fusion
in Figure 6 is a long-wave infrared (LWIR) frame; both of a registered multispectral image pair. The top left image in
captured from a video sequence. The scene is of an indoor Figure 7 is a visible frame and the top right image is a thermal
industrial space. The scene includes a collection of objects that frame; both captured from a video sequence. The scene is of
are detected in both wavebands. The imagery is not spatially an outdoor view from a rollup garage door opening to a distant
aligned perfectly, but the affine warp that was applied to the scene with a vehicle in the foreground. It is captured using the
LWIR image is sufficient. There are some alignment issues same equipment setup as Example 2 and is subject to the same
that are actually the result of optical distortion caused by the image problems as a result. This example is intended to
use of different optics for each sensor and resolution issues illustrate the fusion of imagery that is improperly exposed in
with the LWIR image since it was captured at a quarter of the various parts of both channels. The scene has at least one
resolution of the NIR channel.
 Figure 6 - Example 2 Results – NIR (Top Left), LWIR (Top Right), and Laplacian Fusion (Bottom)

useful pixel at every pixel index. In the fusion image, the This type of image is representative of the large dynamic range
visible band image provides the foreground car and interior of that occurs when looking at a transition between sunlit outdoor
the garage while seamlessly transitioning between the thermal and electrically lit indoor content for NIR imagery.
content in the background of the outdoor portion of the scene.
 Figure 7 - Example 3 Results – NIR (Top Left), LWIR (Top Right), and Laplacian Fusion (Bottom)
channels results in imagery similar to grayscale imaging over
V. ANALYSIS all of the visible channels. The fusion result has increased
The preliminary results are predictable and consistent. As detail in multiple regions, but generally has the same level of
shown in Example 1, the fusion of multiple visible spectral detail as the MATLAB grayscale result.
 Table 1 – Execution times for each example image set
Example Matrix Dimensions Building Sparse Matrix Building gradient field Solving linear system
(pixels) (sec) (sec) (sec)
1 193929 x 193929 0.677151 0.373219 1.536592
2 303849 x 303849 0.956812 0.398306 2.627780
3 1784322 x 1784322 5.831647 1.780139 24.698355
When fusing discontinuous and/or non-visible channels, the The current method for solving the Poisson equation is not
resulting image is subtly mixed and intuitive to understand. acceptable for FPGA implementation in a real-time system.
The technique is capable of seamlessly fusing imagery that has The numerical manipulation requires the entire image to be
regions that are improperly exposed or washed out while not formed into a large matrix which would result in a minimum of
allowing this to affect the output image. Example 2 and one frame of latency. Approximations to the integration of the
Example 3 illustrate very common and reoccurring situations derivative image must be developed to make this fusion
where multiple parts of the image fail to capture information approach viable for night vision goggles.
efficiently. However, the idea that there is always some
information in an image is important to the premise that image REFERENCES
fusion is valuable as a mechanism to enhance information [1] Wang, Z.; Ziou, D.; Armenakis, C.; Li, D.; Li, Q.; "A comparative
density in images. It should be clear from visual analysis that analysis of image fusion methods," IEEE Transactions on Geoscience
the results in Figure 6 show that the small amount of and Remote Sensing, vol.43, no.6, pp. 1391-1402, June 2005. doi:
10.1109/TGRS.2005.846874.
information in the LWIR channel was integrated in to the
[2] Fay, D.A. et al; "Fusion of multi-sensor imagery for night vision: color
visible image even though there was sufficient information in visualization, target learning and search," Proceedings of the Third
the NIR channel and only slightly more information in the International Conference on Information Fusion, 2000. FUSION 2000.
LWIR channel. vol.1, 10-13 July 2000. doi: 10.1109/IFIC.2000.862702.
[3] Waxman, A.M. et al; “Solid-state color night vision: fusion of low-light
The Poisson solver is not a low latency mathematical visible and thermal infrared imagery,” MIT Lincoln Laboratory Journal,
method, but approximation techniques are currently being vol.11, pp.41-60, 1998.
reviewed that are low latency. All other mathematics in this [4] Hui Li; Manjunath, B.S.; Mitra, S.K.; "Multi-sensor image fusion using
design have clear implementations that are low latency with the wavelet transform," IEEE International Conference on Image
exact calculations. The MATLAB PC execution times for each Processing, 1994. ICIP 1994, vol.1, pp.51-55. Nov 13-16 1994. doi:
example are shown in Table 1. 10.1109/ICIP.1994.413273.
[5] Wen, J.; Li, Y.; Gong, H.; “Remote Sensing Image Fusion on Gradient
VI. CONCLUSIONS Field,” Proceedings of the 18th international Conference on Pattern
Recognition. ICPR 2006. vol.3, pp.643-646. Washington, DC, Aug. 20-
The proposed application of the Laplacian blending method 24, 2006. doi: 10.1109/ICPR.2006.995.
appears to provide excellent performance under a variety of [6] Chen, Y.; Blum, R.S.; "Experimental tests of image fusion for night
conditions. The method automatically adapts between the vision, "8th International Conference on Information Fusion, 2005.
scene content of the input images while adding information vol.1, pp. 8, 25-28 July 2005. doi: 10.1109/ICIF.2005.1591895.
from one and/or the other image depending on the Laplacian [7] Waxman, A.; Fay, D.; Ilardi, P.; Savoye, D.; Biehl, R.; Grau, D.; "Sensor
Fused Night Vision: Assessing Image Quality in the Lab and in the
content. Field," 9th International Conference on Information Fusion, 2006, pp.1-
As imaging moves to longer wavelengths, the photon 8, 10-13 July 2006. doi: 10.1109/ICIF.2006.301767.
measurement is dominated more by emitted rather than [8] Petrovic, V.; Xydeas, C.; "Objective image fusion performance
characterization," Tenth IEEE International Conference on Computer
reflected photons. Laplacian fusion is dependent on the basic Vision, 2005. ICCV 2005. vol.2, pp.1866-1871. Oct. 17-21, 2005. doi:
premise that information is stored in the derivative of the 10.1109/ICCV.2005.175.
image. While this dependence is somewhat obvious for visible [9] Fattal, R.; Lischinski, D.; Werman, M.; “Gradient domain high dynamic
wavelengths, it is not as clear for longer wavelengths. range compression,” Proceedings of the 29th Annual Conference on
Emissive imagers may not store their information in the Computer Graphics and interactive Techniques. SIGGRAPH 2002. San
derivatives at the same level as the reflective imagers. As a Antonio, Texas, July 23-26, 2002. New York, NY, pp.249-256. doi:
10.1145/566570.566573.
result, the Laplacian fusion technique may mix channels with
unequal weighting. [10] Levin, A.; Zomet, A.; Peleg, S.; Weiss, Y.; “Seamless image stitching in
the gradient domain,” European Conference on Computer Vision. ECCV
2004. pp.377-389. 2004.
VII. FUTURE WORK
[11] Pérez, P.; Gangnet, M.; Blake, A.; “Poisson image editing.” ACM
This technique relies on boundary conditions to provide Transactions on Graphics. Vol.22, Iss.3, pp313-318, July 2003. doi:
initial conditions in the Poisson equation. To accomplish full 10.1145/882262.882269.
image laplacian fusion, boundary conditions must be [12] Jong Hyun Park; Moon Gi Kang; "Noise reduction for gradient-based
eliminated, made constant, or the effects must be mitigated. multispectral image fusion," Proceedings of International Symposium on
Intelligent Signal Processing and Communication Systems, 2004.
Currently, the implementation uses a single pixel border to ISPACS 2004., pp.185-188, Nov. 18-19, 2004. doi:
create a boundary condition. Each input channel produces a 10.1109/ISPACS.2004.1439042.
different visual result primarily at the boundary.