Attention Mechanisms in Convolutional Neural Networks (CNNs)
Attention mechanisms improve CNNs by helping them focus on the most
important parts of an image, similar to how humans concentrate on specific
parts of a scene. This makes CNNs more effective at tasks like image
classification, object detection, and segmentation.
A. What is Attention in CNNs?
Attention in CNNs mimics the human ability to focus on key areas in an image.
For example, when identifying a person, the model pays more attention to the
face than the background.
B. Why Use Attention in CNNs?
Selective Focus: Helps the model prioritize important parts of the image.
Handle Noise: Reduces the impact of irrelevant or noisy image areas.
Capture Global Context: Goes beyond local details to understand the full
image.
Improves Interpretability: Highlights areas that influenced predictions.
C. Types of Attention Mechanisms
Channel Attention: Focuses on important feature channels (e.g.,
Squeeze-and-Excitation Block).
Spatial Attention: Focuses on key image regions (e.g., PSANet).
Hybrid Attention: Combines both channel and spatial attention (e.g., CBAM).
D. How Attention Works in CNNs
Extract Features: CNN processes the image to generate feature maps.
Generate Attention Weights: Identifies important regions or channels.
Recalibrate Features: Adjusts feature maps using these weights.
Predict: Refined features are used for the final task (e.g., classification).
1. Squeeze-and-Excitation (SE) Block
The SE block introduces channel-wise attention by recalibrating channel
importance. Here's a step-by-step breakdown:
Global Average Pooling (GAP): For each channel in the feature map, GAP
calculates the average value across all spatial positions.
This reduces the feature map size from 𝐶 × 𝐻 × 𝑊 to 𝐶, summarizing spatial
information for each channel.
Fully Connected (FC) Layer with Reduction: The GAP output
(𝐶-dimensional vector) is fed into an FC layer that reduces the number of
channels by a reduction ratio 𝑟 (e.g., 𝑟 = 16). This step compresses information,
forcing the model to focus on essential features.
ReLU Activation: Adds non-linearity to help the model learn complex channel
dependencies.
FC Layer to Restore Dimensions: A second FC layer restores the reduced
channel count back to the original size 𝐶.
Sigmoid Activation: Converts the output into attention weights between 0 and
1 for each channel.
Rescaling: The original feature map is multiplied by these weights
channel-wise, enhancing important channels and suppressing irrelevant ones.
2. Efficient Channel Attention (ECA-Net)
ECA simplifies channel attention by replacing FC layers with a 1D convolution,
making it lightweight and efficient.
Global Average Pooling (GAP): Summarizes spatial information for each
channel, similar to SE.
Adaptive Kernel Size: ECA avoids FC layers by applying a 1D convolution
along the channel dimension. The kernel size 𝑘 for the convolution is adaptively
determined based on the number of channels 𝐶 : 𝑘 = 𝜓(𝐶) where 𝜓(𝐶) is a
function (e.g., logarithmic) that ensures scalability.
1D Convolution: This lightweight operation captures channel dependencies
efficiently without increasing model complexity.
Sigmoid Activation and Rescaling: The output of the 1D convolution is passed
through a sigmoid function, generating channel-wise attention weights.
These weights are applied to the input feature map for channel refinement.
3. Point-wise Spatial Attention (PSANet)
PSANet emphasizes spatial attention by considering relationships between all
points in the feature map.
Feature Map Reduction: The input feature map (𝐶×𝐻×𝑊) is reduced to a
smaller size (𝐶′×𝐻×𝑊) using a convolutional layer. This makes subsequent
computations more efficient.
Collect and Distribute Attention: The reduced feature map is split into two
streams:
● Collect Attention: Determines how much attention each pixel collects
from the entire image.
● Distribute Attention: Determines how much attention each pixel
distributes to other pixels.
Attention Map Generation: Both streams generate attention maps (𝐻×𝑊×
(2𝐻+1) × (2𝑊 +1)) using convolution and non-linear activations.
Feature Refinement: Attention maps are applied to the feature maps,
enhancing pixels based on their global and local importance.
Concatenation and Projection: The refined feature maps are combined and
projected back to match the original input dimensions.
Convolutional Attention Block Module (CBAM) :-
The Convolutional Attention Block (CAB) enhances the learning capability of
deep neural networks by integrating attention mechanisms with convolutional
layers. The primary goal of CAB is to allow the network to focus on the most
relevant features in the input data, such as important spatial regions or specific
channels in a feature map, while ignoring less important details.
The CAB starts with an input feature map 𝑋∈𝑅^(𝐶×𝐻×𝑊), where 𝐶 is the
number of channels, and 𝐻 and 𝑊 are the spatial dimensions (height and width).
The module applies attention mechanisms to determine which features are most
important. There are two common types of attention:
Channel Attention focuses on identifying the most important channels. The
input feature map is processed using global pooling operations, such as average
pooling or max pooling, to summarize the information in each channel.
This summarized information is passed through small fully connected layers
with activation functions to calculate the importance (or attention) of each
channel.
The computed attention values are used to scale the original feature map,
enhancing significant channels and reducing less important ones. By focusing
on the most important channels, CAM improves the representation of key
features in the data.
The Spatial Attention Module (SAM) emphasizes the significance of specific
spatial regions within the feature map, such as areas corresponding to objects or
patterns in an image. SAM highlights the critical spatial locations where
features are most relevant.
The input feature map is pooled along the channel dimension using average
pooling and max pooling to create two spatial maps that summarize information
across all channels. These spatial maps are concatenated and processed through
a convolutional layer to generate an attention map. The generated attention map
is applied to the input feature map, emphasizing important spatial regions and
downplaying less relevant ones. SAM ensures that the network focuses on the
most crucial areas within an image, improving spatial feature extraction.
