Overcomplete Autoencoders with PyTorch
Last Updated :
24 Apr, 2025
Neural networks are used in autoencoders to encode and decode data. They are utilized in many different applications, including data compression, natural language processing, and picture and audio recognition. Autoencoders work by learning a compressed representation of the input data that may be used in a variety of situations.
Overcomplete Autoencoders are a subset of autoencoders that employ a greater number of hidden units than input units. They have demonstrated promise in applications like denoising and feature extraction and have been used to learn complicated, non-linear functions.
Overcomplete Autoencoders
Overcomplete Autoencoders are a type of autoencoders that use more hidden units than the number of input units. This means that the encoder and decoder layers have more units than the input layer. The idea behind using more hidden units is to learn a more complex, non-linear function that can better capture the structure of the data.
Advantages of using Overcomplete Autoencoders include their ability to learn more complex representations of the data, which can be useful for tasks such as feature extraction and denoising. Overcomplete Autoencoders are also more robust to noise and can handle missing data better than traditional autoencoders
However, there are also some disadvantages to using Overcomplete Autoencoders. One of the main disadvantages is that they can be more difficult to train than traditional autoencoders. This is because the extra hidden units can cause overfitting, which can lead to poor performance on new data.
Implementing Overcomplete Autoencoders with PyTorch
To implement Overcomplete Autoencoders with PyTorch, we need to follow several steps:
- Dataset preparation: In order to train the model, we must first prepare the dataset. The loading of the data, it is preliminary processing, and its division into training and test sets are examples of this.
- constructing the architectural model Using PyTorch, we must define the Overcomplete Autoencoder’s architecture. This entails specifying the loss function and the encoder and decoder layers that will be used to train the model.
- Model training: Using the provided dataset, we must train the Overcomplete Autoencoder. The optimization process must be specified, the hyperparameters must be configured, and the model weights must be updated after iterating through the training set of data.
- Evaluation of the model’s performance: When the model has been trained, we must assess how well it performs using the test data. Calculating metrics like the reconstruction error and viewing the model’s output are required for this.
Sure, here’s a step-by-step guide on how to install and implement an Overcomplete Autoencoder with PyTorch:
Step 1: Install PyTorch and Load the required functions
A well-liked deep learning framework called PyTorch offers resources for creating and refining neural networks. Conda, pip, or PyTorch’s source code can all be used for installation. Here is an illustration of using pip to install PyTorch:
pip install torch torchvision
Python3
import torch
import torch.nn as nn
import torchvision.datasets as datasets
import torchvision.transforms as transforms
import torch.optim as optim
from torch.utils.data import DataLoader
import torch.optim as optim
from torch.utils.data import DataLoader
from torchsummary import summary
import matplotlib.pyplot as plt
|
Step 2: Load the Dataset
The MNIST dataset, which consists of grayscale pictures of handwritten numbers, will be used in this example. Using the torchvision library, which offers pre-built datasets and data transformers for popular computer vision datasets, you can obtain the dataset. An illustration of loading the MNIST dataset is provided here:
Python
train_dataset = datasets.MNIST(root='data/',
train=True,
transform=transforms.ToTensor(),
download=True)
test_dataset = datasets.MNIST(root='data/',
train=False,
transform=transforms.ToTensor(),
download=True)
|
Step 3: Define the Model
A particular kind of neural network called an overcomplete autoencoder is made to learn a compressed version of the input data. With an overcomplete autoencoder, there are more hidden units in the encoder than there are input layer units. Due to the network bottleneck caused by this, the encoder is compelled to learn a compressed version of the input data.
Here’s an example of how to define an Overcomplete Autoencoder in PyTorch:
Python
class OvercompleteAutoencoder(nn.Module):
def __init__(self, input_dim, hidden_dim):
super(OvercompleteAutoencoder, self).__init__()
self.encoder = nn.Sequential(
nn.Linear(input_dim, hidden_dim),
nn.ReLU(True),
nn.Linear(hidden_dim, hidden_dim),
nn.ReLU(True),
nn.Linear(hidden_dim, hidden_dim),
nn.ReLU(True),
)
self.decoder = nn.Sequential(
nn.Linear(hidden_dim, hidden_dim),
nn.ReLU(True),
nn.Linear(hidden_dim, hidden_dim),
nn.ReLU(True),
nn.Linear(hidden_dim, input_dim),
nn.Sigmoid()
)
def forward(self, x):
x = self.encoder(x)
x = self.decoder(x)
return x
|
In this example, we define a PyTorch class called Overcomplete Autoencoder that derives from the nn.Module class. Encoder and decoder functions are included in the class, and they are implemented as successive layers of linear and activation functions.
Step 4: Define the Loss function and optimizers
Define a loss function and an optimizer before we can train the Overcomplete Autoencoder. We must We’ll utilize the Adam optimizer and the binary cross-entropy loss function in this case.
Python3
model = OvercompleteAutoencoder(input_dim=784, hidden_dim=1000)
criterion = nn.BCELoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)
|
Step 5: Use GPU if available
Python3
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print('Device is',device)
model.load_state_dict(torch.load(PATH))
model.to(device)
model.to(device)
summary(model, (1,784))
|
Output:
Device is cuda
----------------------------------------------------------------
Layer (type) Output Shape Param #
================================================================
Linear-1 [-1, 1, 1000] 785,000
ReLU-2 [-1, 1, 1000] 0
Linear-3 [-1, 1, 1000] 1,001,000
ReLU-4 [-1, 1, 1000] 0
Linear-5 [-1, 1, 1000] 1,001,000
ReLU-6 [-1, 1, 1000] 0
Linear-7 [-1, 1, 1000] 1,001,000
ReLU-8 [-1, 1, 1000] 0
Linear-9 [-1, 1, 1000] 1,001,000
ReLU-10 [-1, 1, 1000] 0
Linear-11 [-1, 1, 784] 784,784
Sigmoid-12 [-1, 1, 784] 0
================================================================
Total params: 5,573,784
Trainable params: 5,573,784
Non-trainable params: 0
----------------------------------------------------------------
Input size (MB): 0.00
Forward/backward pass size (MB): 0.09
Params size (MB): 21.26
Estimated Total Size (MB): 21.35
----------------------------------------------------------------Step 6: Train the Model
Python
num_epochs = 3
batch_size = 128
train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
for epoch in range(num_epochs):
for batch_idx, (data, _) in enumerate(train_loader):
data=data.to(device)
data = data.view(data.size(0), -1)
optimizer.zero_grad()
recon_data = model(data)
loss = criterion(recon_data, data)
loss.backward()
optimizer.step()
if batch_idx % 100 == 0:
print('Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(epoch,
batch_idx * len(data),
len(train_loader.dataset),
100. * batch_idx / len(train_loader),
loss.item()))
|
Output:
Epoch: 0 [0/60000 (0%)] Loss: 0.000000
Epoch: 0 [12800/60000 (21%)] Loss: 0.000000
Epoch: 0 [25600/60000 (43%)] Loss: 0.000000
Epoch: 0 [38400/60000 (64%)] Loss: 0.000000
Epoch: 0 [51200/60000 (85%)] Loss: 0.000000
Epoch: 1 [0/60000 (0%)] Loss: 0.000000
Epoch: 1 [12800/60000 (21%)] Loss: 0.000000
Epoch: 1 [25600/60000 (43%)] Loss: 0.000000
Epoch: 1 [38400/60000 (64%)] Loss: 0.000000
Epoch: 1 [51200/60000 (85%)] Loss: 0.000000
Epoch: 2 [0/60000 (0%)] Loss: 0.000000
Epoch: 2 [12800/60000 (21%)] Loss: 0.000000
Epoch: 2 [25600/60000 (43%)] Loss: 0.000000
Epoch: 2 [38400/60000 (64%)] Loss: 0.000000
Epoch: 2 [51200/60000 (85%)] Loss: 0.000000
We can begin training the model after defining the optimizer and loss function. We’ll train the model in this example for 10 epochs with a batch size of 128. Every 100 batches, we’ll additionally print the training loss.
Step 7: Evaluate the Model
We may assess the model’s performance using the test dataset after training. In this illustration, we’ll plot some of the input photos and their reconstructed counterparts while also computing the reconstruction loss on the test dataset.
Python
test_loader = DataLoader(test_dataset, batch_size=16, shuffle=False)
model.eval()
with torch.no_grad():
test_loss = 0
for data, _ in test_loader:
data=data.to(device)
data = data.view(data.size(0), -1)
recon_data = model(data)
test_loss += criterion(recon_data, data).item()
test_loss /= len(test_loader.dataset)
print('Test Loss: {:.6f}'.format(test_loss))
data, _ = next(iter(test_loader))
data=data.to(device)
data = data.view(data.size(0), -1)
recon_data = model(data)
fig, axes = plt.subplots(nrows=4, ncols=8, figsize=(16,8))
for i in range(0,4,2):
for j in range(8):
k = k = j if i == 0 else 8+j
axes[i, j].imshow(data[k].cpu().view(28, 28), cmap='gray')
axes[i, j].set_title('Original')
axes[i, j].set_axis_off()
axes[i+1, j].imshow(recon_data[k].cpu().view(28, 28), cmap='gray')
axes[i+1, j].set_title('Reconstruct')
axes[i+1, j].set_axis_off()
fig.tight_layout()
plt.show()
|
Output:
Test Loss: 0.000000
-(4).png)
Output vs Reconstructed image
In this example, we calculate the reconstruction loss on the test dataset and print it out. We also plot some input images and their reconstructions using matplotlib.
Similar Reads
Implement Convolutional Autoencoder in PyTorch with CUDA
Autoencoders are a type of neural network architecture used for unsupervised learning tasks such as data compression, dimensionality reduction, and data denoising. The architecture consists of two main components: an encoder and a decoder. The encoder portion of the network compresses the input data
4 min read
Undercomplete Autoencoder
In the expansive field of machine learning, undercomplete autoencoders have carved out a niche as powerful tools for unsupervised learning, especially in dimensionality reduction and feature extraction. These specialized types of neural networks are designed to compress input data into a lower-dimen
7 min read
Implementing an Autoencoder in PyTorch
Autoencoders are neural networks that learn to compress and reconstruct data. In this guide we’ll walk you through building a simple autoencoder in PyTorch using the MNIST dataset. This approach is useful for image compression, denoising and feature extraction. Implementation of Autoencoder in PyTor
4 min read
ML | AutoEncoder with TensorFlow 2.0
Autoencoders are a type of neural network used for unsupervised learning, particularly for tasks like dimensionality reduction, denoising, and feature extraction. In this article we'll implement a Convolutional Neural Network (CNN)-based autoencoder using TensorFlow and the MNIST dataset Implementin
3 min read
Graph Neural Networks with PyTorch
Graph Neural Networks (GNNs) represent a powerful class of machine learning models tailored for interpreting data described by graphs. This is particularly useful because many real-world structures are networks composed of interconnected elements, such as social networks, molecular structures, and c
4 min read
Implement Deep Autoencoder in PyTorch for Image Reconstruction
Since the availability of staggering amounts of data on the internet, researchers and scientists from industry and academia keep trying to develop more efficient and reliable data transfer modes than the current state-of-the-art methods. Autoencoders are one of the key elements found in recent times
8 min read
Contractive Autoencoder (CAE)
In this article, we will learn about Contractive Autoencoders which come in very handy while extracting features from the images, and how normal autoencoders have been improved to create Contractive Autoencoders. What is Contarctive AutoEncoder?Contractive Autoencoder was proposed by researchers at
5 min read
Computer Vision with PyTorch
PyTorch is a powerful framework applicable to various computer vision tasks. The article aims to enumerate the features and functionalities within the context of computer vision that empower developers to build neural networks and train models. It also demonstrates how PyTorch framework can be utili
6 min read
Autoencoders -Machine Learning
An autoencoder is a type of artificial neural network that learns to represent data in a compressed form and then reconstructs it as closely as possible to the original input. Autoencoders consists of two components: Encoder: This compresses the input into a compact representation and capture the mo
9 min read
How Autoencoders works ?
Autoencoders is a type of neural network used for unsupervised learning particularly for tasks like dimensionality reduction, anomaly detection and feature extraction. It consists of two main parts: an encoder and a decoder. The goal of an autoencoder is to learn a more efficient representation of t
7 min read