What is Embedding Layer ?

The embedding layer converts high-dimensional or categorical data into lower-dimensional numerical vectors that capture meaningful patterns and relationships. It helps machine learning models process complex data more efficiently and is commonly used in tasks such as Natural Language Processing (NLP) and recommendation systems.

Working

Input Representation: Suppose we have a sentence containing words such as "cat", "dog" and "apple". Each word is assigned a unique integer ID based on the vocabulary.
Embedding Mapping: The embedding layer maps each word ID to a dense numerical vector. Instead of representing "cat" as a single number, it represents it as a vector such as [0.2, 0.8, -0.5]. Similarly, "dog" could be represented as [0.3, 0.7, -0.6]. The values in these vectors are learned during model training.
Learning Relationships: During training, the model updates the embedding vectors based on the training objective. Words that occur in similar contexts may develop similar vector representations, allowing the model to capture useful relationships between words.

Example of Learning Relationships

Consider the sentence "The cat chased the mouse."
Across many training examples, the model learns from the contexts in which words such as "cat", "dog" and "mouse" appear.
If "cat" and "dog" occur in similar contexts, their embedding vectors may become more similar, allowing the model to capture relationships between them in the embedding space.

Building a Simple Neural Network with an Embedding Layer

The following is a simple example that helps us understand how to use an embedding layer in Python with TensorFlow. The model utilizes an embedding layer to process input data. Here's how it works:

input_dim refers to the size of the vocabulary, which is the number of unique words.
output_dim specifies the size of each word's vector, also known as the embedding size. For example, each word could be represented as a 128-dimensional vector.
input_length defines the length of each input sequence.

This setup allows the model to transform words into dense vectors of fixed size, which are easier for neural networks to process and understand.

Python

import tensorflow as tf
from tensorflow.keras import layers, models

vocab_size = 10000  # Size of the vocabulary 
embedding_dim = 128  # Dimension of the embedding vector
input_length = 100  # Length of input sequences

model = models.Sequential()
# Embedding layer
model.add(layers.Embedding(input_dim=vocab_size, output_dim=embedding_dim, input_length=input_length))
model.add(layers.GlobalAveragePooling1D())  
model.add(layers.Dense(64, activation='relu'))  
model.add(layers.Dense(1, activation='sigmoid')) 
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

# Check the model summary
model.build(input_shape=(None, input_length)) 
model.summary()

Output:

In the model summary, embedding layers an output shape of (None, 100, 128), meaning it processes input sequences of length 100 and maps each token to a 128-dimensional vector. The layer contains 1,280,000 parameters, which are the embeddings for each word in the vocabulary. These parameters will be learned during training.

Pre-trained Embeddings Models: Word2Vec, GloVe, FastText

Creating embeddings from scratch requires huge amount of datasets and computational power. Pre-trained embeddings makes this process easy:

Word2Vec: It is one of the earliest models to create word embeddings. It learns how different words are related to each other just by looking at different words that surround them in a sentence.
GloVe (Global Vectors for Word Representation): This model mainly focuses on capturing the overall statistics of word occurrences in a large dataset.
FastText: It was developed by Facebook. FastText is more advanced as it also captures the meanings of word parts like prefixes and suffixes. This is helpful for dealing with languages where word forms change such as adding "-ing" or "-ed" to verbs.

Visualizing Embedding Space

Visualizing embedding space helps us to observe how words that are semantically similar are clustered together in the embedding space. For this task, we will use GloVe, a pre-trained word embedding model. We will load the GloVe embeddings, extract the vectors for specific words like "king", "queen", "man", "woman", "boy" and "girl" and then reduce the dimensionality of these vectors to 2D using t-SNE.

You can download the GloVe Embeddings from here.

Python

import numpy as np
import matplotlib.pyplot as plt
from sklearn.manifold import TSNE

# Define the words 
words = ['king', 'queen', 'man', 'woman', 'boy', 'girl']

# Function to load GloVe embeddings
def load_glove_embeddings(file_path):
    embeddings = {}
    with open(file_path, 'r', encoding='utf-8') as f:
        for line in f:
            values = line.strip().split()
            word = values[0]
            vector = np.asarray(values[1:], dtype='float32')
            embeddings[word] = vector
    return embeddings

glove_embeddings = load_glove_embeddings('glove.6B.50d.txt') 

embedding_vectors = []
for word in words:
    if word in glove_embeddings:
        embedding_vectors.append(glove_embeddings[word])

embedding_vectors = np.array(embedding_vectors)

# Use t-SNE for dimensionality reduction to 2D
tsne = TSNE(n_components=2, random_state=42, perplexity=2)
reduced_embeddings_tsne = tsne.fit_transform(embedding_vectors)

# Plot the words in the 2D space
plt.figure(figsize=(8, 6))
for i, word in enumerate(words):
    plt.scatter(reduced_embeddings_tsne[i, 0], reduced_embeddings_tsne[i, 1])
    plt.text(reduced_embeddings_tsne[i, 0] + 0.1, reduced_embeddings_tsne[i, 1] + 0.1, word, fontsize=12)

plt.title('Word Embeddings Visualization (2D)')
plt.show()

Output:

This visualization will help us see how words like "king" and "queen" or "man" and "woman" are placed in proximity, reflecting the semantic relationships between them in the embedding space.

You can download the complete source code from here.

Applications

Word Embeddings: Converts words or tokens into dense vectors that capture their semantic meaning for NLP tasks such as classification, NER and translation.
Collaborative Filtering: Learns user and item embeddings to predict preferences and recommend products, movies, etc.
Text and Image Data: Represents text and images in a shared space for tasks such as image captioning and visual question answering.
Node Embeddings: Learns vector representations of graph nodes for tasks such as node classification and link prediction.

Advantages

Task-Specific Representations: Embeddings can be learned directly for a particular task, allowing the model to develop representations suited to its objective.
Transfer Learning: Pre-trained embeddings can be reused in new models, reducing the need to learn representations entirely from scratch.
Handles Large Vocabularies: Embedding layers can efficiently assign representations to a large number of tokens or categories without manually defining features for each one.
Flexible Embedding Size: The embedding dimension can be selected according to the model and dataset, providing flexibility in balancing representation capacity and model size.

Limitations

Out-of-Vocabulary Inputs: A basic embedding layer cannot directly map an unseen token to a learned vector unless an unknown-token mechanism or another strategy is used.
Context Dependence on the Model: A standard embedding layer assigns the same vector to a token regardless of its context. For example, a word with multiple meanings receives a single learned representation.
Potential Bias: Learned embeddings can inherit or amplify biases present in the data used to train them.
Memory Requirements: The embedding matrix can become large when the number of tokens or categories is very high, increasing the model's memory requirements.
Embedding Dimension Selection: Choosing an unsuitable embedding dimension can affect model performance and resource usage, so it may require experimentation.