Word Embeddings Using FastText

FastText embeddings are a type of word embedding developed by Facebook's AI Research (FAIR) lab. They are based on the idea of subword embeddings, which means that instead of representing words as single entities, FastText breaks them down into smaller components called character n-grams. By doing so, FastText can capture the semantic meaning of morphologically related words, even for out-of-vocabulary words or rare words, making it particularly useful for handling languages with rich morphology or for tasks where out-of-vocabulary words are common. In this article, we will discuss about fastText embeddings' implications in NLP.

What is the need for word embedding in NLP?

Word embeddings are fundamental in NLP for several reasons:

• Dimensionality Reduction: They represent words in a lower-dimensional continuous vector space, making it computationally efficient to handle extensive vocabularies.
• Semantic Similarity: Word embeddings encode semantic relationships, allowing algorithms to understand synonyms, antonyms, and related meanings.
• Contextual Information: By capturing context from surrounding words, embeddings help models understand a word's meaning in context, crucial for tasks like sentiment analysis and named entity recognition.
• Generalization: They generalize well to unseen words, learning from the distributional properties of words in training data.
• Feature Representation: Word embeddings serve as feature representations for machine learning models, enabling the application of various techniques to NLP tasks.
• Efficient Training: Models trained with word embeddings converge faster and often perform better than those using sparse representations.
• Transfer Learning: Pre-trained embeddings, like Word2Vec or GloVe, allow models to leverage knowledge from large corpora, even with limited task-specific data.

Why FastText Embeddings should be used?

FastText offers a significant advantage over traditional word embedding techniques like Word2Vec and GloVe, especially for morphologically rich languages. Here's a breakdown of how FastText addresses the limitations of traditional word embeddings and its implications:

• Utilization of Character-Level Information: FastText takes advantage of character-level information by representing words as the average of embeddings their character n-grams. This approach allows FastText to capture the internal structure of words, including prefixes, suffixes, and roots, which is particularly beneficial for morphologically rich languages where word formations follow specific rules.
• Extension of Word2Vec Model: FastText is an extension of the Word2Vec model, which means inherits the advantages of Word2Vec, such as capturing semantic relationships between words and producing dense vector representations.
• Handling Out-of-Vocabulary Words: One significant limitation of traditional word embeddings is their inability to handle out-of-vocabulary (OOV) words—words that are not present in the training data or vocabulary. Since Word2Vec and GloVe provide embeddings only for words seen during training, encountering an OOV word during inference can pose a challenge.
• FastText's Solution for OOV Words: FastText overcomes the limitation of OOV words by providing embeddings for character n-grams. If an OOV word occurs during inference, FastText can still generate an embedding for it based on its constituent character n-grams. This ability makes FastText more robust and suitable for handling scenarios where encountering new or rare words are common, such as social media data or specialized domains.
• Improved Vector Representations for Morphologically Rich Languages: By leveraging character-level information and providing embeddings for OOV words, FastText significantly improves vector representations for morphologically rich languages. It captures only the semantic meaning but also the internal structure and syntactic relations of words, leading to more accurate and contextually rich embeddings.

Working of FastText Embeddings

FastText embeddings revolutionize natural language processing by leveraging character-level information to generate robust word representations. For instance, consider the word "basketball" with the character n-grams

"<ba, bas, ask, sket, ket, et, etb, tb, tb, bal, all, ll>" and "<basketball>". 

• FastText computes the embedding for "basketball" by averaging the embeddings of these character n-grams along with the word itself. This approach captures both the semantic meaning and the internal structure of the word, making FastText particularly effective for morphologically rich languages.
• During training, FastText utilizes models like Continuous Bag of Words (CBOW) or Skip-gram, which are neural networks trained to predict context given a target word or vice versa. These models optimize neural network parameters to minimize loss, enabling FastText to learn meaningful word representations from large text corpora.

Furthermore, FastText's ability to handle out-of-vocabulary words helps in real-world applications where encountering new or rare words is common. Trained FastText embeddings serve as powerful features for various NLP tasks, facilitating tasks like text classification, sentiment analysis, and machine translation with improved accuracy and efficiency.

Skip-gram Vs CBOW

In the context of FastText embeddings, both Skip-gram and Continuous Bag of Words (CBOW) serve as training methodologies to generate word representations.

Considering the example of the word "basketball," let's compare how Skip-gram and CBOW operate:

Skip-gram:

• Input: Given the target word "basketball," Skip-gram aims to predict its context words, such as "play," "court," "team," etc.
• Training Objective: The model learns to predict the surrounding context words based on the target word.
• Example Usage: Given "basketball" as input, Skip-gram predicts its surrounding context words from a given text corpus.

Continuous Bag of Words (CBOW):

• Input: CBOW takes a window of context words, such as "play," "on," "the," "basketball," "court," as input.
• Training Objective: The model learns to predict the target word "basketball" given its surrounding context.
• Example Usage: With the context words "play," "on," "the," "court" provided as input, CBOW predicts the target word "basketball."In essence, Skip-gram and CBOW differ in their input and output configurations:

Skip-gram predicts context words given the target word. CBOW predicts the target word given its context.

Both methodologies contribute to training FastText embeddings, enabling the model to capture semantic relationships and syntactic structures effectively. The choice between Skip-gram and CBOW depends on the specific requirements of the NLP task and the characteristics of the text corpus being used.

Code Implementation of FastText Embeddings

• This code demonstrates training a FastText model using Gensim and using it to find word embeddings and similar words
• .It begins with importing the necessary libraries and defining a corpus, followed by the training of the FastText model with specified parameters.
• Word embeddings for a specific word ("computer" in this case) are then obtained from the trained model, and the most similar words to "computer" are found based on their embeddings.
• Finally, the word embedding for "computer" and the list of most similar words are printed.

from gensim.models import FastText
from gensim.test.utils import common_texts

# Example corpus (replace with your own corpus)
corpus = common_texts

# Training FastText model
model = FastText(sentences=corpus, vector_size=100, window=5, min_count=1, workers=4, sg=1)

# Example usage: getting embeddings for a word
word_embedding = model.wv['computer']

# Most similar words to a given word
similar_words = model.wv.most_similar('computer')

print("Most similar words to 'computer':", similar_words)

Output:

Most similar words to 'computer': [('user', 0.15659411251544952), ('response', 0.12383826076984406), 
('eps', 0.030704911798238754), ('system', 0.025573883205652237), ('interface', 0.0058587524108588696), 
('survey', -0.03156976401805878), ('minors', -0.0545564740896225), ('human', -0.0668589174747467), 
('time', -0.06855931878089905), ('trees', -0.10636083036661148)]

The output lists words along with their corresponding similarity scores to the word "computer." These scores indicate how semantically close each word is to "computer" within the model's learned vector space.

FastText VS Word2vec: Which is better?

FastText and Word2Vec are both popular tools in natural language processing for generating word embeddings, but they cater to slightly different needs and use cases:

Word2Vec, developed by Google, is renowned for its efficiency and effectiveness in capturing semantic and syntactic word relationships. It uses two architectures (CBOW and Skip-gram) to learn representations and excels in general language modeling tasks.

FastText, developed by Facebook’s AI Research lab, extends Word2Vec's idea by not only learning embeddings for words but also for n-grams of characters within a word. This approach allows FastText to generate better embeddings for rare words or misspelled words by leveraging subword information.

Choosing between FastText and Word2Vec depends on specific requirements: Word2Vec is generally preferred for tasks where there is large well-curated datasets and common vocabulary, whereas FastText shines in handling rare words and morphologically complex languages. For applications which needs robustness to word variations and misspellings, FastText may be the better choice.