Byte-Pair Encoding (BPE) in NLP

Last Updated : 21 Sep, 2026

Byte-Pair Encoding (BPE) is a text tokenization technique in Natural Language Processing. It represents words as smaller units called subwords or tokens by repeatedly merging frequent adjacent symbols during training.

frame_3828

This technique helps in handling rare or unknown words by breaking them into smaller parts that the model has already learned during training. By reducing the vocabulary size, it makes it easier to work with large amounts of text while allowing the model to understand wide variety of languages.

BPE converts text into tokens. The resulting tokens can then be mapped to token IDs and processed by an NLP or language model.

Working

  • BPE starts by representing each word as a sequence of individual characters or initial tokens. It then repeatedly finds the most frequent pair of adjacent tokens and merges them into a single subword token. The process continues until the desired number of merges or vocabulary size is reached.
  • Suppose we have a text corpus with four words: "ab", "bc", "bcd" and "cde".

Step 1: Initialize the vocabulary

Each word is initially split into individual characters.

Vocabulary = {"a", "b", "c", "d", "e"}

Step 2: Count token frequencies

The frequency of each character in the corpus is calculated:

Frequency = {"a": 1, "b": 3, "c": 3, "d": 2, "e": 1}

Step 3: Find the most frequent adjacent pair

BPE examines consecutive tokens and finds the pair that occurs most frequently.

Most frequent pair is "bc" with a frequency of 2.

Step 4: Merge the pair

The pair "bc" is merged into a single subword token.

Merge "b" and "c" to create a new subword unit "bc".

The token sequences are now updated.

For example:

"bcd" : "bc" + "d"

Step 5: Repeat the process

The frequencies of adjacent token pairs are recalculated using the updated sequences. The most frequent pair is then merged again.

For example:

The next merges may produce tokens such as"cd", "de", "ab" or "bcd", depending on their frequencies.

Each learned merge adds a new subword unit to the vocabulary. The process continues until the desired vocabulary size or number of merges is reached.

Step 6: Represent words using the learned subwords

After learning the merge rules, words are represented using the resulting subword tokens.

For example:

"ab" -> "a" + "b"
"bc" -> "bc"
"bcd" -> "bc" + "d"
"cde" -> "c" + "de"

The same learned merge rules can then be applied to new words. This allows BPE to represent common words as larger subwords while breaking rare or unseen words into smaller known units.

Implementation

1. Importing Libraries

We use Counter from the collections module to count the frequency of adjacent token pairs.

Python 
from collections import Counter

2. Counting Adjacent Token Pairs

The get_pair_frequencies() function counts how often each adjacent pair occurs in the current tokenized vocabulary.

Python 
def get_pair_frequencies(vocab):
    pairs = Counter()

    for tokens in vocab.values():
        for i in range(len(tokens) - 1):
            pair = (tokens[i], tokens[i + 1])
            pairs[pair] += 1

    return pairs

Here, vocab contains each word as a sequence of its current tokens. The function examines consecutive tokens and stores their frequencies in a Counter.

3. Merging a Token Pair

The merge_pair() function replaces every occurrence of a selected pair with a single merged token.

Python 
def merge_pair(vocab, pair):
    new_vocab = {}

    for word, tokens in vocab.items():
        new_tokens = []
        i = 0

        while i < len(tokens):
            if i < len(tokens) - 1 and (tokens[i], tokens[i + 1]) == pair:
                new_tokens.append(tokens[i] + tokens[i + 1])
                i += 2
            else:
                new_tokens.append(tokens[i])
                i += 1

        new_vocab[word] = new_tokens

    return new_vocab

For example, if the selected pair is ("b", "c"), the token sequence:

["b", "c", "d"]

becomes:

["bc", "d"]

4. Learning BPE Merge Rules

The learn_bpe() function initializes each word as a sequence of characters and repeatedly finds and merges the most frequent adjacent pair.

Python 
def learn_bpe(corpus, num_merges=3):
    vocab = {
        word: list(word)
        for word in corpus.split()
    }

    merges = []

    for _ in range(num_merges):
        pair_frequencies = get_pair_frequencies(vocab)

        if not pair_frequencies:
            break

        most_frequent = max(
            pair_frequencies,
            key=pair_frequencies.get
        )

        merges.append(most_frequent)

        vocab = merge_pair(vocab, most_frequent)

    return merges

The learned pairs are stored in merges in the order in which they were selected. After each merge, the vocabulary is updated before calculating the frequencies for the next merge.

5. Applying the Learned BPE Rules

The apply_bpe() function applies the learned merge rules to a new word.

Python 
def apply_bpe(word, merges):
    tokens = list(word)

    for pair in merges:
        new_tokens = []
        i = 0

        while i < len(tokens):
            if i < len(tokens) - 1 and (tokens[i], tokens[i + 1]) == pair:
                new_tokens.append(tokens[i] + tokens[i + 1])
                i += 2
            else:
                new_tokens.append(tokens[i])
                i += 1

        tokens = new_tokens

    return tokens

The function starts with individual characters and applies each learned merge rule in order to produce the final subword tokens.

6. Example Usage

We now use a small corpus to learn three BPE merge rules and apply them to the word bcd.

Python 
# Example corpus
corpus = "ab bc bcd cde"

# Learn 3 merge rules
merges = learn_bpe(corpus, num_merges=3)

print("Learned merges:", merges)

# Apply the learned rules to a new word
word = "bcd"
tokens = apply_bpe(word, merges)

print("BPE representation:", tokens)

Output:

Learned merges: [('b', 'c'), ('a', 'b'), ('bc', 'd')]
BPE representation: ['bcd']

The first merge combines b and c because this pair occurs most frequently in the corpus. The subsequent merge rules are learned from the updated token sequences. When the learned rules are applied to bcd, b and c are first merged into bc, followed by bc and d, resulting in the token bcd.

Note: Run the complete code together, as the functions depend on one another.

You can download the complete source code from here.

Applications

  • Large Language Models: Used to split text into subword tokens before processing.
  • Machine Translation: Helps represent words across languages, including rare and morphologically complex words.
  • Text Generation: Provides a manageable vocabulary while allowing new words to be represented through subword combinations.
  • Text Classification: Converts text into subword tokens that can be used as model inputs.
  • Speech and Language Processing: Helps handle varied and previously unseen word forms.

Limitations

  • Token boundaries are based on corpus statistics rather than linguistic meaning.
  • Different training corpora produce different vocabularies and merge rules.
  • Tokenization can be inefficient for some languages or domains if the vocabulary is poorly trained.
  • The learned vocabulary and merge rules are tokenizer-specific.
Comment

Explore