Chapter 5 DICTIONARY

CODING

Dictionary Coding
Dictionary coding is different from Huffman
coding and arithmetic coding.
Both Huffman and arithmetic coding techniques
are based on a statistical model (e.g.,
occurrence probabilities).
In dictionary-based data compression
techniques, a symbol or a string of symbols
generated from a source alphabet is
represented by an index to a dictionary
constructed from the source alphabet.

Dictionary Coding
A dictionary is a list of symbols and strings
of symbols.
There are many examples of this in our daily
lives.
the string September vs. 9,
a social security number vs. a person in the U.S.

Dictionary coding is widely used in text

coding.

Dictionary Coding
Consider English text coding.
The source alphabet includes 26 English letters in both
lower and upper cases, numbers, various punctuation
marks and the space bar.
Huffman or arithmetic coding treats each symbol based
on its occurrence probability. That is, the source is
modeled as a memoryless source.
It is well known, however, that this is not true in many
applications.
In text coding, structure or context plays a significant
role.
Very likely that the letter u appears after the letter q.
Likewise, it is likely that the word concerned will appear after
As far as the weather is.

Dictionary Coding
The strategy of the dictionary coding is to
build a dictionary that contains frequently
occurring symbols and string of symbols.
When a symbol or a string is encountered and it
is contained in the dictionary, it is encoded with
an index to the dictionary.
Otherwise, if not in the dictionary, the symbol or
the string of symbols is encoded in a less
efficient manner.

Dictionary Coding
All the dictionary schemes have an equivalent
statistical scheme, which achieves exactly the
same compression.
Statistical scheme using high-order context
models may outperform dictionary coding (with
high computational complexity).
But, dictionary coding is now superior for fast
speed and economy of memory.
In future, however, statistical scheme may be
preferred [bell 1990].

Formulation of Dictionary Coding

Define dictionary coding in a precise manner
[bell 1990].
We denote a source alphabet by S.
A dictionary consisting of two elements is
defined as D ( P, C ) ,
P is a finite set of phrases generated from the S,
C is a coding function mapping P onto a set of
codewords.

Formulation of Dictionary Coding

The set P is said to be complete if any input string
can be represented by a series of phrases chosen
from the P.
The coding function C is said to obey the prefix
property if there is no codeword that is a prefix of
any other codeword.
For practical usage, i.e., for reversible
compression of any input text, the phrase set P
must be complete and the coding function C must
satisfy the prefix property.

Categorization of DictionaryBased Coding Techniques

The heart of dictionary coding is the
formulation of the dictionary.
A successfully built dictionary results in data
compression; the opposite case may lead to
data expansion.
According to the ways in which dictionaries
are constructed, dictionary coding
techniques can be classified as static or
adaptive.

Static Dictionary Coding

A fixed dictionary,

Produced before the coding process

Used at both the transmitting and receiving ends
It is possible when the knowledge about the source
alphabet and the related strings of symbols, also
known as phrases, is sufficient.

Merit of the static approach: its simplicity.

Its drawbacks lie on

Relatively lower coding efficiency

Less flexibility compared with adaptive dictionary
techniques

Static Dictionary Coding

An example of static algorithms occurs is diagram
coding.
A simple and fast coding technique.
The dictionary contains:
all source symbols and
some frequently used pairs of symbols.

In encoding, two symbols are checked at once to see if

they are in the dictionary.
If so, they are replaced by the index of the two symbols in the
dictionary, and the next pair of symbols is encoded in the next
step.

Static Dictionary Coding

If not, then the index of the first symbol is used to encode the
first symbol. The second symbol is combined with the third
symbol to form a new pair, which is encoded in the next step.

The diagram can be straightforwardly extended

to n-gram. In the extension, the size of the
dictionary increases and so is its coding
efficiency.

Adaptive Dictionary Coding

A completely defined dictionary does not exist
prior to the encoding process and the dictionary is
not fixed.
At the beginning of coding, only an initial dictionary
exists.
It adapts itself to the input during the coding process.

All adaptive dictionary coding algorithms can be

traced back to two different original works by Ziv
and Lempel [ziv 1977, ziv 1978].
The algorithms based on [ziv 1977] are referred to as
the LZ77 algorithms.
Those based on [ziv 1978] the LZ78 algorithms.

Parsing Strategy
Once have a dictionary,
Need to examine the input text and find a string of
symbols that matches an item in the dictionary.
Then the index of the item to the dictionary is encoded.

This process of segmenting the input text into

disjoint strings (whose union equals the input
text) for coding is referred to as parsing.
Obviously, the way to segment the input text into
strings is not unique.

Parsing Strategy
In terms of the highest coding efficiency, optimal
parsing is essentially a shortest-path problem
[bell 1990].
In practice, however, a method called greedy
parsing is used in all the LZ77 and LZ78
algorithms.
With greedy parsing, the encoder searches for the
longest string of symbols in the input that matches an
item in the dictionary at each coding step.
Greedy parsing may not be optimal, but it is simple in
implementation.

Parsing Strategy
Example 6.1
Consider a dictionary, D, whose phrase set
P {a, b, ab, ba, bb, aab, bbb} . The codewords assigned to
these strings are C(a) 10 , C (b) 011 , C (ab) 010 , C (ba) 0101 ,
C (bb) 01 , C (aab) 11 , and C (bbb) 0110.
Now the input text: abbaab.
Using greedy parsing, we then encode the text as
C (ab).C (ba).C (ab) , which is a 10-bit string: 010.0101.010.
In above representations, the periods are used to
indicate the division of segments in the parsing.
This, however, is not an optimum solution. Obviously,
the following parsing will be more efficient, i.e.,
C (a).C (bb).C (aab) , which is a 6- bit string: 10.01.11.

Sliding Window (LZ77) Algorithms

Introduction
The dictionary used is actually a portion of the input text,
which has been recently encoded.
The text that needs to be encoded is compared with the
strings of symbols in the dictionary.
The longest matched string in the dictionary is
characterized by a pointer (sometimes called a token),
which is represented by a triple of data items.
Note that this triple functions as an index to the
dictionary.
In this way, a variable-length string of symbols is
mapped to a fixed-length pointer.

Introduction
There is a sliding window in the LZ77 algorithms. The
window consists of two parts: a search buffer and a lookahead buffer.
The search buffer contains: the portion of the text stream that has
recently been encoded --- the dictionary.
The look-ahead buffer contains: the text to be encoded next.

The window slides through the input text stream from

beginning to end during the entire encoding process.
The size of the search buffer is much larger than that of the
look-ahead buffer.
The sliding window: usually on the order of a few thousand
symbols.
The look-ahead buffer: on the order of several tens to one hundred
symbols.

Encoding and Decoding

Example 6.2
Encoding:

Encoding and Decoding

Decoding:
Now let us see how
the decoder recovers
the string baccbaccgi
from these three
triples.
In Figure 6.5, for
each part, the last
previously encoded
symbol c prior to the
receiving of the three
triples is shaded.

Summary of the LZ77 Approach

The first symbol in the look-ahead buffer is the
symbol or the beginning of a string of symbols to
be encoded at the moment. Let us call it the
symbol s.
In encoding, the search pointer moves to the left,
away from the symbol s, to find a match of the
symbol s in the search buffer. When there are
multiple matches, the match that produces the
longest matched string is chosen. The match is
denoted by a triple <i, j, k>.

Summary of the LZ77 Approach

The first item in the triple, i, is the offset, which is the
distance between the pointer pointing to the symbol
giving the maximum match and the symbol s.
The second item, j, is the length of the matched string.
The third item, k, is the codeword of the symbol
following the matched string in the look-ahead buffer.

At the very beginning, the content of the search

buffer can be arbitrarily selected. For instance, the
symbols in the search buffer may all be the space
symbol.

Summary of the LZ77 Approach

Denote the size of the search buffer by SB, the
size of the look-ahead buffer by L and the size of
the source alphabet by A. Assume that the natural
binary code is used. Then we see that the LZ77
approach encodes variable-length strings of
symbols with fixed-length codewords.
Specifically, the offset i is of coding length log SB ,
the length of matched string j is of coding length
log 2 ( SB L) ,
and the codeword k is of coding length log 2 ( A) , where
the sign a denotes the smallest integer larger than a .
2

Summary of the LZ77 Approach

The length of the matched string is equal to log (SB L)
because the search for the maximum matching
can enter into the look-ahead buffer.
The decoding process is simpler than the
encoding process since there is no comparison
involved in the decoding.
The most recently encoded symbols in the search
buffer serve as the dictionary used in the LZ77
approach. The merit of doing so is that the
dictionary is adapted to the input text well.
2

Summary of the LZ77 Approach

The limitation of the approach is that if the
distance between the repeated patterns in the
input text stream is larger than the size of the
search buffer, then the approach cannot utilize the
structure to compress the text.
A window with a moderate size, say, SB L 8192, can
compress a variety of texts well. Several reasons
have been analyzed in [bell 1990].
Many variations have been made to improve
coding efficiency of the LZ77 approach.

LZ78 Algorithms
Introduction
Limitations with LZ77:
If the distance between two repeated patterns is larger than the
size of the search buffer, the LZ77 algorithms cannot work
efficiently.
The fixed size of the both buffers implies that the matched string
cannot be longer than the sum of the sizes of the two buffers,
meaning another limitation on coding efficiency.
Increasing the sizes of the search buffer and the look-ahead
buffer seemingly will resolve the problems. A close look,
however, reveals that it also leads to increases in the number of
bits required to encode the offset and matched string length as
well as an increase in processing complexity.

Introduction
LZ78:
No use of the sliding window.
Use encoded text as a dictionary which, potentially, does not
have a fixed size.
Each time a pointer (token) is issued, the encoded string is
included in the dictionary.
Once a preset limit to the dictionary size has been reached,
either the dictionary is fixed for the future (if the coding
efficiency is good), or it is reset to zero, i.e., it must be restarted.
Instead of the triples used in the LZ77, only pairs are used in
the LZ78.
Specifically, only the position of the pointer to the matched
string and the symbol following the matched string need to be
encoded.

Encoding and Decoding

Example 6.3
Encoding:
Consider the text stream: baccbaccacbcabccbbacc.
In Table 6.4, in general, as the encoding proceeds, the entries
in the dictionary become longer and longer. First, entries with
single symbols come out, but later, more and more entries with
two symbols show up. After that more and more entries with
three symbols appear. This means that coding efficiency is
increasing.

Decoding:
Since the decoder knows the rule applied in the encoding, it can
reconstruct the dictionary and decode the input text stream from
the received doubles.

Encoding and Decoding

Table 6.4 An encoding example using the LZ78 algorithm
Index

Doubles

Encoded symbols

< 0, C(b) >

< 0, C(a) >

< 0, C(c) >

< 3, 1 >

cb

< 2, 3 >

ac

< 3, 2 >

ca

< 4, 3 >

cbc

< 2, 1 >

ab

< 3, 3 >

cc

10

< 1, 1 >

bb

11

< 5, 3 >

acc

LZW Algorithm
Both the LZ77 and LZ78 approaches, when published in
1977 and 1978, respectively, were theory-oriented.
The effective and practical improvement over the LZ78 in
[welch 1984] brought much attention to the LZ dictionary
coding techniques. The resulting algorithm is referred to
the LZW algorithm.
It removed the second item in the double (the index of the
symbol following the longest matched string) and hence, it
enhanced coding efficiency.
In other words, the LZW only sends the indexes of the
dictionary to the decoder.

LZW Algorithm
Example 6.4
Consider the following input text stream:
accbadaccbaccbacc. We see that the source
alphabet is S={a,b,c,d,}.
Encoding:

LZW Algorithm
Table 6.5 An example of the dictionary coding using the
LZW algorithm
Index

Entry

1
2
3
4

a
b
c
d

5
6
7
8
9
10
11
12
13
14
15

ac
cc
cb
ba
ad
da
acc
cba
accb
bac
cc

Input Smbols

Encoded Index

Initial dictionary

a
C
C
b
a
d
a,c
c,b
a,c,c
b,a,
c,c,

1
3
3
2
1
4
5
7
11
8

LZW Algorithm
Decoding:
Initially, the decoder has the same dictionary (the top four rows
in Table 6.5) as that in the encoder.
Once the first index 1 comes, the decoder decodes a symbol a.
The second index is 3, which indicates that the next symbol is c.
From the rule applied in encoding, the decoder knows further that
a new entry ac has been added to the dictionary with an index 5.

The next index is 3. It is known that the next symbol is also c.

It is also known that the string cc has been added into the
dictionary as the sixth entry.

In this way, the decoder reconstructs the dictionary and

decodes the input text stream.

LZ78 Compression Algorithm (contd)

Dictionary empty ; Prefix empty ; DictionaryIndex 1;
while(characterStream is not empty)
{
Char next character in characterStream;
if(Prefix + Char exists in the Dictionary)
Prefix Prefix + Char ;
else
{
if(Prefix is empty)
CodeWordForPrefix 0 ;
else
CodeWordForPrefix DictionaryIndex for Prefix ;
Output: (CodeWordForPrefix, Char) ;
insertInDictionary( ( DictionaryIndex , Prefix + Char) );
DictionaryIndex++ ;
Prefix empty ;
}
}
if(Prefix is not empty)
{
CodeWordForPrefix DictionaryIndex for Prefix;
Output: (CodeWordForPrefix , ) ;
}

Exercises
1. Encode (i.e., compress) the following strings
using the LZ77 algorithm.
Window size : 5
ABBABCABBBBC

2. Encode (i.e., compress) the following strings

using the LZ78 algorithm.
1.ABBCBCABABCAABCAAB
2.BABAABRRRA
3.AAAAAAAAA

LZ77 - Example
Window size : 5
ABBABCABBBBC
NextSeq Code
A
B
BA
BC
ABB
BBC

(0,0,A)
(0,0,B)
(1,1,A)
(3,1,C)
(3,2,B)
(2,2,C)

Example 1: LZ78 Compression

Encode (i.e., compress) the string ABBCBCABABCAABCAAB using the LZ78 algorithm.

The compressed message is: (0,A)(0,B)(2,C)(3,A)(2,A)(4,A)(6,B)

Note: The above is just a representation, the commas and parentheses are not transmitted;
we will discuss the actual form of the compressed message later on in slide 12.

Example 1: LZ78 Compression (contd)

1. A is not in the Dictionary; insert it
2. B is not in the Dictionary; insert it
3. B is in the Dictionary.
BC is not in the Dictionary; insert it.
4. B is in the Dictionary.
BC is in the Dictionary.
BCA is not in the Dictionary; insert it.
5. B is in the Dictionary.
BA is not in the Dictionary; insert it.
6. B is in the Dictionary.
BC is in the Dictionary.
BCA is in the Dictionary.
BCAA is not in the Dictionary; insert it.
7. B is in the Dictionary.
BC is in the Dictionary.
BCA is in the Dictionary.
BCAA is in the Dictionary.
BCAAB is not in the Dictionary; insert it.

Example 2: LZ78 Compression

Encode (i.e., compress) the string BABAABRRRA using the LZ78 algorithm.

The compressed message is: (0,B)(0,A)(1,A)(2,B)(0,R)(5,R)(2, )

Example 2: LZ78 Compression (contd)

1. B is not in the Dictionary; insert it
2. A is not in the Dictionary; insert it
3. B is in the Dictionary.
BA is not in the Dictionary; insert it.
4. A is in the Dictionary.
AB is not in the Dictionary; insert it.
5. R is not in the Dictionary; insert it.
6. R is in the Dictionary.
RR is not in the Dictionary; insert it.
7. A is in the Dictionary and it is the last input character; output a pair
containing its index: (2, )

Example 3: LZ78 Compression

Encode (i.e., compress) the string AAAAAAAAA using the LZ78 algorithm.

1. A is not in the Dictionary; insert it

2. A is in the Dictionary
AA is not in the Dictionary; insert it
3. A is in the Dictionary.
AA is in the Dictionary.
AAA is not in the Dictionary; insert it.
4. A is in the Dictionary.
AA is in the Dictionary.
AAA is in the Dictionary and it is the last pattern; output a pair containing its index:
(3, )

LZ78 Compression: Number of bits transmitted

Example: Uncompressed String: ABBCBCABABCAABCAAB

Number of bits = Total number of characters * 8
= 18 * 8
= 144 bits

Suppose the codewords are indexed starting from 1:

Compressed string( codewords): (0, A) (0, B) (2, C) (3, A) (2, A) (4, A) (6, B)
Codeword index
1
2
3
4
5
6
7

Each code word consists of an integer and a character:

The character is represented by 8 bits.
The number of bits n required to represent the integer part of the codeword with
index i is given by:

Alternatively number of bits required to represent the integer part of the codeword
with index i is the number of significant bits required to represent the integer i 1

LZ78 Compression: Number of bits transmitted (contd)

Codeword
index
Bits:

(0, A)
(0, B) (2, C) (3, A) (2, A) (4, A) (6, B)
1
2
3
4
5
6
7
(1 + 8) + (1 + 8) + (2 + 8) + (2 + 8) + (3 + 8) + (3 + 8) + (3 + 8) = 71 bits

The actual compressed message is: 0A0B10C11A010A100A110B

where each character is replaced by its binary 8-bit ASCII code.

LZ78 Decompression Algorithm

Dictionary empty ; DictionaryIndex 1 ;
while(there are more (CodeWord, Char) pairs in codestream){
CodeWord next CodeWord in codestream ;
Char character corresponding to CodeWord ;
if(CodeWord = = 0)
String empty ;
else
String string at index CodeWord in Dictionary ;
Output: String + Char ;
insertInDictionary( (DictionaryIndex , String + Char) ) ;
DictionaryIndex++;
}

Summary:

input: (CW, character) pairs

output:

if(CW == 0)
output: currentCharacter
else
output: stringAtIndex CW + currentCharacter
Insert: current output in dictionary

Example 1: LZ78 Decompression

Decode (i.e., decompress) the sequence (0, A) (0, B) (2, C) (3, A) (2, A) (4, A) (6, B)

The decompressed message is: ABBCBCABABCAABCAAB

Example 2: LZ78 Decompression

Decode (i.e., decompress) the sequence (0, B) (0, A) (1, A) (2, B) (0, R) (5, R) (2, )

The decompressed message is: BABAABRRRA

Example 3: LZ78 Decompression

Decode (i.e., decompress) the sequence (0, A) (1, A) (2, A) (3, )

The decompressed message is: AAAAAAAAA

Applications
The CCITT Recommendation V.42 bis is a data
compression standard used in modems that connect
computers with remote users via the GSTN. In the
compressed mode, the LZW algorithm is recommended for
data compression.
In image compression, the LZW finds its application as
well. Specifically, it is utilized in the graphic interchange
format (GIF) which was created to encode graphical
images. GIF is now also used to encode natural images,
though it is not very efficient in this regard. For more
information, readers are referred to [sayood 1996].
The LZW algorithm is also used in the Unix Compress
command.

International Standards for

Lossless Still Image Compression
Lossless Bilevel Still Image Compression
Algorithms
As mentioned above, there are four different
international standard algorithms falling into this
category.
MH (Modified Huffman coding): This algorithm is defined in
CCITT Recommendation T.4 for facsimile coding. It uses
the 1-D RLC technique followed by the modified Huffman
coding technique.

Algorithms
MR (Modified READ (Relative Element Address Designate)
coding): Defined in CCITT Recommendation T.4 for
facsimile coding. It uses the 2-D RLC technique followed by
the modified Huffman coding technique.
MMR (Modified Modified READ coding): Defined in CCITT
Recommendation T.6. It is based on MR, but is modified to
maximize compression.
JBIG (Joint Bilevel Image experts Group coding): Defined in
CCITT Recommendation T.82. It uses an adaptive 2-D
coding model, followed by an adaptive arithmetic coding
technique.

Performance Comparison
The JBIG test image set was used to compare the
performance of the above-mentioned algorithms. The set
contains scanned business documents with different
densities, graphic images, digital halftones, and mixed
(document and halftone) images.
Note that digital halftones, also named (digital) halftone
images, are generated by using only binary devices.
Some small black units are imposed on a white background.
The units may assume different shapes: a circle, a square and so
on.
The more dense the black units in a spot of an image, the darker
the spot appears.
The digital halftoning method has been used for printing gray-level
images in newspapers and books.

Performance Comparison
For bilevel images excluding digital halftones, the
compression ratio achieved by these techniques ranges
from 3 to 100.
The compression ratio increases monotonically in the order
of the following standard algorithms: MH, MR, MMR, JBIG.
For digital halftones, MH, MR, and MMR result in data
expansion, while JBIG achieves compression ratios in the
range of 5 to 20.
This demonstrates that among the techniques, JBIG is the
only one suitable for the compression of digital halftones.

Lossless Multilevel Still Image

Compression
Algorithms
There are two international standards for
multilevel still image compression:
JBIG (Joint Bilevel Image experts Group coding):
Defined in CCITT Recommendation T. 82.
It uses an adaptive arithmetic coding technique.
To encode multilevel images, the JIBG decomposes
multilevel images into bit-planes, then compresses these
bit-planes using its bilevel image compression technique.
To further enhance compression ratio, it uses Gary coding
to represent pixel amplitudes instead of weighted binary
coding.

Algorithms
JPEG (Joint Photographic (image) Experts Group
coding): Defined in CCITT Recommendation T. 81.
For lossless coding (Lossless JPEG), it uses
differential coding technique. The predictive error is
encoded using either Huffman coding or adaptive
arithmetic coding techniques.

Performance Comparison
A set of color test images from the JPEG
standards committee was used for
performance comparison.
The luminance component (Y) is of resolution
720 576 pixels, while the chrominance
components (U and V) are of 360 576 pixels.
The compression ratios calculated are the
combined results for all the three components.
The following observations have been reported.

Performance Comparison
When quantized in 8 bits/pixel, the compression ratios vary
much less for multilevel images than for bilevel images,
and are roughly equal to 2.
When quantized with 5 bits/pixel down to 2 bits/pixel,
compared with the lossless JPEG, the JBIG achieves an
increasingly higher compression ratio, up to a maximum of
29%.
When quantized with 6 bits/pixel, JBIG and lossless JPEG
achieve similar compression ratios.
When quantized with 7 bits/pixel to 8 bits/pixel, the lossless
JPEG achieves a 2.4 % to 2.6 % higher compression ratio
than JBIG.