Minimum Edit

Distance

Definition of Minimum
Edit Distance
Dan Jurafsky

How similar are two strings?

• Spell correction • Computational Biology
• The user typed “graffe” • Align two sequences of nucleotides
Which is closest? AGGCTATCACCTGACCTCCAGGCCGATGCCC
• graf TAGCTATCACGACCGCGGTCGATTTGCCCGAC
• graft • Resulting alignment:
• grail
• giraffe -AGGCTATCACCTGACCTCCAGGCCGA--TGCCC---
TAG-CTATCAC--GACCGC--GGTCGATTTGCCCGAC

• Also for Machine Translation, Information Extraction, Speech Recognition

Dan Jurafsky

Edit Distance
• The minimum edit distance between two strings
• Is the minimum number of editing operations
• Insertion
• Deletion
• Substitution
• Needed to transform one into the other
Dan Jurafsky

Minimum Edit Distance

• Two strings and their alignment:
Dan Jurafsky

Minimum Edit Distance

• If each operation has cost of 1

• Distance between these is 5
• If substitutions cost 2 (Levenshtein)
• Distance between them is 8
Dan Jurafsky

Alignment in Computational Biology

• Given a sequence of bases

AGGCTATCACCTGACCTCCAGGCCGATGCCC
TAGCTATCACGACCGCGGTCGATTTGCCCGAC
• An alignment:
-AGGCTATCACCTGACCTCCAGGCCGA--TGCCC---
TAG-CTATCAC--GACCGC--GGTCGATTTGCCCGAC
• Given two sequences, align each letter to a letter or gap
 Dan Jurafsky

Other uses of Edit Distance in NLP

• Evaluating Machine Translation and speech recognition

R Spokesman confirms senior government adviser was shot
H Spokesman said the senior adviser was shot dead
S I D I
• Named Entity Extraction and Entity Coreference
• IBM Inc. announced today
• IBM profits
• Stanford President John Hennessy announced yesterday
• for Stanford University President John Hennessy
Dan Jurafsky

How to find the Min Edit Distance?

• Searching for a path (sequence of edits) from the start
string to the final string:
• Initial state: the word we’re transforming
• Operators: insert, delete, substitute
• Goal state: the word we’re trying to get to
• Path cost: what we want to minimize: the number of edits
Dan Jurafsky

Minimum Edit as Search

• But the space of all edit sequences is huge!
• We can’t afford to navigate naïvely
• Lots of distinct paths wind up at the same state.
• We don’t have to keep track of all of them
• Just the shortest path to each of those revisted states.
Dan Jurafsky

Defining Min Edit Distance

• For two strings
• X of length n
• Y of length m
• We define D(i,j)
• the edit distance between X[1..i] and Y[1..j]
• i.e., the first i characters of X and the first j characters of Y
• The edit distance between X and Y is thus D(n,m)
Minimum Edit
Distance

Definition of Minimum
Edit Distance
Minimum Edit
Distance

Computing Minimum
Edit Distance
Dan Jurafsky

Dynamic Programming for

Minimum Edit Distance
• Dynamic programming: A tabular computation of D(n,m)
• Solving problems by combining solutions to subproblems.
• Bottom-up
• We compute D(i,j) for small i,j
• And compute larger D(i,j) based on previously computed smaller values
i.e., compute D(i,j) for all i (0 < i < n) and j (0 < j < m).
 Dan Jurafsky

Defining Min Edit Distance (Levenshtein)

• Initialization
D(i,0) = i
D(0,j) = j
• Recurrence Relation:
For each i = 1…M
For each j = 1…N
D(i-1,j) + 1
D(i,j)= min D(i,j-1) + 1
D(i-1,j-1) + 2; if X(i) ≠ Y(j)
0; if X(i) = Y(j)
• Termination:
D(N,M) is distance
Dan Jurafsky

The Edit Distance Table

N 9
O 8
I 7

T 6
N 5
E 4
T 3
N 2
I 1
# 0 1 2 3 4 5 6 7 8 9
# E X E C U T I O N
Dan Jurafsky

The Edit Distance Table

N 9
O 8
I 7
T 6
N 5
E 4
T 3
N 2
I 1
# 0 1 2 3 4 5 6 7 8 9
# E X E C U T I O N
Dan Jurafsky

Edit Distance
N 9
O 8
I 7

T 6
N 5
E 4
T 3
N 2
I 1
# 0 1 2 3 4 5 6 7 8 9
# E X E C U T I O N
Dan Jurafsky

The Edit Distance Table

N 9 8 9 10 11 12 11 10 9 8
O 8 7 8 9 10 11 10 9 8 9
I 7 6 7 8 9 10 9 8 9 10
T 6 5 6 7 8 9 8 9 10 11
N 5 4 5 6 7 8 9 10 11 10
E 4 3 4 5 6 7 8 9 10 9
T 3 4 5 6 7 8 7 8 9 8
N 2 3 4 5 6 7 8 7 8 7
I 1 2 3 4 5 6 7 6 7 8
# 0 1 2 3 4 5 6 7 8 9
# E X E C U T I O N
Minimum Edit
Distance

Computing Minimum
Edit Distance
 Minimum Edit
Distance

Backtrace for Computing

Alignments
Dan Jurafsky

Computing alignments
• Edit distance isn’t sufficient
• We often need to align each character of the two strings to each other
• We do this by keeping a “backtrace”
• Every time we enter a cell, remember where we came from
• When we reach the end,
• Trace back the path from the upper right corner to read off the alignment
Dan Jurafsky

Edit Distance
N 9
O 8
I 7

T 6
N 5
E 4
T 3
N 2
I 1
# 0 1 2 3 4 5 6 7 8 9
# E X E C U T I O N
Dan Jurafsky

MinEdit with Backtrace

 Dan Jurafsky

Adding Backtrace to Minimum Edit Distance

• Base conditions: Termination:

D(i,0) = i D(0,j) = j D(N,M) is distance
• Recurrence Relation:
For each i = 1…M
For each j = 1…N
D(i-1,j) + 1 deletion
D(i,j)= min D(i,j-1) + 1 insertion
D(i-1,j-1) + 2; if X(i) ≠ Y(j) substitution
0; if X(i) = Y(j)
LEFT insertion
ptr(i,j)= DOWN deletion
DIAG substitution
Dan Jurafsky

The Distance Matrix

Every non-decreasing path

x0 …………………… xN

from (0,0) to (M, N)

corresponds to
an alignment
of the two sequences

An optimal alignment is composed of

y0 ……………………………… yM optimal subalignments
Slide adapted from Serafim Batzoglou
Dan Jurafsky

Result of Backtrace
• Two strings and their alignment:
Dan Jurafsky

Performance
• Time:
O(nm)
• Space:
O(nm)
• Backtrace
O(n+m)
 Minimum Edit
Distance

Backtrace for Computing

Alignments
Minimum Edit
Distance

Weighted Minimum Edit

Distance
Dan Jurafsky

Weighted Edit Distance

• Why would we add weights to the computation?
• Spell Correction: some letters are more likely to be mistyped than others
• Biology: certain kinds of deletions or insertions are more likely than
others
Dan Jurafsky

Confusion matrix for spelling errors

Dan Jurafsky
 Dan Jurafsky

Weighted Min Edit Distance

• Initialization:
D(0,0) = 0
D(i,0) = D(i-1,0) + del[x(i)]; 1 < i ≤ N
D(0,j) = D(0,j-1) + ins[y(j)]; 1 < j ≤ M
• Recurrence Relation:
D(i-1,j) + del[x(i)]
D(i,j)= min D(i,j-1) + ins[y(j)]
D(i-1,j-1) + sub[x(i),y(j)]
• Termination:
D(N,M) is distance
 Dan Jurafsky

Where did the name, dynamic

programming, come from?
…The 1950s were not good years for mathematical research. [the] Secretary of
Defense …had a pathological fear and hatred of the word, research…

I decided therefore to use the word, “programming”.

I wanted to get across the idea that this was dynamic, this was multistage… I thought,
let’s … take a word that has an absolutely precise meaning, namely dynamic… it’s
impossible to use the word, dynamic, in a pejorative sense. Try thinking of some
combination that will possibly give it a pejorative meaning. It’s impossible.

Thus, I thought dynamic programming was a good name. It was something not even a
Congressman could object to.”

Richard Bellman, “Eye of the Hurricane: an autobiography” 1984.

Minimum Edit
Distance

Weighted Minimum Edit

Distance
 Minimum Edit
Distance

Minimum Edit Distance in

Computational Biology
Dan Jurafsky

Sequence Alignment

AGGCTATCACCTGACCTCCAGGCCGATGCCC
TAGCTATCACGACCGCGGTCGATTTGCCCGAC

-AGGCTATCACCTGACCTCCAGGCCGA--TGCCC---
TAG-CTATCAC--GACCGC--GGTCGATTTGCCCGAC
Dan Jurafsky

Why sequence alignment?

• Comparing genes or regions from different species
• to find important regions
• determine function
• uncover evolutionary forces
• Assembling fragments to sequence DNA
• Compare individuals to looking for mutations
Dan Jurafsky

Alignments in two fields

• In Natural Language Processing
• We generally talk about distance (minimized)
• And weights
• In Computational Biology
• We generally talk about similarity (maximized)
• And scores
 Dan Jurafsky

The Needleman-Wunsch Algorithm

• Initialization:
D(i,0) = -i * d
D(0,j) = -j * d
• Recurrence Relation:
D(i-1,j) - d
D(i,j)= min D(i,j-1) - d
D(i-1,j-1) + s[x(i),y(j)]
• Termination:
D(N,M) is distance
Dan Jurafsky

The Needleman-Wunsch Matrix

x1 ……………………………… xM
y1 …………………… yN

(Note that the origin is

at the upper left.)

Slide adapted from Serafim Batzoglou

Dan Jurafsky

A variant of the basic algorithm:

• Maybe it is OK to have an unlimited # of gaps in the beginning

and end:

----------CTATCACCTGACCTCCAGGCCGATGCCCCTTCCGGC
GCGAGTTCATCTATCAC--GACCGC--GGTCG--------------

• If so, we don’t want to penalize gaps at the ends

Slide from Serafim Batzoglou

Dan Jurafsky

Different types of overlaps

Example:
2 overlapping“reads” from a
sequencing project

Example:
Search for a mouse gene
within a human chromosome

Slide from Serafim Batzoglou

 Dan Jurafsky

The Overlap Detection variant

x1 ……………………………… xM Changes:
y1 …………………… yN

1. Initialization
For all i, j,
F(i, 0) = 0
F(0, j) = 0

2. Termination
maxi F(i, N)
FOPT = max
maxj F(M, j)
Slide from Serafim Batzoglou
Dan Jurafsky

The Local Alignment Problem

Given two strings x = x1……xM,

y = y1……yN

Find substrings x’, y’ whose similarity

(optimal global alignment value)
is maximum

x = aaaacccccggggtta
y = ttcccgggaaccaacc Slide from Serafim Batzoglou
Dan Jurafsky

The Smith-Waterman algorithm

Idea: Ignore badly aligning regions

Modifications to Needleman-Wunsch:

Initialization:F(0, j) = 0
F(i, 0) = 0

0
Iteration: F(i, j) = max F(i – 1, j) – d
F(i, j – 1) – d
F(i – 1, j – 1) + s(xi, yj)
Slide from Serafim Batzoglou
Dan Jurafsky

The Smith-Waterman algorithm

Termination:
1. If we want the best local alignment…

FOPT = maxi,j F(i, j)

Find FOPT and trace back

2. If we want all local alignments scoring > t

?? For all i, j find F(i, j) > t, and trace back?

Complicated by overlapping local alignments Slide from Serafim Batzoglou

Dan Jurafsky

Local alignment example

A T T A T C
X = ATCAT 0 0 0 0 0 0 0
Y = ATTATC A 0
Let: T 0
m = 1 (1 point for match) C 0
d = 1 (-1 point for del/ins/sub)
A 0
T 0
Dan Jurafsky

Local alignment example

A T T A T C
X = ATCAT 0 0 0 0 0 0 0
Y = ATTATC A 0 1 0 0 1 0 0
T 0 0 2 1 0 2 0
C 0 0 1 1 0 1 3
A 0 1 0 0 2 1 2
T 0 0 2 0 1 3 2
Dan Jurafsky

Local alignment example

A T T A T C
X = ATCAT 0 0 0 0 0 0 0
Y = ATTATC A 0 1 0 0 1 0 0
T 0 0 2 1 0 2 0
C 0 0 1 1 0 1 3
A 0 1 0 0 2 1 2
T 0 0 2 0 1 3 2
Dan Jurafsky

Local alignment example

A T T A T C
X = ATCAT 0 0 0 0 0 0 0
Y = ATTATC A 0 1 0 0 1 0 0
T 0 0 2 1 0 2 0
C 0 0 1 1 0 1 3
A 0 1 0 0 2 1 2
T 0 0 2 0 1 3 2
 Minimum Edit
Distance

Minimum Edit Distance in

Computational Biology