The role of lexical analyzer

token
Source Lexical To semantic
program Parser analysis
Analyzer
getNextToken

Symbol
table
Regular expressions
Ɛ is a regular expression, L(Ɛ) = {Ɛ}
If a is a symbol in ∑then a is a regular
expression, L(a) = {a}
(r) | (s) is a regular expression denoting the
language L(r) ∪ L(s)
 (r)(s) is a regular expression denoting the
language L(r)L(s)
(r)* is a regular expression denoting (L9r))*
(r) is a regular expression denting L(r)
Regular definitions
d1 -> r1
d2 -> r2
…
dn -> rn

 Example:
letter_ -> A | B | … | Z | a | b | … | Z | _
digit -> 0 | 1 | … | 9
id -> letter_ (letter_ | digit)*
Extensions
One or more instances: (r)+
Zero of one instances: r?
Character classes: [abc]

Example:
letter_ -> [A-Za-z_]
digit -> [0-9]
id -> letter_(letter|digit)*
Recognition of tokens
Starting point is the language grammar to
understand the tokens:
stmt -> if expr then stmt
| if expr then stmt else stmt
|Ɛ
expr -> term relop term
| term
term -> id
| number
Recognition of tokens (cont.)
The next step is to formalize the patterns:
digit -> [0-9]
Digits -> digit+
number -> digit(.digits)? (E[+-]? Digit)?
letter -> [A-Za-z_]
id -> letter (letter|digit)*
If -> if
Then -> then
Else -> else
Relop -> < | > | <= | >= | = | <>
We also need to handle whitespaces:
ws -> (blank | tab | newline)+
Transition diagrams
Transition diagram for relop
Transition diagrams (cont.)
Transition diagram for reserved words and
identifiers
Transition diagrams (cont.)
Transition diagram for unsigned numbers
Transition diagrams (cont.)
Transition diagram for whitespace
Architecture of a transition-
diagram-based lexical analyzer
TOKEN getRelop()
{
TOKEN retToken = new (RELOP)
while (1) { /* repeat character processing until a
return or failure occurs */
switch(state) {
case 0: c= nextchar();
if (c == ‘<‘) state = 1;
else if (c == ‘=‘) state = 5;
else if (c == ‘>’) state = 6;
else fail(); /* lexeme is not a relop */
break;
case 1: …
…
case 8: retract();
[Link] = GT;
return(retToken);
}
Lexical Analyzer Generator -
Lex
Lex Source Lexical [Link].c
program Compiler
lex.l

[Link].c
C [Link]
compiler

Sequence
Input stream [Link]
of tokens
Structure of Lex programs

declarations
%%
translation rules Pattern {Action}
%%
auxiliary functions
Example
%{
Int installID() {/* funtion to
/* definitions of manifest constants
install the lexeme, whose first
LT, LE, EQ, NE, GT, GE, character is pointed to by
IF, THEN, ELSE, ID, NUMBER, RELOP */ yytext, and whose length is
%} yyleng, into the symbol table
and return a pointer thereto
*/
/* regular definitions
}
delim [ \t\n]
ws {delim}+
Int installNum() { /* similar to
letter [A-Za-z]
installID, but puts numerical
digit [0-9] constants into a separate
id {letter}({letter}|{digit})* table */
number {digit}+(\.{digit}+)?(E[+-]?{digit}+)? }

%%
{ws} {/* no action and no return */}
if {return(IF);}
then {return(THEN);}
else {return(ELSE);}
{id} {yylval = (int) installID(); return(ID); }
{number} {yylval = (int) installNum();
return(NUMBER);}
…
Finite Automata
Regular expressions = specification
Finite automata = implementation

A finite automaton consists of

An input alphabet 
A set of states S
A start state n
A set of accepting states F  S
A set of transitions state input state

16
Finite Automata
Transition
s1 a s2
Is read
In state s1 on input “a” go to state s2

If end of input

If in accepting state => accept, othewise =>
reject
If no transition possible => reject

17
Finite
A state
Automata State Graphs

• The start state

• An accepting state

a
• A transition

18
A Simple Example
A finite automaton that accepts only “1”

A finite automaton accepts a string if we can follow transitions labeled with the characters in the string from the start to some accepting state

19
Another Simple Example
A finite automaton accepting any number of 1’s followed by a single 0
Alphabet: {0,1}

Check that “1110” is accepted but “110…” is not

20
And Another
Alphabet {0,1}
Example
What language does this recognize?

1 0

0 0

1
1

21
And Another Example
Alphabet still { 0, 1 }
1

The operation of the automaton is not

completely defined by the input
On input “11” the automaton could be in either
state
22
Epsilon Moves
Another kind of transition: -moves

A B

• Machine can move from state A to state B

without reading input

23
Deterministic and
Nondeterministic Automata
Deterministic Finite Automata (DFA)
One transition per input per state
No -moves
Nondeterministic Finite Automata (NFA)
Can have multiple transitions for one input in a
given state
Can have -moves
Finite automata have finite memory
Need only to encode the current state

24
Execution of Finite Automata
A DFA can take only one path through the
state graph
Completely determined by input

NFAs can choose

Whether to make -moves
Which of multiple transitions for a single input
to take

25
Acceptance of NFAs
An NFA can get into multiple states
1

0 1

• Input: 1 0 1

• Rule: NFA accepts if it can get in a final state

26
NFA vs. DFA (1)
NFAs and DFAs recognize the same set of
languages (regular languages)

DFAs are easier to implement

There are no choices to consider

27
NFA vs. DFA (2)
For a given language the NFA can be simpler
than the DFA
1
0 0
NFA
0

1 0
0 0
DFA
1
1
• DFA can be exponentially larger than NFA

28
Regular Expressions to Finite
Automata
High-level sketch

NFA

Regular
expressions DFA

Lexical Table-driven
Specification Implementation of DFA

29
Regular Expressions to NFA (1)
For each kind of rexp, define an NFA
Notation: NFA for rexp A

• For 


• For input a
a

30
Regular Expressions to NFA (2)
For AB

A  B

• For A | B
B 


 A

31
Regular Expressions to NFA (3)
For A*


A


32
Example of RegExp -> NFA
conversion
Consider the regular expression
(1 | 0)*1
The NFA is


 C 1 E 
A B G 1
 0 F H  I J
 D 


33
Next
NFA

Regular
expressions DFA

Lexical Table-driven
Specification Implementation of DFA

34
NFA to DFA. The Trick
Simulate the NFA
Each state of resulting DFA
= a non-empty subset of states of the NFA
Start state
= the set of NFA states reachable through -
moves from NFA start state
Add a transition S a S’ to DFA iff
S’ is the set of NFA states reachable from the
states in S after seeing the input a
 considering -moves as well
35
NFA -> DFA Example


 C 1 E 
A B G 1
 0 F H  I J
 D 

0
0 FGABCDHI
ABCDHI 0 1
1
1 EJGABCDHI

36
NFA to DFA. Remark
An NFA may be in many states at any time

How many different states ?

If there are N states, the NFA must be in

some subset of those N states

How many non-empty subsets are there?

2N - 1 = finitely many, but exponentially many

37
Implementation
A DFA can be implemented by a 2D table T
One dimension is “states”
Other dimension is “input symbols”
For every transition Si a Sk define T[i,a] = k
DFA “execution”
If in state Si and input a, read T[i,a] = k and
skip to state Sk
Very efficient

38
Table Implementation of a DFA
0
0 T
S 0 1
1
1 U

0 1
S T U
T T U
U T U

39