07/23/2020
Lexical Analysis
Zakia Zinat Choudhury
Lecturer
Department of Computer Science & Engineering
University of Rajshahi
The Role of Lexical Analyzer
Lexical analyzer is the first phase of a compiler. It is also known as a scanner. So,
it’s main job is to read the input characters of the source program and group
them into lexemes, and produce a sequence of tokens for each lexeme in the
source program as an output.
Tokens
Lexemes
Source Lexical Syntax
Program Analyzer Analyzer
Request for Tokens
Figure: Interaction between the Lexical analyzer and the Syntax analyzer
2
Zakia Zinat Choudhury, Lecturer, Dept. of CSE, RU
1
07/23/2020
The Role of Lexical Analyzer
Lexical analyzers sometimes are divided into two processes:
a) Scanning consists of the simple processes that do not require tokenization of
the input, such as deletion of comments and compaction of consecutive
whitespace characters into one.
b) Lexical analysis proper is the more complex portion, where the scanner
produces the sequence of tokens as output.
3
Zakia Zinat Choudhury, Lecturer, Dept. of CSE, RU
Why lexical analysis and syntax analysis phases
are separated?
Simplicity of design is the most important consideration. The separation of
lexical and syntax analysis often allows us to simplify at least one of these
tasks.
Compiler efficiency is improved. A separate lexical analyzer allows us to apply
specialized techniques that serve only the lexical task, not the job of parsing.
Compiler portability is enhanced. Input-device-specific peculiarities can be
restricted to the lexical analyzer.
4
Zakia Zinat Choudhury, Lecturer, Dept. of CSE, RU
2
07/23/2020
Convert the source code into stream of tokens
Removing white spaces
Removing the comments
Functions of
Lexical Analyzer Recognizing identifiers and keywords
Recognizing of constants
Show error when the lexeme does not match any
patterns
5
Zakia Zinat Choudhury, Lecturer, Dept. of CSE, RU
Lexemes, Patterns and Tokens
Lexeme:
A lexeme is a sequence of characters in the source program that is matched by
the pattern for a token.
Pattern:
A pattern is a description of the form or rule that describes the set of strings.
Token:
A token is a set of strings over source alphabets. Also a token is a pair consisting
of a token name and an optional attribute value.
Typical tokens are,
1) Identifiers 2) keywords 3) operators 4) special symbols 5)constants
6
Zakia Zinat Choudhury, Lecturer, Dept. of CSE, RU
3
07/23/2020
When more than one lexeme can match a pattern,
the lexical analyzer must provide the subsequent
compiler phases additional information about the
particular lexeme that matched.
For each lexeme, the lexical analyzer produces as
output a token of the form
Attributes for Tokens (token-name, attribute-value)
❖ token-name is an abstract symbol that is used
during syntax analysis
❖ attribute-value points to an entry in the symbol
table for this token.
7
Zakia Zinat Choudhury, Lecturer, Dept. of CSE, RU
An alphabet is any finite set of symbols.
Typical examples of symbols are letters, digits,
and punctuation. The set {0,1} is the binary alphabet.
A string over an alphabet is a finite sequence of
symbols drawn from that alphabet.
For example, banana is a string of length six. The
empty string, denoted ꜫ, is the string of length zero.
Specification of Tokens
A language is any countable set of strings over some
fixed alphabet.
Abstract languages like Ø, the empty set, or {ꜫ},
the set containing only the empty string, are languages
under this definition.
8
Zakia Zinat Choudhury, Lecturer, Dept. of CSE, RU
4
07/23/2020
Terms for Parts of Strings
Prefix
Suffix
Substring
Proper prefixes, suffixes, substrings
Subsequence
9
Zakia Zinat Choudhury, Lecturer, Dept. of CSE, RU
Operations on Languages
Union of two languages L and M is written as
L U M = {s | s is in L or s is in M}
Concatenation of two languages L and M is written as
LM = {st | s is in L and t is in M}
The Kleene Closure of a language L is written as
L* = Zero or more occurrence of language L
The Positive Closure of a language L is written as
L+ = One or more occurrence of language L
10
Zakia Zinat Choudhury, Lecturer, Dept. of CSE, RU
5
07/23/2020
Regular Expressions
Regular expressions have the capability to express finite languages by defining
a pattern for finite strings of symbols.
The grammar defined by regular expressions is known as regular grammar.
The language defined by regular grammar is known as regular language.
11
Zakia Zinat Choudhury, Lecturer, Dept. of CSE, RU
Regular Expressions’ Operations
If r and s are regular expressions denoting the languages L(r) and L(s), then
Union : (r)|(s) is a regular expression denoting L(r) U L(s)
Concatenation : (r)(s) is a regular expression denoting L(r)L(s)
Kleene closure : (r)* is a regular expression denoting (L(r))*
(r) is a regular expression denoting L(r)
12
Zakia Zinat Choudhury, Lecturer, Dept. of CSE, RU
6
07/23/2020
Example of Lexical Analyzer
int position , rate, initial
Symbol Table Manager
position = rate + initial *60; Serial no Variable Name Variable Type
1 position int
2 rate int
Lexical Analyzer
3 initial int
Stream of token
<id,1> <=> <id,2> <+> <id,3> <*> <60>
Zakia Zinat Choudhury, Lecturer, Dept. of CSE, RU 13
Example of Lexical Analyzer
1. int x1;
x=23;
2. /*find the total value x and y*/
int x, y, sum;
sum = x + y ;
printf(“Total = %d\n”, sum);
Zakia Zinat Choudhury, Lecturer, Dept. of CSE, RU 14
7
07/23/2020
Lexical Error
Lexical error is a sequence of characters that does not match the
pattern of any token.
Lexical phase error can be:
Spelling error.
Exceeding length of identifier or numeric constants.
Appearance of illegal characters.
To remove the character that should be present.
To replace a character with an incorrect character.
Transposition of two characters.
15
Zakia Zinat Choudhury, Lecturer, Dept. of CSE, RU
Transition Diagrams
As an intermediate step in the construction of a
lexical analyzer, patterns are converted into
stylized flowcharts, called “transition diagrams”.
Transition diagrams have a collection of nodes
or circles, called states. Each state represents a
condition that could occur during the process of
scanning the input looking for a lexeme that
matches one of several patterns.
Edges are directed from one state of the
transition diagram to another.
Figure: Transition Diagram of Relation Operator
16
Zakia Zinat Choudhury, Lecturer, Dept. of CSE, RU
8
07/23/2020
Finite Automata
Finite Automata(FA) is the simplest machine to recognize patterns.
Finite automata come in two flavors:
(a) Nondeterministic finite automata (NFA) have no restrictions on the labels
of their edges. A symbol can label several edges out of the same state, and
Ꜫ, the empty string, is a possible label.
(b) Deterministic finite automata (DFA) have, for each state, and for each
symbol of its input alphabet exactly one edge with that symbol leaving that
state.
17
Zakia Zinat Choudhury, Lecturer, Dept. of CSE, RU
