COMPILERS BASIC COMPILER FUNCTIONS A compiler accepts a program written in a high level language as input and produces its machine language equivalent as output. For the purpose of compiler construction, a high level programming language is described in terms of a grammar. This grammar specifies the formal description of the syntax or legal statements in the language. Example: Assignment statement in Pascal is defined as: < variable > : = < Expression > The compiler has to match statement written by the programmer to the structure defined by the grammars and generates appropriate object code for each statement. The compilation process is so complex that it is not reasonable to implement it in one single step. It is partitioned into a series of sub-process called phases. A phase is a logically cohesive operation that takes as input one representation of the source program and produces an output of another representation. The basic phases are - Lexical Analysis, Syntax Analysis, and Code Generation. Lexical Analysis: It is the first phase. It is also called scanner. It separates characters of the source language into groups that logically belong together. These groups are called tokens. The usual tokens are: Keyword: Identifiers: Operator symbols: Punctuation symbols:
such as DO or IF, such as x or num, such as <, =, or, +, and such as parentheses or commas.
The output of the lexical analysis is a stream of tokens, which is passed to the next phase; the syntax analyzer or parser. Syntax Analyzer: It groups tokens together into syntactic structure. For example, the three tokens representing A + B might be grouped into a syntactic structure called as expression. Expressions might further be combined to form statements. Often the syntactic structures can be regarded as a tree whose leaves are the tokens. The interior nodes of the tree represent strings of token that logically belong together. Fig. 1 shows the syntax tree for READ statement in PASCAL (read) (id - list) READ
(
id {value}
)
Fig. 1 Syntax Tree Code Generator: It produces the object code by deciding on the memory locations for data, selecting code to access each datum and selecting the registers in which each computation is to be done. Designing a code generator that produces truly efficient object program is one of the most difficult parts of compiler design. In the following sections we discuss the basic elements of a simple compilation
1
process, illustrating this application to the example program in fig. 2. PROGRAM STATS VAR SUM, SUMSQ, I, VALUE, MEAN, VARIANCE : INTEGER BEGIN SUM :=0; SUMSQ : = 0 ; FOR I : = 1 to 100 Do BEGIN READ (VALUE) ; SUM : = SUM + VALUE ; SUMSQ : = SUMSQ + VALUE * VALUE END; MEAN : = SUM DIV 100; VARIANCE : = SUMSQ DIV 100 - MEAN * MEAN ; WRITE (MEAN, VARIANCE) END
Fig. 2 Pascal Program GRAMMARS A grammar for a programming language is a formal description of the syntax of programs and individual statements written in the language. The grammar does not describe the semantics or memory of the various statements. To differentiate between syntax and semantics consider the following example: VAR X, Y : REAL I : INTEGER X:=I+Y;
VAR
I, J, K : INTEGER I:= J+K ;
Fig .3 These two programs statement have identical syntax. Each is an assignment statement; the value to be assigned is given by an expression that consists of two variable names separated by the operator '+'. The semantics of the two statements are quite different. The first statement specifies that the variables in the expressions are to be added using integer arithmetic operations. The second statement specifies a floating-point addition, with the integer operand 2 being connected to floating point before adding. The difference between the statements would be recognized during code generation. Grammar can be written using a number of different notations. Backus-Naur Form (BNF) is one of the methods available. It is simple and widely used. It provides capabilities that are different for most purposes. A BNF grammar consists of a set of rules, each of which defines the syntax of some construct in the programming language. A grammar has four components. They are:
1. A set of tokens, known as terminal symbols non-enclosed in bracket. Example: READ, WRITE
2
2. A set of terminals. The character strings enclosed between the angle brackets (<, >) are called terminal symbols. These are the names of the constructs defined in the grammar. 3. A set of productions where each production consists of a non-terminal called the left side of the production, as "is defined to be" (:: = ), and a sequence of token and/or non-terminal, called the right side of the product. Example: < reads > : : = READ
. 4. A designation of one of the non-terminals as the start symbol. This rule offers two possibilities separated by the symbol, for the syntax of an < id - list > may consist simply of a token id (the notation id denotes an identifier that is recognized by the scanner). The second syntax. Example:
ALPHA ALPHA, BETA
ALPHA is an < id - list > that consist of another < id - list > ALPHA, followed by a comma, followed by an id BETA. Tree: It is also called parse tree or syntax tree. It is convenient to display the analysis of a source statement in terms of a grammar as a tree. Example: READ (VALUE) GRAMMAR: (read) : : = READ ( < id -list>) Example: Assignment statement: SUM : = 0 ; SUM : = + VALUE ; SUM : = - VALUE ; Grammar: < assign > : : = id : = < exp > < exp > : : = < term > | < exp > - < term > < term > : : = < factor > | < term > * < factor > | < term > DIV < factor > < factor > : : = id | int | ( < exp > ) Assign consists of an id followed by the token : = , followed by an expression Fig. 4(a). Show the syntax tree. Expressions are sequence of connected by the operations + and - Fig. 4(b). Show the syntax tree. Term is a sequence of < factor > S connected by * and DIV Fig. 4(c).
A factor may consists of an identifies id or an int (which is also recognized by the scan) or an < exp > enclosed in parenthesis. Fig. 4(d). < assign > id := {variance } Fig. 4 (a)
< exp > < term >
+
< exp >
Fig. 4 (b)
3
< term > | < factor >
Dir X
factor | < term >
id
< factor >
int Id
Fig.4 (c) (< exp > )
Fig. 4 (d)
Fig. 4 Parse Trees For the statement Variance : = SUMSQ Div 100 - MEAN * MEAN ; The list of simplified Pascal grammar is shown in fig.5. 1. < prog >
2. 3. 4. 5.
6. 7. 8. 9. 10. 11. 12.
: : = PROGRAM < program > VAR BEGIN < stmt > - list > END. < prog - name >: : = id < dec - list > : : = < dec > | < dec - list > ; < dec > < dec > : : = < id - list > : < type > < type > : : = integer < id - list > : : = id | < id - list > , id : : = < stmt > ; < stmt > < stmt > : : = < assign > | | < write > | < for > < assign > : : = id : = < exp > < exp > : : = < term > | < exp > + < term > | < exp > - < term > < term > : : = < factor > | < term > | DIV < factor > : : = id ; int | (< exp >)
13. < READ > 14. < write > 15. < for > 16. < index - exp> 17. < body >
::= ::= ::= ::= ::=
READ ( < id - list >) WRITE ( < id - list >) FOR < idex - exp > Do < body > id : = < exp > To ( exp > < start > | BEGIN < start - list > END
Fig. 5 Simplified Pascal Grammar ( < prog >) | PROGRAM
< prog - name > VAR dec - list Id {STATS}
BEGIN
END
< dec > < stmt - list > < id - list >
(id - list)
,
;
: < type > ↓ INTEGER
< stmt > < write >
id {VARIANCE}
< stmt - list > ;
WRITE
( )
4
(id - list ) ; (id - list ) ,
id (MEAN)
< stmt - list >
id < stmt - list >
;
< assign > (id - list ) .
id {VARIANCE}
id id := {VARIANCE} < exp >
;
< assign > (id - list ) , , id {SUM} {SUMSQ}
id {I}
;
id {SMSQ} < stmt >
< start > id : = {mean} | |
< assign >
< assign > id := | | | | Div | | | term | >term> Div id : < exp > | factor | < factor >
factor < term > int
| { 0}
| {SUM} Next Page
int {0}
int id
|
* | | id | [MEAN] id {MEAN}
{100} | {100}
int
id {SUMSQ}
|
|
< for > FOR
Id {I}
Do
< body >
: = To BEGIN END | | < term > | | ; < stmt > | | | int int {I} {100} ; | | < stmy > | id := < read > (SUMSQ id : = {SUM} READ ( < id - list > ) < exp > + < term > < exp > + < term > id | | | {VALUE? < factor > < term > * | | | | | < factor >. id id | { value} | | {value} id id id {SUM} {SUMSQ} {value}
Fig. 6 Parse tree for the Program 1 Parse tree for Pascal program in fig.1 is shown in fig. 6 1 (a) Draw parse trees, according to the grammar in fig. 5 for the following S:
(a) ALPHA
< id - list >
5
| id { ALPHA } (b) ALPHA, BETA, GAMMA
< id - list >
< id - list >
< id - list >
id , {GAMMA}
id , {BETA}
id [ ALPHA ]
2 (a) Draw Parse tree, according to the grammar in fig. 5 for the following < exp > S :
(a) ALPHA + BETA
< exp > | < term > < term > | < factor > + < factor > | | id id {ALPHA}
{BETA}
(b) ALPHA - BETA + GAMMA < exp < exp >
-
< term > | < factor > id {ALPHA}
| id
term
< term > | < factor >
*
factor id
{GAMMA}
{BETA}
(c) ALPHA DIV (BETA + GAMMA) = DELTA < exp >
6
< exp >
-
< term >
< term > | < term >
< factor >
Div
< factor >
{DELTA}
< factor > ( < exp >
)
id {ALPHA} < exp >
+ < term >
< term >
factor
id
id
{BETA}
3.
{GAMMA}
Suppose the rules of the grammar for < exp > and < term > is as follows: < exp > :: = < term > | < exp > * < term> | < exp> Div < term > < term > :: = | < term > + < factor > | < term > - < factor > Draw the parse trees for the following:
(a) A1 + B1 (b) A1 - B1 * G1 (c) A1 + DIV (B1 + G1) - D1
(a) A1 + B1
< exp > | term
< term > factor
+
id {A1} (b) A1 - B1 * G1 < exp > | teerm teerm
-
< factor > | id {B1}
< factor >
factor term
*
factor 7
| id {G1}
id factor {A1} id {B1} (c) A1 DIV (B1 + A1) - D1 < exp > < exp >
DIV
< term > < factor > |
< term >
< term >
-
< factor >
< factor >
id {A1}
(
id {D1}
< exp >
)
< term > < term > < factor >
+
< factor > id {G1}
id {B1} LEXICAL ANALYSIS Lexical Analysis involves scanning the program to be compiled. Scanners are designed to recognize keywords, operations, identifiers, integer, floating point numbers, character strings and other items that are written as part of the source program. Items are recognized directly as single tokens. These tokens could be defined as a part of the grammar. Example: : : = | | : : = A | B | C | . . . | Z : : = 0 | 1 | 2 | . . . | 9 In a such a case the scanner world recognize as tokens the single characters A, B, . . . Z,, 0, 1, . . . 9. The parser could interpret a sequence of such characters as the language construct < ident >. Scanners can perform this function more efficiently. There can be significant saving in compilation time since large part of the source program consists of multiple-character identifiers. It is also possible to restrict the length of identifiers in a scanner than in a passing notion. The scanner generally recognizes both single and multiple character tokens directly. The scanner output consists of sequence of tokens. This token can be considered to have a fixed length code. The fig. 7 gives a list of integer code for each token for the program in fig. 5 in such a type of coding scheme, the PROGRAM is represented by the integer value 1, VAR has the integer value 2 and so on.
Token Program
VAR
BEGIN END END INTEGER FOR
8
Code
1
2
3
4
5
6
7
Token READ WRITE Token := +
To -
Do K
; DIV
: (
, )
Token Code
:= 15
+ 16
17
K 18
DIV 17
( 20
) 21
Token Code
Id 22
Int 23 Fig. 7 Token Coding Scheme
For a keyword or an operator the token loading scheme gives sufficient information. In the case of an identifier, it is also necessary to supply particular identifier name that was scanned. It is true for the integer, floating point values, character-string constant etc. A token specifier can be associated with the type of code for such tokens. This specifier gives the identifier name, integer value, etc., that was found by the scanner. Some scanners enter the identifiers directly into a symbol table. The token specifier for the identifiers may be a pointer to the symbol table entry for that identifier. The functions of a scanner are: The entire program is not scanned at one time. Scanner is a operator as a procedure that is called by the processor when it needs another token. Scanner is responsible for reading the lines of the source program and possible for printing the source listing. The scanner, except for printing as the output listing, ignores comments. Scanner must look into the language characteristics.
Example: FOTRAN : : : PASCAL : : :
Columns 1 - 5 Statement number Column 6 Continuation of line Column 7 . 22 Program statement Blanks function as delimiters for tokens Statement can be continued freely End of statement is indicated by ; (semi column)
Scanners should look into the rules for the formation of tokens.
Example: 'READ': Should not be considered as keyword as it is within quotes. i.e., all string within quotes should not be considered as token.
Blanks are significant within the quoted string. Blanks has important factor to play in different language
Example 1: FORTRAN Statement: Do 10 I = 1, 100 ; Do is a key word, I is identifier, 10 is the statement number Statement: Do 10 I = 1 ;It is an identifier Do 10 I = 1 Note: Blanks are ignored in FORTRAN statement and hence it is a assignment statement. In this case the scanner must look ahead to see if there is a comma (,) before it can decide in the proper identification of the characters
9
Do. Example 2: In FORTRAN keywords may also be used as an identifier. Words such as IF, THEN, and ELSE might represent either keywords or variable names.
IF (THEN .EQ ELSE) THEN IF = THEN ELSE THEN = IF ENDIF Modeling Scanners as Finite Automata Finite automatic provides an easy way to visualize the operation of a scanner. Mathematically, a finite automation consists of a finite set of states and a set of transition from one state to another. Finite automatic is graphically represented. It is shown in fig, State is represented by circle. Arrow indicates the transition from one state to another. Each arrow is labeled with a character or set of characters that can be specified for transition to occur. The starting state has an arrow entering it that is not connected to anything else.
1 State
Final State Fig. 8
Transition
Example: Finite automata to recognize tokens is gives in fig. 9. The corresponding algorithm is given in fig. 10
0-9
1
A-Z B
2 2 1
3
A-Z
Fig. 9 Get first Input-character If Input-character in [ 'A' . . ' Z' ] then begin while Input - character in [ 'A' . . 'Z', ' 0'. . ' 9' ] do begin get next input - character End {while} end {if first is [ 'A' .. ' Z' ] } else return (token-error)
Fig. 10 SYNTACTIC ANALYSIS
10
During syntactic analysis, the source programs are recognized as language constructs described by the grammar being used. Parse tree uses the above process for translation of statements, Parsing techniques are divided into two general classes: -- Bottom up and -- Top down. Top down methods begin with the rule of the grammar that specifies the goal of the analysis ( i.e., the root of the tree), and attempt to construct the tree so that the terminal nodes match the statement being analyzed. Bottom up methods begin with the terminal nodes of the tree and attempt to combine these into successively high - level nodes until the root is reached. OPERATOR PRECEDENCE PARSING The bottom up parsing technique considered is called the operator precedence method. This method is loaded on examining pairs of consecutive operators in the source program and making decisions about which operation should be performed first. Example: A + B * C - D
(1)
The usual procedure of operation multiplication and division has higher precedence over addition and subtraction. Now considering equation (1) the two operators (+ and *), we find that + has lower precedence than *. This is written as +⋖ * [+ has lower precedence *] Similarly ( * and - ), we find that * ⋗ - [* has greater precedence -]. The operation precedence method uses such observations to guide the parsing process.
PROGRAM VAR BEGIN END INTEGER FOR READ WRITE TO DO ; :
≐
≐
⋖⋖⋖
≐≐ ⋗⋗
⋗
⋖ ⋗ ⋗ ⋗⋗ ⋗ ⋗
⋖⋖ ⋖ ⋗
≐
⋖
⋖⋖⋖ ⋖⋖⋖
⋗
IntId
-+: =,
WRITEREASFOR INTEGER
ENDENDBEGINVAR
⋖ ⋗
()DIV*
(2) ::DOTO
A+B*C -D
⋖ ⋖ ⋖
<
⋗ ≐ ≐ ⋗ ⋗⋖ ⋗
⋖⋖ ⋖
⋖⋖⋖⋖
⋖
⋖ ⋖ ⋖ ⋖
11
, := + -
≐ ⋗ ⋗⋗⋗ ⋗⋗⋗ ⋗⋗⋗
⋗⋗ ⋗⋗ ⋗⋗ ⋗⋗ ⋗⋗ ⋗⋗
* DIV ) ( id Int
⋗
⋗ ⋗ ⋗ ⋗
⋖⋖ ⋗⋗ ⋗⋗ ⋗⋗
⋗⋗
⋗ ⋗ ⋗ ⋗
⋗ ⋗ ⋗ ⋗
⋗⋗ ⋗⋗
⋗⋗ ⋗ ⋗ ⋗⋗ ⋗
⋖
⋗⋗ ⋗⋗ ⋗⋗
⋗ ≐ ⋗⋗ ⋗⋗
⋖ ⋖ ⋖ ⋗
⋖ ⋖ ⋖ ⋗
⋖ ⋖ ⋗ ⋖ ⋗ ⋗⋖
⋗ ⋗ ⋖ ⋗
⋗⋖ ⋗ ⋗⋖ ⋗ ⋖⋖ ≐ ⋗ ⋗
⋗⋗ ⋗⋗
≐ ⋖⋖ ⋖⋖ ⋖⋖ ⋖⋖ ⋖⋖ ⋖⋖
⋗ ⋗
Fig 11 Precedence Matrix for the Grammar for fig 5 Equation (2) implies that the sub expression B * C is to be computed before either of the other operations in the expression is performed. In times of the parse tree this means that the * operation appears at a lower level than does either + or -. Thus a bottom up parses should recognize B * C by interpreting it in terms of the grammar, before considering the surrounding terms. The first step in constructing an operator-precedence parser is to determine the precedence relations between the operators of the grammar. Operator is taken to mean any terminal symbol (i.e., any token). We also have precedence relations involving tokens such as BEGIN, READ, id and ( . For the grammar in fig. 5, the precedence relations is given in the fig. 11. Example: PROGRAM
≐
VAR ; These two tokens have equal precedence
Begin ⋖ FOR ; BEGIN has lower precedence over FOR. There are some values which do not follows precedence relations for comparisons. Example: ; ⋗ END and END ⋗ ; i.e., when ; is followed by END, the ' ; ' has higher precedence and when END is followed by ; the END has higher precedence. In all the statements where precedence relation does not exist in the table, two tokens cannot appear together in any legal statement. If such combination occurs during parsing it should be recognized as error. Let us consider some operator precedence for the grammar in fig. 5. Example:
Pascal Statement:
BEGIN READ (VALUE);
These Pascal statements scanned from left to right, one token at a time. For each pair of operators, the precedence relation between them is determined. Fig. 12(a) shows the parser that has identified the portion of the statement delimited by the precedence relations ⋖ and ⋗ to be interpreted in terms of the grammar. (a) . . . BEGIN READ ( id ) ⋖ ≐⋖ ⋗ (b) . . . BEGIN READ ( < N1 > ) ; ⋖ ≐ ≐ ⋗ (c) . . . BEGIN < N2 > ; (d) ... READ
< N2 > (
)
12
id (VALUE)
Fig. 12 According to the grammar id may be considered as < factor > . (rule 12), (rule 9) or a < id-list > (rule 6). In operator precedence phase, it is not necessary to indicate which non-terminal symbol is being recognized. It is interpreted as non-terminal < N1 >. Hence the new version is shown in fig. 12(b). An operator-precedence parser generally uses a stack to save token that have been scanned but not yet parsed, so it can reexamine them in this way. Precedence relations hold only between terminal symbols, so < N1 > is not involved in this process and a relationship is determined between (and). READ () corresponds to rule 13 of the grammar. This rule is the only one that could be applied in recognizing this portion of the program. The sequence is simply interpreted as a sequence of some interpretation < N2 >. Fig. 12(c) shows this interpretation. The parser tree is given in fig. 12(d). Note: (1) The parse tree in fig. 1 and fig. 12 (d) are same except for the name of the non-terminal symbols involved. (2) The name of the non-terminals is arbitrarily chosen. Example:
VARIANCE ; = SUMSQ DIV 100 - MEAN * MEAN (i) . . id 1 : = id 2 Div . . ⋖ ≐ ⋖ ⋗
.
(ii) . . . id 1 : = Div int ⋖ ≐ ⋖ ⋖ ⋗
{SUMSQ}
(iii) . . . id 1 : = Div -
⋖
≐
⋖
⋗
{SUMSQ}
int {100}
(iv) . . . . id 1 : = - id 3 * ⋖ ≐ ⋖ ⋖ ⋗ DIV id2 int {SUMSQ} {100} v) . . . . id 1 : = - * id 4 ⋖ ≐ ⋖ ⋖ ⋖ ⋗
;
id 3 {MEAN}
(vi) . . . id 1 : = - * ⋖ ≐ ⋖ ⋖ ⋗ id 4 {MEAN}
13
(vii) . . . id 1 : = - ⋖ ≐ ⋖ ⋗
*
id 3 {MEAN} (viii) . . . id : = ⋖ ≐ ⋗ (ix)
id 4 {MEAN}
-
. . .
id 1 := {VARIANCE}
DIV id 2 {SUMSQ}
* int id 3 id 4 {100} {MEAN} {MEAN}
SHIFT REDUCE PARSING The operation procedure parsing was developed to shift reduce parsing. This method makes use of a stack to store tokens that have not yet been recognized in terms of the grammar. The actions of the parser are controlled by entries in a table, which is somewhat similar to the precedence matrix. The two main actions of shift reducing parsing are Shift: Push the current token into the stack. Reduce: Recognize symbols on top of the stack according to a rule of a grammar Example:
BEGIN
READ ( id ) . . .
Steps Token Stream 1. . . . BEGIN
Stack READ ( id ) . . .
Shi ft 2. . . . BEGIN
READ ( id ) Shi ft
BEGIN
14
3. . . . BEGIN
READ ( id ) . . . Shi ft
READ BEGIN
4. . . BEGIN
READ ( id ) . . . (
Shi ft
READ BEGIN
5. . . . BEGIN
READ ( id ) . . . id ( READ BEGIN
Shi ft 6. . . . BEGIN
READ ( id ) . . . Shi ft
. < id-list > ( READ BEGIN
Explanation 1. The parser shift (pushing the current token onto the stack) when it encounters BEGIN 2 to 4. The shift pushes the next three tokens onto the stack. 5. The reduce action is invoked. The reduce converts the token on the top of the stack to a non-terminal symbol from the grammar. 6. The shift pushes onto the stack, to be reduced later as part of the READ statement. Note: Shift roughly corresponds to the action taken by an operator – precedence parses when it encounters the relation ⋖ and ≐. Reduce roughly corresponds to the action taken when an operator precedence parser encounters the relation ⋗. RECURSIVE DESCENT PARSING Recursive-Descent is a top-down parsing technique. A recursive-descent parser is made up of a precedence for each non-terminal symbol in the grammar. When a precedence is called it attempts to find a sub-string of the input, beginning with the current token, that can be interpreted as the non-terminal with which the procedure is associated. During this process it may call other procedures, or call itself recursively to search for other non-terminals. If the procedure finds the non-terminal that is its goal, it returns an indication of success to its caller. It also advances the current-token pointer past the sub-string it has just recognized. If the precedence is unable to find a sub-string that can be interpreted as to the desired non-terminal, it returns an indication of failure. Example: < read > : : = READ ( < id - list > ) The procedure for < read > in a recursive descent parser first examiner the next two input, looking for READ and (. If these are found, the procedures for < read > then call the procedure for < id - list >. If that procedure succeeds, the < read > procedure examines the next input token, looking for). If all these tests are successful, the < read > procedure returns an indication of success. Otherwise the procedure returns a failure. There are problems to write a complete set of procedures for the grammar of fig. 15.
15
Example: The procedure for < id - list >, corresponding to rule 6 would be unable to decide between its alternatives since id and < id-list > can begin with id. : : = id | < idlist >, id If the procedure somehow decided to try the second alternative , it would immediately call itself recursively to find an . This causes unending chain. Top-down parsers cannot be directly used with a grammar that contains this kind of immediate left recursion. Similarly the problem occurs for rules 3, 7, 10 and 11. Hence the fig. 13 shows the rules 3, 6, 7, 10 and 11 modification. 3 6 7 10 11
< dec - list > : : < id - list > :: < stmt - list > : : < exp > :: = < term > : : =
= < dec > { ; } = id {; id } = < stmt > { ; < stmt > } < term > { + < term . | -- < term > } < factor > { + < factor > | Div < factor >.}
Fig. 13
Fig. 14 illustrates a recursive-descent parse of the READ statement: READ (VALUE); The modified grammar is considered in the procedure for the non-terminal and < id-list >. It is assumed that TOKEN contains the type of the next input token. PROCEDURE READ BEGIN ROUND : = FALSE If TOKEN + 8 { read } THEN BEGIN advance to next token IF TOKEN + 20 { ( } THEN BEGIN advance to next token IF IDLIST returns success THEN IF token = 21 { ) } THEN BEGIN FOUND : = TRUE advance to next token END { if ) } END { if READ } IF FOUND = TRUE THEN return success else failure end (READ)
Fig. 14 Procedure IDLIST begin FOUND = FALSE if TOKEN = 22 {id} then begin FOUND : = TRUE advance to Next token while (TOKEN = 14 {,}) and (FOUND = TRUE) do
16
begin advance to next token if TOKEN = 22 {id} then advance to next token else FOUND = FALSE End {while} End {if id} if FOUND : = TRUE then return success else return failure end {IDLIST} Fig. 15 The fig. 15 IDLIST procedure shows an error message if ( , ) is not followed by a id. It indicates the failure in the return statement. If the sequence of tokens such as " id, id " could be a legal construct according to the grammar, this recursive-descent technique would not work properly. Fig. 16 shows a graphic representation of the recursive parsing process for the statement being analyzed. (i)
In this part, the READ procedure has been invoked and has examined the tokens READ and ' ( " from the input stream (indicated by the dashed lines). (ii) In this part, the READ has called IDLIST (indicated by the solid line), which has examined the token id. (iii) In this part, the IDLIST has returned to READ indicating success; READ has then examined the input token. Note that the sequence of procedure calls and token examinations has completely defined the structures of the READ statement. The parser tree was constructed beginning at the root, hence the term top-down parsing.
(i)
REA D
(II)
READ (
REA D
READ (
(iii)
IDLIS T
REA D IDLIST READ (
id { Value }
id { Value
Fig. 16
Fig. 17 illustrates a recursive discard parse of the assignment statement. Variance: = SUNSQ DIVISION - MEAN * MEAN The fig. 17 shows the procedures for the non-terminal symbols that are involved in parsing this statement. Procedure ASSIGN begin FOUND = FALSE if TOKEN = 22 {id} then begin
17
advance to Next token if TOKEN = 15 {: =} then begin advance to next token if EXP returns success then FOUND : = TRUE end {if : =} if FOUND : = TRUE then return success else return failure end {ASSIGN} Procedure EXP begin FOUND = FALSE If TERM returns success then begin FOUND: = TRUE while ((TOKEN = 16 {+ } ) or (TOKEN = 17 { - } ) ) and (FOUND = TRUE) do begin advance to next token if TERM returns success then FOUND = FALSE end {while} end {if TERM} if FOUND : = TRUE then return success else return failure end {EXP} Procedure TERM begin FOUND : = FALSE If FACTOR returns success then begin FOUND : = TRUE while ((TOKEN = 18 { * }) or (TOKEN = 19 {DIV }) and (FOUND = TRUE) do begin advance to next token if TERM returns failure then FOUND : = FALSE end {while} end {if FACTOR} if FOUND : = TRUE then return success else return failure end {TERM} Procedure FACTOR
18
begin FOUND : = FALSE if (TOKEN = 22 { id } ) or (TOKEN = 23 {int } ) then begin FOUND : = TRUE advance to next token end { if id or int } else if TOKEN = 20 { ( } then begin advance to next token if EXP returns success then if TOKEN = 21 { ) } then begin (FOUND = TRUE) advance to next token end { if ) } end {if ( } if FOUND : = TRUE then return success else return failure end {FACTOR} Fig. 17 Recursive-Descent Parse of an Assignment Statement A step-by-step representation of the procedure calls and token examination is shown in fig. 1
(i)
ASSIGN
id 1 := { VARIANCE }
(ii)
(iii)ASSIGN
ASSIGN
id 1 : = { VARIANCE }
EXP
id 1 : {VARIANCE}
=
EXP
TERM
(iv)
(v)
id 1 := {VARIANCE}
EXP
id 1 := {VARIANCE}
TERM
FACTOR
(vi)
ASSIGN
EXP
TERM
FACTOR
DIV
ASSIGN
id 1 := {VARIANCE} TERM -
FACTOR
FACTOR
EXP
TERM
DIV
FACTOR
19
id 2
id 2
{SUMSQ}
int
{SUMSQ}
(vii)
id
{100}
2
{SUMSQ}
int {100}
ASSIGN
id 1 := {VARIANCE}
EXP -
TERM
FACTOR
TERM
FACTOR
FACTOR
DIV
id 2
int
{SUMSQ}
id 3
{100}
(viii)
{MEANS}
ASSIGN
id 1 := (VARIANCE} TERM
FACTOR
EXP
FACTOR
TERM
FACTOR
FACTOR
* DIV
id 2 {SUMSQ}
DIV
int {100}
id 3 {MEANS}
id 4 {MEANS}
Fig. 18 Step by step Representation for Variance : = SUMSQ Div 100 - MEAN * Mean GENERATION OF OBJECT CODE After the analysis of system, the object code is to be generated. The code generation technique used in a set of routine, one for each rule or alternative rule in the grammar. The routines that are related to the meaning of he compounding construct in the language is called the semantic routines. When the parser recognizes a portion of the source program according to some rule of the grammar, the corresponding semantic routines are executed. These semantic routines generate object code directly and hence they are referred as code generation routines. The code generation routines that is discussed are designed for the use with the grammar in fig. .5. This grammar is used for code generations to emphasize the point that code generation techniques need not be associated with any particular parsing method. The parsing technique discussed in 1.3 does not follow the constructs specified by this grammar. The operator precedence method ignores certain non-terminal and the recursive-descent method must use slightly modified grammar. The code generation is for the SIC/XE machine. The technique use two data structure: (1) A List (2) A Stack List Count: A variable List count is used to keep a count of the number of items currently in the list. The token specifiers are denoted by ST (token)
20
Example:
id int
ST (id) ; name of the identifier ST (int) ; value of the integer, # 100
The code generation routines create segments of object code for the compiled program. A symbolic representation is given to these codes using SIC assembler language. LC (Location Counter): It is a counter which is updated to reflect the next variable address in the compiled program (exactly as it is in an assembler). Application Process to READ Statement:
(read) + JSUB WORD 1 WORD
< id - list > READ
(
XREAD VALUE
) {VALUE}
Fig. 19(a) Parse Tree for Read Using the rule of the grammar the parser recognizes at each step the left most sub-string of the input that can be interpreted. In an operator precedence parse, the recognition occurs when a sub-string of the input is reduced to some non-terminal . In a recursive-descent parse, the recognition occurs when a procedure returns to its caller, indicating success. Thus the parser first recognizes the id VALUE as an < id - list >, and then recognizes the complete statement as a < read >. The symbolic representation of the object code to be generated for the READ statement is as shown in fig. 19(b). This code consists of a call to a statement XREAD, which world be a part of a standard library associated with the compiler. The subroutine any program that wants to perform a READ operation can call XREAD. XREAD is linked together with the generated object program by a linking loader or a linkage editor. The technique is commonly used for the compilation of statements that perform voluntarily complex functions. The use of a subroutine avoids the repetitive generation of large amounts of in-line code, which makes the object program smaller. The parameter list for XREAD is defined immediately after the JSUB that calls it. The first word is the number of variable that will be assigned values by the READ. The following word gives the addresses of three variables. Fig. 19(c) shows the routines that might be used to accomplish the code generation.
1. < id - list > : : = id add ST (id) to list add 1 to List_count 2. < id - list > : : = < id - list >, id add ST (id) to list add 1 to LC List_Current 3. < read > : : = READ (< id - list >) generate [ + JSUB XREAD ] record external reference to XREAD generate [WORD List - count] for each item on list of do begin remove ST (ITEM) from list
21
generate [WORD ST (ITEM)] end List _count : = 0
Fig. 19 (c) Routine for READ Code Generation The first two statements (1) and (2) correspond to alternative structure for < id - list >, that is < id - list > : : = id | < id - list >, id. In each case the token specifies ST (id) for a new identifier being called to the < id - list > is inserted into the list used by the code-generation routine, and list-count is updated to reflect the insertion. After the entire < id-list > has been parsed, the list contains the token specifiers for all the identifiers that are part of the < id- list >. When the < read > statement is recognized, the token specifiers are removed from the list and used to generate the object code for the READ. Code-generation Process for the Assignment Statement Example: VARIANCE: = SUMSQ DIV 100 - MEAN * MEAN The parser tree for this statement is shown in fig. 20. Most of the work of parsing involves the analysis of the < exp > on the right had side of the " : = " statement.:
< assign > < exp > < exp >
< exp > (term)
< term > < term >
< term >
< factor >
< factor > < factor > id {VARIANCE}
:=
id { SUMSQ }
< factor > DIV
int {100}
_
id
{MEAN}
*
id
{MEAN}
Fig. 20 The parser first recognizes the id SUMSQ as a < factor > and < term > ; then it recognizes the int 100 as a < factor >; then it recognizes SUNSQ DIV 100 as a < term >, and so forth. The order in which the parts of the statements are recognized is the same as the order in which the calculations are to be performed. A code-generation routine is called for each portion of the statement is recognized.
Example; For a rule < term >1: : = < term >
2
* < factor > a code is to be
generated. The subscripts are used to distinguish between the two occurrences of < term > .
22
The code-generation routines perform all arithmetic operations using register A. Hence the multiple < term >2 * < factor > after multiplication is available in register A. Before multiplication one of the operand < term >2 must be located in A-register. The results after multiplication will be left in register A. So we need to keep track of the result left in register A by each segment of code that is generated. This is accomplished by extending the token-specifier idea to non-terminal nodes of the parse tree. The node specifier ST (< term1>) would be set to rA, indicating that the result of the completion is in register A. the variable REGA is used to indicate the highest level node of the parse tree when value is left in register A by the code generated so far. Clearly there can be only one such node at any point in the code-generation process. If the value corresponding to a node is not in register A, the specifier for the node is similar to a token specifier: either a pointer to a symbol table entry for the variable that contains the value or an integer constant. Fig. 21 shows the code-generation routine considering the A-register of the machine.
1.
2.
3.
4.
5.
6.
< assign > : : = id := < exp > GETA (< exp >) generate [ STA ST (id)] REGA : = null :: =< term > ST < exp > : = ST (< term >) if ST < exp > = rA then REGA : = < exp > < exp >1 : : = < exp >2 + < term > if SR (< exp >2) = rA then generate [ADD ST (< term >)] else if ST (< term >) = rA then generate [ADD ST (< exp >2)] else begin GETA (< EXP >2) generate [ADD ST(< term >)] end ST (< exp >1) : = rA REGA : = < exp >1 < exp >1 : : = < exp >2 - < term > if ST (< exp >2) = rA then generate [SUB ST (< term >)] else begin GETA (< EXP >2) generate [ SUB ST (< term >)] end SR (< exp >1) : = rA REGA : = < exp >1 < term > : : = < factor > ST (< term >) : = ST (< factor >) if ST () = rA then REGA : = < term > < term >1 : : = < term >2 * < factor > if ST (< term >2) = rA then
23
7.
9. 10.
generate [ MUL ST (< factor >)] else if S (< factor >) = rA then generate [ MUL ST (< term >2)] else begin GETA (< term >2) generate [ MUL SrT(< factor >)] end ST (< term >1) : = rA REGA : = < term >1 < term > : : = < term >2 DIV < factor > if SR (< term >2) = rA then generate [DIV ST(< factor >)] else begin GETA (< term >2) generate [ DIV ST (< factor >)] end SR (< term >1) : = rA REGA : = < term >1 < factor > : : = id ST (< factor >) : = ST (id) < factor > : : = int ST (< factor >) : = ST (int) < factor > : : = < exp > ST (< factor >) : = ST (< exp >) if ST (< factor >) = rA then REGA : = < factor >
Fig. 21 Code Generation Routines If the node specifies for either operand is rA, the corresponding value is already in register A, the routine simply generates a MUL instruction. The node specifier for the other operand gives the operand address for this MUL. Otherwise, the procedure GETA is called. The GETA procedure is shown in fig. 22. Procedure - GETA (NODE) begin if REGA = null then generate [LDA ST (NODE) ] else if ST (NODE) π rA then begin creates a new looking variable Tempi generate [STA Tempi] record forward reference to Tempi ST (REGA) : = Tempi Generate [LDA ST (NODE)] end (if ≠ rA) ST(NODE) : = rA REGA : = NODE end {GETA }
24
Fig. 22 The procedure GETA generates a LDA instruction to load the values associated to 2 into register A. Before loading the value into A-register, it confirms whether A is null. If it is not null it generates STA instruction to save the contents of register-A into Temp-variable. There can be number of Temp variable like Temp1, Temp2 . . . etc. The temporary variables used during a completion will be assigned storage location at the end of the object program. The node specifies for the node associated with the value previously in register A, indicated by REGA is reset to indicate the temporary variable used. After the necessary instructions are generated, the code-generation routine sets ST (< term >1) and REGA to indicate that the value corresponding to < terms >1 is now in register A. This completes the code-generation action for the * operation. The code-generation routine for ' + ' operation is the same as the ' * ' operation. The routine ' DIV ' and ' - ' are similar except that for these operations it is necessary for the first operand to be in register A. The code generation for < assign > consists of bringing the value to be assigned into register A (using GETA) and then generating a STA instruction. The remaining rules in fig. 21 do not require the generation of any instruction since no computation and data movement is involved. The object code generated for the assignment statement is shown in fig. 22. LDA DIV STA LDA MUL STA LDA SUB STA
SUMSQ * 100 TMP1 MEAN MEAN TMP2 TMP1 TMP2 VARIABLE
Fig. 22 For the grammar < prog > the code-generation routine is shown in fig. 23. When is recognized, storage locations are assigned to any temporary (Temp) variables that have been used. Any references to these variables are then fixed in the object code using the same process performed for forward references by a one-pass assembler. The compiler also generates any modification records required to describe external references to library subroutine. < prog > : : = PROGRAM < prog-name > VAR < dec list > BEGIN < stmp -- list > END. generate [LDL RETADR] generate [RSUB] for each Temp variable used do generate [ Temp RESW 1] insert [ J EXADDR ] {jump to first executable instruction} in bytes 3 - 5 of object program. fix up forward reference to Temp variables generate modification records for external references generates [END]. The < prog-name > generates header information in the object program that is similar to that created from the START and EXTREF as assembler directives. It also generates instructions
25
to save the return address and jump to the first executable instruction in the compiled program. Fig. 24 shows the code generation routine for the grammar < prog-name >. < Program > : : = id generate [START 0] generate [EXTREF XREAD, XWRITE] generate [STL RETADR] add 3 to LC {leave room for jump to first executable instruction} generate [RETADR RESW 1]
Fig. 24 Similar to the previous code-generation routine fig. 25 shows the codegeneration for < dec - list >, < dec > , < write >, < for > , < index - exp > and body. < dec - list > : : = { alternatives } save LC as EXADDR {tentative address of first executable instruction} < dec > : : = > id - list > : < type > for each item on list do begin remove ST (NAME) from list enter LC symbol table as address for NAME generate [ST (NAME) RESW 1] end LIST COUNT : = 0 < write > : : = WRITE ( < id - list > ) generate [ + JSUB XWRITE] record external reference to XWRITE generate [WORD LISTCOUNT] for each item on list do begin remove ST (ITEM) from list generate [WORD ST (ITEM)] end LIST COUNT : = 0 < for > : : = FOR < id ex -- exp > Do < body > POP JUMPADDR from stack {address of jump out of loop} POP ST (INDEX) from stack {index variable} POP LOOPADDR from stack {beginning address of loop} generate [LDA ST (INDEX)] generate [ADD #1] generate [ J LOOPADDR] insert [ JGT LC ] at location JUMPADDR < index - exp > : : = id : = < exp > | TO < exp >2 GETA (< exp >;) Push LC onto stack {beginning addressing loop} Push ST (id) onto stack {index variable}
26
Generate [STA ST (id)] Generate [ COMP ST (< exp > 2)] Push LC onto stack {address of jump out of loop} and 3 to LC [ leave room for jump instruction] REGA : = null
Fig. 25 There are no code-generation for the statements < type > : : = INTEGER < stmt - list > : : = {either alternative} < stmt > : : = {any alternative} < body > : : = {either alternative} For the Pascal program in fig. 1 the complete code-generation process is shown in fig. 26.
1 STATS
START 0 {Program Header} EXTREF XREAD, XREAD, XWRITE STL TETADR {Save return address} J {EXADDR} 2 RETADDR RESW 1 3 SUM RESW 1 SUMSQ RESW 1 I RESW 1 VALUE RESW 1 MEAN RESW 1 VARIANCE RESW 1 5 {EXADDR} LDA #0 {SUM = 0} STA SUM 6 LDA #0 {SUMSQ : = 0} STA SUMSQ 7 LDA #1 {FOR I : = 1 TO 100} {L1} STA I COMP # 100 JGT {L2} 9 + JSUB X READ {READ (VALUE) } WORD 1 WORD VALUE 10 LDA SUM {SUM : = SUM + VALUE} ADD VALUE STA SUM 11 LDA VALUE {SUMSQ : = SUMSQ * VALUE * VALUE} MUL VALUE ADD SUMSQ STA SUMSQ LDA I {END OF FOR LOOP}
27
13 {L2} DIVISION}
ADD J LDA
DIV STA 14 LDA DIV STA LDA MUL STA LDA SUB STA 15 +JSUB WORD WORD WORD LDL RSUB TEMP 1` RESW TEMP 2 RESW END
#1 {L1} SUM
{MEAN : = SUM
# 100 MEAN SUM {VARIABLE : = SUMSQ DIV # 100 100 - MEAN * MEAN} TEMP1 MEAN MEAN TEMP2 TEMP1 TEMP2 VARIANCE XWRITE {WRITE (MEAN, VARIANCE) } 2 MEAN VARIABLE RETADR 1 1
{WORKING VARIABLE USED}
Fig. 25 Object Code Generated for Pascal Program 8.1 MACHINE DEPENDENT COMPILER FEATURES At an elementary level, all the code generation is machine dependent. This is because, we must know the instruction set of a computer to generate code for it. There are many more complex issues involved. They are:
Allocation of register Rearrangement of machine instruction to improve efficiency of execution
Considering an intermediate form of the program being compiled normally does such types of code optimization. In this intermediate form, the syntax and semantics of the source statements have been completely analyzed, but the actual translation into machine code has not yet been performed. It is easier to analyze and manipulate this intermediate code than to perform the operations on either the source program or the machine code. The intermediate form made in a compiler, is not strictly dependent on the machine for which the compiler is designed. 8.1.1
INTERMEDIATE FORM OF THE PROGRAM
The intermediate form that is discussed here represents the executable instruction of the program with a sequence of quadruples. Each quadruples of the form Operation, OP1, OP2, result. Where Operation - is some function to be performed by the object code
28
OP 1 & OP2 - are the operands for the operation and Result - designation when the resulting value is to be placed. Example 1:
SUM : = SUM + VALUE could be represented as + , SUM, Value, i, i1 := i1 , , SUM
The entry i1, designates an intermediate result (SUM + VALUE); the second quadruple assigns the value of this intermediate result to SUM. Assignment is treated as a separate operation ( : =). Example 2 : VARIANCE : = SUMSQ, DIV 100 -- MEAN * MEAN DIV, SUMSQ, #100, i1 *, MEAN, MEAN, i2 -, i1, i2, i3
::= Note:
i3
VARIABLE
,
Quadruples appears in the order in which the corresponding object code instructions are to be executed. This greatly simplifies the task of analyzing the code for purposes of optimization. It is also easy to translate into machine instructions.
For the source program in Pascal shown in fig. 1. The corresponding quadruples are shown in fig. 27. The READ and WRITE statements are represented with a CALL operation, followed by PARM quadruples that specify the parameters of the READ or WRITE. The JGT operation in quadruples 4 in fig. 27 compares the values of its two operands and jumps to quadruple 15 if the first operand is greater than the second. The J operation in quadruples 14 jumps unconditionally to quadruple 4.
Line Operation OP 1 1. 2. 3. 4. 5. 6. 7. 9. 10. 11. 12. 13. 14. 15. 16. 17. 1 19. 20. 21.
:= := := JGT CALL PARAM + ;= * + := + := J DIV := DIV * := CALL
OP 2
#0 #0 #1 I XREAD SUM i1 VALUE SUMSQ i3 I i4
#100
Result
SUM SUMSQ I (15)
Pascal Statement
SUM : = 0 SUMSQ : = 0 FOR I : = 1 to 100 READ (VALUE)
VALUE VALUE i1 SUM VALUE i2 i2 i3 SUMSQ #1 i4 I (4) #100 i5 MEAN #100 i6 MEAN i7 i7 i8
SUM i5 SUMSQ MEAN i6 i8 XWRITE
SUM : = SUM + VALUE SUMSQ : = SUMSQ + VALUE * VALUE End of FOR loop MEAN : = SUM DIV 100 VARIANCE : = SUMSQ DIV 100 - MEAN * MEAN VARIANCE WRITE (MEAN, VALIANCE
29
22. 23.
PARAM PARAM
MEAN VARIANCE
Fig. .27 Intermediate Code for the Pascal Program 8.1.2
MACHINE - DEPENDENT CODE OPTIMIZATION
There are several different possibilities for performing machine-dependent code optimization . -- Assignment and use of registers: Here we concentrate the use of registers as instruction operand. The bottleneck in all computers to perform with high speed is the access of data from memory. If machine instructions use registers as operands the speed of operation is much faster. Therefore, we would prefer to keep in registers all variables and intermediate result that will be used later in the program. There are rarely as many registers available as we would like to use. The problem then becomes which register value to replace when it is necessary to assign a register for some other purpose. On reasonable approach is to scan the program for the next point at which each register value would be used. The value that will not be needed for the longest time is the one that should be replaced. If the register that is being reassigned contains the value of some variable already stored in memory, the value can simply be discarded. Otherwise, this value must be saved using a temporary variable. This is one of the functions performed by the GETA procedure. In using register assignment, a compiler must also consider control flow of the program. If they are jump operations in the program, the register content may not have the value that is intended. The contents may be changed. Usually the existence of jump instructions creates difficulty in keeping track of registers contents. One way to deal with the problem is to divide the problem into basic blocks. A basic block is a sequence of quadruples with one entry point, which is at the beginning of the block, one exit point, which is at the end of the block, and no jumps within the blocks. Since procedure calls can have unpredictable effects as register contents, a CALL operation is usually considered to begin a new basic block. The assignment and use of registers within a basic block can follow as described previously. When control passes from one block to another, all values currently held in registers are saved in temporary variables. For the problem is fig. .27, the quadruples can be divided into five blocks. They are:
A : 1-3
Block -- A
Quadruples 1 - 3
Block -- B
Quadruples 4
Block -- C
Quadruples 5 - 14
C : 5 - 14
Block -- D
Quadruples 15 - 20
D : 15 -
Block -- E
Quadruples 21 - 23
E : 21 -
B:4
Fig. 28 Fig. 28 shows the basic blocks of the flow group for the quadruples in fig. 27. An arrow from one block to another indicates that control can pass directly from one quadruple to another. This kind of representation is called a flow group.
30
-- Rearranging quadruples before machine code generation: Example :
1) 2) 3) 4)
DIV * :=
LDA DIV STA LDA MUL STA
SUMSQ # 100 MEAN MEAN i2 i1 i2 i3 i3 VARIANCE
SUMSQ # 100 T1 MEAN MEAN T2
SUB STA
i1
LDA T1 T2 VARIANCE
Fig. 29 Fig. 29 shows a typical generation of machine code from the quadruples using only a single register. Note that the value of the intermediate result, is calculated first and stored in temporary variable T1. Then the value of i2 is calculated subtracting i2 from ii. Even though i2 value is in the register, it is not possible to perform the subtraction operation. It is necessary to store the value of i 2 in another temporary variable T2 and then load the value of i1 from T1 into register A before performing the subtraction. The optimizing compiler could rearrange the quadruples so that the second operand of the subtraction is computed first. This results in reducing two memory accesses. Fig. 29 shows the rearrangements. * DIV :=
MEAN SUMSQ i1 i3 LDA MUL STA LDA DIV SUB STA
MEAN i2 # 100 i1 i2 i3 VARIANCE
MEAN MEAN T1 SUMSQ # 100 T1 VARIANCE
Fig. 29 Rearrangement of Quadruples for Code Optimization -- Characteristics and Instructions of Target Machine: These may be special loop control instructions or addressing modes that can be used to create more efficient object code. On some computers there are high-level machine instructions that can perform complicated functions such as calling procedure and manipulating data structures in a single operation. Some computers have multiple functional blocks. The source code must be rearranged to use all the blocks or most of the blocks concurrently. This is possible if the result of one block does not depend on the result of the other. There are some systems where the data flow can be arranged between blocks without storing the intermediate data in any register. An optimizing compiler for such a machine could rearrange object code instructions to take advantage of these properties.
31
Machine Independent Compiler Features Machine independent compilers describe the method for handling structured variables such as arrays. Problems involved in compiling a block-structured language indicate some possible solution. 3.1 STRUCTURED VARIABLES Structured variables discussed here are arrays, records, strings and sets. The primarily consideration is the allocation of storage for such variable and then the generation of code to reference then. Arrays: In Pascal array declaration (i) Single dimension array:
A: ARRAY [ 1 . . 10] OF INTEGER
If each integer variable occupies one word of memory, then we require 10 words of memory to store this array. In general an array declaration is ARRAY [ l .. u ] OF INTEGER Memory word allocated = ( u - l + 1) words. (ii)
Two dimension array :
B : ARRAY [ 0 .. 3, 1 . . 3 ] OF INTEGER
In this type of declaration total word memory required is 0 to 3 = 4 ; 1 - 3 = 3 ; 4 x 3 = 12 word memory locations. In general: ARRAY [ l1 .. u1, l2 . . u2.] OF INTEGER Requires ( u1 - l1 + 1) * ( u2 - l2 + 1) Memory words The data is stored in memory in two different ways. They are row-major and column major. All array elements that have the same value of the first subscript are stored in contiguous locations. This is called row-major order. It is shown in fig. 30(a). Another way of looking at this is to scan the words of the array in sequence and observe the subscript values. In row-major order, the right most subscript varies most rapidly. 0,1
0,2
0,3
Row 0
0,4
0,5
0,1
1,2
1,3
1,4
1,5
Row 1
2,1
2,2
2,3
2,4
2,5
...
Row 2
Fig. 30 (a) Fig. 30(b) shows the column major way of storing the data in memory. All elements that have the same value of the second subscript are stored together; this is called column major order. In other words, the column major order, the left most subscript varies most rapidly. To refer to an element, we must calculate the address of the referenced element relative to the base address of the array. Compiler would generate code to place the relative address in an index register. Index addressing mode is made easier to access the desired array element. (1) One Dimensional Array: On a SIC machine to access A [6], the address is calculated by starting address of data + size of each data * number of preceding data. i.e. Assuming the starting address is 1000H Size of each data is 3 bytes on SIC machine Number of preceding data is 5 Therefore the address for A [ 6 ] is = 1000 + 3 * 5 = 1015. In general for A: ARRAY [ l . . u ] of integer, if each array element occupies W bytes of storage and if the value of the subscript is S, then the relative address of the referred element A[ S ] is given by W * ( S - l ). The code generation to perform such a calculation is shown in fig. 31.
32
The notation A[ i2 ] in quadruple 3 specifies that the generated machine code should refer to A using index addressing after having placed the value A: ARRAY [ 1 . . 10 ] OF INTEGER . . . A[ I ] : = S (1)
+ :=
I i1 #5
#1 #3
i1 i2 A [ i1 ]
Fig. 31 Code Generation for Single Dimension Array of i2 in the Index Register (2) Multi-Dimensional Array: In multi-dimensional array we assume row major order. To access element B[ 2,3 ] of the matrix B[ 6, 4 ], we must skip over two complete rows before arriving at the beginning of row 2. Each row contains 6 elements so we have to skip 6 x 2 = 12 array elements before we come to the beginning of row 2 to arrive at B[ 2, 3 ]. To skip over the first two elements of row 2 to arrive at B[ 2, 3 ]. This makes a total of 12 + 2 = 14 elements between the beginning of the array and element B[2, 3 ]. If each element occurs 3 byte as in SIC, the B[2, 3] is located relating at 14 x 3 = 42 address within the array. Generally the two dimensional array can be written as B ; ARRAY [ l1 . . . u1, l1 . . . u1, ] OF INTEGER The code to perform such an array reference is shown in fig. 32. B : ARRAY [ 0 . . 3, 1 . . 6 ] OF INTEGER . . B[I, J] : = 5 1)
* -+ * :=
I j i1 i3 #5
#6 #1 i2 #3
i1 i2 i3 i4 A [ i1 ]
Fig. 32 Code Generation for Two Dimensional Array The symbol - table entry for an array usually specifies the following:
The type of the elements in the array The number of dimensions declared The lower and upper limit for each subscript.
This information is sufficient for the compiler to generate the code required for array reference. Some of the languages line FORTRAN 90, the values of ROWS and COLUMNS are not known at completion time. The compiler cannot directly generate code. Then, the compiler create a descriptor called dope vector for the array. The descriptor includes space for storing the lower and upper bounds for each array subscript. When storage is allocated for the array, the values of these bounds are computed and stored in the descriptor. The generated code for one array reference uses the values from the descriptor to calculate relative addresses as required. The descriptor may also include the number of dimension for the array, the type of the array
33
elements and a pointer to the beginning of the array. This information can be useful if the allocated array is passed as a parameter to another procedure. In the compilation of other structured variables like recode, string and sets the same type of storage allocations are required. The compiler must store information concerning the structure of the variable and use the information to generate code to access components of the structure and it must construct a description for situation in which the required conformation is not known at compilation time. 8.3.1
MACHINE - INDEPENDENT CODE OPTIMIZATION
One important source of code optimization is the elimination of common subexpressions. These are sub-expressions that appear at more than one port in the program and that compute the same value. Let us consider the example in fig. 33. x, y : ARRAY [ 0 . . 10, 1 . . 10 ] OF INTEGER . . . FOR I : = 1 TO 10 DO X [ I, 2 * J - 1 ] : = [ I, 2 * J }
Fig. 33(a) The sub-expression 2 * J is calculated twice. An optimizing compiler should generate code so that the multiplication is performed only once and the result is used in both places. Common sub-expressions are usually detected through the analysis of an intermediate form of the program. This intermediate form is shown in fig. 33(b).
Line Operation 1. 2. 3. 4. 5. 6. 7. 9. 10. 11. 12. 13. 14. 15. 16. 17. 1 19. 20.
:= JGT * * --+ * -* * -+ * := + := J
OP 1 #1 I I i1 #2 i3 i4 i2 i6 I i8 #2 i10 i9 i12 y [ i13 } #1 i14
OP 2 #10 #1 #10 J #1 #1 i5 #3 #1 #10 J 3 1 i11 i11 #3 I
I (20) i1 i2 i3 i4 i5 i6 i7 i8 i9 i10 i12 i13 x [ i17 ] i17 I (2)
Result
Pascal Statement
[Loop initialization] [Subscript calculation for x]
[Subscript Calculation for y]
[Assignment Operation] [End of Loop] [Next Statement]
Fig. 33(b) 34
Examining fig. 33(b), the sequence of quadruples, we observe that quadruples 5 and 12 are the same except for the name of the intermediate result produced. The operand J is not changed in value between quadruples 5 and 12. It is not possible to reach quadruple 12 without passing through quadruple 5 first because the quadruples are part of the same basic block. Therefore, quadruples 5 and 12 compute the same value. This means we can delete quadruple 12 and replace any reference to its result ( i10 ), with the reference to i3, the result of quadruple 5. this information eliminates the duplicate calculation of 2 * J which we identified previously as a common expression in the source statement. After the substitution of i3 for i10 , quadruples 6 and 13 are the same except for the name of the result. Hence the quadruple 13 can be removed and substitute i4 for i11 wherever it is used. Similarly quadruple 10 and 11 can be removed because they are equivalent to quadruples 3 and 4.
Line Operation 1. 2. 3. 4. 5. 6. 7. 9. 10. 11. 12. 13. 14. 15. 16.
OP 1
:= JGT * * + * +
#1 I I i1 #2 i3 i4 i2 i6 i2
* := + := J
i12 y [ i13 ] #1 i14
OP 2 #10 #1 #10 J #1 #1 i5 #3 i4 #3 I
I (16) i1 i2 i3 i4 i5 i6 i7 i12 i13 x [i7 ] i14 I (2)
Result
Pascal Statement
[Loop initialization] [Subscript calculation for x]
[Subscript Calculation for y] [assignment Operation] [End of Loop] [Next Statement]
Fig. 34 Names i1 have been left unchanged, except for the substitutions first described, to make the compromise with fig. 33(b) easier. This optimized code has only 15 quadruples and hence the time taken is reduced. Another method of code optimization is the removal of loop invariants. There are subexpressions within the loop whose values do not change from one iteration of the loop to the next. Thus the values can be calculated once, before the loop is entered, rather than being recalculated for each iteration. In the example shows in fig. 33(a), the loop-invariant computation is the term 2 * J [quadruple 5 fig. 34]. The result of this computation depends only on the operand J, which does not change the value during the execution of the loop. Thus we can move quadruple 5 in fig. 34 to a point immediately before the loop is entered. A similar arrangement can be applied to quadruples 6 and 7. Fig. 35 shows the sequence of quadruples that result from these modification. The total number of quadruples remains the same as fig. 34, however, the number of quadruples within the body of the loop has been reduced from 14 to 11. Our modification have reduced the total number of quadruples for one execution of the FOR from 181 [Fig. 23 (b) ], to 114 [Fig 25], which saves a substantial amount of time.
35
Line Operation 1. 2. 3. 4. 5. 6. 7. 9. 10. 11. 12. 13. 14. 15. 16.
* := JGT * + * + * := + := J
OP 1 #2 i3 i4 #1 I I i1 i2 i6 i2 i12 y [ i13 ] #1 i14
OP 2 J #1 #1 #10 #1 #10 i5 #3 i4 #3 I
i3 i4 i5 I (16) i1 i2 i6 i7 i12 i13 x [i7 ] i14 I (5)
Result
Pascal Statement
{Commutation of invariants} {Loop Initialization} {Subscript calculation for x} {Subscript Calculation for y} {assignment Operation} {End of Loop} {Next Statement}
Fig. 35 -- The optimization can be obtained by rewriting the source program. Example; The statement in fig. 36(a) could be written as shown in fig. 36 (b). FOR I : = 1 To 10 Do x [ I, 2 * J - 1 ] : = y [ I, 2 * J ]
Fig. 36(a) T1 : = 2 * J ; T2 : = T1 -- 1 ; FOR : = 1 To 10 Do x [ I, T2 ] : = y[ I, T1]
Fig. 36(b) This would achieve only a part of the benefits realized by the optimization process described. Some time the statement in fig. 36(a) is preferable because it is clearer than the modified version involving T1 and T2. An optimizing compiler should allow the programmer to write source code that is clearer and easy to read and it should compile such a program into machine code that is efficient to execute. -- Code optimization of another source is the substitution of a more efficient operation for a less efficient one. Example: The FORTRAN statement: Do 10 I = 1, 20 ; To calculate the first 20 power of 2 and store it in TABLE ( I ) = 2 * * I ; TABLE In each iteration of the loop, the constant 2 is raised to the power I. The quadruples are shown in fig. 37(a). Exponentiation is represented with the operation EXP. This computation can be performed more efficiently. Here, in each iteration of the loop, the value of I is incremented by 1. The value of 2 * * I for the current iteration can be found by multiplying the value for the previous iteration by 2. This method of computing 2 * * I is much more efficient than performing series of multiplication or using a logarithms technique.
36
This technique is shown in fig. 37(b).
Line Operation 1. 2. 3. 4. 5. 6. 7.
:= EXP -* := + := JLE
OP 1 #1 #2 I i2 i1 I i4 I
OP 2
Result
Pascal Statement
I {Loop Initialization} i1 { Calculation of 2 * i5 {Subscript calculation } i3 TABLE [ i2] {Assignment Operation} i4 {End of the Loop} I i3
I #1 #3 #1 #20
Fig. 37(a) Line Operation 1. 2. 3. 4. 5. 6. 7. 9.
:= :: = := * + := + := JLE
OP 1 #1 # (-3) #1 i1 i3 i1 I i4 I
OP 2
#2 #3 #1 #20
Result
i1 i3 I i1 i3 TABLE [ i3] i4 I (4)
Pascal Statement
{Initialize temporaries} {Loop Initialization} { Calculation of 2 * * I } {Subscript calculation } {Assignment Operation} {End of the Loop}
Fig. 37(b) STORAGE ALLOCATION All the program defined variable, temporary variable, including the location used to save the return address use simple type of storage assignment called static allocation. When recursively procedures are called, static allocation cannot be used. This is explained with an example. Fig. 38(a) shows the operating system calling the program MAIN. The return address from register 'L' is stored as a static memory location RETADR within MAIN.
SYSTEM (1)
SYSTEM
MAIN
(1)
SYSTEM MAIN
CALL SUB
RETADR
MAIN
CALL SUB RETADR
RETADR
(2)
(a)
(1)
(2) SUB CALL SUB
(b)
(3) RETADR
RETADR
37
(c)
(c)
Fig. 38 In fig. 38(b) MAIN has called the procedure SUB. The return address for the call has been stored at a fixed location within SUB (invocation 2). If SUB now calls itself recursively as shown in fig. 38(c), a problem occurs. SUB stores the return address for invocation 3 into RETADR from register L. This destroys the return address for invocation 2. As a result, there is no possibility of ever making a correct return to MAIN. There is no provision of saving the register contents. When the recursive call is made, variable within SUB may set few variables. These variables may be destroyed. However, these previous values may be needed by invocation 2 or SUB after the return from the recursive call. Hence it is necessary to preserve the previous values of any variables used by SUB, including parameters, temporaries, return addresses, register save areas etc., when a recursive call is made. This is accomplished with a dynamic storage allocation technique. In this technique, each procedure call creates an activation record that contains storage for all the variables used by the procedure. If the procedure is called recursively, another activation record is created. Each activation record is associated with a particular invocation of the procedure, not with the itself. An activation record is not deleted until a return has been made from the corresponding invocation. Activation records are typically allocated on a stack, with the correct record at the tip of the stack. It is shown in fig. 39(a). Fig. 39(a) corresponds to fig. 39(b). The procedure MAIN has been called; its activation record appears on the stack. The base register B has been set to indicate the starting address of this correct activation record. The first word in an activation record would normally contain a pointer PREV to the previous record on the stack. Since the record is the first, the pointer value is null. The second word of the activation record contain a portion NEXT to the first unused word of the stack, which will be the starting address for the next activation record created. The third word contain the return address for this invocation of the procedure, and the necessary words contain the values of variables used by the procedure.
SYSTEMS MAIN
RETADR NEXT 0
Stack Fig. 39 (a) SYSTEM (1)
MAIN
CALL SUB
Variables For SUB RETADR NEXT PREV Variable B For MAIN RETADR NE XT 0 stacl
38
SUB Stack
Fig. 39(b) In fig. 39 (b), MAIN has called the procedure SUB. A new activation record has been created on the top of the stack, with register B set to indicate this new current record. The pointers PREV and NEXT in the time records have been set as shown.
SYSTEM (1)
MAIN
Variables For SUB RETADR NEXT
CALL SUB
PREV B Variable For MAIN RETADR NEXT
CALL SUB
Fig. 39 (c)
PREV Variable For MAIN RETADR NEXT 0
Stack
In fig. 39(c), SUB has called itself recursively another activation record has been created for this current invocation for SUB. Note that the return address and variable values for the two invocations of SUB are kept separate by this process. When a procedure returns to its caller, the current activation record (which corresponds to the most recent invocation) is deleted. The pointer PREV in the deleted record is used to reestablish the previous activation record as the current one, and execution continues.
SYSTEM (1)
MAIN
CALL SUB
SUB
Fig. 39(d)
Variables For SUB RETADR NEXT PREV B Variable For MAIN RETADR NEXT 0
Stack
Fig. 39(d) shows the stack as it would appear after SUB returns from the recursive call. Register B has been reset to point to the instruction record for the previous invocation of SUB.
39
The return address and all the variable values in this activation record are exactly the same as they were before the recursive call. This technique is called automatic allocation of storage. When the technique is used the compiler must generate code for the reference to variables using some sort of relative addressing. In our example the compiler assigns to each variable an address that is relative to the beginning of the activation record, instead of an actual location within the object program. The address of the current activation record is, by convention contained in register B, so a reference to a variable is translated as an instruction that uses base relative addressing. The displacement in this instruction is the relative address of the variable within the activation record. The compiler must also generate additional code to manage the activation records themselves. At the beginning of each procedure there must be code to create a new activation record, linking it to the previous one and setting the appropriate pointers as shown in fig. 39. This code is often called a prologue for the procedure. At the end of the procedure, there must be code to delete the current activation record, resulting pointers as needed. This code is called an epilogue. Example: IN FOTRAN 90 :ALLOCATE (MATRIX (ROWS, COLUMNS) ) allocation storage for the dynamic array MATRIX with the specified dimensions. DE-ALLOCATE MATRIX releases the storage assigned to MATRIX by a previous ALLOCATE. IN PASCAL: NEW (P) allocates storage for a variable and sets the pointer P to indicate the variable just created. DISPOSE (P)
releases the storage that was previously assigned to the variable pointed to by P. In C : MALLCO (SIZE) ; allocate a block of specified size . . . FREE (P) ; frees the storage indicated by pointer P. A variable that is dynamically allocated in this way does not occupy a fixed location in an activation record, so it cannot be referenced directly using base relative addressing. Such a variable is usually accessed using indirect addressing through a pointer variable P. Since P does occupy a fixed location in the activation record, it can be addressed in the usual way. The mechanism to allocate a storage memory to a variable can be done in any of the following ways: A NEW or MALLOC statement would be translated into a request by the operating system for an area of storage of the required size. The required allocation is handled through a run-time support procedure associated with the compiler. With this method, a large block of free storage called a heap is obtained from the operating system at the beginning of the program. Allocations of the storage from the heap are managed by the run-time procedure. In some systems, the program need not free memory for storage. A runtime garbage collection procedure scans the pointer in the program and reclaims areas from the heap that are no longer used. 8.3.3
BLOCK - STRUCTURED LANGUAGE
40
A block is a unit that can be divided in a language. It is a portion of a program that has the ability to declare its own identifiers. This definition of a block is also meet the units such as procedures and functions. Let us consider a Pascal program with number of procedure blocks as shown in fig. 40. Each procedure corresponds to a block. Note that blocks are rested within other blocks. Example: Procedures B and D are rested within procedure A and procedure C is rested within procedure B. Each block may contain a declaration of variables. A block may also refer to variables that are defined in any block that contains it, provided the same names are not redefined in the inner block. Variables cannot be used outside the block in which they are declared. In compiling a program written in a blocks structured language, it is convenient to number the blocks as shown in fig. 40. As the beginning of each new block is recognized, it is assigned the next block number in sequence. The compiler can then construct a table that describes the block structure. It is illustrated in fig. 41. The blocklevel entry gives the nesting depth for each block. The outer most block number that is one greater than that of the surrounding block.
PROCEDURE A ; VAR X, Y, Z : INTEGER ; : PROCEDURE B ; VAR W, X, Y : REAL ; : PROCEDURE C ; VAR W, V : INTEGER ; : 3 END { C }; : END { B }; : PROCEDURE D ; VAR X, Z : CHAR ; . 2 . END { D}; END { A};
1 2
Fig. 40 Nested Blocks in a Program Block Surrounding Name Number Level Block A 1 1 -B 2 2 1 C 3 3 2 D 4 2 1 Fig. 41 Since a name can be declared more than once in a program (by different blocks), each symbol-table entry for an identifier must contain the number of the declaring block. A declaration of an identifier is legal if there has been no previous declaration of that identifier by
41
the current block, so there can be several symbolic table entries for the same name. The entries that represent declaration of the same name by different blocks can be linked together in the symbol table with a chain of pointers. When a reference to an identifier appears in the source program, the compiler must first check the symbol table for a definition of that identifier by the current block. If not such definition is found, the compiler looks for a definition by the block that surrounds the current one, then by the block that surrounds that and so on. If the outermost block is reached without finding a definition of the identifier, then the reference is an error. The search process just described can easily be implemented within a symbol table that uses hashed addressing. The hashing function is used to locate one definition of the identifier. The chain of definitions for that identifier is then searched for the appropriate entry. Most block-structured languages make use of automatic storage allocation. The variables that are defined by a block are stored in an activation record that is created each time the block is entered. If a statement refers to a variable that is declared within the current block, this variable is present in the current activation record, so it can be accessed in the usual way. It is possible to refer to a variable that is declared in some surrounding block. In that case, the most recent activation record for that block must be located to access the variable.
Activation Record for
Activation Record for C Activation Record for
C C B
B Stack (a)
A
Activation Record for C
C
(b) Activatio
B A
Fig. 42 Use of Display for Procedure A data structure called display is used to access a variable in surrounding blocks. The display contains pointers to the most recent activation records for the current block and for all blocks that surround the current one in the source program. When a block refers to a variable that is declared in some surrounding block, the generated object code uses the display to find the activation record that contains this variable. Example: When a procedure calls itself recursively thus an activation record is created on the stack as a result of the call. Assume procedure C calls itself recursively. It is shown in fig. 42(b) the record for C is created on the stack as a result of the call. Any reference to a variable declared by C should use this most recent activation record ; the display pointer for C is changed accordingly. Variables that correspond to the previous invocation of C are not accessible for the movement, so there is no display pointer to this activation record.
Activation Record for C Activation Record for B Activation
Record for A
Activation
42
D A Stack
Display
Fig 42(c) Now if procedure 'C' call procedure D the resulting stack and display are as illustrated in fig. 42(c) . An activation record for D has been created in the usual way and added to the stack. Note, that the display now contains only two pointers: one each to the activation records for D and A. This is because procedure D cannot refer to variable in B or C, except through parameters that are passed to it, even though it is called from C. According to the rules for the scope of names in as block-structured language, procedure D can refer only to variable that are declared by D or by some block that contains D in the source program. 8.4 COMPILER DESIGN OPTIONS The compiler design is briefly discussed in this section. The compiler is divided into single pass and multi pass compilers. 4.1. COMPILER PASSES One pass compiler for a subset of the Pascal language was discussed in section 1. In this design the parsing process drove the compiler. The lexical scanner was called when the parser needed another input token and a code-generation routine was invoked as the parser recognized each language construct. The code optimization techniques discussed cannot be applied in total to one-pass compiler without intermediate code-generation. One pass compiler is efficient to generate the object code. One pass compiler cannot be used for translation for all languages. FORTRAN and PASCAL language programs have declaration of variable at the beginning of the program. Any variable that is not declared is assigned characteristic by default. One pass compiler may fix the formal reference jump instruction without problem as in one pass assembler. But it is difficult to fix if the declaration of an identifier appears after it has been used in the program as in some programming languages. Example:
X:=Y*Z
If all the variables x, y and z are of type INTEGER, the object code for this statement might consist of a simple integer multiplication followed by storage of the result. If the variable are a mixture of REAL and INTEGER types, one or more conversion operations will need to be included in the object code, and floating point arithmetic instructions may be used. Obviously the compiler cannot decide what machine instructions to generate for this statement unless instruction about the operands is available. The statement may even be illegal for certain combinations of operand types. Thus a language that allows forward reference to data items cannot be compiled in one pass. Some programming language requires more than two passes. Example : ALGOL-98 requires at least 3 passes.
43
There are a number of factors that should be considered in deciding between one pass and multi pass compiler designs. (1) One Pass Compiles: Speed of compilation is considered important. Computer running students jobs tend to spend a large amount of time performing compilations. The resulting object code is usually executed only once or twice for each compilation, these test runs are not normally very short. In such an environment, improvement in the speed of compilation can lead to significant benefit in system performance and job turn around time. (2) Multi-Pass Compiles: If programs are executed many times for each compilation or if they process large amount of data, then speed of executive becomes more important than speed of compilation. In a case, we might prefer a multi-pass compiler design that could incorporate sophisticated code-optimization technique. Multi-pass compilers are also used when the amount of memory, or other systems resources, is severely limited. The requirements of each pass can be kept smaller if the work by compilation is divided into several passes. Other factors may also influence the design of the compiler. If a compiler is divided into several passes, each pass becomes simpler and therefore, easier to understand, read and test. Different passes can be assigned to different programmers and can be written and tested in parallel, which shortens the overall time require for compiler construction. INTERPRETERS An interpreter processes a source program written in a high-level language. The main difference between compiler and interpreter is that interpreters execute a version of the source program directly, instead of translating it into machine code. An interpreter performs lexical and syntactic analysis functions just like compiler and then translates the source program into an internal form. The internal form may also be a sequence of quadruples. After translating the source program into an internal form, the interpreter executes the operations specified by the program. During this phase, an interpreter can be viewed as a set of subtractions. The internal form of the program drives the execution of this subtraction. The major differences b/w interpreter and compiler are: Interpreters
Compilers
44
1) The process of translating a source program into some internal form is simpler and faster 2) Execution of the translated program is much slower. 3) Debugging facilities can be easily provided. 4) During execution the interpreter produce symbolic dumps of data values, trace of program execution related to the source statement. 5) Program testing can be done effectively using interpreter as the operation on different data can be traced. 6) Easy to handle dynamic scoping
The process of translating a source program into some internal form is slower than interpreter. Executing machine code is much faster. Provision of bugging facilities are difficult and complicated. The compiler does not produce symbolic dumps of date value. Debugging tools are required for trace the program.
It is difficult to test as the compiler execution file gives the final result. Difficult to scooping
handle
dynamic
Most programming languages can be either compiled or interpreted successfully. However, some languages are particularly well suited to the use of interpreter. Compilers usually generate calls to library routines to perform function such as I/O and complex conversion operations. In such cases, an interpreter might be performed because of its speed of translation. Most of the execution time for the standard program would be consumed by the standard library routines. These routines would be the same, regardless of whether a compiler or an interpreter is used. In some languages the type of a variable can change during the execution of a program. Dynamic scoping is used, in which the variable that are referred to by a function or a subroutines are determined by the sequence of calls made during execution, not by the nesting of blocks in the source program. It is difficult to compile such language efficiently and allow for dynamic changes in the types of variables and the scope of names. These features can be more easily handled by an interpreter that provides delayed binding of symbolic variable names to data types and locations. 4.3 P-CODE COMPILERS P-Code compilers also called byte of code compilers are very similar in concept to interpreters. A P-code compiler, intermediate form is the machine language for a hypothetical computers, often called pscudo-machine or P-machine. The process of using such a P-code is
shown in fig, 43.
45
The main advantage of this approach is portability of software. It is not necessary for the compiler to generate different code for different computers, because the P-code object program can be executed on any machine that has a P-code interpreter. Even the compiler itself can be transported if it is written in the language that it compiles. To accomplish this, the source version of the compiler is compiled into P-code; this P-code can then be interpreted on another compiler. In this way P-code compiler can be used without modification as a wide variety of system if a Pcode interpreter is written for each different machine.
Source Program
P-Code Compiler
Compiler
Object Program P - Code Execute
P - Code Interprete r
Fig. 43 The design of a P-machine and the associated P-code is often related to the requirements of the language being compiled. For example, the P-code for a Pascal compiler might include single P-instructions that perform: Array subscript calculation Handle the details of procedure entry and exit and Perform elementary operation on sets This simplifies the code generation process, leading to a smaller and more efficient compiler. The P-code object program is often much smaller than a corresponding machine code program. This is particularly useful on machines with severely limited memory size. The interpretive execution of P-code program may be much slower than the execution of the equivalent machine code. Many P-code compilers are designed for a single user running on a dedicated micro-computer systems. In that case, the speed of execution may be relatively insignificant because the limiting factor is system performance may be the response time and " think time " of the user. If execution speed is important, some P-code compilers support the use of machinelanguage subtraction. By rewriting a small number of commonly used routines in machine language, rather than P-code, it is often possible to improve the performance. Of course, this approach sacrifices some of the portability associated with the use of P-code compilers. 8.4.2
COMPILER-COMPILERS
Compiler-Compiler is software tool that can be used to help in the task of compiler construction. Such tools are also called Compiler Generators or Translator - writing system.
46
The process of using a typical compiler-compiler is shown in fig. 44. The compiler writer provides a description of the language to be translated. This description may consists of a set of lexical rules for defining tokens and a grammar for the source language. Some compilercompilers use this information to generate a scanner and a parses directly. Others create tables for use by standard table-driven scanning and parsing routines that are supplies by the compiler compiler.
Lexical Ruler Grammar
CompilerCompiler
Scanner Parser
Semantic Routines
Code Generator Fig. 44
The compiler writer also provides a set of semantic or code-generation routines. There is one such routine for each rule of the grammar. The parser each time it recognizes the language construct described by the associated rule calls this routine. Some compiler-compiler can parse a longer section of the program before calling a semantic routine. In that case, an internal form of the statements that have been analyzed, such as a portion of the parse tree, may be passed to the semantic routine. This approach is often used when code optimization is to be performed. Compiler-compilers frequently provide special languages, notations, data structures, and other similar facilities that can be used in the writing of semantic routines. The main advantage of using a compiler-compiler is case of compiler construction and testing. The amount of work required from the user varies considerably from one compiler to another depending upon the degree of flexibility provided. Compilers that are generated in this way tend to require more memory and compile programs more slowly than hand written compilers. However, the object code generated by the compiler may actually be better when a compiler-compiler is used. Because of the automatic construction of scanners and parsers and the special tools provided for writing semantic routines, the compiler writer is freed from many of the mechanical details of compiler construction. The write can therefore focus more attention on good code generation and optimization.
47
LEX INTRODUCTION Lex source is a table of regular expressions and corresponding program fragments. The table is translated to a program that reads an input stream, copying it to an output stream and partitioning the input into strings that match the given expressions. As each such string is recognized the corresponding program fragment is executed. The recognition of the expressions is performed by a deterministic finite automaton generated by Lex. The program fragments written by the user are executed in the order in which the corresponding regular expressions occur in the input stream. The lexical analysis programs written with Lex accept ambiguous specifications and choose the longest match possible at each input point. If necessary, substantial look ahead is performed on the input, but the input stream will be backed up to the end of the current partition, so that the user has general freedom to manipulate it. Lex is a program generator designed for lexical processing of character input streams. It accepts a high-level, problem oriented specification for character string matching, and produces a program in a general purpose language which recognizes regular expressions. The regular expressions are specified by the user in the source specifications given to Lex. The Lex written code recognizes these expressions in an input stream and partitions the input stream into strings matching the expressions. At the boundaries between strings program sections provided by the user are executed. The Lex source file associates the regular expressions and the program fragments. As each expression appears in the input to the program written by Lex, the corresponding fragment is executed. Lex is not a complete language, but rather a generator representing a new language feature which can be added to different programming languages, called ``host languages.'' Just as general purpose languages can produce code to run on different computer hardware, Lex can write code in different host languages. The host language is used for the output code generated by Lex and also for the program fragments added by the user. Compatible run-time libraries for the different host languages are also provided. This makes Lex adaptable to different environments and different users. Each application may be directed to the combination of hardware and host language appropriate to the task, the user's background, and the properties of local implementations. Several tools have been built for constructing lexical analyzers from special purpose notations based on regular expression. Lex is widely used tool to specify lexical analyzers for a variety of languages. We refer to the tool as the Lex compiler and to its input specification as the Lex language. Lex is generally used in the manner depicted in fig. 9.1. First a specification of a lexical analyzer is prepared by creating a program Lex-l in the Lex language. Lex-l is run through the Lex compiler to produce a C program Lex.YY.C. The program Lex.YY.C consists of a tabular representation of a transition diagram constructed from the regular expressions of lex.l, together with a standard routine that uses the table to recognize LEXEMER. The lexical analyses phase reads the characters in the source program and groups them into a stream of tokens in which each token represents a logically cohesive sequence of characters, such as an identifier, a keyword (if, while, etc.) a punctuation character or a multi-character operator like : = . The character sequence forming a token is called the lexeme for the token. The actions associated with regular
48
expressions in lex - a are pieces of C code and are carried over directly to lex. YY.C. Finally, lex .YY.C is run through the C compiler to produce an object program a.out.
Lex Source Program Lex Lex - 1 Compiler
Lex.YY.C
C Compiler
Lex.YY.C
a. out
Fig. 9.1 Creating a Lexical Analyzer with Lex Lex Source The general format of Lex source is: {definitions} %% {rules} %% {user subroutines} where the definitions and the user subroutines are often omitted. The second %% is optional, but the first is required to mark the beginning of the rules. The absolute minimum Lex program is thus %% (no definitions, no rules) which translates into a program which copies the input to the output unchanged. In the outline of Lex programs shown above, the rules represent the user's control decisions; they are a table, in which the left column contains regular expressions and the right column contains actions, program fragments to be executed when the expressions are recognized. Thus an individual rule might appear integer printf ("found keyword INT"); to look for the string integer in the input stream and print the message “found keyword INT'' whenever it appears. In this example the host procedural language is C and the C library function printf is used to print the string. The end of the expression is indicated by the first blank or tab character. If the action is merely a single C expression, it can just be given on the right side of the line; if it is compound, or takes more than a line, it should be enclosed in braces. As a slightly more useful example, suppose it is desired to change a number of words from British to American spelling. Lex rules such as colour printf("color"); mechanise printf("mechanize"); petrol printf("gas"); would be a start. These rules are not quite enough, since the word petroleum would become gas; a way of dealing with this will be described later. Lex Regular Expressions A regular expression specifies a set of strings to be matched. It contains text characters (which match the corresponding characters in the strings being compared) and operator characters (which specify repetitions, choices, and other features). The letters of the alphabet and the digits are always text characters; thus the regular expression integer matches the string integer wherever it appears and the expression A123Ba looks for the string A123Ba.
49
For a trivial example, consider a program to delete from the input all blanks or tabs at the ends of lines %% [ \t]+$ ; is all that is required. The program contains a %% delimiter to mark the beginning of the rules, and one rule. This rule contains a regular expression which matches one or more instances of the characters blank or tab (written \t for visibility, in accordance with the C language convention) just prior to the end of a line. The brackets indicate the character class made of blank and tab; the + indicates “one or more ...''; and the $ indicates “end of line''. No action is specified, so the program generated by Lex (yylex) will ignore these characters. Everything else will be copied. To change any remaining string of blanks or tabs to a single blank, add another rule: %% [ \t]+$ ; [ \t]+ printf(" "); The finite automation generated for this source will scan for rules at once, observing at the termination of the string of blanks or tabs whether or not there is a newline character, and executing the desired rule action. The first rule matches all strings of blanks or tabs at the end of lines, and the second rule all remaining strings of blanks or tabs. In the program written by Lex, the user's fragments (representing the actions to be performed as each regular expression is found) are gathered as cases of a switch. The automaton interpreter directs the control flow. Opportunity is provided for the user to insert either declarations or additional statements in the routine containing the actions, or to add subroutines outside this action routine. Operators: The operator characters are: " \ [ ] ^ - ? . * + | ( ) $ / { } % < >” and if they are to be used as text characters, an escape should be used. The quotation mark operator (") indicates that whatever is contained between a pair of quotes is to be taken as text characters. Thus xyz"++" matches the string xyz++ when it appears. Note that a part of a string may be quoted. harmless but unnecessary to quote an ordinary text character; the expression "xyz++"
It is
is the same as the one above. Thus by quoting every non-alphanumeric character being used as a text character, the user can avoid remembering the list. An operator character may also be turned into a text character by preceding it with \ as in xyz\+\+ which is another, less readable, equivalent of the above expressions. Another use of the quoting mechanism is to get a blank into an expression; normally, as explained above, blanks or tabs end a rule. Any blank character not contained within [] must be quoted. Several normal C escapes with \ are recognized: \n is new line, \t is tab, and \b is backspace. To enter \ itself, use \\. Since new line is illegal in an expression, \n must be used; it is not required to escape tab and backspace. Every character but blank, tab, new line and the list above is always a text character.
50
Character Classes Classes of characters can be specified using the operator pair []. The construction [abc] matches a single character, which may be a, b, or c. Within square brackets, most operator meanings are ignored. Only three characters are special: these are \ - and ^. The - character indicates ranges. For example, [a-z0-9<>_] indicates the character class containing all the lower case letters, the digits, the angle brackets, and underline. Ranges may be given in either order. Using - between any pair of characters which are not both upper case letters, both lower case letters, or both digits is implementation dependent and will get a warning message. (E.g., [0-z] in ASCII is many more characters than it is in EBCDIC). If it is desired to include the character - in a character class, it should be first or last; thus [-+0-9] matches all the digits and the two signs. In character classes, the ^ operator must appear as the first character after the left bracket; it indicates that the resulting string is to be complemented with respect to the computer character set. Thus [^abc] matches all characters except a, b, or c, including all special or control characters; or [^a-zA-Z] is any character which is not a letter. The \ character provides the usual escapes within character class brackets. Arbitrary Character To match almost any character, the operator character is the class of all characters except new line. Escaping into octal is possible although non-portable: [\40-\176] matches all printable characters in the ASCII character set, from octal 40 (blank) to octal 176 (tilde). Optional expressions: The operator ? Indicates an optional element of an expression. Thus ab?c matches either ac or abc. Repeated Expressions Repetitions of classes are indicated by the operators * and +. a* is any number of consecutive a characters, including no character; while a+ is one or more instances of a. For example, [a-z]+
51
is all strings of lower case letters. And [A-Za-z][A-Za-z0-9]* indicates all alphanumeric strings with a leading alphabetic character. This is a typical expression for recognizing identifiers in computer languages. Alternation and Grouping The operator | indicates alternation: (ab|cd) matches either ab or cd. Note that parentheses are used for grouping, although they are not necessary on the outside level; ab|cd would have sufficed. Parentheses can be used for more complex expressions: (ab|cd+)?(ef)* matches such strings as abefef, efefef, cdef, or cddd; but not abc, abcd, or abcdef. Context Sensitivity Lex will recognize a small amount of surrounding context. The two simplest operators for this are ^ and $. If the first character of an expression is ^, the expression will only be matched at the beginning of a line (after a newline character, or at the beginning of the input stream). This can never conflict with the other meaning of ^, complementation of character classes, since that only applies within the [] operators. If the very last character is $, the expression will only be matched at the end of a line (when immediately followed by newline). The latter operator is a special case of the / operator character, which indicates trailing context. The expression ab/cd matches the string ab, but only if followed by cd. Thus ab$ is the same as ab/\n Left context is handled in Lex by start conditions. If a rule is only to be executed when the Lex automaton interpreter is in start condition x, the rule should be prefixed by using the angle bracket operator characters. If we considered “being at the beginning of a line'' to be start condition ONE, then the ^ operator would be equivalent to Start conditions are explained more fully latter. Repetitions and Definitions The operators {} specify either repetitions (if they enclose numbers) or definition expansion (if they enclose a name). For example {digit} looks for a predefined string named digit and inserts it at that point in the expression. The definitions are given in the first part of the Lex input, before the rules. In contrast, a{1,5}
52
looks for 1 to 5 occurrences of a. Finally, initial % is special, being the separator for Lex source segments. The user will often want to know the actual text that matched some expression like [a-z]+. Lex leaves this text in an external character array named yytext. Thus, to print the name found, a rule like [a-z]+ printf("%s", yytext); will print the string in yytext. The C function printf accepts a format argument and data to be printed; in this case, the format is “print string'' (% indicating data conversion, and %s indicating string type), and the data are the characters in yytext. So this just places the matched string on the output. This action is so common that it may be written as ECHO: [a-z]+ ECHO; is the same as the above. Since the default action is just to print the characters found, one might ask why give a rule, like this one, which merely specifies the default action? Such rules are often required to avoid matching some other rule which is not desired. For example, if there is a rule which matches read it will normally match the instances of read contained in bread or readjust; to avoid this, a rule of the form [a-z]+ is needed. This is explained further below. Sometimes it is more convenient to know the end of what has been found; hence Lex also provides a count yyleng of the number of characters matched. To count both the number of words and the number of characters in words in the input, the user might write [a-zA-Z]+ {words++; chars += yyleng;} which accumulates in chars the number of characters in the words recognized. The last character in the string matched can be accessed by yytext[yyleng-1] Occasionally, a Lex action may decide that a rule has not recognized the correct span of characters. Two routines are provided to aid with this situation. First, yymore () can be called to indicate that the next input expression recognized is to be tacked on to the end of this input. Normally, the next input string would overwrite the current entry in yytext. Second, yyless (n) may be called to indicate that not all the characters matched by the currently successful expression are wanted right now. The argument n indicates the number of characters in yytext to be retained. Further characters previously matched are returned to the input. This provides the same sort of look ahead offered by the / operator, but in a different form. Example: Consider a language which defines a string as a set of characters between quotation (") marks, and provides that to include a "in a string it must be preceded by a \. The regular expression which matches that is somewhat confusing, so that it might be preferable to write \"[^"]* { if (yytext[yyleng-1] == '\\') yymore(); else ... normal user processing } which will, when faced with a string such as "abc\" “def" first match the five characters "abc\”; then the call to yymore() will cause the next part of the string, "def, to be tacked on the end. Note that the final quote terminating the string should be picked up in the code labeled ``normal processing''. The function yyless () might be used to reprocess text in various circumstances. Consider the C problem of distinguishing the ambiguity of “=-a''. Suppose it is desired to treat this as “=- a'' but print a message. A rule might be
53
=-[a-zA-Z] { printf("Op (=-) ambiguous\n"); yyless(yyleng-1); ... action for =- ... } which prints a message, returns the letter after the operator to the input stream, and treats the operator as ``=-''. Alternatively it might be desired to treat this as ``= -a''. To do this, just return the minus sign as well as the letter to the input: =-[a-zA-Z] { printf("Op (=-) ambiguous\n"); yyless(yyleng-2); ... action for = ... } will perform the other interpretation. Note that the expressions for the two cases might more easily be written =-/[A-Za-z] in the first case and =/-[A-Za-z] in the second; no backup would be required in the rule action. It is not necessary to recognize the whole identifier to observe the ambiguity. The possibility of “=-3'', however, makes =-/[^ \t\n] a still better rule. In addition to these routines, Lex also permits access to the I/O routines it uses. They are: 1) input() which returns the next input character; 2) output(c) which writes the character c on the output; and 3) unput(c) pushes the character c back onto the input stream to be read later by input(). By default these routines are provided as macro definitions, but the user can override them and supply private versions. These routines define the relationship between external files and internal characters, and must all be retained or modified consistently. They may be redefined, to cause input or output to be transmitted to or from strange places, including other programs or internal memory; but the character set used must be consistent in all routines; a value of zero returned by input must mean end of file; and the relationship between unput and input must be retained or the Lex lookahead will not work. Lex does not look ahead at all if it does not have to, but every rule ending in + * ? or $ or containing / implies lookahead. Lookahead is also necessary to match an expression that is a prefix of another expression. See below for a discussion of the character set used by Lex. The standard Lex library imposes a 100 character limit on backup. Another Lex library routine that the user will sometimes want to redefine is yywrap() which is called whenever Lex reaches an end-of-file. If yywrap returns a 1, Lex continues with the normal wrapup on end of input. Sometimes, however, it is convenient to arrange for more input to arrive from a new source. In this case, the user should provide a yywrap which arranges for new input and returns 0. This instructs Lex to continue processing. The default yywrap always returns 1. This routine is also a convenient place to print tables, summaries, etc. at the end of a program. Note that it is not possible to write a normal rule which recognizes end-of-file; the only access to this condition is through yywrap. In fact, unless a private version of input() is supplied a file containing nulls cannot be handled, since a value of 0 returned by input is taken to be end-of-file.
54
Ambiguous Source Rules Lex can handle ambiguous specifications. When more than one expression can match the current input, Lex chooses as follows: 1) The longest match is preferred. 2) Among rules which matched the same number of characters, the rule given first is preferred. Thus, suppose the rules integer keyword action ...; [a-z]+ identifier action ...; to be given in that order. If the input is integers, it is taken as an identifier, because [a-z]+ matches 8 characters while integer matches only 7. If the input is integer, both rules match 7 characters, and the keyword rule is selected because it was given first. Anything shorter (e.g. int) will not match the expression integer and so the identifier interpretation is used. The principle of preferring the longest match makes rules containing expressions like .* dangerous. For example, ' *' might seem a good way of recognizing a string in single quotes. But it is an invitation for the program to read far ahead, looking for a distant single quote. Presented with the input 'first' quoted string here, 'second' here the above expression will match 'first' quoted string here, 'second' which is probably not what was wanted. A better rule is of the form '[^'\n]*' which, on the above input, will stop after 'first'. The consequences of errors like this are mitigated by the fact that the . operator will not match newline. Thus expressions like .* stop on the current line. Don't try to defeat this with expressions like (.|\n)+ or equivalents; the Lex generated program will try to read the entire input file, causing internal buffer overflows. Note that Lex is normally partitioning the input stream, not searching for all possible matches of each expression. This means that each character is accounted for once and only once. For example, suppose it is desired to count occurrences of both she and he in an input text. Some Lex rules to do this might be she s++; he h++; \n | . ; where the last two rules ignore everything besides he and she. Remember that . does not include newline. Since she includes he, Lex will normally not recognize the instances of he included in she, since once it has passed a she those characters are gone. Sometimes the user would like to override this choice. The action REJECT means ``go do the next alternative.'' It causes whatever rule was second choice after the current rule to be executed. The position of the input pointer is adjusted accordingly. Suppose the user really wants to count the included instances of he: she he \n .
{s++; REJECT;} {h++; REJECT;} | ;
these rules are one way of changing the previous example to do just that. After counting each expression, it is rejected; whenever appropriate, the other expression will then be counted. In this example, of course, the user could note that she includes he but not vice versa, and omit the
55
REJECT action on he; in other cases, however, it would not be possible a priori to tell which input characters were in both classes. Consider the two rules a[bc]+ { ... ; REJECT;} a[cd]+ { ... ; REJECT;} If the input is ab, only the first rule matches, and on ad only the second matches. The input string accb matches the first rule for four characters and then the second rule for three characters. In contrast, the input accd agrees with the second rule for four characters and then the first rule for three. In general, REJECT is useful whenever the purpose of Lex is not to partition the input stream but to detect all examples of some items in the input, and the instances of these items may overlap or include each other. Suppose a digram table of the input is desired; normally the digrams overlap, that is the word the is considered to contain both th and he. Assuming a two-dimensional array named digram to be incremented, the appropriate source is %% [a-z][a-z] { digram[yytext[0]][yytext[1]]++; REJECT; } . ; \n ; where the REJECT is necessary to pick up a letter pair beginning at every character, rather than at every other character. Lex Source Definitions Remember the format of the Lex source: {definitions} %% {rules} %% {user routines} So far only the rules have been described. The user needs additional options, though, to define variables for use in his program and for use by Lex. These can go either in the definitions section or in the rules section. Remember that Lex is turning the rules into a program. Any source not intercepted by Lex is copied into the generated program. There are three classes of such things. 1) Any line which is not part of a Lex rule or action which begins with a blank or tab is copied into the Lex generated program. Such source input prior to the first %% delimiter will be external to any function in the code; if it appears immediately after the first %%, it appears in an appropriate place for declarations in the function written by Lex which contains the actions. This material must look like program fragments, and should precede the first Lex rule. As a side effect of the above, lines which begin with a blank or tab, and which contain a comment, are passed through to the generated program. This can be used to include comments in either the Lex source or the generated code. The comments should follow the host language convention.
56
2) Anything included between lines containing only %{ and %} is copied out as above. The delimiters are discarded. This format permits entering text like preprocessor statements that must begin in column 1, or copying lines that do not look like programs. 3) Anything after the third %% delimiter, regardless of formats, etc., is copied out after the Lex output. Definitions intended for Lex are given before the first %% delimiter. Any line in this section not contained between %{ and %}, and beginning in column 1, is assumed to define Lex substitution strings. The format of such lines is name translation and it causes the string given as a translation to be associated with the name. The name and translation must be separated by at least one blank or tab, and the name must begin with a letter. The translation can then be called out by the {name} syntax in a rule. Using {D} for the digits and {E} for an exponent field, for example, might abbreviate rules to recognize numbers: D [0-9] E [DEde][-+]?{D}+ %% {D}+ printf("integer"); {D}+"."{D}*({E})? | {D}*"."{D}+({E})? | {D}+{E} Note the first two rules for real numbers; both require a decimal point and contain an optional exponent field, but the first requires at least one digit before the decimal point and the second requires at least one digit after the decimal point. To correctly handle the problem posed by a Fortran expression such as 35.EQ.I, which does not contain a real number, a context-sensitive rule such as [0-9]+/"."EQ printf("integer"); could be used in addition to the normal rule for integers. The definitions section may also contain other commands, including the selection of a host language, a character set table, a list of start conditions, or adjustments to the default size of arrays within Lex itself for larger source programs. Lex and Yacc If you want to use Lex with Yacc, note that what Lex writes is a program named yylex(), the name required by Yacc for its analyzer. Normally, the default main program on the Lex library calls this routine, but if Yacc is loaded, and its main program is used, Yacc will call yylex(). In this case each Lex rule should end with return(token); where the appropriate token value is returned. An easy way to get access to Yacc's names for tokens is to compile the Lex output file as part of the Yacc output file by placing the line #include "lex.yy.c" in the last section of Yacc input. Supposing the grammar to be named “good'' and the lexical rules to be named “better'' the UNIX command sequence can just be: yacc good lex better cc y.tab.c -ly -ll The Yacc library (-ly) should be loaded before the Lex library, to obtain a main program which invokes the Yacc parser. The generations of Lex and Yacc programs can be done
57
in either order. 9.5 Examples 1. Write a Lex source program to copy an input file while adding 3 to every positive number divisible by 7. %% int k; [0-9]+ { k = atoi(yytext); if (k%7 == 0) printf("%d", k+3); else printf("%d",k); } to do just that. The rule [0-9]+ recognizes strings of digits; atoi converts the digits to binary and stores the result in k. The operator % (remainder) is used to check whether k is divisible by 7; if it is, it is incremented by 3 as it is written out. It may be objected that this program will alter such input items as 49.63 or X7. Furthermore, it increments the absolute value of all negative numbers divisible by 7. To avoid this, just add a few more rules after the active one, as here: %% int k; -?[0-9]+ { k = atoi(yytext); printf("%d", k%7 == 0 ? k+3 : k); } -?[0-9.]+ ECHO; [A-Za-z][A-Za-z0-9]+ ECHO; Numerical strings containing a “.'' or preceded by a letter will be picked up by one of the last two rules, and not changed. The if-else has been replaced by a C conditional expression to save space; the form a?b:c means “if a then b else c''. 2. Write a Lex program that histograms the lengths of words, where a word is defined as a string of letters. int lengs[100]; %% [a-z]+ lengs[yyleng]++; . | \n ; %% yywrap() { int i; printf("Length No. words\n"); for(i=0; i<100; i++) if (lengs[i] > 0) printf("%5d%10d\n",i,lengs[i]); return(1); } This program accumulates the histogram, while producing no output. At the end of the input it prints the table. The final statement return(1); indicates that Lex is to perform wrapup. If
58
yywrap returns zero (false) it implies that further input is available and the program is to continue reading and processing. To provide a yywrap that never returns true causes an infinite loop. THE LOOK AHEAD OPERATOR Lexical analyzers for certain programming language constructs need to look ahead beyond the end of a lexeme before they can determine a token with certainty. Example: Fortran DO Statement DO 5 I = 1.25 DO 5 I = 1,25 In Fortran, blanks are not significant outside of comments and Hollerith strings; Hence removing all the blanks in the above statements appears to the lexical analyzer as: DO5I = 1.25 DO5I = 1,25 In the first statement, we cannot tell until we see the decimal point that the string DO is part of the identifier DO5I. In the second statement, DO is a keyword by itself. In lex, a pattern of the form r1/r2 can be written. Where r1 & r2 - regular expressions and / is the operator. It means a string in r1 is followed by the string r2 with a ' / ' between the two strings indicated the right context for a match. It is used only to restrict a match not to be part of the match. Example: A lex specification that recognizes the keyword DO in the context above is DO/({letter} ; {digit } ) * = ( { letter} | {digit} ) * , With this specification, the lexical analyzer will look ahead in its input buffer for a sequence of letters and digits followed by an equal sign followed by letters and digit followed by a comma to be sure that it did not have an assignment statement. Then only the characters D and O, preceding the look ahead operator / would be part of the lexeme that was matched. After a successful match, yytext points to the D and yyleng = 2. Note that this simple look ahead pattern allows DO to be recognized when followed by garbage, like Z4 = 6Q, but it will never recognize DO that is part of an identifier. Example: Fortran IF Statement The look ahead operator can be used to cope with another difficult lexical analysis problem in Fortran i.e., distinguishing keywords from identifiers. IF ( I, J ) = 3 It is a perfect Fortran assignment statement, not a logical if statement. One way to specify the keyword IF using Lex is to define its possible right context using the look ahead operator. Logical if statement is IF (condition) statement From of logical - if - statement is IF (condition) THEN Then - block ELSE else - block END IF
59
We note that every unlabeled Fortran statement begins with a letter and that every right parentheses used for subscripting or operand grouping must be followed by an operator symbol such as =, + or comma, another right parentheses or the end of the statement. A letter cannot follow such a right parentheses. In this situation, to confirm that IF is a keyword rather than an array name we scan forward looking for a right parentheses followed by a letter before seeing a new time character. This pattern for the keyword IF can be written as IF / \ ( . * \ ) {letter} The . (dot) stands for "any character but new line" and the back slashes in front of the parentheses till Lex to treat them literally, not as meta symbols for grouping in regular expressions. The IF statements in Fortran can be solved by seeing IF ( . to determine whether IF has been declared an array. We scan for the full pattern indicated above if it has been so declared. Such texts makes the automatic implementation of a lexical analyzer from a Lex specification harder, and they may even cost time in the long run, since frequent checks must be made by the program simulating a transition diagram to determine whether any such tests must be made. Programs: The lex programs are written in a file with a dot extension. Example first.l. The program is executed in the following manner: % lex < filename.l > % cc lex.yy.c –o -ll The lex translates the lex specification into a C source file called lex.yy.c which we compile with the lex library –ll. We then execute the resulting program to check that it works as we expect. The exeution is as following: 1. If –o is not given run the program by using ./a.out 2. If –o is given run the program by using execfilename. 3. Enter the data after execution. 4. Use ^d to terminate the program and give result. There are two programs written for the above requirement. The one is to give the text online after execution. When carriage return is pressed the yylex is executed and the printf statement is carried on displaying on the screen the total number of vowels and consonants. This program is used to read the given text from a file. The file is opened in the read mode and the yylex function calls the base program to count the number of vowels and consonants. After it cover across the EOF, the C-program is executed to print the number of vowels and consonants. 1. Write a lex program to find the number of vowels and consonants. %{ /*
to find vowels and consonents*/ int vowels = 0; int consonents = 0;
%} %% [ \t\n]+ [aeiouAEIOU] vowels++; [bcdfghjklmnpqrstvwxyzBCDFGHJKLMNPQRSTVWXYZ] consonents++; . %% main() {
60
yylex(); printf(" The number of vowels = %d\n", vowels); printf(" number of consonents = %d \n", consonents); return(0); } The same program can be executed by giving alternative grammar. It is as follows: Here a file is opened which is given as a argument and reads to text and counts the number of vowels and consonants. %{ unsigned int vowelcount=0; unsigned int consocount=0; %} vowel [aeiouAEIOU] consonant [bcdfghjklmnpqrstvwxyzBCDFGHJKLMNPQRSTVWXYZ] eol \n %% {vowel} { vowelcount++;} {consonant} { consocount++; } %% main(int argc,char *argv[]) { if(argc > 1) { FILE *fp; fp=fopen(argv[1],"r"); if(!fp) { fprintf(stderr,"could not open %s\n",argv[1]); exit(1); } yyin=fp; } yylex(); printf(" vowelcount=%u consonantcount=%u\n ",vowelcount,consocount); return(0); }
2. Write a Lex program to count the number of words, characters, blanks and lines in a given text. %{
unsigned int charcount=0; int wordcount=0; int linecount=0; int blankcount =0;
%} word[^ \t\n]+ eol \n %% [ ] blankcount++; {word} { wordcount++; charcount+=yyleng;}
61
{eol} {charcount++; linecount++;} . { ECHO; charcount++;} %% main(argc, argv) int argc; char **argv; { if(argc > 1) { FILE *file; file = fopen(argv[1],"r"); if(!file) { fprintf(stderr, "could not open %s\n", argv[1]); exit(1); } yyin = file; yylex(); printf("\nThe number of characters = %u\n", charcount); printf("The number of wordcount = %u\n", wordcount); printf("The number of linecount = %u\n", linecount); printf("The number of blankcount = %u\n", blankcount); return(0); } else printf(" Enter the file name along with the program \n"); } 3. Write a lex program to find the number of positive integer, negative integer, positive floating positive number and negative floating point number. %{
int int int int
posnum negnum posflo negflo
= = = =
0; 0; 0; 0;
%} %% [\n\t ]; ([0-9]+) {posnum++;} -?([0-9]+) {negnum++; } ([0-9]*\.[0-9]+) { posflo++; } -?([0-9]*\.[0-9]+) { negflo++; } . ECHO; %% main() { yylex(); printf("Number of positive numbers = %d\n", posnum); printf("number of negative numbers = %d\n", negnum); printf("number of floting positive number = %d\n", posflo); printf("number of floating negative number = %d\n", negflo); }
62
4. Write a lex program to find the given c program has right number of brackets. Count the number of comments. Check for while loop. %{ /* int int int int int int int
find main, comments, {, (, ), } */ comments=0; opbr=0; clbr=0; opfl=0; clfl=0; j=0; k=0;
%} %% "main()" j=1; "/*"[ \t].*[ \t]"*/" comments++; "while("[0-9a-zA-Z]*")"[ \t]*\n"{"[ \t]*.*"}" k=1; ^[ \t]*"{"[ \t]*\n ^[ \t]*"}" k=1; "(" opbr++; ")" clbr++; "{" opfl++; "}" clfl++; [^ \t\n]+ . ECHO; %% main(argc, argv) int argc; char *argv[]; { if (argc > 1) { FILE *file; file = fopen(argv[1], "r"); if (!file) { printf("error opeing a file \n"); exit(1); } yyin = file; } yylex(); if(opbr != clbr) printf("open brackets is not equal to close brackets\n"); if(opfl != clfl) printf("open flower brackets is not equal to close flower brackets\n"); printf(" the number of comments = %d\n",comments); if (!j) printf("there is no main function \n"); if (k) printf("there is loop\n"); else printf("there is no valid for loop\n"); return(0); }
63
5. Write a lex program to replace scanf with READ and printf with WRITE statement also find the number of scanf and printf. %{ int pc=0,sc=0; %} %% printf fprintf(yyout,"WRITE");pc++; scanf fprintf(yyout,"READ");sc++; . ECHO; %% main(int argc,char* argv[]) { if(argc!=3) { printf("\nUsage: %s \n",argv[0]); return; } yyin=fopen(argv[1],"r"); yyout=fopen(argv[2],"w"); yylex(); printf("\nno. of printfs:%d\nno. of scanfs:%d\n",pc,sc); } 6. Write a lex program to find whether the given expression is valid. %{
#include int valid=0,ctr=0,oc = 0;
%} NUM [0-9]+ OP [+*/-] %% {NUM}({OP}{NUM})+ {
valid = 1; for(ctr = 0;yytext[ctr];ctr++) { switch(yytext[ctr]) { case '+': case '-': case '*': case '/': oc++; } } } {NUM}\n {printf("\nOnly a number.");} \n { if(valid) printf("valid \n operatorcount = %d",oc); else printf("Invalid"); valid = oc = 0;ctr=0; } %% main() { yylex(); }
64
/*
Another solution for the same problem
*/
%{ int oprc=0,digc=0,top=-1,flag=0; char stack[20]; %} digit [0-9]+ opr [+*/-] %% [ \n\t]+ ['('] {stack[++top]='(';} [')'] {flag=1; if(stack[top]=='('&&(top!=-1)) top--; else { printf("\n Invalid expression\n"); exit(0); } } {digit} {digc++;} {opr}/['('] { oprc++; printf("%s",yytext);} {opr}/{digit} {oprc++; printf("%s",yytext);} . {printf("Invalid "); exit(0);} %% main() { yylex(); if((digc==oprc+1||digc==oprc) && top==-1) { printf("VALID"); printf("\n oprc=%d\n digc=%d\n",oprc,digc); } else printf("INVALID"); } 7.Write a lex program to find the given sentence is simple or compound. %{ int flag=0; %} %% (" "[aA][nN][dD]" ")|(" "[oO][rR]" ")|(" "[bB][uU][tT]" ") flag=1; . ; %% main() {yylex(); if (flag==1) printf("COMPOUND SENTENCE \n"); else printf("SIMPLE SENTENCE \n"); } 8. Write a lex program to find the number of valid identifiers. %{
65
int count=0; %} %% (" int ")|(" float ")|(" double ")|(" char ") { int ch; ch = input(); for(;;) { if (ch==',') {count++;} else if(ch==';') {break;} ch = input(); } count++; } %% main(int argc,char *argv[]) { yyin=fopen(argv[1],"r"); yylex(); printf("the no of identifiers used is %d\n",count); }
Exercises 1. 2. 3. 4. 5.
How is Lexical analyzer created with Lex? How does Lex behave in concert with aparser? Why should the lex select the longest prefix match pattern? Why should lexical analyzer need to llo ahead to determine a token? Explain how the lexical analyzer will recognize IF statement in FOTRAN?
66
YACC Introduction Yacc provides a general tool for describing the input to a computer program. The Yacc user specifies the structures of his input, together with code to be invoked as each such structure is recognized. Yacc turns such a specification into a subroutine that handles the input process; frequently, it is convenient and appropriate to have most of the flow of control in the user's application handled by this subroutine. The input subroutine produced by Yacc calls a user-supplied routine to return the next basic input item. Thus, the user can specify his input in terms of individual input characters or in terms of higher level constructs such as names and numbers. The user supplied routine may also handle idiomatic features such as comment and continuation conventions, which typically defy easy grammatical specification. Yacc is written in portable C. Yacc provides a general tool for imposing structure on the input to a computer program. User prepares a specification of the input process; this includes rules describing the input structure, code to be invoked when these rules are recognized, and a low-level routine to do the basic input. Yacc then generates a function to control the input process. This function, called a parser, calls the user-supplied low-level input routine (the lexical analyzer) to pick up the basic items (called tokens) from the input stream. These tokens are organized according to the input structure rules, called grammar rules; when one of these rules has been recognized, then user code supplied for this rule, an action, is invoked; actions have the ability to return values and make use of the values of other actions. Yacc is written in a portable dialect of C and the actions, and output subroutine, are in C as well. Moreover, many of the syntactic conventions of Yacc follow C. The heart of the input specification is a collection of grammar rules. Each rule describes an allowable structure and gives it a name. For example, one grammar rule might be date : month_name day ',' year Here, date, month_name, day, and year represent structures of interest in the input process; presumably, month_name, day, and year are defined elsewhere. The comma ``,'' is enclosed in single quotes; this implies that the comma is to appear literally in the input. The colon and semicolon merely serve as punctuation in the rule, and have no significance in controlling the input. Thus, with proper definitions, the input July 4, 1776 might be matched by the above rule. An important part of the input process is carried out by the lexical analyzer. This user routine reads the input stream, recognizing the lower level structures, and communicates these tokens to the parser. For historical reasons, a structure recognized by the lexical analyzer is called a terminal symbol, while the structure recognized by the parser is called a nonterminal symbol. To avoid confusion, terminal symbols will usually be referred to as tokens.
67
There is considerable leeway (flexibility) in deciding whether to recognize structures using the lexical analyzer or grammar rules. For example, the rules month_name : 'J' 'a' 'n' ; month_name : 'F' 'e' 'b' ; ... month_name : 'D' 'e' 'c' ; might be used in the above example. The lexical analyzer would only need to recognize individual letters, and month_name would be a nonterminal symbol. Such low-level rules tend to waste time and space, and may complicate the specification beyond Yacc's ability to deal with it. Usually, the lexical analyzer would recognize the month names, and return an indication that a month_name was seen; in this case, month_name would be a token. Literal characters such as ``,'' must also be passed through the lexical analyzer, and are also considered tokens. Specification files are very flexible. It is relatively easy to add to the above example the rule date: month '/' day '/' year; allowing 7 / 4 / 1776 as a synonym for July 4, 1776. In most cases, this new rule could be ``slipped in'' to a working system with minimal effort, and little danger of disrupting existing input. The input being read may not conform to the specifications. These input errors are detected as early as is theoretically possible with a left-to-right scan; thus, not only is the chance of reading and computing with bad input data substantially reduced, but the bad data can usually be quickly found. Error handling, provided as part of the input specifications, permits the reentry of bad data, or the continuation of the input process after skipping over the bad data. In some cases, Yacc fails to produce a parser when given a set of specifications. For example, the specifications may be self contradictory, or they may require a more powerful recognition mechanism than that available to Yacc. The former cases represent design errors; the latter cases can often be corrected by making the lexical analyzer more powerful, or by rewriting some of the grammar rules. While Yacc cannot handle all possible specifications, its power compares favorably with similar systems; moreover, the constructions which are difficult for Yacc to handle are also frequently difficult for human beings to handle. Some users have reported that the discipline of formulating valid Yacc specifications for their input revealed errors of conception or design early in the program development. 1. Basic Specifications Every specification file consists of three sections: the declarations, (grammar) rules, and programs. The sections are separated by double percent ``%%'' marks. (The percent ``%'' is generally used in Yacc specifications as an escape character.) In other words, a full specification file looks like declarations %% rules %% programs
68
The declaration section may be empty. Moreover, if the programs section is omitted, the second %% mark may be omitted also; thus, the smallest legal Yacc specification is %% rules Blanks, tabs, and newlines are ignored except that they may not appear in names or multicharacter reserved symbols. Comments may appear wherever a name is legal; they are enclosed in /* . . . */, as in C and PL/I. The rules section is made up of one or more grammar rules. A grammar rule has the form: A : BODY ; A represents a nonterminal name, and BODY represents a sequence of zero or more names and literals. The colon and the semicolon are Yacc punctuation. Names may be of arbitrary length, and may be made up of letters, dot ``.'', underscore ``_'', and non-initial digits. Upper and lower case letters are distinct. The names used in the body of a grammar rule may represent tokens or nonterminal symbols. A literal consists of a character enclosed in single quotes ``'''. As in C, the backslash ``\'' is an escape character within literals, and all the C escapes are recognized. Thus '\n' newline '\r' return '\'' single quote ``''' '\\' backslash ``\'' '\t' tab '\b' backspace '\f' form feed '\xxx' ``xxx'' in octal For a number of technical reasons, the NUL character ('\0' or 0) should never be used in grammar rules. If there are several grammar rules with the same left hand side, the vertical bar ``|'' can be used to avoid rewriting the left hand side. In addition, the semicolon at the end of a rule can be dropped before a vertical bar. Thus the grammar rules A : B C D ; A : E F ; A : G ; can be given to Yacc as A : B C D | E F | G ; It is not necessary that all grammar rules with the same left side appear together in the grammar rules section, although it makes the input much more readable, and easier to change. If a nonterminal symbol matches the empty string, this can be indicated in the obvious way: empty : ; Names representing tokens must be declared; this is most simply done by writing %token name1, name2 . . .
69
in the declarations section. Every name not defined in the declarations section is assumed to represent a non-terminal symbol. Every non-terminal symbol must appear on the left side of at least one rule. Of all the nonterminal symbols, one, called the start symbol, has particular importance. The parser is designed to recognize the start symbol; thus, this symbol represents the largest, most general structure described by the grammar rules. By default, the start symbol is taken to be the left hand side of the first grammar rule in the rules section. It is possible, and in fact desirable, to declare the start symbol explicitly in the declarations section using the % start keyword: %start symbol The end of the input to the parser is signaled by a special token, called the endmarker. If the tokens up to, but not including, the endmarker form a structure which matches the start symbol, the parser function returns to its caller after the end-marker is seen; it accepts the input. If the endmarker is seen in any other context, it is an error. It is the job of the user-supplied lexical analyzer to return the endmarker when appropriate; see section 3, below. Usually the endmarker represents some reasonably obvious I/O status, such as ``end-of-file'' or ``end-of-record''. 2: Actions: With each grammar rule, the user may associate actions to be Yacc: Yet Another Compiler-Compiler performed each time the rule is recognized in the input process. These actions may return values, and may obtain the values returned by previous actions. Moreover, the lexical analyzer can return values for tokens, if desired. An action is an arbitrary C statement, and as such can do input and output, call subprograms, and alter external vectors and variables. An action is specified by one or more statements, enclosed in curly braces ``{'' and ``}''. For example, A : '(' B ')' { hello( 1, "abc" ); } and XXX : YYY ZZZ { printf("a message\n"); flag = 25; } are grammar rules with actions. To facilitate easy communication between the actions and the parser, the action statements are altered slightly. The symbol ``dollar sign'' ``$'' is used as a signal to Yacc in this context. To return a value, the action normally sets the pseudo-variable ``$$'' to some value. For example, an action that does nothing but return the value 1 is { $$ = 1; } To obtain the values returned by previous actions and the lexical analyzer, the action may use the pseudo-variables $1, $2, . . ., which refer to the values returned by the components of the right side of a rule, reading from left to right. Thus, if the rule is
70
A
:
B C D ;
for example, then $2 has the value returned by C, and $3 the value returned by D. As a more concrete example, consider the rule expr : '(' expr ')' ; The value returned by this rule is usually the value of the expr in parentheses. This can be indicated by expr : '(' expr ')' { $$ = $2 ; } By default, the value of a rule is the value of the first element in it ($1). Thus, grammar rules of the form A : B ; frequently need not have an explicit action. In the examples above, all the actions came at the end of their rules. Sometimes, it is desirable to get control before a rule is fully parsed. Yacc permits an action to be written in the middle of a rule as well as at the end. This rule is assumed to return a value, accessible through the usual mechanism by the actions to the right of it. In turn, it may access the values returned by the symbols to its left. Thus, in the rule A : B { $$ = 1; } C { x = $2; y = $3; } ; the effect is to set x to 1, and y to the value returned by C. Actions that do not terminate a rule are actually handled by Yacc by manufacturing a new nonterminal symbol name, and a new rule matching this name to the empty string. The interior action is the action triggered off by recognizing this added rule. Yacc actually treats the above example as if it had been written: $ACT
:
/* empty */ { $$ = 1; }
; A
:
B $ACT C { x = $2; y = $3; }
; In many applications, output is not done directly by the actions; rather, a data structure, such as a parse tree, is constructed in memory, and transformations are applied to it before output is generated. Parse trees are particularly easy to construct, given routines to build and maintain the tree structure desired. For example, suppose there is a C function node, written so that the call node( L, n1, n2 ) creates a node with label L, and descendants n1 and n2, and returns the index of the newly created node. Then parse tree can be built by supplying actions such as: expr : expr '+' expr { $$ = node( '+', $1, $3 ); } in the specification. The user may define other variables to be used by the actions. Declarations and definitions can appear in the declarations section, enclosed in the marks ``%{'' and ``%}''. These
71
declarations and definitions have global scope, so they are known to the action statements and the lexical analyzer. For example, %{ int variable = 0; %} could be placed in the declarations section, making variable accessible to all of the actions. The Yacc parser uses only names beginning in ``yy''; the user should avoid such names. In these examples, all the values are integers: a discussion of values of other types will be found in Section 10. 3: Lexical Analysis The user must supply a lexical analyzer to read the input stream and communicate tokens (with values, if desired) to the parser. The lexical analyzer is an integer-valued function called yylex. The user must supply a lexical analyzer to read the input stream and communicate tokens (with values, if desired) to the parser. The lexical analyzer is an integer-valued function called yylex. The parser and the lexical analyzer must agree on these token numbers in order for communication between them to take place. The numbers may be chosen by Yacc, or chosen by the user. In either case, the ``# define'' mechanism of C is used to allow the lexical analyzer to return these numbers symbolically. For example, suppose that the token name DIGIT has been defined in the declarations section of the Yacc specification file. The relevant portion of the lexical analyzer might look like: yylex(){ extern int yylval; int c; ... c = getchar(); ... switch( c ) { ... case '0': case '1': ... case '9': yylval = c-'0'; return( DIGIT ); ... } ... The intent is to return a token number of DIGIT, and a value equal to the numerical value of the digit. Provided that the lexical analyzer code is placed in the programs section of the specification file, the identifier DIGIT will be defined as the token number associated with the token DIGIT. This mechanism leads to clear, easily modified lexical analyzers; the only pitfall is the need to avoid using any token names in the grammar that are reserved or significant in C or the parser; for example, the use of token names ‘if’ or ‘while’ will almost certainly cause severe difficulties when the lexical analyzer is compiled. The token name error is reserved for error handling, and should not be used naively.
72
As mentioned above, the token numbers may be chosen by Yacc or by the user. In the default situation, the numbers are chosen by Yacc. The default token number for a literal character is the numerical value of the character in the local character set. Other names are assigned token numbers starting at 257. To assign a token number to a token (including literals), the first appearance of the token name or literal in the declarations section can be immediately followed by a nonnegative integer. This integer is taken to be the token number of the name or literal. Names and literals not defined by this mechanism retain their default definition. It is important that all token numbers be distinct. For historical reasons, the end marker must have token number 0 or negative. This token number cannot be redefined by the user; thus, all lexical analyzers should be prepared to return 0 or negative as a token number upon reaching the end of their input. A very useful tool for constructing lexical analyzers is the Lex program developed by Mike Lesk. [8] These lexical analyzers are designed to work in close harmony with Yacc parsers. The specifications for these lexical analyzers use regular expressions instead of grammar rules. Lex can be easily used to produce quite complicated lexical analyzers, but there remain some languages (such as FORTRAN) which do not fit any theoretical framework, and whose lexical analyzers must be crafted by hand. 4: How the Parser Works Yacc turns the specification file into a C program, which parses the input according to the specification given. The algorithm used to go from the specification to the parser is complex, and will not be discussed here (see the references for more information). The parser itself, however, is relatively simple, and understanding how it works, while not strictly necessary, will nevertheless make treatment of error recovery and ambiguities much more comprehensible. The parser produced by Yacc consists of a finite state machine with a stack. The parser is also capable of reading and remembering the next input token (called the lookahead token). The current state is always the one on the top of the stack. The states of the finite state machine are given small integer labels; initially, the machine is in state 0, the stack contains only state 0, and no lookahead token has been read. The machine has only four actions available to it, called shift, reduce, accept, and error. A move of the parser is done as follows: 1. Based on its current state, the parser decides whether it needs a lookahead token to decide what action should be done; if it needs one, and does not have one, it calls yylex to obtain the next token. 2. Using the current state, and the lookahead token if needed, the parser decides on its next action, and carries it out. This may result in states being pushed onto the stack, or popped off the stack, and in the lookahead token being processed or left alone. The shift action is the most common action the parser takes. Whenever a shift action is taken, there is always a lookahead token. For example, in state 56 there may be an action: IF
shift 34
73
which says, in state 56, if the lookahead token is IF, the current state (56) is pushed down on the stack, and state 34 becomes the current state (on the top of the stack). The look ahead token is cleared. The reduce action keeps the stack from growing without bounds. Reduce actions are appropriate when the parser has seen the right hand side of a grammar rule, and is prepared to announce that it has seen an instance of the rule, replacing the right hand side by the left hand side. It may be necessary to consult the lookahead token to decide whether to reduce, but usually it is not; in fact, the default action (represented by a ``.'') is often a reduce action. Reduce actions are associated with individual grammar rules. Grammar rules are also given small integer numbers, leading to some confusion. The action .
reduce 18
refers to grammar rule 18, while the action IF shift 34 refers to state 34. Suppose the rule being reduced is A : x y z ; The reduce action depends on the left hand symbol (A in this case), and the number of symbols on the right hand side (three in this case). To reduce, first pop off the top three states from the stack (In general, the number of states popped equals the number of symbols on the right side of the rule). In effect, these states were the ones put on the stack while recognizing x, y, and z, and no longer serve any useful purpose. After popping these states, a state is uncovered which was the state the parser was in before beginning to process the rule. Using this uncovered state, and the symbol on the left side of the rule, perform what is in effect a shift of A. A new state is obtained, pushed onto the stack, and parsing continues. There are significant differences between the processing of the left hand symbol and an ordinary shift of a token, however, so this action is called a goto action. In particular, the lookahead token iscleared by a shift, and is not affected by a goto. In any case, the uncovered state contains an entry such as: A goto 20 causing state 20 to be pushed onto the stack, and become the current state. n effect, the reduce action ``turns back the clock'' in the parse, popping the states off the stack to go back to the state where the right hand side of the rule was first seen. The parser then behaves as if it had seen the left side at that time. If the right hand side of the rule is empty, no states are popped off of the stack: the uncovered state is in fact the current state. The reduce action is also important in the treatment of user-supplied actions and values. When a rule is reduced, the code supplied with the rule is executed before the stack is adjusted. In addition to the stack holding the states, another stack, running in parallel with it, holds the values returned from the lexical analyzer and the actions. When a shift takes place, the external variable yylval is copied onto the value stack. After the return from the user code, the reduction is carried out. When the goto action is done, the external variable yyval is copied onto the value stack. The pseudo-variables $1, $2, etc., refer to the value stack. The other two parser actions are conceptually much simpler. The accept action indicates that the entire input has been seen and that it matches the specification. This action appears only when the lookahead token is the endmarker, and indicates that the parser has successfully done its job. The error action, on the other hand, represents a place where the parser can no longer continue parsing according to the specification. The input tokens it has seen, together with the lookahead token, cannot be followed by anything that would result in a legal input. The parser reports an error, and attempts to recover the situation and resume parsing: the error recovery (as opposed to the detection of error).
74
Example1: Consider the specification %token DING DONG DELL %% rhyme : sound place ; sound : DING DONG ; place : DELL ; When Yacc is invoked with the -v option, a file called y.output is produced, with a human-readable description of the parser. The y.output file corresponding to the above grammar (with some statistics stripped off the end) is: state 0 $accept : _rhyme $end DING shift 3 . error rhyme goto 1 sound goto 2 state 1 $accept : rhyme_$end $end accept . error state 2 rhyme : sound_place DELL shift 5 . error place goto 4 state 3 sound : DING_DONG DONG shift 6 . error state 4 rhyme : sound place_ (1) . reduce 1 state 5 place : DELL_ (3) . reduce 3 state 6 sound : DING DONG_ (2) . reduce 2 Notice that, in addition to the actions for each state, there is a description of the parsing rules being processed in each state. The _ character is used to indicate what has been seen, and what is yet to come, in each rule. Suppose the input is DING DONG DELL. It is instructive to follow the steps of the parser while processing this input.
75
nitially, the current state is state 0. The parser needs to refer to the input in order to decide between the actions available in state 0, so the first token, DING, is read, becoming the lookahead token. The action in state 0 on DING is is ``shift 3'', so state 3 is pushed onto the stack, and the lookahead token is cleared. State 3 becomes the current state. The next token, DONG, is read, becoming the lookahead token. The action in state 3 on the token DONG is ``shift 6'', so state 6 is pushed onto the stack, and the lookahead is cleared. The stack now contains 0, 3, and 6. In state 6, without even consulting the lookahead, the parser reduces by rule 2. sound : DING DONG This rule has two symbols on the right hand side, so two states, 6 and 3, are popped off of the stack, uncovering state 0. Consulting the description of state 0, looking for a goto on sound, sound goto 2 is obtained; thus state 2 is pushed onto the stack, becoming the current state. In state 2, the next token, DELL, must be read. The action is ``shift 5'', so state 5 is pushed onto the stack, which now has 0, 2, and 5 on it, and the lookahead token is cleared. In state 5, the only action is to reduce by rule 3. This has one symbol on the right hand side, so one state, 5, is popped off, and state 2 is uncovered. The goto in state 2 on place, the left side of rule 3, is state 4. Now, the stack contains 0, 2, and 4. In state 4, the only action is to reduce by rule 1. There are two symbols on the right, so the top two states are popped off, uncovering state 0 again. In state 0, there is a goto on rhyme causing the parser to enter state 1. In state 1, the input is read; the endmarker is obtained, indicated by ``$end'' in the y.output file. The action in state 1 when the endmarker is seen is to accept, successfully ending the parse. The reader is urged to consider how the parser works when confronted with such incorrect strings as DING DONG DONG, DING DONG, DING DONG DELL DELL, etc. A few minutes spend with this and other simple examples will probably be repaid when problems arise in more complicated contexts. 5: Ambiguity and Conflicts A set of grammar rules is ambiguous if there is some input string that can be structured in two or more different ways. For example, the grammar rule expr
:
expr '-' expr
is a natural way of expressing the fact that one way of forming an arithmetic expression is to put two other expressions together with a minus sign between them. Unfortunately, this grammar rule does not completely specify the way that all complex inputs should be structured. For example, if the input is expr - expr - expr the rule allows this input to be structured as either ( expr - expr ) - expr or as expr - ( expr - expr ) (The first is called left association, the second right association). Yacc detects such ambiguities when it is attempting to build the parser. It is instructive to consider the problem that confronts the parser when it is given an input such as
76
expr - expr - expr When the parser has read the second expr, the input that it has seen: expr - expr matches the right side of the grammar rule above. The parser could reduce the input by applying this rule; after applying the rule; the input is reduced to expr (the left side of the rule). The parser would then read the final part of the input: - expr and again reduce. The effect of this is to take the left associative interpretation. Alternatively, when the parser has seen expr - expr it could defer the immediate application of the rule, and continue reading the input until it had seen expr - expr - expr It could then apply the rule to the rightmost three symbols, reducing them to expr and leaving expr - expr Now the rule can be reduced once more; the effect is to take the right associative interpretation. Thus, having read expr - expr the parser can do two legal things, a shift or a reduction, and has no way of deciding between them. This is called a shift / reduce conflict. It may also happen that the parser has a choice of two legal reductions; this is called a reduce / reduce conflict. Note that there are never any ``Shift/shift'' conflicts. When there are shift/reduce or reduce/reduce conflicts, Yacc still produces a parser. It does this by selecting one of the valid steps wherever it has a choice. A rule describing which choice to make in a given situation is called a disambiguating rule. Yacc invokes two disambiguating rules by default: 1. In a shift/reduce conflict, the default is to do the shift. 2. In a reduce/reduce conflict, the default is to reduce by the earlier grammar rule (in the input sequence). Rule 1 implies that reductions are deferred whenever there is a choice, in favor of shifts. Rule 2 gives the user rather crude control over the behavior of the parser in this situation, but reduce/reduce conflicts should be avoided whenever possible. Conflicts may arise because of mistakes in input or logic, or because the grammar rules, while consistent, require a more complex parser than Yacc can construct. The use of actions within rules can also cause conflicts, if the action must be done before the parser can be sure which rule is being recognized. In these cases, the application of disambiguating rules is inappropriate, and leads to an incorrect parser. For this reason, Yacc always reports the number of shift/reduce and reduce/reduce conflicts resolved by Rule 1 and Rule 2.
77
In general, whenever it is possible to apply disambiguating rules to produce a correct parser, it is also possible to rewrite the grammar rules so that the same inputs are read but there are no conflicts. For this reason, most previous parser generators have considered conflicts to be fatal errors. Our experience has suggested that this rewriting is somewhat unnatural, and produces slower parsers; thus, Yacc will produce parsers even in the presence of conflicts. As an example of the power of disambiguating rules, consider a fragment from a programming language involving an ``if-then-else'' construction: stat
: | ;
IF '(' cond ')' stat IF '(' cond ')' stat ELSE stat
In these rules, IF and ELSE are tokens, cond is a nonterminal symbol describing conditional (logical) expressions, and stat is a nonterminal symbol describing statements. The first rule will be called the simple-if rule, and the second the if-else rule. These two rules form an ambiguous construction, since input of the form IF ( C1 ) IF ( C2 ) S1 ELSE S2 can be structured according to these rules in two ways: IF ( C1 ) { IF ( C2 ) S1 } ELSE S2 or IF ( C1 ) { IF ( C2 ) S1 ELSE S2 } The second interpretation is the one given in most programming languages having this construct. Each ELSE is associated with the last preceding ``un-ELSE'd'' IF. In this example, consider the situation where the parser has seen IF ( C1 ) IF ( C2 ) S1 and is looking at the ELSE. It can immediately reduce by the simple-if rule to get IF ( C1 ) stat and then read the remaining input, ELSE S2 and reduce IF ( C1 ) stat ELSE S2 by the if-else rule. This leads to the first of the above groupings of the input. On the other hand, the ELSE may be shifted, S2 read, and then the right hand portion of IF ( C1 ) IF ( C2 ) S1 ELSE S2 can be reduced by the if-else rule to get IF ( C1 ) stat
78
which can be reduced by the simple-if rule. This leads to the second of the above groupings of the input, which is usually desired. Once again the parser can do two valid things - there is a shift/reduce conflict. The application of disambiguating rule 1 tells the parser to shift in this case, which leads to the desired grouping. This shift/reduce conflict arises only when there is a particular current input symbol, ELSE, and particular inputs already seen, such as IF ( C1 ) IF ( C2 ) S1 In general, there may be many conflicts, and each one will be associated with an input symbol and a set of previously read inputs. The previously read inputs are characterized by the state of the parser. The conflict messages of Yacc are best understood by examining the verbose (-v) option output file. For example, the output corresponding to the above conflict state might be: 23: shift/reduce conflict (shift 45, reduce 18) on ELSE state 23 stat : IF ( cond ) stat_ (18) stat : IF ( cond ) stat_ELSE stat ELSE shift 45 . reduce 18 The first line describes the conflict, giving the state and the input symbol. The ordinary state description follows, giving the grammar rules active in the state, and the parser actions. Recall that the underline marks the portion of the grammar rules which has been seen. Thus in the example, in state 23 the parser has seen input corresponding to IF ( cond ) stat and the two grammar rules shown are active at this time. The parser can do two possible things. If the input symbol is ELSE, it is possible to shift into state 45. State 45 will have, as part of its description, the line stat : IF ( cond ) stat ELSE_stat since the ELSE will have been shifted in this state. Back in state 23, the alternative action, described by ``.'', is to be done if the input symbol is not mentioned explicitly in the above actions; thus, in this case, if the input symbol is not ELSE, the parser reduces by grammar rule 18: stat : IF '(' cond ')' stat Once again, notice that the numbers following ``shift'' commands refer to other states, while the numbers following ``reduce'' commands refer to grammar rule numbers. In the y.output file, the rule numbers are printed after those rules which can be reduced. In most one states, there will be at most reduce action possible in the state, and this will be the default command. The user who encounters unexpected shift/reduce conflicts will probably want to look at the verbose output to decide whether the default actions are appropriate. In really tough cases, the user might need to know more about the behavior and construction of the parser than can be covered here. In this case, one of the theoretical references[2, 3, 4] might be consulted; the services of a local guru might also be appropriate. 6: Precedence There is one common situation where the rules given above for resolving conflicts are not sufficient; this is in the parsing of arithmetic expressions. Most of the commonly used constructions for arithmetic expressions can be naturally described by the notion of precedence levels for operators, together with information about left or right associatively. It turns out that ambiguous grammars with appropriate disambiguating rules can be used to create parsers that are faster and easier to write than parsers constructed from unambiguous grammars. The basic notion is to write grammar rules of the form expr : expr OP expr
79
and expr : UNARY expr for all binary and unary operators desired. This creates a very ambiguous grammar, with many parsing conflicts. As disambiguating rules, the user specifies the precedence, or binding strength, of all the operators, and the associativity of the binary operators. This information is sufficient to allow Yaccto resolve the parsing conflicts in accordance with these rules, and construct a parser that realizes the desired precedences and associativities. The precedences and associativities are attached to tokens in the declarations section. This is done by a series of lines beginning with a Yacc keyword: %left, %right, or %nonassoc, followed by a list of tokens. All of the tokens on the same line are assumed to have the same precedence level and associativity; the lines are listed in order of increasing precedence or binding strength. Thus, %left '+' '-' %left '*' '/' describes the precedence and associativity of the four arithmetic operators. Plus and minus are left associative, and have lower precedence than star and slash, which are also left associative. The keyword %right is used to describe right associative operators, and the keyword %nonassoc is used to describe operators, like the operator .LT. in Fortran, that may not associate with themselves; thus, A .LT. B .LT. C is illegal in Fortran, and such an operator would be described with the keyword %nonassoc in Yacc. As an example of the behavior of these declarations, the description %right '=' %left '+' '-' %left '*' '/' %% expr : expr '=' expr | expr '+' expr | expr '-' expr | expr '*' expr | expr '/' expr | NAME ; might be used to structure the input a = b = c*d - e - f*g as follows: a = ( b = ( ((c*d)-e) - (f*g) ) ) When this mechanism is used, unary operators must, in general, be given a precedence. Sometimes a unary operator and a binary operator have the same symbolic representation, but different precedences. An example is unary and binary '-'; unary minus may be given the same strength as multiplication, or even higher, while binary minus has a lower strength than multiplication. The keyword, %prec, changes the precedence level associated with a particular grammar rule. %prec appears immediately after the body of the grammar rule, before the action or closing semicolon, and is followed by a token name or literal. It causes the precedence of the grammar rule to become that of the following token name or literal. For example, to make unary minus have the same precedence as multiplication the rules might resemble: %left '+' '-' %left '*' '/' %% expr : expr '+' expr | expr '-' expr
80
| expr '*' expr | expr '/' expr | '-' expr %prec '*' | NAME ; A token declared by %left, %right, and %nonassoc need not be, but may be, declared by %token as well. The precedence and associatively are used by Yacc to resolve parsing conflicts; they give rise to disambiguating rules. Formally, the rules work as follows: 1.
The precedences and associativities are recorded for those tokens and literals that have them. 2. A precedence and associativity is associated with each grammar rule; it is the precedence and associativity of the last token or literal in the body of the rule. If the %prec construction is used, it overrides this default. Some grammar rules may have no precedence and associativity associated with them. 3. When there is a reduce/reduce conflict, or there is a shift/reduce conflict and either the input symbol or the grammar rule has no precedence and associativity, then the two disambiguating rules given at the beginning of the section are used, and the conflicts are reported. 4. If there is a shift/reduce conflict, and both the grammar rule and the input character have precedence and associativity associated with them, then the conflict is resolved in favor of the action (shift or reduce) associated with the higher precedence. If the precedences are the same, then the associativity is used; left associative implies reduce, right associative implies shift, and nonassociating implies error. Conflicts resolved by precedence are not counted in the number of shift/reduce and reduce/reduce conflicts reported by Yacc. This means that mistakes in the specification of precedences may disguise errors in the input grammar; it is a good idea to be sparing with precedences, and use them in an essentially ``cookbook'' fashion, until some experience has been gained. The y.output file is very useful in deciding whether the parser is actually doing what was intended. 7: Error Handling Error handling is an extremely difficult area, and many of the problems are semantic ones. When an error is found, for example, it may be necessary to reclaim parse tree storage, delete or alter symbol table entries, and, typically, set switches to avoid generating any further output. It is seldom acceptable to stop all processing when an error is found; it is more useful to continue scanning the input to find further syntax errors. This leads to the problem of getting the parser ``restarted'' after an error. A general class of algorithms to do this involves discarding a number of tokens from the input string, and attempting to adjust the parser so that input can continue. To allow the user some control over this process, Yacc provides a simple, but reasonably general, feature. The token name ``error'' is reserved for error handling. This name can be used in grammar rules; in effect, it suggests places where errors are expected, and recovery might take place. The parser pops its stack until it enters a state where the token ``error'' is legal. It then behaves as if the token ``error'' were the current lookahead token, and performs the action encountered. The lookahead token is then reset to the token that caused the error. If no special error rules have been specified, the processing halts when an error is detected.
81
In order to prevent a cascade of error messages, the parser after detecting an error, remains in error state until three tokens have been successfully read and shifted. If an error is detected when the parser is already in error state, no message is given, and the input token is quietly deleted. As an example, a rule of the form stat : error would, in effect, mean that on a syntax error the parser would attempt to skip over the statement in which the error was seen. More precisely, the parser will scan ahead, looking for three tokens that might legally follow a statement, and start processing at the first of these; if the beginnings of statements are not sufficiently distinctive, it may make a false start in the middle of a statement, and end up reporting a second error where there is in fact no error. Actions may be used with these special error rules. These actions might attempt to reinitialize tables, reclaim symbol table space, etc. Error rules such as the above are very general, but difficult to control. Somewhat easier are rules such as stat : error ';' Here, when there is an error, the parser attempts to skip over the statement, but will do so by skipping to the next ';'. All tokens after the error and before the next ';' cannot be shifted, and are discarded. When the ';' is seen, this rule will be reduced, and any ``cleanup'' action associated with it performed. Another form of error rule arises in interactive applications, where it may be desirable to permit a line to be reentered after an error. A possible error rule might be input : error '\n' { printf( "Reenter last line: " ); } input { $$ = $4; } There is one potential difficulty with this approach; the parser must correctly process three input tokens before it admits that it has correctly resynchronized after the error. If the reentered line contains an error in the first two tokens, the parser deletes the offending tokens, and gives no message; this is clearly unacceptable. For this reason, there is a mechanism that can be used to force the parser to believe that an error has been fully recovered from. The statement yyerrok ; in an action resets the parser to its normal mode. The last example is better written input : error '\n' { yyerrok; printf( "Reenter last line: " ); } input { $$ = $4; } ; As mentioned above, the token seen immediately after the ``error'' symbol is the input token at which the error was discovered. Sometimes, this is inappropriate; for example, an error recovery action might take upon itself the job of finding the correct place to resume input. In this case, the previous lookahead token must be cleared. The statement yyclearin ; in an action will have this effect. For example, suppose the action after error were to call some sophisticated resynchronization routine, supplied by the user, that attempted to advance the input to the beginning of the next valid statement. After this routine was called, the next token returned by yylex would presumably be the first token in a legal statement; the old, illegal token must be discarded, and the error state reset. This could be done by a rule like stat : error { resynch(); yyerrok ;
82
yyclearin ; } ; These mechanisms are admittedly crude, but do allow for a simple, fairly effective recovery of the parser from many errors; moreover, the user can get control to deal with the error actions required by other portions of the program. Left Recursion The algorithm used by the Yacc parser encourages so called ``left recursive'' grammar rules: rules of the form name : name rest_of_rule ; These rules frequently arise when writing specifications of sequences and lists: list : item | list ',' item ; and seq : item | seq item ; In each of these cases, the first rule will be reduced for the first item only, and the second rule will be reduced for the second and all succeeding items. With right recursive rules, such as seq : item | item seq ; the parser would be a bit bigger, and the items would be seen, and reduced, from right to left. More seriously, an internal stack in the parser would be in danger of overflowing if a very long sequence were read. Thus, the user should use left recursion wherever reasonable. It is worth considering whether a sequence with zero elements has any meaning, and if so, consider writing the sequence specification with an empty rule: seq : /* empty */ | seq item ; Once again, the first rule would always be reduced exactly once, before the first item was read, and then the second rule would be reduced once for each item read. Permitting empty equences often leads to increased generality. However, conflicts might arise if Yacc is asked to decide which empty sequence it has seen, when it hasn't seen enough to know! Lexical Tie-ins Some lexical decisions depend on context. For example, the lexical analyzer might want to delete blanks normally, but not within quoted strings. Or names might be entered into a symbol table in declarations, but not in expressions. One way of handling this situation is to create a global flag that is examined by the lexical analyzer, and set by actions. For example, suppose a program consists of 0 or more declarations, followed by 0 or more statements. Consider: %{ int dflag; %} ... other declarations ... %% prog : decls stats ; decls : /* empty */ { dflag = 1; }
83
| ; stats :
decls declaration /* empty */ { dflag = 0; } stats statement
| ; ... other rules ... The flag dflag is now 0 when reading statements, and 1 when reading declarations, except for the first token in the first statement. This token must be seen by the parser before it can tell that the declaration section has ended and the statements have begun. In many cases, this single token exception does not affect the lexical scan. This kind of ``backdoor'' approach can be elaborated to a noxious degree. Nevertheless, it represents a way of doing some things that are difficult, if not impossible, to do otherwise. Reserved Words Some programming languages permit the user to use words like ``if'', which are normally reserved, as label or variable names,provided that such use does not conflict with the legal use of these names in the programming language. This is extremely hard to do in the framework of Yacc; it is difficult to pass information to the lexical analyzer telling it ``this instance of `if' is a keyword, and that instance is a variable''. The user can make a stab at it, using the mechanism described in the last subsection, but it is difficult. A number of ways of making this easier are under advisement. Until then, it is better that the keywords be reserved; that is, be forbidden for use as variable names. There are powerful stylistic reasons for preferring this, anyway. Example This example gives the complete Yacc specification for a small desk calculator; the desk calculator has 26 registers, labeled ``a'' through ``z'', and accepts arithmetic expressions made up of the operators +, -, *, /, % (mod operator), & (bitwise and), | (bitwise or), and assignment. If an expression at the top level is an assignment, the value is not printed; otherwise it is. As in C, an integer that begins with 0 (zero) is assumed to be octal; otherwise, it is assumed to be decimal. As an example of a Yacc specification, the desk calculator does a reasonable job of showing how precedence and ambiguities are used, and demonstrating simple error recovery. The major oversimplifications are that the lexical analysis phase is much simpler than for most applications, and the output is produced immediately, line by line. Note the way that decimal and octal integers are read in by the grammar rules; This job is probably better done by the lexical analyzer. %{ # include # include int regs[26]; int base; %} %start list %token DIGIT LETTER %left '|' %left '&' %left '+' '-' %left '*' '/' '%' %left UMINUS /* supplies precedence for unary minus */ %% /* beginning of rules section */
84
list : /* empty */ | list stat '\n' | list error '\n' { yyerrok; } ; stat :
expr { printf( "%d\n", $1 ); } | LETTER '=' expr { regs[$1] = $3; } ; expr : '(' expr ')' { $$ = $2; } | expr '+' expr { $$ = $1 + $3; } | expr '-' expr { $$ = $1 - $3; } | expr '*' expr { $$ = $1 * $3; } | expr '/' expr { $$ = $1 / $3; } | expr '%' expr { $$ = $1 % $3; } | expr '&' expr { $$ = $1 & $3; } | expr '|' expr { $$ = $1 | $3; } | '-' expr %prec UMINUS { $$ = - $2; } | LETTER { $$ = regs[$1]; } | number ; number
: DIGIT { $$ = $1; base = ($1==0) ? 8 : 10; } | number DIGIT { $$ = base * $1 + $2; } ;
%%
/* start of programs */
yylex() { /* /* /*
/* lexical analysis routine */ returns LETTER for a lower case letter, yylval = 0 through 25 */ return DIGIT for a digit, yylval = 0 through 9 */ all other characters are returned immediately */
int c; while( (c=getchar()) == ' ' ) {/* skip blanks */ }
85
/* c is now nonblank */ if( islower( c ) ) { yylval = c - 'a'; return ( LETTER ); } if( isdigit( c ) ) { yylval = c - '0'; return( DIGIT ); } return( c ); } Problems: The Yacc programs can be executed in two ways. The yacc program itself will have c-code which passes the tokens. The program has to convert the typed number to digit and pass the number to the yacc program. Then the yacc program can be executed by giving the command: yacc The output of this execution results in y.tab.c file. This file can be compiled to get the executable file. The compilation is as follows: cc y.tab.c –o -o is optional. If –o is not used the a.out will be the executable file. Else outfilename will be the executable file. 5. The yacc program gets the tokens from the lex program. Hence a lex program has be written to pass the tokens to the yacc. That means we have to follow different procedure to get the executable file. i. The lex program is fist compiled using lex compiler to get lex.yy.c. ii. The yacc program is compiled using yacc compiler to get y.tab.c. iii. Using c compiler b+oth the lex and yacc intermediate files are compiled with the lex library function. cc y.tab.c lex.yy.c –ll. iv. If necessary out file name can be included during compiling with –o option. 1. Write a Yacc program to test validity of a simple expression with +, - , /, and *. /* Lex program that passes tokens */ %{ #include "y.tab.h" extern int yyparse(); %} %% [0-9]+ { return NUM;} [a-zA-Z_][a-zA-Z_0-9]* { return IDENTIFIER;} [+-] {return ADDORSUB;} [*/] {return PROORDIV;} [)(] {return yytext[0];} [\n] {return '\n';} %% int main()
86
{
yyparse();
}
/* Yacc program to check for valid expression */ %{ #include extern int yyerror(char * s); extern int yylex(); %} %token NUM %token ADDORSUB %token PROORDIV %token IDENTIFIER %% input : | input line ; line : '\n' | exp '\n' { printf("valid"); } | error '\n' { yyerrok; } ; exp : exp ADDORSUB term | term ; term : term PROORDIV factor | factor ; factor : NUM | IDENTIFIER | '(' exp ')' ; %% int yyerror(char *s) { printf("%s","INVALID\n"); }
/* yacc program that gets token from the c porogram */ %{ #include #include %} %token NUMBER LETTER %left '+' '-' %left '*' '/' %% line:line expr '\n' {printf("\nVALID\n");} | line '\n' | |error '\n' { yyerror ("\n INVALID"); yyerrok;} ; expr:expr '+' expr |expr '-' expr |expr '*'expr |expr '/' expr
87
| NUMBER | LETTER ;
%% main() { yyparse(); } yylex() { char c; while((c=getchar())==' '); if(isdigit(c)) return NUMBER; if(isalpha(c)) return LETTER; return c; } yyerror(char *s) { printf("%s",s); }
2. Write a Yacc program to recognize validity of a nested ‘IF’ control statement and display levels of nesting in the nested if. /* Lex program to pass tokens */ %{ #include “y.tab.h” %} digit [0-9] num {digit} + (“.” {digit}+)? binopr [+-/*%^=> <&|”= =”| “!=” | “>=” | “<=” unopr [~!] char [a-zA-Z_] id {char}({digit} | {char})* space [ \t] %% {space} ; {num} return num; { binopr } return binopr; { unopr } return unopr; { id} return id “if” return if . return yytext[0]; %% NUMBER {DIGIT}+
/* Yacc program to check for the valid expression */ %{ #include int cnt; %} %token binopr
88
%token unop %token num %token id %token if %% foo: if_stat { printf(“valid: count = %d\n”, cnt); cnt = 0; exit(0); } | error { printf(“Invalid \n”); } if_stat: token_if ‘(‘ cond ‘)’ comp_stat {cnt++;} cond: expr ; expr: sim_exp | ‘(‘ expr ‘)’ | expr binop factor | unop factor ; factor: sim_exp | ‘(‘ expr ‘)’ ; sim_exp: num | id ; sim_stat: expr ‘;’ | if ; stat_list: sim_stat | stat_list sim_stat ; comp_stat: sim_stat | ‘{‘ stat_list ‘}’ ; %% main() { yyparse(); } yyerror(char *s) { printf(“%s\n”, s); exit(0); }
3. Write a Yacc program to recognize a valid arithmetic expression that uses +, - , / , *. %{ #include #include %}
89
% token num % left '+' '-' % left '*' '/' %% st : st expn '\n' {printf ("valid \n"); } | | st '\n' | error '\n' { yyerror ("Invalid \n"); } ; %% void main() { yyparse (); return 0 ; } yylex() { char c; while (c = getch () ) == ' ') if (is digit (c)) return num; return c; } yyerror (char *s) { printf("%s", s); } 4. Write a yacc program to recognize an valid variable which starts with letter followed by a digit. The letter should be in lowercase only.
/*
Lex program to send tokens to the yacc program
*/
%{ #include "y.tab.h" %} %% [0-9] return digit; [a-z] return letter; [\n] return yytext[0]; . return 0; %% /*
Yacc program to validate the given variable
*/
%{ #include 90
%} % token digit letter; %% ident : expn '\n' { printf ("valid\n"); exit (0); } ; expn : letter | expn letter | expn digit | error { yyerror ("invalid \n"); exit (0); } ; %% main() { yyparse(); } yyerror (char *s) { printf("%s", s); } /*
Yacc program which has c program to pass tokens
*/
%{ #include #include %} %token LETTER DIGIT %% st:st LETTER DIGIT '\n' {printf("\nVALID");} | st '\n' | | error '\n' {yyerror("\nINVALID");yyerrok;} ; %% main() { yyparse(); } yylex() { char c; while((c=getchar())==' '); if(islower(c)) return LETTER; if(isdigit(c)) return DIGIT; return c; } yyerror(char *s) { printf("%s",s); }
91
5.Write a yacc program to evaluate an expression (simple calculator program).
/* %{
Lex program to send tokens to the Yacc program
*/
#include" y.tab.h" expern int yylval; %} %% [0-9] digit char[_a-zA-Z] id {char} ({ char } | {digit })* %% {digit}+ {yylval = atoi (yytext); return num; } {id} return name [ \t] ; \n return 0; . return yytext [0]; %% /* %{
Yacc Program to work as a calculator
*/
#include #include #include %} % token num name % left '+' '-' % left '*' '/' % left unaryminus %% st
: name '=' expn | expn { printf ("%d\n" $1); } ; expn : num { $$ = $1 ; } | expn '+' num { $$ = $1 + $3; } | expn '-' num { $$ = $1 - $3; } | expn '*' num { $$ = $1 * $3; } | expn '/' num { if (num == 0) { printf ("div by zero \n"); exit (0); }
92
else { $$ = $1 / $3; } | '(' expn ')' { $$ = $2; } ; %% main() { yyparse(); } yyerror (char *s) { printf("%s", s); } 5. Write a yacc program to recognize the grammar { anb for n >= 0}. /* %{
Lex program to pass tokens to yacc program
*/
#include "y.tab.h" %} [a] { return a ; printf("returning A to yacc \n"); } [b] return b [\n] return yytex[0]; . return error; %% /*
Yacc program to check the given expression
*/
%{ #include %} % token a b error %% input : line | error ; line : expn '\n' { printf(" valid new line char \n"); } ; expn : aa expn bb | aa ; aa : aa a |a ; bb : bb b |b ; 93
error : error { yyerror ( " " ) ; } %% main() { yyparse(); } yyerror (char *s) { printf("%s", s); } /* Yacc to evaluate the expression and has c program for tokens */ %{ /* 6b.y
{A^NB
N >=0}
*/
#include %} %token A B %% st:st reca endb '\n' {printf("String belongs to grammar\n");} | st endb '\n' {printf("String belongs to grammar\n");} | st '\n' | error '\n' {yyerror ("\nDoes not belong to grammar\n");yyerrok;} | ; reca: reca enda | enda; enda:A; endb:B; %% main() { yyparse(); } yylex() { char c; while((c=getchar())==' '); if(c=='a') return A; if(c=='b') return B; return c; } yyerror(char *s) { fprintf(stdout,"%s",s); }
7. Write a program to recognize the grammar { anbn | n >= 0 }
94
/*
Lex program to send tokens to yacc program
*/
%{ #include "y.tab.h" %} [a] {return A ; printf("returning A to yacc \n"); } [b] return B [\n] return yytex[0]; . return error; %% /*
yacc program that evaluates the expression */
%{ #include %} % token a b error %% input : line | error ; line : expn '\n' { printf(" valid new line char \n"); } ; expn : aa expn bb | ; error : error { yyerror ( " " ) ; } %% main() { yyparse(); } yyerror (char *s) { printf("%s", s); } /*
Yacc program which has its own c program to send tokens */
%{ /* 7b.y
{A^NB^N
N >=0}
*/
#include %} %token A B %%
95
st:st reca endb '\n' | st '\n' | | error '\n'
{printf("String belongs to grammar\n");} {printf("N value is 0,belongs to grammar\n");}
{yyerror ("\nDoes not belong to grammar\n");yyerrok;} ; reca: enda reca endb | enda; enda:A; endb:B; %% main() { yyparse(); } yylex() { char c; while((c=getchar())==' '); if(c=='a') return A; if(c=='b') return B; return c; } yyerror(char *s) { fprintf(stdout,"%s",s); } 8. Write a Yacc program t identify a valid IF statement or IF-THEN-ELSE statement.
/*
Lex program to send tokens to yacc program
*/
%{ #include "y.tab.h" %} CHAR [a-zA-Z0-9] %x CONDSTART %% <*>[ ] ; <*>[ \t\n]+ ; <*><> return 0; if return(IF); else return(ELSE); then return(THEN); \( {BEGIN(CONDSTART);return('(');} {CHAR}+ return COND; \) {BEGIN(INITIAL);return(')');} {CHAR}+ return(STAT) ; %% /*
Yacc program to check for If and IF Then Else statement
*/
%{
96
#include %} %token IF COND THEN STAT ELSE %% Stat:IF '(' COND ')' THEN STAT {printf("\n VALId Statement");} | IF '(' COND ')' THEN STAT ELSE STAT {printf("\n VALID Statement");} | ; %% main() { printf("\n enter statement "); yyparse(); } yyerror (char *s) { printf("%s",s); } /*
Yacc program that has c program to send tokens
*/
%{ #include #include %} %token if simple % noassoc reduce % noassoc else %% start : start st ‘\n’ | ; st : simple | if_st ; if_st : if st %prec reduce { printf (“simple\n”); } | if st else st {printf (“if_else \n”); } ; %% int yylex() { int c; c = getchar(); switch ( c ) { case ‘i’ : return if; case ‘s’ : return simple; case ‘e’ : return else; default : return c; } } main ()
97
{ yy parse(); } yyerror (char *s)
{ printf("%s", s); } References
1. B. W. Kernighan and D. M. Ritchie, The C Programming Language, Prentice-Hall, Englewood Cliffs, New Jersey, 1978. 2. A. V. Aho and S. C. Johnson, "LR Parsing," Comp. Surveys, vol. 6, no. 2, pp. 99-124, June 1974. 3. A. V. Aho, S. C. Johnson, and J. D. Ullman, "Deterministic Parsing of Ambiguous Grammars," Comm. Assoc. Comp. Mach., vol. 18, no. 8, pp. 441452, August 1975. 4. A. V. Aho and J. D. Ullman, Principles of Compiler Design, AddisonWesley, Reading, Mass., 1977. 5. S. C. Johnson, "Lint, a C Program Checker," Comp. Sci. Tech. Rep. No. 65, 1978 .]. updated version TM 78-1273-3
98
