xtc.parser

Class Tokenizer

java.lang.Object

xtc.tree.Visitor

xtc.parser.GrammarVisitor

xtc.parser.Tokenizer

public class Tokenizer
extends GrammarVisitor

Visitor to recognize token-level productions.

This visitor recognizes the boundary between hierarchical and lexical syntax:

This visitor recognizes all lexical productions. A production is lexical if it is text-only or if it is void and only references other lexical productions (if any). As a result, a lexical production may not contain parser actions, semantic actions that reference CodeGenerator.VALUE, or bindings to CodeGenerator.VALUE. This visitor traverses the grammar, starting with its public productions. When encountering a lexical production it does not traverse into the production; rather, if the lexical production also consumes the input, this visitor marks the production as token-level. This visitor ensures that all void productions are correctly annotated with the Properties.CONSUMER property, indicating whether they consume the input.

This visitor does not change a token-level production's type, so that later parser generator phases can still distinguish between productions that used to be text-only and that used to be void. It does however, remove any Properties.TEXT_ONLY property.

This visitor assumes that the entire grammar is contained in a single module and that text-only productions have been marked as such. It may perform faster if the grammar has been annotated with its real root.

Version:

$Revision: 1.12 $

Author:

Robert Grimm

See Also:

TextTester, RootFinder