xtc.parser
Class Tokenizer

java.lang.Object
  xtc.tree.Visitor
      xtc.parser.GrammarVisitor
          xtc.parser.Tokenizer

public class Tokenizer
extends GrammarVisitor

Visitor to recognize token-level productions.

This visitor recognizes the boundary between hierarchical and lexical syntax:

This visitor recognizes all lexical productions. A production is lexical if it is text-only or if it is void and only references other lexical productions (if any). As a result, a lexical production may not contain parser actions, semantic actions that reference CodeGenerator.VALUE, or bindings to CodeGenerator.VALUE.

This visitor traverses the grammar, starting with its public productions. When encountering a lexical production it does not traverse into the production; rather, if the lexical production also consumes the input, this visitor marks the production as token-level.

This visitor ensures that all void productions are correctly annotated with the Properties.CONSUMER property, indicating whether they consume the input.

This visitor does not change a token-level production's type, so that later parser generator phases can still distinguish between productions that used to be text-only and that used to be void. It does however, remove any Properties.TEXT_ONLY property.

This visitor assumes that the entire grammar is contained in a single module and that text-only productions have been marked as such. It may perform faster if the grammar has been annotated with its real root.

Version:

$Revision: 1.12 $

Author:

Robert Grimm

See Also:

TextTester, RootFinder

Nested Class Summary

static class

Tokenizer.Tester Visitor to determine which productions are lexical.

Field Summary

Fields inherited from class xtc.parser.GrammarVisitor

analyzer, isBound, isLastElement, isPredicate, isRepeatedOnce, isTopLevel, isVoided, needsSequence, runtime, transformInPlace

Constructor Summary

Tokenizer(Runtime runtime, Analyzer analyzer)
Create a new tokenizer.

Method Summary

static void	markToken(Production p, boolean verbose) Mark the specified production as token-level.
java.lang.Object	visit(Module m) Visit the specified grammar.
Element	visit(NonTerminal nt) Visit the specified nonterminal.

Methods inherited from class xtc.parser.GrammarVisitor

visit, visit, visit, visit, visit, visit, visit, visit, visit, visit, visit, visit, visit, visit, visit

Methods inherited from class xtc.tree.Visitor

dispatch, equals, hashCode, iterate, map, mapInPlace, unableToVisit, visit

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Tokenizer

public Tokenizer(Runtime runtime,
                 Analyzer analyzer)

Create a new tokenizer.

Parameters:

runtime - The runtime.

analyzer - The analyzer utility.

Method Detail

visit

public java.lang.Object visit(Module m)

Visit the specified grammar.

Overrides:

visit in class GrammarVisitor

visit

public Element visit(NonTerminal nt)

Visit the specified nonterminal.

markToken

public static void markToken(Production p,
                             boolean verbose)

Mark the specified production as token-level. This method sets the specified production's Properties.TOKEN property and removes any Properties.TEXT_ONLY property. It does, however, not adjust the production's type to Token.

Parameters:

p - The production.