Practical difference between parser rules and lexer rules in ANTLR?

parsing antlr lexical-analysis
Tony the Pony picture Tony the Pony · Nov 28, 2010 · Viewed 8.7k times · Source

I understand the theory behind separating parser rules and lexer rules in theory, but what are the practical differences between these two statements in ANTLR:

my_rule: ... ;

MY_RULE: ... ;

Do they result in different AST trees? Different performance? Potential ambiguities?

Answer

Bart Kiers picture Bart Kiers · Nov 28, 2010

Jen wrote:

... what are the practical differences between these two statements in ANTLR ...

MY_RULE will be used to tokenize your input source. It represents a fundamental building block of your language.

my_rule is called from the parser, it consists of zero or more other parser rules or tokens produced by the lexer.

That's the difference.

Jen wrote:

Do they result in different AST trees? Different performance? ...

The parser builds the AST using tokens produced by the lexer, so the questions make no sense (to me). A lexer merely "feeds" the parser a 1 dimensional stream of tokens.

Related questions

lexers vs parsers

Are lexers and parsers really that different in theory? It seems fashionable to hate regular expressions: coding horror, another blog post. However, popular lexing based tools: pygments, geshi, or prettify, all use regular expressions. They seem to lex anything... When …

Read More
Using ANTLR 3.3?

I'm trying to get started with ANTLR and C# but I'm finding it extraordinarily difficult due to the lack of documentation/tutorials. I've found a couple half-hearted tutorials for older versions, but it seems there have been some major changes …

Read More
ANTLR4: Whitespace handling

I have seen many ANTLR grammars that use whitespace handling like this: WS: [ \n\t\r]+ -> skip; // or WS: [ \n\t\r]+ -> channel(HIDDEN); So the whitespaces are thrown away respectively send to the hidden channel. …

Read More