ANTLR lexer mismatches tokens

Posted by Barry Brown on Stack Overflow See other posts from Stack Overflow or by Barry Brown
Published on 2010-04-09T06:18:24Z Indexed on 2010/04/09 6:23 UTC
Read the original article Hit count: 521

Filed under:

antlr

|

lexer

|

compiler

|

parser

|

tokenizing

I have a simple ANTLR grammar, which I have stripped down to its bare essentials to demonstrate this problem I'm having. I am using ANTLRworks 1.3.1.

grammar sample;

assignment  :  IDENT ':=' NUM ';' ;

IDENT       :  ('a'..'z')+ ;

NUM         :  ('0'..'9')+ ;

WS          :  (' '|'\n'|'\t'|'\r')+ {$channel=HIDDEN;} ;

Obviously, this statement is accepted by the grammar:

x := 99;

But this one also is:

x := @!$()()%99***;

Output from the ANTLRworks Interpreter:

ANTLR Interpreter diagram

What am I doing wrong? Even other sample grammars that come with ANTLR (such as the CMinus grammar) exhibit this behavior.

© Stack Overflow or respective owner

Related posts about antlr

Antlr question: cannot get Antlr tool to compile simple file from ANTLRWorks

as seen on Stack Overflow - Search for 'Stack Overflow'
Here is the grammar file: grammar fred; test : 'fred'; Here is the batch file to launch the tool: SET JAVA_HOME=C:\Program Files\Java\jdk1.6.0_24 SET PATH=%PATH%;%JAVA_HOME%\bin SET ANTLR_HOME=c:/users/don/workspace/antlrAssign/lib/ java -cp %ANTLR_HOME%/antlr-3.3-complete.jar antlr.Tool… >>> More
Improving ANTLR DSL parse-error messages

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm working on a domain-specific language (DSL) for non-programmers. Non-programmers make a lot of grammar mistakes: they misspell keywords, they don't close parentheses, they don't terminate blocks, etc. I'm using ANTLR to generate my parser; it provides a nifty mechanism for handling RecognitionExceptions… >>> More
ANTLR - accessing token values in c/c++

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello, I am trying to parse integers and to access their value in antlr 3.2. I already found out how to do this in Java: //token definition INT : '0'..'9'+; //rule to access token value: start : val=INT {Integer x = Integer.valueOf( $val.text ).intValue(); } ; ... but I couldn't… >>> More
Removing Left Recursion in ANTLR

as seen on Stack Overflow - Search for 'Stack Overflow'
As is explained in http://stackoverflow.com/questions/2652060/removing-left-recursion , there are two ways to remove the left recursion. Modify the original grammar to remove the left recursion using some procedure Write the grammar originally not to have the left recursion What people normally… >>> More
Jar not found when executing class

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi there, I'm working through the ANTLR book and there are many examples that should be easy to compile using the command line. Some information to get te problem: antlr-3.2.jar contains the ANTLR classes. I added the antlr-3.2.jar to the CLASSPATH environment variable (Windows 7) and when compiling… >>> More

Related posts about lexer

Error in running script [closed]

as seen on Programmers - Search for 'Programmers'
I'm trying to run heathusf_v1.1.0.tar.gz found here I installed tcsh to make build_heathusf work. But, when I run ./build_heathusf, I get the following (I'm running that on a Fedora Linux system from Terminal): $ ./build_heathusf Compiling programs to build a library of image processing functions… >>> More
problem string recursion antlr lexer token

as seen on Stack Overflow - Search for 'Stack Overflow'
How do I build a token in lexer that can handle recursion inside as this string: ${*anythink*${*anything*}*anythink*} ? thanks >>> More
Lexer antlr3 token problem

as seen on Stack Overflow - Search for 'Stack Overflow'
Can I construct a token ENDPLUS: '+' (options (greedy = false;):.) * '+' ; being considered by the lexer only if it is preceded by a token PREwithout including in ENDPLUS? PRE: '<<' ; Thanks. >>> More
grammar parser lexer antlr letteral

as seen on Stack Overflow - Search for 'Stack Overflow'
What's the difference between this grammar: ... if_statement : 'if' condition 'then' statement 'else' statement 'end_if'; ... and this: ... if_statement : IF condition THEN statement ELSE statement END_IF; ... IF : 'if'; THEN: 'then'; ELSE: 'else'; END_IF: 'end_if'; .... ? If there is any… >>> More
ANTLR lexer mismatches tokens

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a simple ANTLR grammar, which I have stripped down to its bare essentials to demonstrate this problem I'm having. I am using ANTLRworks 1.3.1. grammar sample; assignment : IDENT ':=' NUM ';' ; IDENT : ('a'..'z')+ ; NUM : ('0'..'9')+ ; WS : (' '|'\n'|'\t'|'\r')+… >>> More