Antlr Lexer Quoted String Problem
Posted
by Loki
on Stack Overflow
See other posts from Stack Overflow
or by Loki
Published on 2010-05-23T05:27:58Z
Indexed on
2010/05/23
5:30 UTC
Read the original article
Hit count: 577
I'm trying to build a lexer to tokenize lone words and quoted strings. I got the following:
STRING: QUOTE (options {greedy=false;} : . )* QUOTE ;
WS : SPACE+ { $channel = HIDDEN; } ;
WORD : ~(QUOTE|SPACE)+ ;
For the corner cases, it needs to parse:
"string" word1" word2
As three tokens: "string" as STRING and word1" and word2 as WORD. Basically, if there is a last quote, it needs to be part of the WORD were it is. If the quote is surrounded by white spaces, it should be a WORD.
I tried this rule for WORD, without success:
WORD: ~(QUOTE|SPACE)+
| (~(QUOTE|SPACE)* QUOTE ~QUOTE*)=> ~(QUOTE|SPACE)* QUOTE ~(QUOTE|SPACE)* ;
© Stack Overflow or respective owner