How do I lex this input?

Posted by etheros on Stack Overflow See other posts from Stack Overflow or by etheros
Published on 2010-05-09T17:46:54Z Indexed on 2010/05/09 17:48 UTC
Read the original article Hit count: 227

Filed under:

antlr

|

lexical-analysis

I currently have a working, simple language implemented in Java using ANTLR. What I want to do is embed it in plain text, in a similar fashion to PHP.

For example:

Lorem ipsum dolor sit amet
<% print('consectetur adipiscing elit'); %>
Phasellus volutpat dignissim sapien.

I anticipate that the resulting token stream would look something like:

CDATA OPEN PRINT OPAREN APOS STRINGAPOS CPARENT SEMI CLOSE CDATA

How can I achieve this, or is there a better way?

© Stack Overflow or respective owner

Related posts about antlr

Antlr question: cannot get Antlr tool to compile simple file from ANTLRWorks

as seen on Stack Overflow - Search for 'Stack Overflow'
Here is the grammar file: grammar fred; test : 'fred'; Here is the batch file to launch the tool: SET JAVA_HOME=C:\Program Files\Java\jdk1.6.0_24 SET PATH=%PATH%;%JAVA_HOME%\bin SET ANTLR_HOME=c:/users/don/workspace/antlrAssign/lib/ java -cp %ANTLR_HOME%/antlr-3.3-complete.jar antlr.Tool… >>> More
Improving ANTLR DSL parse-error messages

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm working on a domain-specific language (DSL) for non-programmers. Non-programmers make a lot of grammar mistakes: they misspell keywords, they don't close parentheses, they don't terminate blocks, etc. I'm using ANTLR to generate my parser; it provides a nifty mechanism for handling RecognitionExceptions… >>> More
ANTLR - accessing token values in c/c++

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello, I am trying to parse integers and to access their value in antlr 3.2. I already found out how to do this in Java: //token definition INT : '0'..'9'+; //rule to access token value: start : val=INT {Integer x = Integer.valueOf( $val.text ).intValue(); } ; ... but I couldn't… >>> More
Removing Left Recursion in ANTLR

as seen on Stack Overflow - Search for 'Stack Overflow'
As is explained in http://stackoverflow.com/questions/2652060/removing-left-recursion , there are two ways to remove the left recursion. Modify the original grammar to remove the left recursion using some procedure Write the grammar originally not to have the left recursion What people normally… >>> More
Jar not found when executing class

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi there, I'm working through the ANTLR book and there are many examples that should be easy to compile using the command line. Some information to get te problem: antlr-3.2.jar contains the ANTLR classes. I added the antlr-3.2.jar to the CLASSPATH environment variable (Windows 7) and when compiling… >>> More

Related posts about lexical-analysis

What follows after lexical analysis?

as seen on Programmers - Search for 'Programmers'
I'm working on a toy compiler (for some simple language like PL/0) and I have my lexer up and running. At this point I should start working on building the parse tree, but before I start I was wondering: How much information can one gather from just the string of tokens? Here's what I gathered so… >>> More
Fuzzy Regex, Text Processing, Lexical Analysis?

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm not quite sure what terminology to search for, so my title is funky... Here is the workflow I've got: Semi-structured documents are scanned to file. The files are OCR'd to text. The text is parsed into Python objects The objects are serialized (to SQL, JSON, whatever) for use. The documents… >>> More
lexical analysis gives only one output?

as seen on Stack Overflow - Search for 'Stack Overflow'
I tested this example(lexe.java), but it gave me only one output. I gave this text as a reader: public class LexeTest{ private int a = 14; } And the nextToken() function is : public Category nextToken () { if (inp.findWithinHorizon (tokenPat, 0) == null) return Category… >>> More
What causes a lexical analysis error for eclipse in jsp EL validation?

as seen on Stack Overflow - Search for 'Stack Overflow'
I've got some EL code inside of a JSP tag. The line starts as follows: <c:if test="${pageContext.request.serverName eq \'localhost\'}"> Eclipse throws up an error on this, saying: Unable to analyze EL expression due to lexical analysis error I'm unsure what this even means. Is this… >>> More
Possible typos in ECMAScript 5 specification?

as seen on Stack Overflow - Search for 'Stack Overflow'
Does anybody know why, at the end of section 7.6 of the ECMA-262, 5th Edition specification, the nonterminals UnicodeLetter, UnicodeCombiningMark, UnicodeDigit, UnicodeconnectorPunctuation, and UnicodeEscapeSequence are not followed by two colons? From section 5.1.6: Nonterminal symbols are shown… >>> More