Good library for search text tokenization
- by Chris Dutrow
Looking to tokenize some text in the same or similar way in which a search engine would do it.
The reason we are doing this is so that we can run some statistical analysis on the tokens. The language we are using is python, so would prefer a library in that language, but could probably set something up to use another language if necessary.
…