Read huge free text docs in one file for lucene indexing
- by Jun
I have heaps of free text news docs in one big file. The structure of each news doc is like:
(Header line) Category, Doc1, Date (day, month, year)
(body text)
...
...
...
(Header line) Category, Doc2, Date (day, month, year)
(body text)
...
...
...
If I extract each doc from the big file, it costs too much time and not efficient. Therefore, I decide to read the file line by line and feed information to lucene the same time.
I write c# code to index each doc to lucene like:
Streamreader sr = new Streamreader(file);
string line = "";
while((line = sr.ReadLine()) != null)
{
How can I tell this line is a doc header line from text line
and get the metadata and all the text lines of a doc for lucene to index.
Also, the text is read by OCR which can not give correct line-separating.
Captions are mixed with content text
iterate the process till the end of the file
}
with thanks
