Search in big text log files

Posted by 0xFF on Stack Overflow See other posts from Stack Overflow or by 0xFF
Published on 2010-05-19T11:27:22Z Indexed on 2010/05/19 11:30 UTC
Read the original article Hit count: 204

Filed under:
|

Hi,

let's say you have an game server which creating text log files of gamers actions, and from time to time you need to lookup something in those logs files (like investigating an scam or loosing an item). Just for example you have 100 files and each file have size between 20MB and 50MB - How you would search them quickly?

What I have already tried to do is create several threads and each invidual thread will map his own file to memory (let say memory should not be problem if it not exceed 500MB of ram) perform search here, result was something around 1 second per file :

File:a26.log - read in: 0.891, lines: 625282, matches: 78848

Is there better way how to do that ? - because it seems to me kinda slow. thanks.

(java was used for this case)

© Stack Overflow or respective owner

Related posts about java

Related posts about textfiles