Search in big text log files
Posted
by 0xFF
on Stack Overflow
See other posts from Stack Overflow
or by 0xFF
Published on 2010-05-19T11:27:22Z
Indexed on
2010/05/19
11:30 UTC
Read the original article
Hit count: 204
Hi,
let's say you have an game server which creating text log files of gamers actions, and from time to time you need to lookup something in those logs files (like investigating an scam or loosing an item). Just for example you have 100 files and each file have size between 20MB and 50MB - How you would search them quickly?
What I have already tried to do is create several threads and each invidual thread will map his own file to memory (let say memory should not be problem if it not exceed 500MB of ram) perform search here, result was something around 1 second per file :
File:a26.log - read in: 0.891, lines: 625282, matches: 78848
Is there better way how to do that ? - because it seems to me kinda slow. thanks.
(java was used for this case)
© Stack Overflow or respective owner