Merging and sorting multiple files with "sort"

Posted by NewbiZ on Super User See other posts from Super User or by NewbiZ
Published on 2010-06-03T13:52:26Z Indexed on 2010/06/03 13:56 UTC
Read the original article Hit count: 265

Filed under:
|

Hello,

I have a bunch of text logfiles in the following format:

ID          (17 characters)
Timestamp   (14 characters YYYYmmddHHMMSS e.g. "20060210100040" -> 2006/02/10 10:00:40)
Random data (? characters)
end of line

The files are already sorted by timestamp. I need to get 1 log file with all the logs from multiple logs files, sorted by timestamp. Note that the log files are really huge, around 3-4G each (and there are dozens of them) I tried the following command:

sort -s -m -t '|' -k1n,1n +17 -o data_sort.txt *.TXT

Here is how I ended up with this command:

-s     : don't bother with tie results
-m     : merge all logs files
-t '|' : there is no | in my logs, so the whole line should be field 1
-k1n,1n: sort on the first field as a numeric value
+17    : the timestamp starts at index 17
-o     : output file

Actually... it fails miserably. The output file data_sort.txt is just the concatenation of all files, not sorted at all :(

I would greatly appreciate if anyone could provide any help on this problem!

Thanks

© Super User or respective owner

Related posts about linux

Related posts about sort