Merging and sorting multiple files with "sort"
Posted
by NewbiZ
on Super User
See other posts from Super User
or by NewbiZ
Published on 2010-06-03T13:52:26Z
Indexed on
2010/06/03
13:56 UTC
Read the original article
Hit count: 265
Hello,
I have a bunch of text logfiles in the following format:
ID (17 characters)
Timestamp (14 characters YYYYmmddHHMMSS e.g. "20060210100040" -> 2006/02/10 10:00:40)
Random data (? characters)
end of line
The files are already sorted by timestamp. I need to get 1 log file with all the logs from multiple logs files, sorted by timestamp. Note that the log files are really huge, around 3-4G each (and there are dozens of them) I tried the following command:
sort -s -m -t '|' -k1n,1n +17 -o data_sort.txt *.TXT
Here is how I ended up with this command:
-s : don't bother with tie results
-m : merge all logs files
-t '|' : there is no | in my logs, so the whole line should be field 1
-k1n,1n: sort on the first field as a numeric value
+17 : the timestamp starts at index 17
-o : output file
Actually... it fails miserably. The output file data_sort.txt is just the concatenation of all files, not sorted at all :(
I would greatly appreciate if anyone could provide any help on this problem!
Thanks
© Super User or respective owner