How can I remove old log entries from a log file and archive them somewhere else in Linux?
Posted
by
Mike B
on Server Fault
See other posts from Server Fault
or by Mike B
Published on 2012-12-07T21:51:12Z
Indexed on
2012/12/07
23:10 UTC
Read the original article
Hit count: 239
CentOS 4.x
I apologize in advance if this is not the appropriate place to ask this question. It pertains to a linux server / IT admin task.
I've got a log file on an old CentOS 4.x server and I want to remove log entries older than a certain date and place them in a new file for archive.
Here's an example of the log format:
2012-06-07 22:32:01,289 ABC:0|Foo|Foo2|4.4|1234|Some Event|123|blah blah blah
2012-06-07 22:32:03,289 ABC:0|Foo|Foo2|4.4|1234|Some Event|123|blah blah blah
2012-06-07 22:32:04,289 ABC:0|Foo|Foo2|4.4|1234|Some Event|123|
2012-06-07 22:32:10,289 ABC:0|Foo|Foo2|4.4|1234|Some Event|123|blah blah blah
2012-06-07 22:32:12,289 ABC:0|Foo|Foo2|4.4|1234|Some Event|123|blah blah blah
2012-06-07 22:32:15,289 ABC:0|Foo|Foo2|4.4|1234|Some Event|123|
2012-06-07 22:32:40,289 ABC:0|Foo|Foo2|4.4|1234|Some Event|123|blah blah blah
2012-06-07 22:32:58,289 ABC:0|Foo|Foo2|4.4|1234|Some Event|123|blah blah blah
2012-06-07 22:33:01,289 ABC:0|Foo|Foo2|4.4|1234|Some Event|123|
2012-06-07 22:33:01,289 ABC:0|Foo|Foo2|4.4|1234|Some Event|123|blah blah blah
2012-06-07 22:33:02,289 ABC:0|Foo|Foo2|4.4|1234|Some Event|123|
Essentially, I'm looking for a one-liner that will do the following:
- Find any events older than a provided YYYY-MM-DD and remove them from the primary log file.
- Take the deleted events from step 1 and put them in a new log file
- (Optional) Compress the new archive log file holding the deleted events.
I'm aware that there are log rotate tools that do this but this should just be a one-time task so I'd prefer not to set that up.
Additional notes:
- If the date part it tricky or too resource intensive, an alternative would be to just keep the last X number of lines and move the rest. I was originally thinking of something like
tail -n 10000 > newfile.txt
but that would mean moving the "good" logs to a new file and then doing a name swap... and then I'd still need to remove the "good" entries from the archive. - This particular log file is pretty large (1 GB) so I'd prefer the task to be as resource and time efficient as possible.
- The extra pipes in the log concern me and I'm not sure if I'd need extra protection in the commands to avoid that from causing problems.
© Server Fault or respective owner