Monitor a log file on Linux and send each line to another program

Posted by mlambie on Server Fault See other posts from Server Fault or by mlambie
Published on 2011-03-15T06:27:18Z Indexed on 2011/03/15 8:11 UTC
Read the original article Hit count: 259

Filed under:

ubuntu

|

logging

|

swatch

|

apt-cacher-ng

I run an apt-cacher-ng server on Ubuntu Linux which writes logs in the following format:

1299745593|O|149406|XXX.XXX.XXX.XXX|uburep/pool/main/t/tiff/libtiff4_3.9.2-2ubuntu0.4_amd64.deb
1299745593|O|10154976|XXX.XXX.XXX.XXX|uburep/pool/main/l/linux-firmware/linux-firmware_1.34.4_all.deb     
1299748529|O|39368|XXX.XXX.XXX.XXX|uburep/pool/main/n/nagios-nrpe/nagios-nrpe-server_2.12-4ubuntu1_amd64.deb 
1300155440|O|680100|XXX.XXX.XXX.XXX|uburep/pool/main/t/tzdata/tzdata_2011c-0ubuntu0.10.04_all.deb

It shows the timestamp, direction (in or out), byte count, IP and filename.

Every time a line is written to it, I'd like to also send that line to another program. I will have this program insert the line into a database so that I can crunch some statistics about how much bandwidth we're saving through operating a caching server.

I do not want to cat the log file every X minutes (via cron) looking for new entries as it'd be somewhat computationally uneconomical. Instead I'd prefer to have a daemon monitor the log, and when a change is detected, each line is sent to my database-insertion script.

Will swatch achieve this, or are there better options?

© Server Fault or respective owner

Related posts about ubuntu

Cannot update, apt-get cannot fetch index files

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I have a fresh install of Ubuntu 11.10 from the iso 'ubuntu-11.10-desktop-amd64.iso'. I installed this in VMWare Fusion 4.1.1 running on OSX 10.7.3. When setting up the VM, I allowed easy install to take care of creating my user and installing VMWare tools. No problems during installation, everything… >>> More
'sudo apt-get update' error on Ubuntu 12.04

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
When I type in 'sudo apt-get update' I get this mohd-arafat-hossain@TUD:~$ sudo apt-get update [sudo] password for mohd-arafat-hossain: Ign http://bd.archive.ubuntu.com precise InRelease Ign http://bd.archive.ubuntu.com precise-updates InRelease Ign http://bd.archive.ubuntu… >>> More
Opening Skype, Opera, OpenOffice logs me off

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
Whats common among Skype, Opera, OpenOffice in Ubuntu ? Whenever I open these applications I get logged off and shows back me the login screen. This started happening since the 10.10 upgrade. Forgot to mention : Yes, its x64.Each time I open these applications, the UI shows and then crashes. I… >>> More
Update information outdated

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I have a warning triangle that my update information is outdated, the last update was 12 days ago. I use Ubuntu 11.10. A run of sudo apt-get update produces the following output: Ign http://ppa.launchpad.net oneiric InRelease Ign http://de.archive.ubuntu.com oneiric InRelease … >>> More
ubuntu/apt-get update said "Failed to Fetch http:// .... 404 not found"

as seen on Server Fault - Search for 'Server Fault'
Hi all, I'm trying to run apt-get update on ubuntu 9.10 I've configured my proxy server and I can access the internet without any problem: /etc/apt# wget "http://www.google.com" Resolving (...) Proxy request sent, awaiting response... 200 OK Length: 292 [text/html] Saving to: `index.html' 100%[=================================================================================================================================>]… >>> More

Related posts about logging

log file is not getting created using JDK logging with Commons-logging

as seen on Stack Overflow - Search for 'Stack Overflow'
When I run the TestJcLLoggingService class log messages are coming to Console but no log file is created, please help me if you know the answer. two source files are pasted below. TestJcLLoggingService.java package com.amadeus.psp.pasd.logging; import org.apache.commons.logging.Log; import org… >>> More
Python logging before you run logging.basicConfig?

as seen on Stack Overflow - Search for 'Stack Overflow'
It appears that if you invoke logging.info() BEFORE you run logging.basicConfig, the logging.basicConfig call doesn't have any effect. In fact, no logging occurs. Where is this behavior documented? I don't really understand. >>> More
Python: combine logging and wx so that logging stream is redirectet to stdout/stderr frame

as seen on Stack Overflow - Search for 'Stack Overflow'
Here's the thing: I'm trying to combine the logging module with wx.App()'s redirect feature. My intention is to log to a file AND to stderr. But I want stderr/stdout redirected to a separate frame as is the feature of wx.App. My test code: import logging import wx class MyFrame(wx.Frame): … >>> More
Redirect logging output using custom logging handler

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi Guys, I'm using a module in my python app that writes a lot a of messages using the logging module. Initially I was using this in a console application and it was pretty easy to get the logging output to display on the console using a console handler. Now I've developed a GUI version of my app… >>> More
Java Logger API

as seen on Oracle Blogs - Search for 'Oracle Blogs'
This is a more like a tip rather than technical write up and serves as a quick intro for newbies. The logger API helps to diagnose application level or JDK level issues at runtime. There are 7 levels which decide the detailing in logging (SEVERE, WARNING, INFO, CONFIG, FINE, FINER, FINEST)… >>> More