Real-time log parsing and reporting

Posted by Alienfluid on Server Fault See other posts from Server Fault or by Alienfluid
Published on 2011-08-05T03:51:41Z Indexed on 2012/10/13 21:41 UTC
Read the original article Hit count: 462

Filed under:

nginx

|

logging

|

http

|

mongodb

|

parse

We have a small project we are working on part-time that runs on Nginx/MongoDB on Ubuntu 10.04 LTS Server. We'd like to be able to see reports on things like server load, requests/sec, response time, DB load, DB response time, etc. Is there an open source or free (as in beer) tool that can parse such logs and provide a real-time report?

I looked into Splunk briefly, but I wanted to see if there are any others that are highly recommended.

© Server Fault or respective owner

Related posts about nginx

ModSecurity compile error on nginx

as seen on Server Fault - Search for 'Server Fault'
I'm trying to install ModSecurity on nginx with the following instructions : wget https://github.com/SpiderLabs/ModSecurity/archive/master.zip unzip master cd ModSecurity-master ./autogen.sh ./configure --enable-standalone-module And i got the following error : Checking plataform... Identified… >>> More
Cannot Start Nginx Compiled from Source

as seen on Server Fault - Search for 'Server Fault'
I am trying to compile Nginx from source based on the original compiled Nginx server running on my DigitalOcean server ( Ubuntu-14.04 64x ) but with a few extra modules. I can get everything installed smoothly but I can not get it to start. I am sure the ini is correct because I copied the original… >>> More
Nginx http_mp4_module seam installed but dont work

as seen on Server Fault - Search for 'Server Fault'
I try to use the http_mp4_module on my Ubuntu server but that didnt seem to work at all. When i check nginx -V i get : nginx version: nginx/1.1.19 TLS SNI support enabled configure arguments: --prefix=/etc/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-client-body-temp-path=/var/lib/nginx/body… >>> More
GeoIP and Nginx

as seen on Server Fault - Search for 'Server Fault'
I have a nginx with geoip, but it is not working rightly. The issue is the next: Nginx are getting geodata from $_SERVER['REMOTE_ADDR'] instead of $_SERVER['HTTP_X_HAPROXY_IP'], which have the real client ip. So, the reported geodata belongs to my server ip instead of client ip. Does anybody where… >>> More
Nginx1.6.1: Installing from source not running corectly

as seen on Server Fault - Search for 'Server Fault'
I just installed Nginx 1.6.1 from source but it didn't seem to installed correctly. Nginx is running if I service nginx status but when I do nginx -v it outputs command not found. Regular HTML page shows fine and there is no error in the error logs. I am on AWS Ec2 linux AMI. Here is my /etc/init… >>> More

Related posts about logging

log file is not getting created using JDK logging with Commons-logging

as seen on Stack Overflow - Search for 'Stack Overflow'
When I run the TestJcLLoggingService class log messages are coming to Console but no log file is created, please help me if you know the answer. two source files are pasted below. TestJcLLoggingService.java package com.amadeus.psp.pasd.logging; import org.apache.commons.logging.Log; import org… >>> More
Python logging before you run logging.basicConfig?

as seen on Stack Overflow - Search for 'Stack Overflow'
It appears that if you invoke logging.info() BEFORE you run logging.basicConfig, the logging.basicConfig call doesn't have any effect. In fact, no logging occurs. Where is this behavior documented? I don't really understand. >>> More
Python: combine logging and wx so that logging stream is redirectet to stdout/stderr frame

as seen on Stack Overflow - Search for 'Stack Overflow'
Here's the thing: I'm trying to combine the logging module with wx.App()'s redirect feature. My intention is to log to a file AND to stderr. But I want stderr/stdout redirected to a separate frame as is the feature of wx.App. My test code: import logging import wx class MyFrame(wx.Frame): … >>> More
Redirect logging output using custom logging handler

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi Guys, I'm using a module in my python app that writes a lot a of messages using the logging module. Initially I was using this in a console application and it was pretty easy to get the logging output to display on the console using a console handler. Now I've developed a GUI version of my app… >>> More
Java Logger API

as seen on Oracle Blogs - Search for 'Oracle Blogs'
This is a more like a tip rather than technical write up and serves as a quick intro for newbies. The logger API helps to diagnose application level or JDK level issues at runtime. There are 7 levels which decide the detailing in logging (SEVERE, WARNING, INFO, CONFIG, FINE, FINER, FINEST)… >>> More