Alternative databases to use when putting IIS Logs into a database using LogParser

Posted by Robin Day on Stack Overflow See other posts from Stack Overflow or by Robin Day
Published on 2010-06-18T14:43:59Z Indexed on 2010/06/18 14:53 UTC
Read the original article Hit count: 297

Filed under:

database

|

iis

|

simpledb

|

bigtable

|

logfiles

We have run some scripts that use LogParser to dump our IIS logs into a SQL Server database.

We can then query this to get simple stats on hits, usage etc. It's also good when linking it to error log databases and performance counter database to compare usage with errors, etc.

Having implemented this for just one system and for the last 2-3 weeks we already have a 5GB database with around 10 million records.

This is making any queries to this database quite slow and will no doubt cause storage issues if we continue to log as we are.

Can anyone suggest any alternative databases that we could use for this data that would be more efficient for such logs? I'd be particularly interested in any experience of Google's BigTable or Amazon's SimbleDB.

Are either of these suitable for reporting queries? COUNTs, GROUP BYs, PIVOTs?

© Stack Overflow or respective owner

Related posts about database

SQL SERVER Retrieve and Explore Database Backup without Restoring Database Idera virtual database

as seen on Dot net Slackers - Search for 'Dot net Slackers'
I recently downloaded Ideras SQL virtual database, and tested it. There are a few things about this tool which caught my attention.My ScenarioIt is quite common in real life that sometimes observing or retrieving older data is necessary; however, it had changed as time passed by. The full database… >>> More
Cloning A Database On The Same Server Using Rman Duplicate From Active Database

as seen on Oracle Blogs - Search for 'Oracle Blogs'
To clone a database using Rman we used to require an existing Rman backup, on 11g we can clone databases using the "from active" database option. In this case we do not require an existing backup, the active datafiles will be used as the source for the clone. In order to clone with the source database… >>> More
cPickle ImportError: No module named multiarray

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello, I'm using cPickle to save my Database into file. The code looks like that: def Save_DataBase(): import cPickle from scipy import * from numpy import * a=Results.VersionName #filename='D:/results/'+a[a.find('/')+1:-a.find('/')-2]+Results.AssType[:3]+str(random.randint(0,100))+Results.Distribution+"… >>> More
SQL SERVER – 2008 – Introduction to Snapshot Database – Restore From Snapshot

as seen on SQL Authority - Search for 'SQL Authority'
Snapshot database is one of the most interesting concepts that I have used at some places recently. Here is a quick definition of the subject from Book On Line: A Database Snapshot is a read-only, static view of a database (the source database). Multiple snapshots can exist on a source database and… >>> More
OTN ???? ?????? ???????

as seen on Oracle Blogs - Search for 'Oracle Blogs'
Database ?? Database ??????? Database ?????????? Java WebLogic Server/????????·???? SOA/BPM/????? ???????/???? ID??/?????? ?????EPM/BI EPM/BI ??????? EPM/BI ???? OS/??? ???? ????? MySQL Database ?? ???? ?? ????????? ??? ?? ORACLE MASTER… >>> More

Related posts about iis

How to find and fix issue with Pound and HAProxy

as seen on Server Fault - Search for 'Server Fault'
Pound sits in front of HAProxy (on the same box) to perform SSL off-load. Requests are passed to 127.0.0.1:80 where HAProxy then balances the requests across backend servers for a hosted ASP .NET web app. A user is getting HTTP error 500 (Internal Server Error) returned to their browser this morning… >>> More
How to migrate deploy RESTful WCF Web Service from IIS 6.0 to IIS 7.0

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi all, I have a WCF Restful Web Service. It works find under VS and IIS 6.0. Now I want to move it to another work station with IIS 7.0 on it. I tried to copy all the deploy file from IIS 6 to IIS 7, but it cannot be accessed by other client, except for the request from it's own machine. I don't… >>> More
Server with IIS and Apache - how to SSL encrypt Apache with IIS

as seen on Server Fault - Search for 'Server Fault'
I have a Windows Server 2003 box already setup and working with IIS 6. IIS is set to serve a site out over both HTTP and HTTPS connections using default ports. For various reasons I need to set Apache up on the same server and it needs to serve its pages to end-users as SSL encrypted HTTPS pages… >>> More
Having IIS remote management problem with my vista machine managing Server 2008 IIS 7.5

as seen on Server Fault - Search for 'Server Fault'
I am trying to use IIS 7 Remote Management installed on Vista Ultimate SP1. Connection is to IIS 7.5 on Windows Server 2008 Webserver R2. Tried on both full & core install. When I connect up, the console wants to download and install new features. Microsoft.Web.Management.IisClient 7.5.0.0 Microsoft… >>> More
MVC4 App opens with directory listing and gives a 404 for any direct URL's entered in the browser

as seen on Pro Webmasters - Search for 'Pro Webmasters'
I've just deployed a previously (on my local IIS) working MVC4 app to IIS 7.5 on the dev server. After tweaking this and that - one knows how these things get forgotten - the app finally launches, but shows a directory listing of the app root. Clicking on most links there works, opening the directory… >>> More