Alternative databases to use when putting IIS Logs into a database using LogParser
- by Robin Day
We have run some scripts that use LogParser to dump our IIS logs into a SQL Server database.
We can then query this to get simple stats on hits, usage etc. It's also good when linking it to error log databases and performance counter database to compare usage with errors, etc.
Having implemented this for just one system and for the last 2-3 weeks we already have a 5GB database with around 10 million records.
This is making any queries to this database quite slow and will no doubt cause storage issues if we continue to log as we are.
Can anyone suggest any alternative databases that we could use for this data that would be more efficient for such logs? I'd be particularly interested in any experience of Google's BigTable or Amazon's SimbleDB.
Are either of these suitable for reporting queries? COUNTs, GROUP BYs, PIVOTs?