How can I parse Amazon S3 log files?

Posted by artlung on Pro Webmasters See other posts from Pro Webmasters or by artlung
Published on 2010-07-09T00:45:15Z Indexed on 2011/11/29 18:04 UTC
Read the original article Hit count: 464

Filed under:

analytics

|

amazon-s3

|

logging

What are the best options for parsing Amazon S3 (Simple Storage) log files? I've turned on logging and now I have log files that look like this:

858e709ba90996df37d6f5152650086acb6db14a67d9aaae7a0f3620fdefb88f files.example.com [08/Jul/2010:10:31:42 +0000] 68.114.21.105 65a011a29cdf8ec533ec3d1ccaae921c 13880FBC9839395C REST.GET.OBJECT example.com/blog/wp-content/uploads/2006/10/kitties_we_cant_stop_here_this_is_bat_country.jpg "GET /example.com/blog/wp-content/uploads/2006/10/kitties_we_cant_stop_here_this_is_bat_country.jpg HTTP/1.1" 200 - 32957 32957 12 10 "http://atlanta.craigslist.org/forums/?act=Q&ID=163218891" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.19) Gecko/2010031422 Firefox/3.0.19" -

What are the best options for automating the log files? I'm not using any other Amazon services other than S3.

© Pro Webmasters or respective owner

Related posts about analytics

Google Analytics www 301 causing issues with In-Page Analytics

as seen on Pro Webmasters - Search for 'Pro Webmasters'
The closest question I could find to my problem is This one. The similarity is: I have a profile in Google Analytics (GA) that has been collecting data for a year. The domain setting in GA is "http://example.com". The site, however, will redirect any non-www request, to www.example.com, via a… >>> More
Real-time Big Data Analytics is a reality for StubHub with Oracle Advanced Analytics

as seen on Oracle Blogs - Search for 'Oracle Blogs'
What can you use for a comprehensive platform for real-time analytics? How can you process big data volumes for near-real-time recommendations and dramatically reduce fraud? Learn in this video what Stubhub achieved with Oracle R Enterprise from the Oracle Advanced Analytics option to Oracle Database… >>> More
Documentation in Oracle Retail Analytics, Release 13.3

as seen on Oracle Blogs - Search for 'Oracle Blogs'
The 13.3 Release of Oracle Retail Analytics is now available on the Oracle Software Delivery Cloud and from My Oracle Support. The Oracle Retail Analytics 13.3 release introduced significant new functionality with its new Customer Analytics module. The Customer Analytics module enables… >>> More
Google Analytics - Can I download the logs?

as seen on Server Fault - Search for 'Server Fault'
Can I download the raw logs of my Google Analytics data? I want to do some processing on the data. >>> More
Custom Tracking with Google Analytics

as seen on Stack Overflow - Search for 'Stack Overflow'
I am trying to figure out how to use my google analytics account, and do custom tracking on certain links and such, but following the technical information on the help site on google isn't getting me anywhere. Has anyone done something like this? Point me in the right direction. >>> More

Related posts about amazon-s3

Amazon S3 Tips: Quickly Add/Modify HTTP Headers To All Files Recursively

as seen on Tech Dreams - Search for 'Tech Dreams'
Amazon S3 is an dead cheap cloud storage service that offers unlimited storage in pay as you use model. Recently we moved all the images and other static files(scripts & css) of Tech Dreams to Amazon S3 to reduce load on VPS server. Amazon S3 is cheap, but monthly bill will shoot up if images/static… >>> More
Alternative to Amazon's S3 service?

as seen on Super User - Search for 'Super User'
Just wondering if there is good alternative to Amazon's S3 service? I like S3 but the bandwidth cost is high. I looked at CouldFiles from Rackspace but the cost is even higher. I don't mind prepaying or having monthly payment in order to reduce the bandwidth cost greatly. Thank you for any help >>> More
Alternative to Amazon’s S3 service?

as seen on Stack Overflow - Search for 'Stack Overflow'
Just wondering if there is good alternative to Amazon's S3 service? I like S3 but the bandwidth cost is high. I looked at CouldFiles from Rackspace but the cost is even higher. I don't mind prepaying or having monthly payment in order to reduce the bandwidth cost greatly. Thank you for any help >>> More
Saving Python Complex Data Types to Amazon S3

as seen on Stack Overflow - Search for 'Stack Overflow'
Can Python class data be saved to S3 without marshalling? I am trying to cut down of I/O operations until necessary. >>> More
Filesize with SWFUpload and Amazon S3

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello all, I'm currently using SWFUpload to upload files to my S3 bucket. And it's working great. I'm using the script from a website here: http://www.anedix.com/news/article/50 Again, the upload to my S3 works fine, however, I've been running into an issue when attempting to upload larger… >>> More