Reading in gzipped data from S3 in Ruby

Posted by Evan Zamir on Stack Overflow See other posts from Stack Overflow or by Evan Zamir
Published on 2013-10-31T21:50:41Z Indexed on 2013/10/31 21:53 UTC
Read the original article Hit count: 238

Filed under:

ruby

|

amazon-web-services

|

amazon-s3

My company has data messages (json) stored in gzipped files on Amazon S3. I want to use Ruby to iterate through the files and do some analytics. I started to use the 'aws/s3' gem, and get get each file as an object:

#<AWS::S3::S3Object:0x4xxx4760 '/my.company.archive/data/msg/20131030093336.json.gz'>

But once I have this object, I do not know how to unzip it or even access the data inside of it.

© Stack Overflow or respective owner

Related posts about ruby

Setting up Rails to work with sqlserver

as seen on Stack Overflow - Search for 'Stack Overflow'
Ok I followed the steps for setting up ruby and rails on my Vista machine and I am having a problem connecting to the database. Contents of database.yml development: adapter: sqlserver database: APPS_SETUP Host: WindowsVT06\SQLEXPRESS Username: se Password: paswd Run rake db:migrate… >>> More
marshal data too short!!!

as seen on Stack Overflow - Search for 'Stack Overflow'
My application requires to keep large data objects in session. There are like 3-4 data objects each created by parsing a csv containing 150 X 20 cells having strings of 3-4 characters. My application shows this error- "marshal data too short". I tried this- Deleting the old session table. Deleting… >>> More
Sinatra and XML POST request

as seen on Stack Overflow - Search for 'Stack Overflow'
I don't know is it my mistake or no. So i have that code: <code> post '/singin/get_token' do content_type :xml puts request.body.read puts xmlRequest xmlRequest = REXML::Document.new(request.body.read) ... </code> And when i post something like that: <code> <?xml… >>> More
how to change ruby path from /usr/bin/ruby to /usr/local/bin/ruby

as seen on Stack Overflow - Search for 'Stack Overflow'
reading around the various ruby install tutorials it's required to change path from /usr/bin/ruby to /usr/local/bin/ruby but i cant seem to be able to do it. Ultimately i want to install Ruby 1.9.2, should i uninstall 1.8.7 or what? i tried to install Ruby 1.9.2 with macports, the installation seemed… >>> More
strange bundler error: tar_input.rb:49:in `initialize': not in gzip format (Zlib::GzipFile::Error) o

as seen on Stack Overflow - Search for 'Stack Overflow'
i am getting a strange bundler error when running bundle pack with bundler 0.9.12 any ideas? (see pastie for a better formatted code: http://pastie.org/881328 ) /opt/ruby-enterprise-1.8.7-2010.01/lib/ruby/site_ruby/1.8/rubygems/package/tar_input.rb:49:in `initialize': not in gzip format (Zlib::GzipFile::Error) … >>> More

Related posts about amazon-web-services

amazon web services and sql server support

as seen on Server Fault - Search for 'Server Fault'
Hi All, I have built my application using sql server 2008 and .net framework 3.5 I am looking for a sclable hosting service and have come to think of amazon web services. Does amazon also support hosting of sql server 2008 databases? What hosting services do you advise Thank you. >>> More
Unable to list owned images and running instances from Amazon Web Services using Zend Framework

as seen on Stack Overflow - Search for 'Stack Overflow'
I am using Zend Framework's library to manage EC2 instances and AMI. However I can't list the AMI's I own and can't list existing EC2 instances. $ec2Instance = new Zend_Service_Amazon_Ec2_Instance($awsAccessKey, $awsSecretKey); $instances = $ec2Instance ->describe(); $ec2Instance -describe()… >>> More
Amazon Web services - retrieving a wishlist

as seen on Stack Overflow - Search for 'Stack Overflow'
I've been tinkering with Yahoo Pipes and the Amazon E-Commerce Service (ECS) SDK to retrieve my wishlist. The problem is that although I can get all the items on my wishlist just fine, it seems to include items that I've deleted too. Has anyone else used this API and noticed this? Is there a way… >>> More
Amazon Web Services: A Developer Primer

as seen on Internet.com - Search for 'Internet.com'
With Amazon Web Services (AWS), developers get both scalable services that they can use to architect their applications and the flexibility to run any software on Amazon's compute cloud. >>> More
amazon web services

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, Has anyone of you worked on amazon web services? I just wanted to retrieve the citations of a given book. Is this possible using the aws? Is aws free? Thanks. >>> More