Fast ruby http library for large XML downloads

Posted by Vlad Zloteanu on Stack Overflow See other posts from Stack Overflow or by Vlad Zloteanu
Published on 2010-05-14T11:45:59Z Indexed on 2010/05/15 10:44 UTC
Read the original article Hit count: 349

Filed under:

ruby

|

http

|

ruby-on-rails

|

large-file-download

I am consuming various XML-over-HTTP web services returning large XML files (> 2MB). What would be the fastest ruby http library to reduce the 'downloading' time?

Required features:

both GET and POST requests
gzip/deflate downloads (Accept-Encoding: deflate, gzip) - very important

I am thinking between:

open-uri
Net::HTTP
curb

but you can also come with other suggestions.

P.S. To parse the response, I am using a pull parser from Nokogiri, so I don't need an integrated solution like rest-client or hpricot.

© Stack Overflow or respective owner

Related posts about ruby

Setting up Rails to work with sqlserver

as seen on Stack Overflow - Search for 'Stack Overflow'
Ok I followed the steps for setting up ruby and rails on my Vista machine and I am having a problem connecting to the database. Contents of database.yml development: adapter: sqlserver database: APPS_SETUP Host: WindowsVT06\SQLEXPRESS Username: se Password: paswd Run rake db:migrate… >>> More
marshal data too short!!!

as seen on Stack Overflow - Search for 'Stack Overflow'
My application requires to keep large data objects in session. There are like 3-4 data objects each created by parsing a csv containing 150 X 20 cells having strings of 3-4 characters. My application shows this error- "marshal data too short". I tried this- Deleting the old session table. Deleting… >>> More
Sinatra and XML POST request

as seen on Stack Overflow - Search for 'Stack Overflow'
I don't know is it my mistake or no. So i have that code: <code> post '/singin/get_token' do content_type :xml puts request.body.read puts xmlRequest xmlRequest = REXML::Document.new(request.body.read) ... </code> And when i post something like that: <code> <?xml… >>> More
how to change ruby path from /usr/bin/ruby to /usr/local/bin/ruby

as seen on Stack Overflow - Search for 'Stack Overflow'
reading around the various ruby install tutorials it's required to change path from /usr/bin/ruby to /usr/local/bin/ruby but i cant seem to be able to do it. Ultimately i want to install Ruby 1.9.2, should i uninstall 1.8.7 or what? i tried to install Ruby 1.9.2 with macports, the installation seemed… >>> More
strange bundler error: tar_input.rb:49:in `initialize': not in gzip format (Zlib::GzipFile::Error) o

as seen on Stack Overflow - Search for 'Stack Overflow'
i am getting a strange bundler error when running bundle pack with bundler 0.9.12 any ideas? (see pastie for a better formatted code: http://pastie.org/881328 ) /opt/ruby-enterprise-1.8.7-2010.01/lib/ruby/site_ruby/1.8/rubygems/package/tar_input.rb:49:in `initialize': not in gzip format (Zlib::GzipFile::Error) … >>> More

Related posts about http

Cannot update, apt-get cannot fetch index files

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I have a fresh install of Ubuntu 11.10 from the iso 'ubuntu-11.10-desktop-amd64.iso'. I installed this in VMWare Fusion 4.1.1 running on OSX 10.7.3. When setting up the VM, I allowed easy install to take care of creating my user and installing VMWare tools. No problems during installation, everything… >>> More
the size of apt-get update lists is too big

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I ran a clean install to Ubuntu 12.04 and so far everything has been working well. I especially commend the Ubuntu team for this release. I only noticed that the size of repository update is now about ~13MB. Normally, it is about this size for the first time you run apt-get update after a clean install… >>> More
Quantal: Broken apt-index, cant fix dependencies

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I can't seem to add/remove/update packages Ubuntu software update has a notice about partial upgrades but fails Seems to be similar to this problem $ sudo apt-get update Ign http://archive.ubuntu.com quantal InRelease Ign http://security.ubuntu.com precise-security InRelease Ign… >>> More
12.04: Apt-Get Update: failure to fetch; can't connect to any sources

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I realize there are dozens of "apt-get update: failure to fetch" questions (I read through all I could find), but my present circumstance is unique to 12.04 and it affects all sources; not just launchpad. Additionally, I've tried several different servers in Europe and the U.S. as well as the "main… >>> More
How can I remove the Translation entries in apt?

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
This is the output of aptitude update: Ign http://archive.canonical.com natty InRelease Ign http://extras.ubuntu.com natty InRelease Ign http://dl.google.com stable InRelease Ign http://security.ubuntu.com natty-security InRelease Hit http://deb.torproject.org natty InRelease Get:1 http://dl.google… >>> More