What are the best practices for importing large datasets into MongoDB?

Posted by snl on Stack Overflow See other posts from Stack Overflow or by snl
Published on 2010-05-25T17:13:39Z Indexed on 2010/05/25 17:21 UTC
Read the original article Hit count: 191

Filed under:

ruby-on-rails

|

ruby

|

mongodb

|

mongoid

We are just giving MongoDB a test run and have set up a Rails 3 app with Mongoid. What are the best practices for inserting large datasets into MongoDB? To flesh out a scenario: Say, I have a book model and want to import several million records from a CSV file.

I suppose this needs to be done in the console, so this may possibly not be a Ruby-specific question.

Edited to add: I assume it makes a huge difference whether the imported data includes associations or is supposed to go into one model only. Any comments on either scenario welcome.

© Stack Overflow or respective owner

Related posts about ruby-on-rails

Ruby on Rails - How can I start? [closed]

as seen on Programmers - Search for 'Programmers'
I have misconception in understanding the relationship between Ruby language and Ruby on Rails Framework. Because of 'I am an absolute beginner' in web development I have no idea if I have to grasp the fundamentals of Ruby before I go with Ruby on Rails! I also want to ask who is behind both Ruby… >>> More
DES3 decryption in Ruby on Rails

as seen on Stack Overflow - Search for 'Stack Overflow'
My RoR server receives a string, that was encrypted in C++ application using des3 with base64 encoding The cipher object is created so: cipher = OpenSSL::Cipher::Cipher::new("des3") cipher.key = key_str cipher.iv = iv_str key_str and iv_str: are string representations of key and initialization… >>> More
Ruby on rails: Image downloads with Authentication/Authorization/Time outs

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi Guys, I'm having few doubts on implementing file downloads. I'm creating an app where I use attachment_fu with Amazon s3 to upload files. Things are working pretty well so far on uploading side. Now its the time to start the file downloads. Here is what I need, a logged in user search and browse… >>> More
Ruby on Rails deployment, on "thin" server with lot of attachments

as seen on Stack Overflow - Search for 'Stack Overflow'
A lot of PDFs are stored inside MySQL as a BLOB field for each PDF file. The average file size is 500K each. The Rails app will stream the :binary data as file downloads, where there is a user click on the download link. Assume there is a maximum of 5 users downloading 5 PDFs concurrently, what… >>> More
Apply Behavior Driven Development to Ruby on Rails with Rspec

as seen on Internet.com - Search for 'Internet.com'
Applying the Behavior Driven Development (BDD) methodology to your Rails development takes you beyond traditional Test Driven Development. Learn how to do it with the Ruby-based BDD framework RSpec. >>> More

Related posts about ruby

Setting up Rails to work with sqlserver

as seen on Stack Overflow - Search for 'Stack Overflow'
Ok I followed the steps for setting up ruby and rails on my Vista machine and I am having a problem connecting to the database. Contents of database.yml development: adapter: sqlserver database: APPS_SETUP Host: WindowsVT06\SQLEXPRESS Username: se Password: paswd Run rake db:migrate… >>> More
marshal data too short!!!

as seen on Stack Overflow - Search for 'Stack Overflow'
My application requires to keep large data objects in session. There are like 3-4 data objects each created by parsing a csv containing 150 X 20 cells having strings of 3-4 characters. My application shows this error- "marshal data too short". I tried this- Deleting the old session table. Deleting… >>> More
Sinatra and XML POST request

as seen on Stack Overflow - Search for 'Stack Overflow'
I don't know is it my mistake or no. So i have that code: <code> post '/singin/get_token' do content_type :xml puts request.body.read puts xmlRequest xmlRequest = REXML::Document.new(request.body.read) ... </code> And when i post something like that: <code> <?xml… >>> More
how to change ruby path from /usr/bin/ruby to /usr/local/bin/ruby

as seen on Stack Overflow - Search for 'Stack Overflow'
reading around the various ruby install tutorials it's required to change path from /usr/bin/ruby to /usr/local/bin/ruby but i cant seem to be able to do it. Ultimately i want to install Ruby 1.9.2, should i uninstall 1.8.7 or what? i tried to install Ruby 1.9.2 with macports, the installation seemed… >>> More
strange bundler error: tar_input.rb:49:in `initialize': not in gzip format (Zlib::GzipFile::Error) o

as seen on Stack Overflow - Search for 'Stack Overflow'
i am getting a strange bundler error when running bundle pack with bundler 0.9.12 any ideas? (see pastie for a better formatted code: http://pastie.org/881328 ) /opt/ruby-enterprise-1.8.7-2010.01/lib/ruby/site_ruby/1.8/rubygems/package/tar_input.rb:49:in `initialize': not in gzip format (Zlib::GzipFile::Error) … >>> More