Processing large recordsets in Rails

Posted by japancheese on Stack Overflow See other posts from Stack Overflow or by japancheese
Published on 2010-04-08T21:08:24Z Indexed on 2010/04/08 21:13 UTC
Read the original article Hit count: 209

Filed under:
|
|

Hello, I'm trying to perform a daily operation on a larger than normal dataset (2m+ records). However, Rails seems to take a very long time performing operations on such a dataset. Operations like

Dataset.all.each do |data|
  ...
end

take a very long time to complete (I assume this is because it can't fit all the items into memory at once, right?).

Does anyone have any strategies on how I could handle this situation? I know SQL would probably speed up the process, but I'm looking to use the Rails environment as I can do many more complicated things to the data than I can with just SQL statements.

© Stack Overflow or respective owner

Related posts about ruby-on-rails

Related posts about scaling