Using delayed_job to process file uploads across multiple servers

Posted by Steve Klabnik on Stack Overflow See other posts from Stack Overflow or by Steve Klabnik
Published on 2010-03-25T20:40:15Z Indexed on 2010/03/25 20:43 UTC
Read the original article Hit count: 267

Filed under:

rails

|

delayed-job

|

fileupload

Does anyone have any good resources on how to do this?

Basically, I'm working on a project (in Rails) where people can upload files. They might be big. I'd like to process them using delayed_job before sending them to S3. I'd also like to do this processing on a separate job queue server, rather than on the webserver itself. I'd rather not have to upload the files to the webserver, then transfer them to the job queue server, and then upload them to S3 if I don't have to.

Thanks.

© Stack Overflow or respective owner

Related posts about rails

Ruby on Rails: url_for :back leads to NoMethodError for back_url

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi all, I'm trying to use url_for(:back) to create a redirect leading back to a previous page upon a user's logging in. I've had it working successfully for when the user just goes to the login page on his or her own. However, when the user is redirected to the login page due to accessing a page… >>> More
Why am I getting "ArgumentError: wrong number of arguments (1 for 0)" when running my rails function

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm stumped on what's causing this. I get this error and stack trace in all my functional tests where I call 'post'. Here is the full stack trace: 7) Error: test_should_validate(UsersControllerTest): ArgumentError: wrong number of arguments (1 for 0) /Users/hisham/src/rails/ftuBackend/vendor/rails/actionpack/lib/action_controller/routing/route… >>> More
Spork servers super slow (>3m) to start for RSpec & Cucumber BDD

as seen on Stack Overflow - Search for 'Stack Overflow'
I recently installed a fresh development setup on my laptop and now notice that my instances of spork take several minutes to start up. This is also most likely of the RSpec and Cucumber tests start up times running super slow. I ran in diagnostic mode with the -d flag and received the output below… >>> More
trying to convert rails 2.3.4 app to rails 3 but views not appearing

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a simple rails 2.3.4 application I am trying to get running with rails 3.0.0-beta2. I have the server running and all my links and navigation showing, however for some reason the content of my views is not displaying. When I navigate to Site/index the html in my index.html.erb in my site… >>> More
Create new Rails project with previews rails version

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello, The latest version of rails installed on my mac is 3.0.0.beta gem list returns rails (3.0.0.beta, 2.3.5, 2.3.4, 2.3.2, 2.2.2, 1.2.6) I want to create a new project using version 2.3.5 So I run: rails _2.3.5_ myProject But it returns the following error $ rails _2.3.5_ photosbackup /Library/Ruby/Site/1… >>> More

Related posts about delayed-job

Use Delayed::Job to manage multiple job queues

as seen on Stack Overflow - Search for 'Stack Overflow'
I want to use Delayed::Job (or perhaps a more appropriate job queue to my problem) to dispatch jobs to multiple background daemons. I have several background daemons that carry out different responsibilities. Each one is interested in different jobs in the queue from the Rails app. Is this possible… >>> More
Rails Delayed Job & Library Class

as seen on Stack Overflow - Search for 'Stack Overflow'
Hey we have a library class (lib/Mixpanel) that calls delayed job as follows: class Mixpanel attr_accessor :options attr_accessor :event def track!() .. dj = send_later :access_api # also tried with self.send_later .. end def access_api .. end The problem is that when… >>> More
set up delayed job with rails

as seen on Stack Overflow - Search for 'Stack Overflow'
What is the difference between send_later and perform action? >>> More
An affordable way to use multiple Delayed::Job queues

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a Ruby on Rails app that needs process many background jobs simultaneously: anywhere from 5-6 at a time to up to 50-60 at a time depending on the time of day. Right now my app is running on Heroku, which charges $.05/hour per worker, regardless of how much CPU or memory the worker is using… >>> More
Delayed_job: undefined method `handle_asynchronously' for #<Class:0x4601b08>

as seen on Stack Overflow - Search for 'Stack Overflow'
After deploying DelayedJob as a gem, I got the following error: undefined method `handle_asynchronously' for #<Class:0x4601b08> Same code works when I deploy DelayedJob as a plugin. I have tried the solution suggested by this SO question and I still get the same error. I am using DelayedJob… >>> More