Rails running multiple delayed_job - lock tables

Posted by pepernik on Stack Overflow See other posts from Stack Overflow or by pepernik
Published on 2010-04-25T08:17:28Z Indexed on 2010/04/25 8:23 UTC
Read the original article Hit count: 311

Hey. I use delayed_job for background processing. I have 8 CPU server, MySQL and I start 7 delayed_job processes

RAILS_ENV=production script/delayed_job -n 7 start 

Q1: I'm wondering is it possible that 2 or more delayed_job processes start processing the same process (the same record-row in the database delayed_jobs). I checked the code of the delayed_job plugin but can not find the lock directive in a way it should be.

I think each process should lock the database table before executing an UPDATE on lock_by column. They lock the record simply by updating the locked_by field (UPDATE delayed_jobs SET locked_by...). Is that really enough? No locking needed? Why? I know that UPDATE has higher priority than SELECT but I think this does not have the effect in this case.

My understanding of the multy-threaded situation is:

Process1: Get waiting job X. [OK]
Process2: Get waiting jobs X. [OK]
Process1: Update locked_by field. [OK]
Process2: Update locked_by field. [OK]
Process1: Get waiting job X. [Already processed]
Process2: Get waiting jobs X. [Already processed]

I think in some cases more jobs can get the same information and can start processing the same process.

Q2: Is 7 delayed_jobs a good number for 8CPU server? Why yes/not.

Thx 10x!

© Stack Overflow or respective owner

Related posts about delayed-job

Related posts about multithreading