How can I configure different worker pools using celery?
Posted
by
Chris R
on Stack Overflow
See other posts from Stack Overflow
or by Chris R
Published on 2011-01-12T04:49:30Z
Indexed on
2011/01/12
4:53 UTC
Read the original article
Hit count: 223
I need to deploy a queued execution service with (generally) the following three classes of worker:
A periodic, low-priority job class that takes a long time and can be processed serially; these jobs should only use 0..2 workers in the system at most.
A periodic, deadline-sensitive job class that take a short to medium amount of time (say, topping out at 5 minutes)
An ad-hoc job class, that is higher priority than #1, but can interleave with #2. Any workers from class #2 that are inactive when this type of job comes in should handle it, without ever starving the pool of workers for #2
All three job classes are the same task, the only difference between them is how they're requested; they'll take the same input and generate the same output, but each one has different performance guarantees.
How can I implement this using celery?
© Stack Overflow or respective owner
