Best Work Queue service for distributed clusters

Posted by onewheelgood on Stack Overflow See other posts from Stack Overflow or by onewheelgood
Published on 2010-05-27T07:45:49Z Indexed on 2010/05/27 7:51 UTC
Read the original article Hit count: 291

Filed under:
|

Hi there.

I require a simple work queue type system for asynchronous task management.

I have looked at both beanstalkd and gearman. However, both these seem to assume that the client and the queue server are on the same network, and therefore that there will always be a reliable network between them.

I need one that can support the client and server being in different places in the world, and be able to manage temporary loss of network connection between clusters.

Ideally, this would work in such a way where I post a job to a local proxy that attempts to send it to the main queue server. If there is no network connection, it would try again later, however it would not lose the job or delay the client.

Any recommendations?

© Stack Overflow or respective owner

Related posts about message-queue

Related posts about distributed