Long running, polling, queueing process for Python. What's the best stuff to use?

Posted by Bialecki on Stack Overflow See other posts from Stack Overflow or by Bialecki
Published on 2010-04-20T14:45:45Z Indexed on 2010/04/20 15:43 UTC
Read the original article Hit count: 264

Filed under:

Feel free to close and/or redirect if this has been asked, but here's my situation:

I've got an application that will require doing a bunch of small units of work (polling a web service until something is done, then parsing about 1MB worth of XML and putting it in a database). I want to have a simple async queueing mechanism that'll poll for work to do in a queue, execute the units of work that need to be done, and have the flexibility to allow for spawning multiple worker processes so these units of work can be done in parallel. (Bonus if there's some kind of event framework that would also me to listen for when work is complete.)

I'm sure there is stuff to do this. Am I describing Twisted? I poked through the documentation, I'm just not sure exactly how my problems maps onto their framework, but I haven't spent much time with it. Should I just look at the multiprocess libraries in Python? Something else?

© Stack Overflow or respective owner

Related posts about python