Asynchronous daemon processing / ORM interaction with Django

Posted by perrierism on Stack Overflow See other posts from Stack Overflow or by perrierism
Published on 2010-05-10T19:47:26Z Indexed on 2010/05/11 7:24 UTC
Read the original article Hit count: 441

I'm looking for a way to do asynchronous data processing with a daemon that uses Django ORM. However, the ORM isn't thread-safe; it's not thread-safe to try to retrieve / modify django objects from within threads. So I'm wondering what the correct way to achieve asynchrony is?

Basically what I need to accomplish is taking a list of users in the db, querying a third party api and then making updates to user-profile rows for those users. As a daemon or background process. Doing this in series per user is easy, but it takes too long to be at all scalable. If the daemon is retrieving and updating the users through the ORM, how do I achieve processing 10-20 users at a time? I would use a standard threading / queue system for this but you can't thread interactions like

models.User.objects.get(id=foo) ...

Django itself is an asynchronous processing system which makes asynchronous ORM calls(?) for each request, so there should be a way to do it? I haven't found anything in the documentation so far.

Cheers

© Stack Overflow or respective owner

Related posts about django

Related posts about django-orm