Asynchronous daemon processing / ORM interaction with Django
Posted
by perrierism
on Stack Overflow
See other posts from Stack Overflow
or by perrierism
Published on 2010-05-10T19:47:26Z
Indexed on
2010/05/11
7:24 UTC
Read the original article
Hit count: 445
I'm looking for a way to do asynchronous data processing with a daemon that uses Django ORM. However, the ORM isn't thread-safe; it's not thread-safe to try to retrieve / modify django objects from within threads. So I'm wondering what the correct way to achieve asynchrony is?
Basically what I need to accomplish is taking a list of users in the db, querying a third party api and then making updates to user-profile rows for those users. As a daemon or background process. Doing this in series per user is easy, but it takes too long to be at all scalable. If the daemon is retrieving and updating the users through the ORM, how do I achieve processing 10-20 users at a time? I would use a standard threading / queue system for this but you can't thread interactions like
models.User.objects.get(id=foo) ...
Django itself is an asynchronous processing system which makes asynchronous ORM calls(?) for each request, so there should be a way to do it? I haven't found anything in the documentation so far.
Cheers
© Stack Overflow or respective owner