Python - Help with multiprocessing / threading basics.
- by orokusaki
I haven't ever used multi-threading, and I decided to learn it today. I was reluctant to ever use it before, but when I tried it out it seemed way to easy, which makes me wary.
Are there any gotchas in my code, or is it really that simple?
import uuid
import time
import multiprocessing
def sleep_then_write(content):
time.sleep(5)
f = open(unicode(uuid.uuid4()), 'w')
f.write(content)
f.close()
if __name__ == '__main__':
for i in range(3):
p = multiprocessing.Process(target=sleep_then_write,
args=('Hello World',))
p.start()
My primary purpose of using threading would be to offload multiple images to S3 after re-sizing them, all at the same time. Is that a reasonable task for Python's multiprocessing? I've read a lot about certain types of tasks not really getting any gain from using threading in Python due to the GIL, but it seems that multiprocessing completely removes that worry, yes?
I can imagine a case where 50 users hit the system and it spawns 150 Python interpreters. I can also imagine that wouldn't be good on a production server. How can something like that be avoided?
Finally (but most important): How can I return control back to the caller of the new processes? I need to be able to continue with returning an HTTP response and content back to the user and then have the processes continue doing there work after the user of my website is done with the transaction.
