Python Multiprocessing Question

Posted by Avan on Stack Overflow See other posts from Stack Overflow or by Avan
Published on 2010-12-23T02:47:23Z Indexed on 2010/12/23 2:54 UTC
Read the original article Hit count: 247

Filed under:
|

My program has 2 parts divided into the core and downloader. The core handles all the app logic while the downloader just downloads urls. Right now, I am trying to use the python multiprocessing module to accomplish the task of the core as a process and the downloader as a process.

The first problem I noticed was that if I spawn the downloader process from the core process so that the downloader is the child process and the core is the parent, the core process(parent) is blocked until the child is finished. I do not want this behaivor though. I would like to have a core process and a downloader process that are both able to execute their code and communicate between each other.

example

...
def main():
    jobQueue = Queue()
    jobQueue.put("http://google.com)
    d = Downloader(jobQueue)
    p = Process(target=d.start())
    p.start()

if __name__ == '__main__':
    freeze_support()
    main()

where Downloader's start() just takes the url out of the queue and downloads it.

In order to have the 2 processes unblocked, would I need to create 2 processes from the parent process and then share something between them?

© Stack Overflow or respective owner

Related posts about python

Related posts about multiprocessing