Python Multiprocessing Question
Posted
by
Avan
on Stack Overflow
See other posts from Stack Overflow
or by Avan
Published on 2010-12-23T02:47:23Z
Indexed on
2010/12/23
2:54 UTC
Read the original article
Hit count: 249
python
|multiprocessing
My program has 2 parts divided into the core and downloader. The core handles all the app logic while the downloader just downloads urls. Right now, I am trying to use the python multiprocessing module to accomplish the task of the core as a process and the downloader as a process.
The first problem I noticed was that if I spawn the downloader process from the core process so that the downloader is the child process and the core is the parent, the core process(parent) is blocked until the child is finished. I do not want this behaivor though. I would like to have a core process and a downloader process that are both able to execute their code and communicate between each other.
example
...
def main():
jobQueue = Queue()
jobQueue.put("http://google.com)
d = Downloader(jobQueue)
p = Process(target=d.start())
p.start()
if __name__ == '__main__':
freeze_support()
main()
where Downloader's start() just takes the url out of the queue and downloads it.
In order to have the 2 processes unblocked, would I need to create 2 processes from the parent process and then share something between them?
© Stack Overflow or respective owner