Python IPC, popen too slow

Posted by UnableToLoad on Stack Overflow See other posts from Stack Overflow or by UnableToLoad
Published on 2012-05-30T22:38:47Z Indexed on 2012/05/30 22:40 UTC
Read the original article Hit count: 237

Filed under:
|
|
|

i need to run a subprocess (./myProgram) form python script and get output, actually i do this:

 import subprocess
 proc = subprocess.Popen('./generate_out',
                   shell=False,
                   stdout=subprocess.PIPE,
                   )

 while proc.poll() is None:
    out = proc.stdout.readline()
    data = doStuff(out)
    print(data)

but is slow, sometimes pass a lot of time between the output produced by ./generate_out and the print(data), knowing that my doStuff() function is very fast, i think there is some buffer slowing down my pipe...

Notes: ./generate_out, generates potentially an unlimited number of lines of finite length each. It seems that when too few chars are put in the pipe between the two processes nothing happens, then when enough is produced i get a huge print (non the expected behaviour!) sometimes i wait many seconds (10-20 and more) between generate_out print and python print)

what can i do? maybe communicate() is faster? anithing else?

Thank you a lot!

© Stack Overflow or respective owner

Related posts about python

Related posts about ipc