Improve efficiency when using parallel to read from compressed stream
- by Yoga
Is another question extended from the previous one [1]
I have a compressed file and stream them to feed into a python program, e.g.
bzcat data.bz2 | parallel --no-notice -j16 --pipe python parse.py > result.txt
The parse.py can read from stdin continusuoly and print to stdout
My ec2 instance is 16 cores but from the top command it is…