Parallelize Bash Script

Posted by thelsdj on Stack Overflow See other posts from Stack Overflow or by thelsdj
Published on 2008-09-01T16:47:53Z Indexed on 2010/06/10 1:42 UTC
Read the original article Hit count: 238

Filed under:

Lets say I have a loop in bash:

for foo in `some-command`
do
   do-something $foo
done

do-something is cpu bound and I have a nice shiny 4 core processor. I'd like to be able to run up to 4 do-something's at once.

The naive approach seems to be:

for foo in `some-command`
do
   do-something $foo &
done

This will run all do-somethings at once, but there are a couple downsides, mainly that do-something may also have some significant I/O which performing all at once might slow down a bit. The other problem is that this code block returns immediately, so no way to do other work when all the do-somethings are finished.

How would you write this loop so there are always X do-somethings running at once?

© Stack Overflow or respective owner

Related posts about bash