Minimal "Task Queue" with stock Linux tools to leverage Multicore CPU

Posted by Manuel on Stack Overflow See other posts from Stack Overflow or by Manuel
Published on 2009-05-06T23:14:57Z Indexed on 2010/06/10 20:13 UTC
Read the original article Hit count: 286

Filed under:
|
|

What is the best/easiest way to build a minimal task queue system for Linux using bash and common tools?

I have a file with 9'000 lines, each line has a bash command line, the commands are completely independent.

command 1 > Logs/1.log
command 2 > Logs/2.log
command 3 > Logs/3.log
...

My box has more than one core and I want to execute X tasks at the same time. I searched the web for a good way to do this. Apparently, a lot of people have this problem but nobody has a good solution so far.

It would be nice if the solution had the following features:

  • can interpret more than one command (e.g. command; command)
  • can interpret stream redirects on the lines (e.g. ls > /tmp/ls.txt)
  • only uses common Linux tools

Bonus points if it works on other Unix-clones without too exotic requirements.

© Stack Overflow or respective owner

Related posts about bash

Related posts about queue