Minimal "Task Queue" with stock Linux tools to leverage Multicore CPU
Posted
by Manuel
on Stack Overflow
See other posts from Stack Overflow
or by Manuel
Published on 2009-05-06T23:14:57Z
Indexed on
2010/06/10
20:13 UTC
Read the original article
Hit count: 283
What is the best/easiest way to build a minimal task queue system for Linux using bash and common tools?
I have a file with 9'000 lines, each line has a bash command line, the commands are completely independent.
command 1 > Logs/1.log
command 2 > Logs/2.log
command 3 > Logs/3.log
...
My box has more than one core and I want to execute X tasks at the same time. I searched the web for a good way to do this. Apparently, a lot of people have this problem but nobody has a good solution so far.
It would be nice if the solution had the following features:
- can interpret more than one command (e.g.
command; command
) - can interpret stream redirects on the lines (e.g.
ls > /tmp/ls.txt
) - only uses common Linux tools
Bonus points if it works on other Unix-clones without too exotic requirements.
© Stack Overflow or respective owner