Running commands in parallel with a limit of simultaneous number of commands

Posted by Vi on Super User See other posts from Super User or by Vi
Published on 2010-06-17T11:47:04Z Indexed on 2010/06/17 11:54 UTC
Read the original article Hit count: 396

Filed under:

linux

|

bash

|

limit

|

background-job

Sequential: for i in {1..1000}; do do_something $i; done - too slow

Parallel: for i in {1..1000}; do do_something $i& done - too much load

How to run commands in parallel, but not more than, for example, 20 instances per moment?

Now usually using hack like for i in {1..1000}; do do_something $i& sleep 5; done, but this is not a good solution.

© Super User or respective owner

Related posts about linux

apt-get install and update fail

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I've got a problem with apt-get update and apt-get install ... commands . every time update or installing fails and errors are : Get:1 http://dl.google.com stable Release.gpg [198B] Ign http://dl.google.com/linux/chrome/deb/ stable/main Translation-en_US Get:2 http://dl… >>> More
kernel module compiling error

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
sh@ubuntu:/home/ccpp/helloworld$ make gcc-4.6 -O2 -DMODULE -D_KERNEL_ -W -Wall -Wstrict-prototypes -Wmissing-prototypes -isystem /lib/modules/`uname -r`/build/include -c -o hello-1.o hello-1.c hello-1.c:4:0: warning: "MODULE" redefined [enabled by default] <command-line>:0:0: note: this is… >>> More
Build-Essentials installation failing

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I am having trouble accessing the several critical header files that show to be a part of the build process. The "Ubuntu Software Center" shows "Build Essentials" as installed: Next I did the following two commands, which did not improve the problem: ~$ sudo apt-get install build-essential [sudo]… >>> More
Updating Debian kernel

as seen on Super User - Search for 'Super User'
I'm trying to update my Debian machine to 2.6.32-46 (which is the new stable). However, after doing apt-get update my apt-cache search linux-image shows me: linux-headers-2.6.32-5-486 - Header files for Linux 2.6.32-5-486 linux-headers-2.6.32-5-686-bigmem - Header files for Linux 2.6.32-5-686-bigmem linux-headers-2… >>> More
Serial connection over a single USB cable (Windows to linux, or linux to linux)

as seen on Server Fault - Search for 'Server Fault'
I'm helping out with a project for an embedded device that only has USB and no serial. This device is running Linux. These days, when we need to connect to a serial port on a device we typically use a USB to serial adapter (on something like a phone system or a load balancing device, etc). I would… >>> More

Related posts about bash

launching a program from bash causes bash to go to new prompt

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
When I run a program from the console, e.g. me@box:~$ firefox I expect the console to log error messages (I think this is std out or std err?) and other items from the program, firefox in this case. But today I notice that bash just opens the program and goes to a new prompt, e.g. me@box:~$… >>> More
How to debug a .bash_profile

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I was updating my .bash_profile, and unfortunetly I made a few updates and now I am getting: env: bash: No such file or directory env: bash: No such file or directory env: bash: No such file or directory env: bash: No such file or directory env: bash: No such file or directory -bash: tar: command… >>> More
Every command fails with "command not found" after changing .bash_profile?

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I was updating my .bash_profile, and unfortunetly I made a few updates and now I am getting: env: bash: No such file or directory env: bash: No such file or directory env: bash: No such file or directory env: bash: No such file or directory env: bash: No such file or directory -bash: tar: command… >>> More
Is there any fundamental difference between piping in mac and linux?

as seen on Super User - Search for 'Super User'
ps -e | grep bash sample output from a linux machine: 1128 pts/14 00:00:00 bash 7491 pts/7 00:00:00 bash 12651 pts/14 00:00:00 bash 16145 pts/2 00:00:00 bash sample output from a mac machine: 58352 ttys000 0:00.09 login -pfl username /bin/bash -c exec -la bash /bin/bash 58353 ttys000… >>> More
why is $0 set to -bash?

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
First login process name seems to be set to "-bash", but if I subshell then it becomes "bash". for example: root@nowere:~# echo $0 -bash root@nowere:~# bash root@nowere:~# echo $0 bash -bash is causing some scripts to fail, such as . /usr/share/debconf/confmodule exec /usr/share/debconf/frontend… >>> More