Gearman too many processes issue
Posted
by
Roman Newaza
on Server Fault
See other posts from Server Fault
or by Roman Newaza
Published on 2013-10-22T08:20:57Z
Indexed on
2013/10/22
9:58 UTC
Read the original article
Hit count: 733
I use Net_Gearman from PECL, Gearmand 1.1.11 and Gearman Manager. Every time I add background job, I can see new worker listed with no Function, nor Id in Ggearman-Monitor:
If I add many messages in the bash loop, after some time it becomes very slow.
for i in $(seq 0 9999); do php Client.php && echo $i; done
Yesterday, the situation was even worse - I had many error messages in Gearmand log regarding Too many open files and once I added --file-descriptors=49152 as an option and swithched to 1.1.11 from 1.0.6, these errors gone.
Here is lsof -p $(cat /var/run/gearman/gearmand.pid) output:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
gearmand 2020 gearman cwd DIR 8,2 4096 2 /
gearmand 2020 gearman rtd DIR 8,2 4096 2 /
gearmand 2020 gearman txt REG 8,2 3852472 3672962 /opt/sbin/gearmand
gearmand 2020 gearman mem REG 8,2 52120 9961752 /lib/x86_64-linux-gnu/libnss_files-2.15.so
gearmand 2020 gearman mem REG 8,2 47680 9961756 /lib/x86_64-linux-gnu/libnss_nis-2.15.so
gearmand 2020 gearman mem REG 8,2 97248 9961768 /lib/x86_64-linux-gnu/libnsl-2.15.so
gearmand 2020 gearman mem REG 8,2 35680 9961750 /lib/x86_64-linux-gnu/libnss_compat-2.15.so
gearmand 2020 gearman mem REG 8,2 92720 9964871 /lib/x86_64-linux-gnu/libz.so.1.2.3.4
gearmand 2020 gearman mem REG 8,2 109288 11014600 /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25
gearmand 2020 gearman mem REG 8,2 1030512 9961759 /lib/x86_64-linux-gnu/libm-2.15.so
gearmand 2020 gearman mem REG 8,2 1930616 9964982 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
gearmand 2020 gearman mem REG 8,2 382896 9964977 /lib/x86_64-linux-gnu/libssl.so.1.0.0
gearmand 2020 gearman mem REG 8,2 1815224 9961748 /lib/x86_64-linux-gnu/libc-2.15.so
gearmand 2020 gearman mem REG 8,2 88384 9964865 /lib/x86_64-linux-gnu/libgcc_s.so.1
gearmand 2020 gearman mem REG 8,2 962656 11014043 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
gearmand 2020 gearman mem REG 8,2 199600 11016157 /usr/lib/x86_64-linux-gnu/libmemcached.so.11.0.0
gearmand 2020 gearman mem REG 8,2 31752 9961755 /lib/x86_64-linux-gnu/librt-2.15.so
gearmand 2020 gearman mem REG 8,2 14768 9961763 /lib/x86_64-linux-gnu/libdl-2.15.so
gearmand 2020 gearman mem REG 8,2 414280 9183971 /usr/lib/libboost_program_options.so.1.46.1
gearmand 2020 gearman mem REG 8,2 283832 9183656 /usr/lib/libevent-2.0.so.5.1.4
gearmand 2020 gearman mem REG 8,2 664504 11014432 /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6
gearmand 2020 gearman mem REG 8,2 135366 9961757 /lib/x86_64-linux-gnu/libpthread-2.15.so
gearmand 2020 gearman mem REG 8,2 3534240 9175810 /usr/lib/libmysqlclient.so.18.1.0
gearmand 2020 gearman mem REG 8,2 149280 9961760 /lib/x86_64-linux-gnu/ld-2.15.so
gearmand 2020 gearman 0u CHR 1,3 0t0 1029 /dev/null
gearmand 2020 gearman 1u CHR 1,3 0t0 1029 /dev/null
gearmand 2020 gearman 2u CHR 1,3 0t0 1029 /dev/null
gearmand 2020 gearman 3w REG 8,2 9381897 3409366 /var/log/gearman-job-server/gearman.log
gearmand 2020 gearman 4r FIFO 0,8 0t0 38869143 pipe
gearmand 2020 gearman 5w FIFO 0,8 0t0 38869143 pipe
gearmand 2020 gearman 6u 0000 0,9 0 6826 anon_inode
gearmand 2020 gearman 7u unix 0xffff880230fdf500 0t0 38869144 socket
gearmand 2020 gearman 8u unix 0xffff880230fdde40 0t0 38869145 socket
gearmand 2020 gearman 9u IPv4 38869146 0t0 TCP localhost:4730 (LISTEN)
gearmand 2020 gearman 10r FIFO 0,8 0t0 38869147 pipe
gearmand 2020 gearman 11w FIFO 0,8 0t0 38869147 pipe
gearmand 2020 gearman 12u 0000 0,9 0 6826 anon_inode
gearmand 2020 gearman 13u unix 0xffff880230fde4c0 0t0 38869148 socket
gearmand 2020 gearman 14u unix 0xffff880230fdeb40 0t0 38869149 socket
gearmand 2020 gearman 15r FIFO 0,8 0t0 38869150 pipe
gearmand 2020 gearman 16w FIFO 0,8 0t0 38869150 pipe
gearmand 2020 gearman 17u 0000 0,9 0 6826 anon_inode
gearmand 2020 gearman 18u 0000 0,9 0 6826 anon_inode
gearmand 2020 gearman 19u unix 0xffff880230fdb400 0t0 38869151 socket
gearmand 2020 gearman 20u unix 0xffff880230fdaa40 0t0 38869152 socket
gearmand 2020 gearman 21r FIFO 0,8 0t0 38869153 pipe
gearmand 2020 gearman 22w FIFO 0,8 0t0 38869153 pipe
gearmand 2020 gearman 23u unix 0xffff880203cfce00 0t0 38868290 socket
gearmand 2020 gearman 24u unix 0xffff880203cfdb00 0t0 38868291 socket
gearmand 2020 gearman 25r FIFO 0,8 0t0 38868292 pipe
gearmand 2020 gearman 26w FIFO 0,8 0t0 38868292 pipe
gearmand 2020 gearman 27u 0000 0,9 0 6826 anon_inode
gearmand 2020 gearman 28u unix 0xffff880203cf9040 0t0 38868293 socket
gearmand 2020 gearman 29u unix 0xffff880203cfaa40 0t0 38868294 socket
gearmand 2020 gearman 30r FIFO 0,8 0t0 38868295 pipe
gearmand 2020 gearman 31w FIFO 0,8 0t0 38868295 pipe
gearmand 2020 gearman 32u IPv4 38868324 0t0 TCP localhost:4730->localhost:57954 (ESTABLISHED)
gearmand 2020 gearman 33u IPv4 38868325 0t0 TCP localhost:4730->localhost:57955 (ESTABLISHED)
gearmand 2020 gearman 34u IPv4 38901247 0t0 TCP localhost:4730->localhost:38594 (ESTABLISHED)
gearmand 2020 gearman 35u IPv4 38868327 0t0 TCP localhost:4730->localhost:57957 (ESTABLISHED)
gearmand 2020 gearman 36u IPv4 38867483 0t0 TCP localhost:4730->localhost:57959 (ESTABLISHED)
gearmand 2020 gearman 37u IPv4 38867484 0t0 TCP localhost:4730->localhost:57958 (ESTABLISHED)
gearmand 2020 gearman 38u IPv4 38901248 0t0 TCP localhost:4730->localhost:38595 (CLOSE_WAIT)
gearmand 2020 gearman 39u IPv4 38901249 0t0 TCP localhost:4730->localhost:38597 (ESTABLISHED)
gearmand 2020 gearman 40u IPv4 38869201 0t0 TCP localhost:4730->localhost:57979 (ESTABLISHED)
gearmand 2020 gearman 41u IPv4 38900437 0t0 TCP localhost:4730->localhost:38599 (ESTABLISHED)
gearmand 2020 gearman 42u IPv4 38900438 0t0 TCP localhost:4730->localhost:38602 (ESTABLISHED)
gearmand 2020 gearman 43u IPv4 38868375 0t0 TCP localhost:4730->localhost:57987 (ESTABLISHED)
gearmand 2020 gearman 44u IPv4 38900468 0t0 TCP localhost:4730->localhost:38606 (CLOSE_WAIT)
gearmand 2020 gearman 45u IPv4 38868381 0t0 TCP localhost:4730->localhost:57999 (ESTABLISHED)
gearmand 2020 gearman 46u IPv4 38868388 0t0 TCP localhost:4730->localhost:58007 (ESTABLISHED)
gearmand 2020 gearman 47u IPv4 38868393 0t0 TCP localhost:4730->localhost:58011 (ESTABLISHED)
gearmand 2020 gearman 48u IPv4 38903950 0t0 TCP localhost:4730->localhost:38609 (ESTABLISHED)
gearmand 2020 gearman 49u IPv4 38870276 0t0 TCP localhost:4730->localhost:58019 (ESTABLISHED)
gearmand 2020 gearman 50u IPv4 38903955 0t0 TCP localhost:4730->localhost:38613 (ESTABLISHED)
gearmand 2020 gearman 51u IPv4 38900477 0t0 TCP localhost:4730->localhost:38617 (CLOSE_WAIT)
gearmand 2020 gearman 52u IPv4 38867630 0t0 TCP localhost:4730->localhost:58031 (ESTABLISHED)
gearmand 2020 gearman 53u IPv4 38867633 0t0 TCP localhost:4730->localhost:58035 (ESTABLISHED)
gearmand 2020 gearman 54u IPv4 38867636 0t0 TCP localhost:4730->localhost:58039 (ESTABLISHED)
gearmand 2020 gearman 55u IPv4 38900536 0t0 TCP localhost:4730->localhost:38619 (ESTABLISHED)
gearmand 2020 gearman 56u IPv4 38868419 0t0 TCP localhost:4730->localhost:58047 (ESTABLISHED)
gearmand 2020 gearman 57u IPv4 38869263 0t0 TCP localhost:4730->localhost:58051 (ESTABLISHED)
gearmand 2020 gearman 58u IPv4 38900537 0t0 TCP localhost:4730->localhost:38621 (ESTABLISHED)
gearmand 2020 gearman 59u IPv4 38869271 0t0 TCP localhost:4730->localhost:58059 (ESTABLISHED)
gearmand 2020 gearman 60u IPv4 38900538 0t0 TCP localhost:4730->localhost:38623 (ESTABLISHED)
gearmand 2020 gearman 61u IPv4 38870319 0t0 TCP localhost:4730->localhost:58067 (ESTABLISHED)
gearmand 2020 gearman 62u IPv4 38900540 0t0 TCP localhost:4730->localhost:38628 (ESTABLISHED)
gearmand 2020 gearman 63u IPv4 38869289 0t0 TCP localhost:4730->localhost:58075 (ESTABLISHED)
...
gearmand 2020 gearman 2229u IPv4 38903885 0t0 TCP localhost:4730->localhost:38572 (ESTABLISHED)
gearmand 2020 gearman 2230u IPv4 38901211 0t0 TCP localhost:4730->localhost:38576 (ESTABLISHED)
gearmand 2020 gearman 2234u IPv4 38901237 0t0 TCP localhost:4730->localhost:38588 (ESTABLISHED)
© Server Fault or respective owner
