Search Results

Search found 16 results on 1 pages for 'gearman'.

Page 1/1 | 1 

• Gearman too many processes issue

  - by Roman Newaza

  I use Net_Gearman from PECL, Gearmand 1.1.11 and Gearman Manager. Every time I add background job, I can see new worker listed with no Function, nor Id in Ggearman-Monitor: If I add many messages in the bash loop, after some time it becomes very slow. for i in $(seq 0 9999); do php Client.php && echo $i; done Yesterday, the situation was even worse - I had many error messages in Gearmand log regarding Too many open files and once I added --file-descriptors=49152 as an option and swithched to 1.1.11 from 1.0.6, these errors gone. Here is lsof -p $(cat /var/run/gearman/gearmand.pid) output: COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME gearmand 2020 gearman cwd DIR 8,2 4096 2 / gearmand 2020 gearman rtd DIR 8,2 4096 2 / gearmand 2020 gearman txt REG 8,2 3852472 3672962 /opt/sbin/gearmand gearmand 2020 gearman mem REG 8,2 52120 9961752 /lib/x86_64-linux-gnu/libnss_files-2.15.so gearmand 2020 gearman mem REG 8,2 47680 9961756 /lib/x86_64-linux-gnu/libnss_nis-2.15.so gearmand 2020 gearman mem REG 8,2 97248 9961768 /lib/x86_64-linux-gnu/libnsl-2.15.so gearmand 2020 gearman mem REG 8,2 35680 9961750 /lib/x86_64-linux-gnu/libnss_compat-2.15.so gearmand 2020 gearman mem REG 8,2 92720 9964871 /lib/x86_64-linux-gnu/libz.so.1.2.3.4 gearmand 2020 gearman mem REG 8,2 109288 11014600 /usr/lib/x86_64-linux-gnu/libsasl2.so.2.0.25 gearmand 2020 gearman mem REG 8,2 1030512 9961759 /lib/x86_64-linux-gnu/libm-2.15.so gearmand 2020 gearman mem REG 8,2 1930616 9964982 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 gearmand 2020 gearman mem REG 8,2 382896 9964977 /lib/x86_64-linux-gnu/libssl.so.1.0.0 gearmand 2020 gearman mem REG 8,2 1815224 9961748 /lib/x86_64-linux-gnu/libc-2.15.so gearmand 2020 gearman mem REG 8,2 88384 9964865 /lib/x86_64-linux-gnu/libgcc_s.so.1 gearmand 2020 gearman mem REG 8,2 962656 11014043 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16 gearmand 2020 gearman mem REG 8,2 199600 11016157 /usr/lib/x86_64-linux-gnu/libmemcached.so.11.0.0 gearmand 2020 gearman mem REG 8,2 31752 9961755 /lib/x86_64-linux-gnu/librt-2.15.so gearmand 2020 gearman mem REG 8,2 14768 9961763 /lib/x86_64-linux-gnu/libdl-2.15.so gearmand 2020 gearman mem REG 8,2 414280 9183971 /usr/lib/libboost_program_options.so.1.46.1 gearmand 2020 gearman mem REG 8,2 283832 9183656 /usr/lib/libevent-2.0.so.5.1.4 gearmand 2020 gearman mem REG 8,2 664504 11014432 /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6 gearmand 2020 gearman mem REG 8,2 135366 9961757 /lib/x86_64-linux-gnu/libpthread-2.15.so gearmand 2020 gearman mem REG 8,2 3534240 9175810 /usr/lib/libmysqlclient.so.18.1.0 gearmand 2020 gearman mem REG 8,2 149280 9961760 /lib/x86_64-linux-gnu/ld-2.15.so gearmand 2020 gearman 0u CHR 1,3 0t0 1029 /dev/null gearmand 2020 gearman 1u CHR 1,3 0t0 1029 /dev/null gearmand 2020 gearman 2u CHR 1,3 0t0 1029 /dev/null gearmand 2020 gearman 3w REG 8,2 9381897 3409366 /var/log/gearman-job-server/gearman.log gearmand 2020 gearman 4r FIFO 0,8 0t0 38869143 pipe gearmand 2020 gearman 5w FIFO 0,8 0t0 38869143 pipe gearmand 2020 gearman 6u 0000 0,9 0 6826 anon_inode gearmand 2020 gearman 7u unix 0xffff880230fdf500 0t0 38869144 socket gearmand 2020 gearman 8u unix 0xffff880230fdde40 0t0 38869145 socket gearmand 2020 gearman 9u IPv4 38869146 0t0 TCP localhost:4730 (LISTEN) gearmand 2020 gearman 10r FIFO 0,8 0t0 38869147 pipe gearmand 2020 gearman 11w FIFO 0,8 0t0 38869147 pipe gearmand 2020 gearman 12u 0000 0,9 0 6826 anon_inode gearmand 2020 gearman 13u unix 0xffff880230fde4c0 0t0 38869148 socket gearmand 2020 gearman 14u unix 0xffff880230fdeb40 0t0 38869149 socket gearmand 2020 gearman 15r FIFO 0,8 0t0 38869150 pipe gearmand 2020 gearman 16w FIFO 0,8 0t0 38869150 pipe gearmand 2020 gearman 17u 0000 0,9 0 6826 anon_inode gearmand 2020 gearman 18u 0000 0,9 0 6826 anon_inode gearmand 2020 gearman 19u unix 0xffff880230fdb400 0t0 38869151 socket gearmand 2020 gearman 20u unix 0xffff880230fdaa40 0t0 38869152 socket gearmand 2020 gearman 21r FIFO 0,8 0t0 38869153 pipe gearmand 2020 gearman 22w FIFO 0,8 0t0 38869153 pipe gearmand 2020 gearman 23u unix 0xffff880203cfce00 0t0 38868290 socket gearmand 2020 gearman 24u unix 0xffff880203cfdb00 0t0 38868291 socket gearmand 2020 gearman 25r FIFO 0,8 0t0 38868292 pipe gearmand 2020 gearman 26w FIFO 0,8 0t0 38868292 pipe gearmand 2020 gearman 27u 0000 0,9 0 6826 anon_inode gearmand 2020 gearman 28u unix 0xffff880203cf9040 0t0 38868293 socket gearmand 2020 gearman 29u unix 0xffff880203cfaa40 0t0 38868294 socket gearmand 2020 gearman 30r FIFO 0,8 0t0 38868295 pipe gearmand 2020 gearman 31w FIFO 0,8 0t0 38868295 pipe gearmand 2020 gearman 32u IPv4 38868324 0t0 TCP localhost:4730->localhost:57954 (ESTABLISHED) gearmand 2020 gearman 33u IPv4 38868325 0t0 TCP localhost:4730->localhost:57955 (ESTABLISHED) gearmand 2020 gearman 34u IPv4 38901247 0t0 TCP localhost:4730->localhost:38594 (ESTABLISHED) gearmand 2020 gearman 35u IPv4 38868327 0t0 TCP localhost:4730->localhost:57957 (ESTABLISHED) gearmand 2020 gearman 36u IPv4 38867483 0t0 TCP localhost:4730->localhost:57959 (ESTABLISHED) gearmand 2020 gearman 37u IPv4 38867484 0t0 TCP localhost:4730->localhost:57958 (ESTABLISHED) gearmand 2020 gearman 38u IPv4 38901248 0t0 TCP localhost:4730->localhost:38595 (CLOSE_WAIT) gearmand 2020 gearman 39u IPv4 38901249 0t0 TCP localhost:4730->localhost:38597 (ESTABLISHED) gearmand 2020 gearman 40u IPv4 38869201 0t0 TCP localhost:4730->localhost:57979 (ESTABLISHED) gearmand 2020 gearman 41u IPv4 38900437 0t0 TCP localhost:4730->localhost:38599 (ESTABLISHED) gearmand 2020 gearman 42u IPv4 38900438 0t0 TCP localhost:4730->localhost:38602 (ESTABLISHED) gearmand 2020 gearman 43u IPv4 38868375 0t0 TCP localhost:4730->localhost:57987 (ESTABLISHED) gearmand 2020 gearman 44u IPv4 38900468 0t0 TCP localhost:4730->localhost:38606 (CLOSE_WAIT) gearmand 2020 gearman 45u IPv4 38868381 0t0 TCP localhost:4730->localhost:57999 (ESTABLISHED) gearmand 2020 gearman 46u IPv4 38868388 0t0 TCP localhost:4730->localhost:58007 (ESTABLISHED) gearmand 2020 gearman 47u IPv4 38868393 0t0 TCP localhost:4730->localhost:58011 (ESTABLISHED) gearmand 2020 gearman 48u IPv4 38903950 0t0 TCP localhost:4730->localhost:38609 (ESTABLISHED) gearmand 2020 gearman 49u IPv4 38870276 0t0 TCP localhost:4730->localhost:58019 (ESTABLISHED) gearmand 2020 gearman 50u IPv4 38903955 0t0 TCP localhost:4730->localhost:38613 (ESTABLISHED) gearmand 2020 gearman 51u IPv4 38900477 0t0 TCP localhost:4730->localhost:38617 (CLOSE_WAIT) gearmand 2020 gearman 52u IPv4 38867630 0t0 TCP localhost:4730->localhost:58031 (ESTABLISHED) gearmand 2020 gearman 53u IPv4 38867633 0t0 TCP localhost:4730->localhost:58035 (ESTABLISHED) gearmand 2020 gearman 54u IPv4 38867636 0t0 TCP localhost:4730->localhost:58039 (ESTABLISHED) gearmand 2020 gearman 55u IPv4 38900536 0t0 TCP localhost:4730->localhost:38619 (ESTABLISHED) gearmand 2020 gearman 56u IPv4 38868419 0t0 TCP localhost:4730->localhost:58047 (ESTABLISHED) gearmand 2020 gearman 57u IPv4 38869263 0t0 TCP localhost:4730->localhost:58051 (ESTABLISHED) gearmand 2020 gearman 58u IPv4 38900537 0t0 TCP localhost:4730->localhost:38621 (ESTABLISHED) gearmand 2020 gearman 59u IPv4 38869271 0t0 TCP localhost:4730->localhost:58059 (ESTABLISHED) gearmand 2020 gearman 60u IPv4 38900538 0t0 TCP localhost:4730->localhost:38623 (ESTABLISHED) gearmand 2020 gearman 61u IPv4 38870319 0t0 TCP localhost:4730->localhost:58067 (ESTABLISHED) gearmand 2020 gearman 62u IPv4 38900540 0t0 TCP localhost:4730->localhost:38628 (ESTABLISHED) gearmand 2020 gearman 63u IPv4 38869289 0t0 TCP localhost:4730->localhost:58075 (ESTABLISHED) ... gearmand 2020 gearman 2229u IPv4 38903885 0t0 TCP localhost:4730->localhost:38572 (ESTABLISHED) gearmand 2020 gearman 2230u IPv4 38901211 0t0 TCP localhost:4730->localhost:38576 (ESTABLISHED) gearmand 2020 gearman 2234u IPv4 38901237 0t0 TCP localhost:4730->localhost:38588 (ESTABLISHED)

  Read the article

• Compiling Gearman PHP Library for CentOS 5.8

  - by Andrew Ellis

  I've been trying to get Gearman compiled on CentOS 5.8 all afternoon. Unfortunately I am restricted to this version of CentOS by my CTO and how he has our entire network configured. I think it's simply because we don't have enough resources to upgrade our network... But anyways, the problem at hand. I have searched through Server Fault, Stack Overflow, Google, and am unable to locate a working solution. What I have below is stuff I have pieced together from my searching. Searches have told said to install the following via yum: yum -y install --enablerepo=remi boost141-devel libgearman-devel e2fsprogs-devel e2fsprogs gcc44 gcc-c++ To get the Boost headers working correctly I did this: cp -f /usr/lib/boost141/* /usr/lib/ cp -f /usr/lib64/boost141/* /usr/lib64/ rm -f /usr/include/boost ln -s /usr/include/boost141/boost /usr/include/boost With all of the dependancies installed and paths setup I then download and compile gearmand-1.1.2 just fine. wget -O /tmp/gearmand-1.1.2.tar.gz https://launchpad.net/gearmand/1.2/1.1.2/+download/gearmand-1.1.2.tar.gz cd /tmp && tar zxvf gearmand-1.1.2.tar.gz ./configure && make -j8 && make install That works correctly. So now I need to install the Gearman library for PHP. I have attempted through PECL and downloading the source directly, both result in the same error: checking whether to enable gearman support... yes, shared not found configure: error: Please install libgearman What I don't understand is I installed the libgearman-devel package which also installed the core libgearman. The installation installs libgearman-devel-0.14-3.el5.x86_64, libgearman-devel-0.14-3.el5.i386, libgearman-0.14-3.el5.x86_64, and libgearman-0.14-3.el5.i386. Is it possible the package version is lower than what is required? I'm still poking around with this, but figured I'd throw this up to see if anyone has a solution while I continue to research a fix. Thanks!

  Read the article

• Simulating Google Appengine's Task Queue with Gearman

  - by sotangochips

  One of the characteristics I love most about Google's Task Queue is its simplicity. More specifically, I love that it takes a URL and some parameters and then posts to that URL when the task queue is ready to execute the task. This structure means that the tasks are always executing the most current version of the code. Conversely, my gearman workers all run code within my django project -- so when I push a new version live, I have to kill off the old worker and run a new one so that it uses the current version of the code. My goal is to have the task queue be independent from the code base so that I can push a new live version without restarting any workers. So, I got to thinking: why not make tasks executable by url just like the google app engine task queue? The process would work like this: User request comes in and triggers a few tasks that shouldn't be blocking. Each task has a unique URL, so I enqueue a gearman task to POST to the specified URL. The gearman server finds a worker, passes the url and post data to a worker The worker simply posts to the url with the data, thus executing the task. Assume the following: Each request from a gearman worker is signed somehow so that we know it's coming from a gearman server and not a malicious request. Tasks are limited to run in less than 10 seconds (There would be no long tasks that could timeout) What are the potential pitfalls of such an approach? Here's one that worries me: The server can potentially get hammered with many requests all at once that are triggered by a previous request. So one user request might entail 10 concurrent http requests. I suppose I could have a single worker with a sleep before every request to rate-limit. Any thoughts?

  Read the article

• Compiling Gearman extension for PHP on OS-X Snow Leopard with Zend Server Community Edition

  - by GrumpyCanuck

  I was wondering if anyone else has figured out how to solve this problem. Whether I install the extension via PECL or compile it by hand, I get the same error: PHP Warning: PHP Startup: Unable to load dynamic library '/usr/local/zend/lib/php_extensions/gearman.so' - dlopen(/usr/local/zend/lib/php_extensions/gearman.so, 9): no suitable image found. Did find: /usr/local/zend/lib/php_extensions/gearman.so: mach-o, but wrong architecture in Unknown on line 0 I have no idea how to fix this problem and I can only bang my head into my desk so many times.

  Read the article

• Any way to access Gearman administration?

  - by webdestroya

  I want to be able to query a gearman server to determine how many instances of a worker I have running (basically I want to make sure that RunTaskA is available and RunTaskB is available if there are no workers handling those tasks, I want to be able to send an alert out. Is there any way to do this? Also: Mad props if you know of a PHP way to query the gearman server.

  Read the article

• gearman - what is it

  - by abson

  What is gearman and where and in what is it used?

  Read the article

• Gearman client doBackground always returns GEARMAN_TIMEOUT

  - by Ascherer

  So, ive got a simple gearman system running right now, with a worker running. The worker basically just takes the payload (a random number in this case, and is supposed to echo it back to the screen. Literally an echo, not returning to the client. The client sends the random number. Im trying to do a $client->doBackground( 'post', 65482, md5(uniqid())); but its coming back with a 47 error (GEARMAN_TIMEOUT) every time. getErrNo() returns 0, error() returns something about GEARMAN_TIMEOUT However, when i change it to just $client->do(blah, blah, blah), it works just fine. I've even occasionally seen it where the worker still echo's the number, even after getting the timeout error... public function execute() { $method = 'do'; if( !$this->getBlock() ) { $method .= ( $this->getPriority() == 'Normal' ? '' : $this->getPriority() ) . 'Background'; } else { $method .= $this->getPriority(); } echo "Method: $method \t Worker: {$this->getName()} \t Payload: {$this->getPayload()} \t Hash: {$this->getHash()}\n"; $this->setResult( $this->getClient() ->$method( $this->getName(), $this->getPayload(), $this->getHash() ) ); if( $this->getClient()->returnCode() != GEARMAN_SUCCESS ) { echo "Code: " . $this->getClient()->returnCode() . "\t" . GEARMAN_TIMEOUT . "\n"; } }

  Read the article

• Scheduling with gearman vs. cron?

  - by Industrial

  Hi everybody, I have noticed a lot of people discussing Gearman and it's scheduling features making it enable to distribute work onto other servers. However, I have not yet seen a comparison to native cronjobs. What are the differences between cron and Gearman?

  Read the article

• Gearman: Run both client and worker on same machine?

  - by Industrial

  Hi everybody, I just manage to install gearman on my local Ubuntu test environment. I wonder if I could in any way run both the worker and the client at the same computer, while in development stage? Thanks a lot!

  Read the article

• Proper way of utilizing gearman with my php application.

  - by luckytaxi

  For now I just want to use Gearman for background processing. For example, I need to email a recipient that they have a private message waiting for them once the sender submits their message into the DB. I assume I can run the worker/client and server on my primary server but I have no problems offloading some of the tasks to a different web server. Anyways, my question is how do I handle multiple "functions?" Let's say I need a job that handles the email portion and a job to handle image manipulation. Can I have multiple functions in the worker? I've followed a couple of examples I found online but each example only shows one function being initialized. Do I have to start up multiple "workers" to handle multiple functions?

  Read the article

• How to properly use Gearman with my PHP application?

  - by luckytaxi

  For now I just want to use Gearman for background processing. For example, I need to email a recipient that they have a private message waiting for them once the sender submits their message into the DB. I assume I can run the worker/client and server on my primary server but I have no problems offloading some of the tasks to a different web server. Anyways, my question is how do I handle multiple "functions?" Let's say I need a job that handles the email portion and a job to handle image manipulation. Can I have multiple functions in the worker? I've followed a couple of examples I found online but each example only shows one function being initialized. Do I have to start up multiple "workers" to handle multiple functions?

  Read the article

• Python Daemon Subprocess not working at boot

  - by Adam Richardson

  I am attempting to write a python daemon that will launch at boot. The goal of the script is to receive a job from our gearman load balancing server and complete the job. I am using the python-daemon module from pypi (http://pypi.python.org/pypi/python-daemon/). The nature of the job that it is completing is converting images in the orf (olympus raw image format) to jpeg. In order to accomplish this an outside program is used, ufraw in this case. The problem comes in when I start the daemon at boot, if I launch from the shell it runs perfectly and completes the work. When it starts at boot it is unable to launch the subprocess command. commandString = '/usr/bin/ufraw-batch --interpolation=four-color --wb=camera --compression=100 --output="' + outfile + '" --out-type=jpg --overwrite "' + infile + '"' args = shlex.split(commandString) process = subprocess.Popen(args).wait() I am not sure what I am doing wrong. Thanks for any help.

  Read the article

• How can I best implement 'cache until further notice' with memcache in multiple tiers?

  - by ajreal

  the term "client" used here is not referring to client's browser, but client server Before cache workflow 1. client make a HTTP request --> 2. server process --> 3. store parsed results into memcache for next use (cache indefinitely) --> 4. return results to client --> 5. client get the result, store into client's local memcache with TTL After cache workflow 1. another client make a HTTP request --> 2. memcache found return memcache results to client --> 3. client get the result, store into client's local memcache with TTL TTL = time to live Is possible for me to know when the data was updated, and to expire relevant memcache(s) accordingly. However, the pitfalls on client site cache TTL Any data update before the TTL is not pick-up by client memcache. In reverse manner, where there is no update, client memcache still expire after the TTL First request (or concurrent requests) after cache TTL will get throttle as it need to repeat the "Before cache workflow" In the event where client require several HTTP requests on a single web page, it could be very bad in performance. Ideal solution should be client to cache indefinitely until further notice. Here are the three proposals about futher notice Proposal 1 : Make use on HTTP header (current implementation) 1. client sent HTTP request last modified time header 2. server check if last data modified time=last cache time return status 304 3. client based on header to decide further processing GOOD? ---- - save some parsing for client - lesser data transfer BAD? ---- - fire a HTTP request is still slow - server end still need to process lots of requests Proposal 2 : Consistently issue a HTTP request to check all data group last modified time 1. client fire a HTTP request 2. server to return last modified time for all data group 3. client compare local last cache time with the result 4. if data group last cache time < server last modified time then request again for that data group only GOOD? ---- - only fetch what is no up-to-date - less requests for server BAD? ---- - every web page require a HTTP request Proposal 3 : Tell client when new data is available (Push) 1. when server end notice there is a change on a data group 2. notify clients on the changes 3. help clients to fetch again data 4. then reset client local memcache after data is parsed GOOD? ---- - let the cache act/behave like a true cache BAD? ---- - encourage race condition My preference is on proposal 3, and something like Gearman could be ideal Where there is a change, Gearman server to sent the task to multiple clients (workers). Am I crazy? (I know my first question is a bit crazy)

  Read the article

• Facebook like on demand meta content scraper

  - by Tobias

  you guys ever saw that FB scrapes the link you post on facebook (status, message etc.) live right after you paste it in the link field and displays various metadata, a thumb of the image, various images from the a page link or a video thumb from a video related link (like youtube). any ideas how one would copy this function? i'm thinking about a couple gearman workers or even better just javascript that does a xhr requests and parses the content based on regex's or something similar... any ideas? any links? did someone already tried to do the same and wrapped it in a nice class? anything? :) thanks!

  Read the article

• Best Work Queue service for distributed clusters

  - by onewheelgood

  Hi there. I require a simple work queue type system for asynchronous task management. I have looked at both beanstalkd and gearman. However, both these seem to assume that the client and the queue server are on the same network, and therefore that there will always be a reliable network between them. I need one that can support the client and server being in different places in the world, and be able to manage temporary loss of network connection between clusters. Ideally, this would work in such a way where I post a job to a local proxy that attempts to send it to the main queue server. If there is no network connection, it would try again later, however it would not lose the job or delay the client. Any recommendations?

  Read the article

• Using NFS for scalable PHP/MySQL web application

  - by Jeroen Moons

  Here's the situation: I have a PHP/MySQL web application that accepts user uploads (pdf files). From these pdf files' pages a preview image is made on the fly and presented to the web app's users. Some pdfs might be on the large side, most will be under 50 MB but some extreme cases could be as large as a few hundred MB. A little waiting for the preview image for large pdf files is acceptable but no more than a minute let's say. Everything is running on one server for now, but soon the app will hit the server's limit on both storage and processing power. My idea to solve the problem: To deal with this situation I had the idea of having one or more pdf processing servers as needed, and one or more file storage servers. These two types of servers are mounted to the server on which the actual app runs using NFS. The app could then use GearMan to delegate pdf processing tasks to these processing servers. The processing server can mount the storage server and read the file stored there, process it and write its output to that server. The servers I'm talking about will be amazon ec2 instances. The web app returns a link to the resulting pdf preview image on the storage server that was used which can then be used on the front end to show the image to the user. My question: I have zero experience with apps that use multiple servers, is this idea viable or is there a better way to do it? Is an NFS setup fast and reliable enough for this situation?

  Read the article

1