Celery - minimize memory consuption

Posted by Andrew on Server Fault See other posts from Server Fault or by Andrew
Published on 2010-12-03T14:29:16Z Indexed on 2011/06/28 8:24 UTC
Read the original article Hit count: 305

Filed under:

python

|

memory-usage

|

profiling

We have ~300 celeryd processes running under Ubuntu 10.4 64-bit , in idle every process takes ~19mb RES, ~174mb VIRT, thus - it's around 6GB of RAM in idle for all processes. In active state - process takes up to 100mb of RES and ~300mb VIRT

Every process uses minidom(xml files are < 500kb, simple structure) and urllib.

Quetions is - how can we decrease RAM consuption - at least for idle workers, probably some celery or python options may help? How to determine which part takes most of memory?

© Server Fault or respective owner

Related posts about python

unmet dependencies in Ubuntu 12.04

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I tried today to install a dvb-card on my Ubuntu 12.04 (Linux blauhai-linux 3.2.0-25-generic #40-Ubuntu SMP Wed May 23 20:30:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux ). The installation failed with an error. After that, i tried to install python (it was already installed but i got this error): linux:~$… >>> More
How can I get sikuli-ide to work?

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I installed sikuli-ide with sudo apt-get install sikuli-ide Everything was fine until I tried to start it from the terminal. I typed sikuli-ide But the only response I got was [info] locale: en_US The application was not started, furthermore there is no desktop file and sikuli-ide does not… >>> More
Getting PATH right for python after MacPorts install

as seen on Super User - Search for 'Super User'
I can't import some python libraries (PIL, psycopg2) that I just installed with MacPorts. I looked through these forums, and tried to adjust my PATH variable in $HOME/.bash_profile in order to fix this but it did not work. I added the location of PIL and psycopg2 to PATH. I know that Terminal is… >>> More
call python with system() in R to run a python script emulating the python console

as seen on Stack Overflow - Search for 'Stack Overflow'
I want to pass a chunk of Python code to Python in R with something like system('python ...'), and I'm wondering if there is an easy way to emulate the python console in this case. For example, suppose the code is "print 'hello world'", how can I get the output like this in R? >>> print… >>> More
Python - Calling a non python program from python?

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, I am currently struggling to call a non python program from a python script. I have a ~1000 files that when passed through this C++ program will generate ~1000 outputs. Each output file must have a distinct name. The command I wish to run is of the form: program_name -input -output -o1 -o2… >>> More

Related posts about memory-usage

SQL Server 2005 high memory usage and performance problems

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi there guys. I have this ASP.NET/SQLServer2005 website running on a production server (Win2003, QuadCore, 4GB). The site runs smoothly normally, but after 2-3 weeks I notice a slow performance on the site (especifically in one particular page). Also I notice that the SQL Server process is using… >>> More
Coldfusion on VPS, how much JVM heap memory?

as seen on Server Fault - Search for 'Server Fault'
Recently I got a VPS server and I'm running Coldfusion, the website was running fine until it got more and more traffic and I started to encounter 'OutOfMemory' exceptions. I thought simply to rise the memory of the VPS server, but this didn't help. After doing some Google searches I found a setting… >>> More
Design approach, string table data, variables, stl memory usage

as seen on Stack Overflow - Search for 'Stack Overflow'
I have an old structure class like this: typedef vector<vector<string>> VARTYPE_T; which works as a single variable. This variable can hold from one value over a list to data like a table. Most values are long,double, string or double [3] for coordinates (x,y,z). I just convert them as… >>> More
Normal memory usage in Rails

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm wondering how much memory usage is normal for a ruby process in a rails application? I really need something to benchmark against. In my dev environment WEBrick a single ruby process uses about 61mb to handle 10 simultaneous requests going non stop. In my prod environment Apache2+Passenger starts… >>> More
C++ memory usage

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi. I have pointer str: char * str = new char[10]; I'm using this memory for store data. How can I allocate more bytes for str and don't lost old data from str? >>> More