How to understand the memory usage and load average in linux server
- by Tim
Hi,
I am using a linux server which has 128GB of memory and 24 cores. I use top to see how much it is used. Its output is pasted at the end of the post. Here are two questions:
(1) I see that each of the running processes occupies a very small percentage of memory (%MEM no more than 0.2%, and most just 0.0%), but how the total memory is almost used as in the fourth line of output ("Mem: 130766620k total, 130161072k used, 605548k free, 919300k buffers")? The sum of used percentage of memory over all processes seems unlikely to achieve almost 100%, doesn't it?
(2) how to understand the load average on the first line ("load average: 14.04, 14.02, 14.00")?
Thanks and regards!
Edit:
Thanks!
I also really like to hear some rough numbers based on used percentage of memory to determine if a server is heavily loaded, since I once became the one who cramed the server without understanding the current load.
Is swap regarded as almost the same as memory? For example, when memory and swap are almost of same size, if the memory is almost running out but the swap is still largely free, may I just view it as if the used percentage of memory + swap is still not high and run other new processes?
How would you consider together CPU or memory (or memory + swap) usage? Do you become worried if either of them reaches too high or both?
Output of top:
$ top
top - 12:45:33 up 19 days, 23:11, 18 users, load average: 14.04, 14.02, 14.00
Tasks: 484 total, 12 running, 472 sleeping, 0 stopped, 0 zombie
Cpu(s): 36.7%us, 19.7%sy, 0.0%ni, 43.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 130766620k total, 130161072k used, 605548k free, 919300k buffers
Swap: 63111312k total, 500556k used, 62610756k free, 124437752k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6529 sanchez 18 -2 1075m 219m 13m S 100 0.2 13760:23 MATLAB
13210 timothy 18 -2 48336 37m 1216 R 100 0.0 3:56.75 absurdity
13888 timothy 18 -2 48336 37m 1204 R 100 0.0 2:04.89 absurdity
14542 timothy 18 -2 48336 37m 1196 R 100 0.0 1:08.34 absurdity
14544 timothy 18 -2 2888 2076 400 R 100 0.0 1:06.14 gatherData
6183 sanchez 18 -2 1133m 195m 13m S 100 0.2 13676:04 MATLAB
6795 sanchez 18 -2 1079m 210m 13m S 100 0.2 13734:26 MATLAB
10178 timothy 18 -2 48336 37m 1204 R 100 0.0 11:33.93 absurdity
12438 timothy 18 -2 48336 37m 1216 R 100 0.0 5:38.17 absurdity
13661 timothy 18 -2 48336 37m 1216 R 100 0.0 2:44.13 absurdity
14098 timothy 18 -2 48336 37m 1204 R 100 0.0 1:58.31 absurdity
14335 timothy 18 -2 48336 37m 1196 R 100 0.0 1:08.93 absurdity
14765 timothy 18 -2 48336 37m 1196 R 99 0.0 0:32.57 absurdity
13445 timothy 18 -2 48336 37m 1216 R 99 0.0 3:01.37 absurdity
28990 root 20 0 0 0 0 S 2 0.0 65:50.21 pdflush
12141 tim 18 -2 19380 1660 1024 R 1 0.0 0:04.04 top
1240 root 15 -5 0 0 0 S 0 0.0 16:07.11 kjournald
9019 root 20 0 296m 4460 2616 S 0 0.0 82:19.51 kdm_greet
1 root 20 0 4028 728 592 S 0 0.0 0:03.11 init
2 root 15 -5 0 0 0 S 0 0.0 0:00.00 kthreadd
3 root RT -5 0 0 0 S 0 0.0 0:01.01 migration/0
4 root 15 -5 0 0 0 S 0 0.0 0:08.13 ksoftirqd/0
5 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/0
6 root RT -5 0 0 0 S 0 0.0 17:27.31 migration/1
7 root 15 -5 0 0 0 S 0 0.0 0:01.21 ksoftirqd/1
8 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/1
9 root RT -5 0 0 0 S 0 0.0 10:02.56 migration/2
10 root 15 -5 0 0 0 S 0 0.0 0:00.34 ksoftirqd/2
11 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/2
12 root RT -5 0 0 0 S 0 0.0 4:29.53 migration/3
13 root 15 -5 0 0 0 S 0 0.0 0:00.34 ksoftirqd/3