Detecting source of memory usage on a Linux box
- by apeace
I have a toy Linux box with 256mb RAM running Ubuntu 10.04.1 LTS. Here is the output of free -m:
total used free shared buffers cached
Mem: 245 122 122 0 19 64
-/+ buffers/cache: 38 206
Swap: 511 0 511
Unless I'm reading this wrong, 122mb is being used and only 84mb of that is disk cache. Here are all processes I'm running sorted by memory usage (ps -eo pmem,pcpu,rss,vsize,args | sort -k 1 -r):
%MEM %CPU RSS VSZ COMMAND
5.0 0.0 12648 633140 node /home/node/main/sites.js
1.5 0.0 3884 251736 /usr/sbin/console-kit-daemon --no-daemon
1.3 0.0 3328 77108 sshd: apeace [priv]
0.9 0.0 2344 19624 -bash
0.7 0.0 1776 23620 /sbin/init
0.6 0.0 1624 77108 sshd: apeace@pts/0
0.6 0.0 1544 9940 redis-server /etc/redis/redis.conf
0.6 0.0 1524 25848 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 103:105
0.5 0.0 1324 119880 rsyslogd -c4
0.4 0.0 1084 49308 /usr/sbin/sshd
0.4 0.0 1028 44376 /usr/sbin/exim4 -bd -q30m
0.3 0.0 904 6876 ps -eo pmem,pcpu,rss,vsize,args
0.3 0.0 888 21124 cron
0.3 0.0 868 23472 dbus-daemon --system --fork
0.2 0.0 732 19624 -bash
0.2 0.0 628 6128 /sbin/getty -8 38400 tty1
0.2 0.0 628 16952 upstart-udev-bridge --daemon
0.2 0.0 564 16800 udevd --daemon
0.2 0.0 552 16796 udevd --daemon
0.2 0.0 548 16796 udevd --daemon
0.0 0.0 0 0 [xenwatch]
0.0 0.0 0 0 [xenbus]
0.0 0.0 0 0 [sync_supers]
0.0 0.0 0 0 [netns]
0.0 0.0 0 0 [migration/3]
0.0 0.0 0 0 [migration/2]
0.0 0.0 0 0 [migration/1]
0.0 0.0 0 0 [migration/0]
0.0 0.0 0 0 [kthreadd]
0.0 0.0 0 0 [kswapd0]
0.0 0.0 0 0 [kstriped]
0.0 0.0 0 0 [ksoftirqd/3]
0.0 0.0 0 0 [ksoftirqd/2]
0.0 0.0 0 0 [ksoftirqd/1]
0.0 0.0 0 0 [ksoftirqd/0]
0.0 0.0 0 0 [ksnapd]
0.0 0.0 0 0 [kseriod]
0.0 0.0 0 0 [kjournald]
0.0 0.0 0 0 [khvcd]
0.0 0.0 0 0 [khelper]
0.0 0.0 0 0 [kblockd/3]
0.0 0.0 0 0 [kblockd/2]
0.0 0.0 0 0 [kblockd/1]
0.0 0.0 0 0 [kblockd/0]
0.0 0.0 0 0 [flush-202:1]
0.0 0.0 0 0 [events/3]
0.0 0.0 0 0 [events/2]
0.0 0.0 0 0 [events/1]
0.0 0.0 0 0 [events/0]
0.0 0.0 0 0 [crypto/3]
0.0 0.0 0 0 [crypto/2]
0.0 0.0 0 0 [crypto/1]
0.0 0.0 0 0 [crypto/0]
0.0 0.0 0 0 [cpuset]
0.0 0.0 0 0 [bdi-default]
0.0 0.0 0 0 [async/mgr]
0.0 0.0 0 0 [aio/3]
0.0 0.0 0 0 [aio/2]
0.0 0.0 0 0 [aio/1]
0.0 0.0 0 0 [aio/0]
Now, I know that ps is not the best for viewing process memory usage, but that's because it tends to report more memory than is actually being used...meaning no matter how you look at it all my processes combined shouldn't be using near 122mb, even if you account for the disk cache.
What's more, memory usage is growing all the time. I've had to restart my server once a week, because once my 256mb fills up it starts swapping, which it wouldn't do just for disk cache. Shouldn't there be some way for me to see the culprit?!
I'm new to server admin, so please if there's something obvious I'm overlooking point it out to me.
Just for good measure, the output of cat /proc/meminfo:
MemTotal: 251140 kB
MemFree: 124604 kB
Buffers: 20536 kB
Cached: 66136 kB
SwapCached: 0 kB
Active: 65004 kB
Inactive: 37576 kB
Active(anon): 15932 kB
Inactive(anon): 164 kB
Active(file): 49072 kB
Inactive(file): 37412 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 524284 kB
SwapFree: 524284 kB
Dirty: 8 kB
Writeback: 0 kB
AnonPages: 15916 kB
Mapped: 10668 kB
Shmem: 188 kB
Slab: 18604 kB
SReclaimable: 10088 kB
SUnreclaim: 8516 kB
KernelStack: 536 kB
PageTables: 1444 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 649852 kB
Committed_AS: 64224 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 752 kB
VmallocChunk: 34359737600 kB
DirectMap4k: 262144 kB
DirectMap2M: 0 kB
EDIT: I had misinterpreted the meaning of free -m at first. But even so: the important thing is that my OS eventually begins to use swap memory if I don't restart my server, which disk caching wouldn't do. So where do I look to see what is using all this memory?
