I have a custom built Ubuntu 11.04 server with a 6 disk software RAID 10 primary drive. On it I'm primarily running a PostgreSQL and a few other utilities that stream data from the web. I often find after a few hours of uptime the server starts to lag with all kinds of processes. For example, it may take 10-15 seconds after log-in to get a shell prompt. It might take 5-10 seconds for top to come up. An ls might take a second or two.
When I look at top there is almost no CPU usage. There's a fair amount of memory used by the PostgreSQL server but not enough to bleed into swap.
I have no idea where to go from here, other than to suspect the RAID10 (I've only ever had software RAID 1's before).
Edit: Output from top:
top - 11:56:03 up 1:46, 3 users, load average: 0.89, 0.73, 0.72
Tasks: 119 total, 1 running, 118 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.2%us, 0.0%sy, 0.0%ni, 93.5%id, 6.2%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 16325596k total, 3478248k used, 12847348k free, 20880k buffers
Swap: 19534176k total, 0k used, 19534176k free, 3041992k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1747 woodsp 20 0 109m 10m 4888 S 1 0.1 0:42.70 python
357
root 20 0 0 0 0 S 0 0.0 0:00.40 jbd2/sda3-8
1
root 20 0 24324 2284 1344 S 0 0.0 0:00.84 init
2
root 20 0 0 0 0 S 0 0.0 0:00.00 kthreadd
3
root 20 0 0 0 0 S 0 0.0 0:00.24 ksoftirqd/0
6
root RT 0 0 0 0 S 0 0.0 0:00.00 migration/0
7
root RT 0 0 0 0 S 0 0.0 0:00.01 watchdog/0
8
root RT 0 0 0 0 S 0 0.0 0:00.00 migration/1
10
root 20 0 0 0 0 S 0 0.0 0:00.02 ksoftirqd/1
12
root RT 0 0 0 0 S 0 0.0 0:00.01 watchdog/1
13
root RT 0 0 0 0 S 0 0.0 0:00.00 migration/2
14
root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/2:0
15
root 20 0 0 0 0 S 0 0.0 0:00.00 ksoftirqd/2
16
root RT 0 0 0 0 S 0 0.0 0:00.01 watchdog/2
17
root RT 0 0 0 0 S 0 0.0 0:00.00 migration/3
18
root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/3:0
19
root 20 0 0 0 0 S 0 0.0 0:00.02 ksoftirqd/3
20
root RT 0 0 0 0 S 0 0.0 0:00.01 watchdog/3
21
root 0 -20 0 0 0 S 0 0.0 0:00.00 cpuset
22
root 0 -20 0 0 0 S 0 0.0 0:00.00 khelper
23
root 20 0 0 0 0 S 0 0.0 0:00.00 kdevtmpfs
24
root 0 -20 0 0 0 S 0 0.0 0:00.00 netns
26
root 20 0 0 0 0 S 0 0.0 0:00.00 sync_supers
df -h
rpsharp@ncp-skookum:~$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 1.8T 549G 1.2T 32% /
udev 7.8G 4.0K 7.8G 1% /dev
tmpfs 3.2G 492K 3.2G 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 7.8G 0 7.8G 0% /run/shm
/dev/sda2 952M 128K 952M 1% /boot/efi
/dev/md0 5.5T 562G 4.7T 11% /usr/local
free -m
psharp@ncp-skookum:~$ free -m
total used free shared buffers cached
Mem: 15942 3409 12533 0 20 2983
-/+ buffers/cache: 405 15537
Swap: 19076 0 19076
tail -50 /var/log/syslog
Jul 3 06:31:32 ncp-skookum rsyslogd: [origin software="rsyslogd" swVersion="5.8.6" x-pid="1070" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Jul 3 06:39:01 ncp-skookum CRON[14211]: (root) CMD ( [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) ! -execdir fuser -s {} 2>/dev/null \; -delete)
Jul 3 06:40:01 ncp-skookum CRON[14223]: (smmsp) CMD (test -x /etc/init.d/sendmail && /usr/share/sendmail/sendmail cron-msp)
Jul 3 07:00:01 ncp-skookum CRON[14328]: (woodsp) CMD (/home/woodsp/bin/mail_tweetupdate # email an update)
Jul 3 07:00:01 ncp-skookum CRON[14327]: (smmsp) CMD (test -x /etc/init.d/sendmail && /usr/share/sendmail/sendmail cron-msp)
Jul 3 07:00:28 ncp-skookum sendmail[14356]: q63E0SoZ014356: from=woodsp, size=2328, class=0, nrcpts=2, msgid=<
[email protected]>, relay=woodsp@localhost
Jul 3 07:00:29 ncp-skookum sm-mta[14357]: q63E0Si6014357: from=<
[email protected]>, size=2569, class=0, nrcpts=2, msgid=<
[email protected]>, proto=ESMTP, daemon=MTA-v4, relay=localhost [127.0.0.1]
Jul 3 07:00:29 ncp-skookum sendmail[14356]: q63E0SoZ014356: to=Spencer Wood <
[email protected]>,Martin Lacayo <
[email protected]>, ctladdr=woodsp (1004/1005), delay=00:00:01, xdelay=00:00:01, mailer=relay, pri=62328, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (q63E0Si6014357 Message accepted for delivery)
Jul 3 07:00:29 ncp-skookum sm-mta[14359]: STARTTLS=client, relay=mx3.stanford.edu., version=TLSv1/SSLv3, verify=FAIL, cipher=DHE-RSA-AES256-SHA, bits=256/256
Jul 3 07:00:29 ncp-skookum sm-mta[14359]: q63E0Si6014357: to=<
[email protected]>,<
[email protected]>, ctladdr=<
[email protected]> (1004/1005), delay=00:00:01, xdelay=00:00:00, mailer=esmtp, pri=152569, relay=mx3.stanford.edu. [171.67.219.73], dsn=2.0.0, stat=Sent (Ok: queued as 8F3505802AC)
Jul 3 07:09:08 ncp-skookum CRON[14396]: (root) CMD ( [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) ! -execdir fuser -s {} 2>/dev/null \; -delete)
Jul 3 07:17:01 ncp-skookum CRON[14438]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jul 3 07:20:01 ncp-skookum CRON[14453]: (smmsp) CMD (test -x /etc/init.d/sendmail && /usr/share/sendmail/sendmail cron-msp)
Jul 3 07:39:01 ncp-skookum CRON[14551]: (root) CMD ( [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) ! -execdir fuser -s {} 2>/dev/null \; -delete)
Jul 3 07:40:01 ncp-skookum CRON[14562]: (smmsp) CMD (test -x /etc/init.d/sendmail && /usr/share/sendmail/sendmail cron-msp)
Jul 3 08:00:01 ncp-skookum CRON[14668]: (smmsp) CMD (test -x /etc/init.d/sendmail && /usr/share/sendmail/sendmail cron-msp)
Jul 3 08:09:01 ncp-skookum CRON[14724]: (root) CMD ( [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) ! -execdir fuser -s {} 2>/dev/null \; -delete)
Jul 3 08:17:01 ncp-skookum CRON[14766]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jul 3 08:20:01 ncp-skookum CRON[14781]: (smmsp) CMD (test -x /etc/init.d/sendmail && /usr/share/sendmail/sendmail cron-msp)
Jul 3 08:39:01 ncp-skookum CRON[14881]: (root) CMD ( [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) ! -execdir fuser -s {} 2>/dev/null \; -delete)
Jul 3 08:40:01 ncp-skookum CRON[14892]: (smmsp) CMD (test -x /etc/init.d/sendmail && /usr/share/sendmail/sendmail cron-msp)
Output of hdparm -t /dev/sd{a,b,c,d,e,f} This looks suspicious?
/dev/sda:
Timing buffered disk reads: 2 MB in 4.84 seconds = 423.39 kB/sec
/dev/sdb:
Timing buffered disk reads: 420 MB in 3.01 seconds = 139.74 MB/sec
/dev/sdc:
Timing buffered disk reads: 390 MB in 3.00 seconds = 129.87 MB/sec
/dev/sdd:
Timing buffered disk reads: 416 MB in 3.00 seconds = 138.51 MB/sec
/dev/sde:
Timing buffered disk reads: 422 MB in 3.00 seconds = 140.50 MB/sec
/dev/sdf:
Timing buffered disk reads: 416 MB in 3.01 seconds = 138.26 MB/sec
