Search Results

Search found 138 results on 6 pages for 'shr deadman'.

Page 6/6 | < Previous Page | 2 3 4 5 6 

• e2fsck extremly slow, although enough memory exists

  - by kaefert

  I've got this external USB-Disk: kaefert@blechmobil:~$ lsusb -s 2:3 Bus 002 Device 003: ID 0bc2:3320 Seagate RSS LLC As can be seen in this dmesg output, there are some problems that prevents that disk from beeing mounted: kaefert@blechmobil:~$ dmesg | grep sdb [ 114.474342] sd 5:0:0:0: [sdb] 732566645 4096-byte logical blocks: (3.00 TB/2.72 TiB) [ 114.475089] sd 5:0:0:0: [sdb] Write Protect is off [ 114.475092] sd 5:0:0:0: [sdb] Mode Sense: 43 00 00 00 [ 114.475959] sd 5:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 114.477093] sd 5:0:0:0: [sdb] 732566645 4096-byte logical blocks: (3.00 TB/2.72 TiB) [ 114.501649] sdb: sdb1 [ 114.502717] sd 5:0:0:0: [sdb] 732566645 4096-byte logical blocks: (3.00 TB/2.72 TiB) [ 114.504354] sd 5:0:0:0: [sdb] Attached SCSI disk [ 116.804408] EXT4-fs (sdb1): ext4_check_descriptors: Checksum for group 3976 failed (47397!=61519) [ 116.804413] EXT4-fs (sdb1): group descriptors corrupted! So I went and fired up my favorite partition manager - gparted, and told it to verify and repair the partition sdb1. This made gparted call e2fsck (version 1.42.4 (12-Jun-2012)) e2fsck -f -y -v /dev/sdb1 Although gparted called e2fsck with the "-v" option, sadly it doesn't show me the output of my e2fsck process (bugreport https://bugzilla.gnome.org/show_bug.cgi?id=467925 ) I started this whole thing on Sunday (2012-11-04_2200) evening, so about 48 hours ago, this is what htop says about it now (2012-11-06-1900): PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command 3704 root 39 19 1560M 1166M 768 R 98.0 19.5 42h56:43 e2fsck -f -y -v /dev/sdb1 Now I found a few posts on the internet that discuss e2fsck running slow, for example: http://gparted-forum.surf4.info/viewtopic.php?id=13613 where they write that its a good idea to see if the disk is just that slow because maybe its damaged, and I think these outputs tell me that this is not the case in my case: kaefert@blechmobil:~$ sudo hdparm -tT /dev/sdb /dev/sdb: Timing cached reads: 3562 MB in 2.00 seconds = 1783.29 MB/sec Timing buffered disk reads: 82 MB in 3.01 seconds = 27.26 MB/sec kaefert@blechmobil:~$ sudo hdparm /dev/sdb /dev/sdb: multcount = 0 (off) readonly = 0 (off) readahead = 256 (on) geometry = 364801/255/63, sectors = 5860533160, start = 0 However, although I can read quickly from that disk, this disk speed doesn't seem to be used by e2fsck, considering tools like gkrellm or iotop or this: kaefert@blechmobil:~$ iostat -x Linux 3.2.0-2-amd64 (blechmobil) 2012-11-06 _x86_64_ (2 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 14,24 47,81 14,63 0,95 0,00 22,37 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0,59 8,29 2,42 5,14 43,17 160,17 53,75 0,30 39,80 8,72 54,42 3,95 2,99 sdb 137,54 5,48 9,23 0,20 587,07 22,73 129,35 0,07 7,70 7,51 16,18 2,17 2,04 Now I researched a little bit on how to find out what e2fsck is doing with all that processor time, and I found the tool strace, which gives me this: kaefert@blechmobil:~$ sudo strace -p3704 lseek(4, 41026998272, SEEK_SET) = 41026998272 write(4, "\212\354K[_\361\3nl\212\245\352\255jR\303\354\312Yv\334p\253r\217\265\3567\325\257\3766"..., 4096) = 4096 lseek(4, 48404766720, SEEK_SET) = 48404766720 read(4, "\7t\260\366\346\337\304\210\33\267j\35\377'\31f\372\252\ffU\317.y\211\360\36\240c\30`\34"..., 4096) = 4096 lseek(4, 41027002368, SEEK_SET) = 41027002368 write(4, "\232]7Ws\321\352\t\1@[+5\263\334\276{\343zZx\352\21\316`1\271[\202\350R`"..., 4096) = 4096 lseek(4, 48404770816, SEEK_SET) = 48404770816 read(4, "\17\362r\230\327\25\346//\210H\v\311\3237\323K\304\306\361a\223\311\324\272?\213\tq \370\24"..., 4096) = 4096 lseek(4, 41027006464, SEEK_SET) = 41027006464 write(4, "\367yy>x\216?=\324Z\305\351\376&\25\244\210\271\22\306}\276\237\370(\214\205G\262\360\257#"..., 4096) = 4096 lseek(4, 48404774912, SEEK_SET) = 48404774912 read(4, "\365\25\0\21|T\0\21}3t_\272\373\222k\r\177\303\1\201\261\221$\261B\232\3142\21U\316"..., 4096) = 4096 ^CProcess 3704 detached around 16 of these lines every second, so 4 read and 4 write operations every second, which I don't consider to be a lot.. And finally, my question: Will this process ever finish? If those numbers from fseek (48404774912) represent bytes, that would be something like 45 gigabytes, with this beeing a 3 terrabyte disk, which would give me 134 days to go, if the speed stays constant, and he scans the disk like this completly and only once. Do you have some advice for me? I have most of the data on that disk elsewhere, but I've put a lot of hours into sorting and merging it to this disk, so I would prefer to getting this disk up and running again, without formatting it anew. I don't think that the hardware is damaged since the disk is only a few months and since I can't see any I/O errors in the dmesg output. UPDATE: I just looked at the strace output again (2012-11-06_2300), now it looks like this: lseek(4, 1419860611072, SEEK_SET) = 1419860611072 read(4, "3#\f\2447\335\0\22A\355\374\276j\204'\207|\217V|\23\245[\7VP\251\242\276\207\317:"..., 4096) = 4096 lseek(4, 43018145792, SEEK_SET) = 43018145792 write(4, "]\206\231\342Y\204-2I\362\242\344\6R\205\361\324\177\265\317C\334V\324\260\334\275t=\10F."..., 4096) = 4096 lseek(4, 1419860615168, SEEK_SET) = 1419860615168 read(4, "\262\305\314Y\367\37x\326\245\226\226\320N\333$s\34\204\311\222\7\315\236\336\300TK\337\264\236\211n"..., 4096) = 4096 lseek(4, 43018149888, SEEK_SET) = 43018149888 write(4, "\271\224m\311\224\25!I\376\16;\377\0\223H\25Yd\201Y\342\r\203\271\24eG<\202{\373V"..., 4096) = 4096 lseek(4, 1419860619264, SEEK_SET) = 1419860619264 read(4, ";d\360\177\n\346\253\210\222|\250\352T\335M\33\260\320\261\7g\222P\344H?t\240\20\2548\310"..., 4096) = 4096 lseek(4, 43018153984, SEEK_SET) = 43018153984 write(4, "\360\252j\317\310\251G\227\335{\214`\341\267\31Y\202\360\v\374\307oq\3063\217Z\223\313\36D\211"..., 4096) = 4096 So this number of the lseeks before the reads, like 1419860619264 are already a lot bigger, standing for 1.29 terabytes if the numbers are bytes, so it doesn't seem to be a linear progress on a big scale, maybe there are only some areas that need work, that have big gaps in between them. (times are in CET)

  Read the article

• Ubuntu server has slow performance

  - by Rich

  I have a custom built Ubuntu 11.04 server with a 6 disk software RAID 10 primary drive. On it I'm primarily running a PostgreSQL and a few other utilities that stream data from the web. I often find after a few hours of uptime the server starts to lag with all kinds of processes. For example, it may take 10-15 seconds after log-in to get a shell prompt. It might take 5-10 seconds for top to come up. An ls might take a second or two. When I look at top there is almost no CPU usage. There's a fair amount of memory used by the PostgreSQL server but not enough to bleed into swap. I have no idea where to go from here, other than to suspect the RAID10 (I've only ever had software RAID 1's before). Edit: Output from top: top - 11:56:03 up 1:46, 3 users, load average: 0.89, 0.73, 0.72 Tasks: 119 total, 1 running, 118 sleeping, 0 stopped, 0 zombie Cpu(s): 0.2%us, 0.0%sy, 0.0%ni, 93.5%id, 6.2%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 16325596k total, 3478248k used, 12847348k free, 20880k buffers Swap: 19534176k total, 0k used, 19534176k free, 3041992k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1747 woodsp 20 0 109m 10m 4888 S 1 0.1 0:42.70 python 357 root 20 0 0 0 0 S 0 0.0 0:00.40 jbd2/sda3-8 1 root 20 0 24324 2284 1344 S 0 0.0 0:00.84 init 2 root 20 0 0 0 0 S 0 0.0 0:00.00 kthreadd 3 root 20 0 0 0 0 S 0 0.0 0:00.24 ksoftirqd/0 6 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/0 7 root RT 0 0 0 0 S 0 0.0 0:00.01 watchdog/0 8 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/1 10 root 20 0 0 0 0 S 0 0.0 0:00.02 ksoftirqd/1 12 root RT 0 0 0 0 S 0 0.0 0:00.01 watchdog/1 13 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/2 14 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/2:0 15 root 20 0 0 0 0 S 0 0.0 0:00.00 ksoftirqd/2 16 root RT 0 0 0 0 S 0 0.0 0:00.01 watchdog/2 17 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/3 18 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/3:0 19 root 20 0 0 0 0 S 0 0.0 0:00.02 ksoftirqd/3 20 root RT 0 0 0 0 S 0 0.0 0:00.01 watchdog/3 21 root 0 -20 0 0 0 S 0 0.0 0:00.00 cpuset 22 root 0 -20 0 0 0 S 0 0.0 0:00.00 khelper 23 root 20 0 0 0 0 S 0 0.0 0:00.00 kdevtmpfs 24 root 0 -20 0 0 0 S 0 0.0 0:00.00 netns 26 root 20 0 0 0 0 S 0 0.0 0:00.00 sync_supers df -h rpsharp@ncp-skookum:~$ df -h Filesystem Size Used Avail Use% Mounted on /dev/sda3 1.8T 549G 1.2T 32% / udev 7.8G 4.0K 7.8G 1% /dev tmpfs 3.2G 492K 3.2G 1% /run none 5.0M 0 5.0M 0% /run/lock none 7.8G 0 7.8G 0% /run/shm /dev/sda2 952M 128K 952M 1% /boot/efi /dev/md0 5.5T 562G 4.7T 11% /usr/local free -m psharp@ncp-skookum:~$ free -m total used free shared buffers cached Mem: 15942 3409 12533 0 20 2983 -/+ buffers/cache: 405 15537 Swap: 19076 0 19076 tail -50 /var/log/syslog Jul 3 06:31:32 ncp-skookum rsyslogd: [origin software="rsyslogd" swVersion="5.8.6" x-pid="1070" x-info="http://www.rsyslog.com"] rsyslogd was HUPed Jul 3 06:39:01 ncp-skookum CRON[14211]: (root) CMD ( [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) ! -execdir fuser -s {} 2>/dev/null \; -delete) Jul 3 06:40:01 ncp-skookum CRON[14223]: (smmsp) CMD (test -x /etc/init.d/sendmail && /usr/share/sendmail/sendmail cron-msp) Jul 3 07:00:01 ncp-skookum CRON[14328]: (woodsp) CMD (/home/woodsp/bin/mail_tweetupdate # email an update) Jul 3 07:00:01 ncp-skookum CRON[14327]: (smmsp) CMD (test -x /etc/init.d/sendmail && /usr/share/sendmail/sendmail cron-msp) Jul 3 07:00:28 ncp-skookum sendmail[14356]: q63E0SoZ014356: from=woodsp, size=2328, class=0, nrcpts=2, msgid=<[email protected]>, relay=woodsp@localhost Jul 3 07:00:29 ncp-skookum sm-mta[14357]: q63E0Si6014357: from=<[email protected]>, size=2569, class=0, nrcpts=2, msgid=<[email protected]>, proto=ESMTP, daemon=MTA-v4, relay=localhost [127.0.0.1] Jul 3 07:00:29 ncp-skookum sendmail[14356]: q63E0SoZ014356: to=Spencer Wood <[email protected]>,Martin Lacayo <[email protected]>, ctladdr=woodsp (1004/1005), delay=00:00:01, xdelay=00:00:01, mailer=relay, pri=62328, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (q63E0Si6014357 Message accepted for delivery) Jul 3 07:00:29 ncp-skookum sm-mta[14359]: STARTTLS=client, relay=mx3.stanford.edu., version=TLSv1/SSLv3, verify=FAIL, cipher=DHE-RSA-AES256-SHA, bits=256/256 Jul 3 07:00:29 ncp-skookum sm-mta[14359]: q63E0Si6014357: to=<[email protected]>,<[email protected]>, ctladdr=<[email protected]> (1004/1005), delay=00:00:01, xdelay=00:00:00, mailer=esmtp, pri=152569, relay=mx3.stanford.edu. [171.67.219.73], dsn=2.0.0, stat=Sent (Ok: queued as 8F3505802AC) Jul 3 07:09:08 ncp-skookum CRON[14396]: (root) CMD ( [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) ! -execdir fuser -s {} 2>/dev/null \; -delete) Jul 3 07:17:01 ncp-skookum CRON[14438]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Jul 3 07:20:01 ncp-skookum CRON[14453]: (smmsp) CMD (test -x /etc/init.d/sendmail && /usr/share/sendmail/sendmail cron-msp) Jul 3 07:39:01 ncp-skookum CRON[14551]: (root) CMD ( [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) ! -execdir fuser -s {} 2>/dev/null \; -delete) Jul 3 07:40:01 ncp-skookum CRON[14562]: (smmsp) CMD (test -x /etc/init.d/sendmail && /usr/share/sendmail/sendmail cron-msp) Jul 3 08:00:01 ncp-skookum CRON[14668]: (smmsp) CMD (test -x /etc/init.d/sendmail && /usr/share/sendmail/sendmail cron-msp) Jul 3 08:09:01 ncp-skookum CRON[14724]: (root) CMD ( [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) ! -execdir fuser -s {} 2>/dev/null \; -delete) Jul 3 08:17:01 ncp-skookum CRON[14766]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Jul 3 08:20:01 ncp-skookum CRON[14781]: (smmsp) CMD (test -x /etc/init.d/sendmail && /usr/share/sendmail/sendmail cron-msp) Jul 3 08:39:01 ncp-skookum CRON[14881]: (root) CMD ( [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) ! -execdir fuser -s {} 2>/dev/null \; -delete) Jul 3 08:40:01 ncp-skookum CRON[14892]: (smmsp) CMD (test -x /etc/init.d/sendmail && /usr/share/sendmail/sendmail cron-msp) Output of hdparm -t /dev/sd{a,b,c,d,e,f} This looks suspicious? /dev/sda: Timing buffered disk reads: 2 MB in 4.84 seconds = 423.39 kB/sec /dev/sdb: Timing buffered disk reads: 420 MB in 3.01 seconds = 139.74 MB/sec /dev/sdc: Timing buffered disk reads: 390 MB in 3.00 seconds = 129.87 MB/sec /dev/sdd: Timing buffered disk reads: 416 MB in 3.00 seconds = 138.51 MB/sec /dev/sde: Timing buffered disk reads: 422 MB in 3.00 seconds = 140.50 MB/sec /dev/sdf: Timing buffered disk reads: 416 MB in 3.01 seconds = 138.26 MB/sec

  Read the article

• How to stop a ICMP attack?

  - by cumhur onat

  We are under a heavy icmp flood attack. Tcpdump shows the result below. Altough we have blocked ICMP with iptables tcpdump still prints icmp packets. I've also attached iptables configuration and "top" result. Is there any thing I can do to completely stop icmp packets? [root@server downloads]# tcpdump icmp -v -n -nn tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes 03:02:47.810957 IP (tos 0x0, ttl 49, id 16007, offset 0, flags [none], proto: ICMP (1), length: 56) 80.227.64.183 > 77.92.136.196: ICMP redirect 94.201.175.188 to host 80.227.64.129, length 36 IP (tos 0x0, ttl 124, id 31864, offset 0, flags [none], proto: ICMP (1), length: 76) 77.92.136.196 > 94.201.175.188: [|icmp] 03:02:47.811559 IP (tos 0x0, ttl 49, id 16010, offset 0, flags [none], proto: ICMP (1), length: 56) 80.227.64.183 > 77.92.136.196: ICMP redirect 94.201.175.188 to host 80.227.64.129, length 36 IP (tos 0x0, ttl 52, id 31864, offset 0, flags [none], proto: ICMP (1), length: 76) 77.92.136.196 > 94.201.175.188: [|icmp] 03:02:47.811922 IP (tos 0x0, ttl 49, id 16012, offset 0, flags [none], proto: ICMP (1), length: 56) 80.227.64.183 > 77.92.136.196: ICMP redirect 94.201.175.188 to host 80.227.64.129, length 36 IP (tos 0x0, ttl 122, id 31864, offset 0, flags [none], proto: ICMP (1), length: 76) 77.92.136.196 > 94.201.175.188: [|icmp] 03:02:47.812485 IP (tos 0x0, ttl 49, id 16015, offset 0, flags [none], proto: ICMP (1), length: 56) 80.227.64.183 > 77.92.136.196: ICMP redirect 94.201.175.188 to host 80.227.64.129, length 36 IP (tos 0x0, ttl 126, id 31864, offset 0, flags [none], proto: ICMP (1), length: 76) 77.92.136.196 > 94.201.175.188: [|icmp] 03:02:47.812613 IP (tos 0x0, ttl 49, id 16016, offset 0, flags [none], proto: ICMP (1), length: 56) 80.227.64.183 > 77.92.136.196: ICMP redirect 94.201.175.188 to host 80.227.64.129, length 36 IP (tos 0x0, ttl 122, id 31864, offset 0, flags [none], proto: ICMP (1), length: 76) 77.92.136.196 > 94.201.175.188: [|icmp] 03:02:47.812992 IP (tos 0x0, ttl 49, id 16018, offset 0, flags [none], proto: ICMP (1), length: 56) 80.227.64.183 > 77.92.136.196: ICMP redirect 94.201.175.188 to host 80.227.64.129, length 36 IP (tos 0x0, ttl 122, id 31864, offset 0, flags [none], proto: ICMP (1), length: 76) 77.92.136.196 > 94.201.175.188: [|icmp] 03:02:47.813582 IP (tos 0x0, ttl 49, id 16020, offset 0, flags [none], proto: ICMP (1), length: 56) 80.227.64.183 > 77.92.136.196: ICMP redirect 94.201.175.188 to host 80.227.64.129, length 36 IP (tos 0x0, ttl 52, id 31864, offset 0, flags [none], proto: ICMP (1), length: 76) 77.92.136.196 > 94.201.175.188: [|icmp] 03:02:47.814092 IP (tos 0x0, ttl 49, id 16023, offset 0, flags [none], proto: ICMP (1), length: 56) 80.227.64.183 > 77.92.136.196: ICMP redirect 94.201.175.188 to host 80.227.64.129, length 36 IP (tos 0x0, ttl 120, id 31864, offset 0, flags [none], proto: ICMP (1), length: 76) 77.92.136.196 > 94.201.175.188: [|icmp] 03:02:47.814233 IP (tos 0x0, ttl 49, id 16024, offset 0, flags [none], proto: ICMP (1), length: 56) 80.227.64.183 > 77.92.136.196: ICMP redirect 94.201.175.188 to host 80.227.64.129, length 36 IP (tos 0x0, ttl 120, id 31864, offset 0, flags [none], proto: ICMP (1), length: 76) 77.92.136.196 > 94.201.175.188: [|icmp] 03:02:47.815579 IP (tos 0x0, ttl 49, id 16025, offset 0, flags [none], proto: ICMP (1), length: 56) 80.227.64.183 > 77.92.136.196: ICMP redirect 94.201.175.188 to host 80.227.64.129, length 36 IP (tos 0x0, ttl 50, id 31864, offset 0, flags [none], proto: ICMP (1), length: 76) 77.92.136.196 > 94.201.175.188: [|icmp] 03:02:47.815726 IP (tos 0x0, ttl 49, id 16026, offset 0, flags [none], proto: ICMP (1), length: 56) 80.227.64.183 > 77.92.136.196: ICMP redirect 94.201.175.188 to host 80.227.64.129, length 36 IP (tos 0x0, ttl 50, id 31864, offset 0, flags [none], proto: ICMP (1), length: 76) 77.92.136.196 > 94.201.175.188: [|icmp] 03:02:47.815890 IP (tos 0x0, ttl 49, id 16027, offset 0, flags [none], proto: ICMP (1), length: 56) 80.227.64.183 > 77.92.136.196: ICMP redirect 94.201.175.188 to host 80.227.64.129, length 36 iptables configuration: [root@server etc]# iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination ofis tcp -- anywhere anywhere tcp dpt:mysql ofis tcp -- anywhere anywhere tcp dpt:ftp DROP icmp -- anywhere anywhere Chain FORWARD (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination DROP icmp -- anywhere anywhere Chain ofis (2 references) target prot opt source destination ACCEPT all -- OUR_OFFICE_IP anywhere DROP all -- anywhere anywhere top: top - 03:12:19 up 400 days, 15:43, 3 users, load average: 1.49, 1.67, 2.61 Tasks: 751 total, 3 running, 748 sleeping, 0 stopped, 0 zombie Cpu(s): 8.2%us, 1.0%sy, 0.0%ni, 87.9%id, 2.1%wa, 0.1%hi, 0.7%si, 0.0%st Mem: 32949948k total, 26906844k used, 6043104k free, 4707676k buffers Swap: 10223608k total, 0k used, 10223608k free, 14255584k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 36 root 39 19 0 0 0 R 100.8 0.0 17:03.56 ksoftirqd/11 10552 root 15 0 11408 1460 676 R 5.7 0.0 0:00.04 top 7475 lighttpd 15 0 304m 22m 15m S 3.8 0.1 0:05.37 php-cgi 1294 root 10 -5 0 0 0 S 1.9 0.0 380:54.73 kjournald 3574 root 15 0 631m 11m 5464 S 1.9 0.0 0:00.65 node 7766 lighttpd 16 0 302m 19m 14m S 1.9 0.1 0:05.70 php-cgi 10237 postfix 15 0 52572 2216 1692 S 1.9 0.0 0:00.02 scache 1 root 15 0 10372 680 572 S 0.0 0.0 0:07.99 init 2 root RT -5 0 0 0 S 0.0 0.0 0:16.72 migration/0 3 root 34 19 0 0 0 S 0.0 0.0 0:00.06 ksoftirqd/0 4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0 5 root RT -5 0 0 0 S 0.0 0.0 1:10.46 migration/1 6 root 34 19 0 0 0 S 0.0 0.0 0:01.11 ksoftirqd/1 7 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/1 8 root RT -5 0 0 0 S 0.0 0.0 2:36.15 migration/2 9 root 34 19 0 0 0 S 0.0 0.0 0:00.19 ksoftirqd/2 10 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/2 11 root RT -5 0 0 0 S 0.0 0.0 3:48.91 migration/3 12 root 34 19 0 0 0 S 0.0 0.0 0:00.20 ksoftirqd/3 13 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/3 uname -a [root@server etc]# uname -a Linux thisis.oursite.com 2.6.18-238.19.1.el5 #1 SMP Fri Jul 15 07:31:24 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux arp -an [root@server downloads]# arp -an ? (77.92.136.194) at 00:25:90:04:F0:90 [ether] on eth0 ? (192.168.0.2) at 00:25:90:04:F0:91 [ether] on eth1 ? (77.92.136.193) at 00:23:9C:0B:CD:01 [ether] on eth0

  Read the article

• More interruptions than cpu context switches

  - by Christopher Valles

  I have a machine running Debian GNU/Linux 5.0.8 (lenny) 8 cores and 12Gb of RAM. We have one core permanently around 40% ~ 60% wait time and trying to spot what is happening I realized that we have more interruptions than cpu context switches. I found that the normal ratio between context switch and interruptions is around 10x more context switching than interruptions but on my server the values are completely different. backend1:~# vmstat -s 12330788 K total memory 12221676 K used memory 3668624 K active memory 6121724 K inactive memory 109112 K free memory 3929400 K buffer memory 4095536 K swap cache 4194296 K total swap 7988 K used swap 4186308 K free swap 44547459 non-nice user cpu ticks 702408 nice user cpu ticks 13346333 system cpu ticks 1607583668 idle cpu ticks 374043393 IO-wait cpu ticks 4144149 IRQ cpu ticks 3994255 softirq cpu ticks 0 stolen cpu ticks 4445557114 pages paged in 2910596714 pages paged out 128642 pages swapped in 267400 pages swapped out 3519307319 interrupts 2464686911 CPU context switches 1306744317 boot time 11555115 forks Any ideas if that is an issue? And in that case, how can I spot the cause and fix it? Update Following the instructions of the comments and focusing on the core stuck in wait I checked the processes attached to that core and below you can find the list: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ P COMMAND 24 root RT -5 0 0 0 S 0 0.0 0:03.42 7 migration/7 25 root 15 -5 0 0 0 S 0 0.0 0:04.78 7 ksoftirqd/7 26 root RT -5 0 0 0 S 0 0.0 0:00.00 7 watchdog/7 34 root 15 -5 0 0 0 S 0 0.0 1:18.90 7 events/7 83 root 15 -5 0 0 0 S 0 0.0 1:10.68 7 kblockd/7 291 root 15 -5 0 0 0 S 0 0.0 0:00.00 7 aio/7 569 root 15 -5 0 0 0 S 0 0.0 0:00.00 7 ata/7 1545 root 15 -5 0 0 0 S 0 0.0 0:00.00 7 ksnapd 1644 root 15 -5 0 0 0 S 0 0.0 0:36.73 7 kjournald 1725 root 16 -4 16940 1152 488 S 0 0.0 0:00.00 7 udevd 2342 root 20 0 8828 1140 956 S 0 0.0 0:00.00 7 sh 2375 root 20 0 8848 1220 1016 S 0 0.0 0:00.00 7 locate 2421 root 30 10 8896 1268 1016 S 0 0.0 0:00.00 7 updatedb.findut 2430 root 30 10 58272 49m 616 S 0 0.4 0:17.44 7 sort 2431 root 30 10 3792 448 360 S 0 0.0 0:00.00 7 frcode 2682 root 15 -5 0 0 0 S 0 0.0 3:25.98 7 kjournald 2683 root 15 -5 0 0 0 S 0 0.0 0:00.64 7 kjournald 2687 root 15 -5 0 0 0 S 0 0.0 1:31.30 7 kjournald 3261 root 15 -5 0 0 0 S 0 0.0 2:30.56 7 kondemand/7 3364 root 20 0 3796 596 476 S 0 0.0 0:00.00 7 acpid 3575 root 20 0 8828 1140 956 S 0 0.0 0:00.00 7 sh 3597 root 20 0 8848 1216 1016 S 0 0.0 0:00.00 7 locate 3603 root 30 10 8896 1268 1016 S 0 0.0 0:00.00 7 updatedb.findut 3612 root 30 10 58272 49m 616 S 0 0.4 0:27.04 7 sort 3655 root 20 0 11056 2852 516 S 0 0.0 5:36.46 7 redis-server 3706 root 20 0 19832 1056 816 S 0 0.0 0:01.64 7 cron 3746 root 20 0 3796 580 484 S 0 0.0 0:00.00 7 getty 3748 root 20 0 3796 580 484 S 0 0.0 0:00.00 7 getty 7674 root 20 0 28376 1000 736 S 0 0.0 0:00.00 7 cron 7675 root 20 0 8828 1140 956 S 0 0.0 0:00.00 7 sh 7708 root 30 10 58272 49m 616 S 0 0.4 0:03.36 7 sort 22049 root 20 0 8828 1136 956 S 0 0.0 0:00.00 7 sh 22095 root 20 0 8848 1220 1016 S 0 0.0 0:00.00 7 locate 22099 root 30 10 8896 1264 1016 S 0 0.0 0:00.00 7 updatedb.findut 22108 root 30 10 58272 49m 616 S 0 0.4 0:44.55 7 sort 22109 root 30 10 3792 452 360 S 0 0.0 0:00.00 7 frcode 26927 root 20 0 8828 1140 956 S 0 0.0 0:00.00 7 sh 26947 root 20 0 8848 1216 1016 S 0 0.0 0:00.00 7 locate 26951 root 30 10 8896 1268 1016 S 0 0.0 0:00.00 7 updatedb.findut 26960 root 30 10 58272 49m 616 S 0 0.4 0:10.24 7 sort 26961 root 30 10 3792 452 360 S 0 0.0 0:00.00 7 frcode 27952 root 20 0 65948 3028 2400 S 0 0.0 0:00.00 7 sshd 30731 root 20 0 0 0 0 S 0 0.0 0:01.34 7 pdflush 31204 root 20 0 0 0 0 S 0 0.0 0:00.24 7 pdflush 21857 deploy 20 0 1227m 2240 868 S 0 0.0 2:44.22 7 nginx 21858 deploy 20 0 1228m 2784 868 S 0 0.0 2:42.45 7 nginx 21862 deploy 20 0 1228m 2732 868 S 0 0.0 2:43.90 7 nginx 21869 deploy 20 0 1228m 2840 868 S 0 0.0 2:44.14 7 nginx 27994 deploy 20 0 19372 2216 1380 S 0 0.0 0:00.00 7 bash 28493 deploy 20 0 331m 32m 16m S 4 0.3 0:00.40 7 apache2 21856 deploy 20 0 1228m 2844 868 S 0 0.0 2:43.64 7 nginx 3622 nobody 30 10 21156 10m 916 D 0 0.1 4:42.31 7 find 7716 nobody 30 10 12268 1280 888 D 0 0.0 0:43.50 7 find 22116 nobody 30 10 12612 1696 916 D 0 0.0 6:32.26 7 find 26968 nobody 30 10 12268 1284 888 D 0 0.0 1:56.92 7 find Update As suggested I take a look at /proc/interrupts and below the info there: CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 0: 35 0 0 1469085485 0 0 0 0 IO-APIC-edge timer 1: 0 0 0 8 0 0 0 0 IO-APIC-edge i8042 8: 0 0 0 1 0 0 0 0 IO-APIC-edge rtc0 9: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi acpi 12: 0 0 0 105 0 0 0 0 IO-APIC-edge i8042 16: 0 0 0 0 0 0 0 580212114 IO-APIC-fasteoi 3w-9xxx, uhci_hcd:usb1 18: 0 0 142 0 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb6, ehci_hcd:usb7 19: 9 0 0 0 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb3, uhci_hcd:usb5 21: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb2 23: 0 0 0 0 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4, ehci_hcd:usb8 1273: 0 0 1600400502 0 0 0 0 0 PCI-MSI-edge eth0 1274: 0 0 0 0 0 0 0 0 PCI-MSI-edge ahci NMI: 0 0 0 0 0 0 0 0 Non-maskable interrupts LOC: 214252181 69439018 317298553 21943690 72562482 56448835 137923978 407514738 Local timer interrupts RES: 27516446 16935944 26430972 44957009 24935543 19881887 57746906 24298747 Rescheduling interrupts CAL: 10655 10705 10685 10567 10689 10669 10667 396 function call interrupts TLB: 529548 462587 801138 596193 922202 747313 2027966 946594 TLB shootdowns TRM: 0 0 0 0 0 0 0 0 Thermal event interrupts THR: 0 0 0 0 0 0 0 0 Threshold APIC interrupts SPU: 0 0 0 0 0 0 0 0 Spurious interrupts ERR: 0 All the values seems more or less the same for all the cores but this one IO-APIC-fasteoi 3w-9xxx, uhci_hcd:usb1 only affects to the core 7 (the same with the wait time of 40% ~ 60%) could be something attached to the usb port causing the issue? Thanks in advanced

  Read the article

• High memory usage on the server - can't determine the process

  - by HTF

  I've noticed high memory usage on the server. Details: OS: CentOS 6.3 - x86_64 Web server: Nginx with PHP-FPM The server is generating PDF documents so the traffic is minimum. top: # top -b -n 1 -a top - 10:04:51 up 21 days, 18:57, 1 user, load average: 0.00, 0.00, 0.00 Tasks: 92 total, 1 running, 91 sleeping, 0 stopped, 0 zombie Cpu(s): 0.3%us, 0.2%sy, 0.0%ni, 99.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 3923092k total, 3720380k used, 202712k free, 133904k buffers Swap: 4194296k total, 12k used, 4194284k free, 147404k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 15855 www-data 20 0 199m 4952 2128 S 0.0 0.1 0:00.06 php-fpm 15853 www-data 20 0 199m 4940 2028 S 0.0 0.1 0:00.06 php-fpm 15850 www-data 20 0 199m 4928 2020 S 0.0 0.1 0:00.05 php-fpm 15851 www-data 20 0 199m 4888 2020 S 0.0 0.1 0:00.06 php-fpm 15852 www-data 20 0 199m 4852 2020 S 0.0 0.1 0:00.06 php-fpm 15857 www-data 20 0 198m 4716 2020 S 0.0 0.1 0:00.06 php-fpm 17553 root 20 0 97816 3860 2924 S 0.0 0.1 0:00.03 sshd 15849 root 20 0 198m 3460 1072 S 0.0 0.1 0:00.12 php-fpm 13441 nginx 20 0 65608 2968 1604 S 0.0 0.1 0:02.06 nginx 13440 nginx 20 0 65608 2964 1600 S 0.0 0.1 0:01.87 nginx 17561 root 20 0 105m 1944 1488 S 0.0 0.0 0:00.01 bash 1150 xfs 20 0 20980 1784 704 S 0.0 0.0 0:00.13 xfs 15863 root 20 0 179m 1424 1028 S 0.0 0.0 0:00.00 rsyslogd 1 root 20 0 19224 1360 1088 S 0.0 0.0 0:17.96 init 1201 nrpe 20 0 40928 1288 704 S 0.0 0.0 3:57.64 nrpe 13226 root 20 0 114m 1216 612 S 0.0 0.0 0:00.01 crond 6691 root 20 0 64068 1156 488 S 0.0 0.0 0:09.59 sshd 13439 root 20 0 65104 1128 292 S 0.0 0.0 0:00.00 nginx 19026 root 20 0 15040 1116 844 R 0.0 0.0 0:00.00 top 451 root 16 -4 11052 1096 316 S 0.0 0.0 0:00.02 udevd 1174 root 18 -2 11048 1064 288 S 0.0 0.0 0:00.00 udevd 1175 root 18 -2 11048 1064 288 S 0.0 0.0 0:00.00 udevd 1065 root 16 -4 93168 824 560 S 0.0 0.0 0:16.00 auditd 1165 root 20 0 4056 564 480 S 0.0 0.0 0:00.00 mingetty 1167 root 20 0 4056 564 480 S 0.0 0.0 0:00.00 mingetty 1169 root 20 0 4056 564 480 S 0.0 0.0 0:00.00 mingetty 1171 root 20 0 4056 564 480 S 0.0 0.0 0:00.00 mingetty 1163 root 20 0 4056 560 480 S 0.0 0.0 0:00.00 mingetty 1176 root 20 0 4056 560 480 S 0.0 0.0 0:00.00 mingetty 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd 3 root RT 0 0 0 0 S 0.0 0.0 0:11.75 migration/0 4 root 20 0 0 0 0 S 0.0 0.0 44:30.28 ksoftirqd/0 5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0 6 root RT 0 0 0 0 S 0.0 0.0 0:03.51 watchdog/0 7 root RT 0 0 0 0 S 0.0 0.0 0:11.63 migration/1 8 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/1 9 root 20 0 0 0 0 S 0.0 0.0 11:35.50 ksoftirqd/1 10 root RT 0 0 0 0 S 0.0 0.0 0:03.34 watchdog/1 11 root 20 0 0 0 0 S 0.0 0.0 1:36.68 events/0 12 root 20 0 0 0 0 S 0.0 0.0 1:50.57 events/1 13 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cgroup 14 root 20 0 0 0 0 S 0.0 0.0 0:00.00 khelper 15 root 20 0 0 0 0 S 0.0 0.0 0:00.00 netns 16 root 20 0 0 0 0 S 0.0 0.0 0:00.00 async/mgr 17 root 20 0 0 0 0 S 0.0 0.0 0:00.00 pm 18 root 20 0 0 0 0 S 0.0 0.0 0:07.86 sync_supers 19 root 20 0 0 0 0 S 0.0 0.0 0:10.38 bdi-default 20 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/0 21 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/1 22 root 20 0 0 0 0 S 0.0 0.0 0:04.35 kblockd/0 23 root 20 0 0 0 0 S 0.0 0.0 0:04.18 kblockd/1 24 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kacpid 25 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kacpi_notify 26 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kacpi_hotplug 27 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata/0 28 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata/1 29 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ata_aux 30 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksuspend_usbd 31 root 20 0 0 0 0 S 0.0 0.0 0:00.00 khubd 32 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kseriod 33 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md/0 34 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md/1 35 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md_misc/0 36 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md_misc/1 37 root 20 0 0 0 0 S 0.0 0.0 0:00.48 khungtaskd 38 root 20 0 0 0 0 S 0.0 0.0 1:07.52 kswapd0 39 root 25 5 0 0 0 S 0.0 0.0 0:00.00 ksmd 40 root 39 19 0 0 0 S 0.0 0.0 0:22.00 khugepaged 41 root 20 0 0 0 0 S 0.0 0.0 0:00.00 aio/0 42 root 20 0 0 0 0 S 0.0 0.0 0:00.00 aio/1 43 root 20 0 0 0 0 S 0.0 0.0 0:00.00 crypto/0 44 root 20 0 0 0 0 S 0.0 0.0 0:00.00 crypto/1 49 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthrotld/0 50 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthrotld/1 52 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kpsmoused 53 root 20 0 0 0 0 S 0.0 0.0 0:00.00 usbhid_resumer 83 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kstriped 233 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_0 234 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_1 321 root 20 0 0 0 0 S 0.0 0.0 0:00.00 virtio-blk 359 root 20 0 0 0 0 S 0.0 0.0 0:03.24 kdmflush 360 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdmflush 380 root 20 0 0 0 0 S 0.0 0.0 0:20.64 jbd2/dm-0-8 381 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ext4-dio-unwrit 382 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ext4-dio-unwrit 694 root 20 0 0 0 0 S 0.0 0.0 0:00.00 vballoon 697 root 20 0 0 0 0 S 0.0 0.0 0:00.00 virtio-net 818 root 20 0 0 0 0 S 0.0 0.0 0:00.00 jbd2/vda1-8 819 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ext4-dio-unwrit 820 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ext4-dio-unwrit 851 root 20 0 0 0 0 S 0.0 0.0 0:06.96 kauditd 1013 root 20 0 0 0 0 S 0.0 0.0 0:15.45 flush-253:0 ps: # ps aux --sort -vsz | head USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND www-data 13213 0.0 0.1 204416 4772 ? S 08:28 0:00 php-fpm: pool default www-data 13214 0.0 0.1 204416 4776 ? S 08:28 0:00 php-fpm: pool default www-data 13215 0.0 0.1 204416 4832 ? S 08:28 0:00 php-fpm: pool default www-data 13216 0.0 0.1 204416 4776 ? S 08:28 0:00 php-fpm: pool default www-data 13218 0.0 0.1 204416 4956 ? S 08:28 0:00 php-fpm: pool default free: #free -m total used free shared buffers cached Mem: 3831 3530 300 0 130 143 -/+ buffers/cache: 3256 574 Swap: 4095 0 4095 When I stooped Nginx, PHP-FPM the memory usage was still the same. Could you help me to investigate what is consuming the memory on the system? Regards

  Read the article

• Why are my Opteron cores running at only 75% capacity each? (25% CPU idle)

  - by Tim Cooper

  We've just taken delivery of a powerful 32-core AMD Opteron server with 128Gb. We have 2 x 6272 CPU's with 16 cores each. We are running a big long-running java task on 30 threads. We have the NUMA optimisations for Linux and java turned on. Our Java threads are mainly using objects that are private to that thread, sometimes reading memory that other threads will be reading, and very very occasionally writing or locking shared objects. We can't explain why the CPU cores are 25% idle. Below is a dump of "top": top - 23:06:38 up 1 day, 23 min, 3 users, load average: 10.84, 10.27, 9.62 Tasks: 676 total, 1 running, 675 sleeping, 0 stopped, 0 zombie Cpu(s): 64.5%us, 1.3%sy, 0.0%ni, 32.9%id, 1.3%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 132138168k total, 131652664k used, 485504k free, 92340k buffers Swap: 5701624k total, 230252k used, 5471372k free, 13444344k cached ... top - 22:37:39 up 23:54, 3 users, load average: 7.83, 8.70, 9.27 Tasks: 678 total, 1 running, 677 sleeping, 0 stopped, 0 zombie Cpu0 : 75.8%us, 2.0%sy, 0.0%ni, 22.2%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu1 : 77.2%us, 1.3%sy, 0.0%ni, 21.5%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu2 : 77.3%us, 1.0%sy, 0.0%ni, 21.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu3 : 77.8%us, 1.0%sy, 0.0%ni, 21.2%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu4 : 76.9%us, 2.0%sy, 0.0%ni, 21.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu5 : 76.3%us, 2.0%sy, 0.0%ni, 21.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu6 : 12.6%us, 3.0%sy, 0.0%ni, 84.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu7 : 8.6%us, 2.0%sy, 0.0%ni, 89.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu8 : 77.0%us, 2.0%sy, 0.0%ni, 21.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu9 : 77.0%us, 2.0%sy, 0.0%ni, 21.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu10 : 77.6%us, 1.7%sy, 0.0%ni, 20.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu11 : 75.7%us, 2.0%sy, 0.0%ni, 21.4%id, 1.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu12 : 76.6%us, 2.3%sy, 0.0%ni, 21.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu13 : 76.6%us, 2.3%sy, 0.0%ni, 21.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu14 : 76.2%us, 2.6%sy, 0.0%ni, 15.9%id, 5.3%wa, 0.0%hi, 0.0%si, 0.0%st Cpu15 : 76.6%us, 2.0%sy, 0.0%ni, 21.5%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu16 : 73.6%us, 2.6%sy, 0.0%ni, 23.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu17 : 74.5%us, 2.3%sy, 0.0%ni, 23.2%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu18 : 73.9%us, 2.3%sy, 0.0%ni, 23.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu19 : 72.9%us, 2.6%sy, 0.0%ni, 24.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu20 : 72.8%us, 2.6%sy, 0.0%ni, 24.5%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu21 : 72.7%us, 2.3%sy, 0.0%ni, 25.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu22 : 72.5%us, 2.6%sy, 0.0%ni, 24.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu23 : 73.0%us, 2.3%sy, 0.0%ni, 24.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu24 : 74.7%us, 2.7%sy, 0.0%ni, 22.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu25 : 74.5%us, 2.6%sy, 0.0%ni, 22.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu26 : 73.7%us, 2.0%sy, 0.0%ni, 24.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu27 : 74.1%us, 2.3%sy, 0.0%ni, 23.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu28 : 74.1%us, 2.3%sy, 0.0%ni, 23.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu29 : 74.0%us, 2.0%sy, 0.0%ni, 24.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu30 : 73.2%us, 2.3%sy, 0.0%ni, 24.5%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu31 : 73.1%us, 2.0%sy, 0.0%ni, 24.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 132138168k total, 131711704k used, 426464k free, 88336k buffers Swap: 5701624k total, 229572k used, 5472052k free, 13745596k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 13865 root 20 0 122g 112g 3.1g S 2334.3 89.6 20726:49 java 27139 jayen 20 0 15428 1728 952 S 2.6 0.0 0:04.21 top 27161 sysadmin 20 0 15428 1712 940 R 1.0 0.0 0:00.28 top 33 root 20 0 0 0 0 S 0.3 0.0 0:06.24 ksoftirqd/7 131 root 20 0 0 0 0 S 0.3 0.0 0:09.52 events/0 1858 root 20 0 0 0 0 S 0.3 0.0 1:35.14 kondemand/0 A dump of the java stack confirms that none of the threads are anywhere near the few places where locks are used, nor are they anywhere near any disk or network i/o. I had trouble finding a clear explanation of what 'top' means by "idle" versus "wait", but I get the impression that "idle" means "no more threads that need to be run" but this doesn't make sense in our case. We're using a "Executors.newFixedThreadPool(30)". There are a large number of tasks pending and each task lasts for 10 seconds or so. I suspect that the explanation requires a good understanding of NUMA. Is the "idle" state what you see when a CPU is waiting for a non-local access? If not, then what is the explanation?

  Read the article

• e2fsck extremely slow, although enough memory exists

  - by kaefert

  I've got this external USB-Disk: kaefert@blechmobil:~$ lsusb -s 2:3 Bus 002 Device 003: ID 0bc2:3320 Seagate RSS LLC As can be seen in this dmesg output, there is some problem that prevents that disk from beeing mounted: kaefert@blechmobil:~$ dmesg ... [ 113.084079] usb 2-1: new high-speed USB device number 3 using ehci_hcd [ 113.217783] usb 2-1: New USB device found, idVendor=0bc2, idProduct=3320 [ 113.217787] usb 2-1: New USB device strings: Mfr=2, Product=3, SerialNumber=1 [ 113.217790] usb 2-1: Product: Expansion Desk [ 113.217792] usb 2-1: Manufacturer: Seagate [ 113.217794] usb 2-1: SerialNumber: NA4J4N6K [ 113.435404] usbcore: registered new interface driver uas [ 113.455315] Initializing USB Mass Storage driver... [ 113.468051] scsi5 : usb-storage 2-1:1.0 [ 113.468180] usbcore: registered new interface driver usb-storage [ 113.468182] USB Mass Storage support registered. [ 114.473105] scsi 5:0:0:0: Direct-Access Seagate Expansion Desk 070B PQ: 0 ANSI: 6 [ 114.474342] sd 5:0:0:0: [sdb] 732566645 4096-byte logical blocks: (3.00 TB/2.72 TiB) [ 114.475089] sd 5:0:0:0: [sdb] Write Protect is off [ 114.475092] sd 5:0:0:0: [sdb] Mode Sense: 43 00 00 00 [ 114.475959] sd 5:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 114.477093] sd 5:0:0:0: [sdb] 732566645 4096-byte logical blocks: (3.00 TB/2.72 TiB) [ 114.501649] sdb: sdb1 [ 114.502717] sd 5:0:0:0: [sdb] 732566645 4096-byte logical blocks: (3.00 TB/2.72 TiB) [ 114.504354] sd 5:0:0:0: [sdb] Attached SCSI disk [ 116.804408] EXT4-fs (sdb1): ext4_check_descriptors: Checksum for group 3976 failed (47397!=61519) [ 116.804413] EXT4-fs (sdb1): group descriptors corrupted! ... So I went and fired up my favorite partition manager - gparted, and told it to verify and repair the partition sdb1. This made gparted call e2fsck (version 1.42.4 (12-Jun-2012)) e2fsck -f -y -v /dev/sdb1 Although gparted called e2fsck with the "-v" option, sadly it doesn't show me the output of my e2fsck process (bugreport https://bugzilla.gnome.org/show_bug.cgi?id=467925 ) I started this whole thing on Sunday (2012-11-04_2200) evening, so about 48 hours ago, this is what htop says about it now (2012-11-06-1900): PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command 3704 root 39 19 1560M 1166M 768 R 98.0 19.5 42h56:43 e2fsck -f -y -v /dev/sdb1 Now I found a few posts on the internet that discuss e2fsck running slow, for example: http://gparted-forum.surf4.info/viewtopic.php?id=13613 where they write that its a good idea to see if the disk is just that slow because maybe its damaged, and I think these outputs tell me that this is not the case in my case: kaefert@blechmobil:~$ sudo hdparm -tT /dev/sdb /dev/sdb: Timing cached reads: 3562 MB in 2.00 seconds = 1783.29 MB/sec Timing buffered disk reads: 82 MB in 3.01 seconds = 27.26 MB/sec kaefert@blechmobil:~$ sudo hdparm /dev/sdb /dev/sdb: multcount = 0 (off) readonly = 0 (off) readahead = 256 (on) geometry = 364801/255/63, sectors = 5860533160, start = 0 However, although I can read quickly from that disk, this disk speed doesn't seem to be used by e2fsck, considering tools like gkrellm or iotop or this: kaefert@blechmobil:~$ iostat -x Linux 3.2.0-2-amd64 (blechmobil) 2012-11-06 _x86_64_ (2 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 14,24 47,81 14,63 0,95 0,00 22,37 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0,59 8,29 2,42 5,14 43,17 160,17 53,75 0,30 39,80 8,72 54,42 3,95 2,99 sdb 137,54 5,48 9,23 0,20 587,07 22,73 129,35 0,07 7,70 7,51 16,18 2,17 2,04 Now I researched a little bit on how to find out what e2fsck is doing with all that processor time, and I found the tool strace, which gives me this: kaefert@blechmobil:~$ sudo strace -p3704 lseek(4, 41026998272, SEEK_SET) = 41026998272 write(4, "\212\354K[_\361\3nl\212\245\352\255jR\303\354\312Yv\334p\253r\217\265\3567\325\257\3766"..., 4096) = 4096 lseek(4, 48404766720, SEEK_SET) = 48404766720 read(4, "\7t\260\366\346\337\304\210\33\267j\35\377'\31f\372\252\ffU\317.y\211\360\36\240c\30`\34"..., 4096) = 4096 lseek(4, 41027002368, SEEK_SET) = 41027002368 write(4, "\232]7Ws\321\352\t\1@[+5\263\334\276{\343zZx\352\21\316`1\271[\202\350R`"..., 4096) = 4096 lseek(4, 48404770816, SEEK_SET) = 48404770816 read(4, "\17\362r\230\327\25\346//\210H\v\311\3237\323K\304\306\361a\223\311\324\272?\213\tq \370\24"..., 4096) = 4096 lseek(4, 41027006464, SEEK_SET) = 41027006464 write(4, "\367yy>x\216?=\324Z\305\351\376&\25\244\210\271\22\306}\276\237\370(\214\205G\262\360\257#"..., 4096) = 4096 lseek(4, 48404774912, SEEK_SET) = 48404774912 read(4, "\365\25\0\21|T\0\21}3t_\272\373\222k\r\177\303\1\201\261\221$\261B\232\3142\21U\316"..., 4096) = 4096 ^CProcess 3704 detached around 16 of these lines every second, so 4 read and 4 write operations every second, which I don't consider to be a lot.. And finally, my question: Will this process ever finish? If those numbers from fseek (48404774912) represent bytes, that would be something like 45 gigabytes, with this beeing a 3 terrabyte disk, which would give me 134 days to go, if the speed stays constant, and e2fsck scans the disk like this completly and only once. Do you have some advice for me? I have most of the data on that disk elsewhere, but I've put a lot of hours into sorting and merging it to this disk, so I would prefer to getting this disk up and running again, without formatting it anew. I don't think that the hardware is damaged since the disk is only a few months and since I can't see any I/O errors in the dmesg output. UPDATE: I just looked at the strace output again (2012-11-06_2300), now it looks like this: lseek(4, 1419860611072, SEEK_SET) = 1419860611072 read(4, "3#\f\2447\335\0\22A\355\374\276j\204'\207|\217V|\23\245[\7VP\251\242\276\207\317:"..., 4096) = 4096 lseek(4, 43018145792, SEEK_SET) = 43018145792 write(4, "]\206\231\342Y\204-2I\362\242\344\6R\205\361\324\177\265\317C\334V\324\260\334\275t=\10F."..., 4096) = 4096 lseek(4, 1419860615168, SEEK_SET) = 1419860615168 read(4, "\262\305\314Y\367\37x\326\245\226\226\320N\333$s\34\204\311\222\7\315\236\336\300TK\337\264\236\211n"..., 4096) = 4096 lseek(4, 43018149888, SEEK_SET) = 43018149888 write(4, "\271\224m\311\224\25!I\376\16;\377\0\223H\25Yd\201Y\342\r\203\271\24eG<\202{\373V"..., 4096) = 4096 lseek(4, 1419860619264, SEEK_SET) = 1419860619264 read(4, ";d\360\177\n\346\253\210\222|\250\352T\335M\33\260\320\261\7g\222P\344H?t\240\20\2548\310"..., 4096) = 4096 lseek(4, 43018153984, SEEK_SET) = 43018153984 write(4, "\360\252j\317\310\251G\227\335{\214`\341\267\31Y\202\360\v\374\307oq\3063\217Z\223\313\36D\211"..., 4096) = 4096 So the numbers in the lseek lines before the reads, like 1419860619264 are already a lot bigger, standing for 1.29 terabytes if those numbers are bytes, so it doesn't seem to be a linear progress on a big scale, maybe there are only some areas that need work, that have big gaps in between them. UPDATE2: Okey, big disappointment, the numbers are back to very small again (2012-11-07_0720) lseek(4, 52174548992, SEEK_SET) = 52174548992 read(4, "\374\312\22\\\325\215\213\23\0357U\222\246\370v^f(\312|f\212\362\343\375\373\342\4\204mU6"..., 4096) = 4096 lseek(4, 46603526144, SEEK_SET) = 46603526144 write(4, "\370\261\223\227\23?\4\4\217\264\320_Am\246CQ\313^\203U\253\274\204\277\2564n\227\177\267\343"..., 4096) = 4096 so either e2fsck goes over the data multiple times, or it just hops back and forth multiple times. Or my assumption that those numbers are bytes is wrong. UPDATE3: Since it's mentioned here http://forums.fedoraforum.org/showthread.php?t=282125&page=2 that you can testisk while e2fsck is running, i tried that, though not with a lot of success. When asking testdisk to display the data of my partition, this is what I get: TestDisk 6.13, Data Recovery Utility, November 2011 Christophe GRENIER <[email protected]> http://www.cgsecurity.org 1 P Linux 0 4 5 45600 40 8 732566272 Can't open filesystem. Filesystem seems damaged. And this is what strace currently gives me (2012-11-07_1030) lseek(4, 212460343296, SEEK_SET) = 212460343296 read(4, "\315Mb\265v\377Gn \24\f\205EHh\2349~\330\273\203\3375\206\10\r3=W\210\372\352"..., 4096) = 4096 lseek(4, 47347830784, SEEK_SET) = 47347830784 write(4, "]\204\223\300I\357\4\26\33+\243\312G\230\250\371*m2U\t_\215\265J \252\342Pm\360D"..., 4096) = 4096 (times are in CET)

  Read the article

• High CPU usage with Team Speak 3.0.0-rc2

  - by AlexTheBird

  The CPU usage is always around 40 percent. I use push-to-talk and I had uninstalled pulseaudio. Now I use Alsa. I don't even have to connect to a Server. By simply starting TS the cpu usage goes up 40 percent and stays there. The CPU usage of 3.0.0-rc1 [Build: 14468] is constantly 14 percent. This is the output of top, mpstat and ps aux while I am running TS3 ... of course: alexandros@alexandros-laptop:~$ top top - 18:20:07 up 2:22, 3 users, load average: 1.02, 0.85, 0.77 Tasks: 163 total, 1 running, 162 sleeping, 0 stopped, 0 zombie Cpu(s): 5.3%us, 1.9%sy, 0.1%ni, 91.8%id, 0.7%wa, 0.1%hi, 0.1%si, 0.0%st Mem: 2061344k total, 964028k used, 1097316k free, 69116k buffers Swap: 3997688k total, 0k used, 3997688k free, 449032k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2714 alexandr 20 0 206m 31m 24m S 37 1.6 0:12.78 ts3client_linux 868 root 20 0 47564 27m 10m S 8 1.4 3:21.73 Xorg 1 root 20 0 2804 1660 1204 S 0 0.1 0:00.53 init 2 root 20 0 0 0 0 S 0 0.0 0:00.00 kthreadd 3 root RT 0 0 0 0 S 0 0.0 0:00.01 migration/0 4 root 20 0 0 0 0 S 0 0.0 0:00.45 ksoftirqd/0 5 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/0 6 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/1 7 root 20 0 0 0 0 S 0 0.0 0:00.08 ksoftirqd/1 8 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/1 9 root 20 0 0 0 0 S 0 0.0 0:01.17 events/0 10 root 20 0 0 0 0 S 0 0.0 0:00.81 events/1 11 root 20 0 0 0 0 S 0 0.0 0:00.00 cpuset 12 root 20 0 0 0 0 S 0 0.0 0:00.00 khelper 13 root 20 0 0 0 0 S 0 0.0 0:00.00 async/mgr 14 root 20 0 0 0 0 S 0 0.0 0:00.00 pm 16 root 20 0 0 0 0 S 0 0.0 0:00.00 sync_supers 17 root 20 0 0 0 0 S 0 0.0 0:00.00 bdi-default 18 root 20 0 0 0 0 S 0 0.0 0:00.00 kintegrityd/0 19 root 20 0 0 0 0 S 0 0.0 0:00.00 kintegrityd/1 20 root 20 0 0 0 0 S 0 0.0 0:00.05 kblockd/0 21 root 20 0 0 0 0 S 0 0.0 0:00.02 kblockd/1 22 root 20 0 0 0 0 S 0 0.0 0:00.00 kacpid 23 root 20 0 0 0 0 S 0 0.0 0:00.00 kacpi_notify 24 root 20 0 0 0 0 S 0 0.0 0:00.00 kacpi_hotplug 25 root 20 0 0 0 0 S 0 0.0 0:00.99 ata/0 26 root 20 0 0 0 0 S 0 0.0 0:00.92 ata/1 27 root 20 0 0 0 0 S 0 0.0 0:00.00 ata_aux 28 root 20 0 0 0 0 S 0 0.0 0:00.00 ksuspend_usbd 29 root 20 0 0 0 0 S 0 0.0 0:00.00 khubd alexandros@alexandros-laptop:~$ mpstat Linux 2.6.32-32-generic (alexandros-laptop) 16.06.2011 _i686_ (2 CPU) 18:20:15 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle 18:20:15 all 5,36 0,09 1,91 0,68 0,07 0,06 0,00 0,00 91,83 alexandros@alexandros-laptop:~$ ps aux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.0 2804 1660 ? Ss 15:58 0:00 /sbin/init root 2 0.0 0.0 0 0 ? S 15:58 0:00 [kthreadd] root 3 0.0 0.0 0 0 ? S 15:58 0:00 [migration/0] root 4 0.0 0.0 0 0 ? S 15:58 0:00 [ksoftirqd/0] root 5 0.0 0.0 0 0 ? S 15:58 0:00 [watchdog/0] root 6 0.0 0.0 0 0 ? S 15:58 0:00 [migration/1] root 7 0.0 0.0 0 0 ? S 15:58 0:00 [ksoftirqd/1] root 8 0.0 0.0 0 0 ? S 15:58 0:00 [watchdog/1] root 9 0.0 0.0 0 0 ? S 15:58 0:01 [events/0] root 10 0.0 0.0 0 0 ? S 15:58 0:00 [events/1] root 11 0.0 0.0 0 0 ? S 15:58 0:00 [cpuset] root 12 0.0 0.0 0 0 ? S 15:58 0:00 [khelper] root 13 0.0 0.0 0 0 ? S 15:58 0:00 [async/mgr] root 14 0.0 0.0 0 0 ? S 15:58 0:00 [pm] root 16 0.0 0.0 0 0 ? S 15:58 0:00 [sync_supers] root 17 0.0 0.0 0 0 ? S 15:58 0:00 [bdi-default] root 18 0.0 0.0 0 0 ? S 15:58 0:00 [kintegrityd/0] root 19 0.0 0.0 0 0 ? S 15:58 0:00 [kintegrityd/1] root 20 0.0 0.0 0 0 ? S 15:58 0:00 [kblockd/0] root 21 0.0 0.0 0 0 ? S 15:58 0:00 [kblockd/1] root 22 0.0 0.0 0 0 ? S 15:58 0:00 [kacpid] root 23 0.0 0.0 0 0 ? S 15:58 0:00 [kacpi_notify] root 24 0.0 0.0 0 0 ? S 15:58 0:00 [kacpi_hotplug] root 25 0.0 0.0 0 0 ? S 15:58 0:00 [ata/0] root 26 0.0 0.0 0 0 ? S 15:58 0:00 [ata/1] root 27 0.0 0.0 0 0 ? S 15:58 0:00 [ata_aux] root 28 0.0 0.0 0 0 ? S 15:58 0:00 [ksuspend_usbd] root 29 0.0 0.0 0 0 ? S 15:58 0:00 [khubd] root 30 0.0 0.0 0 0 ? S 15:58 0:00 [kseriod] root 31 0.0 0.0 0 0 ? S 15:58 0:00 [kmmcd] root 34 0.0 0.0 0 0 ? S 15:58 0:00 [khungtaskd] root 35 0.0 0.0 0 0 ? S 15:58 0:00 [kswapd0] root 36 0.0 0.0 0 0 ? SN 15:58 0:00 [ksmd] root 37 0.0 0.0 0 0 ? S 15:58 0:00 [aio/0] root 38 0.0 0.0 0 0 ? S 15:58 0:00 [aio/1] root 39 0.0 0.0 0 0 ? S 15:58 0:00 [ecryptfs-kthrea] root 40 0.0 0.0 0 0 ? S 15:58 0:00 [crypto/0] root 41 0.0 0.0 0 0 ? S 15:58 0:00 [crypto/1] root 48 0.0 0.0 0 0 ? S 15:58 0:03 [scsi_eh_0] root 50 0.0 0.0 0 0 ? S 15:58 0:00 [scsi_eh_1] root 53 0.0 0.0 0 0 ? S 15:58 0:00 [kstriped] root 54 0.0 0.0 0 0 ? S 15:58 0:00 [kmpathd/0] root 55 0.0 0.0 0 0 ? S 15:58 0:00 [kmpathd/1] root 56 0.0 0.0 0 0 ? S 15:58 0:00 [kmpath_handlerd] root 57 0.0 0.0 0 0 ? S 15:58 0:00 [ksnapd] root 58 0.0 0.0 0 0 ? S 15:58 0:03 [kondemand/0] root 59 0.0 0.0 0 0 ? S 15:58 0:02 [kondemand/1] root 60 0.0 0.0 0 0 ? S 15:58 0:00 [kconservative/0] root 61 0.0 0.0 0 0 ? S 15:58 0:00 [kconservative/1] root 213 0.0 0.0 0 0 ? S 15:58 0:00 [scsi_eh_2] root 222 0.0 0.0 0 0 ? S 15:58 0:00 [scsi_eh_3] root 234 0.0 0.0 0 0 ? S 15:58 0:00 [scsi_eh_4] root 235 0.0 0.0 0 0 ? S 15:58 0:01 [usb-storage] root 255 0.0 0.0 0 0 ? S 15:58 0:00 [jbd2/sda5-8] root 256 0.0 0.0 0 0 ? S 15:58 0:00 [ext4-dio-unwrit] root 257 0.0 0.0 0 0 ? S 15:58 0:00 [ext4-dio-unwrit] root 290 0.0 0.0 0 0 ? S 15:58 0:00 [flush-8:0] root 318 0.0 0.0 2316 888 ? S 15:58 0:00 upstart-udev-bridge --daemon root 321 0.0 0.0 2616 1024 ? S<s 15:58 0:00 udevd --daemon root 526 0.0 0.0 0 0 ? S 15:58 0:00 [kpsmoused] root 528 0.0 0.0 0 0 ? S 15:58 0:00 [led_workqueue] root 650 0.0 0.0 0 0 ? S 15:58 0:00 [radeon/0] root 651 0.0 0.0 0 0 ? S 15:58 0:00 [radeon/1] root 652 0.0 0.0 0 0 ? S 15:58 0:00 [ttm_swap] root 654 0.0 0.0 2612 984 ? S< 15:58 0:00 udevd --daemon root 656 0.0 0.0 0 0 ? S 15:58 0:00 [hd-audio0] root 657 0.0 0.0 2612 916 ? S< 15:58 0:00 udevd --daemon root 674 0.6 0.0 0 0 ? S 15:58 0:57 [phy0] syslog 715 0.0 0.0 34812 1776 ? Sl 15:58 0:00 rsyslogd -c4 102 731 0.0 0.0 3236 1512 ? Ss 15:58 0:02 dbus-daemon --system --fork root 740 0.0 0.1 19088 3380 ? Ssl 15:58 0:00 gdm-binary root 744 0.0 0.1 18900 4032 ? Ssl 15:58 0:01 NetworkManager avahi 749 0.0 0.0 2928 1520 ? S 15:58 0:00 avahi-daemon: running [alexandros-laptop.local] avahi 752 0.0 0.0 2928 544 ? Ss 15:58 0:00 avahi-daemon: chroot helper root 753 0.0 0.1 4172 2300 ? S 15:58 0:00 /usr/sbin/modem-manager root 762 0.0 0.1 20584 3152 ? Sl 15:58 0:00 /usr/sbin/console-kit-daemon --no-daemon root 836 0.0 0.1 20856 3864 ? Sl 15:58 0:00 /usr/lib/gdm/gdm-simple-slave --display-id /org/gnome/DisplayManager/Display1 root 856 0.0 0.1 4836 2388 ? S 15:58 0:00 /sbin/wpa_supplicant -u -s root 868 2.3 1.3 36932 27924 tty7 Rs+ 15:58 3:22 /usr/bin/X :0 -nr -verbose -auth /var/run/gdm/auth-for-gdm-a46T4j/database -nolisten root 891 0.0 0.0 1792 564 tty4 Ss+ 15:58 0:00 /sbin/getty -8 38400 tty4 root 901 0.0 0.0 1792 564 tty5 Ss+ 15:58 0:00 /sbin/getty -8 38400 tty5 root 908 0.0 0.0 1792 564 tty2 Ss+ 15:58 0:00 /sbin/getty -8 38400 tty2 root 910 0.0 0.0 1792 568 tty3 Ss+ 15:58 0:00 /sbin/getty -8 38400 tty3 root 913 0.0 0.0 1792 564 tty6 Ss+ 15:58 0:00 /sbin/getty -8 38400 tty6 root 917 0.0 0.0 2180 1072 ? Ss 15:58 0:00 acpid -c /etc/acpi/events -s /var/run/acpid.socket daemon 924 0.0 0.0 2248 432 ? Ss 15:58 0:00 atd root 927 0.0 0.0 2376 900 ? Ss 15:58 0:00 cron root 950 0.0 0.0 11736 1372 ? Ss 15:58 0:00 /usr/sbin/winbindd root 958 0.0 0.0 11736 1184 ? S 15:58 0:00 /usr/sbin/winbindd root 974 0.0 0.1 6832 2580 ? Ss 15:58 0:00 /usr/sbin/cupsd -C /etc/cups/cupsd.conf root 1078 0.0 0.0 1792 564 tty1 Ss+ 15:58 0:00 /sbin/getty -8 38400 tty1 gdm 1097 0.0 0.0 3392 772 ? S 15:58 0:00 /usr/bin/dbus-launch --exit-with-session root 1112 0.0 0.1 19216 3292 ? Sl 15:58 0:00 /usr/lib/gdm/gdm-session-worker root 1116 0.0 0.1 5540 2932 ? S 15:58 0:01 /usr/lib/upower/upowerd root 1131 0.0 0.1 6308 3824 ? S 15:58 0:00 /usr/lib/policykit-1/polkitd 108 1163 0.0 0.2 16788 4360 ? Ssl 15:58 0:01 /usr/sbin/hald root 1164 0.0 0.0 3536 1300 ? S 15:58 0:00 hald-runner root 1188 0.0 0.0 3612 1256 ? S 15:58 0:00 hald-addon-input: Listening on /dev/input/event6 /dev/input/event5 /dev/input/event2 root 1194 0.0 0.0 3612 1224 ? S 15:58 0:00 /usr/lib/hal/hald-addon-rfkill-killswitch root 1200 0.0 0.0 3608 1240 ? S 15:58 0:00 /usr/lib/hal/hald-addon-generic-backlight root 1202 0.0 0.0 3616 1236 ? S 15:58 0:02 hald-addon-storage: polling /dev/sr0 (every 2 sec) root 1204 0.0 0.0 3616 1236 ? S 15:58 0:00 hald-addon-storage: polling /dev/sdb (every 2 sec) root 1211 0.0 0.0 3624 1220 ? S 15:58 0:00 /usr/lib/hal/hald-addon-cpufreq 108 1212 0.0 0.0 3420 1200 ? S 15:58 0:00 hald-addon-acpi: listening on acpid socket /var/run/acpid.socket 1000 1222 0.0 0.1 24196 2816 ? Sl 15:58 0:00 /usr/bin/gnome-keyring-daemon --daemonize --login 1000 1240 0.0 0.3 28228 7312 ? Ssl 15:58 0:00 gnome-session 1000 1274 0.0 0.0 3284 356 ? Ss 15:58 0:00 /usr/bin/ssh-agent /usr/bin/dbus-launch --exit-with-session gnome-session 1000 1277 0.0 0.0 3392 772 ? S 15:58 0:00 /usr/bin/dbus-launch --exit-with-session gnome-session 1000 1278 0.0 0.0 3160 1652 ? Ss 15:58 0:00 /bin/dbus-daemon --fork --print-pid 5 --print-address 7 --session 1000 1281 0.0 0.2 8172 4636 ? S 15:58 0:00 /usr/lib/libgconf2-4/gconfd-2 1000 1287 0.0 0.5 24228 10896 ? Ss 15:58 0:03 /usr/lib/gnome-settings-daemon/gnome-settings-daemon 1000 1290 0.0 0.1 6468 2364 ? S 15:58 0:00 /usr/lib/gvfs/gvfsd 1000 1293 0.0 0.6 38104 13004 ? S 15:58 0:03 metacity 1000 1296 0.0 0.1 30280 2628 ? Ssl 15:58 0:00 /usr/lib/gvfs//gvfs-fuse-daemon /home/alexandros/.gvfs 1000 1301 0.0 0.0 3344 988 ? S 15:58 0:03 syndaemon -i 0.5 -k 1000 1303 0.0 0.1 8060 3488 ? S 15:58 0:00 /usr/lib/gvfs/gvfs-gdu-volume-monitor root 1306 0.0 0.1 15692 3104 ? Sl 15:58 0:00 /usr/lib/udisks/udisks-daemon 1000 1307 0.4 1.0 50748 21684 ? S 15:58 0:34 python -u /usr/share/screenlets/DigiClock/DigiClockScreenlet.py 1000 1308 0.0 0.9 35608 18564 ? S 15:58 0:00 python /usr/share/screenlets-manager/screenlets-daemon.py 1000 1309 0.0 0.3 19524 6468 ? S 15:58 0:00 /usr/lib/policykit-1-gnome/polkit-gnome-authentication-agent-1 1000 1311 0.0 0.5 37412 11788 ? S 15:58 0:01 gnome-power-manager 1000 1312 0.0 1.0 50772 22628 ? S 15:58 0:03 gnome-panel 1000 1313 0.1 1.5 102648 31184 ? Sl 15:58 0:10 nautilus root 1314 0.0 0.0 5188 996 ? S 15:58 0:02 udisks-daemon: polling /dev/sdb /dev/sr0 1000 1315 0.0 0.6 51948 12464 ? SL 15:58 0:01 nm-applet --sm-disable 1000 1317 0.0 0.1 16956 2364 ? Sl 15:58 0:00 /usr/lib/gvfs/gvfs-afc-volume-monitor 1000 1318 0.0 0.3 20164 7792 ? S 15:58 0:00 bluetooth-applet 1000 1321 0.0 0.1 7260 2384 ? S 15:58 0:00 /usr/lib/gvfs/gvfs-gphoto2-volume-monitor 1000 1323 0.0 0.5 37436 12124 ? S 15:58 0:00 /usr/lib/notify-osd/notify-osd 1000 1324 0.0 1.9 197928 40456 ? Ssl 15:58 0:06 /home/alexandros/.dropbox-dist/dropbox 1000 1329 0.0 0.3 20136 7968 ? S 15:58 0:00 /usr/bin/gnome-screensaver --no-daemon 1000 1331 0.0 0.1 7056 3112 ? S 15:58 0:00 /usr/lib/gvfs/gvfsd-trash --spawner :1.6 /org/gtk/gvfs/exec_spaw/0 root 1340 0.0 0.0 2236 1008 ? S 15:58 0:00 /sbin/dhclient -d -sf /usr/lib/NetworkManager/nm-dhcp-client.action -pf /var/run/dhcl 1000 1348 0.0 0.1 42252 3680 ? Ssl 15:58 0:00 /usr/lib/bonobo-activation/bonobo-activation-server --ac-activate --ior-output-fd=19 1000 1384 0.0 1.7 80244 35480 ? Sl 15:58 0:02 /usr/bin/python /usr/lib/deskbar-applet/deskbar-applet/deskbar-applet --oaf-activate- 1000 1388 0.0 0.5 26196 11804 ? S 15:58 0:01 /usr/lib/gnome-panel/wnck-applet --oaf-activate-iid=OAFIID:GNOME_Wncklet_Factory --oa 1000 1393 0.1 0.5 25876 11548 ? S 15:58 0:08 /usr/lib/gnome-applets/multiload-applet-2 --oaf-activate-iid=OAFIID:GNOME_MultiLoadAp 1000 1394 0.0 0.5 25600 11140 ? S 15:58 0:03 /usr/lib/gnome-applets/cpufreq-applet --oaf-activate-iid=OAFIID:GNOME_CPUFreqApplet_F 1000 1415 0.0 0.5 39192 11156 ? S 15:58 0:01 /usr/lib/gnome-power-manager/gnome-inhibit-applet --oaf-activate-iid=OAFIID:GNOME_Inh 1000 1417 0.0 0.7 53544 15488 ? Sl 15:58 0:00 /usr/lib/gnome-applets/mixer_applet2 --oaf-activate-iid=OAFIID:GNOME_MixerApplet_Fact 1000 1419 0.0 0.4 23816 9068 ? S 15:58 0:00 /usr/lib/gnome-panel/notification-area-applet --oaf-activate-iid=OAFIID:GNOME_Notific 1000 1488 0.0 0.3 20964 7548 ? S 15:58 0:00 /usr/lib/gnome-disk-utility/gdu-notification-daemon 1000 1490 0.0 0.1 6608 2484 ? S 15:58 0:00 /usr/lib/gvfs/gvfsd-burn --spawner :1.6 /org/gtk/gvfs/exec_spaw/1 1000 1510 0.0 0.1 6348 2084 ? S 15:58 0:00 /usr/lib/gvfs/gvfsd-metadata 1000 1531 0.0 0.3 19472 6616 ? S 15:58 0:00 /usr/lib/gnome-user-share/gnome-user-share 1000 1535 0.0 0.4 77128 8392 ? Sl 15:58 0:00 /usr/lib/evolution/evolution-data-server-2.28 --oaf-activate-iid=OAFIID:GNOME_Evoluti 1000 1601 0.0 0.5 69576 11800 ? Sl 15:59 0:00 /usr/lib/evolution/2.28/evolution-alarm-notify 1000 1604 0.0 0.7 33924 15888 ? S 15:59 0:00 python /usr/share/system-config-printer/applet.py 1000 1701 0.0 0.5 37116 11968 ? S 15:59 0:00 update-notifier 1000 1892 4.5 7.0 406720 145312 ? Sl 17:11 3:09 /opt/google/chrome/chrome 1000 1896 0.0 0.1 69812 3680 ? S 17:11 0:02 /opt/google/chrome/chrome 1000 1898 0.0 0.6 91420 14080 ? S 17:11 0:00 /opt/google/chrome/chrome --type=zygote 1000 1916 0.2 1.3 140780 27220 ? Sl 17:11 0:12 /opt/google/chrome/chrome --type=extension --disable-client-side-phishing-detection - 1000 1918 0.7 1.8 155720 37912 ? Sl 17:11 0:31 /opt/google/chrome/chrome --type=extension --disable-client-side-phishing-detection - 1000 1921 0.0 1.0 135904 21052 ? Sl 17:11 0:02 /opt/google/chrome/chrome --type=extension --disable-client-side-phishing-detection - 1000 1927 6.5 3.6 194604 74960 ? Sl 17:11 4:32 /opt/google/chrome/chrome --type=renderer --disable-client-side-phishing-detection -- 1000 2156 0.4 0.7 48344 14896 ? Rl 18:03 0:04 gnome-terminal 1000 2157 0.0 0.0 1988 712 ? S 18:03 0:00 gnome-pty-helper 1000 2158 0.0 0.1 6504 3860 pts/0 Ss 18:03 0:00 bash 1000 2564 0.2 0.1 6624 3984 pts/1 Ss+ 18:17 0:00 bash 1000 2711 0.0 0.0 4208 1352 ? S 18:19 0:00 /bin/bash /home/alexandros/Programme/TeamSpeak3-Client-linux_x86_back/ts3client_runsc 1000 2714 36.5 1.5 210872 31960 ? SLl 18:19 0:18 ./ts3client_linux_x86 1000 2743 0.0 0.0 2716 1068 pts/0 R+ 18:20 0:00 ps aux Output of vmstat: alexandros@alexandros-laptop:~$ vmstat procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 0 1093324 69840 449496 0 0 27 10 476 667 6 2 91 1 Output of lsusb alexandros@alexandros-laptop:~$ lspci 00:00.0 Host bridge: Silicon Integrated Systems [SiS] 671MX 00:01.0 PCI bridge: Silicon Integrated Systems [SiS] PCI-to-PCI bridge 00:02.0 ISA bridge: Silicon Integrated Systems [SiS] SiS968 [MuTIOL Media IO] (rev 01) 00:02.5 IDE interface: Silicon Integrated Systems [SiS] 5513 [IDE] (rev 01) 00:03.0 USB Controller: Silicon Integrated Systems [SiS] USB 1.1 Controller (rev 0f) 00:03.1 USB Controller: Silicon Integrated Systems [SiS] USB 1.1 Controller (rev 0f) 00:03.3 USB Controller: Silicon Integrated Systems [SiS] USB 2.0 Controller 00:05.0 IDE interface: Silicon Integrated Systems [SiS] SATA Controller / IDE mode (rev 03) 00:06.0 PCI bridge: Silicon Integrated Systems [SiS] PCI-to-PCI bridge 00:07.0 PCI bridge: Silicon Integrated Systems [SiS] PCI-to-PCI bridge 00:0d.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+ (rev 10) 00:0f.0 Audio device: Silicon Integrated Systems [SiS] Azalia Audio Controller 01:00.0 VGA compatible controller: ATI Technologies Inc Mobility Radeon X2300 02:00.0 Ethernet controller: Atheros Communications Inc. AR5001 Wireless Network Adapter (rev 01) The Team Speak log file : 2011-06-19 19:04:04.223522|INFO | | | Logging started, clientlib version: 3.0.0-rc2 [Build: 14642] 2011-06-19 19:04:04.761149|ERROR |SoundBckndIntf| | /home/alexandros/Programme/TeamSpeak3-Client-linux_x86_back/soundbackends/libpulseaudio_linux_x86.so error: NOT_CONNECTED 2011-06-19 19:04:05.871770|INFO |ClientUI | | Failed to init text to speech engine 2011-06-19 19:04:05.894623|INFO |ClientUI | | TeamSpeak 3 client version: 3.0.0-rc2 [Build: 14642] 2011-06-19 19:04:05.895421|INFO |ClientUI | | Qt version: 4.7.2 2011-06-19 19:04:05.895571|INFO |ClientUI | | Using configuration location: /home/alexandros/.ts3client/ts3clientui_qt.conf 2011-06-19 19:04:06.559596|INFO |ClientUI | | Last update check was: Sa. Jun 18 00:08:43 2011 2011-06-19 19:04:06.560506|INFO | | | Checking for updates... 2011-06-19 19:04:07.357869|INFO | | | Update check, my version: 14642, latest version: 14642 2011-06-19 19:05:52.978481|INFO |PreProSpeex | 1| Speex version: 1.2rc1 2011-06-19 19:05:54.055347|INFO |UIHelpers | | setClientVolumeModifier: 10 -8 2011-06-19 19:05:54.057196|INFO |UIHelpers | | setClientVolumeModifier: 11 2 Thanks for taking the time to read my message. UPDATE: Thanks to nickguletskii's link I googled for "alsa cpu usage" (without quotes) and it brought me to a forum. A user wrote that by directly selecting the hardware with "plughw:x.x" won't impact the performance of the system. I have selected it in the TS 3 configuration and it worked. But this solution is not optimal because now no other program can access the sound output. If you need any further information or my question is unclear than please tell me.

  Read the article

• MPM Prefork Apache Uses Absurd Amount of Memory

  - by Charlie JM

  Help! My apache processes are all using 115MB of memory on startup. Relevant information: Linux version (uname -a) Linux 2.6.31-14-generic-pae #48-Ubuntu SMP Fri Oct 16 15:22:42 UTC 2009 i686 GNU/Linux Apache version (/usr/sbin/apache2 -v) Server version: Apache/2.2.8 (Ubuntu) Server built: Mar 9 2010 20:45:36 Top display (top -u www-data) PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 23377 www-data 20 0 115m 94m 3908 S 28 1.6 0:04.59 apache2 23375 www-data 20 0 119m 99m 5892 S 9 1.6 0:05.04 apache2 23324 www-data 20 0 116m 96m 5144 S 2 1.6 0:04.73 apache2 23283 www-data 20 0 115m 95m 4480 S 1 1.6 0:04.89 apache2 23259 www-data 20 0 116m 96m 5380 S 0 1.6 0:05.55 apache2 23370 www-data 20 0 115m 94m 4396 S 0 1.6 0:04.75 apache2 23229 www-data 20 0 116m 96m 6096 S 0 1.6 0:05.43 apache2 ... and so on ... Memory map (pmap $(pidof apache2)) (actually, just one apache2 process) Most of the memory is [anon], see line 5 23324: /usr/sbin/apache2 -k start 08048000 332K r-x-- /usr/sbin/apache2 0809b000 8K rw--- /usr/sbin/apache2 0809d000 12K rw--- [ anon ] 093a0000 92812K rw--- [ anon ] b5b6c000 4K rw--- [ anon ] b5b6d000 512K rw-s- [ shmid=0x13528003 ] b5fa8000 16K r-x-- /lib/tls/i686/cmov/libnss_dns-2.7.so b5fac000 8K rw--- /lib/tls/i686/cmov/libnss_dns-2.7.so b5fae000 120K r-x-- /usr/lib/php5/20060613+lfs/suhosin.so b5fcc000 16K rw--- /usr/lib/php5/20060613+lfs/suhosin.so b5fd0000 4K rw--- [ anon ] b5fd1000 76K r-x-- /usr/lib/php5/20060613+lfs/pdo.so b5fe4000 8K rw--- /usr/lib/php5/20060613+lfs/pdo.so b5fe6000 92K r-x-- /usr/lib/php5/20060613+lfs/mysqli.so b5ffd000 8K rw--- /usr/lib/php5/20060613+lfs/mysqli.so b5fff000 1648K r-x-- /usr/lib/libmysqlclient.so.15.0.0 b619b000 268K rw--- /usr/lib/libmysqlclient.so.15.0.0 b61de000 4K rw--- [ anon ] b61f0000 92K r-x-- /usr/lib/libxcb.so.1.0.0 b6207000 4K rw--- /usr/lib/libxcb.so.1.0.0 b6208000 164K r-x-- /usr/lib/libfontconfig.so.1.3.0 b6231000 4K rw--- /usr/lib/libfontconfig.so.1.3.0 b6232000 124K r-x-- /usr/lib/libjpeg.so.62.0.0 b6251000 4K rw--- /usr/lib/libjpeg.so.62.0.0 b6252000 136K r-x-- /usr/lib/libpng12.so.0.15.0 b6274000 4K rw--- /usr/lib/libpng12.so.0.15.0 b6275000 60K r-x-- /usr/lib/libXpm.so.4.11.0 b6284000 4K rw--- /usr/lib/libXpm.so.4.11.0 b6285000 912K r-x-- /usr/lib/libX11.so.6.2.0 b6369000 12K rw--- /usr/lib/libX11.so.6.2.0 b636c000 424K r-x-- /usr/lib/libfreetype.so.6.3.16 b63d6000 12K rw--- /usr/lib/libfreetype.so.6.3.16 b63d9000 236K r-x-- /usr/lib/libt1.so.5.1.1 b6414000 12K rw--- /usr/lib/libt1.so.5.1.1 b6417000 84K rw--- [ anon ] b642c000 116K r-x-- /usr/lib/libgd.so.2.0.0 b6449000 128K rw--- /usr/lib/libgd.so.2.0.0 b6469000 16K rw--- [ anon ] b646d000 88K r-x-- /usr/lib/php5/20060613+lfs/gd.so b6483000 16K rw--- /usr/lib/php5/20060613+lfs/gd.so b6487000 192K r-x-- /usr/lib/libidn.so.11.5.30 b64b7000 4K rw--- /usr/lib/libidn.so.11.5.30 b64b8000 232K r-x-- /usr/lib/libcurl.so.4.0.1 b64f2000 4K rw--- /usr/lib/libcurl.so.4.0.1 b64f8000 44K r-x-- /usr/lib/php5/20060613+lfs/mysql.so b6503000 4K rw--- /usr/lib/php5/20060613+lfs/mysql.so b6504000 268K r-x-- /usr/lib/libgmp.so.3.4.2 b6547000 4K rw--- /usr/lib/libgmp.so.3.4.2 b6548000 648K r-x-- /usr/lib/libclamav.so.5.0.4 b65ea000 44K rw--- /usr/lib/libclamav.so.5.0.4 b65f8000 52K r-x-- /usr/lib/php5/20060613+lfs/curl.so b6605000 4K rw--- /usr/lib/php5/20060613+lfs/curl.so b6606000 148K r-x-- /usr/lib/libmcrypt.so.4.4.7 b662b000 8K rw--- /usr/lib/libmcrypt.so.4.4.7 b662d000 28K rw--- [ anon ] b6634000 24K r-x-- /usr/lib/php5/20060613+lfs/pdo_mysql.so b663a000 4K rw--- /usr/lib/php5/20060613+lfs/pdo_mysql.so b663b000 16K r-x-- /usr/lib/libXdmcp.so.6.0.0 b663f000 4K rw--- /usr/lib/libXdmcp.so.6.0.0 b6640000 12K r-x-- /usr/lib/php5/20060613+lfs/clamav.so b6643000 4K rw--- /usr/lib/php5/20060613+lfs/clamav.so b6644000 1036K r-x-- /usr/lib/libc-client.so.2007.0 b6747000 28K rw--- /usr/lib/libc-client.so.2007.0 b674e000 4K rw--- [ anon ] b6750000 24K r-x-- /usr/lib/libltdl.so.3.1.6 b6756000 4K rw--- /usr/lib/libltdl.so.3.1.6 b6757000 32K r-x-- /usr/lib/php5/20060613+lfs/mcrypt.so b675f000 4K rw--- /usr/lib/php5/20060613+lfs/mcrypt.so b6760000 88K r-x-- /usr/lib/php5/20060613+lfs/imap.so b6776000 4K rw--- /usr/lib/php5/20060613+lfs/imap.so b6777000 104K r-x-- /usr/local/lib/libssh2.so b6791000 4K rw--- /usr/local/lib/libssh2.so b6792000 1324K r-x-- /usr/lib/ZendOptimizer.so b68dd000 68K rw--- /usr/lib/ZendOptimizer.so b68ee000 20K rw--- [ anon ] b68f3000 8K r-x-- /usr/lib/libXau.so.6.0.0 b68f5000 4K rw--- /usr/lib/libXau.so.6.0.0 b68f6000 52K r-x-- /usr/lib/php5/20060613+lfs/ssh2.so b6903000 4K rw--- /usr/lib/php5/20060613+lfs/ssh2.so b6904000 252K r---- /usr/lib/locale/en_US.utf8/LC_CTYPE b6974000 64K rw-s- /dev/zero (deleted) b6984000 36K r-x-- /lib/tls/i686/cmov/libnss_files-2.7.so b698d000 8K rw--- /lib/tls/i686/cmov/libnss_files-2.7.so b698f000 32K r-x-- /lib/tls/i686/cmov/libnss_nis-2.7.so b6997000 8K rw--- /lib/tls/i686/cmov/libnss_nis-2.7.so b6999000 28K r-x-- /lib/tls/i686/cmov/libnss_compat-2.7.so b69a0000 8K rw--- /lib/tls/i686/cmov/libnss_compat-2.7.so b69a2000 36K r-x-- /lib/libpam.so.0.81.6 b69ab000 4K rw--- /lib/libpam.so.0.81.6 b69ac000 28K r--s- /usr/lib/gconv/gconv-modules.cache b69b3000 8K r-x-- /usr/lib/apache2/modules/mod_userdir.so b69b5000 4K rw--- /usr/lib/apache2/modules/mod_userdir.so b69b6000 148K r-x-- /usr/lib/apache2/modules/mod_ssl.so b69db000 8K rw--- /usr/lib/apache2/modules/mod_ssl.so b69dd000 8K rw--- [ anon ] b69df000 8K r-x-- /usr/lib/apache2/modules/mod_setenvif.so b69e1000 4K rw--- /usr/lib/apache2/modules/mod_setenvif.so b69e2000 1128K r-x-- /usr/lib/libxml2.so.2.6.31 b6afc000 20K rw--- /usr/lib/libxml2.so.2.6.31 b6b01000 4K rw--- [ anon ] b6b02000 80K r-x-- /lib/tls/i686/cmov/libnsl-2.7.so b6b16000 8K rw--- /lib/tls/i686/cmov/libnsl-2.7.so b6b18000 8K rw--- [ anon ] b6b1a000 140K r-x-- /lib/tls/i686/cmov/libm-2.7.so b6b3d000 8K rw--- /lib/tls/i686/cmov/libm-2.7.so b6b3f000 60K r-x-- /lib/libbz2.so.1.0.4 b6b4e000 4K rw--- /lib/libbz2.so.1.0.4 b6b4f000 4K r-x-- /usr/lib/libxcb-xlib.so.0.0.0 b6b50000 4K rw--- /usr/lib/libxcb-xlib.so.0.0.0 b6b51000 56K r-x-- /usr/lib/apache2/modules/mod_rewrite.so b6b5f000 4K rw--- /usr/lib/apache2/modules/mod_rewrite.so b6b60000 5060K r-x-- /usr/lib/apache2/modules/libphp5.so b7051000 208K rw--- /usr/lib/apache2/modules/libphp5.so b7085000 20K rw--- [ anon ] b708a000 28K r-x-- /usr/lib/apache2/modules/mod_negotiation.so b7091000 4K rw--- /usr/lib/apache2/modules/mod_negotiation.so b7092000 12K r-x-- /usr/lib/apache2/modules/mod_mime.so b7095000 4K rw--- /usr/lib/apache2/modules/mod_mime.so b7096000 36K r-x-- /usr/lib/apache2/modules/mod_include.so b709f000 4K rw--- /usr/lib/apache2/modules/mod_include.so b70a0000 4K r-x-- /usr/lib/apache2/modules/mod_env.so b70a1000 4K rw--- /usr/lib/apache2/modules/mod_env.so b70a2000 4K r-x-- /usr/lib/apache2/modules/mod_dir.so b70a3000 4K rw--- /usr/lib/apache2/modules/mod_dir.so b70a4000 20K r-x-- /usr/lib/apache2/modules/mod_cgi.so b70a9000 4K rw--- /usr/lib/apache2/modules/mod_cgi.so b70aa000 28K r-x-- /usr/lib/apache2/modules/mod_autoindex.so b70b1000 4K rw--- /usr/lib/apache2/modules/mod_autoindex.so b70b2000 4K r-x-- /usr/lib/apache2/modules/mod_authz_user.so b70b3000 4K rw--- /usr/lib/apache2/modules/mod_authz_user.so b70b4000 8K r-x-- /usr/lib/apache2/modules/mod_authz_host.so b70b6000 4K rw--- /usr/lib/apache2/modules/mod_authz_host.so b70b7000 8K r-x-- /usr/lib/apache2/modules/mod_authz_groupfile.so b70b9000 4K rw--- /usr/lib/apache2/modules/mod_authz_groupfile.so b70ba000 8K rw--- [ anon ] b70bc000 12K r-x-- /lib/libgpg-error.so.0.3.0 b70bf000 4K rw--- /lib/libgpg-error.so.0.3.0 b70c0000 4K rw--- [ anon ] b70c1000 8K r-x-- /lib/libkeyutils-1.2.so b70c3000 4K rw--- /lib/libkeyutils-1.2.so b70c4000 28K r-x-- /usr/lib/libkrb5support.so.0.1 b70cb000 4K rw--- /usr/lib/libkrb5support.so.0.1 b70cc000 136K r-x-- /usr/lib/libk5crypto.so.3.1 b70ee000 4K rw--- /usr/lib/libk5crypto.so.3.1 b70ef000 300K r-x-- /lib/libgcrypt.so.11.2.3 b713a000 8K rw--- /lib/libgcrypt.so.11.2.3 b713c000 80K r-x-- /usr/lib/libz.so.1.2.3.3 b7150000 4K rw--- /usr/lib/libz.so.1.2.3.3 b7151000 4K rw--- [ anon ] b7152000 60K r-x-- /usr/lib/libtasn1.so.3.0.12 b7161000 4K rw--- /usr/lib/libtasn1.so.3.0.12 b7162000 160K r-x-- /usr/lib/libgssapi_krb5.so.2.2 b718a000 4K rw--- /usr/lib/libgssapi_krb5.so.2.2 b718b000 8K r-x-- /lib/libcom_err.so.2.1 b718d000 4K rw--- /lib/libcom_err.so.2.1 b718e000 556K r-x-- /usr/lib/libkrb5.so.3.3 b7219000 8K rw--- /usr/lib/libkrb5.so.3.3 b721b000 1192K r-x-- /usr/lib/i686/cmov/libcrypto.so.0.9.8 b7345000 84K rw--- /usr/lib/i686/cmov/libcrypto.so.0.9.8 b735a000 16K rw--- [ anon ] b735e000 248K r-x-- /usr/lib/i686/cmov/libssl.so.0.9.8 b739c000 16K rw--- /usr/lib/i686/cmov/libssl.so.0.9.8 b73a0000 452K r-x-- /usr/lib/libgnutls.so.13.9.1 b7411000 20K rw--- /usr/lib/libgnutls.so.13.9.1 b7416000 88K r-x-- /usr/lib/libsasl2.so.2.0.22 b742c000 4K rw--- /usr/lib/libsasl2.so.2.0.22 b742d000 60K r-x-- /lib/tls/i686/cmov/libresolv-2.7.so b743c000 8K rw--- /lib/tls/i686/cmov/libresolv-2.7.so b743e000 8K rw--- [ anon ] b7440000 8K r-x-- /lib/tls/i686/cmov/libdl-2.7.so b7442000 8K rw--- /lib/tls/i686/cmov/libdl-2.7.so b7444000 36K r-x-- /lib/tls/i686/cmov/libcrypt-2.7.so b744d000 8K rw--- /lib/tls/i686/cmov/libcrypt-2.7.so b744f000 160K rw--- [ anon ] b7477000 28K r-x-- /lib/tls/i686/cmov/librt-2.7.so b747e000 8K rw--- /lib/tls/i686/cmov/librt-2.7.so b7480000 12K r-x-- /lib/libuuid.so.1.2 b7483000 4K rw--- /lib/libuuid.so.1.2 b7484000 124K r-x-- /usr/lib/libexpat.so.1.5.2 b74a3000 8K rw--- /usr/lib/libexpat.so.1.5.2 b74a5000 396K r-x-- /usr/lib/libsqlite3.so.0.8.6 b7508000 8K rw--- /usr/lib/libsqlite3.so.0.8.6 b750a000 120K r-x-- /usr/lib/libpq.so.5.1 b7528000 4K rw--- /usr/lib/libpq.so.5.1 b7529000 1172K r-x-- /usr/lib/libdb-4.6.so b764e000 8K rw--- /usr/lib/libdb-4.6.so b7650000 4K rw--- [ anon ] b7651000 48K r-x-- /usr/lib/liblber-2.4.so.2.0.5 b765d000 4K rw--- /usr/lib/liblber-2.4.so.2.0.5 b765e000 244K r-x-- /usr/lib/libldap_r-2.4.so.2.0.5 b769b000 4K rw--- /usr/lib/libldap_r-2.4.so.2.0.5 b769c000 8K rw--- [ anon ] b769e000 1316K r-x-- /lib/tls/i686/cmov/libc-2.7.so b77e7000 4K r---- /lib/tls/i686/cmov/libc-2.7.so b77e8000 8K rw--- /lib/tls/i686/cmov/libc-2.7.so b77ea000 12K rw--- [ anon ] b77ed000 80K r-x-- /lib/tls/i686/cmov/libpthread-2.7.so b7801000 8K rw--- /lib/tls/i686/cmov/libpthread-2.7.so b7803000 8K rw--- [ anon ] b7805000 136K r-x-- /usr/lib/libapr-1.so.0.2.11 b7827000 4K rw--- /usr/lib/libapr-1.so.0.2.11 b7828000 4K rw--- [ anon ] b7829000 100K r-x-- /usr/lib/libaprutil-1.so.0.2.11 b7842000 4K rw--- /usr/lib/libaprutil-1.so.0.2.11 b7843000 152K r-x-- /usr/lib/libpcre.so.3.12.1 b7869000 4K rw--- /usr/lib/libpcre.so.3.12.1 b786a000 4K r-x-- /usr/lib/apache2/modules/mod_authz_default.so b786b000 4K rw--- /usr/lib/apache2/modules/mod_authz_default.so b786c000 4K r-x-- /usr/lib/apache2/modules/mod_authn_file.so b786d000 4K rw--- /usr/lib/apache2/modules/mod_authn_file.so b786e000 24K r-x-- /usr/lib/apache2/modules/mod_auth_digest.so b7874000 4K rw--- /usr/lib/apache2/modules/mod_auth_digest.so b7875000 8K r-x-- /usr/lib/apache2/modules/mod_auth_basic.so b7877000 4K rw--- /usr/lib/apache2/modules/mod_auth_basic.so b7878000 8K r-x-- /usr/lib/apache2/modules/mod_alias.so b787a000 4K rw--- /usr/lib/apache2/modules/mod_alias.so b787b000 8K rw--- [ anon ] b787d000 4K r-x-- [ anon ] b787e000 104K r-x-- /lib/ld-2.7.so b7898000 8K rw--- /lib/ld-2.7.so bfd68000 76K rwx-- [ stack ] bfd7b000 8K rw--- [ anon ] total 119008K I have no idea what's going on. I've tried adjusting the usual parameters (MaxClients, MaxRequestsPerClient, etc, but those don't do anything.) Note, also, that this is memory usage on startup - it doesn't grow, it just starts like this and then stays more or less constant. Ideas?

  Read the article

• high load average, high wait, dmesg raid error messages (debian nfs server)

  - by John Stumbles

  Debian 6 on HP proliant (2 CPU) with raid (2*1.5T RAID1 + 2*2T RAID1 joined RAID0 to make 3.5T) running mainly nfs & imapd (plus samba for windows share & local www for previewing web pages); with local ubuntu desktop client mounting $HOME, laptops accessing imap & odd files (e.g. videos) via nfs/smb; boxes connected 100baseT or wifi via home router/switch uname -a Linux prole 2.6.32-5-686 #1 SMP Wed Jan 11 12:29:30 UTC 2012 i686 GNU/Linux Setup has been working for months but prone to intermittently going very slow (user experience on desktop mounting $HOME from server, or laptop playing videos) and now consistently so bad I've had to delve into it to try to find what's wrong(!) Server seems OK at low load e.g. (laptop) client (with $HOME on local disk) connecting to server's imapd and nfs mounting RAID to access 1 file: top shows load ~ 0.1 or less, 0 wait but when (desktop) client mounts $HOME and starts user KDE session (all accessing server) then top shows e.g. top - 13:41:17 up 3:43, 3 users, load average: 9.29, 9.55, 8.27 Tasks: 158 total, 1 running, 157 sleeping, 0 stopped, 0 zombie Cpu(s): 0.4%us, 0.4%sy, 0.0%ni, 49.0%id, 49.7%wa, 0.0%hi, 0.5%si, 0.0%st Mem: 903856k total, 851784k used, 52072k free, 171152k buffers Swap: 0k total, 0k used, 0k free, 476896k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 3935 root 20 0 2456 1088 784 R 2 0.1 0:00.02 top 1 root 20 0 2028 680 584 S 0 0.1 0:01.14 init 2 root 20 0 0 0 0 S 0 0.0 0:00.00 kthreadd 3 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/0 4 root 20 0 0 0 0 S 0 0.0 0:00.12 ksoftirqd/0 5 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/0 6 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/1 7 root 20 0 0 0 0 S 0 0.0 0:00.16 ksoftirqd/1 8 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/1 9 root 20 0 0 0 0 S 0 0.0 0:00.42 events/0 10 root 20 0 0 0 0 S 0 0.0 0:02.26 events/1 11 root 20 0 0 0 0 S 0 0.0 0:00.00 cpuset 12 root 20 0 0 0 0 S 0 0.0 0:00.00 khelper 13 root 20 0 0 0 0 S 0 0.0 0:00.00 netns 14 root 20 0 0 0 0 S 0 0.0 0:00.00 async/mgr 15 root 20 0 0 0 0 S 0 0.0 0:00.00 pm 16 root 20 0 0 0 0 S 0 0.0 0:00.02 sync_supers 17 root 20 0 0 0 0 S 0 0.0 0:00.02 bdi-default 18 root 20 0 0 0 0 S 0 0.0 0:00.00 kintegrityd/0 19 root 20 0 0 0 0 S 0 0.0 0:00.00 kintegrityd/1 20 root 20 0 0 0 0 S 0 0.0 0:00.02 kblockd/0 21 root 20 0 0 0 0 S 0 0.0 0:00.08 kblockd/1 22 root 20 0 0 0 0 S 0 0.0 0:00.00 kacpid 23 root 20 0 0 0 0 S 0 0.0 0:00.00 kacpi_notify 24 root 20 0 0 0 0 S 0 0.0 0:00.00 kacpi_hotplug 25 root 20 0 0 0 0 S 0 0.0 0:00.00 kseriod 28 root 20 0 0 0 0 S 0 0.0 0:04.19 kondemand/0 29 root 20 0 0 0 0 S 0 0.0 0:02.93 kondemand/1 30 root 20 0 0 0 0 S 0 0.0 0:00.00 khungtaskd 31 root 20 0 0 0 0 S 0 0.0 0:00.18 kswapd0 32 root 25 5 0 0 0 S 0 0.0 0:00.00 ksmd 33 root 20 0 0 0 0 S 0 0.0 0:00.00 aio/0 34 root 20 0 0 0 0 S 0 0.0 0:00.00 aio/1 35 root 20 0 0 0 0 S 0 0.0 0:00.00 crypto/0 36 root 20 0 0 0 0 S 0 0.0 0:00.00 crypto/1 203 root 20 0 0 0 0 S 0 0.0 0:00.00 ksuspend_usbd 204 root 20 0 0 0 0 S 0 0.0 0:00.00 khubd 205 root 20 0 0 0 0 S 0 0.0 0:00.00 ata/0 206 root 20 0 0 0 0 S 0 0.0 0:00.00 ata/1 207 root 20 0 0 0 0 S 0 0.0 0:00.14 ata_aux 208 root 20 0 0 0 0 S 0 0.0 0:00.01 scsi_eh_0 dmesg suggests there's a disk problem: .............. (previous episode) [13276.966004] raid1:md0: read error corrected (8 sectors at 489900360 on sdc7) [13276.966043] raid1: sdb7: redirecting sector 489898312 to another mirror [13279.569186] ata4.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0 [13279.569211] ata4.00: irq_stat 0x40000008 [13279.569230] ata4.00: failed command: READ FPDMA QUEUED [13279.569257] ata4.00: cmd 60/08:00:00:6a:05/00:00:23:00:00/40 tag 0 ncq 4096 in [13279.569262] res 41/40:00:05:6a:05/00:00:23:00:00/40 Emask 0x409 (media error) <F> [13279.569306] ata4.00: status: { DRDY ERR } [13279.569321] ata4.00: error: { UNC } [13279.575362] ata4.00: configured for UDMA/133 [13279.575388] ata4: EH complete [13283.169224] ata4.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0 [13283.169246] ata4.00: irq_stat 0x40000008 [13283.169263] ata4.00: failed command: READ FPDMA QUEUED [13283.169289] ata4.00: cmd 60/08:00:00:6a:05/00:00:23:00:00/40 tag 0 ncq 4096 in [13283.169294] res 41/40:00:07:6a:05/00:00:23:00:00/40 Emask 0x409 (media error) <F> [13283.169331] ata4.00: status: { DRDY ERR } [13283.169345] ata4.00: error: { UNC } [13283.176071] ata4.00: configured for UDMA/133 [13283.176104] ata4: EH complete [13286.224814] ata4.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0 [13286.224837] ata4.00: irq_stat 0x40000008 [13286.224853] ata4.00: failed command: READ FPDMA QUEUED [13286.224879] ata4.00: cmd 60/08:00:00:6a:05/00:00:23:00:00/40 tag 0 ncq 4096 in [13286.224884] res 41/40:00:06:6a:05/00:00:23:00:00/40 Emask 0x409 (media error) <F> [13286.224922] ata4.00: status: { DRDY ERR } [13286.224935] ata4.00: error: { UNC } [13286.231277] ata4.00: configured for UDMA/133 [13286.231303] ata4: EH complete [13288.802623] ata4.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0 [13288.802646] ata4.00: irq_stat 0x40000008 [13288.802662] ata4.00: failed command: READ FPDMA QUEUED [13288.802688] ata4.00: cmd 60/08:00:00:6a:05/00:00:23:00:00/40 tag 0 ncq 4096 in [13288.802693] res 41/40:00:05:6a:05/00:00:23:00:00/40 Emask 0x409 (media error) <F> [13288.802731] ata4.00: status: { DRDY ERR } [13288.802745] ata4.00: error: { UNC } [13288.808901] ata4.00: configured for UDMA/133 [13288.808927] ata4: EH complete [13291.380430] ata4.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0 [13291.380453] ata4.00: irq_stat 0x40000008 [13291.380470] ata4.00: failed command: READ FPDMA QUEUED [13291.380496] ata4.00: cmd 60/08:00:00:6a:05/00:00:23:00:00/40 tag 0 ncq 4096 in [13291.380501] res 41/40:00:05:6a:05/00:00:23:00:00/40 Emask 0x409 (media error) <F> [13291.380577] ata4.00: status: { DRDY ERR } [13291.380594] ata4.00: error: { UNC } [13291.386517] ata4.00: configured for UDMA/133 [13291.386543] ata4: EH complete [13294.347147] ata4.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0 [13294.347169] ata4.00: irq_stat 0x40000008 [13294.347186] ata4.00: failed command: READ FPDMA QUEUED [13294.347211] ata4.00: cmd 60/08:00:00:6a:05/00:00:23:00:00/40 tag 0 ncq 4096 in [13294.347217] res 41/40:00:06:6a:05/00:00:23:00:00/40 Emask 0x409 (media error) <F> [13294.347254] ata4.00: status: { DRDY ERR } [13294.347268] ata4.00: error: { UNC } [13294.353556] ata4.00: configured for UDMA/133 [13294.353583] sd 3:0:0:0: [sdc] Unhandled sense code [13294.353590] sd 3:0:0:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [13294.353599] sd 3:0:0:0: [sdc] Sense Key : Medium Error [current] [descriptor] [13294.353610] Descriptor sense data with sense descriptors (in hex): [13294.353616] 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 [13294.353635] 23 05 6a 06 [13294.353644] sd 3:0:0:0: [sdc] Add. Sense: Unrecovered read error - auto reallocate failed [13294.353657] sd 3:0:0:0: [sdc] CDB: Read(10): 28 00 23 05 6a 00 00 00 08 00 [13294.353675] end_request: I/O error, dev sdc, sector 587557382 [13294.353726] ata4: EH complete [13294.366953] raid1:md0: read error corrected (8 sectors at 489900544 on sdc7) [13294.366992] raid1: sdc7: redirecting sector 489898496 to another mirror and they're happening quite frequently, which I guess is liable to account for the performance problem(?) # dmesg | grep mirror [12433.561822] raid1: sdc7: redirecting sector 489900464 to another mirror [12449.428933] raid1: sdb7: redirecting sector 489900504 to another mirror [12464.807016] raid1: sdb7: redirecting sector 489900512 to another mirror [12480.196222] raid1: sdb7: redirecting sector 489900520 to another mirror [12495.585413] raid1: sdb7: redirecting sector 489900528 to another mirror [12510.974424] raid1: sdb7: redirecting sector 489900536 to another mirror [12526.374933] raid1: sdb7: redirecting sector 489900544 to another mirror [12542.619938] raid1: sdc7: redirecting sector 489900608 to another mirror [12559.431328] raid1: sdc7: redirecting sector 489900616 to another mirror [12576.553866] raid1: sdc7: redirecting sector 489900624 to another mirror [12592.065265] raid1: sdc7: redirecting sector 489900632 to another mirror [12607.621121] raid1: sdc7: redirecting sector 489900640 to another mirror [12623.165856] raid1: sdc7: redirecting sector 489900648 to another mirror [12638.699474] raid1: sdc7: redirecting sector 489900656 to another mirror [12655.610881] raid1: sdc7: redirecting sector 489900664 to another mirror [12672.255617] raid1: sdc7: redirecting sector 489900672 to another mirror [12672.288746] raid1: sdc7: redirecting sector 489900680 to another mirror [12672.332376] raid1: sdc7: redirecting sector 489900688 to another mirror [12672.362935] raid1: sdc7: redirecting sector 489900696 to another mirror [12674.201177] raid1: sdc7: redirecting sector 489900704 to another mirror [12698.045050] raid1: sdc7: redirecting sector 489900712 to another mirror [12698.089309] raid1: sdc7: redirecting sector 489900720 to another mirror [12698.111999] raid1: sdc7: redirecting sector 489900728 to another mirror [12698.134006] raid1: sdc7: redirecting sector 489900736 to another mirror [12719.034376] raid1: sdc7: redirecting sector 489900744 to another mirror [12734.545775] raid1: sdc7: redirecting sector 489900752 to another mirror [12734.590014] raid1: sdc7: redirecting sector 489900760 to another mirror [12734.624050] raid1: sdc7: redirecting sector 489900768 to another mirror [12734.647308] raid1: sdc7: redirecting sector 489900776 to another mirror [12734.664657] raid1: sdc7: redirecting sector 489900784 to another mirror [12734.710642] raid1: sdc7: redirecting sector 489900792 to another mirror [12734.721919] raid1: sdc7: redirecting sector 489900800 to another mirror [12734.744732] raid1: sdc7: redirecting sector 489900808 to another mirror [12734.779330] raid1: sdc7: redirecting sector 489900816 to another mirror [12782.604564] raid1: sdb7: redirecting sector 1242934216 to another mirror [12798.264153] raid1: sdc7: redirecting sector 1242935080 to another mirror [13245.832193] raid1: sdb7: redirecting sector 489898296 to another mirror [13261.376929] raid1: sdb7: redirecting sector 489898304 to another mirror [13276.966043] raid1: sdb7: redirecting sector 489898312 to another mirror [13294.366992] raid1: sdc7: redirecting sector 489898496 to another mirror although the arrays are still running on all disks - they haven't given up on any yet: # cat /proc/mdstat Personalities : [raid1] [raid0] md10 : active raid0 md0[0] md1[1] 3368770048 blocks super 1.2 512k chunks md1 : active raid1 sde2[2] sdd2[1] 1464087824 blocks super 1.2 [2/2] [UU] md0 : active raid1 sdb7[0] sdc7[2] 1904684920 blocks super 1.2 [2/2] [UU] unused devices: <none> So I think I have some idea what the problem is but I am not a linux sysadmin expert by the remotest stretch of the imagination and would really appreciate some clue checking here with my diagnosis and what do I need to do: obviously I need to source another drive for sdc. (I'm guessing I could buy a larger drive if the price is right: I'm thinking that one day I'll need to grow the size of the array and that would be one less drive to replace with a larger one) then use mdadm to fail out the existing sdc, remove it and fit the new drive fdisk the new drive with the same size partition for the array as the old one had use mdadm to add the new drive into the array that sound OK?

  Read the article

• Did I lose my RAID again?

  - by BarsMonster

  Hi! A little history: 2 years ago I was really excited to find out that mdadm is so powerful that it even can reshape arrays, so you can start with a smaller array and then grow it as you need. I've bought 3x1Tb drives and made a RAID-5. It was fine for a year. Then I bought 2x more, and tried to reshape to RAID-6 out of 5 drives, and due to some mess with superblock versions, lost all content. Had to rebuild it from scratch, but 2Tb of data were gone. Yesterday I bought 2 more drives, and this time I had everything: properly built array, UPS. I've disabled write intent map, added 2 new drives as spares and run a command to grow array to 7-disks. It started working, but speed was ridiculously slow, ~100kb/sec. After processing first 37Mb at such an amazing speed, one of old HDDs fails. I properly shutdown the PC and disconnected the failed drive. After bootup it appeared that it recreated the intent map as it was still in mdadm config, so I removed it from config and rebooted again. Now all I see is that all mdadm processes deadlock, and don't do anything. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1937 root 20 0 12992 608 444 D 0 0.1 0:00.00 mdadm 2283 root 20 0 12992 852 704 D 0 0.1 0:00.01 mdadm 2287 root 20 0 0 0 0 D 0 0.0 0:00.01 md0_reshape 2288 root 18 -2 12992 820 676 D 0 0.1 0:00.01 mdadm And all I see in mdstat is: $ cat /proc/mdstat Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md0 : active raid6 sdb1[1] sdg1[4] sdf1[7] sde1[6] sdd1[0] sdc1[5] 2929683456 blocks super 1.2 level 6, 1024k chunk, algorithm 2 [7/6] [UU_UUUU] [>....................] reshape = 0.0% (37888/976561152) finish=567604147.2min speed=0K/sec I've already tried mdadm 2.6.7, 3.1.4 and 3.2 - nothing helps. Did I lose my data again? Any suggestions on how can I make this work? OS is Ubuntu Server 10.04.2. PS. Needless to say, the data is inaccessible - I cannot mount /dev/md0 to save the most valuable data. You can see my disappointment - the very specific thing I was excited about failed twice taking 5Tb of my data with it. Update: It appears there is some nice info in kern.log: 21:38:48 ...: [ 166.522055] raid5: reshape will continue 21:38:48 ...: [ 166.522085] raid5: device sdb1 operational as raid disk 1 21:38:48 ...: [ 166.522091] raid5: device sdg1 operational as raid disk 4 21:38:48 ...: [ 166.522097] raid5: device sdf1 operational as raid disk 5 21:38:48 ...: [ 166.522102] raid5: device sde1 operational as raid disk 6 21:38:48 ...: [ 166.522107] raid5: device sdd1 operational as raid disk 0 21:38:48 ...: [ 166.522111] raid5: device sdc1 operational as raid disk 3 21:38:48 ...: [ 166.523942] raid5: allocated 7438kB for md0 21:38:48 ...: [ 166.524041] 1: w=1 pa=2 pr=5 m=2 a=2 r=7 op1=0 op2=0 21:38:48 ...: [ 166.524050] 4: w=2 pa=2 pr=5 m=2 a=2 r=7 op1=0 op2=0 21:38:48 ...: [ 166.524056] 5: w=3 pa=2 pr=5 m=2 a=2 r=7 op1=0 op2=0 21:38:48 ...: [ 166.524062] 6: w=4 pa=2 pr=5 m=2 a=2 r=7 op1=0 op2=0 21:38:48 ...: [ 166.524068] 0: w=5 pa=2 pr=5 m=2 a=2 r=7 op1=0 op2=0 21:38:48 ...: [ 166.524073] 3: w=6 pa=2 pr=5 m=2 a=2 r=7 op1=0 op2=0 21:38:48 ...: [ 166.524079] raid5: raid level 6 set md0 active with 6 out of 7 devices, algorithm 2 21:38:48 ...: [ 166.524519] RAID5 conf printout: 21:38:48 ...: [ 166.524523] --- rd:7 wd:6 21:38:48 ...: [ 166.524528] disk 0, o:1, dev:sdd1 21:38:48 ...: [ 166.524532] disk 1, o:1, dev:sdb1 21:38:48 ...: [ 166.524537] disk 3, o:1, dev:sdc1 21:38:48 ...: [ 166.524541] disk 4, o:1, dev:sdg1 21:38:48 ...: [ 166.524545] disk 5, o:1, dev:sdf1 21:38:48 ...: [ 166.524550] disk 6, o:1, dev:sde1 21:38:48 ...: [ 166.524553] ...ok start reshape thread 21:38:48 ...: [ 166.524727] md0: detected capacity change from 0 to 2999995858944 21:38:48 ...: [ 166.524735] md: reshape of RAID array md0 21:38:48 ...: [ 166.524740] md: minimum _guaranteed_ speed: 1000 KB/sec/disk. 21:38:48 ...: [ 166.524745] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for reshape. 21:38:48 ...: [ 166.524756] md: using 128k window, over a total of 976561152 blocks. 21:39:05 ...: [ 166.525013] md0: 21:42:04 ...: [ 362.520063] INFO: task mdadm:1937 blocked for more than 120 seconds. 21:42:04 ...: [ 362.520068] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 21:42:04 ...: [ 362.520073] mdadm D 00000000ffffffff 0 1937 1 0x00000000 21:42:04 ...: [ 362.520083] ffff88002ef4f5d8 0000000000000082 0000000000015bc0 0000000000015bc0 21:42:04 ...: [ 362.520092] ffff88002eb5b198 ffff88002ef4ffd8 0000000000015bc0 ffff88002eb5ade0 21:42:04 ...: [ 362.520100] 0000000000015bc0 ffff88002ef4ffd8 0000000000015bc0 ffff88002eb5b198 21:42:04 ...: [ 362.520107] Call Trace: 21:42:04 ...: [ 362.520133] [<ffffffffa0224892>] get_active_stripe+0x312/0x3f0 [raid456] 21:42:04 ...: [ 362.520148] [<ffffffff81059ae0>] ? default_wake_function+0x0/0x20 21:42:04 ...: [ 362.520159] [<ffffffffa0228413>] make_request+0x243/0x4b0 [raid456] 21:42:04 ...: [ 362.520169] [<ffffffffa0221a90>] ? release_stripe+0x50/0x70 [raid456] 21:42:04 ...: [ 362.520179] [<ffffffff81084790>] ? autoremove_wake_function+0x0/0x40 21:42:04 ...: [ 362.520188] [<ffffffff81414df0>] md_make_request+0xc0/0x130 21:42:04 ...: [ 362.520194] [<ffffffff81414df0>] ? md_make_request+0xc0/0x130 21:42:04 ...: [ 362.520205] [<ffffffff8129f8c1>] generic_make_request+0x1b1/0x4f0 21:42:04 ...: [ 362.520214] [<ffffffff810f6515>] ? mempool_alloc_slab+0x15/0x20 21:42:04 ...: [ 362.520222] [<ffffffff8116c2ec>] ? alloc_buffer_head+0x1c/0x60 21:42:04 ...: [ 362.520230] [<ffffffff8129fc80>] submit_bio+0x80/0x110 21:42:04 ...: [ 362.520236] [<ffffffff8116c849>] submit_bh+0xf9/0x140 21:42:04 ...: [ 362.520244] [<ffffffff8116f124>] block_read_full_page+0x274/0x3b0 21:42:04 ...: [ 362.520251] [<ffffffff81172c90>] ? blkdev_get_block+0x0/0x70 21:42:04 ...: [ 362.520258] [<ffffffff8110d875>] ? __inc_zone_page_state+0x35/0x40 21:42:04 ...: [ 362.520265] [<ffffffff810f46d8>] ? add_to_page_cache_locked+0xe8/0x160 21:42:04 ...: [ 362.520272] [<ffffffff81173d78>] blkdev_readpage+0x18/0x20 21:42:04 ...: [ 362.520279] [<ffffffff810f484b>] __read_cache_page+0x7b/0xe0 21:42:04 ...: [ 362.520285] [<ffffffff81173d60>] ? blkdev_readpage+0x0/0x20 21:42:04 ...: [ 362.520290] [<ffffffff81173d60>] ? blkdev_readpage+0x0/0x20 21:42:04 ...: [ 362.520297] [<ffffffff810f57dc>] do_read_cache_page+0x3c/0x120 21:42:04 ...: [ 362.520304] [<ffffffff810f5909>] read_cache_page_async+0x19/0x20 21:42:04 ...: [ 362.520310] [<ffffffff810f591e>] read_cache_page+0xe/0x20 21:42:04 ...: [ 362.520317] [<ffffffff811a6cb0>] read_dev_sector+0x30/0xa0 21:42:04 ...: [ 362.520324] [<ffffffff811a7fcd>] amiga_partition+0x6d/0x460 21:42:04 ...: [ 362.520331] [<ffffffff811a7938>] check_partition+0x138/0x190 21:42:04 ...: [ 362.520338] [<ffffffff811a7a7a>] rescan_partitions+0xea/0x2f0 21:42:04 ...: [ 362.520344] [<ffffffff811744c7>] __blkdev_get+0x267/0x3d0 21:42:04 ...: [ 362.520350] [<ffffffff81174650>] ? blkdev_open+0x0/0xc0 21:42:04 ...: [ 362.520356] [<ffffffff81174640>] blkdev_get+0x10/0x20 21:42:04 ...: [ 362.520362] [<ffffffff811746c1>] blkdev_open+0x71/0xc0 21:42:04 ...: [ 362.520369] [<ffffffff811419f3>] __dentry_open+0x113/0x370 21:42:04 ...: [ 362.520377] [<ffffffff81253f8f>] ? security_inode_permission+0x1f/0x30 21:42:04 ...: [ 362.520385] [<ffffffff8114de3f>] ? inode_permission+0xaf/0xd0 21:42:04 ...: [ 362.520391] [<ffffffff81141d67>] nameidata_to_filp+0x57/0x70 21:42:04 ...: [ 362.520398] [<ffffffff8115207a>] do_filp_open+0x2da/0xba0 21:42:04 ...: [ 362.520406] [<ffffffff811134a8>] ? unmap_vmas+0x178/0x310 21:42:04 ...: [ 362.520414] [<ffffffff8115dbfa>] ? alloc_fd+0x10a/0x150 21:42:04 ...: [ 362.520421] [<ffffffff81141769>] do_sys_open+0x69/0x170 21:42:04 ...: [ 362.520428] [<ffffffff811418b0>] sys_open+0x20/0x30 21:42:04 ...: [ 362.520437] [<ffffffff810121b2>] system_call_fastpath+0x16/0x1b 21:42:04 ...: [ 362.520446] INFO: task mdadm:2283 blocked for more than 120 seconds. 21:42:04 ...: [ 362.520450] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 21:42:04 ...: [ 362.520454] mdadm D 0000000000000000 0 2283 2212 0x00000000 21:42:04 ...: [ 362.520462] ffff88002cca7d98 0000000000000086 0000000000015bc0 0000000000015bc0 21:42:04 ...: [ 362.520470] ffff88002ededf78 ffff88002cca7fd8 0000000000015bc0 ffff88002ededbc0 21:42:04 ...: [ 362.520478] 0000000000015bc0 ffff88002cca7fd8 0000000000015bc0 ffff88002ededf78 21:42:04 ...: [ 362.520485] Call Trace: 21:42:04 ...: [ 362.520495] [<ffffffff81543a97>] __mutex_lock_slowpath+0xf7/0x180 21:42:04 ...: [ 362.520502] [<ffffffff8154397b>] mutex_lock+0x2b/0x50 21:42:04 ...: [ 362.520508] [<ffffffff8117404d>] __blkdev_put+0x3d/0x190 21:42:04 ...: [ 362.520514] [<ffffffff811741b0>] blkdev_put+0x10/0x20 21:42:04 ...: [ 362.520520] [<ffffffff811741f3>] blkdev_close+0x33/0x60 21:42:04 ...: [ 362.520527] [<ffffffff81145375>] __fput+0xf5/0x210 21:42:04 ...: [ 362.520534] [<ffffffff811454b5>] fput+0x25/0x30 21:42:04 ...: [ 362.520540] [<ffffffff811415ad>] filp_close+0x5d/0x90 21:42:04 ...: [ 362.520546] [<ffffffff81141697>] sys_close+0xb7/0x120 21:42:04 ...: [ 362.520553] [<ffffffff810121b2>] system_call_fastpath+0x16/0x1b 21:42:04 ...: [ 362.520559] INFO: task md0_reshape:2287 blocked for more than 120 seconds. 21:42:04 ...: [ 362.520563] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 21:42:04 ...: [ 362.520567] md0_reshape D ffff88003aee96f0 0 2287 2 0x00000000 21:42:04 ...: [ 362.520575] ffff88003cf05a70 0000000000000046 0000000000015bc0 0000000000015bc0 21:42:04 ...: [ 362.520582] ffff88003aee9aa8 ffff88003cf05fd8 0000000000015bc0 ffff88003aee96f0 21:42:04 ...: [ 362.520590] 0000000000015bc0 ffff88003cf05fd8 0000000000015bc0 ffff88003aee9aa8 21:42:04 ...: [ 362.520597] Call Trace: 21:42:04 ...: [ 362.520608] [<ffffffffa0224892>] get_active_stripe+0x312/0x3f0 [raid456] 21:42:04 ...: [ 362.520616] [<ffffffff81059ae0>] ? default_wake_function+0x0/0x20 21:42:04 ...: [ 362.520626] [<ffffffffa0226f80>] reshape_request+0x4c0/0x9a0 [raid456] 21:42:04 ...: [ 362.520634] [<ffffffff81084790>] ? autoremove_wake_function+0x0/0x40 21:42:04 ...: [ 362.520644] [<ffffffffa022777a>] sync_request+0x31a/0x3a0 [raid456] 21:42:04 ...: [ 362.520651] [<ffffffff81052713>] ? __wake_up+0x53/0x70 21:42:04 ...: [ 362.520658] [<ffffffff814156b1>] md_do_sync+0x621/0xbb0 21:42:04 ...: [ 362.520668] [<ffffffff810387b9>] ? default_spin_lock_flags+0x9/0x10 21:42:04 ...: [ 362.520675] [<ffffffff8141640c>] md_thread+0x5c/0x130 21:42:04 ...: [ 362.520681] [<ffffffff81084790>] ? autoremove_wake_function+0x0/0x40 21:42:04 ...: [ 362.520688] [<ffffffff814163b0>] ? md_thread+0x0/0x130 21:42:04 ...: [ 362.520694] [<ffffffff81084416>] kthread+0x96/0xa0 21:42:04 ...: [ 362.520701] [<ffffffff810131ea>] child_rip+0xa/0x20 21:42:04 ...: [ 362.520707] [<ffffffff81084380>] ? kthread+0x0/0xa0 21:42:04 ...: [ 362.520713] [<ffffffff810131e0>] ? child_rip+0x0/0x20 21:42:04 ...: [ 362.520718] INFO: task mdadm:2288 blocked for more than 120 seconds. 21:42:04 ...: [ 362.520721] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 21:42:04 ...: [ 362.520725] mdadm D 0000000000000000 0 2288 1 0x00000000 21:42:04 ...: [ 362.520733] ffff88002cca9c18 0000000000000086 0000000000015bc0 0000000000015bc0 21:42:04 ...: [ 362.520741] ffff88003aee83b8 ffff88002cca9fd8 0000000000015bc0 ffff88003aee8000 21:42:04 ...: [ 362.520748] 0000000000015bc0 ffff88002cca9fd8 0000000000015bc0 ffff88003aee83b8 21:42:04 ...: [ 362.520755] Call Trace: 21:42:04 ...: [ 362.520763] [<ffffffff81543a97>] __mutex_lock_slowpath+0xf7/0x180 21:42:04 ...: [ 362.520771] [<ffffffff812a6d50>] ? exact_match+0x0/0x10 21:42:04 ...: [ 362.520777] [<ffffffff8154397b>] mutex_lock+0x2b/0x50 21:42:04 ...: [ 362.520783] [<ffffffff811742c8>] __blkdev_get+0x68/0x3d0 21:42:04 ...: [ 362.520790] [<ffffffff81174650>] ? blkdev_open+0x0/0xc0 21:42:04 ...: [ 362.520795] [<ffffffff81174640>] blkdev_get+0x10/0x20 21:42:04 ...: [ 362.520801] [<ffffffff811746c1>] blkdev_open+0x71/0xc0 21:42:04 ...: [ 362.520808] [<ffffffff811419f3>] __dentry_open+0x113/0x370 21:42:04 ...: [ 362.520815] [<ffffffff81253f8f>] ? security_inode_permission+0x1f/0x30 21:42:04 ...: [ 362.520821] [<ffffffff8114de3f>] ? inode_permission+0xaf/0xd0 21:42:04 ...: [ 362.520828] [<ffffffff81141d67>] nameidata_to_filp+0x57/0x70 21:42:04 ...: [ 362.520834] [<ffffffff8115207a>] do_filp_open+0x2da/0xba0 21:42:04 ...: [ 362.520841] [<ffffffff810ff0e1>] ? lru_cache_add_lru+0x21/0x40 21:42:04 ...: [ 362.520848] [<ffffffff8111109c>] ? do_anonymous_page+0x11c/0x330 21:42:04 ...: [ 362.520855] [<ffffffff81115d5f>] ? handle_mm_fault+0x31f/0x3c0 21:42:04 ...: [ 362.520862] [<ffffffff8115dbfa>] ? alloc_fd+0x10a/0x150 21:42:04 ...: [ 362.520868] [<ffffffff81141769>] do_sys_open+0x69/0x170 21:42:04 ...: [ 362.520874] [<ffffffff811418b0>] sys_open+0x20/0x30 21:42:04 ...: [ 362.520882] [<ffffffff810121b2>] system_call_fastpath+0x16/0x1b 21:44:04 ...: [ 482.520065] INFO: task mdadm:1937 blocked for more than 120 seconds. 21:44:04 ...: [ 482.520071] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 21:44:04 ...: [ 482.520077] mdadm D 00000000ffffffff 0 1937 1 0x00000000 21:44:04 ...: [ 482.520087] ffff88002ef4f5d8 0000000000000082 0000000000015bc0 0000000000015bc0 21:44:04 ...: [ 482.520096] ffff88002eb5b198 ffff88002ef4ffd8 0000000000015bc0 ffff88002eb5ade0 21:44:04 ...: [ 482.520104] 0000000000015bc0 ffff88002ef4ffd8 0000000000015bc0 ffff88002eb5b198 21:44:04 ...: [ 482.520112] Call Trace: 21:44:04 ...: [ 482.520139] [<ffffffffa0224892>] get_active_stripe+0x312/0x3f0 [raid456] 21:44:04 ...: [ 482.520154] [<ffffffff81059ae0>] ? default_wake_function+0x0/0x20 21:44:04 ...: [ 482.520165] [<ffffffffa0228413>] make_request+0x243/0x4b0 [raid456] 21:44:04 ...: [ 482.520175] [<ffffffffa0221a90>] ? release_stripe+0x50/0x70 [raid456] 21:44:04 ...: [ 482.520185] [<ffffffff81084790>] ? autoremove_wake_function+0x0/0x40 21:44:04 ...: [ 482.520194] [<ffffffff81414df0>] md_make_request+0xc0/0x130 21:44:04 ...: [ 482.520201] [<ffffffff81414df0>] ? md_make_request+0xc0/0x130 21:44:04 ...: [ 482.520212] [<ffffffff8129f8c1>] generic_make_request+0x1b1/0x4f0 21:44:04 ...: [ 482.520221] [<ffffffff810f6515>] ? mempool_alloc_slab+0x15/0x20 21:44:04 ...: [ 482.520229] [<ffffffff8116c2ec>] ? alloc_buffer_head+0x1c/0x60 21:44:04 ...: [ 482.520237] [<ffffffff8129fc80>] submit_bio+0x80/0x110 21:44:04 ...: [ 482.520244] [<ffffffff8116c849>] submit_bh+0xf9/0x140 21:44:04 ...: [ 482.520252] [<ffffffff8116f124>] block_read_full_page+0x274/0x3b0 21:44:04 ...: [ 482.520258] [<ffffffff81172c90>] ? blkdev_get_block+0x0/0x70 21:44:04 ...: [ 482.520266] [<ffffffff8110d875>] ? __inc_zone_page_state+0x35/0x40 21:44:04 ...: [ 482.520273] [<ffffffff810f46d8>] ? add_to_page_cache_locked+0xe8/0x160 21:44:04 ...: [ 482.520280] [<ffffffff81173d78>] blkdev_readpage+0x18/0x20 21:44:04 ...: [ 482.520286] [<ffffffff810f484b>] __read_cache_page+0x7b/0xe0 21:44:04 ...: [ 482.520293] [<ffffffff81173d60>] ? blkdev_readpage+0x0/0x20 21:44:04 ...: [ 482.520299] [<ffffffff81173d60>] ? blkdev_readpage+0x0/0x20 21:44:04 ...: [ 482.520306] [<ffffffff810f57dc>] do_read_cache_page+0x3c/0x120 21:44:04 ...: [ 482.520313] [<ffffffff810f5909>] read_cache_page_async+0x19/0x20 21:44:04 ...: [ 482.520319] [<ffffffff810f591e>] read_cache_page+0xe/0x20 21:44:04 ...: [ 482.520327] [<ffffffff811a6cb0>] read_dev_sector+0x30/0xa0 21:44:04 ...: [ 482.520334] [<ffffffff811a7fcd>] amiga_partition+0x6d/0x460 21:44:04 ...: [ 482.520341] [<ffffffff811a7938>] check_partition+0x138/0x190 21:44:04 ...: [ 482.520348] [<ffffffff811a7a7a>] rescan_partitions+0xea/0x2f0 21:44:04 ...: [ 482.520355] [<ffffffff811744c7>] __blkdev_get+0x267/0x3d0 21:44:04 ...: [ 482.520361] [<ffffffff81174650>] ? blkdev_open+0x0/0xc0 21:44:04 ...: [ 482.520367] [<ffffffff81174640>] blkdev_get+0x10/0x20 21:44:04 ...: [ 482.520373] [<ffffffff811746c1>] blkdev_open+0x71/0xc0 21:44:04 ...: [ 482.520380] [<ffffffff811419f3>] __dentry_open+0x113/0x370 21:44:04 ...: [ 482.520388] [<ffffffff81253f8f>] ? security_inode_permission+0x1f/0x30 21:44:04 ...: [ 482.520396] [<ffffffff8114de3f>] ? inode_permission+0xaf/0xd0 21:44:04 ...: [ 482.520403] [<ffffffff81141d67>] nameidata_to_filp+0x57/0x70 21:44:04 ...: [ 482.520410] [<ffffffff8115207a>] do_filp_open+0x2da/0xba0 21:44:04 ...: [ 482.520417] [<ffffffff811134a8>] ? unmap_vmas+0x178/0x310 21:44:04 ...: [ 482.520426] [<ffffffff8115dbfa>] ? alloc_fd+0x10a/0x150 21:44:04 ...: [ 482.520432] [<ffffffff81141769>] do_sys_open+0x69/0x170 21:44:04 ...: [ 482.520438] [<ffffffff811418b0>] sys_open+0x20/0x30 21:44:04 ...: [ 482.520447] [<ffffffff810121b2>] system_call_fastpath+0x16/0x1b 21:44:04 ...: [ 482.520458] INFO: task mdadm:2283 blocked for more than 120 seconds. 21:44:04 ...: [ 482.520462] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 21:44:04 ...: [ 482.520467] mdadm D 0000000000000000 0 2283 2212 0x00000000 21:44:04 ...: [ 482.520475] ffff88002cca7d98 0000000000000086 0000000000015bc0 0000000000015bc0 21:44:04 ...: [ 482.520483] ffff88002ededf78 ffff88002cca7fd8 0000000000015bc0 ffff88002ededbc0 21:44:04 ...: [ 482.520490] 0000000000015bc0 ffff88002cca7fd8 0000000000015bc0 ffff88002ededf78 21:44:04 ...: [ 482.520498] Call Trace: 21:44:04 ...: [ 482.520508] [<ffffffff81543a97>] __mutex_lock_slowpath+0xf7/0x180 21:44:04 ...: [ 482.520515] [<ffffffff8154397b>] mutex_lock+0x2b/0x50 21:44:04 ...: [ 482.520521] [<ffffffff8117404d>] __blkdev_put+0x3d/0x190 21:44:04 ...: [ 482.520527] [<ffffffff811741b0>] blkdev_put+0x10/0x20 21:44:04 ...: [ 482.520533] [<ffffffff811741f3>] blkdev_close+0x33/0x60 21:44:04 ...: [ 482.520541] [<ffffffff81145375>] __fput+0xf5/0x210 21:44:04 ...: [ 482.520547] [<ffffffff811454b5>] fput+0x25/0x30 21:44:04 ...: [ 482.520554] [<ffffffff811415ad>] filp_close+0x5d/0x90 21:44:04 ...: [ 482.520560] [<ffffffff81141697>] sys_close+0xb7/0x120 21:44:04 ...: [ 482.520568] [<ffffffff810121b2>] system_call_fastpath+0x16/0x1b 21:44:04 ...: [ 482.520574] INFO: task md0_reshape:2287 blocked for more than 120 seconds. 21:44:04 ...: [ 482.520578] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 21:44:04 ...: [ 482.520582] md0_reshape D ffff88003aee96f0 0 2287 2 0x00000000 21:44:04 ...: [ 482.520590] ffff88003cf05a70 0000000000000046 0000000000015bc0 0000000000015bc0 21:44:04 ...: [ 482.520597] ffff88003aee9aa8 ffff88003cf05fd8 0000000000015bc0 ffff88003aee96f0 21:44:04 ...: [ 482.520605] 0000000000015bc0 ffff88003cf05fd8 0000000000015bc0 ffff88003aee9aa8 21:44:04 ...: [ 482.520612] Call Trace: 21:44:04 ...: [ 482.520623] [<ffffffffa0224892>] get_active_stripe+0x312/0x3f0 [raid456] 21:44:04 ...: [ 482.520633] [<ffffffff81059ae0>] ? default_wake_function+0x0/0x20 21:44:04 ...: [ 482.520643] [<ffffffffa0226f80>] reshape_request+0x4c0/0x9a0 [raid456] 21:44:04 ...: [ 482.520651] [<ffffffff81084790>] ? autoremove_wake_function+0x0/0x40 21:44:04 ...: [ 482.520661] [<ffffffffa022777a>] sync_request+0x31a/0x3a0 [raid456] 21:44:04 ...: [ 482.520668] [<ffffffff81052713>] ? __wake_up+0x53/0x70 21:44:04 ...: [ 482.520675] [<ffffffff814156b1>] md_do_sync+0x621/0xbb0 21:44:04 ...: [ 482.520685] [<ffffffff810387b9>] ? default_spin_lock_flags+0x9/0x10 21:44:04 ...: [ 482.520692] [<ffffffff8141640c>] md_thread+0x5c/0x130 21:44:04 ...: [ 482.520699] [<ffffffff81084790>] ? autoremove_wake_function+0x0/0x40 21:44:04 ...: [ 482.520705] [<ffffffff814163b0>] ? md_thread+0x0/0x130 21:44:04 ...: [ 482.520711] [<ffffffff81084416>] kthread+0x96/0xa0 21:44:04 ...: [ 482.520718] [<ffffffff810131ea>] child_rip+0xa/0x20 21:44:04 ...: [ 482.520725] [<ffffffff81084380>] ? kthread+0x0/0xa0 21:44:04 ...: [ 482.520730] [<ffffffff810131e0>] ? child_rip+0x0/0x20 21:44:04 ...: [ 482.520735] INFO: task mdadm:2288 blocked for more than 120 seconds. 21:44:04 ...: [ 482.520739] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 21:44:04 ...: [ 482.520743] mdadm D 0000000000000000 0 2288 1 0x00000000 21:44:04 ...: [ 482.520751] ffff88002cca9c18 0000000000000086 0000000000015bc0 0000000000015bc0 21:44:04 ...: [ 482.520759] ffff88003aee83b8 ffff88002cca9fd8 0000000000015bc0 ffff88003aee8000 21:44:04 ...: [ 482.520767] 0000000000015bc0 ffff88002cca9fd8 0000000000015bc0 ffff88003aee83b8 21:44:04 ...: [ 482.520774] Call Trace: 21:44:04 ...: [ 482.520782] [<ffffffff81543a97>] __mutex_lock_slowpath+0xf7/0x180 21:44:04 ...: [ 482.520790] [<ffffffff812a6d50>] ? exact_match+0x0/0x10 21:44:04 ...: [ 482.520797] [<ffffffff8154397b>] mutex_lock+0x2b/0x50 21:44:04 ...: [ 482.520804] [<ffffffff811742c8>] __blkdev_get+0x68/0x3d0 21:44:04 ...: [ 482.520810] [<ffffffff81174650>] ? blkdev_open+0x0/0xc0 21:44:04 ...: [ 482.520816] [<ffffffff81174640>] blkdev_get+0x10/0x20 21:44:04 ...: [ 482.520822] [<ffffffff811746c1>] blkdev_open+0x71/0xc0 21:44:04 ...: [ 482.520829] [<ffffffff811419f3>] __dentry_open+0x113/0x370 21:44:04 ...: [ 482.520837] [<ffffffff81253f8f>] ? security_inode_permission+0x1f/0x30 21:44:04 ...: [ 482.520843] [<ffffffff8114de3f>] ? inode_permission+0xaf/0xd0 21:44:04 ...: [ 482.520850] [<ffffffff81141d67>] nameidata_to_filp+0x57/0x70 21:44:04 ...: [ 482.520857] [<ffffffff8115207a>] do_filp_open+0x2da/0xba0 21:44:04 ...: [ 482.520864] [<ffffffff810ff0e1>] ? lru_cache_add_lru+0x21/0x40 21:44:04 ...: [ 482.520871] [<ffffffff8111109c>] ? do_anonymous_page+0x11c/0x330 21:44:04 ...: [ 482.520878] [<ffffffff81115d5f>] ? handle_mm_fault+0x31f/0x3c0 21:44:04 ...: [ 482.520885] [<ffffffff8115dbfa>] ? alloc_fd+0x10a/0x150 21:44:04 ...: [ 482.520891] [<ffffffff81141769>] do_sys_open+0x69/0x170 21:44:04 ...: [ 482.520897] [<ffffffff811418b0>] sys_open+0x20/0x30 21:44:04 ...: [ 482.520905] [<ffffffff810121b2>] system_call_fastpath+0x16/0x1b 21:46:04 ...: [ 602.520053] INFO: task mdadm:1937 blocked for more than 120 seconds. 21:46:04 ...: [ 602.520059] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 21:46:04 ...: [ 602.520065] mdadm D 00000000ffffffff 0 1937 1 0x00000000 21:46:04 ...: [ 602.520075] ffff88002ef4f5d8 0000000000000082 0000000000015bc0 0000000000015bc0 21:46:04 ...: [ 602.520084] ffff88002eb5b198 ffff88002ef4ffd8 0000000000015bc0 ffff88002eb5ade0 21:46:04 ...: [ 602.520091] 0000000000015bc0 ffff88002ef4ffd8 0000000000015bc0 ffff88002eb5b198 21:46:04 ...: [ 602.520099] Call Trace: 21:46:04 ...: [ 602.520127] [<ffffffffa0224892>] get_active_stripe+0x312/0x3f0 [raid456] 21:46:04 ...: [ 602.520142] [<ffffffff81059ae0>] ? default_wake_function+0x0/0x20 21:46:04 ...: [ 602.520153] [<ffffffffa0228413>] make_request+0x243/0x4b0 [raid456] 21:46:04 ...: [ 602.520162] [<ffffffffa0221a90>] ? release_stripe+0x50/0x70 [raid456] 21:46:04 ...: [ 602.520171] [<ffffffff81084790>] ? autoremove_wake_function+0x0/0x40 21:46:04 ...: [ 602.520180] [<ffffffff81414df0>] md_make_request+0xc0/0x130 21:46:04 ...: [ 602.520187] [<ffffffff81414df0>] ? md_make_request+0xc0/0x130 21:46:04 ...: [ 602.520197] [<ffffffff8129f8c1>] generic_make_request+0x1b1/0x4f0 21:46:04 ...: [ 602.520206] [<ffffffff810f6515>] ? mempool_alloc_slab+0x15/0x20 21:46:04 ...: [ 602.520215] [<ffffffff8116c2ec>] ? alloc_buffer_head+0x1c/0x60 21:46:04 ...: [ 602.520222] [<ffffffff8129fc80>] submit_bio+0x80/0x110 21:46:04 ...: [ 602.520229] [<ffffffff8116c849>] submit_bh+0xf9/0x140 21:46:04 ...: [ 602.520237] [<ffffffff8116f124>] block_read_full_page+0x274/0x3b0 21:46:04 ...: [ 602.520244] [<ffffffff81172c90>] ? blkdev_get_block+0x0/0x70 21:46:04 ...: [ 602.520252] [<ffffffff8110d875>] ? __inc_zone_page_state+0x35/0x40 21:46:04 ...: [ 602.520259] [<ffffffff810f46d8>] ? add_to_page_cache_locked+0xe8/0x160 21:46:04 ...: [ 602.520266] [<ffffffff81173d78>] blkdev_readpage+0x18/0x20 21:46:04 ...: [ 602.520273] [<ffffffff810f484b>] __read_cache_page+0x7b/0xe0 21:46:04 ...: [ 602.520279] [<ffffffff81173d60>] ? blkdev_readpage+0x0/0x20 21:46:04 ...: [ 602.520285] [<ffffffff81173d60>] ? blkdev_readpage+0x0/0x20 21:46:04 ...: [ 602.520292] [<ffffffff810f57dc>] do_read_cache_page+0x3c/0x120 21:46:04 ...: [ 602.520300] [<ffffffff810f5909>] read_cache_page_async+0x19/0x20 21:46:04 ...: [ 602.520306] [<ffffffff810f591e>] read_cache_page+0xe/0x20 21:46:04 ...: [ 602.520314] [<ffffffff811a6cb0>] read_dev_sector+0x30/0xa0 21:46:04 ...: [ 602.520321] [<ffffffff811a7fcd>] amiga_partition+0x6d/0x460 21:46:04 ...: [ 602.520328] [<ffffffff811a7938>] check_partition+0x138/0x190 21:46:04 ...: [ 602.520335] [<ffffffff811a7a7a>] rescan_partitions+0xea/0x2f0 21:46:04 ...: [ 602.520342] [<ffffffff811744c7>] __blkdev_get+0x267/0x3d0 21:46:04 ...: [ 602.520348] [<ffffffff81174650>] ? blkdev_open+0x0/0xc0 21:46:04 ...: [ 602.520354] [<ffffffff81174640>] blkdev_get+0x10/0x20 21:46:04 ...: [ 602.520359] [<ffffffff811746c1>] blkdev_open+0x71/0xc0 21:46:04 ...: [ 602.520367] [<ffffffff811419f3>] __dentry_open+0x113/0x370 21:46:04 ...: [ 602.520375] [<ffffffff81253f8f>] ? security_inode_permission+0x1f/0x30 21:46:04 ...: [ 602.520383] [<ffffffff8114de3f>] ? inode_permission+0xaf/0xd0 21:46:04 ...: [ 602.520390] [<ffffffff81141d67>] nameidata_to_filp+0x57/0x70 21:46:04 ...: [ 602.520397] [<ffffffff8115207a>] do_filp_open+0x2da/0xba0 21:46:04 ...: [ 602.520404] [<ffffffff811134a8>] ? unmap_vmas+0x178/0x310 21:46:04 ...: [ 602.520413] [<ffffffff8115dbfa>] ? alloc_fd+0x10a/0x150 21:46:04 ...: [ 602.520419] [<ffffffff81141769>] do_sys_open+0x69/0x170 21:46:04 ...: [ 602.520425] [<ffffffff811418b0>] sys_open+0x20/0x30 21:46:04 ...: [ 602.520434] [<ffffffff810121b2>] system_call_fastpath+0x16/0x1b 21:46:04 ...: [ 602.520443] INFO: task mdadm:2283 blocked for more than 120 seconds. 21:46:04 ...: [ 602.520447] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 21:46:04 ...: [ 602.520451] mdadm D 0000000000000000 0 2283 2212 0x00000000 21:46:04 ...: [ 602.520460] ffff88002cca7d98 0000000000000086 0000000000015bc0 0000000000015bc0 21:46:04 ...: [ 602.520468] ffff88002ededf78 ffff88002cca7fd8 0000000000015bc0 ffff88002ededbc0 21:46:04 ...: [ 602.520475] 0000000000015bc0 ffff88002cca7fd8 0000000000015bc0 ffff88002ededf78 21:46:04 ...: [ 602.520483] Call Trace: 21:46:04 ...: [ 602.520492] [<ffffffff81543a97>] __mutex_lock_slowpath+0xf7/0x180 21:46:04 ...: [ 602.520500] [<ffffffff8154397b>] mutex_lock+0x2b/0x50 21:46:04 ...: [ 602.520506] [<ffffffff8117404d>] __blkdev_put+0x3d/0x190 21:46:04 ...: [ 602.520512] [<ffffffff811741b0>] blkdev_put+0x10/0x20 21:46:04 ...: [ 602.520518] [<ffffffff811741f3>] blkdev_close+0x33/0x60 21:46:04 ...: [ 602.520526] [<ffffffff81145375>] __fput+0xf5/0x210 21:46:04 ...: [ 602.520533] [<ffffffff811454b5>] fput+0x25/0x30 21:46:04 ...: [ 602.520539] [<ffffffff811415ad>] filp_close+0x5d/0x90 21:46:04 ...: [ 602.520545] [<ffffffff81141697>] sys_close+0xb7/0x120 21:46:04 ...: [ 602.520552] [<ffffffff810121b2>] system_call_fastpath+0x16/0x1b

  Read the article

• MySQL on Linux out of memory

  - by Sunrays

  OS: Redhat Enterprise Linux Server Release 5.3 (Tikanga) Architecture: Intel Xeon 64Bit MySQL Server 5.5.20 Enterprise Server advanced edition. Application: Liferay. My database size is 200MB. RAM is 64GB. The memory consumption increases gradually and we run out of memory. Then only rebooting releases all the memory, but then process of memory consumption starts again and reaches 63-64GB in less than a day. Parameters detail: key_buffer_size=16M innodb_buffer_pool_size=3GB inndb_buffer_pool_instances=3 max_connections=1000 innodb_flush_method=O_DIRECT innodb_change_buffering=inserts read_buffer_size=2M read_rnd_buffer_size=256K It's a serious production server issue that I am facing. What could be the reason behind this and how to resolve. This is the report of 2pm today, after Linux was rebooted yesterday @ around 10pm. Output of free -m total used free shared buffers cached Mem: 64455 22053 42402 0 1544 1164 -/+ buffers/cache: 19343 45112 Swap: 74998 0 74998 Output of vmstat 2 5 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 0 43423976 1583700 1086616 0 0 1 173 22 27 1 1 98 0 0 2 0 0 43280200 1583712 1228636 0 0 0 146 1265 491 2 2 96 1 0 0 0 0 43421940 1583724 1087160 0 0 0 138 1469 738 2 1 97 0 0 1 0 0 43422604 1583728 1086736 0 0 0 5816 1615 934 1 1 97 0 0 0 0 0 43422372 1583732 1086752 0 0 0 2784 1323 545 2 1 97 0 0 Output of top -n 3 -b top - 14:16:22 up 16:32, 5 users, load average: 0.79, 0.77, 0.93 Tasks: 345 total, 1 running, 344 sleeping, 0 stopped, 0 zombie Cpu(s): 1.0%us, 0.9%sy, 0.0%ni, 98.1%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 66002772k total, 22656292k used, 43346480k free, 1582152k buffers Swap: 76798724k total, 0k used, 76798724k free, 1163616k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 6434 mysql 15 0 4095m 841m 5500 S 113.5 1.3 426:53.69 mysqld 1 root 15 0 10344 680 572 S 0.0 0.0 0:03.09 init 2 root RT -5 0 0 0 S 0.0 0.0 0:00.01 migration/0 3 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0 4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0 5 root RT -5 0 0 0 S 0.0 0.0 0:00.01 migration/1 6 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1 7 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/1 8 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/2 9 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/2 10 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/2 11 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/3 12 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/3 13 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/3 14 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/4 15 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/4 16 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/4 17 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/5 18 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/5 19 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/5 20 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/6 21 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/6 22 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/6 23 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/7 24 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/7 25 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/7 26 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/8 27 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/8 28 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/8 29 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/9 30 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/9 31 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/9 32 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/10 33 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/10 34 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/10 35 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/11 36 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/11 37 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/11 38 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/12 39 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/12 40 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/12 41 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/13 42 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/13 43 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/13 44 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/14 45 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/14 46 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/14 47 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/15 48 root 34 19 0 0 0 S 0.0 0.0 0:00.01 ksoftirqd/15 49 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/15 50 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/16 51 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/16 52 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/16 53 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/17 54 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/17 55 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/17 56 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/18 57 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/18 58 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/18 59 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/19 60 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/19 61 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/19 62 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/20 63 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/20 64 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/20 65 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/21 66 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/21 67 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/21 68 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/22 69 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/22 70 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/22 71 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/23 72 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/23 73 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/23 74 root 10 -5 0 0 0 S 0.0 0.0 0:00.02 events/0 75 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 events/1 76 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 events/2 77 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 events/3 78 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 events/4 79 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 events/5 80 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 events/6 81 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 events/7 82 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 events/8 83 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 events/9 84 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 events/10 85 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 events/11 86 root 10 -5 0 0 0 S 0.0 0.0 0:00.01 events/12 87 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 events/13 88 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 events/14 89 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 events/15 90 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 events/16 91 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 events/17 92 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 events/18 93 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 events/19 94 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 events/20 95 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 events/21 96 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 events/22 97 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 events/23 98 root 10 -5 0 0 0 S 0.0 0.0 0:00.01 khelper 615 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kthread 643 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kblockd/0 644 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kblockd/1 645 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kblockd/2 646 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kblockd/3 647 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kblockd/4 648 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kblockd/5 649 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kblockd/6 650 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kblockd/7 651 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kblockd/8 652 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kblockd/9 653 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kblockd/10 654 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kblockd/11 655 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kblockd/12 656 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kblockd/13 657 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kblockd/14 658 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kblockd/15 659 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kblockd/16 660 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kblockd/17 661 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kblockd/18 662 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kblockd/19 663 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kblockd/20 664 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kblockd/21 665 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kblockd/22 666 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kblockd/23 667 root 17 -5 0 0 0 S 0.0 0.0 0:00.00 kacpid 840 root 17 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/0 841 root 18 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/1 842 root 19 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/2 843 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/3 844 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/4 845 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/5 846 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/6 847 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/7 848 root 12 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/8 849 root 13 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/9 850 root 13 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/10 851 root 13 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/11 852 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/12 853 root 16 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/13 854 root 17 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/14 855 root 18 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/15 856 root 19 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/16 857 root 19 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/17 858 root 19 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/18 859 root 19 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/19 860 root 19 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/20 861 root 19 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/21 862 root 19 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/22 863 root 19 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/23 866 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 khubd 868 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kseriod 1118 root 23 0 0 0 0 S 0.0 0.0 0:00.00 pdflush 1119 root 15 0 0 0 0 S 0.0 0.0 0:00.11 pdflush 1120 root 19 -5 0 0 0 S 0.0 0.0 0:00.00 kswapd0 1121 root 19 -5 0 0 0 S 0.0 0.0 0:00.00 kswapd1 1122 root 19 -5 0 0 0 S 0.0 0.0 0:00.00 aio/0 1123 root 19 -5 0 0 0 S 0.0 0.0 0:00.00 aio/1 1124 root 19 -5 0 0 0 S 0.0 0.0 0:00.00 aio/2 1125 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 aio/3 1126 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 aio/4 1127 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 aio/5 1128 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 aio/6 1129 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 aio/7 1130 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 aio/8 1131 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 aio/9 1132 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 aio/10 1133 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 aio/11 1134 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 aio/12 1135 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 aio/13 1136 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 aio/14 1137 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 aio/15 1138 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 aio/16 1139 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 aio/17 1140 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 aio/18 1141 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 aio/19 1142 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 aio/20 1143 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 aio/21 1144 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 aio/22 1145 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 aio/23 1308 root 11 -5 0 0 0 S 0.0 0.0 0:00.00 kpsmoused 1566 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 ata/0 1567 root 10 -5 0 0 0 S 0.0 0.0 0:00.27 ata/1 1568 root 10 -5 0 0 0 S 0.0 0.0 0:02.39 ata/2 1569 root 10 -5 0 0 0 S 0.0 0.0 0:00.07 ata/3 1570 root 10 -5 0 0 0 S 0.0 0.0 0:00.72 ata/4 1571 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 ata/5 1572 root 10 -5 0 0 0 S 0.0 0.0 0:00.15 ata/6 1573 root 10 -5 0 0 0 S 0.0 0.0 0:00.07 ata/7 1574 root 10 -5 0 0 0 S 0.0 0.0 0:00.06 ata/8 1575 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 ata/9 1576 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 ata/10 1577 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 ata/11 1578 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 ata/12 1579 root 10 -5 0 0 0 S 0.0 0.0 0:00.14 ata/13 1580 root 10 -5 0 0 0 S 0.0 0.0 0:01.56 ata/14 1581 root 10 -5 0 0 0 S 0.0 0.0 0:00.04 ata/15 1582 root 10 -5 0 0 0 S 0.0 0.0 0:00.40 ata/16 1583 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 ata/17 1584 root 10 -5 0 0 0 S 0.0 0.0 0:00.11 ata/18 1585 root 10 -5 0 0 0 S 0.0 0.0 0:00.03 ata/19 1586 root 10 -5 0 0 0 S 0.0 0.0 0:00.02 ata/20 1587 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 ata/21 1588 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 ata/22 1589 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 ata/23 1590 root 13 -5 0 0 0 S 0.0 0.0 0:00.00 ata_aux 1616 root 10 -5 0 0 0 S 0.0 0.0 0:17.20 scsi_eh_0 1617 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_1 1668 root 11 -5 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_2 1669 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 qla2xxx_2_dpc 1670 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 scsi_wq_2 1671 root 11 -5 0 0 0 S 0.0 0.0 0:00.00 fc_wq_2 1672 root 12 -5 0 0 0 S 0.0 0.0 0:00.00 fc_dl_2 1673 root 12 -5 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_3 1674 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 qla2xxx_3_dpc 1675 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 scsi_wq_3 1676 root 11 -5 0 0 0 S 0.0 0.0 0:00.00 fc_wq_3 1677 root 12 -5 0 0 0 S 0.0 0.0 0:00.00 fc_dl_3 1728 root 12 -5 0 0 0 S 0.0 0.0 0:00.00 kstriped 1829 root 10 -5 0 0 0 S 0.0 0.0 1:09.14 kjournald 1857 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kauditd 1891 root 11 -4 13008 1188 388 S 0.0 0.0 0:00.40 udevd 4555 root 11 -5 0 0 0 S 0.0 0.0 0:00.00 kmpathd/0 4556 root 13 -5 0 0 0 S 0.0 0.0 0:00.00 kmpathd/1 4557 root 13 -5 0 0 0 S 0.0 0.0 0:00.00 kmpathd/2 4558 root 14 -5 0 0 0 S 0.0 0.0 0:00.00 kmpathd/3 4559 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 kmpathd/4 4560 root 16 -5 0 0 0 S 0.0 0.0 0:00.00 kmpathd/5 4561 root 16 -5 0 0 0 S 0.0 0.0 0:00.00 kmpathd/6 4562 root 17 -5 0 0 0 S 0.0 0.0 0:00.00 kmpathd/7 4563 root 18 -5 0 0 0 S 0.0 0.0 0:00.00 kmpathd/8 4564 root 19 -5 0 0 0 S 0.0 0.0 0:00.00 kmpathd/9 4565 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 kmpathd/10 4566 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 kmpathd/11 4567 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kmpathd/12 4568 root 13 -5 0 0 0 S 0.0 0.0 0:00.00 kmpathd/13 4569 root 13 -5 0 0 0 S 0.0 0.0 0:00.00 kmpathd/14 4570 root 13 -5 0 0 0 S 0.0 0.0 0:00.00 kmpathd/15 4571 root 14 -5 0 0 0 S 0.0 0.0 0:00.00 kmpathd/16 4572 root 14 -5 0 0 0 S 0.0 0.0 0:00.00 kmpathd/17 4573 root 14 -5 0 0 0 S 0.0 0.0 0:00.00 kmpathd/18 4574 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 kmpathd/19 4575 root 16 -5 0 0 0 S 0.0 0.0 0:00.00 kmpathd/20 4576 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 kmpathd/21 4577 root 16 -5 0 0 0 S 0.0 0.0 0:00.00 kmpathd/22 4578 root 16 -5 0 0 0 S 0.0 0.0 0:00.00 kmpathd/23 4579 root 18 -5 0 0 0 S 0.0 0.0 0:00.00 kmpath_handlerd 4734 root 13 -5 0 0 0 S 0.0 0.0 0:00.00 kjournald 4736 root 10 -5 0 0 0 S 0.0 0.0 0:04.82 kjournald 4744 root 13 -5 0 0 0 S 0.0 0.0 0:00.00 kjournald 5238 root RT 0 87584 3648 2768 S 0.0 0.0 0:03.60 multipathd 5537 root 11 -4 27328 812 580 S 0.0 0.0 0:00.14 auditd 5539 root 7 -8 81804 768 616 S 0.0 0.0 0:00.04 audispd 5564 root 15 0 5904 632 512 S 0.0 0.0 0:00.10 syslogd 5567 root 15 0 3800 432 344 S 0.0 0.0 0:00.01 klogd 5579 root 18 0 10728 384 244 S 0.0 0.0 0:00.42 irqbalance 5592 rpc 18 0 8048 584 464 S 0.0 0.0 0:00.00 portmap 5625 root 18 0 11032 768 632 S 0.0 0.0 0:00.00 rpc.statd 5681 root 11 -5 0 0 0 S 0.0 0.0 0:00.00 rpciod/0 5682 root 11 -5 0 0 0 S 0.0 0.0 0:00.00 rpciod/1 5683 root 12 -5 0 0 0 S 0.0 0.0 0:00.00 rpciod/2 5684 root 12 -5 0 0 0 S 0.0 0.0 0:00.00 rpciod/3 5685 root 12 -5 0 0 0 S 0.0 0.0 0:00.00 rpciod/4 5686 root 12 -5 0 0 0 S 0.0 0.0 0:00.00 rpciod/5 5687 root 12 -5 0 0 0 S 0.0 0.0 0:00.00 rpciod/6 5688 root 12 -5 0 0 0 S 0.0 0.0 0:00.00 rpciod/7 5689 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 rpciod/8 5690 root 12 -5 0 0 0 S 0.0 0.0 0:00.00 rpciod/9 5691 root 12 -5 0 0 0 S 0.0 0.0 0:00.00 rpciod/10 5692 root 13 -5 0 0 0 S 0.0 0.0 0:00.00 rpciod/11

  Read the article

• New CentOS/cPanel servers showing high load averages at idle

  - by Jax

  I have taken delivery of two identically specced CentOS/cPanel servers, showing the same behaviour of a resting load average of 1.30, 1.21, 1.16 and yet the CPU is sitting 100% idle. Hardware: Xeon(R) CPU E3-1270 4GB RAM Behavior:- top shows CPU 99.9% idle virtually no disk IO Some command output :- uname -a Linux server.myserver.com 2.6.18-308.4.1.el5PAE #1 SMP Tue Apr 17 17:47:38 EDT 2012 i686 i686 i386 GNU/Linux top top - 10:37:50 up 1:47, 1 user, load average: 1.28, 1.20, 1.17 Tasks: 199 total, 1 running, 198 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni, 99.9%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 4125104k total, 438764k used, 3686340k free, 25788k buffers Swap: 2096440k total, 0k used, 2096440k free, 291080k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1 root 15 0 2160 640 552 S 0.0 0.0 0:00.89 init 2 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/0 3 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0 4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0 5 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/1 6 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1 7 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/1 8 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/2 9 root 35 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/2 10 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/2 11 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/3 12 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/3 13 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/3 14 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/4 15 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/4 16 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/4 17 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/5 18 root 38 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/5 19 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/5 20 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/6 21 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/6 22 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/6 23 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/7 24 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/7 25 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/7 26 root 10 -5 0 0 0 S 0.0 0.0 0:06.42 events/0 27 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 events/1 28 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 events/2 29 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 events/3 30 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 events/4 31 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 events/5 32 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 events/6 33 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 events/7 34 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 khelper 35 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kthread 45 root 13 -5 0 0 0 S 0.0 0.0 0:00.00 kblockd/0 46 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kblockd/1 47 root 14 -5 0 0 0 S 0.0 0.0 0:00.00 kblockd/2 48 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kblockd/3 49 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kblockd/4 50 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kblockd/5 51 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kblockd/6 52 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kblockd/7 53 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kacpid 189 root 11 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/0 190 root 11 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/1 191 root 12 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/2 192 root 12 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/3 193 root 13 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/4 194 root 13 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/5 195 root 14 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/6 196 root 14 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/7 199 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 khubd ps axf PID TTY STAT TIME COMMAND 1 ? Ss 0:00 init [3] 2 ? S< 0:00 [migration/0] 3 ? SN 0:00 [ksoftirqd/0] 4 ? S< 0:00 [watchdog/0] 5 ? S< 0:00 [migration/1] 6 ? SN 0:00 [ksoftirqd/1] 7 ? S< 0:00 [watchdog/1] 8 ? S< 0:00 [migration/2] 9 ? SN 0:00 [ksoftirqd/2] 10 ? S< 0:00 [watchdog/2] 11 ? S< 0:00 [migration/3] 12 ? SN 0:00 [ksoftirqd/3] 13 ? S< 0:00 [watchdog/3] 14 ? S< 0:00 [migration/4] 15 ? SN 0:00 [ksoftirqd/4] 16 ? S< 0:00 [watchdog/4] 17 ? S< 0:00 [migration/5] 18 ? SN 0:00 [ksoftirqd/5] 19 ? S< 0:00 [watchdog/5] 20 ? S< 0:00 [migration/6] 21 ? SN 0:00 [ksoftirqd/6] 22 ? S< 0:00 [watchdog/6] 23 ? S< 0:00 [migration/7] 24 ? SN 0:00 [ksoftirqd/7] 25 ? S< 0:00 [watchdog/7] 26 ? S< 0:06 [events/0] 27 ? S< 0:00 [events/1] 28 ? S< 0:00 [events/2] 29 ? S< 0:00 [events/3] 30 ? S< 0:00 [events/4] 31 ? S< 0:00 [events/5] 32 ? S< 0:00 [events/6] 33 ? S< 0:00 [events/7] 34 ? S< 0:00 [khelper] 35 ? S< 0:00 [kthread] 45 ? S< 0:00 \_ [kblockd/0] 46 ? S< 0:00 \_ [kblockd/1] 47 ? S< 0:00 \_ [kblockd/2] 48 ? S< 0:00 \_ [kblockd/3] 49 ? S< 0:00 \_ [kblockd/4] 50 ? S< 0:00 \_ [kblockd/5] 51 ? S< 0:00 \_ [kblockd/6] 52 ? S< 0:00 \_ [kblockd/7] 53 ? S< 0:00 \_ [kacpid] 189 ? S< 0:00 \_ [cqueue/0] 190 ? S< 0:00 \_ [cqueue/1] 191 ? S< 0:00 \_ [cqueue/2] 192 ? S< 0:00 \_ [cqueue/3] 193 ? S< 0:00 \_ [cqueue/4] 194 ? S< 0:00 \_ [cqueue/5] 195 ? S< 0:00 \_ [cqueue/6] 196 ? S< 0:00 \_ [cqueue/7] 199 ? S< 0:00 \_ [khubd] 201 ? S< 0:00 \_ [kseriod] 301 ? S 0:00 \_ [khungtaskd] 302 ? S 0:00 \_ [pdflush] 303 ? S 0:00 \_ [pdflush] 304 ? S< 0:00 \_ [kswapd0] 305 ? S< 0:00 \_ [aio/0] 306 ? S< 0:00 \_ [aio/1] 307 ? S< 0:00 \_ [aio/2] 308 ? S< 0:00 \_ [aio/3] 309 ? S< 0:00 \_ [aio/4] 310 ? S< 0:00 \_ [aio/5] 311 ? S< 0:00 \_ [aio/6] 312 ? S< 0:00 \_ [aio/7] 472 ? S< 0:00 \_ [kpsmoused] 551 ? S< 0:00 \_ [ata/0] 552 ? S< 0:00 \_ [ata/1] 553 ? S< 0:00 \_ [ata/2] 554 ? S< 0:00 \_ [ata/3] 555 ? S< 0:00 \_ [ata/4] 556 ? S< 0:00 \_ [ata/5] 557 ? S< 0:00 \_ [ata/6] 558 ? S< 0:00 \_ [ata/7] 559 ? S< 0:00 \_ [ata_aux] 569 ? S< 0:00 \_ [scsi_eh_0] 570 ? S< 0:00 \_ [scsi_eh_1] 571 ? S< 0:00 \_ [scsi_eh_2] 572 ? S< 0:00 \_ [scsi_eh_3] 573 ? S< 0:00 \_ [scsi_eh_4] 574 ? S< 0:00 \_ [scsi_eh_5] 593 ? S< 0:00 \_ [kstriped] 630 ? S< 0:00 \_ [kjournald] 655 ? S< 0:00 \_ [kauditd] 1860 ? S< 0:00 \_ [kmpathd/0] 1861 ? S< 0:00 \_ [kmpathd/1] 1862 ? S< 0:00 \_ [kmpathd/2] 1863 ? S< 0:00 \_ [kmpathd/3] 1864 ? S< 0:00 \_ [kmpathd/4] 1865 ? S< 0:00 \_ [kmpathd/5] 1866 ? S< 0:00 \_ [kmpathd/6] 1867 ? S< 0:00 \_ [kmpathd/7] 1868 ? S< 0:00 \_ [kmpath_handlerd] 1902 ? S< 0:00 \_ [kjournald] 1904 ? S< 0:00 \_ [kjournald] 1906 ? S< 0:00 \_ [kjournald] 1908 ? S< 0:00 \_ [kjournald] 1910 ? S< 0:00 \_ [kjournald] 2184 ? S< 0:00 \_ [iscsi_eh] 2288 ? S< 0:00 \_ [cnic_wq] 2298 ? S< 0:00 \_ [bnx2i_thread/0] 2299 ? S< 0:00 \_ [bnx2i_thread/1] 2300 ? S< 0:00 \_ [bnx2i_thread/2] 2301 ? S< 0:00 \_ [bnx2i_thread/3] 2302 ? S< 0:00 \_ [bnx2i_thread/4] 2303 ? S< 0:00 \_ [bnx2i_thread/5] 2304 ? S< 0:00 \_ [bnx2i_thread/6] 2305 ? S< 0:00 \_ [bnx2i_thread/7] 2330 ? S< 0:00 \_ [ib_addr] 2359 ? S< 0:00 \_ [ib_mcast] 2360 ? S< 0:00 \_ [ib_inform] 2361 ? S< 0:00 \_ [local_sa] 2371 ? S< 0:00 \_ [iw_cm_wq] 2381 ? S< 0:00 \_ [ib_cm/0] 2382 ? S< 0:00 \_ [ib_cm/1] 2383 ? S< 0:00 \_ [ib_cm/2] 2384 ? S< 0:00 \_ [ib_cm/3] 2385 ? S< 0:00 \_ [ib_cm/4] 2386 ? S< 0:00 \_ [ib_cm/5] 2387 ? S< 0:00 \_ [ib_cm/6] 2388 ? S< 0:00 \_ [ib_cm/7] 2398 ? S< 0:00 \_ [rdma_cm] 2684 ? S< 0:00 \_ [bond0] 2882 ? S< 0:00 \_ [bond1] 3195 ? S< 0:00 \_ [kondemand/0] 3197 ? S< 0:00 \_ [kondemand/1] 3198 ? S< 0:00 \_ [kondemand/2] 3199 ? S< 0:00 \_ [kondemand/3] 3200 ? S< 0:00 \_ [kondemand/4] 3201 ? S< 0:00 \_ [kondemand/5] 3202 ? S< 0:00 \_ [kondemand/6] 3203 ? S< 0:00 \_ [kondemand/7] 688 ? S<s 0:00 /sbin/udevd -d 2425 ? S<Lsl 0:00 iscsiuio 2432 ? Ss 0:00 iscsid 2434 ? S<Ls 0:00 iscsid 3061 ? S<sl 0:00 auditd 3063 ? S<sl 0:00 \_ /sbin/audispd 3121 ? Ss 0:00 syslogd -m 0 3124 ? Ss 0:00 klogd -x 3220 ? Ss 0:00 irqbalance 3278 ? Ss 0:00 dbus-daemon --system 3324 ? Ss 0:00 /usr/sbin/acpid 3337 ? Ss 0:00 hald 3338 ? S 0:00 \_ hald-runner 3345 ? S 0:00 \_ hald-addon-acpi: listening on acpid socket /var/run/acpid.socket 3349 ? S 0:00 \_ hald-addon-keyboard: listening on /dev/input/event1 3360 ? S 0:00 \_ hald-addon-storage: polling /dev/sr0 3413 ? Ssl 0:00 automount 3435 ? Ssl 0:00 /usr/sbin/named -u named 3466 ? Ss 0:00 /usr/sbin/sshd 4072 ? Ss 0:00 \_ sshd: root@pts/0 4078 pts/0 Ss 0:00 \_ -bash 5436 pts/0 R+ 0:00 \_ ps axf 3484 ? Ss 0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid 3500 ? SLs 0:00 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g 3514 ? S 0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --pid-file=/var/lib/mysql/server.myserver.com.pid 3575 ? Sl 0:00 \_ /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --log-error=/var/lib/mysql/server.myserver.com.err --pid-fil 3687 ? Ss 0:00 /usr/sbin/exim -bd -q1h 3709 ? Ss 0:00 /usr/sbin/dovecot 3710 ? S 0:00 \_ dovecot-auth 3725 ? S 0:00 \_ pop3-login 3726 ? S 0:00 \_ pop3-login 3727 ? S 0:00 \_ imap-login 3728 ? S 0:00 \_ imap-login 3729 ? Ss 0:00 /usr/local/apache/bin/httpd -k start -DSSL 4326 ? S 0:00 \_ /usr/bin/perl /usr/local/cpanel/bin/leechprotect 4332 ? S 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL 4333 ? S 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL 4334 ? S 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL 4335 ? S 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL 4336 ? S 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL 4337 ? S 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL 4382 ? S 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL 4383 ? S 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL 4384 ? S 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL 5389 ? S 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL 5390 ? S 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL 3741 ? Ss 0:00 pure-ftpd (SERVER) 3746 ? S 0:00 /usr/sbin/pure-authd -s /var/run/ftpd.sock -r /usr/sbin/pureauth 3759 ? Ss 0:00 crond 3772 ? Ss 0:00 /usr/sbin/atd 3909 ? S 0:00 cpsrvd (SSL) - waiting for connections 5435 ? Z 0:00 \_ [cpsrvd-ssl] <defunct> 3931 ? S 0:00 queueprocd - wait to process a task 3948 ? S 0:00 tailwatchd 3954 ? SN 0:00 cpanellogd - sleeping for logs 4003 ? Ss 0:00 ./nimbus /opt/nimsoft 4016 ? S 0:00 \_ nimbus(controller) 4053 ? Sl 0:00 \_ nimbus(spooler) 4066 ? S 0:00 \_ nimbus(hdb) 4069 ? S 0:00 \_ nimbus(cdm) 4070 ? S 0:00 \_ nimbus(processes) 4023 ? S 0:00 /usr/sbin/smartd -q never 4027 tty1 Ss+ 0:00 /sbin/mingetty tty1 4028 tty2 Ss+ 0:00 /sbin/mingetty tty2 4029 tty3 Ss+ 0:00 /sbin/mingetty tty3 4030 tty4 Ss+ 0:00 /sbin/mingetty tty4 4031 tty5 Ss+ 0:00 /sbin/mingetty tty5 4033 tty6 Ss+ 0:00 /sbin/mingetty tty6 4035 ttyS1 Ss+ 0:00 /sbin/agetty -h -L ttyS1 19200 vt100 vmstat 10 6 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 0 3718136 25684 257424 0 0 8 3 127 189 0 0 100 0 0 0 0 0 3718136 25700 257420 0 0 0 7 1013 1500 0 0 100 0 0 0 0 0 3718136 25700 257424 0 0 0 1 1013 1551 0 0 100 0 0 0 0 0 3718136 25700 257424 0 0 0 0 1012 1469 0 0 100 0 0 1 0 0 3712680 25716 257424 0 0 0 2 1013 1542 0 0 100 0 0 0 0 0 3718376 25740 257424 0 0 0 46 1017 1534 0 0 100 0 0 Can anyone advise me as to what is the cause of and how I may resolve this behaviour? A kernel/driver conflict perhaps? I don't see any processes in R or D state that might inflate the load averages artificially, I realise it may be considered low in an 8 thread system but its higher at idle than any normal behaviour I've previously come across. Thanks in advance for your time. Edit: iotop Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND 26 be/3 root 0.00 B/s 0.00 B/s 0.00 % 0.29 % [events/0] 3205 be/3 root 0.00 B/s 0.00 B/s 0.00 % 0.10 % [kondemand/2] 3208 be/3 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kondemand/5] 3209 be/3 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kondemand/6] 3207 be/3 root 0.00 B/s 0.00 B/s 0.10 % 0.00 % [kondemand/4] 3210 be/3 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kondemand/7] 3227 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % irqbalance 3288 be/3 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rpciod/1] 3287 be/3 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rpciod/0] 3206 be/3 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kondemand/3] 3069 be/3 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % auditd 3070 be/2 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % audispd 655 be/3 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kauditd] 3619 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % automount 3 be/7 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/0] 3068 be/3 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % auditd 29 be/3 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [events/3] 4 rt/3 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [watchdog/0] 7 rt/3 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [watchdog/1] 10 rt/3 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [watchdog/2] 13 rt/3 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [watchdog/3] 16 rt/3 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [watchdog/4] 19 rt/3 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [watchdog/5] 22 rt/3 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [watchdog/6] 25 rt/3 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [watchdog/7] 27 be/3 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [events/1] 28 be/3 root 0.00 B/s 0.00 B/s 0.29 % 0.00 % [events/2] 30 be/3 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [events/4] 31 be/3 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [events/5] 32 be/3 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [events/6] 33 be/3 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [events/7] 34 be/3 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [khelper] 35 be/3 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kthread] 45 be/3 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kblockd/0]

  Read the article

< Previous Page | 2 3 4 5 6