I'm looking into why our ubuntu web server hung this morning and see that there was some package upgrades a few hours prior to the problem.
I was able to ssh into the system and get a snapshot from top:
top - 08:13:54 up 210 days, 8:25, 2 users, load average: 433.30, 422.40, 375.70
Tasks: 1192 total, 381 running, 810 sleeping, 0 stopped, 1 zombie
Cpu(s): 0.5%us, 6.1%sy, 0.0%ni, 93.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 49549772k total, 48518392k used, 1031380k free, 960152k buffers
Swap: 11595768k total, 279368k used, 11316400k free, 39355664k cached
This is a 16 processor system, so I would typically expect a load in the low teens.
I tried to restart apache, which didn't work, and subsequently had to do a hard reboot to get it working again (which it is).
One thing I found was that the server did an unattended package update.
Is it possible that upgrading php or curl (which our web sites use) might have caused apache to stop responding?
Here is the snip from the unattended-upgrades.log from this morning:
2012-09-18 06:48:30,076 INFO Initial blacklisted packages:
2012-09-18 06:48:30,076 INFO Starting unattended upgrades script
2012-09-18 06:48:30,077 INFO Allowed origins are: ["['Ubuntu', 'lucid-security']"]
2012-09-18 06:49:37,017 INFO Packages that are upgraded: gnupg-curl php5-dev linux-server dhcp3-common linux-libc-dev php5-curl gpgv gnupg linux-headers-server linux-image-server php5 php5-mysql php-pear php5-cli php5-common libapache2-mod-php5 dhcp3-client
2012-09-18 06:49:37,018 INFO Writing dpkg log to '/var/log/unattended-upgrades/unattended-upgrades-dpkg_2012-09-18_06:49:37.017909.log'
