EC2 instance suddenly refusing SSH connections and won't respond to ping

Posted by Chris on Server Fault See other posts from Server Fault or by Chris
Published on 2010-10-01T19:59:48Z Indexed on 2011/01/28 23:28 UTC
Read the original article Hit count: 400

Filed under:
|
|
|
|

My instance was running fine and this morning I was able to access a Ruby on Rails app hosted on it. An hour later I suddenly wasn't able to access my site, my SSH connection attempts were refused and the server wasn't even responding to ping. I didn't change anything on my system during that hour and reboots aren't fixing it.

I've never had any problems connecting or pinging the system before.

Can someone please help? This is on my production system!

  • OS: CentOS 5
  • AMI ID: ami-10b55379
  • Type: m1.small


[] ~% ssh -v *****@meeteor.com 
OpenSSH_5.2p1, OpenSSL 0.9.8l 5 Nov 2009 
debug1: Reading configuration data /etc/ssh_config 
debug1: Connecting to meeteor.com [184.73.235.191] port 22. 
debug1: connect to address 184.73.235.191 port 22: Connection refused 
ssh: connect to host meeteor.com port 22: Connection refused 
[] ~% ping meeteor.com 
PING meeteor.com (184.73.235.191): 56 data bytes 
Request timeout for icmp_seq 0 
Request timeout for icmp_seq 1 
Request timeout for icmp_seq 2 
^C 
--- meeteor.com ping statistics --- 
4 packets transmitted, 0 packets received, 100.0% packet loss 
[] ~% 

========= System Log =========

Restarting system. 
Linux version 2.6.16-xenU ([email protected]) (gcc version 4.0.1 20050727 (Red Hat 4.0.1-5)) #1 SMP Mon May 28 03:41:49 SAST 2007 
BIOS-provided physical RAM map: 
Xen: 0000000000000000 - 000000006a400000 (usable) 
980MB HIGHMEM available. 
727MB LOWMEM available. 
NX (Execute Disable) protection: active 
IRQ lockup detection disabled 
Built 1 zonelists 
Kernel command line: root=/dev/sda1 ro 4 
Enabling fast FPU save and restore... done. 
Enabling unmasked SIMD FPU exception support... done. 
Initializing CPU#0 
PID hash table entries: 4096 (order: 12, 65536 bytes) 
Xen reported: 2599.998 MHz processor. 
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes) 
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes) 
Software IO TLB disabled 
vmalloc area: ee000000-f53fe000, maxmem 2d7fe000 
Memory: 1718700k/1748992k available (1958k kernel code, 20948k reserved, 620k data, 144k init, 1003528k highmem) 
Checking if this processor honours the WP bit even in supervisor mode... Ok. 
Calibrating delay using timer specific routine.. 5202.30 BogoMIPS (lpj=26011526) 
Mount-cache hash table entries: 512 
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line) 
CPU: L2 Cache: 1024K (64 bytes/line) 
Checking 'hlt' instruction... OK. 
Brought up 1 CPUs 
migration_cost=0 
Grant table initialized 
NET: Registered protocol family 16 
Brought up 1 CPUs 
xen_mem: Initialising balloon driver. 
highmem bounce pool size: 64 pages 
VFS: Disk quotas dquot_6.5.1 
Dquot-cache hash table entries: 1024 (order 0, 4096 bytes) 
Initializing Cryptographic API 
io scheduler noop registered 
io scheduler anticipatory registered (default) 
io scheduler deadline registered 
io scheduler cfq registered 
i8042.c: No controller found. 
RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize 
Xen virtual console successfully installed as tty1 
Event-channel device installed. 
netfront: Initialising virtual ethernet driver. 
mice: PS/2 mouse device common for all mice 
md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27 
md: bitmap version 4.39 
NET: Registered protocol family 2 
Registering block device major 8 
IP route cache hash table entries: 65536 (order: 6, 262144 bytes) 
TCP established hash table entries: 262144 (order: 9, 2097152 bytes) 
TCP bind hash table entries: 65536 (order: 7, 524288 bytes) 
TCP: Hash tables configured (established 262144 bind 65536) 
TCP reno registered 
TCP bic registered 
NET: Registered protocol family 1 
NET: Registered protocol family 17 
NET: Registered protocol family 15 
Using IPI No-Shortcut mode 
md: Autodetecting RAID arrays. 
md: autorun ... 
md: ... autorun DONE. 
kjournald starting. Commit interval 5 seconds 
EXT3-fs: mounted filesystem with ordered data mode. 
VFS: Mounted root (ext3 filesystem) readonly. 
Freeing unused kernel memory: 144k freed 
*************************************************************** 
*************************************************************** 
** WARNING: Currently emulating unsupported memory accesses ** 
** in /lib/tls glibc libraries. The emulation is ** 
** slow. To ensure full performance you should ** 
** install a 'xen-friendly' (nosegneg) version of ** 
** the library, or disable tls support by executing ** 
** the following as root: ** 
** mv /lib/tls /lib/tls.disabled ** 
** Offending process: init (pid=1) ** 
*************************************************************** 
*************************************************************** 
Pausing... 5Pausing... 4Pausing... 3Pausing... 2Pausing... 1Continuing... 
INIT: version 2.86 booting 
Welcome to CentOS release 5.4 (Final) 
Press 'I' to enter interactive startup. 
Setting clock : Fri Oct 1 14:35:26 EDT 2010 [ OK ] 
Starting udev: [ OK ] 
Setting hostname localhost.localdomain: [ OK ] 
No devices found 
Setting up Logical Volume Management: [ OK ] 
Checking filesystems 
Checking all file systems. 
[/sbin/fsck.ext3 (1) -- /] fsck.ext3 -a /dev/sda1 
/dev/sda1: clean, 275424/1310720 files, 1161123/2621440 blocks 
[ OK ] 
Remounting root filesystem in read-write mode: [ OK ] 
Mounting local filesystems: [ OK ] 
Enabling local filesystem quotas: [ OK ] 
Enabling /etc/fstab swaps: [ OK ] 
INIT: Entering runlevel: 4 
Entering non-interactive startup 
Starting background readahead: [ OK ] 
Applying ip6tables firewall rules: modprobe: FATAL: Module ip6_tables not found. 
ip6tables-restore v1.3.5: ip6tables-restore: unable to initializetable 'filter' 
Error occurred at line: 3 
Try `ip6tables-restore -h' or 'ip6tables-restore --help' for more information. 
[FAILED] 
Applying iptables firewall rules: [ OK ] 
Loading additional iptables modules: ip_conntrack_netbios_ns [ OK ] 
Bringing up loopback interface: [ OK ] 
Bringing up interface eth0: 
Determining IP information for eth0... done. 
[ OK ] 
Starting auditd: [FAILED] 
Starting irqbalance: [ OK ] 
Starting portmap: [ OK ] 
FATAL: Module lockd not found. 
Starting NFS statd: [ OK ] 
Starting RPC idmapd: FATAL: Module sunrpc not found. 
FATAL: Error running install command for sunrpc 
Error: RPC MTAB does not exist. 
Starting system message bus: [ OK ] 
Starting Bluetooth services:[ OK ] 
[ OK ] 
Can't open RFCOMM control socket: Address family not supported by protocol 
Mounting other filesystems: [ OK ] 
Starting PC/SC smart card daemon (pcscd): [ OK ] 
Starting hidd: Can't open HIDP control socket: Address family not supported by protocol 
[FAILED] 
Starting autofs: Starting automount: automount: test mount forbidden or incorrect kernel protocol version, kernel protocol version 5.00 or above required. 
[FAILED] 
[FAILED] 
Starting sshd: [ OK ] 
Starting cups: [ OK ] 
Starting sendmail: [ OK ] 
Starting sm-client: [ OK ] 
Starting console mouse services: no console device found[FAILED] 
Starting crond: [ OK ] 
Starting xfs: [ OK ] 
Starting anacron: [ OK ] 
Starting atd: [ OK ] 
% Total % Received % Xferd Average Speed Time Time Time Current 
Dload Upload Total Spent Left Speed 
100 390 100 390 0 0 58130 0 --:--:-- --:--:-- --:--:-- 58130 
100 390 100 390 0 0 56984 0 --:--:-- --:--:-- --:--:-- 0 
Starting yum-updatesd: [ OK ] 
Starting Avahi daemon... [ OK ] 
Starting HAL daemon: [ OK ] 
Starting OSSEC: [ OK ] 
Starting smartd: [ OK ] 
c 
CentOS release 5.4 (Final) 
Kernel 2.6.16-xenU on an i686 
domU-12-31-39-00-C4-97 login: 
INIT: Id "2" respawning too fast: disabled for 5 minutes 
INIT: Id "3" respawning too fast: disabled for 5 minutes 
INIT: Id "4" respawning too fast: disabled for 5 minutes 
INIT: Id "5" respawning too fast: disabled for 5 minutes 
INIT: Id "6" respawning too fast: disabled for 5 minutes 

© Server Fault or respective owner

Related posts about ssh

Related posts about amazon-ec2