Weird vps server issue
Posted
by
anon-user0
on Server Fault
See other posts from Server Fault
or by anon-user0
Published on 2012-06-27T10:53:55Z
Indexed on
2012/06/27
15:18 UTC
Read the original article
Hit count: 1363
I have an unmanaged linux vps Ubuntu 11.10 (Oneiric Ocelot). I have LNMP installed. Also php-fpm php-apc, varnish, memcache.
I have (or rather had) several live sites on it.
under normal load the server uses ~700 mb memory. But since last night its using only 20mb~ memory and a lot of the services seems to be down (according to htop) I only see nginx working and mysql starts up and goes does every few minutes on a loop.
Here are some information on the server that might help you help me:
root@server:~# uname -a
Linux server 2.6.18-308.el5.028stab099.3 #1 SMP Wed Mar 7 15:56:00 MSK 2012 i686 i686 i386 GNU/Linux
-
root@server:~# ifconfig -a
lo Link encap:Local Loopback
LOOPBACK MTU:16436 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
venet0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet addr:127.0.0.2 P-t-P:127.0.0.2 Bcast:0.0.0.0 Mask:255.255.255.255
UP BROADCAST POINTOPOINT RUNNING NOARP MTU:1500 Metric:1
RX packets:12515 errors:0 dropped:0 overruns:0 frame:0
TX packets:9541 errors:0 dropped:1 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:7191214 (7.1 MB) TX bytes:536726 (536.7 KB)
venet0:0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet addr:176.31.158.78 P-t-P:176.31.158.78 Bcast:0.0.0.0 Mask:255.255.255.255
UP BROADCAST POINTOPOINT RUNNING NOARP MTU:1500 Metric:1
-
root@server:~# netstat -l
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 *:http-alt *:* LISTEN
tcp 0 0 *:ssh *:* LISTEN
tcp6 0 0 [::]:http-alt [::]:* LISTEN
tcp6 0 0 [::]:ssh [::]:* LISTEN
Active UNIX domain sockets (only servers)
Proto RefCnt Flags Type State I-Node Path
unix 2 [ ACC ] STREAM LISTENING 9307368 @/com/ubuntu/upstart
-
-
htop: http://i.stack.imgur.com/NHKYX.png
EDIT: Stressed. mind was not working adding log:
root@server:~# less /var/log/syslog
Jun 27 05:27:42 server syslogd 1.5.0#6ubuntu1: restart.
Jun 27 05:39:01 server CRON[9298]: (root) CMD ([ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) -delete)
Jun 27 05:40:01 server CRON[9463]: (smmsp) CMD (test -x /etc/init.d/sendmail && /usr/share/sendmail/sendmail cron-msp)
Jun 27 05:46:21 server sm-msp-queue[9480]: q5R1R7Ue004056: to=root, ctladdr=root (0/0), delay=00:19:14, xdelay=00:06:18, mailer=relay, pri=122407, relay=[127.0.0.1] [127.0.0.1], dsn=4.0.0, stat=Deferred: Connection timed out with [127.0.0.1]
Jun 27 05:52:39 server sm-msp-queue[9480]: q5QMk7S9009582: to=root, ctladdr=root (0/0), delay=03:06:32, xdelay=00:06:18, mailer=relay, pri=842407, relay=[127.0.0.1] [127.0.0.1], dsn=4.0.0, stat=Deferred: Connection timed out with [127.0.0.1]
Jun 27 06:00:01 server CRON[15671]: (smmsp) CMD (test -x /etc/init.d/sendmail && /usr/share/sendmail/sendmail cron-msp)
Jun 27 06:06:22 server sm-msp-queue[15690]: q5R1R7Ue004056: to=root, ctladdr=root (0/0), delay=00:39:15, xdelay=00:06:18, mailer=relay, pri=212407, relay=[127.0.0.1] [127.0.0.1], dsn=4.0.0, stat=Deferred: Connection timed out with [127.0.0.1]
Jun 27 06:09:01 server CRON[18114]: (root) CMD ([ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) -delete)
Jun 27 06:12:40 server sm-msp-queue[15690]: q5QMk7S9009582: to=root, ctladdr=root (0/0), delay=03:26:33, xdelay=00:06:18, mailer=relay, pri=932407, relay=[127.0.0.1] [127.0.0.1], dsn=4.0.0, stat=Deferred: Connection timed out with [127.0.0.1]
Jun 27 06:20:02 server CRON[21888]: (smmsp) CMD (test -x /etc/init.d/sendmail && /usr/share/sendmail/sendmail cron-msp)
Jun 27 06:26:22 server sm-msp-queue[21907]: q5R1R7Ue004056: to=root, ctladdr=root (0/0), delay=00:59:15, xdelay=00:06:18, mailer=relay, pri=302407, relay=[127.0.0.1] [127.0.0.1], dsn=4.0.0, stat=Deferred: Connection timed out with [127.0.0.1]
Jun 27 06:27:02 server CRON[24021]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Jun 27 06:32:40 server sm-msp-queue[21907]: q5QMk7S9009582: to=root, ctladdr=root (0/0), delay=03:46:33, xdelay=00:06:18, mailer=relay, pri=1022407, relay=[127.0.0.1] [127.0.0.1], dsn=4.0.0, stat=Deferred: Connection timed out with [127.0.0.1]
Jun 27 06:39:01 server CRON[27941]: (root) CMD ([ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) -delete)
Jun 27 06:40:02 server CRON[28110]: (smmsp) CMD (test -x /etc/init.d/sendmail && /usr/share/sendmail/sendmail cron-msp)
Jun 27 06:46:22 server sm-msp-queue[28125]: q5R1R7Ue004056: to=root, ctladdr=root (0/0), delay=01:19:15, xdelay=00:06:18, mailer=relay, pri=392407, relay=[127.0.0.1] [127.0.0.1], dsn=4.0.0, stat=Deferred: Connection timed out with [127.0.0.1]
Jun 27 06:52:40 server sm-msp-queue[28125]: q5QMk7S9009582: to=root, ctladdr=root (0/0), delay=04:06:33, xdelay=00:06:18, mailer=relay, pri=1112407, relay=[127.0.0.1] [127.0.0.1], dsn=4.0.0, stat=Deferred: Connection timed out with [127.0.0.1]
Jun 27 06:52:40 server sm-msp-queue[28125]: q5QMk7S9009582: q5R2e4uo028125: sender notify: Warning: could not send message for past 4 hours
Jun 27 06:52:44 server sm-msp-queue[28125]: q5R2e4uo028125: to=root, delay=00:00:04, xdelay=00:00:04, mailer=relay, pri=33690, relay=[127.0.0.1] [127.0.0.1], dsn=4.0.0, stat=Deferred: Connection timed out with [127.0.0.1]
Jun 27 07:00:02 server CRON[1543]: (smmsp) CMD (test -x /etc/init.d/sendmail && /usr/share/sendmail/sendmail cron-msp)
Jun 27 07:06:21 server sm-msp-queue[1560]: q5R2e4uo028125: to=root, delay=00:13:41, xdelay=00:06:18, mailer=relay, pri=123690, relay=[127.0.0.1] [127.0.0.1], dsn=4.0.0, stat=Deferred: Connection timed out with [127.0.0.1]
Jun 27 07:09:01 server CRON[3986]: (root) CMD ([ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) -delete)
Jun 27 07:12:39 server sm-msp-queue[1560]: q5R1R7Ue004056: to=root, ctladdr=root (0/0), delay=01:45:32, xdelay=00:06:18, mailer=relay, pri=482407, relay=[127.0.0.1] [127.0.0.1], dsn=4.0.0, stat=Deferred: Connection timed out with [127.0.0.1]
Jun 27 07:18:57 server sm-msp-queue[1560]: q5QMk7S9009582: to=root, ctladdr=root (0/0), delay=04:32:50, xdelay=00:06:18, mailer=relay, pri=1202407, relay=[127.0.0.1] [127.0.0.1], dsn=4.0.0, stat=Deferred: Connection timed out with [127.0.0.1]
Jun 27 07:20:02 server CRON[7760]: (smmsp) CMD (test -x /etc/init.d/sendmail && /usr/share/sendmail/sendmail cron-msp)
Jun 27 07:26:22 server sm-msp-queue[7775]: q5R2e4uo028125: to=root, delay=00:33:42, xdelay=00:06:18, mailer=relay, pri=213690, relay=[127.0.0.1] [127.0.0.1], dsn=4.0.0, stat=Deferred: Connection timed out with [127.0.0.1]
Jun 27 07:27:01 server CRON[9887]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Jun 27 07:32:40 server sm-msp-queue[7775]: q5R1R7Ue004056: to=root, ctladdr=root (0/0), delay=02:05:33, xdelay=00:06:18, mailer=relay, pri=572407, relay=[127.0.0.1] [127.0.0.1], dsn=4.0.0, stat=Deferred: Connection timed out with [127.0.0.1]
Jun 27 07:38:58 server sm-msp-queue[7775]: q5QMk7S9009582: to=root, ctladdr=root (0/0), delay=04:52:51, xdelay=00:06:18, mailer=relay, pri=1292407, relay=[127.0.0.1] [127.0.0.1], dsn=4.0.0, stat=Deferred: Connection timed out with [127.0.0.1]
Jun 27 07:39:01 server CRON[13813]: (root) CMD ([ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth :
root@server:~# df -h Filesystem Size Used Avail Use% Mounted on /dev/simfs 20G 2.3G 18G 12% /
-
Jun 26 16:22:41 server varnishd[1413]: Child (32425) died signal=3
Jun 26 16:22:41 server varnishd[1413]: child (21687) Started
Jun 26 16:22:41 server varnishd[1413]: Child (21687) said Child starts
Jun 26 16:22:41 server varnishd[1413]: Child (21687) said SMF.s0 mmap'ed 1073741824 bytes of 1073741824
Jun 26 16:34:28 server -- MARK --
Jun 26 16:54:29 server -- MARK --
Jun 26 17:14:29 server -- MARK --
Jun 26 17:34:29 server -- MARK --
Jun 26 17:54:29 server -- MARK --
Jun 26 18:14:29 server -- MARK --
Jun 26 18:34:29 server -- MARK --
Jun 26 18:54:29 server -- MARK --
Jun 26 19:14:29 server -- MARK --
Jun 26 19:34:29 server -- MARK --
Jun 26 19:54:29 server -- MARK --
Jun 26 20:14:29 server -- MARK --
Jun 26 20:34:29 server -- MARK --
Jun 26 20:48:12 server exiting on signal 15
Jun 26 20:51:58 server syslogd 1.5.0#6ubuntu1: restart.
Jun 26 20:52:01 server varnishd[1324]: Platform: Linux,2.6.18-308.el5.028stab099.3,i686,-sfile,-smalloc,-hcritbit
Jun 26 21:11:58 server -- MARK --
Jun 26 21:31:58 server -- MARK --
Jun 26 21:51:58 server -- MARK --
Jun 26 22:11:58 server -- MARK --
Jun 26 22:31:58 server -- MARK --
Jun 26 22:51:58 server -- MARK --
Jun 26 23:11:58 server -- MARK --
Jun 26 23:31:58 server -- MARK --
Jun 26 23:51:58 server -- MARK --
Jun 27 00:11:58 server -- MARK --
Jun 27 00:23:42 server exiting on signal 15
Jun 27 02:21:10 server syslogd 1.5.0#6ubuntu1: restart.
Jun 27 02:21:12 server varnishd[1341]: Platform: Linux,2.6.18-308.el5.028stab099.3,i686,-sfile,-smalloc,-hcritbit
Jun 27 02:41:10 server -- MARK --
Jun 27 02:46:41 server syslogd 1.5.0#6ubuntu1: restart.
Jun 27 03:20:44 server syslogd 1.5.0#6ubuntu1: restart.
Jun 27 03:20:46 server varnishd[1238]: Platform: Linux,2.6.18-308.el5.028stab099.3,i686,-sfile,-smalloc,-hcritbit
Jun 27 03:20:46 server varnishd[1238]: child (1239) Started
Jun 27 03:20:46 server varnishd[1238]: Child (1239) said Child starts
Jun 27 03:20:46 server varnishd[1238]: Child (1239) said SMF.s0 mmap'ed 1073741824 bytes of 1073741824
Jun 27 03:32:52 server exiting on signal 15
Jun 27 03:33:16 server syslogd 1.5.0#6ubuntu1: restart.
Jun 27 03:33:31 server varnishd[1372]: Platform: Linux,2.6.18-308.el5.028stab099.3,i686,-sfile,-smalloc,-hcritbit
Jun 27 03:53:16 server -- MARK --
Jun 27 04:13:16 server -- MARK --
Jun 27 04:33:16 server -- MARK --
Jun 27 04:53:16 server -- MARK --
Jun 27 05:13:16 server -- MARK --
Jun 27 05:27:42 server syslogd 1.5.0#6ubuntu1: restart.
Jun 27 05:53:17 server -- MARK --
Jun 27 06:13:17 server -- MARK --
Jun 27 06:33:17 server -- MARK --
Jun 27 06:53:17 server -- MARK --
Jun 27 07:13:17 server -- MARK --
Jun 27 07:33:17 server -- MARK --
Jun 27 07:53:17 server -- MARK --
Jun 27 08:13:17 server -- MARK --
Jun 27 08:33:17 server -- MARK --
Jun 27 08:53:17 server -- MARK --
Jun 27 09:13:17 server -- MARK --
Jun 27 09:33:17 server -- MARK --
Jun 27 09:53:17 server -- MARK --
Jun 27 10:13:17 server -- MARK --
Jun 27 10:33:17 server -- MARK --
Jun 27 10:53:17 server -- MARK --
Jun 27 11:13:17 server -- MARK --
Jun 27 11:33:17 server -- MARK --
Jun 27 11:53:18 server -- MARK --
Jun 27 12:13:18 server -- MARK --
Jun 27 12:33:18 server -- MARK --
Jun 27 12:53:18 server -- MARK --
Jun 27 13:13:18 server -- MARK --
Jun 27 13:33:18 server -- MARK --
Jun 27 13:53:18 server -- MARK --
Jun 27 14:13:18 server -- MARK --
Jun 27 14:33:18 server -- MARK --
Jun 27 14:53:18 server -- MARK --
--
root@server:~# cat /var/log/nginx/error.log
2012/06/27 03:32:54 [alert] 1199#0: worker process 1203 exited on signal 9
2012/06/27 03:32:54 [alert] 1199#0: worker process 1200 exited on signal 9
2012/06/27 03:32:54 [alert] 1199#0: worker process 1201 exited on signal 9
2012/06/27 03:32:54 [alert] 1199#0: worker process 1202 exited on signal 9
root@server:~# cat /var/log/nginx/access.log
31.210.99.87 - - [27/Jun/2012:09:09:08 +0400] "GET /w00tw00t.at.ISC.SANS.DFind:) HTTP/1.1" 400 172 "-" "-"
88.191.138.103 - - [27/Jun/2012:13:27:08 +0400] "GET /cms/cmx.jsp HTTP/1.1" 301 184 "-" "-"
88.191.138.103 - - [27/Jun/2012:13:27:08 +0400] "GET /iesvc/iesvc.jsp HTTP/1.1" 301 184 "-" "-"
88.191.138.103 - - [27/Jun/2012:13:27:08 +0400] "GET /cmd2/index.jsp HTTP/1.1" 301 184 "-" "-"
88.191.138.103 - - [27/Jun/2012:13:27:09 +0400] "GET /cmd/index.jsp HTTP/1.1" 301 184 "-" "-"
58.97.147.197 - - [27/Jun/2012:17:17:19 +0400] "GET / HTTP/1.1" 301 184 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.56 Safari/536.5"
58.97.147.197 - - [27/Jun/2012:17:17:37 +0400] "GET / HTTP/1.1" 301 184 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.56 Safari/536.5"
58.97.147.197 - - [27/Jun/2012:17:17:38 +0400] "-" 400 0 "-" "-"
58.97.147.197 - - [27/Jun/2012:17:17:38 +0400] "-" 400 0 "-" "-"
58.97.147.197 - - [27/Jun/2012:17:17:48 +0400] "-" 400 0 "-" "-"
-
root@server:~# cat /var/log/daemon.log
Jun 26 20:48:10 server xinetd[1177]: Exiting...
Jun 26 20:51:58 server xinetd[1174]: Reading included configuration file: /etc/xinetd.d/daytime [file=/etc/xinetd.d/daytime] [line=28]
Jun 26 20:51:58 server xinetd[1174]: Reading included configuration file: /etc/xinetd.d/discard [file=/etc/xinetd.d/discard] [line=26]
Jun 26 20:51:58 server xinetd[1174]: Reading included configuration file: /etc/xinetd.d/echo [file=/etc/xinetd.d/echo] [line=25]
Jun 26 20:51:58 server xinetd[1174]: Reading included configuration file: /etc/xinetd.d/time [file=/etc/xinetd.d/time] [line=26]
Jun 26 20:51:58 server xinetd[1174]: removing chargen
Jun 26 20:51:58 server xinetd[1174]: removing chargen
Jun 26 20:51:58 server xinetd[1174]: removing daytime
Jun 26 20:51:58 server xinetd[1174]: removing daytime
Jun 26 20:51:58 server xinetd[1174]: removing discard
Jun 26 20:51:58 server xinetd[1174]: removing discard
Jun 26 20:51:58 server xinetd[1174]: removing echo
Jun 26 20:51:58 server xinetd[1174]: removing echo
Jun 26 20:51:58 server xinetd[1174]: removing time
Jun 26 20:51:58 server xinetd[1174]: removing time
Jun 26 20:51:58 server xinetd[1174]: xinetd Version 2.3.14 started with libwrap loadavg options compiled in.
Jun 26 20:51:58 server xinetd[1174]: Started working: 0 available services
Jun 26 20:52:01 server vnstatd[1330]: vnStat daemon 1.11 started.
Jun 26 20:52:01 server vnstatd[1330]: Monitoring: venet0
Jun 27 00:23:41 server xinetd[1174]: Exiting...
Jun 27 02:21:12 server vnstatd[1349]: vnStat daemon 1.11 started.
Jun 27 02:21:12 server vnstatd[1349]: Monitoring: venet0
Jun 27 03:20:44 server xinetd[1166]: attribute: disable should not be in default section [file=/etc/xinetd.conf] [line=12]
Jun 27 03:20:44 server xinetd[1166]: Reading included configuration file: /etc/xinetd.d/chargen [file=/etc/xinetd.conf] [line=15]
Jun 27 03:20:44 server xinetd[1166]: Reading included configuration file: /etc/xinetd.d/daytime [file=/etc/xinetd.d/daytime] [line=28]
Jun 27 03:20:44 server xinetd[1166]: Reading included configuration file: /etc/xinetd.d/discard [file=/etc/xinetd.d/discard] [line=26]
Jun 27 03:20:44 server xinetd[1166]: Reading included configuration file: /etc/xinetd.d/echo [file=/etc/xinetd.d/echo] [line=25]
Jun 27 03:20:44 server xinetd[1166]: Reading included configuration file: /etc/xinetd.d/time [file=/etc/xinetd.d/time] [line=26]
Jun 27 03:20:44 server xinetd[1166]: removing chargen
Jun 27 03:20:44 server xinetd[1166]: removing chargen
Jun 27 03:20:44 server xinetd[1166]: removing daytime
Jun 27 03:20:44 server xinetd[1166]: removing daytime
Jun 27 03:20:44 server xinetd[1166]: removing discard
Jun 27 03:20:44 server xinetd[1166]: removing discard
Jun 27 03:20:44 server xinetd[1166]: removing echo
Jun 27 03:20:44 server xinetd[1166]: removing echo
Jun 27 03:20:44 server xinetd[1166]: removing time
Jun 27 03:20:44 server xinetd[1166]: removing time
Jun 27 03:20:44 server xinetd[1166]: xinetd Version 2.3.14 started with libwrap loadavg options compiled in.
Jun 27 03:20:44 server xinetd[1166]: Started working: 0 available services
Jun 27 03:20:46 server vnstatd[1249]: vnStat daemon 1.11 started.
Jun 27 03:20:46 server vnstatd[1249]: Monitoring: venet0
Jun 27 03:32:41 server xinetd[1166]: Exiting...
Jun 27 03:33:32 server vnstatd[1380]: vnStat daemon 1.11 started.
Jun 27 03:33:32 server vnstatd[1380]: Monitoring: venet0
root@server:~#
-
Anything else you need let me know
© Server Fault or respective owner
