I'm running a Debian server and a couple of days ago my rsyslog started to behave very weird, the daemon is running but it doesn't seem to do anything. Many people use the system but I'm the only one with (legal) root access.
I'm using the default rsyslogd configuration (if you think is relevant I'll attach it, but it's the one that comes with the package).
After I rotated all the log files, they have remained empty:
# ls -l /var/log/*.log
-rw-r--r-- 1 root root 0 Jun 27 00:25 /var/log/alternatives.log
-rw-r----- 1 root adm 0 Jun 26 13:03 /var/log/auth.log
-rw-r----- 1 root adm 0 Jun 26 13:03 /var/log/daemon.log
-rw-r--r-- 1 root root 0 Jun 27 00:25 /var/log/dpkg.log
-rw-r----- 1 root adm 0 Jun 26 13:03 /var/log/kern.log
-rw-r----- 1 root adm 0 Jun 26 13:03 /var/log/lpr.log
-rw-r----- 1 root adm 0 Jun 26 13:03 /var/log/mail.log
-rw-r----- 1 root adm 0 Jun 26 13:03 /var/log/user.log
Any try to force a log writing does not have any effect:
# logger hey
# ls -l /var/log/messages
-rw-r----- 1 root adm 0 Jun 26 13:03 /var/log/messages
Lsof shows that rsyslogd does not have any log files opened:
# lsof -p 1855
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
rsyslogd 1855 root cwd DIR 202,0 4096 2 /
rsyslogd 1855 root rtd DIR 202,0 4096 2 /
rsyslogd 1855 root txt REG 202,0 342076 21649 /usr/sbin/rsyslogd
rsyslogd 1855 root mem REG 202,0 38556 32153 /lib/i386-linux-gnu/i686/cmov/libnss_nis-2.13.so
rsyslogd 1855 root mem REG 202,0 79728 32165 /lib/i386-linux-gnu/i686/cmov/libnsl-2.13.so
rsyslogd 1855 root mem REG 202,0 26456 32163 /lib/i386-linux-gnu/i686/cmov/libnss_compat-2.13.so
rsyslogd 1855 root mem REG 202,0 297500 1061058 /usr/lib/rsyslog/imuxsock.so
rsyslogd 1855 root mem REG 202,0 42628 32170 /lib/i386-linux-gnu/i686/cmov/libnss_files-2.13.so
rsyslogd 1855 root mem REG 202,0 22784 1061106 /usr/lib/rsyslog/imklog.so
rsyslogd 1855 root mem REG 202,0 1401000 32169 /lib/i386-linux-gnu/i686/cmov/libc-2.13.so
rsyslogd 1855 root mem REG 202,0 30684 32175 /lib/i386-linux-gnu/i686/cmov/librt-2.13.so
rsyslogd 1855 root mem REG 202,0 9844 32157 /lib/i386-linux-gnu/i686/cmov/libdl-2.13.so
rsyslogd 1855 root mem REG 202,0 117009 32154 /lib/i386-linux-gnu/i686/cmov/libpthread-2.13.so
rsyslogd 1855 root mem REG 202,0 79980 17746 /usr/lib/libz.so.1.2.3.4
rsyslogd 1855 root mem REG 202,0 18836 1061094 /usr/lib/rsyslog/lmnet.so
rsyslogd 1855 root mem REG 202,0 117960 31845 /lib/i386-linux-gnu/ld-2.13.so
rsyslogd 1855 root 0u unix 0xebe8e800 0t0 640 /dev/log
rsyslogd 1855 root 3u FIFO 0,5 0t0 2474 /dev/xconsole
rsyslogd 1855 root 4u unix 0xebe8e400 0t0 645 /var/spool/postfix/dev/log
rsyslogd 1855 root 5r REG 0,3 0 4026532176 /proc/kmsg
I was so frustrated that even reinstall the rsyslog package, but it still refuses to log anything:
# apt-get remove --purge rsyslog
# apt-get install rsyslog
I thought someone had hacked the system, so run rkhunter, chkrootkit, unhide in an attempt to find hide processes / ports and nmap in a remote host to compare with the ports shown by netstat. And I know this doesn't mean anything, but all looks ok. The system also have an iptables firewall that is very restrictive with incoming / outgoing connections.
This is driving me crazy, any idea what is going on here?
[EDIT - disk space info]
# df -h
Filesystem Size Used Avail Use% Mounted on
rootfs 24G 22G 629M 98% /
/dev/root 24G 22G 629M 98% /
devtmpfs 10M 112K 9.9M 2% /dev
tmpfs 76M 48K 76M 1% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 151M 40K 151M 1% /tmp
tmpfs 151M 0 151M 0% /run/shm
