Tracking down rogue disk usage

Posted by Amadan on Server Fault See other posts from Server Fault or by Amadan
Published on 2014-08-20T03:45:14Z Indexed on 2014/08/20 4:23 UTC
Read the original article Hit count: 416

Filed under:
|
|

I found several other questions regarding the theory behind my problem (e.g. this, this), but I don't know how to apply the answers to my machine.

# du -hsx /
11000283    /
# df -kT /
Filesystem               Type 1K-blocks      Used Available Use% Mounted on
/dev/mapper/csisv13-root ext4 516032952 361387456 128432532  74% /

There is a big difference between 11G (du) and 345G (df). Where are the remaining 334G?

It's not in deleted files. There was only one, it was short, and I truncated it just in case. This is what remains:

# lsof -a +L1 /
COMMAND    PID   USER   FD   TYPE DEVICE SIZE/OFF NLINK     NODE NAME
zabbix_ag 4902 zabbix    1w   REG  252,0        0     0 28836028 /var/log/zabbix-agent/zabbix_agentd.log.1 (deleted)
zabbix_ag 4902 zabbix    2w   REG  252,0        0     0 28836028 /var/log/zabbix-agent/zabbix_agentd.log.1 (deleted)
zabbix_ag 4906 zabbix    1w   REG  252,0        0     0 28836028 /var/log/zabbix-agent/zabbix_agentd.log.1 (deleted)
zabbix_ag 4906 zabbix    2w   REG  252,0        0     0 28836028 /var/log/zabbix-agent/zabbix_agentd.log.1 (deleted)
zabbix_ag 4907 zabbix    1w   REG  252,0        0     0 28836028 /var/log/zabbix-agent/zabbix_agentd.log.1 (deleted)
zabbix_ag 4907 zabbix    2w   REG  252,0        0     0 28836028 /var/log/zabbix-agent/zabbix_agentd.log.1 (deleted)
zabbix_ag 4908 zabbix    1w   REG  252,0        0     0 28836028 /var/log/zabbix-agent/zabbix_agentd.log.1 (deleted)
zabbix_ag 4908 zabbix    2w   REG  252,0        0     0 28836028 /var/log/zabbix-agent/zabbix_agentd.log.1 (deleted)
zabbix_ag 4909 zabbix    1w   REG  252,0        0     0 28836028 /var/log/zabbix-agent/zabbix_agentd.log.1 (deleted)
zabbix_ag 4909 zabbix    2w   REG  252,0        0     0 28836028 /var/log/zabbix-agent/zabbix_agentd.log.1 (deleted)
zabbix_ag 4910 zabbix    1w   REG  252,0        0     0 28836028 /var/log/zabbix-agent/zabbix_agentd.log.1 (deleted)
zabbix_ag 4910 zabbix    2w   REG  252,0        0     0 28836028 /var/log/zabbix-agent/zabbix_agentd.log.1 (deleted)

I rebooted to see if fsck does anything. But, from /var/log/boot.log, it seems there are no issues:

/dev/mapper/server-root: clean, 3936097/32768000 files, 125368568/131064832 blocks

Thinking maybe someone overzealously reserved root space, I checked the master record:

# tune2fs -l /dev/mapper/server-root 
tune2fs 1.42 (29-Nov-2011)
Filesystem volume name:   <none>
Last mounted on:          /
Filesystem UUID:          86430ade-cea7-46ce-979c-41769a41ecbe
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags:         signed_directory_hash 
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              32768000
Block count:              131064832
Reserved block count:     6553241
Free blocks:              5696264
Free inodes:              28831903
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      992
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
Flex block group size:    16
Filesystem created:       Fri Feb  1 13:44:04 2013
Last mount time:          Tue Aug 19 16:56:13 2014
Last write time:          Fri Feb  1 13:51:28 2013
Mount count:              9
Maximum mount count:      -1
Last checked:             Fri Feb  1 13:44:04 2013
Check interval:           0 (<none>)
Lifetime writes:          1215 GB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:           256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
First orphan inode:       28836028
Default directory hash:   half_md4
Directory Hash Seed:      bca55ff5-f530-48d1-8347-25c004f66d43
Journal backup:           inode blocks

The system is:

# uname -a
Linux server 3.2.0-67-generic #101-Ubuntu SMP Tue Jul 15 17:46:11 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.04
DISTRIB_CODENAME=precise
DISTRIB_DESCRIPTION="Ubuntu 12.04.2 LTS"

Does anyone have any tips on what exactly to do to find and hopefully reclaim the missing space?

© Server Fault or respective owner

Related posts about ubuntu

Related posts about disk