Disk (EXT4) suddenly empty without any sign of why
- by Ohnomydisk
I have a Ubuntu 10.04 server with several disks in it. The disks are setup with a union filesystem, which presents them all as one logical /home.
A few days ago, one of the disks appears to have suddenly 'become empty', for lack of better explanation. The amount of data on the /home mount almost halved within minutes - the disk appears to have had just over 400 GB of data prior to 'becoming empty'.
I have absolutely no idea what happened. I was not using the server at the other time, but there are half a dozen other users who may have been (without root access and without the ability to hose a whole disk).
I've ran SMART tests on the disk and it comes back clean. The filesystem checks fine (it has 12 GB used now, as some user software continued downloading after the incident).
All I know is that around around midnight on October 19, the disk usage changed dramatically:
The data points are every 15 minutes, and the full loss occured between captures:
2012-10-18 23:58:03.399647 - has 953.97/2059.07 GB [46.33 percent]
2012-10-19 00:13:15.909010 - has 515.18/2059.07 GB [25.02 percent]
Other than that, I have not much to go off :-(
I know that:
There's nothing interesting in log files at that time
Nobody appeared to be logged in via SSH at the time it occured (most users do not even use SSH)
The server was online through whatever occured (3 months uptime)
None of the other disks were affected and everything else on the server looks completely normal
I have tried using "extundelete" on the disk and it didn't really find anything (some temporary files, but they looked new anyway)
I am completely at a loss to what could have caused this. I was initially thinking maybe root escalation exploit, but even if someone did maliciously "rm" the disk contents, it would take more than 15 minutes for 400 GB?
