input / output error, drives randomly refusing to read / write
- by ILMV
I have an issue with one of our servers running Ubuntu 10.04, it is running BackupPC and collects backups from various machines / servers around the building.
On the 8th minute (12:08, 12:18, 12:28 etc) the backups are transferred to an external hard drive, we have three and rotate one drive for another everyday.
The problem we are having is we are randomly experiencing input / output errors, when this happens you cannot read / write to the drive, it hasn't unmounted so I can cd to the mount point /media/backup1.
The drives are not faulty as it's happening on all of them, so I'm at a loss as to what the problem could be, here is an example of the many errors we get:
gzip: stdout: Input/output error
/var/lib/backuppc/backuppc_offline: line 47: /media/backup1/Tue/offline.log: Input/output error
ls: cannot access /media/backup1/Tue/incr_1083_host1.something.co.uk.tar.gz: Input/output error
ls: cannot access /media/backup1/Tue/incr_1088_host1.something.co.uk.tar.gz: Input/output error
ls: cannot access /media/backup1/Tue/incr_1089_host1.something.co.uk.tar.gz: Input/output error
ls: cannot access /media/backup1/Tue/incr_1090_host1.something.co.uk.tar.gz: Input/output error
/var/lib/backuppc/backuppc_offline: line 39: /media/backup1/Tue/offline.log: Input/output error
/var/lib/backuppc/backuppc_offline: line 44: /media/backup1/Tue/offline.log: Input/output error
/var/lib/backuppc/backuppc_offline: line 45: /media/backup1/Tue/incr_1090_host1.something.co.uk.tar.gz: Input/output error
/var/lib/backuppc/backuppc_offline: line 47: /media/backup1/Tue/offline.log: Input/output error
ls: cannot access /media/backup1/Tue/incr_591_tech2.something.co.uk.tar.gz: Input/output error
/var/lib/backuppc/backuppc_offline: line 44: /media/backup1/Tue/offline.log: Input/output error
/var/lib/backuppc/backuppc_offline: line 45: /media/backup1/Tue/incr_591_tech2.something.co.uk.tar.gz: Input/output error
/var/lib/backuppc/backuppc_offline: line 47: /media/backup1/Tue/offline.log: Input/output error
ls: cannot access /media/backup1/Tue/incr_592_tech3.something.co.uk.tar.gz: Input/output error
ls: cannot access /media/backup1/Tue/incr_593_tech3.something.co.uk.tar.gz: Input/output error
/var/lib/backuppc/backuppc_offline: line 44: /media/backup1/Tue/offline.log: Input/output error
/var/lib/backuppc/backuppc_offline: line 45: /media/backup1/Tue/incr_593_tech3.something.co.uk.tar.gz: Input/output error
/var/lib/backuppc/backuppc_offline: line 47: /media/backup1/Tue/offline.log: Input/output error
EDIT » Resolved
So it turns out Quamis was right, even though I didn't think it was possible it was actually a problem with the drive. You see we have three drives all formatted to ext2, on two of them we were getting I/O errors frequently, I cam back to Quamis' answer and discovered the fsck command, so ran it against the problems drives:
fsck /dev/sdb1
This found and fixed a load of problems on the drive, most probably caused by power outages / unsafe removal of drives etc, as the drives are in the xt2 format they aren't journalled and thus aren't protected against such issues.
Drives are now working beautifully, thanks all! :D
