Hey gurus, need some help badly with this one.
I run a server with a 6Tb md raid5 volume built over 7*1Tb disks.
I've had to shut down the server lately and when it went back up, 2 out of the 7 disks used for the raid volume had lost its conf :
dmesg :
[ 10.184167] sda: sda1 sda2 sda3 // System disk
[ 10.202072] sdb: sdb1
[ 10.210073] sdc: sdc1
[ 10.222073] sdd: sdd1
[ 10.229330] sde: sde1
[ 10.239449] sdf: sdf1
[ 11.099896] sdg: unknown partition table
[ 11.255641] sdh: unknown partition table
All 7 disks have same geometry and were configured alike :
dmesg :
Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x1e7481a5
Device Boot Start End Blocks Id System
/dev/sdb1 1 121601 976760001 fd Linux raid autodetect
All 7 disks (sdb1, sdc1, sdd1, sde1, sdf1, sdg1, sdh1) were used in a md raid5 xfs volume.
When booting, md, which was (obviously) out of sync kicked in and automatically started rebuilding over the 7 disks, including the two "faulty" ones; xfs tried to do some shenanigans as well:
dmesg :
[ 19.566941] md: md0 stopped.
[ 19.817038] md: bind<sdc1>
[ 19.817339] md: bind<sdd1>
[ 19.817465] md: bind<sde1>
[ 19.817739] md: bind<sdf1>
[ 19.817917] md: bind<sdh>
[ 19.818079] md: bind<sdg>
[ 19.818198] md: bind<sdb1>
[ 19.818248] md: md0: raid array is not clean -- starting background reconstruction
[ 19.825259] raid5: device sdb1 operational as raid disk 0
[ 19.825261] raid5: device sdg operational as raid disk 6
[ 19.825262] raid5: device sdh operational as raid disk 5
[ 19.825264] raid5: device sdf1 operational as raid disk 4
[ 19.825265] raid5: device sde1 operational as raid disk 3
[ 19.825267] raid5: device sdd1 operational as raid disk 2
[ 19.825268] raid5: device sdc1 operational as raid disk 1
[ 19.825665] raid5: allocated 7334kB for md0
[ 19.825667] raid5: raid level 5 set md0 active with 7 out of 7 devices, algorithm 2
[ 19.825669] RAID5 conf printout:
[ 19.825670] --- rd:7 wd:7
[ 19.825671] disk 0, o:1, dev:sdb1
[ 19.825672] disk 1, o:1, dev:sdc1
[ 19.825673] disk 2, o:1, dev:sdd1
[ 19.825675] disk 3, o:1, dev:sde1
[ 19.825676] disk 4, o:1, dev:sdf1
[ 19.825677] disk 5, o:1, dev:sdh
[ 19.825679] disk 6, o:1, dev:sdg
[ 19.899787] PM: Starting manual resume from disk
[ 28.663228] Filesystem "md0": Disabling barriers, not supported by the underlying device
[ 28.663228] XFS mounting filesystem md0
[ 28.884433] md: resync of RAID array md0
[ 28.884433] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
[ 28.884433] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for resync.
[ 28.884433] md: using 128k window, over a total of 976759936 blocks.
[ 29.025980] Starting XFS recovery on filesystem: md0 (logdev: internal)
[ 32.680486] XFS: xlog_recover_process_data: bad clientid
[ 32.680495] XFS: log mount/recovery failed: error 5
[ 32.682773] XFS: log mount failed
I ran fdisk and flagged sdg1 and sdh1 as fd.
I tried to reassemble the array but it didnt work: no matter what was in mdadm.conf, it still uses sdg and sdh instead of sdg1 and sdh1.
I checked in /dev and I see no sdg1 and and sdh1, shich explains why it wont use it.
I just don't know why those partitions are gone from /dev and how to readd those...
blkid :
/dev/sda1: LABEL="boot" UUID="519790ae-32fe-4c15-a7f6-f1bea8139409" TYPE="ext2"
/dev/sda2: TYPE="swap"
/dev/sda3: LABEL="root" UUID="91390d23-ed31-4af0-917e-e599457f6155" TYPE="ext3"
/dev/sdb1: UUID="2802e68a-dd11-c519-e8af-0d8f4ed72889" TYPE="mdraid"
/dev/sdc1: UUID="2802e68a-dd11-c519-e8af-0d8f4ed72889" TYPE="mdraid"
/dev/sdd1: UUID="2802e68a-dd11-c519-e8af-0d8f4ed72889" TYPE="mdraid"
/dev/sde1: UUID="2802e68a-dd11-c519-e8af-0d8f4ed72889" TYPE="mdraid"
/dev/sdf1: UUID="2802e68a-dd11-c519-e8af-0d8f4ed72889" TYPE="mdraid"
/dev/sdg: UUID="2802e68a-dd11-c519-e8af-0d8f4ed72889" TYPE="mdraid"
/dev/sdh: UUID="2802e68a-dd11-c519-e8af-0d8f4ed72889" TYPE="mdraid"
fdisk -l :
Disk /dev/sda: 40.0 GB, 40020664320 bytes
255 heads, 63 sectors/track, 4865 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x8c878c87
Device Boot Start End Blocks Id System
/dev/sda1 * 1 12 96358+ 83 Linux
/dev/sda2 13 134 979965 82 Linux swap / Solaris
/dev/sda3 135 4865 38001757+ 83 Linux
Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x1e7481a5
Device Boot Start End Blocks Id System
/dev/sdb1 1 121601 976760001 fd Linux raid autodetect
Disk /dev/sdc: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0xc9bdc1e9
Device Boot Start End Blocks Id System
/dev/sdc1 1 121601 976760001 fd Linux raid autodetect
Disk /dev/sdd: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0xcc356c30
Device Boot Start End Blocks Id System
/dev/sdd1 1 121601 976760001 fd Linux raid autodetect
Disk /dev/sde: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0xe87f7a3d
Device Boot Start End Blocks Id System
/dev/sde1 1 121601 976760001 fd Linux raid autodetect
Disk /dev/sdf: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0xb17a2d22
Device Boot Start End Blocks Id System
/dev/sdf1 1 121601 976760001 fd Linux raid autodetect
Disk /dev/sdg: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x8f3bce61
Device Boot Start End Blocks Id System
/dev/sdg1 1 121601 976760001 fd Linux raid autodetect
Disk /dev/sdh: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0xa98062ce
Device Boot Start End Blocks Id System
/dev/sdh1 1 121601 976760001 fd Linux raid autodetect
I really dont know what happened nor how to recover from this mess.
Needless to say the 5TB or so worth of data sitting on those disks are very valuable to me...
Any idea any one?
Did anybody ever experienced a similar situation or know how to recover from it ?
Can someone help me? I'm really desperate... :x
