Linux software raid fails to include one device for one RAID1 array
- by user1389890
One of my four Linux software raid arrays drops one of its two devices when I reboot my system. The other three arrays work fine. I am running RAID1 on kernel version 2.6.32-5-amd64. Every time I reboot, /dev/md2 comes up with only one device. I can manually add the device by saying $ sudo mdadm /dev/md2 --add /dev/sdc1. This works fine, and mdadm confirms that the device has been re-added as follows:
mdadm: re-added /dev/sdc1
After adding the device and and allowing the array time to resynch, this is what the output of $ cat /proc/mdstat looks like:
Personalities : [raid1]
md3 : active raid1 sda4[0] sdb4[1]
244186840 blocks super 1.2 [2/2] [UU]
md2 : active raid1 sdc1[0] sdd1[1]
732574464 blocks [2/2] [UU]
md1 : active raid1 sda3[0] sdb3[1]
722804416 blocks [2/2] [UU]
md0 : active raid1 sda1[0] sdb1[1]
6835520 blocks [2/2] [UU]
unused devices: <none>
Then after I reboot, this is what the output of $ cat /proc/mdstat looks like:
Personalities : [raid1]
md3 : active raid1 sda4[0] sdb4[1]
244186840 blocks super 1.2 [2/2] [UU]
md2 : active raid1 sdd1[1]
732574464 blocks [2/1] [_U]
md1 : active raid1 sda3[0] sdb3[1]
722804416 blocks [2/2] [UU]
md0 : active raid1 sda1[0] sdb1[1]
6835520 blocks [2/2] [UU]
unused devices: <none>
During reboot, here is the output of $ sudo cat /var/log/syslog | grep mdadm :
Jun 22 19:00:08 rook mdadm[1709]: RebuildFinished event detected on md device /dev/md2
Jun 22 19:00:08 rook mdadm[1709]: SpareActive event detected on md device /dev/md2, component device /dev/sdc1
Jun 22 19:00:20 rook kernel: [ 7819.446412] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:00:20 rook kernel: [ 7819.446415] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:00:20 rook kernel: [ 7819.446782] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:00:20 rook kernel: [ 7819.446785] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:00:20 rook kernel: [ 7819.515844] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:00:20 rook kernel: [ 7819.515847] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:00:20 rook kernel: [ 7819.606829] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:00:20 rook kernel: [ 7819.606832] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:03:48 rook kernel: [ 8027.855616] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:03:48 rook kernel: [ 8027.855620] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:03:48 rook kernel: [ 8027.855950] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:03:48 rook kernel: [ 8027.855952] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:03:49 rook kernel: [ 8027.962169] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:03:49 rook kernel: [ 8027.962171] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:03:49 rook kernel: [ 8028.054365] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:03:49 rook kernel: [ 8028.054368] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:10:23 rook kernel: [ 9.588662] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:10:23 rook kernel: [ 9.588664] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:10:23 rook kernel: [ 9.601990] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:10:23 rook kernel: [ 9.601991] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:10:23 rook kernel: [ 9.602693] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:10:23 rook kernel: [ 9.602695] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:10:23 rook kernel: [ 9.605981] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:10:23 rook kernel: [ 9.605983] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:10:23 rook kernel: [ 9.606138] mdadm: sending ioctl 800c0910 to a partition!
Jun 22 19:10:23 rook kernel: [ 9.606139] mdadm: sending ioctl 800c0910 to a partition!
Jun 22 19:10:48 rook mdadm[1737]: DegradedArray event detected on md device /dev/md2
Here is the mdadm.conf file:
ARRAY /dev/md0 metadata=0.90 UUID=92121d42:37f46b82:926983e9:7d8aad9b
ARRAY /dev/md1 metadata=0.90 UUID=9c1bafc3:1762d51d:c1ae3c29:66348110
ARRAY /dev/md2 metadata=0.90 UUID=98cea6ca:25b5f305:49e8ec88:e84bc7f0
ARRAY /dev/md3 metadata=1.2 name=rook:3 UUID=ca3fce37:95d49a09:badd0ddc:b63a4792
I also ran $ sudo smartctl -t long /dev/sdc and no hardware issues were detected. As long as I do not reboot, /dev/md2 seems to work fine. Does anyone have any suggestions?
Here is the output of $ sudo mdadm -E /dev/sdc1 after re-adding the device and letting it resync:
/dev/sdc1:
Magic : a92b4efc
Version : 0.90.00
UUID : 98cea6ca:25b5f305:49e8ec88:e84bc7f0 (local to host rook)
Creation Time : Sun Jul 13 08:05:55 2008
Raid Level : raid1
Used Dev Size : 732574464 (698.64 GiB 750.16 GB)
Array Size : 732574464 (698.64 GiB 750.16 GB)
Raid Devices : 2
Total Devices : 2
Preferred Minor : 2
Update Time : Mon Jun 24 07:42:49 2013
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Checksum : 5fd6cc13 - correct
Events : 180998
Number Major Minor RaidDevice State
this 0 8 33 0 active sync /dev/sdc1
0 0 8 33 0 active sync /dev/sdc1
1 1 8 49 1 active sync /dev/sdd1
Here is the output of $ sudo mdadm -D /dev/md2 after re-adding the device and letting it resync:
/dev/md2:
Version : 0.90
Creation Time : Sun Jul 13 08:05:55 2008
Raid Level : raid1
Array Size : 732574464 (698.64 GiB 750.16 GB)
Used Dev Size : 732574464 (698.64 GiB 750.16 GB)
Raid Devices : 2
Total Devices : 2
Preferred Minor : 2
Persistence : Superblock is persistent
Update Time : Mon Jun 24 07:42:49 2013
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
UUID : 98cea6ca:25b5f305:49e8ec88:e84bc7f0 (local to host rook)
Events : 0.180998
Number Major Minor RaidDevice State
0 8 33 0 active sync /dev/sdc1
1 8 49 1 active sync /dev/sdd1
