One of my four Linux software raid arrays drops one of its two devices when I reboot my system. The other three arrays work fine. I am running RAID1 on kernel version 2.6.32-5-amd64. Every time I reboot, /dev/md2 comes up with only one device. I can manually add the device by saying $ sudo mdadm /dev/md2 --add /dev/sdc1. This works fine, and mdadm confirms that the device has been re-added as follows:

mdadm: re-added /dev/sdc1

After adding the device and and allowing the array time to resynch, this is what the output of $ cat /proc/mdstat looks like:

Personalities : [raid1] 
md3 : active raid1 sda4[0] sdb4[1]
      244186840 blocks super 1.2 [2/2] [UU]

md2 : active raid1 sdc1[0] sdd1[1]
      732574464 blocks [2/2] [UU]

md1 : active raid1 sda3[0] sdb3[1]
      722804416 blocks [2/2] [UU]

md0 : active raid1 sda1[0] sdb1[1]
      6835520 blocks [2/2] [UU]

unused devices: <none>

Then after I reboot, this is what the output of $ cat /proc/mdstat looks like:

Personalities : [raid1] 
md3 : active raid1 sda4[0] sdb4[1]
      244186840 blocks super 1.2 [2/2] [UU]

md2 : active raid1 sdd1[1]
      732574464 blocks [2/1] [_U]

md1 : active raid1 sda3[0] sdb3[1]
      722804416 blocks [2/2] [UU]

md0 : active raid1 sda1[0] sdb1[1]
      6835520 blocks [2/2] [UU]

unused devices: <none>

During reboot, here is the output of $ sudo cat /var/log/syslog | grep mdadm :

Jun 22 19:00:08 rook mdadm[1709]: RebuildFinished event detected on md device /dev/md2
Jun 22 19:00:08 rook mdadm[1709]: SpareActive event detected on md device /dev/md2, component device /dev/sdc1
Jun 22 19:00:20 rook kernel: [ 7819.446412] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:00:20 rook kernel: [ 7819.446415] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:00:20 rook kernel: [ 7819.446782] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:00:20 rook kernel: [ 7819.446785] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:00:20 rook kernel: [ 7819.515844] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:00:20 rook kernel: [ 7819.515847] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:00:20 rook kernel: [ 7819.606829] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:00:20 rook kernel: [ 7819.606832] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:03:48 rook kernel: [ 8027.855616] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:03:48 rook kernel: [ 8027.855620] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:03:48 rook kernel: [ 8027.855950] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:03:48 rook kernel: [ 8027.855952] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:03:49 rook kernel: [ 8027.962169] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:03:49 rook kernel: [ 8027.962171] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:03:49 rook kernel: [ 8028.054365] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:03:49 rook kernel: [ 8028.054368] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:10:23 rook kernel: [    9.588662] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:10:23 rook kernel: [    9.588664] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:10:23 rook kernel: [    9.601990] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:10:23 rook kernel: [    9.601991] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:10:23 rook kernel: [    9.602693] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:10:23 rook kernel: [    9.602695] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:10:23 rook kernel: [    9.605981] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:10:23 rook kernel: [    9.605983] mdadm: sending ioctl 1261 to a partition!
Jun 22 19:10:23 rook kernel: [    9.606138] mdadm: sending ioctl 800c0910 to a partition!
Jun 22 19:10:23 rook kernel: [    9.606139] mdadm: sending ioctl 800c0910 to a partition!
Jun 22 19:10:48 rook mdadm[1737]: DegradedArray event detected on md device /dev/md2

Here is the mdadm.conf file:

ARRAY /dev/md0 metadata=0.90 UUID=92121d42:37f46b82:926983e9:7d8aad9b
ARRAY /dev/md1 metadata=0.90 UUID=9c1bafc3:1762d51d:c1ae3c29:66348110
ARRAY /dev/md2 metadata=0.90 UUID=98cea6ca:25b5f305:49e8ec88:e84bc7f0
ARRAY /dev/md3 metadata=1.2 name=rook:3 UUID=ca3fce37:95d49a09:badd0ddc:b63a4792

I also ran $ sudo smartctl -t long /dev/sdc and no hardware issues were detected. As long as I do not reboot, /dev/md2 seems to work fine. Does anyone have any suggestions?

Here is the output of $ sudo mdadm -E /dev/sdc1 after re-adding the device and letting it resync:

/dev/sdc1:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 98cea6ca:25b5f305:49e8ec88:e84bc7f0 (local to host rook)
  Creation Time : Sun Jul 13 08:05:55 2008
     Raid Level : raid1
  Used Dev Size : 732574464 (698.64 GiB 750.16 GB)
     Array Size : 732574464 (698.64 GiB 750.16 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 2

    Update Time : Mon Jun 24 07:42:49 2013
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 5fd6cc13 - correct
         Events : 180998


      Number   Major   Minor   RaidDevice State
this     0       8       33        0      active sync   /dev/sdc1

   0     0       8       33        0      active sync   /dev/sdc1
   1     1       8       49        1      active sync   /dev/sdd1

Here is the output of $ sudo mdadm -D /dev/md2 after re-adding the device and letting it resync:

/dev/md2:
        Version : 0.90
  Creation Time : Sun Jul 13 08:05:55 2008
     Raid Level : raid1
     Array Size : 732574464 (698.64 GiB 750.16 GB)
  Used Dev Size : 732574464 (698.64 GiB 750.16 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 2
    Persistence : Superblock is persistent

    Update Time : Mon Jun 24 07:42:49 2013
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           UUID : 98cea6ca:25b5f305:49e8ec88:e84bc7f0 (local to host rook)
         Events : 0.180998

    Number   Major   Minor   RaidDevice State
       0       8       33        0      active sync   /dev/sdc1
       1       8       49        1      active sync   /dev/sdd1