here my problem:

I am running ubuntu 12.04 on a raid10 (4 disks), on top of which I installed an lvm with two volume groups (one for /, one for /home).

The layout of the disks are as follows:

Disk /dev/sda: 1500.3 GB, 1500301910016 bytes
255 heads, 63 sectors/track, 182401 cylinders, total 2930277168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0003f3b6


   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *          63      481949      240943+  83  Linux
/dev/sda2          481950  2910640634  1455079342+  fd  Linux raid autodetect
/dev/sda3      2910640635  2930272064     9815715   82  Linux swap / Solaris

Disk /dev/sdb: 1500.3 GB, 1500301910016 bytes
255 heads, 63 sectors/track, 182401 cylinders, total 2930277168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00069785


   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1              63  2910158684  1455079311   fd  Linux raid autodetect
/dev/sdb2      2910158685  2930272064    10056690   82  Linux swap / Solaris

Disk /dev/sdc: 1500.3 GB, 1500301910016 bytes
255 heads, 63 sectors/track, 182401 cylinders, total 2930277168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1              63  2910158684  1455079311   fd  Linux raid autodetect
/dev/sdc2      2910158685  2930272064    10056690   82  Linux swap / Solaris

Disk /dev/sdd: 1500.3 GB, 1500301910016 bytes
255 heads, 63 sectors/track, 182401 cylinders, total 2930277168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000f14de


   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1              63  2910158684  1455079311   fd  Linux raid autodetect
/dev/sdd2      2910158685  2930272064    10056690   82  Linux swap / Solaris

The first disk (/dev/sda) contains the /boot partition on /dev/sda1. I use grub2 to boot the system off this partition.

On top of this raid10 I installed two volume groups, one for /, one for /home.

This system worked well, I even exchanged two disks during the last two years. It always worked. But not this time.

For the first time, /dev/sda broke. I do not know if this is an issue – I know I would have struggled anyways to overcome the problem with /boot installed on that disk and grub2 installed on the mbr of /dev/sda.

Anyways, I did what I always did:

• start knoppix

• fire up the raid

  sudo mdadm --examine -scan
  

  which returns

  ARRAY /dev/md127 UUID=0dbf4558:1a943464:132783e8:19cdff95
  

  start it up

  sudo mdadm --assemble /dev/md127
  

• fail the failing disk (smart event)

  sudo mdadm /dev/md127 --fail /dev/sda2
  

• remove the failing disk

  sudo mdadm /dev/md127 --remove /dev/sda2
  

• stop the raid

  sudo mdadm -S /dev/md127
  

• take out the disk
• replace it with a new one
• create the same partitions as on the failling one

• add it to the raid

  sudo mdadm --assemble /dev/md127
  sudo mdadm /dev/md127 --add /dev/sda2
  

• wait 4 hours

All looks fine:

cat /proc/mdstat

returns:

Personalities : [raid10] 
md127 : active raid10 sda2[0] sdd1[3] sdc1[2] sdb1[1]
      2910158464 blocks 64K chunks 2 near-copies [4/4] [UUUU]

unused devices: <none>

and

sudo mdadm --detail /dev/md127

returns

/dev/md127:
        Version : 0.90
  Creation Time : Wed Jun 10 13:08:46 2009
     Raid Level : raid10
     Array Size : 2910158464 (2775.34 GiB 2980.00 GB)
  Used Dev Size : 1455079232 (1387.67 GiB 1490.00 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 127
    Persistence : Superblock is persistent

    Update Time : Thu Mar 21 16:27:40 2013
          State : clean 
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : near=2
     Chunk Size : 64K

           UUID : 0dbf4558:1a943464:132783e8:19cdff95 (local to host Microknoppix)
         Events : 0.4824680

    Number   Major   Minor   RaidDevice State
       0       8        2        0      active sync   /dev/sda2
       1       8       17        1      active sync   /dev/sdb1
       2       8       33        2      active sync   /dev/sdc1
       3       8       49        3      active sync   /dev/sdd1

However, there is no trace of the volume groups. Rebooting into knoppix does not help Restarting the old system (I actually replugged and re-added the failing disk for that – the system begins to start, but then fails to see the / partition – no wonder if the volume group is gone) does not help.

sudo vgscan, sudo vgdisplay, sudo lvs, sudo lvdisplay, sudo vgscan –mknodes all returned No volume groups found.

I am completely at a loss. Can anyone tell me if and how I can recover my data?

Thanks in advance!