Cluster Node Recovery Using Second Node in Solaris Cluster
- by Onur Bingul
Assumptions:Node 0a is the cluster node that has crashed and could not boot anymore.Node 0b is the node in cluster and in production with services active.Both nodes have their boot disk mirrored via SDS/SVM.We have many options to clone the boot disk from node 0b:- make a copy via network using the ufsdump command and pipe to ufsrestore - make a copy inserting the disk locally on node 0b and creating the third mirror with SDS- make a copy inserting the disk locally on node 0b using dd commandIn this procedure we are going to use dd command (from my experience this is the best option).Bare in mind that in the examples provided we work on Sun Fire V240 systems which have SCSI internal disks. In the case of Fibre Channel (FC) internal disks you must pay attention to the unique identifier, or World Wide Name (WWN), associated with each FC disk (in this case take a look at infodoc #40133 in order to recreate the device tree correctly).Procedure:On node 0b the boot disk is c1t0d0 (c1t1d0 mirror) and this is the VTOC:* Partition Tag Flags Sector Count Sector Mount Directory 0 2 00 0 2106432 2106431 1 3 01 2106432 74630784 76737215 2 5 00 0 143349312 143349311 4 7 00 76737216 50340672 127077887 5 4 00 127077888 14683968 141761855 6 0 00 141761856 1058304 142820159 7 0 00 142820160 529152 143349311We will insert the new disk on node 0b and it will be seen as c1t2d0.1) On node 0b we make a copy via dd from disk c1t0d0s2 to disk c1t2d0s2# dd if=/dev/rdsk/c1t0d0s2 of=/dev/rdsk/c1t2d0s2 bs=8192kA copy of a 72GB disk will take approximately about 45 minutes.Note: as an alternative to make identical copy of root over network follow Document ID: 47498Title: Sun[TM] Cluster 3.0: How to Rebuild a node with Veritas Volume Manager2) Perform an fsck on disk c1t2d0 data slices: 1. fsck -o f /dev/rdsk/c1t2d0s0 (root) 2. fsck -o f /dev/rdsk/c1t2d0s4 (/var) 3. fsck -o f /dev/rdsk/c1t2d0s5 (/usr) 4. fsck -o f /dev/rdsk/c1t2d0s6 (/globaldevices)3) Mount the root file system in order to edit following files for changing the node name:# mount /dev/dsk/c1t2d0s0 /mntChange the hostname from 0b to 0a:# cd /mnt/etc# vi hosts # vi hostname.bge0 # vi hostname.bge2 # vi nodename 4) Change the /mnt/etc/vfstab from the actual:/dev/md/dsk/d201 - - swap - no -/dev/md/dsk/d200 /dev/md/rdsk/d200 / ufs 1 no -/dev/md/dsk/d205 /dev/md/rdsk/d205 /usr ufs 1 no logging/dev/md/dsk/d204 /dev/md/rdsk/d204 /var ufs 1 no logging#/dev/md/dsk/d206 /dev/md/rdsk/d206 /globaldevices ufs 2 yes loggingswap - /tmp tmpfs - yes -/dev/md/dsk/d206 /dev/md/rdsk/d206 /global/.devices/node@2 ufs 2 noglobalto this (unencapsulate disk from SDS/SVM):/dev/dsk/c1t0d0s1 - - swap - no -/dev/dsk/c1t0d0s0 /dev/rdsk/c1t0d0s0 / ufs 1 no -/dev/dsk/c1t0d0s5 /dev/rdsk/c1t0d0s5 /usr ufs 1 no logging/dev/dsk/c1t0d0s4 /dev/rdsk/c1t0d0s4 /var ufs 1 no logging#/dev/md/dsk/d206 /dev/md/rdsk/d206 /globaldevices ufs 2 yes loggingswap - /tmp tmpfs - yes -/dev/dsk/c1t0d0s6 /dev/rdsk/c1t0d0s6 /global/.devices/node@1 ufs 2 no globalIt is important that global device partition (slice 6) in the new vfstab will point to the physical partition of the disk (in our case slice 6).Be careful with the name you use for the new disk. In this case we define it as c1t0d0 because we will insert it as target 0 in node 0a.But this could be different based on the configuration you are working on.5) Remove following entry from /mnt/etc/system (part of unencapsulation procedure):rootdev:/pseudo/md@0:0,200,blk6) Correct the link shared -> ../../global/.devices/node@2/dev/md/shared in order to point to the nodeid of node 0a (in our case nodeid 1):# cd /mnt/dev/mdhow it is now.... node 0b has nodeid 2lrwxrwxrwx 1 root root 42 Mar 10 2005 shared ->../../global/.devices/node@2/dev/md/shared# rm shared# ln -s ../../global/.devices/node@1/dev/md/shared sharedhow is going to be... with nodeid 1 for node 0alrwxrwxrwx 1 root root 42 Mar 10 2005 shared ->../../global/.devices/node@1/dev/md/shared7) Change nodeid (in our case from 2 to 1):# cd /mnt/etc/cluster# vi nodeid8) Change the file /mnt/etc/path_to_inst in order to reflect the correct nodeid for node 0a:# cd /mnt/etc# vi path_to_instChange entries from node@2 to node@1 with the vi command ":%s/node@2/node@1/g"9) Write the bootblock to the disk... just in case:# /usr/sbin/installboot /usr/platform/sun4u/lib/fs/ufs/bootblk /dev/rdsk/c1t2d0s0Now the disk is ready to be inserted in node 0a in order to bootup the node.10) Bootup node 0a with command "boot -sx"... this is becasue we need to make some changes in ccr files in order to recreate did environment.11) Modify cluster ccr:# cd /etc/cluster/ccr# rm did_instances# rm did_instances.bak# vi directory - remove the did_instances line.# /usr/cluster/lib/sc/ccradm -i /etc/cluster/ccr/directory # grep ccr_gennum /etc/cluster/ccr/directory ccr_gennum -1 # /usr/cluster/lib/sc/ccradm -i /etc/cluster/ccr/infrastructure # grep ccr_gennum /etc/cluster/ccr/infrastructure ccr_gennum -112) Bring the node 0a down again to the ok prompt and then issue the command "boot -r"Now the node will join the cluster and from scstat and metaset command you can verify functionality. Next step is to encapsulate the boot disk in SDS/SVM and create the mirrors.In our case node 0b has metadevice name starting from d200. For this reason on node 0a we need to create metadevice starting from d100. This is just an example, you can have different names.The important thing to remember is that metadevice boot disks have different names on each node.13) Remove metadevice pointing to the boot and mirror disks (inherit from node 0b):# metaclear -r -f d200# metaclear -r -f d201# metaclear -r -f d204# metaclear -r -f d205# metaclear -r -f d206verify from metastat that no metadevices are set for boot and mirror disks.14) Encapsulate the boot disk:# metainit -f d110 1 1 c1t0d0s0# metainit d100 -m d110# metaroot d10015) Reboot node 0a.16) Create all the metadevice for slices remaining on boot disk# metainit -f d111 1 1 c1t0d0s1# metainit d101 -m d111# metainit -f d114 1 1 c1t0d0s4# metainit d104 -m d114# metainit -f d115 1 1 c1t0d0s5# metainit d105 -m d115# metainit -f d116 1 1 c1t0d0s6# metainit d106 -m d11617) Edit the vfstab in order to specifiy metadevices created:old:/dev/dsk/c1t0d0s1 - - swap - no -/dev/md/dsk/d100 /dev/md/rdsk/d100 / ufs 1 no -/dev/dsk/c1t0d0s5 /dev/rdsk/c1t0d0s5 /usr ufs 1 no logging/dev/dsk/c1t0d0s4 /dev/rdsk/c1t0d0s4 /var ufs 1 no logging#/dev/md/dsk/d206 /dev/md/rdsk/d206 /globaldevices ufs 2 yes loggingswap - /tmp tmpfs - yes -/dev/dsk/c1t0d0s6 /dev/rdsk/c1t0d0s6 /global/.devices/node@1 ufs 2 no globalnew:/dev/md/dsk/d101 - - swap - no -/dev/md/dsk/d100 /dev/md/rdsk/d100 / ufs 1 no -/dev/md/dsk/d105 /dev/md/rdsk/d105 /usr ufs 1 no logging/dev/md/dsk/d104 /dev/md/rdsk/d104 /var ufs 1 no logging#/dev/md/dsk/106 /dev/md/rdsk/d106 /globaldevices ufs 2 yes loggingswap - /tmp tmpfs - yes -/dev/md/dsk/d106 /dev/md/rdsk/d106 /global/.devices/node@1 ufs 2 noglobal18) Reboot node 0a in order to check new SDS/SVM boot configuration.19) Label the mirror disk c1t1d0 with the VTOC of boot disk c1t0d0:# prtvtoc /dev/dsk/c1t0d0s2 > /var/tmp/VTOC_c1t0d0 # fmthard -s /var/tmp/VTOC_c1t0d0 /dev/rdsk/c1t1d0s220) Put DB replica on slice 7 of disk c1t1d0:# metadb -a -c 3 /dev/dsk/c1t1d0s721) Create metadevice for mirror disk c1t1d0 and attach the new mirror side:# metainit d120 1 1 c1t1d0s0# metattach d100 d120# metainit d121 1 1 c1t1d0s1# metattach d101 d121# metainit d124 1 1 c1t1d0s4# metattach d104 d124# metainit d125 1 1 c1t1d0s5# metattach d105 d125# metainit d126 1 1 c1t1d0s6# metattach d106 d126
