Last night when I was working for a test Disaster Recovery project, we had planned to recover around 100 virtual machines as part of testing the DR environment. We are not using any kind of DR automated solution such as SRM. Hence, we perform manual work for recovery such as mounting the volumes, register VMs, etc.

During the recovery process I started mounting Snap volumes to recovery site. I noticed few unusual scenario which should not be in case of recovery the replicated volumes. Here are the points which made me to rethink that certainly there is some issue with some specific LUNs.

• While Adding replicated datastore using Add datastore wizard, VMFS label in Snapshot Volume tab was showing blank which should not be.
• When I clicked Next, found that Resignation Option was not showing in few LUNs, and it was creating a datastore like a new datastore rather a replicated volumes.
• Datastore size was showing Zero.

After suspecting this issue, I dig down to find out the solution. I followed few initial approaches to troubleshoot the issue from VMware and Storage end, still issue was unresolved.

Later, I jumped to the VMFS partitioning and got to know that either VMFS partition was corrupt or not created.

In this blog, I am going to explain the steps which I followed to recreate the VMFS partition, after which my issue was resolved and resignature option was visible. Now I was able to see replicated data in these volumes too.

Basically, I executed these 4 commands to recreate the VMFS partition. However, I will explain this in below steps:

# partedUtil getptbl /vmfs/devices/disks/naa.6001738c7c8053de000000000004aa70

# partedUtil getUsableSectors /vmfs/devices/disks/naa.6001738c7c8053de000000000004aa70

# partedUtil mklabel /vmfs/devices/disks/naa.6001738c7c8053de000000000004aa70 gpt

# partedUtil setptbl /vmfs/devices/disks/naa.6001738c7c8053de000000000004aa70 gpt "1 2048 4194303966 AA31E02A400F11DB9590000C2911D1B8 0"

Let’s take a quick insight of these commands in detail.

• Go to any ESXi host in the cluster on which impacted LUN is mounted. Connect to ESXi command line using Putty.
• Get the NAA ID of impacted LUN using any of esxcli command or vCenter GUI.
• Run below esxcli command to get the current status of partition table.
  • partedUtil getptbl /vmfs/devices/disks/naa.60017vminsight000000000004aa6b

If your partition table is already exist or not corrupt, then you will get output like below.

If Partition table is corrupt, then partition blocks and information highlighted in above picture will be blank. I found only below output and third line was missing for the impacted LUNs which was clearly indicating the VMFS partition issue.

[root@vminsight-esxi-01:-] partedUtil getptbl /vmfs/devices/disks/naa.6001738c7c8053de000000000004aa70
gpt
261083 255 63 4194304000

Recreate VMFS Partition

Find out the Starting Block and End Block which you will use to create VMFS partition of impacted LUN.

Run below command to find starting block of VMFS partition.

[root@vminsight-esxi-01:-] offset="128 2048"; for dev in `esxcfg-scsidevs -l | grep "Console Device:" | awk {'print $3'}`; do disk=$dev; echo $disk; partedUtil getptbl $disk; { for i in `echo $offset`; do echo "Checking offset found at $i:"; hexdump -n4 -s $((0x100000+(512*$i))) $disk; hexdump -n4 -s $((0x1300000+(512*$i))) $disk; hexdump -C -n 128 -s $((0x130001d + (512*$i))) $disk; done; } | grep -B 1 -A 5 d00d; echo "---------------------"; done
Output:
--------------------- /vmfs/devices/disks/eui.00173800648c00a0
gpt
251053 255 63 4033167360
1 2048 4033167326 AA31E02A400F11DB9590000C2911D1B8 vmfs 0
Checking offset found at 2048:
0200000 d00d c001
0200004 1400000 f15e 2fab
1400004 0140001d  4c 45 58 5f 53 68 61 72  65 64 5f 49 42 4d 5f 58  |vi-datastore-01| 0140002d  49 56 32 5f 30 30 32 00  00 00 00 00 00 00 00 00  |IV2_002.........|
---------------------

*Note the value which is mentioned in red above “2048”. This is beginning block value which you will use to create partition of impacted volume.

Run below command to find starting and end block of VMFS partition.

[root@vminsight-esxi-01:-] partedUtil getUsableSectors /vmfs/devices/disks/naa.6001738c7c8053de000000000004aa70
34
4194303966

*Note the value which is mentioned in red above “4194303966”. This is beginning block value which you will use to create partition of impacted volume.

• There is possibility that you may get error “Unknown partition table on disk” while executing above command. It means that you need to label the table as a GPT partition table using below command.

[root@vminsight-esxi-01:-] partedUtil mklabel /vmfs/devices/disks/naa.6001738c7c8053de000000000004aa70 gpt

• Now again rerun below command and get End block of VMFS Partition.

[root@vminsight-esxi-01:-] partedUtil getUsableSectors /vmfs/devices/disks/naa.6001738c7c8053de000000000004aa70
34
4194303966

Note down the information from above commands which you will use to set and crate the partition.

LUN ID - naa.6001738c7c8053de000000000004aa70
Starting Block – 2048
End Block – 4194303966

• Execute Command using above information to set partition table.

[root@vminsight-esxi-01:-] partedUtil setptbl /vmfs/devices/disks/naa.6001738c7c8053de000000000004aa70 gpt "1 2048 4194303966 AA31E02A400F11DB9590000C2911D1B8 0"

Re-run command to get the partition table status: -

[root@vminsight-esxi-01:-] partedUtil getptbl /vmfs/devices/disks/naa.6001738c7c8053de000000000004aa70
Output:
gpt
261083 255 63 4194304000
1 2048 4194303966 AA31E02A400F11DB9590000C2911D1B8 vmfs 0

Now you have partition and sector information available. Bring a smile on your face and initiate add datastore wizard again. Now you will able to see Snapshot Volume information as below.

Click Next to add datastore. Resignature options are also available now.