vSphere handling of LUNs detected as snapshot LUNs (1011387)

Symptoms

• A datastore is missing, though you can still see the LUN presented to the host. 
  
• When forcing a LUN that is formatted as VMFS-5 to mount (either with the name or UUID), you may see errors similar to:
  
  # esxcli storage vmfs snapshot mount -l DEV-LUN03
No unresolved VMFS snapshots with volume label 'DEV-LUN03' found.

# esxcli storage vmfs snapshot mount -u 4f1d6367-34a3d2d2-f46b-14feb5cc149a
No unresolved VMFS snapshots with volume UUID '4f1d6367-34a3d2d2-f46b-14feb5cc149a' found.
  
• In the vmkernel or messages log , you see messages similar to:
  
  LVM: 8445: Device naa.0012345012345678:1 detected to be a snapshot:

LVM: 8445: Device eui.0017380012020364:1 detected to be a snapshot:
LVM: 8452: queried disk ID: <type 1, len 17, lun 36, devType 0, scsi 0, h(id) 7683208289187576905>
LVM: 8459: on-disk disk ID: <type 1, len 17, lun 17, devType 0, scsi 0, h(id) 7683208289187576905>
• When force mounting a VMFS datastore, you may experience these symptoms:
  
  
  • Other hosts in the same datacenter cannot mount that VMFS datastore from vCenter Server.
    
  • The Resolve VMFS Volumes automatic task, seen in the task list of the vCenter Client, generates the error:
    
    Error: Cannot change the host configuration.
Error Stack
Call "HostStorageSystem.ResolveMultipleUnresolvedVmfsVolumes" for object "storageVolume" on vCenter Server "MyVC" failed.
    Note: Where storageVolume is the name of the datastore and MyVC is the name of your vCenter Server.

Purpose

This article provides information on how snapshot LUN detection is handled by ESXi/ESX 4.x and ESXi 5.x.

Cause

As of ESXi/ESX 4.0, it is no longer necessary to handle snapshot LUNs via the CLI. Resignature and Force-Mount operations have full GUI support and vCenter Server does VMFS rescans on all hosts after a resignature operation.

The snapshot LUNs issue is caused when the ESXi/ESX host cannot confirm the identity of the LUN with what it expects to see in the VMFS metadata. This can be caused by replaced SAN hardware, firmware upgrades, SAN replication, DR tests, and some HBA firmware upgrades. Some ESXi/ESX host upgrades from 3.5 to 4.x (due to the change in naming convention from mpx to naa) have also been known to cause this. However, this is a rare occurrence. For more/related information, see Managing Duplicate VMFS Datastores in the ESX Configuration Guide for ESXi/ESX 4.x and in the vSphere Storage Guide for ESXi 5.x.

Force mounting a VMFS datastore may fail if:

1. Multiple ESXi/ESX 4.x and 5.0 hosts are managed by the same vCenter Server and these hosts are in the same datacenter.
2. A snapshot LUN containing a VMFS datastore is presented to all these ESXi/ESX hosts.
3. One of these ESXi/ESX hosts has force mounted the VMFS datastore that resides on this snapshot LUN.
4. A second ESXi/ESX host is attempting to do an operation at the same time.

When one ESXi/ESX host force mounts a VMFS datastore residing on a LUN which has been detected as a snapshot, an object is added to the datacenter grouping in the vCenter Server database to represent that datastore.

When a second ESXi/ESX host attempts to do the same operation on the same VMFS datastore, the operation fails because an object already exists within the same datacenter grouping in the vCenter Server database.

Since an object already exists, vCenter Server does not allow mounting the datastore on any other ESXi/ESX host residing in that same datacenter.

Resolution

For more information on how snapshot LUNs were handled in earlier releases, see:

• VMFS Volume Can Be Erroneously Recognized as a Snapshot (6482648) 
• Resignaturing VMFS3 volumes on VMware ESX 3.x via the VMware Infrastructure Client (9453805)

As of ESXi/ESX 4.0, it is no longer necessary to handle snapshot LUNs via the CLI. Resignature and Force-Mount operations have full GUI support and vCenter Server does VMFS rescans on all hosts after a resignature operation.

There are three methods used to mount the VMFS volume on an ESXi/ESX host:

• Using the VMware vSphere Client
• Using the VMware vSphere Web Client
• Using the command line/Tech Support Mode console. For more information, see:
  • Unable to connect to an ESX host using Secure Shell (SSH) (1003807)
  • Tech Support Mode for Emergency Support (1003677)
  • Using Tech Support Mode in ESXi 4.1 and ESXi 5.0. (1017910)
    

ESXi 5.x

vSphere Client

1. Log into the vSphere Client and select the server from the inventory panel.
2. In the Configuration tab's Hardware panel, click Storage. 
3. Click Add Storage.
4. Select the Disk/LUN storage type.
5. Click Next.
6. From the list of LUNs, select the LUN that has a datastore name displayed in the VMFS Label column.
   
   Note: The name present in the VMFS Label column indicates that the LUN is a copy that contains a copy of an existing VMFS datastore.
   
7. Click Next.
8. Under Mount Options, these options are displayed:
   
   
   • Keep Existing Signature: Persistently mount the LUN (for example, mount LUN across reboots)
   • Assign a New Signature: Resignature the LUN
   • Format the disk: Reformat the LUN
     
     Notes:
     
     • The Format the disk option deletes any existing data on the LUN.
     • Before attempting to resignature, make sure that there are no virtual machines running off that VMFS volume on any other host, as those virtual machines may become invalid in the vCenter Server inventory and will have to be registered again on their respective hosts.
       
9. Select the desired option for your volume.
10. In the Ready to Complete page, review the datastore configuration information.
11. Click Finish.

vSphere Web Client

1. Log into the vSphere Web Client and navigate to vCenter Home.
2. Click Datastores in the menu on the left.
3. In the Objects tab click the Create a new datastore icon in the top left.
4. Type the datastore name and if required, select the placement location for the datastore.
5. Select VMFS as the datastore type.
6. From the list of storage devices, select the device that has a specific value displayed in the Snapshot Volume column.
   The value present in the Snapshot Volume column indicates that the device is a copy that contains a copy of an existing VMFS datastore.
7. Under Mount Options, select the desired option (Keep existing, Assign new, Format) for your volume. and click Next.
8. Review the datastore configuration information.
9. Click Finish.

Command line

The esxcli command is used on the command line.

• To list the volumes detected as snapshots, run this command:
  
  # esxcli storage vmfs snapshot list
  
  The output appears similar to:
  
  49d22e2e-996a0dea-b555-001f2960aed8
   Volume Name: VMFS_1
   VMFS UUID: 4e26f26a-9fe2664c-c9c7-000c2988e4dd
   Can mount: true
   Reason for un-mountability:
   Can resignature: true
   Reason for non-resignaturability:
   Unresolved Extent Count: 1
  
• To mount a snapshot/replica LUN that is persistent across reboots, run this command:
  
  # esxcli storage vmfs snapshot mount -l label|-u uuid
  
  For example:
  
  # esxcli storage vmfs snapshot mount -l "VMFS_1"
# esxcli storage vmfs snapshot mount -u "49d22e2e-996a0dea-b555-001f2960aed8"
  
• To mount a snapshot/replica LUN that is not persistent across reboots, run this command:
  
  # esxcli storage vmfs snapshot mount -n -l label|-u uuid
  
  For example:
  
  # esxcli storage vmfs snapshot mount -n -l "VMFS_1"
# esxcli storage vmfs snapshot mount -n -u "49d22e2e-996a0dea-b555-001f2960aed8"
  
• To resignature a snapshot/replica LUN, run this command:
  
  # esxcli storage vmfs snapshot resignature -l label|-u uuid
  
  For example:
  
  # esxcli storage vmfs snapshot resignature -l "VMFS_1"
# esxcli storage vmfs snapshot resignature -u "49d22e2e-996a0dea-b555-001f2960aed8"

Notes:
To view the datastores again in vCenter Server, you may have to perform a rescan of the storage adapters on all ESXi/ESX hosts that the datastore is presented to or a refresh of the storage view.
If you are having trouble identifying the affected datastore, in the vSphere client, check the storage view of another ESX/ESXi host that still has the datastore mounted correctly. This will then allow you to correlate VMFS datastore name with NAA LUN identifier.

ESXi/ESX 4.x

vSphere Client

1. Log into the vSphere Client and select the server from the inventory panel.
2. Click the Configuration tab.
3. Click Storage in the Hardware panel.
4. Click Add Storage.
5. Select the Disk/LUN storage type.
6. Click Next.
7. From the list of LUNs, select the LUN that has a datastore name displayed in the VMFS Label column and click Next.
   
   Note: The name present in the VMFS Label column indicates that the LUN is a copy that contains a copy of an existing VMFS datastore.
   
8. Under Mount Options, these options are displayed:
   
   
   • Keep Existing Signature: The VMFS Metadata will not change. The volume will be detected as a snapshot after the reboot. All other hosts accessing this device/LUN will continue to see this as a snapshot volume. (To mount it persistently, see theCommand Line section.)
   • Assign a New Signature: Resignature the LUN
   • Format the disk: Reformat the LUN
     
     Notes:
     
     • The Format the disk option deletes any existing data on the LUN.
     • Before attempting to resignature, make sure that there are no virtual machines running off that VMFS volume on any other host, as those virtual machines may become invalid in the vCenter Server inventory and will have to be registered again on their respective hosts.
       
9. Select the desired option for your volume.
10. On the Ready to Complete page, review the datastore configuration information.
11. Click Finish.

Command line

The esxcfg-volume command is used on the command line.

• To list the volumes that are detected as snapshots/replicas, run this command:
  
  # esxcfg-volume -l
  
  The output appears similar to:
  
  VMFS3 UUID/label: 49d22e2e-996a0dea-b555-001f2960aed8/VMFS_1
Can mount: Yes
Can resignature: Yes
Extent name: naa.60a98000503349394f3450667a744245:1 range: 0 - 97023 (MB)
  
  In this example, the datastore UUID is 49d22e2e-996a0dea-b555-001f2960aed8 and its last label is VMFS_1.
  
• To mount the volume without performing a resignaturing of that volume (this volume is unmounted when the ESX host is rebooted), run this command:
  
  # esxcfg-volume -m VMFS_UUID|label
  
  For example:
  
  # esxcfg-volume -m "VMFS_1"
# esxcfg-volume -m "49d22e2e-996a0dea-b555-001f2960aed8"
  
• To mount the volume without performing a resignaturing of that volume (this volume is mounted when the ESX host is rebooted), run this command:
  
  # esxcfg-volume -M VMFS_UUID|label
  
  For example:
  
  # esxcfg-volume -M "VMFS_1"
# esxcfg-volume -M "49d22e2e-996a0dea-b555-001f2960aed8"
  
• To resignature the volume (the volume is mounted immediately after the resignature), run this command:
  
  # esxcfg-volume -r VMFS_UUID|label
  
  For example:
  
  # esxcfg-volume -r "VMFS_1"
# esxcfg-volume -r "49d22e2e-996a0dea-b555-001f2960aed8"

Notes:
To view the datastores again in vCenter Server, you may have to perform a rescan of the storage adapters on all ESXi/ESX hosts that the datastore is presented to or a refresh of the storage view.
If you are having trouble identifying the affected datastore, in the vSphere client, check the storage view of another ESX/ESXi host that still has the datastore mounted correctly. This will then allow you to correlate VMFS datastore name with NAA LUN identifier.

ESXi/ESX 3.5

1. In the VMware Infrastructure Client, select the host in the inventory.
2. Click the Configuration tab and click Advanced Settings.
3. Select LVM in the left pane and set LVM.DisallowSnapshotLUN to 0 in the right pane.
   
   Warning: When LVM.DisallowSnapshotLUN is set to 0, do not present any snapshot LUNs to the ESXi/ESX host because data corruption may result. For more information, see the SAN Configuration Guide for your version for ESXi/ESX.
   
4. Rescan all VMFS volumes.
   
   After the rescan, all VMFS volumes are available.

Additional Information

There are three important notes to keep in mind when force mounting VMFS volumes on ESXi/ESX 4.x and 5.0 hosts:

• A VMFS datastore can be mounted only if it does not collide with an already mounted VMFS datastore that has the same UUID (that is, VMFS signature). In other words, if the original LUN that contains the original VMFS datastore together with its snapshot LUN are both presented to the ESXi/ESX host simultaneously, force mounting the VMFS datastore from the snapshot LUN on this same ESXi/ESX host is not allowed. In this case, the only way to mount that VMFS datastore from the snapshot LUN is to resignature and choose a new label for that datastore.
• In a datacenter that has multiple ESXi/ESX hosts, a VMFS datastore on a snapshot LUN can be force mounted on any one of these hosts. When this datastore is mounted on a given host, that datastore cannot be force mounted on any other host using the vCenter Server user interface. To mount that datastore on other hosts, use the steps provided above.
• If a LUN has been force mounted using the process above you will be unable to unmount the lun. To resolve this issue, file a support request.

For additional information about how a VMFS volume can be detected as a snapshot, see Snapshot LUN detection in ESX 3.x and ESX 4.x (1011385).

Source:-

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1011387