Using VPLEX Metro with VMware HA

Purpose

This article provides information about Partner Supported Products for the VPLEX Metro Distributed Virtual Volume with VMware HA and vMotion. This configuration was verified by and is directly supported by EMC.

Note: This means that this solution is not directly supported by VMware. For issues with this configuration, contact EMC directly. VMware offers best effort support to the partner to fix vSphere-related problems that may be found in the field. It is the partner's responsibility to verify that the configuration functions with future major and minor releases of vSphere, as VMware does not guarantee that compatibility with future releases is maintained.

Resolution

What is EMC VPLEX?

EMC VPLEX is a federation solution that can be stretched across two geographically dispersed datacenters separated by synchronous distances (maximum distance of separation = 100 km, maximum round trip latency = 5 msec). It provides simultaneous access to storage devices at two sites by creating a VPLEX Distributed Virtual Volume that is supported on each side by a VPLEX Cluster.

Each VPLEX Cluster is itself highly available, scaling from two directors per VPLEX Cluster to eight directors per VPLEX Cluster, where each director is supported by independent power supplies, fans, and interconnects, making each VPLEX Cluster resilient to single point failures.

For more information about Distributed Virtual Volume, see the Additional Information section in this article.

Validated Configuration

This graphic illustrates the configuration validated by EMC and VMware using VMware HA with a VPLEX Metro Deployment.

Supported Use Cases for VMWARE HA CLUSTER over distance with VPLEX Metro

This table shows the supported use cases with VPLEX configuration:

Case	Support
Simultaneous access to a shared Distributed Virtual Volume from two separate ESX clusters	Supported
HA between a host in ESX cluster 1/Data Center 1 to a host in ESX cluster 1/Data Center 2 leveraging the shared Distributed Virtual volume	Supported
Stretching a single vSphere cluster between sites leveraging a Distributed Virtual volume with VMware High Availability (HA) and Distributed Resource Scheduler (DRS)	Supported
VPLEX Metro and vSphere cluster in tandem with vSphere 4.1 DRS VM-Host Affinity Rules	Not Supported

For additional details of Distributed Virtual Volumes on VPLEX, see the Best Practices section of this article.

The DRS VM Host Affinity feature was introduced in vSphere 4.1.

Configuration Requirements

These requirements must be satisfied to support this configuration:

• The maximum round trip latency on both the IP network and the Fibre-Channel Network between the two sites must not exceed 5 milliseconds (ms). The IP network supports the VMware ESX/ESXi hosts and the VPLEX Management Console, and the Fibre Channel network is required by inter-cluster links connecting the two VPLEX Clusters within VPLEX Metro.
• The ESX servers in both data centers must have a private network on the same IP subnet and broadcast domain
• Any IP subnet used by the virtual machine that resides on it, must be accessible from ESX servers in both data centers. This requirement is important so that clients accessing VMs running on ESX servers on both sides are able to function smoothly upon any VMware HA triggered VM restart events
• The data storage locations including the boot device used by the virtual machines must be active and accessible from ESX servers in both data centers
• vCenter Server must be able to connect to ESX servers in both data centers
• The VMware datastore for the VMs running in the ESX Cluster are provisioned on Distributed Virtual Volumes
• The maximum number of hosts in the HA cluster must not exceed 32 hosts, with 4 hosts on each site.

Note:

• The ESX servers forming the VMware HA cluster could be distributed on two sites.  HA Cluster can start a virtual machine on surviving ESX host, the ESX host access the Distributed Virtual Volume through storage path at its site.
• VPLEX 5.0 and ESX 4.1 are tested in this configuration without the VPLEX Witness

For any additional requirement for VPLEX Distributed Virtual Volumes, see the EMC VPLEX Best practices document.

Understanding Preferred Site for Distributed Virtual Volumes

For each Distributed Virtual Volume, VPLEX defines a detach rule. When there is a communication failure between the two clusters in the VPLEX Metro, this detach rule identifies which VPLEX cluster in a VPLEX Metro should detach its mirror leg, thereby allowing service to continue. The detach rule effectively defines a preferred site if VPLEX clusters lose communication with each other. The purpose of having a defined preferred site is to ensure that there is no possibility of a "split brain" caused by both VPLEX Clusters continuing to allow I/O during communication failure.

After a complete communication failure between the two VPLEX Clusters, the preferred site continues to provide service to the Distributed Virtual Volume. The other VPLEX cluster will suspend I/O service to the volume and referred to as the non-preferred site. The detach rule is at the Distributed Virtual Volume level and hence at any given site could be the preferred site for some Distributed Virtual Volume and non-preferred site for others. A VPLEX Metro instance can support up to 5000 Distributed Virtual Volumes and each such volume has its own detach rule. It is therefore possible for the same VPLEX Cluster (and therefore the ESX servers connected to it) to be on the preferred site with respect to one Distributed Virtual Volume but to be on the non-preferred site with respect to another Distributed Virtual Volume.

There are two conditions that can cause the VPLEX clusters to lose communication:

1. Total VPLEX Cluster failure at one site (Failure of all directors in a VPLEX Cluster): A complete VPLEX Cluster failure triggers the detach rule behaviors since the surviving VPLEX Cluster does not have the ability to determine distinguish between interlink communication loss and VPLEX Cluster failure. As a result, Distributed Virtual Volumes whose preferred site is the surviving VPLEX Cluster will continue to run without interruption. The Distributed Virtual Volumes, whose preferred site is the failed VPLEX Cluster site, will enter into I/O suspension until manual intervention is performed.
2. Failure of the inter-cluster communication links (VPLEX Cluster partition): The VPLEX Cluster partition case will also trigger the execution of the detach rule. Each Distributed Virtual Volume will allow I/O to continue on its preferred site and suspend I/O on its non-preferred site.

When the VPLEX Cluster failure or VPLEX Cluster partition condition is resolved, the VPLEX Metro Distributed Virtual Volume gets re-established enabling I/O on both VPLEX Metro sites.

Tested Scenarios

Terminology

• VMs running in the preferred site: This refers to VMs running on the preferred site for the Metro Distributed Virtual Volume supporting the datastore for those VMs.
• VMs running in the non-preferred site: This refers to VMs running on the non-preferred site for the Metro Distributed Virtual Volume supporting the datastore for those VMs.

Scenario	VPLEX Behavior	VMware HA Impact
Single VPLEX back-end (BE) path failure	VPLEX will switch to alternate paths to the same BE Array and continue to provide access to the Metro Distributed Virtual Volumes exposed to the ESX Servers.	None.
Single VPLEX front-end (FE) path failure	The ESX server will be expected to use alternate paths to the Metro Distributed Virtual Volumes.	None.
BE Array failure (preferred site for a Metro Distributed Virtual Volume)	VPLEX will continue to provide access to the Metro Distributed Virtual Volume through the non-preferred site BE array. When access to the array is restored, the storage volumes from the preferred site BE array will be resynchronized automatically.	None.
BE array failure (non-preferred site for a Metro Distributed Virtual Volume)	VPLEX will continue to provide access to the Metro Distributed Virtual Volume using the preferred site BE array. When access to the array is restored, the storage volumes from the non-preferred site BE array will be rebuilt automatically.	None.
Single front-end switch failure (preferred site for a Metro Distributed Virtual Volume)	VPLEX will continue to provide access to the Metro Distributed Virtual Volume via alternate paths to the same VPLEX Cluster from the ESX Server.	None.
Single front-end switch failure (non-preferred site for a Metro Distributed Virtual Volume)	VPLEX will continue to provide access to the Metro Distributed Virtual Volume via alternate paths to the same VPLEX Cluster from the ESX Server.	None.
VPLEX director failure	VPLEX will continue to provide access to the Metro Distributed Virtual Volume through front-end paths available through other directors on the same VPLEX cluster.	None.
Complete site failure (where the preferred site for a Metro Distributed Virtual Volume is in the site that has failed)	VPLEX will suspend I/O on the Metro Distributed Virtual Volume on the non-preferred site. Once it is determined by the administrator that the site has failed, and it is not a case of inter-site communication failure, the volumes on the non-preferred site can be unsuspended ("resumed") using the device resume-link-down command. Note that this process is manualintentionally. While the automated resuming of I/O works in the site failure, it does not work in the VPLEX Cluster Partition case. Issuing the unsuspend command automatically on the non-preferred site would cause both sites to become simultaneously read-writeable creating a potential split brain condition.	VMs running in preferred site: VMware HA will attempt to bring up the failed VMs (up to 5 times) on the ESX Servers supported by the non-preferred site for the Metro Distributed Virtual Volumes. These attempts will fail until the volumes are unsuspended on the non-preferred site. If the HA maximum restart limit is reached, the failed VMs need to be manually restarted on the non-preferred site. VMs running in non-preferred site: These VMs will see the I/O as being suspended and the guest OS may hang during this time. If VM monitoring is turned on, the guest OS' would attempt to be reset (maximum resets is based on the VM Monitoring policy), but the attempts will fail until volume is 'unsuspended'. If the maximum reset limit of VM Monitoring is reached, the failed VMs will need to be restarted manually.
Complete site failure (where the non-preferred site for a Metro Distributed Virtual Volume is in the site that has failed)	VPLEX will continue to provide I/O access to the preferred site.	VMs running in preferred site: No impact. VMs running in non-preferred site: Given that the ESX Servers have failed, the VMs running on those ESX Servers also fail. VMware HA will automatically restart the VMs on the ESX Servers supported by the preferred site for the Metro Distributed Virtual Volume and no administrative action is necessary.
Add ESX Server(s) to the cluster	After the ESX Servers are registered and added to the appropriate VPLEX view, VPLEX will provide access to the provisioned Metro Distributed Virtual Volumes to the newly added host.	None.
Remove ESX Server(s) from the cluster	After the ESX Servers are removed from the appropriate VPLEX view and deregistered, the ESX Server can be removed.	When the ESX Servers are placed into Maintenance Mode, the VMs on the ESX host are vacated by vCenter using vMotion if DRS is enabled in the vSphere cluster. If DRS is not enabled, the VMs should be moved manually before putting the host into Maintenance Mode.
Multiple ESX Server failure(s) - power off.	None.	VMware HA will restart the VMs on any of the surviving ESX Servers within the VMware HA Cluster, as long as the VMware HA Admission Control Policy and HA release limits (at most 4 simultaneous Hosts failures) are not exceeded.
Multiple ESX Server failure(s) -Network disconnect	None.	VMware HA will restart the VMs on any of the surviving ESX Servers within the VMware HA Cluster, as long as the isolation response policy is set to "Power Off" or "Shutdown". The HA Admission Control Policy and the HA release limits (maximum of 4 simultaneous host failures) should be also abided by. If the isolation response policy is to "leave powered-on" then the VMs will continue to run on the isolated host and will be accessible for management once the host is un-isolated.
Single ESX Server and a VPLEX director failure at same site	The surviving VPLEX directors on the VPLEX cluster with the failed director will continue to provide access to the Metro Distributed Virtual Volumes.	There is no impact to VMs running on the surviving ESX Servers. VMs running on the failed ESX Server will be restarted by VMware HA on the surviving ESX Servers that are part of the vSphere Cluster using VMware HA.
Single director and back-end path failure at same site	The surviving VPLEX directors on the VPLEX cluster with the failed director will continue to provide access to the virtual volumes. VPLEX will switch to alternate paths (if available) to the same back-end and continue to provide access to the Metro Distributed Virtual Volumes.	None.
ESX Server all paths down (encountered when the ESX Server loses access to its storage volumes i.e. VPLEX Volumes in this case).	None.	Ideally the I/Os in the ESX host should resume automatically once the paths are restored. In case that does not happen, the host may need to be rebooted to resume the I/Os. If the ESX Server is restarted, this will cause VMware HA to restart the failed VMs on other surviving ESX Servers within the VMware HA cluster.
VPLEX inter-site link failure; vSphere cluster management network intact.	VPLEX will transition Distributed Virtual Volumes on the non-preferred site to the I/O suspension state. On the preferred site, the Distributed Virtual Volumes will continue to provide access. Note that in this case, I/O at the non-preferred site should not be manually unsuspended. In this case, given that both VPLEX Clusters survive, the preferred site will continue to allow I/O. Unsuspending I/O on the non-preferred site will result in the same Distributed Virtual Volume to be read-writeable on both legs creating a potential split brain condition. By restoring the inter-site links, the Distributed Virtual Volume will become unsuspended on the non-preferred site.	VMs running in preferred site: No impact. VMs running in non-preferred site: These VMs will see all I/Os as suspended and the guest OS may hang during that time. If VM Monitoring is turned on, the VM would attempt to be reset (maximum resets is based on the VM Monitoring policy) on the same host, but the attempts will fail until the Distributed Virtual Volumes are manually 'unsuspended'. If the maximum reset limit of VM Monitoring is reached, the failed VMs will need to be restarted manually. The datastore will not be marked as "unavailable", as the path is still active but no I/Os will be processed.
Complete Dual Site failure.	Upon power on of a single VPLEX Cluster, VPLEX will intentionally keep all Distributed Virtual Volumes in the suspended state even if it is the preferred site until such time as it is able to reconnect to the other site or unless the administrator manually resumes I/Os on these volumes using the device resume-link-down command. This behavior is to account for the possibility that I/Os have continued on the other site (either automatically, if the other site was preferred or manually, if the other site was non-preferred) and thereby protect against data corruption.	If ESX servers are powered back on after all the Distributed Virtual Volumes are manually resumed, the HA Cluster will recover from “Total Cluster Failure” by powering-on all failed VMs automatically. So the recovery will be automated if the Distributed Virtual Volumes and ESX server startup order is maintained. If the startup order is not maintained, the failed VMs need to be restarted manually.
Director failure at one site (preferred site for a given Distributed Virtual Volume) and BE array failure at the other site (Secondary site for a given Distributed Virtual Volume)	The surviving VPLEX directors within the VPLEX cluster with the failed director will continue to provide access to the Metro Distributed Virtual Volumes. VPLEX will continue to provide access to the Metro Distributed Virtual Volumes using the preferred site BE array.	None.
VPLEX inter-site link intact; vSphere cluster management network failure.	None.	VMs on each site will continue running on their respective hosts. NOTE: Since there is a network  partition, VMware HA will attempt to restart the VMs on the other site (where the VM is not currently running) for the Metro Distributed Virtual Volumes but will fail (after 5 maximum retries - NOTE: This is configurable). This failure is desired and expected, and is governed via VMware file-level locking. Once the network partition is corrected, HA will need to be reconfigured on the cluster to fix the cluster network partition.
VPLEX inter-site link failure; vSphere cluster management network failure.	VPLEX will suspend I/O on the non-preferred site for a given Distributed Virtual Volume. The volumes will continue to have access on the Distributed Virtual Volume on its preferred site. Note that in this case, I/O at the non-preferred site should not be manually unsuspended. In this case, given that both VPLEX Clusters survive, the preferred site will continue to allow I/O. Unsuspending I/O on the non-preferred site will result in the same Metro Distributed Virtual Volume to be read-writeable on both legs creating a potential split brain condition. By restoring the inter-site networks, the Distributed Virtual Volume will become unsuspended on the non-preferred site.	VMs running in preferred site: The powered-on VMs will continue to run. This is a HA split brain situation; the non-preferred site would think that the hosts of preferred site are dead and would try to restart the powered-on VMs of the preferred site. The power on attempts will fail because of VMFS file-level locks. By default HA will attempt five retries to power-on the VMs. VMs running in non-preferred site: These VMs will see their I/O as suspended; the guest OS may hang during this time. If VM Monitoring is turned on, the VMs would attempt to reset (maximum resets is based on the VM Monitoring policy) on the same host, but the attempts will fails until the volume is unsuspended.  In case the maximum reset attempts are reached, manual intervention will be needed to reset the failed VMs. The datastore will not be marked as "unavailable" as the path is still active, but no I/Os will be processed. This is also a HA split-brain situation where preferred site hosts will think that the hosts of non-preferred sites are dead and will try to restart the failed VMs. However the restart attempts will fail as the running VMs of the non-preferred hosts are holding the VMFS file locks. Once the network partition is corrected, HA will need to be reconfigured on the cluster to fix the HA split brain situation.

In these failure modes described above involving VPLEX Cluster failures, after the VPLEX Cluster joins back into the VPLEX Metro instance after recovery. If a Distributed Virtual Volume was running I/O on the peer site (either because this was the preferred site or because the administrator had manually chosen to resume I/Os), the joining VPLEX Cluster will recognize this and immediately provide the latest data back to the hosts accessing the same Distributed Virtual Volume through the joining VPLEX Cluster. Any stale data in the joining VPLEX Cluster is discarded and / or overwritten.

Best practice documents

EMC best practice documents for VPLEX are available at http://powerlink.emc.com/

Disclaimer: The partner products referenced in this article are hardware devices that are developed and supported by stated partners. Use of these products are also governed by the end user license agreements of the partners. You must obtain the application, support, and licensing for using these products from the partners. For more information, see Support Information in this article.

Additional Information

Term	Definition
Metro Distributed Virtual Volume	A VPLEX virtual volume with complete, synchronized, copies of data (mirrors), exposed through 2 geographically separated VPLEX clusters. Distributed Virtual Volumes can be simultaneously accessed by servers at two separate data centers.
Preferred Site	Distributed Virtual Volume, VPLEX defines a detach rule. When there is a communication failure between the two clusters in a VPLEX Metro, this detach rule effectively identifies which VPLEX Cluster in a VPLEX Metro should detach its mirror leg and thereby allowing service to continue. This VPLEX Cluster, which in the presence of failures, allows I/Os to continue is referred to as the preferred site for that particular Distributed Virtual Volume. Note: The other VPLEX Cluster will suspend I/O and is referred to as the non-preferred site for that Distributed Virtual Volume.

Source:-

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1026692

Hardware and firmware requirements for 64-bit guest operating systems

Purpose

This article explains the host machine hardware and firmware requirements for installing virtual machines with 64-bit guest operating systems.

Note: 64-bit guests are not supported for ESX versions 2.5.x and earlier.

For information about installing Guest Operating Systems, see the Guest Operating System Installation Guide.

Resolution

VMware's virtual machine monitor has traditionally used segmentation to provide isolation between the guest operating system and the virtual machine monitor. This is necessary because the guest operating system and virtual machine monitor share the linear address space.

To determine whether or not your CPU is suitable for running 64-bit guest operating systems, VMware distributes a free utility. You can download the tool from our web site: Processor Check for 64-Bit Compatibility .

Note: This utility is provided without support from VMware and does not require ESX/ESXi, VMware Server or VMware Workstation to perform the check.

ESX/ESXi, VMware Workstation, Player, and VMware Server require a 64-bit CPU to run a 64-bit guest operating system.

VMware Fusion requires a 64-bit processor to run a 64-bit guest operating system or to run any Mac OS Guest. Mac OS Guests require hardware virtualization (VT), which VMware supports only on 64-bit processors. The Intel Core Solo and Core Duo processors are 32-bit, but the Core 2 Duo and newer processors are 64-bit.

Additional Information

Note: The information supplied in this section applies to older product versions.

Workstation 5.5 - 6.x, Player, and VMware Server 1.x support virtual machines with 64-bit guest operating systems only on host machines that have one of these supported 64-bit processors:

• AMD Athlon 64, revision D or later
• AMD Opteron, revision E or later
• AMD Turion 64, revision E or later
• AMD Sempron, 64-bit-capable, revision D or later (experimental support)
• Intel EM64T VT-enabled processors (experimental support)

Note: In shopping for a processor that is compatible with Workstation or VMware Server 64-bit guests, you may be unable to determine the revision numbers of a given vendor's offering of AMD Athlon 64, Opteron, Turion 64, or Sempron processors. At this time, the only reliable way to determine whether or not any of these processors is a revision supported by VMware software is by noting the manufacturing technology (CMOS). Any of the AMD Athlon 64, Opteron, Turion 64, or Sempron processors whose manufacturing technology is 90nm SOI (.09 micron SOI) is compatible with Workstation or VMware Server 64-bit guests. Using this information, you should be able to determine — with the help of your vendor, or from the AMD web site — whether a particular Athlon 64 or Opteron processor model is compatible with Workstation or VMware Server 64-bit guests.

Note: The preceding link was correct as of June 21, 2012. If you find the link is broken, provide feedback and a VMware employee will update the link.

AMD

Segmentation support is missing from the initial AMD64 processors (that is, revision C and earlier) while running in long mode. As a result, AMD64 processors prior to revision D do not have an efficient mechanism for isolating the virtual machine monitor from 64-bit guest operating systems. A limited form of segmentation was reintroduced in long mode, in revision D AMD64 processors. As a result, AMD64 processors must be revision D or later to run 64-bit guest operating systems.

Note: Because AMD Opteron and Turion processors do not ship in revision D, AMD Opteron and Turion 64 processors must be revision E or later to run 64-bit guest operating systems.

When reading AMD's documentation, revisions may be given in decimal format (rather than hexadecimal format). For example, Revision F may also be referred to as Revision 15.

For more information about AMD64, see http://www.amd.com/us/products/technologies/64-bit-computing/Pages/64-bit-computing.aspx .

Note: The preceding link was correct as of June 21, 2012. If you find the link is broken, provide feedback and a VMware employee will update the link.

Intel

Intel CPUs require EM64T and VT support in the chip and in the BIOS.

Note: If VT is enabled on an ESX host, the vmx flag is found in /proc/cpuinfo.

Intel EM64T CPUs do not have segmentation support in long mode. If the Intel EM64T CPU is VT-capable, it comes with hardware virtualization support (Intel's Virtualization Technology). This hardware virtualization support allows VMware to work around the lack of segmentation, making it possible to run 64-bit guest operating systems on Intel EM64T VT-capable CPUs.

Notes:

• Firmware/BIOS support is required to enable Intel Virtualization Technology. Some systems (particularly laptops) do not have the necessary firmware/BIOS support to enable Intel Virtualization Technology, and cannot run 64-bit guest operating systems.
• In multi-processor systems, all processors must support compatible VT implementations. At a minimum, all processors must implement the same VMCS revision ID. In multi-processor systems with mixed processor steppings, 64-bit guest operating systems may not be supported.
• Many servers that include CPUs with VT support might ship with VT disabled by default, and VT must be enabled manually. If your CPUs support VT but you do not see this option in the BIOS, contact your system vendor to request a BIOS version that lets you enable VT support.

For additional information, see the VMware Communities article Trouble-shooting Intel VT-x Issues.

Source:-
http://kb.vmware.com/selfservice/microsites/microsite.do?cmd=displayKC&docType=kc&externalId=1003945&sliceId=1&docTypeID=DT_KB_1_1

Investigating virtual machine file locks on ESX/ESXi

Solution

To prevent concurrent changes to critical virtual machine files and file systems, ESX hosts establish locks on these files. In certain circumstances these locks may not be released when the virtual machine is powered off. The files cannot be accessed by the servers while locked, and the virtual machine is unable to power on.

These virtual machine files are commonly locked for runtime:

• <VMNAME>.vswp
• <DISKNAME>-flat.vmdk
• <DISKNAME>-<ITERATION>-delta.vmdk
• <VMNAME>.vmx
• <VMNAME>.vmxf
• vmware.log  

Identifying the locked file

To identify the locked file, attempt to power on the virtual machine. During the power on process, an error may display or be written to the virtual machine's logs. The error and the log entry identify the virtual machine and files:

1. Where applicable, open and connect the VMware Infrastructure (VI) or vSphere Client to the respective ESX host, VirtualCenter Server, or the vCenter Server hostname or IP address.
2. Locate the affected virtual machine, and attempt to power it on.
3. Open a remote console window for the virtual machine.
4. If the virtual machine is unable to power on, an error on the remote console screen displays with the name of the affected file.
   
   Note: If an error does not display, proceed to these steps to review the vmware.log file of the virtual machine.
   
   
   1. Log in as root to the ESX host using an SSH client.
   2. Confirm that the virtual machine is registered on the server and obtain the full path to the virtual machine:
      
      
      • On an ESX host, run this command:
        
        # vmware-cmd -l
        
        The output returns a list of the virtual machines registered to the ESX host. Each line contains the full path of a virtual machine's .vmx file. The output will be similar to:
        
        /vmfs/volumes/<UUID>/<VMDIR>/<VMNAME>.vmx
        
        Note: Record this information as it is required in the remainder of this process. This is the <path.vmx>referenced in the remainder of the article. It is also case-sensitive.
      • On an ESXi host, run this command:
        
        # vim-cmd vmsvc/getallvms
        
        The output returns a list of the virtual machines registered to the ESX host. Each line contains the datastore and location within of a virtual machine's .vmx file. The output will be similar to:
        
        [<datastore>] <VMDIR>/<VMNAME>.vmx
        
        Verify that the affected virtual machine appears in this list. If it is not listed, the virtual machine is not registered on this ESX/ESXi host. The host on which the virtual machine is registered typically holds the lock. Ensure that you are connected to the proper host before proceeding.
   3. Move to the virtual machine's directory:
      
      # cd /vmfs/volumes/<datastore>/<VMDIR>
   4. Use a text viewer to read the contents of the vmware.log file. At the end of the file, look for error messages that identify the affected file.

Using the touch utility to determine if the file can be locked

The touch utility is designed to update the access and modification time stamp of the specified file or directory. As such, the command can be used to test the file and directory locking mechanism in the VMFS filesystem, where the procedure is expected to fail on locked files. Using touch is the preferred method because the changes to the resource are minimal.

To test the file or directory locking functionality, run this command:

# touch <filename>

Note: Performing a "touch *" command performs the operation on all files in the current directory.

The touch * command can result in these outcomes:

• If the touch * command succeeds, then the command successfully made changes to the date/time stamp and has verified that the file can and has been locked (then unlocked). At this point, retry the virtual machine power-on operation to see if it succeeds.
• If the touch * command fails with a device or resource busy message, it indicates that a process is maintaining a lock on the file or directory. This may be on any of the ESX hosts which have access to the file. If the message is reported, proceed to the next section.
• If another error message is reported, it may indicate that the metadata pertaining to file or directory locking on VMFS may not be valid or corrupt. If this is the case, collect diagnostic information from the VMware ESX host and submit a support request. For more information, see Collecting diagnostic information for VMware products (1008524) and How to Submit a Support Request.

Locating the lock and removing it

Because a virtual machine can be moved between hosts, the host where the virtual machine is currently registered may not be the host maintaining the file lock. The lock must be released by the ESX host that owns the lock. This host is identified by the MAC address of the primary Service Console interface.

Note: Locked files can also be caused by backup programs keeping a lock on the file while backing up the virtual machine. If there are any issues with the backup it may result in the lock not being removed correctly.
In some cases you may need to disable your backup application or reboot the backup server to clear the hung backup.

This lock can be maintained by either the VMkernel (ESX/ESXi) or the Service Console (ESX) for any hosts connected to the same storage.

Note : VMware ESXi does not utilize a separate Service Console Operating System. This reduces the amount of lock troubleshooting to just the VMkernel. For example, Console OS troubleshooting methods such as using the lsof utility are not applicable to VMware ESXi hosts.

Start with identifying the server whose VMkernel may be locking the file. To identify the server:

1. Report the MAC address of the lock holder by running the command (except on NFS volume):
   
   # vmkfstools -D /vmfs/volumes/<UUID>/<VMDIR>/<LOCKEDFILE.xxx>
   
   Note: Run this command on all commonly locked virtual machine files (as listed at start of Resolution) to ensure that all locked files are identified.
2. For servers prior to VMware ESX/ESXi 4.1, this command writes the output of the above command to the system's logs. From ESX/ESXi 4.1, the output is also displayed on-screen. Included in this output is the MAC address of any host that is locking the .vmdk file. To locate this information, run this command:
   
   # tail /var/log/vmkernel (For ESX)
   # tail /var/log/messages (For ESXi)Note: If there is a high amount of logging activity and you are unable to locate lines similar to the example below, use theless command in lieu of tail. You may press G, once open, to immediately scroll to the bottom of the log's output. Use your arrow, page, or scroll keys to locate the relevant output.
   Look for lines similar to this:
   
   Hostname vmkernel: 17:00:38:46.977 cpu1:1033)Lock [type 10c00001 offset 13058048 v 20, hb offset 3499520
   Hostname vmkernel: gen 532, mode 1, owner 45feb537-9c52009b-e812- 00137266e200 mtime 1174669462]Hostname vmkernel: 17:00:38:46.977 cpu1:1033)Addr <4, 136, 2>, gen 19, links 1, type reg, flags 0x0, uid 0, gid 0, mode 600
   Hostname vmkernel: 17:00:38:46.977 cpu1:1033)len 297795584, nb 142 tbz 0, zla 1, bs 2097152
   Hostname vmkernel: 17:00:38:46.977 cpu1:1033)FS3: 132: <END supp167-w2k3-VC-a3112729.vswp>The second line (in bold) displays the MAC address after the word owner. In this example, the MAC address of the Service Console or vswif0 interface of the offending ESX Server is 00:13:72:66:E2:00. After logging into the server, the process maintaining the lock can be analyzed.
   
   Note: If this process does not reveal the MAC address, or the owner identifier is all zeroes, it is possible that it is a Service Console-based lock, an NFS lock, or a lock generated by another system or product that can use or read VMFS file systems. In other circumstances, the file is locked by a VMkernel child or cartel world and the offending host running the process/world must be rebooted to clear it.
3. To determine if the MAC address corresponds to the host that you are currently logged into, see Identifying the ESX Service Console MAC address (1001167). If it does not, you must establish a console or SSH connection to each host that has access to this virtual machine's files.
   
   After identifying, unregister the virtual machine from the existing host and register it on the host holding the lock and then attempt to power on the virtual machine. You may have to set DRS to manual to ensure that the virtual machine powers on the correct host.
   If the virtual machine still does not get powered on, complete these procedures while logged in to the offending host.
   
   Note: If you have already identified a vmkernel lock on the file, skip the rest of the steps in this section.
4. To check for Service Console-based locks on non-ESXi servers, run this command:
   
   # lsof | grep <name of locked file>
   You will see an output similar to:
   
   COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
   71fd60b6- 3631 root 4r REG 0,9 10737418240 23533 <name of locked file>
   Note: If there is no Service Console process locking the file, you should receive no printed output. If you receive any results, however, file a Support Request to identify the process and to determine the root-cause. If it is a third-party process, however, contact the appropriate vendor in order to determine the root-cause prior to killing the process ID, as it may occur again in the future.
   
   Stop the process ID and its lock using the kill command. From the above example, the process ID is 3631:
   
   # kill 3631
   Warning: Using the kill command will abruptly terminate all the running process for the virtual machine without generating any core dump to analyze the status later so be careful before using this command if you decide to troubleshoot the virtual machine state. To Collect diagnostic information by crashing a virtual machine see KB 2005715.
   
   After stopping the process, you can attempt to power on the virtual machine or access the file/resource.
5. Check if the virtual machine still has a World ID assigned to it:
   
   For ESX/ESXi 4.x, run these commands on all ESX/ESXi hosts:
   
   # cd /tmp
   # vm-support -x
   The output will be similar to:
   
   Available worlds to debug:
   wid=<world id> <name of VM with locked file>
   On the ESX/ESXi host where the virtual machine is still running, kill the virtual machine, which releases the lock on the file. To kill the virtual machine, run this command:
   
   # vm-support -X <world id>
   Where the <world id> is the World ID of the virtual machine with the locked file.
   
   Note: This command takes 5-10 minutes to complete. Answer No to "Can I include a screenshot of the VM", and answerYes to all subsequent questions.
   
   After stopping the process, you can power on the virtual machine or access the file/resource
   
   For ESXi 5. 0, the esxcli command-line utility can be used locally or remotely to display a list of the virtual machines which are currently running on the host.
   Obtain a list of all running virtual machines, identified by their World ID, Cartel ID, display name and path to the .vmx configuration file using this command:
   
   # esxcli vm process list
   The output appears similar to this:
   
   VirtualMachineName
   World ID: 1268395
   Process ID: 0
   VMX Cartel ID: 1264298
   UUID: ab cd ef ...
   Display Name: <VirtualMachineName>
   Config File: </path/VirtualMachineName>.vmx
   Two worlds are listed. The first world number (in this example, 1268395) is the Virtual Machine Monitor (VMM) for vCPU 0. The second world number (in this example, 1264298) is the virtual machine cartel ID.
   On the ESX/ESXi host where the virtual machine is still running, kill the virtual machine, which releases the lock on the file. To kill the virtual machine, run this command:
   
# esxcli vm process kill --type soft --world-id 1268395
   For additional information, see Mapping a virtual machine world number to a virtual machine name (1001101).
6. In ESXi 5.0 and ESXi 4.1, to find the owner of the locked file of a virtual machine, run this command:
   
   # vmkvsitools lsof | grep <Virtual Machine Name>
   You see an output similar to:
   
   11773 vmx 12 46 /vmfs/volumes/<Datastore Name>/VirtualMachineName/VirtualMachineName-flat.vmdk
   
   You can then run this command to get the PID of the process for the virtual machine:
   
   
   ps | grep <PID>
   
   You can kill the process with this command:
   
   kill -9 <PID>To generate the core dump after killing the running virtual machine (but hung and nonresponsive)  use the commandkill -6 <PID> or kill -11 <PID>.
   
   Note: In ESXi 4.1 and ESXi 5.0, you can use the k command in esxtop to send a signal to, and kill, a running virtual machine process. On the ESXi console, enter Tech Support mode and log in as root. For more information, see Tech Support Mode for Emergency Support (1003677).
   
   
   1. Run the esxtop utility using the esxtop command.
   2. Press c to switch to the CPU resource utilization screen.
   3. Press Shift+f to display the list of fields.
   4. Press c to add the column for the Leader World ID.
   5. Identify the target virtual machine by its Name and Leader World ID (LWID).
   6. Press k.
   7. At the World to kill prompt, type in the Leader World ID from step 5 and press Enter.
   8. Wait 30 seconds and validate that the process is not longer listed.

Removing the .lck file (NFS Only)

The files on the virtual machine may be locked via NFS storage. You can identify this by files denoted with .lck.#### (where #### refers to the World ID that has the file lock) at the end of the filename. This is an NFS file lock and is only listed when using the ls -la command as it is hidden file.

Caution: These can be removed safely only if the virtual machine is not running.

Note: VMFS volumes do not have .lck files. The locking mechanism for VMFS volumes is handled within VMFS metadata on the volume.

Determining if the file is being used by a running virtual machine

If the file is being accessed by a running virtual machine, the lock cannot be usurped or removed. It is possible that the lockholder host is running the virtual machine and has become unresponsive, or another running virtual machine has the disk incorrectly added to its configuration prior to power-on attempts.

To determine if the virtual machine processes are running:

1. Determine if the virtual machine is registered on the host:
   
   
   • For ESX, run this command as the root user:
     
     # vmware-cmd -l
     Note: If the virtual machine is registered on more than one ESX host, see Virtual Machines appear to be running or registered on multiple ESX Servers (1005051).
     

   • For ESXi, run this command as the root user:
     
     # vim-cmd vmsvc/getallvms
     
     The output lists the Vmid for each virtual machine registered. Record this information as it is required in the remainder of this process on the ESXi server. 
     
2. Assess the virtual machines current state on the host:
   
   
   • For ESX, run this command:
     
     # vmware-cmd <path.vmx> getstate
     
     
     • If the output from this command is getstate() = on , the virtual machine has become unresponsive. To address this issue, see Troubleshooting a virtual machine that has stopped responding (1007819).
     • If the output from this command is getstate() = off , the ESX host may be unaware of the file lock.
   • For ESXi, run this command:
     
     # vim-cmd vmsvc/power.getstate <vmid>
3. To stop the virtual machine process, see Powering off an unresponsive virtual machine on an ESX host (1004340).

Determining if the .vmdk file is in use by other virtual machines

A lock on the .vmdk file can prevent a virtual machine from starting. However, since virtual machine disk files can be configured for use with any virtual machine, the file may be locked by another virtual machine that is currently running.

To determine if the virtual machine's disk file is configured for use on more than one virtual machine, run this command:

# egrep -i <DISKNAME>.vmdk /vmfs/volumes/*/*/*.vmx

Notes:

• This command attempts to locate the specified disk name among all .vmx configuration files for the virtual machines that are visible to the ESX host. A Device or resource busy message is printed for each virtual machine that is running but not registered to this ESX host. You must run this command on each ESX host in the infrastructure or specifically on ESX hosts that have access to the storage containing the virtual machine's files.
• If any additional virtual machines are configured to use the disk, determine if they are currently running. Powering off the other virtual machine using the disk file releases the lock. You must determine which virtual machine should have ownership of the file, then reconfigure your virtual machines to prevent this error from occurring again.
• As part of their operation, many virtual machine backup solutions temporarily attach the virtual machine's .vmdk files to themselves. In such cases, if the backup fails and/or the host shuts down, the backup virtual machine may still have another virtual machine's vmdk file(s) attached. If that is the case, the other virtual machine is usually powered on first, which then creates a locked file condition when the backup virtual machine is attempted to be powered on. Check using Edit Settings on your backup solution's virtual machine to see if it has a hard disk attached that belongs to a different virtual machine. If it does, power down the backup virtual machine, select the appropriate disk and choose Remove to remove the disk from the virtual machine.
  
  Warning: Do not delete files from disk.

If the .vmdk file is not used by other virtual machines, confirm that there are no VMkernel or Service Console processes locking the file, per the above section, Locating the file lock and removing it . If a host can be determined however the specific offending VMkernel child process ID cannot be identified, the server requires rebooting to clear the lock.

Note: You can also try to migrate the virtual machine to another host and power it on. If that ESX host has the lock for the virtual machine, it should allow you to power it on.

Rebooting the ESX host which is locking the files

By this stage, you have already investigated for identifiable VMkernel and Service Console processes which have maintained locks upon the required files, however an unidentified child process still maintains the lock. You have identified the server via thevmkfstools -D command in earlier steps, the lsof utility yields no offending processes, and no other virtual machines are locking the file.

The server should be restarted to allow the virtual machine to be powered on again.

Note: Collect diagnostic information prior to rebooting if you wish to pursue a root-cause analysis with VMware Technical Support.

Migrate the virtual machines from the server and restart it using these processes:

1. Migrate or vMotion all virtual machines from the host to alternate hosts.
2. When the virtual machines have been evacuated, place the host into maintenance mode and reboot it.
   
   Note: If you have only one ESX host or do not have the ability to vMotion or migrate virtual machines, you must schedule downtime for the affected virtual machines prior to rebooting. When the host has rebooted, start the affected virtual machine again.

Check the integrity of the virtual machine configuration file (.vmx)

For more information on checking the integrity of the virtual machine configuration file, see Verifying ESX/ESXi virtual machine file integrity (1003743).  

Opening a Support Request

If your problem still exists after attempting the steps in this article, contact VMware Technical Support and file a Support Request:

• Gather diagnostic information. For more information, see Collecting diagnostic information for VMware products (1008524).
• File a support request with VMware Support and note this KB Article ID in the problem description. For more information, see How to Submit a Support Request.

Note: For related information, see Cannot power on a virtual machine because the virtual disk cannot be opened (1004232).

Source:-