Using Storage vMotion to migrate a virtual machine with many disks timeout

Symptoms

You may experience these symptoms:

• Storage vMotion fails.
• The Storage vMotion operation fails with a timeout between 5-10% or 90-95% complete.
• On ESX 4.1 you may see the errors:
  
  In hostd.log
  
  v ix: [7196 foundryVM.c:10177]: Error VIX_E_INVALID_ARG in VixVM_CancelOps(): One of the parameters was invalid 'vm:/vmfs/volumes/4e417019-4a3c4130-ed96-a4badb51cd0a/Mail02/Mail02.vmx' opID=9BED9F06-000002BE-9d] Failed to unset VM medatadata: FileIO error: Could not find file : /vmfs/volumes/4e417019-4a3c4130-ed96-a4badb51cd0a/Mail02/Mail02-aux.xml.tmp.
   In vmware.log
  
  vmkernel: 114:03:25:51.489 cpu0:4100)WARNING: FSR: 690: 1313159068180024 S: Maximum switchover time (100 seconds) reached. Failing migration; VM should resume on source.
  vmkernel: 114:03:25:51.489 cpu2:10561)WARNING: FSR: 3281: 1313159068180024 D: The migration exceeded the maximum switchover time of 100 second(s). ESX has preemptively failed the migration to allow the VM to continue running on the source host.
  vmkernel: 114:03:25:51.489 cpu2:10561)WARNING: Migrate: 296: 1313159068180024 D: Failed: Maximum switchover time for migration exceeded(0xbad0109) @0x41800f61cee2

• The vCenter Server logs contain entries similar to:
  
  [yyyy-mm-dd hh:mm:ss.nnn tttt error 'App'] [MIGRATE] (migrateidentifier) vMotion failed: vmodl.fault.SystemError
  [yyyy-mm-dd hh:mm:ss.nnn tttt verbose 'App'] [VpxVmomi] Throw vmodl.fault.SystemError with:
  (vmodl.fault.SystemError) {
  dynamicType = <unset>,
  reason = "Source detected that destination failed to resume.",
  msg = "A general system error occurred: Source detected that destination failed to resume."

Resolution

Note: A virtual machine with many virtual disks might be unable to complete a migration with Storage vMotion. The Storage vMotion process requires time to open, close, and process disks during the final copy phase. Storage vMotion migration of virtual machines with many disks might timeout because of this per-disk overhead.

This timeout occurs when the maximum amount of time for switchover to the destination is exceeded. This may occur if there are a large number of provisioning, migration, or power operations occurring on the same datastore as the Storage vMotion. The virtual machine's disk files are reopened during this time, so disk performance issues or large numbers of disks may lead to timeouts.

The default timeout is 100 seconds, and can be modified by changing the fsr.maxSwitchoverSeconds option in the virtual machine configuration to a larger value. This change must be done with the virtual machine powered down.

 

To modify the fsr.maxSwitchoverSeconds option using the vSphere Client: 

1. Open vSphere Client and connect to the ESX/ESXi host or to vCenter Server.
2. Locate the virtual machine in the inventory.
3. Power off the virtual machine.
4. Right-click the virtual machine and click Edit Settings.
5. Click the Options tab.
6. Select the Advanced: General section.
7. Click the Configuration Parameters button.
   
   Note: The Configuration Parameters button is disabled when the virtual machine is powered on.
   
8. From the Configuration Parameters window, click Add Row.
9. In the Name field, enter the parameter name:
   
   fsr.maxSwitchoverSeconds

10. In the Value field, enter the new timeout value in seconds (for example: 150).
11. Click the OK buttons twice to save the configuration change.
12. Power on the virtual machine.

 

To modify the fsr.maxSwitchoverSeconds option by editing the .vmx file manually: 

 

The virtual machine's .vmx configuration file can be manually edited to add or modify the option. Add the optionfsr.maxSwitchoverSeconds = "<new value>" on its own line.

 

For more information, see Tips for editing a .vmx file (1714).

Note: To edit a virtual machines configuration file you will need to power off the virtual machine, remove it from Inventory, make the changes to the vmx file, add the virtual machine back to inventory, and power the virtual machine on again.

 

Source:-

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1010045

 

Best practices for joining vCenter Servers in Linked Mode

Purpose

This article provides best practices when working with vCenter Server Linked Mode, as well as steps to troubleshoot vCenter Server Linked Mode issues.

Resolution

Best practices

 

When working with vCenter Server Linked Mode issue, follow these best practices:

• If the vCenter Server is joined to a domain, ensure that it can communicate with the Domain Controller. If Domain Controller communication problems exist, remove and add the vCenter Server to the Windows domain.
  
• Ensure that all vCenter Server system times are synchronized with a time difference of no greater than 5 minutes.
  
• Ensure that all vCenter Server are the same version and build. For more information, see Cannot access instances of vCenter Server in Linked Mode configuration after upgrading to vCenter Server 4.1 (1026346).
  
• Ensure that the VirtualCenter Server Service uses an account with rights to logon as a service/batch job.
  
• The vCenter Server Linked Mode Configuration tool must be run by a domain user that is also a local administrator on both machines where vCenter Server is installed.
  
• Different Windows domains for vCenter Servers are permitted only if there is a two-way trust between the two domains. Ensure this is true from both Windows domains.
  
• If User Account Control (UAC) is enabled, be sure to use Run as administrator when starting the vCenter Server Linked Mode Configuration tool.
  
• Ensure that the vCenter Server Windows machine name matches the Domain/DNS name.
  
  Note: Instancename, VimWebServicesUrl, and VimApiUrl keys must match. For more information, see ESX and vCenter Server Installation Guide.
  
• Ensure that the Windows firewall service is running but the firewall is turned off.

Verifying the initial replication

The Jointool/vCenter Server installer does a large set of checks to validate initial replication between instances. Issues with joining two instances are usually due to errors in initial replication. However, after a successful join (especially with more than two total instances in the vCenter Server linked mode group), some instances may not see all instances in the group.

 

To see if ADAM replication is the issue, perform these steps on all concerned vCenter Server machines:

1. Click Start > Administrative Tools > ADSI Edit.
2. Right-click ADSI Edit in the left pane and click Connect to.
3. Under Connection Point in the Distinguished Name box, enter dc=virtualcenter,dc=vmware,dc=int. 
4. Under Computer in the domain or server box, enter localhost:389, then click OK. This opens up a new connection to our application partition in ADAM.
5. Expand Default naming context and drill down clicking the OU=Instances container on the left pane. You see entries (GUIDs) under OU=Instances for the vCenter Servers in your setup.
   
   This list should be identical on every replica (and the primary). It does guarantee that replication will continue to succeed,  but it does indicate that initial replication during installation was successful.

Verifying the Health service status

To verify the Health service status for the LDAP Replication Monitor, install the service-monitoring vSphere Client plugin as part of all vCenter Server installs:

1. In vSphere Client, click Home.
2. Click vCenter Server Service Status in the Administration section. 
   
   Note: If you do not see vCenter Service Status, you have to enable the plugin by clicking Plug-ins > Manage plug-ins.

Troubleshooting replication issues

To troubleshoot replication issues:

1. Click Start > Administrative Tools > Event Viewer.
   • Review the  Event Viewer Log entries for related ADAM instance (VMwareVCMSDS or something similar) events. Record any warning or error messages you find.
   • Example warning messages involving replication are often explicit. For example:
     
     8453 Replication access was denied.1772 The list of RPC servers available for the binding of auto handles has been exhausted.
     Note: This error is often a symptom of firewalls blocking ports (RPC mapper runs on port 135, and needs ports > 1024 to be open on the machine).
     
2. Run Knowledge Consistency Check (KCC) from the command line to confirm replication is the problem. Run KCC on the replica machine:
   
   
   • C:\Windows\ADAM\repadmin.exe /kcc localhost 389 (to confirm local consistency )
   • C:\Windows\ADAM\repadmin.exe /kcc remoteVCFQDNremotePort (to confirm remote primary consistency)
     
     If either of these return an error, inform VMware if you open a Support Request.
     
3. Forcing replication can help diagnose issues. To force replication between ADAM instances:
   
   C:\WINDOWS\ADAM>repadmin /replicate remote-vc:remote-vc-adam-portlocal-vc-fqdn:local-adam-portdc=virtualcenter,dc=vmware,dc=int
   
   This is an example of successful replication:
   
   C:\WINDOWS\ADAM>repadmin /replicate vm08.PDPVC.com:389 vm04.PDPVC.com:389 dc=virtualcenter,dc=vmware,dc=int Positive response:
   Sync from vm04.PDPVC.com:389 to vm08.PDPVC.com:389 completed successfully.
   
   This is an example of failed replication:
   DsBindWithCred to vm04.pdpvc.com failed with status 1753 (0x6d9):There are no more endpoints available from the endpoint mapper
   
4. To verify inbound and outbound replication from one machine, run the command:
   
   repadmin /syncall localhost:vc-ldap-port
   
5. Run directory service tests with dsdiag. This runs a comprehensive list of tests to help diagnose what may have failed with the replication (such as name resolution and or referrals):
   
   (c:\windows\adam or c:\windows\system32) dsdiag /s: localhost:vc-ldap-port

Additional Information

Reporting issues:

• Generate a Virtual Center support bundle, see  Collecting diagnostic information for VMware vCenter Server (1011641).
• The Virtual Center server support bundle contains the diagnostic information to help resolve replication errors between instances.

Source:-

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2005481

Setting Multiple Isolation Response Addresses for VMware High Availability

Details

In VirtualCenter Server 2.0.2 and above, you can specify more than one isolation response address for VMware High Availability (HA). The use of multiple isolation response addresses offers VMware HA a potentially more accurate picture of the network connectivity of a host. There may be situations in which a single isolation address would indicate that a host is in a state of complete isolation from the network, but access to additional isolation addresses would show that only a partial network failure has occurred.

Solution

Multiple isolation response addresses can be specified using the das.isolationaddress0 throughdas.isolationaddress9 options.

 

To specify the values for these options in Virtual Infrastructure Client:

1. Select the HA cluster.
2. On the Summary tab, click Edit Settings.
3. In the Settings dialog, select VMware HA.
4. Click Advanced Options.
5. In the Advanced Options (HA) dialog, enter the option name and the corresponding value: 
   
   
   • Option: das.isolationaddress0
   • Value: A valid IP address other than the default gateway address 
     
     Similarly, you can set more isolation response addresses using das.isolationaddress1 throughdas.isolationaddress9. 
     
6. In the Advanced Options (HA) dialog box, set this option:
   • Option: das.usedefaultisolationaddress
   • Value: false 
     
     Notes:
     • You must set this option if you are setting multiple isolation response addresses. 
     • Additionally you should configure das.usedefaultisolationaddress to false when the default gateway is a device which cannot be pinged.
       
7. Click OK.
8. Click OK.
9. Edit settings for the cluster again.
10. Disable HA and click OK.
11. Edit settings again.
12. Enable HA and click OK.

At the time of VMware HA configuration, if none of the specified addresses can be reached, the configuration operation fails. If only a subset of the specified isolation addresses cannot be reached, then the configuration operation succeeds, but a configuration issue is displayed for the cluster and the cluster becomes yellow. The configuration issue identifies the isolation response addresses that could not be reached.

Although VMware HA supports a maximum of 10 isolation response addresses, in most cases one or two isolation response addresses should be sufficient. Also, the das.isolationaddress option formerly used to set a single isolation response address is still supported.

Note: If you specify multiple isolation response addresses, VMware recommends that you ensure the correct operation of VMware HA by:

• Use the default gateway address and other reliable addresses physically close to the hosts in the cluster.
• Ensure a corresponding isolation address is setup for each service console network that has been created on the ESX Server.
• Change the default failure detection time to 20 seconds or greater. To change the default failure detection time: 
  
  
  1. Select the HA cluster.
  2. On the Summary tab, click Edit Settings.
  3. In the Settings dialog, select VMware HA.
  4. Click Advanced Options.
  5. In the Advanced Options (HA) dialog, enter the option name and the corresponding value: 
     
     
     • Option: das.failuredetectiontime
     • Value: A value in milliseconds that represents the timeout value (20 seconds = 20000 milliseconds)
       
  6. Click OK.
  7. Click OK. 

Notes:

• das.failuredetectiontime does not need to be increased when multiple das.isolationaddress options are specified as the pings occur in parallel.
• das.failuredetectiontime is no longer supported in vCenter Server 5.0. For more information, see vSphere HA Advanced Attributes. 

For information on VMware HA best practices, see Best practices and advanced features for VMware High Availability (1002080).

For further details, see the vSphere Availability guide.

Source of this info:-
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1002117

Disabling Time Synchronization

Details

In the VMware Tools control panel, the time synchronization checkbox is unselected, but you may experience these symptoms:

• When you suspend a virtual machine, the next time you resume that virtual machine it synchronizes the time to adjust it to the host.
• Time is resynchronized when you migrate the virtual machine using vMotion, take a snapshot, restore to a snapshot, shrink the virtual disk, or restart the tools service in the virtual machine (including rebooting the virtual machine).

Solution

The time synchronization checkbox controls only whether time is periodically resynchronized while the virtual machine is running. Even if this box is unselected, VMware Tools by default synchronizes the virtual machine's time after a few specific events that are likely to leave the time incorrect.

There are two methods to completely disable time synchronization in the guest. Perform one of these procedures:

Note: Unlike the normal periodic time sync, which can move the guest clock only forward, these special time syncs can move the guest clock either forward or backward. If you want your guests to never sync backward in time then please disable the options (by selecting "0" for each option).

• Open the virtual machine's configuration file (.vmx) in a text editor and set these options to zero. If the entries do not exist, add them.
  
  Note: 0 = disabled, 1 = enabled.
  
  tools.syncTime = "0"
  time.synchronize.continue = "0"
  time.synchronize.restore = "0"
  time.synchronize.resume.disk = "0"
  time.synchronize.shrink = "0"
  time.synchronize.tools.startup = "0"
  time.synchronize.tools.enable = "0"
  time.synchronize.resume.host = "0"
  
• Select the virtual machine in the VMware Infrastructure Client inventory. On the Summary tab, click Edit Settings, then click the Options tab and select General (under Advanced). Click Configuration Parameters, then click Add Row and add this information:
  
  tools.syncTime = "0"
  time.synchronize.continue = "0"
  time.synchronize.restore = "0"
  time.synchronize.resume.disk = "0"
  time.synchronize.shrink = "0"
  time.synchronize.tools.startup = "0"
  time.synchronize.tools.enable = "0"
  time.synchronize.resume.host = "0"

Notes:

• For GSX and VMware Workstation, you need to use "FALSE" instead of "0". For example:
  
  tools.syncTime = "FALSE"

Source of this Info:-
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1189

Comparing VMware HA 4.x and vSphere HA 5.0

Purpose

This article provides a detailed comparison of VMware High Availability (HA) 4.x and vSphere HA 5.x.

Resolution

Capability	VMware HA 4.x	vSphere HA 5.0
Agents	VMware HA in vSphere 4.x has AAM (Automated Availability Manager) agents	VMware HA in vSphere 5.0 has FDM (Fault Domain Manager) agents installed
Cluster Members	The first five hosts added to the cluster are designated as primary hosts, and all subsequent hosts are designated as secondary hosts. If a primary host is removed from the cluster, VMware HA promotes another (secondary) host to primary status.	Each host in the cluster functions as a master host or a slave host. Only one master host exists per cluster and all other hosts are slave hosts. If the master host fails, is shut down, or is removed from the cluster a new election is held.
Responsibilities	Primary Hosts: Decide where to restart virtual machines. Keep track of failed restart attempts. Determine when it is appropriate to keep trying to restart a virtual machine	Master Host: Monitors the state of slave hosts. If a slave host fails or becomes unreachable, the virtual machines on that host are restarted. Monitors the power state of all protected virtual machines. Manages the lists of cluster hosts and protected virtual machines. Acts as vCenter Server management interface to the cluster and reports the cluster health state.
Host isolation methods	Isolation is based on network only.	Isolation is based on network and storage.
Host Failure Detection	If a host stops receiving heartbeats from all other hosts in the cluster and isolation address for 15 seconds or longer	Master/Slave Host management network is lost. Master host management network lost & cannot ping isolation response IP address. Master host fails.
Slot Size Calculation	The default CPU reservation of each powered-on virtual machine is 256 Mhz.	The default CPU reservation of each powered-on virtual machine is 32 Mhz.
Configuration Files	There are no configuration files for VMware HA.	The configuration file is located at/etc/opt/vmware/fdm/ fdm.cfg includes settings for logging, alerting and the working FDM directory.
Logs	Logs are available in/var/log/vmware/aam	The log file is located in/var/log/vmware/fdm/  FDM uses a single log file per host and supports syslog integration.  The remaining files are located at/etc/opt/vmware/fdm/  hostlist: Includes hostnames, IP Addresses, MAC addresses and heartbeat datastores of hosts participating in the cluster.  compatlist: A compatibility matrix which lists all the hosts with which a Virtual Machine is compatible.  clusterconfig: Includes details such as Host Isolation Response, VM Restart Priority, Admission Controls & any relevant DRS & DPM information.
Heartbeat Datastores	Not available	Datastores are used as a communication channel only when the management network is lost, such as in the case of isolation & network partitioning.

Source:-
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2004401

A virtual machine fails to power on with the error: Thin/TBZ disks cannot be opened in multiwriter mode. VMware ESX cannot open the virtual disk for clustering.

Symptoms

Summary

A virtual machine fails to power on with the error: 

Thin/TBZ disks cannot be opened in multiwriter mode. VMware ESX cannot open the virtual disk for clustering.

Example

Reason: Thin/TBZ disks cannot be opened in multiwriter mode..
Cannot open the disk '/vmfs/volumes/4c549ecd-66066010-e610-002354a2261b/Windows Server 2008/Windows Server 2008.vmdk' or one of the snapshot disks it depends on.
VMware ESX cannot open the virtual disk, "/vmfs/volumes/4c549ecd-66066010-e610-002354a2261b/Windows Server 2008/Windows Server 2008.vmdk" for clustering. Please verify that the virtual disk was created using the 'thick' option.

Impact

The virtual machine fails to power on.

Resolution

This issue usually occurs in configurations where a virtual disk is shared across multiple virtual machines.

The shared virtual disks must be in the eagerzeroedthick disk format to facilitate clustering configurations, such as Microsoft Clustering service and VMware Fault Tolerance. This issue occurs if the disk is not in the correct format. To verify if a virtual disk is zeroedthick/eagerzeroedthick, see Determining if a VMDK is zeroedthick or eagerzeroedthick (1011170).

To resolve this issue, see:

• Enabling clustering features for an existing virtual disk by converting in place (1035823)
• Enabling clustering features for an existing virtual disk by converting while copying (1003491)

Source of this Info:-

http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1033570