Creating a persistent scratch location for ESXi

Symptoms

• When you run the command esxcli software vib list, you see an error similar to:
  
  Traceback (most recent call last):
    File "/usr/lib/vmware/esxcli-software", line 432, in <module>
      setupLogging()
    File "/usr/lib/vmware/esxcli-software", line 78, in setupLogging
      handler = logging.handlers.SysLogHandler(address='/dev/log')
    File "/build/mts/release/bora-504890/bora/build/esx/release/python-
      2.6-lib-zip-stage/504890/visor/pylib/python2.6/logging/handlers.py",
      line 704, in __init__
    File "/build/mts/release/bora-504890/bora/build/esx/release/python-
      2.6-lib-zip-stage/504890/visor/pylib/python2.6/logging/handlers.py",
      line 719, in _connect_unixsocket
    File "<string>", line 1, in connect
      socket.error: [Errno 2] No such file or directory
  
• None of the commands related to esxcli software or esxupdate work.
• Scanning the host using Update Manager fails with the error:
  
  General System Error
  
• Rebooting the host initially resolves the issue, but it reoccurs in a few days.
• Log files are missing in /var/log/ or are not being updated.
• You cannot install HA agent on affected ESXi hosts.
• ESXi is installed on a SD card or USB stick.

Purpose

This article describes the procedure to create a persistent scratch location for ESXi.

VMware recommends that ESXi has a persistent scratch location available for storing temporary data including logs, diagnostic information, and system swap. (This is not a requirement, however.) Persistent scratch space may be provisioned on a Fat16 or VMFS partition accessible by the ESXi host.

Cause

If persistent scratch space is not available, ESXi stores this temporary data on a ramdisk, which is constrained in space. This might be problematic in low-memory situations, but is not critical to the operation of ESXi. Information stored on a ramdisk does not persist across reboots, so troubleshooting information such as logs and core files could be lost. If a persistent scratch location on the host is not configured properly, there will be intermittent issues due to lack of space for temp files and the log files will not be updated.

Scratch space is configured automatically during installation or first boot of an ESXi host, and does not usually need to be manually configured. For more details, see the Additional Information section of this article.

Resolution

Scratch space can be manually configured for an ESXi host using the vSphere Client, the vCLI, PowerCLI, Tech Support Mode, or during scripted installation. Select the method appropriate to your environment. Regardless of the method selected, the configuration is written to the /etc/vmware/locker.conf configuration file for use during the next boot.

Changing the configured scratch location requires a reboot of the ESXi host for the changes to take effect. VMotion or power off all virtual machines on the ESXi host prior to making the change. For more information, see Rebooting an ESX Server host (1003530).

The configured scratch location must point to a directory on a filesystem accessible to the ESXi host, such as a Fat16 or VMFS volume. The location may be a directory on a shared or remote disk as long as the ESXi host is allocated its own separate directory.

 

Caution: Do not share a scratch locker directory among multiple ESXi hosts.

 

Note: It may necessary to do a reboot of the ESXi host before completing these procedures.

Configuring a persistent scratch location using the vSphere Client

You can configure persistent scratch space for ESXi using the vSphere Client:

1. Connect to vCenter Server or the ESXi host using the vSphere Client.
2. Select the ESXi host in the inventory.
3. Click the Configuration tab.
4. Click Storage.
5. Right-click a datastore and select Browse.
6. Create a uniquely-named directory for this ESX host (for example, .locker-ESXHostname)
7. Close the Datastore Browser.
8. Click Advanced Settings under Software. 
9. Select the ScratchConfig section.
10. Change the ScratchConfig.ConfiguredScratchLocation configuration option, specifying the full path to the directory.
    
    For example:
    
    /vmfs/volumes/DatastoreName/.locker-ESXHostname
    
11. Click OK.
12. Put the ESXi host in maintenance mode and reboot for the configuration change to take effect.

Configuring a persistent scratch location using the vCLI

You can configure persistent scratch space for ESXi using the vSphere Command-Line Interface. For more information on usage, see the vSphere Command-Line Interface Documentation.

1. Open a command prompt at the location where the vCLI is installed.
2. Obtain a list of datastores reachable from this ESXi host using the command:
   
   vifs.pl connectoptions --listds
   
   For example:
   
   vifs.pl --server esxhostnameorip --username root --listds
   
3. Create a uniquely-named directory for this ESXi host using the command:
   
   vifs.pl connectoptions --mkdir "[DatastoreName] DirectoryName"
   
   For example:
   
   vifs.pl --server esxhostnameorip --username root --mkdir "[Datastore1] .locker-ESXHostname"
   
4. Check the current value of the ScratchConfig.ConfiguredScratchLocation configuration option using the command:
   
   vicfg-advcfg.pl connectoptions -g ScratchConfig.ConfiguredScratchLocation
   
   For example:
   
   vicfg-advcfg.pl --server esxhostnameorip --username root -g ScratchConfig.ConfiguredScratchLocation
   Value of ScratchConfig.ConfiguredScratchLocation is /path/to/location
   
5. Change the ScratchConfig.ConfiguredScratchLocation configuration option, specifying the full path to the directory created in Step 3, using the command:
   
   vicfg-advcfg.pl connectoptions -s /vmfs/volumes/DatastoreName/DirectoryNameScratchConfig.ConfiguredScratchLocation
   
   For example:
   
   vicfg-advcfg.pl --server esxhostnameorip --username root -s /vmfs/volumes/Datastore1/.locker-ESXHostname ScratchConfig.ConfiguredScratchLocation 
   
6. Put the ESXi host into maintenance mode and reboot for the configuration change to take effect.

Configuring a persistent scratch location using PowerCLI

You can configure persistent scratch space for ESXi using the vSphere PowerCLI interface. For more information on usage, see the vSphere PowerCLI documentation.

 

Note: Before proceeding, ensure that /tmp/scratch exists. If it does not exist, use the command mkdir /tmp/scratch to create it.

 

1. Open a command prompt where the PowerCLI is installed.
2. Connect to the ESXi host using the command:
   
   connect-viserver ESXHostnameOrIP
   
3. Obtain a list of datastores reachable from this ESXi host using the command:
   
   Get-Datastore
   
4. Mount a datastore read/write as a PSDrive using the command:
   
   New-PSDrive -Name "mounteddatastore" -Root \ -PSProvider VimDatastore -Datastore (Get-Datastore "DatastoreName")
   
5. Access the new PSDrive using the command:
   
   Set-Location mounteddatastore:
   
6. Create a uniquely-named directory for this ESXi host using the command:
   
   New-Item "DirectoryName" -ItemType directory
   
   For example:
   
   New-Item ".locker-ESXHostname" -ItemType directory
   
7. Check the current value of the ScratchConfig.ConfiguredScratchLocation configuration option using the command:
   
   Get-VMHostAdvancedConfiguration -Name "ScratchConfig.ConfiguredScratchLocation"
   
8. Change the ScratchConfig.ConfiguredScratchLocation configuration option, specifying the full path to the directory created in Step 6, using the command:
   
   Set-VMHostAdvancedConfiguration -Name "ScratchConfig.ConfiguredScratchLocation" -Value "/vmfs/volumes/DatastoreName/DirectoryName"
   
   For example:
   
   Set-VMHostAdvancedConfiguration -Name "ScratchConfig.ConfiguredScratchLocation" -Value "/vmfs/volumes/Datastore1/.locker-ESXHostname"

9. Put the ESXi host into maintenance mode and reboot for the configuration change to take effect.

Configuring a persistent scratch location using Tech Support Mode

You can configure persistent scratch space for ESXi using Tech Support Mode on ESXi, either via the local console or SSH.

Note: A .filename is a hidden file. To view hidden files, use the ls -a command.

1. Open a console to the ESXi host. For more information, see Using Tech Support Mode in ESXi 4.1 and 5.0 (1017910).
2. Obtain a list of datastores reachable from this ESXi host using the command:
   
   ls /vmfs/volumes
   
3. Create a uniquely-named directory for this ESXi host using the command:
   
   mkdir /vmfs/volumes/DatastoreName/DirectoryName
   
   For example:
   
   mkdir /vmfs/volumes/Datastore1/.locker-ESXHostname
   
4. Check the current value of the ScratchConfig.ConfiguredScratchLocation configuration option using the command:
   
   vim-cmd hostsvc/advopt/view ScratchConfig.ConfiguredScratchLocation
   
   Example output:
   
   (vim.option.OptionValue) [
   (vim.option.OptionValue) {
      dynamicType = <unset>,
      key = "ScratchConfig.ConfiguredScratchLocation",
      value = "/path/to/location",
   }
}
   
5. Change the ScratchConfig.ConfiguredScratchLocation configuration option, specifying the full path to the directory created in Step 3, using the command:

vim-cmd hostsvc/advopt/update ScratchConfig.ConfiguredScratchLocation string /vmfs/volumes/DatastoreName/DirectoryName
   
   For example:
   
   /bin/vim-cmd hostsvc/advopt/update ScratchConfig.ConfiguredScratchLocation string /vmfs/volumes/Datastore1/.locker-ESXHostname
   
6. Put the ESXi host into maintenance mode and reboot for the configuration change to take effect.

Configuring a persistent scratch location during scripted install using kickstart scripts

You can configure persistent scratch space for ESXi during kickstart installation by adding commands to your kickstart scripts which create the directory and change the configuration option. For more information on ESXi scripted deployments, see the ESXi Installable and vCenter Server Setup Guide.

Append these commands to your custom kickstart script:

# Generate a new scratch directory for this host on a Datastore
scratchdirectory=/vmfs/volumes/DatastoreName/.locker-$(hostname 2> /dev/null)-$(esxcfg-info -b 2> /dev/null)

# Create the scratch directory
mkdir -p $scratchdirectory

# Change the advanced configuration option
vim-cmd hostsvc/advopt/update ScratchConfig.ConfiguredScratchLocation string $scratchdirectory

Note: The configured scratch location is activated during startup of the ESXi host. A reboot is required after performing the preceding commands.

Additional Information

Further information on the scratch partition can be found in the Configuring Storage Behavior section of the ESXi Setup Guide for your version of ESXi.

Scratch space is configured automatically during installation or first boot of an ESXi host, and does not usually need to be manually configured. ESXi Installable creates a 4 GB Fat16 partition on the target device during installation if there is sufficient space, and if the device is considered Local.

 

If the host has been running for more than 24 hours, this setting may not stay persistent after performing the preceding steps. In this case, perform these steps:

1. Place the host into Maintenance Mode and reboot the host.
2. Click the Configuration tab.
3. Click Software.
4. Click Advanced Settings.
5. Select the ScratchConfig section.
6. Change the ScratchConfig.ConfiguredScratchLocation configuration option and set it to the full path of the directory.
   
   For example:
   
   /vmfs/volumes/DatastoreName/.locker-ESXHostname
   
7. Click OK.
8. Close the window.
9. Click Advanced Settings to reopen the window.
10. Select the ScratchConfig section.
11. Verify that the ScratchConfig.ConfiguredScratchLocation field contains the scratch setting that you configured in Step 6. If it does, reboot the host.
    
    The host is now using the configured scratch partition.

For more information on determining whether a disk device is considered Local, see Identifying disks when working with VMware ESX (1014953), and review the Is Local flag in the output of the esxcfg-scsidevs -l command. If Local is true, the storage device is not sharable among multiple ESXi hosts. If Local is false, the storage device is remote and potentially sharable among several ESXi hosts. Remote devices are not automatically used for scratch by ESXi.

ESXi selects one of these scratch locations during startup in this order of preference:

1. The location configured in the /etc/vmware/locker.conf configuration file, set by theScratchConfig.ConfiguredScratchLocation configuration option, as in this article.
2. A Fat16 filesystem of at least 4 GB on the Local Boot device.
3. A Fat16 filesystem of at least 4 GB on a Local device.
4. A VMFS Datastore on a Local device, in a .locker/ directory.
5. A ramdisk at /tmp/scratch/

Here are two examples where scratch space may not be automatically defined on persistent storage. In each case, the temporary scratch location will be configured on a ramdisk:

1. ESXi deployed on a Flash or SD device, including a USB key. Scratch partitions are not created on Flash or SD storage devices even if connected during install, due to the potentially limited read/write cycles available.
   
2. ESXi deployed in a Boot from SAN configuration or to a SAS device. A Boot from SAN or SAS LUN is considered Remote, and could potentially be shared among multiple ESXi hosts. Remote devices are not used for scratch to avoid collisions between multiple ESXi hosts.

Source:-

http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1033696

Setting the congestion Threshold Value for Storage I/O Control

Details

The congestion threshold value for a datastore is the upper limit of latency that is allowed for a datastore before Storage I/O Control begins to assign importance to the virtual machine workloads according to their shares. You do not need to adjust the congestion threshold setting in most environments.

 

CAUTION Storage I/O Control will not function correctly unless all datatores that share the same spindles on the array have the same congestion threshold.

Solution

 If you do change the threshold setting, set the value based on the following considerations.

• A higher value typically results in higher aggregate throughput and weaker isolation. Throttling will not occur unless the overall average latency is higher than the threshold
• If throughput is more critical than latency, do not set the value too low. For example, for Fibre Channel disks, a value below 20 ms could lower peak disk throughput. On the other hand, a very high value (above 50 ms) might allow very high latency without any significant gain in overall throughput.
• A lower value will result in lower device latency and stronger virtual machine I/O performance isolation. Stronger isolation means that the shares controls are enforced more often. Lower device latency translates into lower I/O latency for the virtual machines with the highest shares, at the cost of higher I/O latency experienced by the virtual machines with fewer shares.
• If latency is more important, a very low value (lower than 20 ms) will result in lower device latency and better isolation among IOs at the cost of a decrease in aggregate datastore throughput.

Procedure

1. Select a datastore in the vSphere Client inventory and click the Configuration tab.
2. Click Properties.
3. Under Storage I/O Control, select the Enabled check box.
4. Click Advanced to edit the congestion threshold value for the datastore.
   The value must be between 10 and 100. You can click Reset to restore the congestion threshold setting to the default value (30 ms).
5. Click Close. 

For more information see the section "Managing Storage I/O Resources" on the "vSphere Resource Management Guide" (pdf)http://www.vmware.com/support/pubs/.

 

Source:-

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1019687

Sysprep file locations and versions

Symptoms

• When attempting to customize the deployment of a virtual machine, the radio buttons are disabled (grayed out).
• When a virtual machine is deployed from a template, the SID is always the same, despite the fact that you chose the option to generate a new SID during template deployment and guest operating system customization.
• When attempting to create a new virtual machine from a template in vCenter Server, you see the error:
  
  Warning: Windows customization resources were not found on this server
  
• In the guestcust.log file, you see the error:
   
  deploy doesn't contain known sysprep files
  
  
• When you ignore the errors, the virtual machine deployment does not get customized

Purpose

This issue may occur if Microsoft's Sysprep files are not found on the vCenter Server host, are not the correct version, or are not in the location they are expected.

 

This article guides you through the process of determining the correct version of Sysprep to use and the correct locations for these files.

Resolution

Microsoft has a different version of Sysprep for each release and service pack of Windows. You must use the version of Sysprep specific to the operating system you are deploying.The differences are not immediately visible in the packaging and documentation of the service packs, so it is necessary to manually investigate.

 

The contents of the Sysprep deploy.cab file must be extracted to the Sysprep Directory on the vCenter Server host. If the file downloaded from the Microsoft Web Site is a .cab file, the Installing the Microsoft Sysprep Tools in the Virtual Machine Administration guide, details how to install the Sysprep Tools.

 

If the file downloaded from the Microsoft Web Site is a .exe file, these additional steps must be executed to extract the files from the .exe:

1. Open a Windows command prompt. For more information, see Opening a command or shell prompt (1003892).
2. Change to the directory where the .exe file is saved.
3. Enter the name of the .exe file with the /x switch to extract the files. For example:
   
   WindowsServer2003-KB926028-v2-x86-ENU.exe /x
    
4. When prompted, choose a directory for the extracted files.
5. Browse the directory and double-click the deploy.cab file.
   
   Note: In some cases, the deploy.cab file may be located within one of the subfolders created in Step 3.
   
6. Select all the files, and copy them to the Sysprep Directory.

When the contents of the of the Sysprep deploy.cab file have been extracted to the Sysprep Directory on the vCenter Server:

1. Log in to the vCenter Server as an Administrator.
2. Click Start > Programs > Accessories > Windows Explorer.
3. Navigate to the Sysprep Directory as listed in the table below.
4. Right-click on the Sysprep .exe file and choose Properties.
5. Click the Version tab. Record the number at the top next to File Version:.

The table below lists the Sysprep version for the Windows versions that are supported for Image Customization. Compare the Sysprep version number with the Windows version for which it is intended:

Notes:

• If vCenter Server is installed on Windows Server 2008 and above, <directory_path> is%ALLUSERSPROFILE%\VMware\VMware VirtualCenter\Sysprepwhich generally translates toC:\ProgramData\VMware\VMware VirtualCenter\Sysprep by default.
  
  Note: C:\ProgramData may be a hidden folder.
  
• If vCenter Server is installed on any other Windows operating system, <directory_path> is%ALLUSERSPROFILE%\Application Data\VMware\VMware VirtualCenter\Sysprep\ which generally translates to C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\Sysprep\ by default.
• To check the SID of a server deployed from a template, you can use the PsGetSid. For more information, seehttp://technet.microsoft.com/en-us/sysinternals/bb897417.  

 

Windows Version	Sysprep Directory	Sysprep Version
Windows 2000 Server SP4 with Update Rollup 1 Download at http://www.microsoft.com/downloads/details.aspx?FamilyID=0c4bfb06-2824-4d2b-abc1-0e2223133afb Or The updated Deployment Tools are available in the Support\Tools\Deploy.cab file on the Windows 2000 SP4 CD-ROM. To download this file, visit the following Microsoft Web site: http://technet.microsoft.com/en-us/windowsserver/2000/bb735341.aspx	<directory_path>\2k	5.0.2195.2104
Windows XP Pro SP2 Download at http://www.microsoft.com/downloads/details.aspx?FamilyId=3E90DC91-AC56-4665-949B-BEDA3080E0F6	<directory_path>\xp	5.1.2600.2180
Windows 2003 Server SP1 Download at http://www.microsoft.com/downloads/details.aspx?familyid=A34EDCF2-EBFD-4F99-BBC4-E93154C332D6	<directory_path>\svr2003	5.2.3790.1830 (srv03_sp1_rtm.050324-1447)
Windows 2003 Server SP2 Download at http://www.microsoft.com/downloads/details.aspx?FamilyID=93f20bb1-97aa-4356-8b43-9584b7e72556	<directory_path>\svr2003	5.2.3790.3959 (srv03_sp2_rtm.070216-1710)
Windows 2003 Server R2 Download at http://www.microsoft.com/downloads/details.aspx?FamilyID=93f20bb1-97aa-4356-8b43-9584b7e72556&displaylang=en	<directory_path>\svr2003	5.2.3790.3959 (srv03_sp2_rtm.070216-1710)
Windows 2003 x64 Download athttp://www.microsoft.com/downloads/details.aspx?familyid=C2684C95-6864-4091-BC9A-52AEC5491AF7&displaylang=en	<directory_path>\svr2003-64	5.2.3790.3959 (srv03_sp2_rtm.070216-1710)
Windows XP x64 Download athttp://www.microsoft.com/downloads/details.aspx?familyid=C2684C95-6864-4091-BC9A-52AEC5491AF7&displaylang=en	<directory_path>\xp-64	5.2.3790.3959 (srv03_sp2_rtm.070216-1710)
Windows XP Pro SP3 Download at http://www.microsoft.com/downloads/details.aspx?familyid=673a1019-8e3e-4be0-ac31-70dd21b5afa7&displaylang=en	<directory_path>\xp	5.1.2600.5512
Windows Vista System Preparation tools are built into the Windows Vista operating system and do not have to be downloaded.	Not Applicable	Not Applicable
Windows Server 2008 System Preparation tools are built into the Windows Server 2008 operating system and do not have to be downloaded.	Not Applicable	Not Applicable
Windows Server 2008 R2 System Preparation tools are built into the Windows Server 2008 R2 operating system and do not have to be downloaded.	Not Applicable	Not Applicable
Windows 7 System Preparation tools are built into the Windows 7 operating system and do not have to be downloaded.	Not Applicable	Not Applicable
vCenter Server Virtual Appliance 5.0	/etc/vmware-vpx/sysprep/	Not Applicable

 

Source:-

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1005593

ESXI5 Installation Step by Step

Check the Minimum system requirements for installing ESX/ESXi from this KB article:-

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1003661

.VSWP File Size

In every VM .vswp file gets created as soon as you will power on the vm and it's deleted as soon as you will power off the vm. The size of this file depends upon two factors:-
1.)  Allocated Memory
2.)  Reserved Memory

Formula for calculating the .vswp file size:-
Allocated Memory - Reserved Memory

Let's Assume Allocated Memory is 1GB and Reserved Memory is also 1GB in this case the size of .vswp file will be 0. The size of this file will be automatically increased as soon as you reduce the reservation but it is not decreased as soon as you will increase the reservation of a vm memory. But if it is increased it is not reflected in the storage views tab it is only reflected by Browsing that datastore where vm file are stored. Check this attached Screenshot:-

HOW TO ENABLE FT IN NESTED VIRTUALIZATION

VMware Fault Tolerance fails to turn on in a two node cluster

 

Symptoms

• VMware Fault Tolerance (FT) is configured and functional for virtual machines in a two node cluster.
• FT is turned off so that the primary virtual machine can be vMotioned to a different host in the cluster.
• Attempts to turn on FT after vMotioning fail with the errors similar to:

The virtual machine is running in a monitor mode that is incompatible for Fault Tolerance.

The Fault Tolerance configuration of the entity <VM name> has an issue: The virtual machine's current configuration does not support Fault Tolerance.

Purpose

Running FT protected virtual machines in a two node cluster is supported. Problems can occur when there is a need to vMotion the primary virtual machine from one host to the other. As the primary and secondary virtual machines cannot reside on the same host, FT must be turned off so that the secondary virtual machines is destroyed. The primary virtual machines can then be vMotioned to the other host.

This article provides steps to prevent the monitor mode from changing during vMotioning that causes FT to fail.

Resolution

This issue occurs if the monitor mode changes during the vMotion process. FT requires the monitor mode be set to Use Intel VT-x/AMD-V for instruction set virtualization and software for MMU virtualization for the monitor mode to not change during the vMotion process.

The default setting for the virtual machine monitor mode is Automatic and FT sets the monitor mode appropriately behind the scenes. If the hosts in the cluster support the Use Intel VT-x/AMD-V for instruction set virtualization and Intel EPT/AMD RVI for MMU virtualization option, the monitor mode is changed to this during the vMotion process.

To set the monitor mode explicitly to Use Intel VT-x/AMD-V for instruction set virtualization and software for MMU virtualization:

Note: In some instances, the virtual machine needs to be powered off in order to change the monitor mode.

1. Right-click the virtual machine in question and choose Edit Settings.
2. In the virtual machine Properties window, click Options and select the CPU/MMU Virtualization option under the Advanced heading.
3. Select the radio button next to Use Intel VT-x/AMD-V for instruction set virtualization and software for MMU virtualization.
4. Click OK.
5. For the setting to take effect, the virtual machine needs to be power cycled or vMotioned to another host. When this is complete, FT can be turned on for the virtual machine.

Source :-

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2000589