Creating a persistent scratch location for ESXi

Symptoms

• When you run the command esxcli software vib list, you see an error similar to:
  
  Traceback (most recent call last):
    File "/usr/lib/vmware/esxcli-software", line 432, in <module>
      setupLogging()
    File "/usr/lib/vmware/esxcli-software", line 78, in setupLogging
      handler = logging.handlers.SysLogHandler(address='/dev/log')
    File "/build/mts/release/bora-504890/bora/build/esx/release/python-
      2.6-lib-zip-stage/504890/visor/pylib/python2.6/logging/handlers.py",
      line 704, in __init__
    File "/build/mts/release/bora-504890/bora/build/esx/release/python-
      2.6-lib-zip-stage/504890/visor/pylib/python2.6/logging/handlers.py",
      line 719, in _connect_unixsocket
    File "<string>", line 1, in connect
      socket.error: [Errno 2] No such file or directory
  
• None of the commands related to esxcli software or esxupdate work.
• Scanning the host using Update Manager fails with the error:
  
  General System Error
  
• Rebooting the host initially resolves the issue, but it reoccurs in a few days.
• Log files are missing in /var/log/ or are not being updated.
• You cannot install HA agent on affected ESXi hosts.
• ESXi is installed on a SD card or USB stick.

Purpose

This article describes the procedure to create a persistent scratch location for ESXi.

VMware recommends that ESXi has a persistent scratch location available for storing temporary data including logs, diagnostic information, and system swap. (This is not a requirement, however.) Persistent scratch space may be provisioned on a Fat16 or VMFS partition accessible by the ESXi host.

Cause

If persistent scratch space is not available, ESXi stores this temporary data on a ramdisk, which is constrained in space. This might be problematic in low-memory situations, but is not critical to the operation of ESXi. Information stored on a ramdisk does not persist across reboots, so troubleshooting information such as logs and core files could be lost. If a persistent scratch location on the host is not configured properly, there will be intermittent issues due to lack of space for temp files and the log files will not be updated.

Scratch space is configured automatically during installation or first boot of an ESXi host, and does not usually need to be manually configured. For more details, see the Additional Information section of this article.

Resolution

Scratch space can be manually configured for an ESXi host using the vSphere Client, the vCLI, PowerCLI, Tech Support Mode, or during scripted installation. Select the method appropriate to your environment. Regardless of the method selected, the configuration is written to the /etc/vmware/locker.conf configuration file for use during the next boot.

Changing the configured scratch location requires a reboot of the ESXi host for the changes to take effect. VMotion or power off all virtual machines on the ESXi host prior to making the change. For more information, see Rebooting an ESX Server host (1003530).

The configured scratch location must point to a directory on a filesystem accessible to the ESXi host, such as a Fat16 or VMFS volume. The location may be a directory on a shared or remote disk as long as the ESXi host is allocated its own separate directory.

 

Caution: Do not share a scratch locker directory among multiple ESXi hosts.

 

Note: It may necessary to do a reboot of the ESXi host before completing these procedures.

Configuring a persistent scratch location using the vSphere Client

You can configure persistent scratch space for ESXi using the vSphere Client:

1. Connect to vCenter Server or the ESXi host using the vSphere Client.
2. Select the ESXi host in the inventory.
3. Click the Configuration tab.
4. Click Storage.
5. Right-click a datastore and select Browse.
6. Create a uniquely-named directory for this ESX host (for example, .locker-ESXHostname)
7. Close the Datastore Browser.
8. Click Advanced Settings under Software. 
9. Select the ScratchConfig section.
10. Change the ScratchConfig.ConfiguredScratchLocation configuration option, specifying the full path to the directory.
    
    For example:
    
    /vmfs/volumes/DatastoreName/.locker-ESXHostname
    
11. Click OK.
12. Put the ESXi host in maintenance mode and reboot for the configuration change to take effect.

Configuring a persistent scratch location using the vCLI

You can configure persistent scratch space for ESXi using the vSphere Command-Line Interface. For more information on usage, see the vSphere Command-Line Interface Documentation.

1. Open a command prompt at the location where the vCLI is installed.
2. Obtain a list of datastores reachable from this ESXi host using the command:
   
   vifs.pl connectoptions --listds
   
   For example:
   
   vifs.pl --server esxhostnameorip --username root --listds
   
3. Create a uniquely-named directory for this ESXi host using the command:
   
   vifs.pl connectoptions --mkdir "[DatastoreName] DirectoryName"
   
   For example:
   
   vifs.pl --server esxhostnameorip --username root --mkdir "[Datastore1] .locker-ESXHostname"
   
4. Check the current value of the ScratchConfig.ConfiguredScratchLocation configuration option using the command:
   
   vicfg-advcfg.pl connectoptions -g ScratchConfig.ConfiguredScratchLocation
   
   For example:
   
   vicfg-advcfg.pl --server esxhostnameorip --username root -g ScratchConfig.ConfiguredScratchLocation
   Value of ScratchConfig.ConfiguredScratchLocation is /path/to/location
   
5. Change the ScratchConfig.ConfiguredScratchLocation configuration option, specifying the full path to the directory created in Step 3, using the command:
   
   vicfg-advcfg.pl connectoptions -s /vmfs/volumes/DatastoreName/DirectoryNameScratchConfig.ConfiguredScratchLocation
   
   For example:
   
   vicfg-advcfg.pl --server esxhostnameorip --username root -s /vmfs/volumes/Datastore1/.locker-ESXHostname ScratchConfig.ConfiguredScratchLocation 
   
6. Put the ESXi host into maintenance mode and reboot for the configuration change to take effect.

Configuring a persistent scratch location using PowerCLI

You can configure persistent scratch space for ESXi using the vSphere PowerCLI interface. For more information on usage, see the vSphere PowerCLI documentation.

 

Note: Before proceeding, ensure that /tmp/scratch exists. If it does not exist, use the command mkdir /tmp/scratch to create it.

 

1. Open a command prompt where the PowerCLI is installed.
2. Connect to the ESXi host using the command:
   
   connect-viserver ESXHostnameOrIP
   
3. Obtain a list of datastores reachable from this ESXi host using the command:
   
   Get-Datastore
   
4. Mount a datastore read/write as a PSDrive using the command:
   
   New-PSDrive -Name "mounteddatastore" -Root \ -PSProvider VimDatastore -Datastore (Get-Datastore "DatastoreName")
   
5. Access the new PSDrive using the command:
   
   Set-Location mounteddatastore:
   
6. Create a uniquely-named directory for this ESXi host using the command:
   
   New-Item "DirectoryName" -ItemType directory
   
   For example:
   
   New-Item ".locker-ESXHostname" -ItemType directory
   
7. Check the current value of the ScratchConfig.ConfiguredScratchLocation configuration option using the command:
   
   Get-VMHostAdvancedConfiguration -Name "ScratchConfig.ConfiguredScratchLocation"
   
8. Change the ScratchConfig.ConfiguredScratchLocation configuration option, specifying the full path to the directory created in Step 6, using the command:
   
   Set-VMHostAdvancedConfiguration -Name "ScratchConfig.ConfiguredScratchLocation" -Value "/vmfs/volumes/DatastoreName/DirectoryName"
   
   For example:
   
   Set-VMHostAdvancedConfiguration -Name "ScratchConfig.ConfiguredScratchLocation" -Value "/vmfs/volumes/Datastore1/.locker-ESXHostname"

9. Put the ESXi host into maintenance mode and reboot for the configuration change to take effect.

Configuring a persistent scratch location using Tech Support Mode

You can configure persistent scratch space for ESXi using Tech Support Mode on ESXi, either via the local console or SSH.

Note: A .filename is a hidden file. To view hidden files, use the ls -a command.

1. Open a console to the ESXi host. For more information, see Using Tech Support Mode in ESXi 4.1 and 5.0 (1017910).
2. Obtain a list of datastores reachable from this ESXi host using the command:
   
   ls /vmfs/volumes
   
3. Create a uniquely-named directory for this ESXi host using the command:
   
   mkdir /vmfs/volumes/DatastoreName/DirectoryName
   
   For example:
   
   mkdir /vmfs/volumes/Datastore1/.locker-ESXHostname
   
4. Check the current value of the ScratchConfig.ConfiguredScratchLocation configuration option using the command:
   
   vim-cmd hostsvc/advopt/view ScratchConfig.ConfiguredScratchLocation
   
   Example output:
   
   (vim.option.OptionValue) [
   (vim.option.OptionValue) {
      dynamicType = <unset>,
      key = "ScratchConfig.ConfiguredScratchLocation",
      value = "/path/to/location",
   }
}
   
5. Change the ScratchConfig.ConfiguredScratchLocation configuration option, specifying the full path to the directory created in Step 3, using the command:

vim-cmd hostsvc/advopt/update ScratchConfig.ConfiguredScratchLocation string /vmfs/volumes/DatastoreName/DirectoryName
   
   For example:
   
   /bin/vim-cmd hostsvc/advopt/update ScratchConfig.ConfiguredScratchLocation string /vmfs/volumes/Datastore1/.locker-ESXHostname
   
6. Put the ESXi host into maintenance mode and reboot for the configuration change to take effect.

Configuring a persistent scratch location during scripted install using kickstart scripts

You can configure persistent scratch space for ESXi during kickstart installation by adding commands to your kickstart scripts which create the directory and change the configuration option. For more information on ESXi scripted deployments, see the ESXi Installable and vCenter Server Setup Guide.

Append these commands to your custom kickstart script:

# Generate a new scratch directory for this host on a Datastore
scratchdirectory=/vmfs/volumes/DatastoreName/.locker-$(hostname 2> /dev/null)-$(esxcfg-info -b 2> /dev/null)

# Create the scratch directory
mkdir -p $scratchdirectory

# Change the advanced configuration option
vim-cmd hostsvc/advopt/update ScratchConfig.ConfiguredScratchLocation string $scratchdirectory

Note: The configured scratch location is activated during startup of the ESXi host. A reboot is required after performing the preceding commands.

Additional Information

Further information on the scratch partition can be found in the Configuring Storage Behavior section of the ESXi Setup Guide for your version of ESXi.

Scratch space is configured automatically during installation or first boot of an ESXi host, and does not usually need to be manually configured. ESXi Installable creates a 4 GB Fat16 partition on the target device during installation if there is sufficient space, and if the device is considered Local.

 

If the host has been running for more than 24 hours, this setting may not stay persistent after performing the preceding steps. In this case, perform these steps:

1. Place the host into Maintenance Mode and reboot the host.
2. Click the Configuration tab.
3. Click Software.
4. Click Advanced Settings.
5. Select the ScratchConfig section.
6. Change the ScratchConfig.ConfiguredScratchLocation configuration option and set it to the full path of the directory.
   
   For example:
   
   /vmfs/volumes/DatastoreName/.locker-ESXHostname
   
7. Click OK.
8. Close the window.
9. Click Advanced Settings to reopen the window.
10. Select the ScratchConfig section.
11. Verify that the ScratchConfig.ConfiguredScratchLocation field contains the scratch setting that you configured in Step 6. If it does, reboot the host.
    
    The host is now using the configured scratch partition.

For more information on determining whether a disk device is considered Local, see Identifying disks when working with VMware ESX (1014953), and review the Is Local flag in the output of the esxcfg-scsidevs -l command. If Local is true, the storage device is not sharable among multiple ESXi hosts. If Local is false, the storage device is remote and potentially sharable among several ESXi hosts. Remote devices are not automatically used for scratch by ESXi.

ESXi selects one of these scratch locations during startup in this order of preference:

1. The location configured in the /etc/vmware/locker.conf configuration file, set by theScratchConfig.ConfiguredScratchLocation configuration option, as in this article.
2. A Fat16 filesystem of at least 4 GB on the Local Boot device.
3. A Fat16 filesystem of at least 4 GB on a Local device.
4. A VMFS Datastore on a Local device, in a .locker/ directory.
5. A ramdisk at /tmp/scratch/

Here are two examples where scratch space may not be automatically defined on persistent storage. In each case, the temporary scratch location will be configured on a ramdisk:

1. ESXi deployed on a Flash or SD device, including a USB key. Scratch partitions are not created on Flash or SD storage devices even if connected during install, due to the potentially limited read/write cycles available.
   
2. ESXi deployed in a Boot from SAN configuration or to a SAS device. A Boot from SAN or SAS LUN is considered Remote, and could potentially be shared among multiple ESXi hosts. Remote devices are not used for scratch to avoid collisions between multiple ESXi hosts.

Source:-

http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1033696