ESXi 5.0 hosts are marked as Not Responding 60 seconds after being added to vCenter Server (2020100)

Symptoms

• When an ESXi 5.0 host is added to vCenter Server 5.0, the host is added correctly but approximately 60 seconds later it is marked asNot Responding.
• Disabling the ESXi 5.0 firewall allows the host to connect.
• When ESXi 3.x and 4.x hosts are added, they do not exhibit the same behavior.

Cause

The ESXi host sends UDP heartbeats to the vCenter Server, and by default this traffic is sent over port 902. There is also a rule in the ESXi firewall to allow for vCenter Server heartbeat traffic. If the vCenter Server has been configured to send traffic over an alternate port, that traffic will be blocked.

Resolution

There are two methods to resolve this issue:

• Method 1: Add a firewall rule to the ESXi host to allow traffic on the alternate heartbeat port
• Method 2: Configure vCenter Server to use the default port number 902

Note: Before beginning the resolution steps, determine which port vCenter Server is currently using to send traffic.

To determine the traffic port in use:

1. Connect to the ESXi 5.0 host using SSH. For more information, see Using ESXi Shell in ESXi 5.0 and 5.1 (2004746).
2. Use the less or grep command to to determine the port in use:
   
   less /etc/vmware/vpxa/vpxa.cfg
   
   or:
   
   grep serverPort /etc/vmware/vpxa/vpxa.cfg
   
   The port number in use is contained in the serverPort tags. For example:
   
   <vpxa>
<bundleVersion>1000000</bundleVersion>
<datastorePrincipal>root</datastorePrincipal>
<hostIp>xxx.xxx.xxx.xxx</hostIp>
<hostKey>52db3386-b766-889a-d778-da0c8851c81e</hostKey>
<hostPort>443</hostPort>
<licenseExpiryNotificationThreshold>15</licenseExpiryNotificationThreshold>
<memoryCheckerTimeInSecs>30</memoryCheckerTimeInSecs>
<serverIp>xxx.xxx.xxx.xxx</serverIp>
<serverPort> 9020</serverPort>
</vpxa>
   
   In this example, serverPort is set to 9020, instead of the default port number 902.

Method 1: Add a firewall rule to the ESXi host to allow traffic on the alternate heartbeat port

To add a firewall rule to the ESXi host:

1. Connect to the ESXi 5.0 host using SSH. For more information, see Using ESXi Shell in ESXi 5.0 and 5.1 (2004746).
2. Navigate to the /etc/vmware/firewall/ directory:
   
   cd /etc/vmware/firewall/
3. Create and edit a new file named heartbeat.xml using the vi command:
   
   vi heartbeat.xml
4. Enter the configuration info into the file as shown in this example:
   
   <!-- Firewall configuration sample -->
<ConfigRoot>
<service>
<id>nondefheartbeat</id>
<rule id='0000'>
<direction>inbound</direction>
<protocol>udp</protocol>
<porttype>dst</porttype>
<port> 9020</port>
</rule>
<rule id='0001'>
<direction>outbound</direction>
<protocol>udp</protocol>
<porttype>dst</porttype>
<port> 9020</port>
</rule>
<enabled>false</enabled>
<required>false</required>
</service>
</ConfigRoot>
   
   Note: The alternate port number used in this example is 9020. Be sure to use the port number you determined earlier for your configuration.
5. Save and close the file.
6. Enable the new firewall rule by running these commands:
   
   esxcli network firewall unload
esxcli network firewall load
esxcli network firewall refresh

Method 2: Configure vCenter Server to use the default port number 902

Note s:

• Ensure that the heartbeat firewall rule is also set to default port 902 prior to changing the port for vCenter Server.
• Before changing the port number back to the default 902, ensure that no other application installed on vCenter Server is using this port.
• This procedure modifies the Windows registry. Before making any registry modifications, ensure that you have a current and valid backup of the registry and the virtual machine. For more information on backing up and restoring the registry, see the Microsoft Knowledge Base article 136393.
  
  The preceding link was correct as of April 16, 2013. If you find the link is broken, provide feedback and a VMware employee will update the link.

To configure vCenter Server to use port number 902:

1. Stop the VMware VirtualCenter Server service. For more information, see Stopping, starting, or restarting vCenter services (1003895).
2. Click Start > Run, type regedit, and click OK. The Registry Editor window opens.
3. Navigate to:
   
   HKEY_LOCAL_MACHINE\SOFTWARE\VMware, Inc.\VMware VirtualCenter
4. Modify the registry key heartbeatport and change the value to 902.
5. Change the windows firewall to accept port 902: 
   
   
   1. Navigate to Windows Firewall > Allow a program or feature through Windows firewall.
   2. Select VMware vCenter Server > Host Heartbeat.
   3. Click Details and change the port to 902 . 
6. Start the VMware VirtualCenter Server service.
   
   Note: The ESXi host may show as disconnected. Reconnect the host so that the new configuration info is saved to the vpxa.cfg file.

Source:-
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2020100

&src=vmw_so_vex_ragga_1012