ESXi/ESX host disconnects from vCenter Server after adding or connecting it to the inventory (2040630)

Symptoms

• An ESXi/ESX host disconnects from vCenter Server.
• After adding or reconnecting an ESXi/ESX to the vCenter Server inventory, it disconnects 30 to 90 seconds after the task completes.
• Changing the uplink switch port VLAN information for the new IP of the ESXi host before changing the IP of the ESXi host results in the host showing as disconnected in vCenter Server.
• Changing the IP address of the ESXi host using the DCUI without first removing the host from the vCenter Server inventory results in the host showing up as disconnected in vCenter Server.

Purpose

This article helps identify issues with heartbeat traffic between vCenter Server and ESXi/ESX hosts.

The steps provided in this article and the included Knowledge Base article links help determine if heartbeat packets are being dropped or lost.

Cause

This issue is caused by dropped, blocked, or lost heartbeat packets between the vCenter Server and the ESXi/ESX host.

If there is an incorrect configuration of the vCenter Server managed IP address, the host receives the heartbeat from vCenter Server but cannot return it.

It is important to remember that the default heartbeat port is UDP 902, and these packets must be sent between vCenter Server and the ESXi/ESX host for the host to stay connected and remain in the vCenter Server inventory.

If the host is not connecting to vCenter Server at all, the cause may be another network issue.

Resolution

To troubleshoot this issue, ensure that bidirectional heartbeat communications are functioning correctly.

The default port for this communication is UDP 902, but be sure to verify the configured port in the vpxa.cfg file on the host. This file also defines the IP address which manages the host.

Confirm vCenter Server managed IP address

Confirm the vCenter Server managed IP address continuity throughout the environment.

1. Determine the managed IP address of the vCenter Server:
   
   
   1. Connect to the vCenter Server with the vSphere Client.
   2. Go to Administration > vCenter Server Settings > Advanced Settings.
   3. Make note of the IP address in the ManagedIP row.
      
2. Determine the IP address configured for vCenter Server:
   
   
   1. From a console or RDP session to the vCenter Server desktop, open a command prompt.
   2. Run the command:
      
      ipconfig
      
   3. Make note of the IP address and ensure that it matches the managed IP address found in step 1.
      
3. Determine the IP address and port that the ESXi/ESX host is using for heartbeat traffic:
   
   
   1. Connect to the ESXi/ESX host using the vSphere Client.
   2. When the Warning pop-up appears, make note of the IP address which is managing the host.
   3. Connect to the same host using SSH.
   4. Check the vpxa.cfg file for the heartbeat traffic port by running the command:
      
      
      • On ESXi 5.x:
        
        grep -i serverport /etc/vmware/vpxa/vpxa.cfg
        
      • On ESXi/ESX 4.1 and earlier:
        
        grep -i serverport /etc/opt/vmware/vpxa/vpxa.cfg
        
   5. Ensure that the port number matches the default heartbeat port of 902.
   6. Check the vpxa.cfg file for the managed IP address by running the command:
      
      
      • On ESXi 5.x:
        
        grep -i serverIp /etc/vmware/vpxa/vpxa.cfg
        
      • On ESXi/ESX 4.1 and earlier:
        
        grep -i serverIp /etc/opt/vmware/vpxa/vpxa.cfg
        
   7. Ensure that the IP address matches the managed IP address found in step 1.

Connectivity

Test connectivity between vCenter Server and the ESXi/ESX host via the heartbeat network.

Using telnet and netcat, test connectivity over the heartbeat network via the configured heartbeat port (default 902). For more information, see:

• Troubleshooting network and TCP/UDP port connectivity issues on ESXi/ESX (2020669)
• TCP and UDP Ports required to access vCenter Server, ESXi/ESX hosts, and other network components (1012382)

Congestion

Test network congestion:

• To test whether UDP heartbeat packets are being received by vCenter Server within the 60-second timeout period, seeESXi/ESX host disconnects from vCenter Server 60 seconds after connecting (1029919).
• If packets are delayed due to network congestion, see ESXi/ESX host keeps disconnecting and reconnecting when heartbeats are not received by vCenter Server (1005757).
• If NAT is configured, see ESXi hosts connected to vCenter Server using NAT disconnect after 30 to 60 seconds (2019865).

Other troubleshooting areas

• Check the NIC firmware and driver levels on the affected host and compare these with other hosts. For more information, see Determining Network/Storage firmware and driver version in ESXi/ESX 4.x and 5.x (1027206).
• Check the Address Resolution Protocol and Neighbor Discovery cache for inconsistencies. For more information, seeDisplaying the ARP and Neighbor Discovery cache for VMkernel network interfaces (2000130).
• Capture a network trace and analyze for inconsistency. For more information, see Capturing a network trace in ESXi using Tech Support Mode or ESXi Shell (1031186).

Additional Information

For more information on the vpxa.cfg file, see ESXi 5.0 hosts are marked as Not Responding 60 seconds after being added to vCenter Server (2020100).

For more information on similar issues, see:

• vCenter Server IP address change causes ESX hosts to disconnect (1001493)
• Verifying the vCenter Server Managed IP Address (1008030)

Source:-

http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=2040630