VMware vRealize Automation 6.0.x tenants are inaccessible and identity stores disappear (2075011)

Symptoms

In vRealize Automation (formerly known as vCloud Automation Center) 6.0.x, 90 days after deployment of a template you experience issues similar to:

• When attempting to log in to tenant, a blank page is displayed with a Submit button in the upper left corner.
• You receive a System Exception error when accessing the tenant identity store configuration page and the identity store configuration has disappeared.
• Cannot log in to a tenant using an LDAP account.
• Unable to add a new identity store configuration to the affected tenant.
• The tenant identity store disappears from the SSO Administrator login.
• In the catalina.out log file, located at /var/log/vmware/vcac/, you see entries similar to:
  
  12:40:49,190 [tomcat-http--34] [authentication] INFO com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.handleFaultCondition:922 - Failed trying to retrieve token: ns0:RequestFailed: Error occured looking for solution user :: Insufficient access
YYYY-03-18 12:40:49,201 [tomcat-http--34] [authentication] ERROR com.vmware.vcac.platform.service.rest.resolver.ApplicationExceptionHandler.handleUnexpectedException:820 - Failed trying to retrieve token: ns0:RequestFailed: Error occured looking for solution user :: Insufficient access
com.vmware.vim.sso.client.exception.InternalError: Failed trying to retrieve token: ns0:RequestFailed: Error occured looking for solution user :: Insufficient access
  
• In the messages.log file on the Identity Appliance, located at /var/log/, you see entries similar to:
  
  T16:50:18-05:00 lsassd[2913]: GSSAPI Error: The referenced context has expired (Unknown error)
T08:34:41-06:00 vmdird: t@139870073485056: Lockout policy check - password expired. (cn=tenantadmin,cn=users,dc=tenant)
T11:58:03-06:00 lsassd[2943]: GSSAPI Error: The referenced context has expired (Unknown error)
.....

Account "cn=tenantadmin,cn=users,dc=qic" password expired and caused login/bind from IDM to fail.
YYYY-03-18T11:38:46-06:00 denqca3vcacid01 vmdird: t@140689332778752: LoginBlocked DN (cn=tenantadmin,cn=users,dc=tenant), error (9239)(Account access blocked)

Cause

 This issue occurs due to expiry of password after 90 days. 

 

By default, the SSO internal tenant admin password expires in 90 days. This password expiration value can be changed or password expiration disabled. After the password expires, the authentication server cannot log in with the old password.

Note: This issue is internal to vRealize related SSO authentication and does not affect external OpenLDAP, Active Directory, or other LDAP configurations.

The current workflow User Interface does not provide any notification when the password is to expire.

Resolution

This issue is resolved in vRealize Automation 6.1 available at VMware Downloads. For more information, refer to the VMware vCloud Automation Center 6.1 Release Notes.

Note: This issue can persist if an in-place upgrade is done from vRealize Automation (formerly known as vCloud Automation Center) 6.0 to vRealize Automation 6.1. If this is the case, re-apply the workaround mentioned below.

To work around this issue, disable password expiration for the tenant admin account.

Note: If you are not sure about performing the steps below, file a support request with VMware Technical Support and note this Knowledge Base article ID (2075011) in the problem description. For more information on filing a Support Request, see Filing a Support Request in My VMware (2006985).

VMware vRealize Automation (formerly known as vCloud Automation Center) using vCenter Single Sign-On (SSO) for tenant authentication

If VMware vRealize Automation is using vCenter Single Sign-On (SSO) for tenant authentication, perform these steps to disable password expiration:

Note: Replace tenant_name with the URL name of your tenant.

1. Open an SSH connection to vCenter Server.
2. Disable password expiration by running this command:
   
   /opt/likewise/bin/ldapmodify -H ldap://localhost:11711 -x -D "cn=administrator,cn=users,dc=vsphere,dc=local" -W <<EOF
dn: cn=DCAdmins,cn=builtin,dc=vsphere,dc=local
changetype: modify
add: member
member: cn=administrator,cn=users,dc=tenant_name
EOF

Response: modifying entry "cn=DCAdmins,cn=builtin,dc=vsphere,dc=local"

Note: You are prompted for the administrator@vsphere.local password when running this command.
3. Run this command to reset the account control flag:
   
   /opt/likewise/bin/ldapmodify -H ldap://localhost:11711 -x -D "cn=administrator,cn=users,dc=vsphere,dc=local" -W <<EOF
dn: cn=administrator,cn=users,dc=tenant_name
changetype: modify
replace: userAccountControl
userAccountControl: 0
EOF

Response: modifying entry "cn=administrator,cn=users,dc=tenant_name."
   
   Note: You are prompted for the administrator@vsphere.local password when running this command.

Using SSO in the VMware vRealize Automation (formerly known as vCloud Automation Center) Identity Appliance

If you are using SSO in the VMware vRealize Automation Identity Appliance, perform these steps to disable password expiration:

1. Download the attached file, kb_2075011_identity_appliance.tar.gz , to your workstation
2. Using an SCP client, upload kb_2075011_identity_appliance.tar.gz  to /tmp of the VMware vRealize Automation Identity Appliance
3. Open an SSH connection to the VMware vRealize Automation Identity Appliance.
4. Navigate to the temp directory on the Identity Appliance by running cd /tmp command.
5. Extract the contents of kb_2075011_identity_appliance.tar.gz into /tmp by running this command:
   
   tar zxvf kb_2075011_identity_appliance.tar.gz
   
6. After contents are extracted, execute 0_run_me script on the Identity Appliance :
   
   ./0_run_me tenant_name
   
   Note: Replace tenant_name with the URL name of your tenant. Use this command as a model:
   
   
   ./0_run_me vmware
   
7. When prompted, enter the password for Administrator@vsphere.local.

VMware vRealize Automation (formerly known as vCloud Automation Center) using Windows installation of vCenter SSO for tenant authentication

If VMware vRealize Automation is using a Windows installation of vCenter Single Sign-On for tenant authentication, perform these steps to disable password expiration:

1. Open an elevated command prompt.
2. Create a temporary directory by running the command:
   
   mkdir c:\temp
   
3. Change directories by running the command:
   
   cd c:\temp
   
4. Create the UserAccountControl.ldif file in Notepad by running the command:
   
   notepad UserAccountControl.ldif
   
5. Copy and paste the content below into the file:
   
   dn: cn=tenantadmin,cn=users,dc=tenant_name
   changetype: modify
   replace: userAccountControl
   userAccountControl: 0 
-
   
   Notes:
6. • Replace tenant_name with the URL name of your tenant.
   • Ensure you include the hyphen on the last line. (In other words, do not omit the hyphen.)
     
7. Save and close the UserAccountControl.ldif file.
8. Create the PasswordExpiration.ldif file in Notepad by running the command:
   
   notepad PasswordExpiration.ldif
   
9. Copy and paste the content below into the file:
   
   dn: cn=DCAdmins,cn=builtin,dc=vsphere,dc=local
   changetype: modify
   add: member
   member: cn=tenantadmin,cn=users,dc=tenant_name
   -
   
   Notes:
   
   • Replace tenant_name with the URL name of your tenant.
   • Ensure you include the hyphen on the last line. (In other words, do not omit the hyphen.)
10. Save and close the PasswordExpiration.ldif file.
11. To modify the user account control configuration and password expiration using the files created earlier in this procedure, run these commands:
    
    Note: If the ldifde executable is not available, run this command to install:
    
    ServerManagerCmd -i RSAT-ADDS-Tools

    Note: ServerManagerCmd has been deprecated, and is not available in Windows Server 2012. For more information, see Microsoft TechNet.
    
    The preceding link was correct as of September 18, 2014. If you find the link is broken, provide feedback and a VMware employee will update the link.
    
    
    
    1. To modify the password expiration, run the command:
       
       ldifde -i -f PasswordExpiration.ldif -s localhost -t 11711 -a "cn=Administrator,cn=Users,dc=vsphere,dc=local" *
       
       When prompted, enter the password for Administrator@vsphere.local.
    2. To modify the user account control configuration, run the command:
       
       ldifde -i -f UserAccountControl.ldif -s localhost -t 11711 -a "cn=Administrator,cn=Users,dc=vsphere,dc=local" *
       
       When prompted, enter the password for Administrator@vsphere.local.

Impact/Risks

Current work around resets the userAccountControl flag and also modifies the values that determine the length of time for the SSO tenant admin account password to expire. The workaround sets that internal service account password to never expire.

Additional Information

To be alerted when this article is updated, click Subscribe to Document in the Actions box.

See Also

VMware vRealize Automation 6.0.x テナントにアクセスできず ID ストアが消える (2094293)
VMware vRealize Automation 6.0.x 租户不可访问且身份存储消失 (2096659)

Attachments

• kb_2075011_identity_appliance.tar.gz

Source:-
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2075011&src=vmw_so_vex_ragga_1012