PowerCLI - Fails to logon to vCenter VCSA using integrated authentication

I recently made a PowerCLI script to pull reports from 4 vCenters we got in our infra. Script was tested against once vCenter and was working great. So deployed it in out script host to be able to schedule and also configured to run it against all 4 vCenters. Then noticed it's getting stuck when it's at lab vCenter and pops up asking for credentials, which I was not expecting as it was suppose to run it using integrated authentication. Works fine against 3 remaining vCenters when I tried.

I was able to reproduce this issue by 

1) Opened PowerCli as service account that we are using to pull report.

2) tried to connect to Prod vCenter "Connect-viserver ProdvCenter" and it logged in immediately. 

3) "Connect-viserver LabvCenter" brought up credential popups again

 

4) To troubleshoot, I issued connect command again with verberos switch (-v) and cancelled the credential window. Below is the output from it.  

PS C:\Users\vCAdmin> Connect-VIServer -v VMwareLab

VERBOSE: Attempting to connect using SSPI

VERBOSE: Reversely resolved 'vmwarelab' to 'vmwaretest'

VERBOSE: SSPI Kerberos: Acquired credentials for user 'OurDomain\vCAdmin'

VERBOSE: SSPI Kerberos: InitializeSecurityContext failed for target 'host/VMwareLab'. Error code: 0x80090303

VERBOSE: Connect using SSPI was unsuccessful

Connect-VIServer : 2/08/2021 4:30:04 PM Connect-VIServer                Could not determine user name and/or password for server VMwareLab

At line:1 char:1

+ Connect-VIServer -v VMwareLab

+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : NotSpecified: (:) [Connect-VIServer], ViServerConnectionException

    + FullyQualifiedErrorId : ViCore_Login_CredentialNotFound,VMware.VimAutomation.ViCore.Cmdlets.Commands.ConnectVIServer

After comparting the configs, I found the issue and line highlighted in red above is the key. 

1) ProdVC DNS entry is "ProdVCenter.OurDomain.com" and it's joined to AD with same hostname.

2) LabvCenter DNS entry is "vmwarelab.OurDomain.com" and this is what I am trying to connect. But VC is joined to AD with another name vmwaretest.OurDomain.com

Note -: you may need to login as administrator@vsphere.local to be able to view AD Domain section.

So I could just use vmwaretest.OurDomain.com in my script.

If at all you want to use the other name, you can still do that by creating a SPN to connect the host name with AD object. 

In my case I used below command to do this

setspn -A "HOST/vmwaretest.OurDomain.com" vmwarelab

which is 

setspn -A "HOST/<FQDN of vCenter as per Active Directory Domain config of VC>" <alternate name you want to use>

After this , just restart your vCenter which would also allow some time for AD replication,

Thanks!

vCenter SSO User password Expired

We had a vCenter SSO user created for SRM and it's password expired. Here is how you can check it and fix it.

User name is srm@vsphere.local

1) Login to VCSA with SSH and below are commands

root@vcenterserver [ ] cd /usr/lib/vmware-vmafd/bin/

root@vcenterserver [ /usr/lib/vmware-vmafd/bin ]# ./dir-cli user find-by-name --account srm --level 2
Enter password for administrator@vsphere.local:
Account: srm
UPN: srm@VSPHERE.LOCAL
Account disabled: FALSE
Account locked: FALSE
Password never expires: FALSE
Password expired: TRUE

root@vcenterserver [ /usr/lib/vmware-vmafd/bin ]#./dir-cli user modify --account srm  --password-never-expires
Enter password for administrator@vsphere.local:
Password set to never expire for [srm].

root@vcenterserver [ /usr/lib/vmware-vmafd/bin ]#./dir-cli password reset --account srm --password XXXXXXXX 

applmgmt service wont start on PSC Appliace post converge operation

Scenario

We had a vCenter with External PSC. We converged them and converge job was successful execpt a cert warning.
After a week we tried to decommission the old PSC appliance and found that the status is shown in WebClient as "Unknown"

Up on checking we found applmgmt in stopped state. Tried to start it but it failed with below error

[ ~ ]# service-control --status
Running:
 lwsmd pschealth vmafdd vmcad vmdird vmdnsd vmonapi vmware-analytics vmware-certificatemanagement vmware-cis-license vmware-cm vmware-rhttpproxy vmware-sca vmware-sts-idmd vmware-stsd vmware-vapi-endpoint vmware-vmon
Stopped:
 applmgmt vmware-statsmonitor

[ ~ ]# service-control --start applmgmt
Operation not cancellable. Please wait for it to finish...
Performing start operation on service applmgmt...
Error executing start on service applmgmt. Details {
    "detail": [
        {
            "translatable": "An error occurred while starting service '%(0)s'",
            "id": "install.ciscommon.service.failstart",
            "args": [
                "applmgmt"
            ],
            "localized": "An error occurred while starting service 'applmgmt'"
        }
    ],
    "componentKey": null,
    "resolution": null,
    "problemId": null
}
Service-control failed. Error: {
    "detail": [
        {
            "translatable": "An error occurred while starting service '%(0)s'",
            "id": "install.ciscommon.service.failstart",
            "args": [
                "applmgmt"
            ],
            "localized": "An error occurred while starting service 'applmgmt'"
        }
    ],
    "componentKey": null,
    "resolution": null,
    "problemId": null
}


We has this issue on two infrastructures and we could fix it one

FIX that worked on first PSC

# List all disabled services for removal.  
find /etc/systemd/system/ -lname '/dev/null' -exec ls {} \;   
 
# Automatically remove them (or rm each file) 
find /etc/systemd/system/ -lname '/dev/null' -exec rm {} \;  
 
# Relaod systemctl daemon 
systemctl daemon-reload  
 
# Start services or Reboot 
service-control --start --all  

However second PSC was not happy still. So we had to manfully remove the replication manually

Manual Removal of the replication

1) Shutdown both PSC and vCenters and take an offline snap
2) Power on only vCenter. Do not start PSC
3) SSH to vCenter and run below commands

a) List all PSCs connected
]# ./vdcrepadmin -f showservers -h localhost -u administrator -w XXXX
cn=oldpscappliance.mydomain.com,cn=Servers,cn=Sites,cn=Configuration,dc=vsphere,dc=local
cn=vcenter.mydomain.com,cn=Servers,cn=Sites,cn=Configuration,dc=vsphere,dc=local

Note -- XXXX is the SSO password for administrator@vsphere.local 

I can now see two, old PSC appliance and also the vCenter with PSC converged in to it.
Ran below command to make sure vCenter is pointing to converged PSC and not the old appliance

]# /usr/lib/vmware-vmafd/bin/vmafd-cli get-ls-location --server-name localhost
https://vcenter.mydomain.com:443/lookupservice/sdk

Output confirmed that the PSC appliance is not in use. So decided to manually remove the association.

# /bin/cmsso-util unregister --node-pnid oldpscappliance.mydomain.com --username administrator --passwd XXXX

Watch theoutput basically ends like this

2019-11-12T08:29:24.939Z  Running command: ['/usr/lib/vmware-vmafd/bin/dir-cli', 'service', 'list', '--login', 'administrator']
2019-11-12T08:29:25.059Z  Done running command
Stopping all the services ...
All services stopped.
Starting all the services ...
Started all the services.
Success
2019-11-12T08:33:13.071Z  Running command: ['/usr/bin/sed', '-i', '-e', 's/cmsso-util.*/cmsso-util/g', '/var/log/vmware/procstate']

2019-11-12T08:33:13.829Z  Done running command

Login to the vCenter via WebClient and under Administration ->  System Configuration makesure that the old PSC is listed anymore.

You may keep the old PSC appliance for a few days and delete it once it's all good.