PowerCLI - Fails to logon to vCenter VCSA using integrated authentication

I recently made a PowerCLI script to pull reports from 4 vCenters we got in our infra. Script was tested against once vCenter and was working great. So deployed it in out script host to be able to schedule and also configured to run it against all 4 vCenters. Then noticed it's getting stuck when it's at lab vCenter and pops up asking for credentials, which I was not expecting as it was suppose to run it using integrated authentication. Works fine against 3 remaining vCenters when I tried.

I was able to reproduce this issue by 

1) Opened PowerCli as service account that we are using to pull report.

2) tried to connect to Prod vCenter "Connect-viserver ProdvCenter" and it logged in immediately. 

3) "Connect-viserver LabvCenter" brought up credential popups again

 

4) To troubleshoot, I issued connect command again with verberos switch (-v) and cancelled the credential window. Below is the output from it.  

PS C:\Users\vCAdmin> Connect-VIServer -v VMwareLab

VERBOSE: Attempting to connect using SSPI

VERBOSE: Reversely resolved 'vmwarelab' to 'vmwaretest'

VERBOSE: SSPI Kerberos: Acquired credentials for user 'OurDomain\vCAdmin'

VERBOSE: SSPI Kerberos: InitializeSecurityContext failed for target 'host/VMwareLab'. Error code: 0x80090303

VERBOSE: Connect using SSPI was unsuccessful

Connect-VIServer : 2/08/2021 4:30:04 PM Connect-VIServer                Could not determine user name and/or password for server VMwareLab

At line:1 char:1

+ Connect-VIServer -v VMwareLab

+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : NotSpecified: (:) [Connect-VIServer], ViServerConnectionException

    + FullyQualifiedErrorId : ViCore_Login_CredentialNotFound,VMware.VimAutomation.ViCore.Cmdlets.Commands.ConnectVIServer

After comparting the configs, I found the issue and line highlighted in red above is the key. 

1) ProdVC DNS entry is "ProdVCenter.OurDomain.com" and it's joined to AD with same hostname.

2) LabvCenter DNS entry is "vmwarelab.OurDomain.com" and this is what I am trying to connect. But VC is joined to AD with another name vmwaretest.OurDomain.com

Note -: you may need to login as administrator@vsphere.local to be able to view AD Domain section.

So I could just use vmwaretest.OurDomain.com in my script.

If at all you want to use the other name, you can still do that by creating a SPN to connect the host name with AD object. 

In my case I used below command to do this

setspn -A "HOST/vmwaretest.OurDomain.com" vmwarelab

which is 

setspn -A "HOST/<FQDN of vCenter as per Active Directory Domain config of VC>" <alternate name you want to use>

After this , just restart your vCenter which would also allow some time for AD replication,

Thanks!

Horizon View - "Failed to connect to Connection Server" when accessed via LB WIP or DNS alias

Scenario

Horizon View - "Failed to connect to Connection Server" when accessed via LB WIP or DNS alias

Works fine when accessed with server FQDN

Solution

If you are facing this issue after upgrading to view 7.X, you are not alone! And this is not an issue.

It's a new security feature part of 7.X and can be disable by steps mentioned in this KB.

https://kb.vmware.com/s/article/2144768

All you need to do is 

> Create file with the name locked.properties

> Add line "checkOrigin=false" (without quotes)

> Save and copy this to C:\Program Files\VMware\VMware View\Server\sslgateway\conf folder on all your connection servers.

> Reboot them or restart connection service on them one by one like you normally do

VMware Horizon View 7.X desktop “Agent unreachable” status

Scenario -: 

We had a VDI user reporting issues connecting to his VDI machine.

Checking View Admin page shows this VDI machine 

First thing first

1) Checked vCenter and made sure that the VM is up and running, not down or suspended. 

2) I could remote desktop to it and checked services

3) Restarted Agent Service. No luck

4) Rebooted VM, no luck there too.

Started to look at the logs at this point

C:\ProgramData\VMware\VDM\logs

debug-2019-12-12-150326.txt

2019-12-12T15:03:35.940+10:00 DEBUG (25A0-26C4) <Thread-4> [AgentMessageSecurityHandler] Configuring message security (ON).
2019-12-12T15:03:36.033+10:00 DEBUG (25A0-26C4) <Thread-4> [BrokerUpdateUtility] Published CHANGEKEY request
2019-12-12T15:03:51.035+10:00 DEBUG (25A0-26C4) <Thread-4> [BrokerUpdateUtility] Timeout waiting for success response

So looks like it was trying to change the Key, but wasn't successful. So I decided to push it from

Connection server instead 

1) Login to one of out View Connection Servers

2) Opened a CMD as Admin

3) Ran below commands

Cd C:\Program Files\VMware\VMware View\Server\tools\bin

vdmadmin -A -d <Name of the Desktop Pool> -m <Machine Name> -resetkey

4) You should be able to see the Agent Public Key listed there and thats all good. 

5) Wait for a few mins and could can see status reporting :)

Reporting as "Unassigned User Connected" coz it was assigned to someone else and I logged in there with Admin ID. So rebooted VM and all good afterwords.


Update - 20/03/2020

We have see the same issue when few users enabled installed Docker and part of that Hyper-V feature was enabled!

Removed Hyper-V feature from Add/Remove Programs -> Turn Windows Feature On/Off

Rebooted the VDI machine and that brought Agent back online

CISCO ACI + vCenter Integration Error "Create a vSphere Distributed Switch" "Status: The operation is not supported on the object"

"Create a vSphere Distributed Switch"  

"Status: The operation is not supported on the object"

We were using CISCO ACI + vSphere for a while. We now have a new datacentre coming up and there were conversations on ACI vs NSX. But finally decided to give ACI a second chance :(

Current datacenters are on vSphere 6.0 and were linked to ACI with no issues.

We decided to go with latest 6.7 vSphere and Built them. ESXi hosts were added on to vCenter and a management DVS was created manually (we always had management DVS out side ACI!) and ESXi VMK0 was moved there, basically all went as per plan and well.

Then came the ACI integration part... We had 3 vCenters and they were all under same VMM domain. So we decided to add the new VC also there.

ACI Role was created in vCenter for permissions, service account is configured there mapping it to the new role. Time came for the integration and it wasn't happy about something.. After adding the new VC, ACI could see ESXi inventory. But the DVS creation was failing with below error in vCenter

Operation is not supported! Come on VMware what operation! 

Anyways I had to gig though logs to find out. Finally below is what I found in VPXA log

2019-09-06T01:33:50.907Z error vpxd[04740] [Originator@6876 sub=DvsUtils opID=7a4ad61f] Non-VMware DVS [Cisco Systems Inc.: ] is not supported
2019-09-06T01:33:51.003Z warning vpxd[04740] [Originator@6876 sub=dvsKeeper opID=7a4ad61f] DVS name [virtual] not in reserved map of DvsManager instance
2019-09-06T01:33:51.003Z info vpxd[04740] [Originator@6876 sub=vpxLro opID=7a4ad61f] [VpxLRO] -- FINISH task-3903
2019-09-06T01:33:51.003Z info vpxd[04740] [Originator@6876 sub=Default opID=7a4ad61f] [VpxLRO] -- ERROR task-3903 -- group-n41 -- vim.Folder.createDistributedVirtualSwitch: vmodl.fault.NotSupported:


So, the issue was.....
Old VM domains were created log time back and were set to use Cisco Systems Inc. as the vendor. VMware 6.0 just dosent care (for now! But upgrade to 6.5 will fail and if you look got KBs there is a way to modify it with a SQL command. Atelast we havent go that far and we will be migrating all VMs to the new DC once built and old once will be decommessioned!). But starting from 6.5 U1 VMware stopped supporting third parity DVS switches. Instead they've opned up APIs and said these vendors can now use APIs create and consume VDS (VMware Distributed Switches). 

Now getting back to how we fixed it. Our ACI was maintained well was updated to the latest. So we could just create a new VMM domain and could specify VMware there.  All worked well!

Email alerts stopped after vCenter 6 to 6.5 upgrade

Upgraded out vCenter from 6.0 Appliance to 6.5 Appliance last week.
It was one of my smoothest experience with vCenters all these time. Having said that, after two days I realised that the email alerts stopped working. We had same issues when we upgraded 5.5 Windows to 6.0 Appliance and same thing happened. We had to go though a bunch of manual config changes to get it working below is the KB and worth checking
https://kb.vmware.com/s/article/2073849
I was afraid that it's the same case again, but to my surprise it was not the case here.

If you are experiencing slimier issues, follow below steps

1) Open vSphere Web Client
2) Click on vCenter and Click on Configure at the right side
3) Under Settings -> General Click Edit
4) Click on Mail
5) Cut the Mail Server (same in a notepad if required) and type something else there (may be check@email.com) and click OK to same
6) Edit again and replace the Mail server with the actual SMTP FQDN that you cut earlier.

Boom!!

All the emails started flowing to my inbox again and there were a lot of them. Looks like they were queued.

Give it a try... All the best....

vCSA 6.0 to 6.5 Upgrade Fail – Migration Assistant on VUM Server

Upgrading PSC 6.0 appliance to 6.5 worked like a charm.

Next step was vCenter appliance upgrade,which would now consolidate Windows based VUM server to the vCenter appliance. So this requires you to run VMware Migration Assistant on the Windows VUM server.

Every time I start the Migration Assistant it failed with an below error

Error: A problem occurred during authentication to the legacy vCenter Server using the provided credentials. Resolution make sure the vCenter Server is up and running. Verify you have entered the correct credentials.

I was sure that the SSO password is correct. 

Looking at the logs

UpgradeRunner log mentions

2018-06-20T21:48:04.483Z INFO utils.featureState_utils Found FSS AUTOMATED_VUM_UPGRADE=True

2018-06-20T21:48:04.513Z WARNING bindings.vc_binding Cannot get access to VC Inventory http://<VCIP>:80, because of Error: Server has wrong SHA1 thumbprint: 7aa2612f090cdd1212sqw216943489ada70019 (required) != b3de7868330f12511b767a122abb245448d5d (server).

2018-06-20T21:48:04.515Z ERROR UpgradeRunner Cannot log in to legacy vCenter Server

2018-06-20T21:48:04.516Z INFO output.requirements_result_producer Repository is not set, not adding SSO information.

2018-06-20T21:48:04.529Z INFO output.requirements_result_producer Persisting preupgrade result :

FIX

1) Navigate to C:\Program Files (x86)\VMware\Infrastructure\Update Manager

2) Open "VMwareUpdateManagerUtility"

3) Provide vCenter FQDN or IP and Login with "administrator@vsphere.local" & SSO Password

4) Under the left side options list, click on "Re-register to vCenter server" and provide SSO credentials again (administrator@vsphere.local and password)

5) Once done, restart "VMware vSphere Update Manager Service"

6) Try Migration Assistant again

It's happy this time!!

SQL 2000 Service not starting after P2V / VMware tool Install or Update - SQL 2000 - An error 1053 - (the service did not respond to the start or control request in a timely fashion) occurred while performing this service operation on the MSSQLServer service

SQL 2000 Service not starting after P2V / VMware tool Install or Update -

An error 1053 - (the service did not respond to the start or control request in a timely fashion) occurred while performing this service operation on the MSSQLServer service.

Error 1053: The service did not respond to the start or control request in a timely fashion.

I have faced this issue a couple of times. When you get this error, usually you need to check for Service Accounts, try to turn on "Interact With Desktop" etc.

But in this case,

1) Service was working fine on a physical server. After P2V, it is not starting on VM

2) Service was working fine on a VM, and it failed after recent VMware tool upgrade/change. 

Fix

All that you need to do is to make sure that you have the below two DLLs 

msvcr71.dll 
msvcp71.dll

They should be be present under %systemroot%\system32 

If not, you can copy these DLLs from another server running same SQL version, or restore them from a backup. It is noticed that VMware tools sometimes mess up these two DLLs, which make SQL to fail. 

All the best! 

R.Hari

Moving vCenter Server Databases to a different SQL Server / SQL Instance

Assuming that you have documented below mentioned details from Old vCenter Install
1) vCenter and Update Manager Database user passwords
2) RSA_User and RSA_Admin passwords
3) SSO Master Password

Stage 1) - Backup and Restore Databases

• Stop vCenter Single Sign On,VMware vCenter Inventory Service, VMware VirtualCenter Server, VMware VirtualCenter Management Webservices, VMware vSphere Update Manager Service and VMware vSphere Web Client.
• Backup vCenter, Update Manager and RSA Databases and restore in new DB server.
• Create vCenter, Update Manager and RSA_User and RSA_DBA Users with same passwords on new DB server.
• Assign Same permissions for users

Stage 2) - Recreate Agent Jobs

• Open the below location

          vCenter Server 4.x: C:\Program Files\VMware\Infrastructure\VirtualCenter Server\sql
          vCenter Server 5.x: C:\Program Files\VMware\Infrastructure\VirtualCenter Server\sql

• Copy below mentioned files to a new folder on vCenter server/new SQL Server.

         job_schedule1_mssql.sql
         job_schedule2_mssql.sql
         job_schedule3_mssql.sql
         job_dbm_performance_data_mssql.sql
         job_cleanup_events_mssql.sql
         job_property_bulletin_mssql.sql
         job_topn_past_day_mssql.sql
         job_topn_past_month_mssql.sql
         job_topn_past_week_mssql.sql
         job_topn_past_year_mssql.sql

• Open job_schedule1_mssql.sql using notepad and copy script
• On the new SQL server, right click on vCenter Server Database > New Query > Paste the copied script and execute the copied script 

NOTE: Please double check and make sure that you have VCenter server database selected in the Database list and not Master.

• Repete the same for all the scripts.
• Once all three jobs are created, navigate to SQL Server Agent and Refresh to see new Jobs created. 
• Right-click Past Day stats rollup, click Properties.
• Ensure the owner of the job is the same database login used by VirtualCenter to connect to the database.
• Repete the same for Past Month stats rollup and Past Week stats rollup.

Stage 3) - Re-pointing SSO Service to the new SQL server

• Open SSO Install DIR. By default it is C:\Program Files\VMware\Infrastructure\SSOServer and Navigate to \utils
• Open a command prompt as administrator and run below mentioned command 

       CD C:\Program Files\VMware\Infrastructure\SSOServer\utils
       ssocli configure-riat -a configure-db --database-host <NEW_DBServer_NAME/IP> --database-port <New_DBServer_Port> -m <SSOServerMasterPassword>

• Open C:\Program Files\VMware\Infrastructure\SSOServer\webapps\lookupservice\WEB-INF\classes
• Open config file with NotePad
• Check port number in the URL and DB Server Name at the last line. 
• Save the file and exit
• SSO Service is good to start now!

Stage 4) - Re-pointing vCenter Server to new DB Server

• Start > Run and "Odbcad32.exe"
• Under System DSN, Change DB Server settings and Save it.
• Start VMware VirtualCenter Server Service!

Stage 5) - Re-pointing Update Manager Service to the new SQL server

• Start > Run and "%systemdrive%\Windows\SysWoW64\Odbcad32.exe"
• Under System DSN, Change DB Server settings and Save it.
• Start VMware vSphere Update Manager Service!

If all services are started, reboot vCenter so that all the services are restarted once fresh. 

Disable Agent jobs and Databases on old server, you can probably keep them for a couple of days and then to a cleanup. 

Thank you!

R.Hari

Assuming that you have documented below mentioned details from Old vCenter Install

1) Vcenter and Update Manager Database user passwords

2) RSA_User and RSA_Admin passwords

3) SSO Master Password

Stage1) - Move Databases

Stop vCenter Single Sign On, VMware vCenter Inventory Service, VMware VirtualCenter Server, VMware VirtualCenter Management Webservices, VMware vSphere Update Manager Service and VMware vSphere Web Client.

Backup Vcenter, Update Manager and RSA Databases and restore in new DB server.

Create Vcenter, Update Manager and RSA User and RSA DBA Users with Same passwords on new DB server.

Assign Same permissions for users

Stage2) - Recreate Agent Jobs

Open the below location

vCenter Server 4.x: C:\Program Files\VMware\Infrastructure\VirtualCenter Server\sql

vCenter Server 5.x: C:\Program Files\VMware\Infrastructure\VirtualCenter Server\sql

Copy below mentioned files to a new folder

job_schedule1_mssql.sql 

job_schedule2_mssql.sql

job_schedule3_mssql.sql

job_dbm_performance_data_mssql.sql

job_cleanup_events_mssql.sql

job_property_bulletin_mssql.sql

job_topn_past_day_mssql.sql

job_topn_past_month_mssql.sql

job_topn_past_week_mssql.sql

job_topn_past_year_mssql.sql

Open job_schedule1_mssql.sql using notepad and copy script

On the new SQL server, open Vcenter Server Database and Open New Queiry and execute the copied script 

NOTE: Please double check and make sure that you have VCenter server database selected in the Database list and not Master.

Repete the same for all the scripts.

After all three jobs are created, navigate to SQL Server Agent and Refresh to see new Jobs created. 

Right-click Past Day stats rollup, click Properties. Ensure the owner of the job is the same database login used by VirtualCenter to connect to the database.

Repete the same for Past Month stats rollup and Past Week stats rollup.

Stage3) - Repointing SSO Service to the new SQL server

Open SSO Install DIR. By default it is C:\Program Files\VMware\Infrastructure\SSOServer  and Navigate to \utils

Open a command prompt as administrator

CD C:\Program Files\VMware\Infrastructure\SSOServer\utils

ssocli configure-riat -a configure-db --database-host <NEW_DBServer_NAME/IP> --database-port <New_DBServer_Port> -m <SSOServerMasterPassword>

Executing action: 'configure-db'

Updating Database configuration

Generating HA node package

Successfully executed action: 'configure-db'

Open C:\Program Files\VMware\Infrastructure\SSOServer\webapps\lookupservice\WEB-INF\classes

Open config file with NotePad

Check port number in the URL and DB Server Name at the last line. 

Save the file and exit

Try to start SSO Service.

Stage4) - Repointing vCenter Server to new DB Server

Start > Run and  "Odbcad32.exe"

Under System DSN, Change DB Server settings and Save it.

Start VMware VirtualCenter Server

Stage5) - Repointing Update Manager Service to the new SQL server

Start > Run and  "%systemdrive%\Windows\SysWoW64\Odbcad32.exe"

Under System DSN, Change DB Server settings and Save it.

Start VMware vSphere Update Manager

If all services are started, reboot vCenter so that all the services are restared once fresh. 

Veeam backup error “Error: Failed to connect to guest agent. Errors: ‘Cannot connect to the host’s administrative share” / Access Denied when accessing Admin Shares (\\IPaddress\C$)

If your Windows Server 2008 box is in a WorkGroup and you require access to one of the admin shares, it can be a little more complicated than with Server 2003.

I had to setup a Veeam backup job for backing up a VM running Windows 2008 in WorkGroup. Initial backup job with default settings went well and I had the enabled “Application Aware Image Processing” and “Guest File System Indexing”. For this I had setup a local admin account on Windows 2008 and provided the credentials in Veeam backup job. Job failed on the test run with below error

'ServerHostName' Error: Failed to connect to guest agent. Errors: 'Cannot connect to the host's administrative share. Host: [10.0.0.11]. Account: []. Win32 error:Logon failure: unknown user name or bad password. Code: 1326 '

10.0.0.11 is the Server IP and I tried to access \\10.0.0.11\C$ and I got the windows asking for credentials. With the local admin account created and it worked from same server. When I tried remotely from Backup server, It was showing “Access Denied” error when provided user and password. It was quiet surprising because it is a local admin use and that’s the issue then.

Access Denied error eliminated possible causes like

1)      “Server” Service not running (should have thrown “The specified network name no longer available” error)

2)      File and Pinter sharing disabled on NIC properties

3)       Secpol.msc -> user rights assignment -> “Access this computer from network” and  “Deny Access to this computer from network” (error should be something like “Logon failures: the user has not been granted the requested login type at this computer” error.

Also checked few things like Start -> Run -> fsmgmt.msc and Administrative shared looks good here

Then finally the fix

Issue was with UAC and here is a KB articles http://support.microsoft.com/kb/951016 and http://support.microsoft.com/kb/942817 talks about this issue in Vista.

So it is a security feature and From the KB Article:  “How UAC remote restrictions work: To better protect those users who are members of the local Administrators group, we implement UAC restrictions on the network. This mechanism helps prevent against "loopback" attacks. This mechanism also helps prevent local malicious software from running remotely with administrative rights.”

So please be sure about the risks.

 So what I had to do was -

1)      Right Click on PowerShell Icon in the system tray

2)      Right click on “Windows PowerShell”

3)      Run as Administrator 

4)      Execute below command

New-ItemProperty HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System -name "LocalAccountTokenFilterPolicy" -value "1" -propertyType dword

Thanks!

R.Hari

Reconfigure network after P2V – Message “The IP address XXX.XXX.XXX.XXX you have entered for this network adapter is already assigned to another adapter”

I am sure that the below message is pretty familiar to you if you are working on P2V migrations. Basically, when you try to configure PROD/Mail ip back to the VM after P2V, you will come across this message.  

"The IP address XXX.XXX.XXX.XXX you have entered for this network adapter is already assigned to another adapter"

Message says that the earlier NIC with this IP is still there in the VM, but not visible as it is not attached to the Hardware/OS currently. Windows shows the above message to remind you that there was a NIC earlier with the same IP.  I have seen this message after VMware hardware upgrade also..

It is always better to do clean up post a P2V, instead of just skipping this message. Below mentioned the simple, two line step, which you can run on a VM after P2V to clean up these old NIC card devices and associated drives as we no longer need them.

1) Go to Start > Run

2) Type "cmd" and Enter

3) execute below mentioned commands

set devmgr_show_nonpresent_devices=1

start devmgmt.msc

4) Run both the commands in the same command prompt, because the first command actually forces device manager to start with the ability to see non-present devices when second command is executed.

5) Once Device Manager is open, click View > Show Hidden Devices.

6) Expand the Network Adapters and here you will see the old NICs which is displayed as grayed out. 

7) You can remove them one by one. You can also click the check box to remove old drivers, if there is an option. 

8) Now you can configure the new NIC with the IP.

Thank you!

RHari. 

vCenter / Simple install errors when using SQL Server

Error: Database connection has failed.  You can refer to the vm-sso-javaLib.log in the system temporary folder for more information.

You may get the above error when you fill in details on "Database Information" screen and click on next

Issue is actually with the Single Sign On Database. Installer is expecting a database to be present with few table structures in it. It is stated in the instillation wizard if you go one step back!

So what you need to do is

1) do to the install location (DVD or the local copy)

2) Navigate to Single Sign On\DBScripts\SSOServer\schema\mssql (or to the right database folder if you are using databases other than SQL)

3) SKIP THIS STEP IF SQL IS INSTALLED ON THE SAME SERVER === Copy files "rsaIMSLiteMSSQLSetupTablespaces.sql" and 

"rsaIMSLiteMSSQLSetupUsers.sql" to the server running SQL Database4) Open "SQL Management Studio"

5) Double Click and open 

"rsaIMSLiteMSSQLSetupTablespaces.sql". It will get opened in SQL Management Studio.
6) Edit the file location for .mdf and .ldf file locations and execute the script.

7) Once completed, open 

"rsaIMSLiteMSSQLSetupUsers.sql" the same way8) Edit Passwords (better to keep the user names default as RSA_USER and RSA_DBA, just change the passwors) and execute this one too.

Once this two steps are completed,  the SSO installation should be able to proceed without any issues.

===========================================================

Error: The specified user does not have sufficient privileges.  Refer to Installation and Setup guide for more details on the privilege requirements.

Take a close look at the configurations and makesure that you haven't swapped DataBase User and DataBase DBA User credentials! :)