ESXi 6.5 - VMware tools 10.3 BSOD issue with Windows 2012 and higher with hardware version 13

Windows 2012 and higher versions with VMware tools 10.3X version with hardware version 13 (and of-course running on 6.5 ESXi) are pron to a BSOD issue

I got the same results when I tested it my lab.

BSOD would say "bad_pool_caller".

VMware this morning released KB recalling whole 10.3 family of tools. Check below links

https://kb.vmware.com/s/article/57796
https://docs.vmware.com/en/VMware-Tools/10.3/rn/vmware-tools-1030-release-notes.html

Recommendation is to remove it completely and install last stable build, which is 10.2.5

Email alerts stopped after vCenter 6 to 6.5 upgrade

Upgraded out vCenter from 6.0 Appliance to 6.5 Appliance last week.
It was one of my smoothest experience with vCenters all these time. Having said that, after two days I realised that the email alerts stopped working. We had same issues when we upgraded 5.5 Windows to 6.0 Appliance and same thing happened. We had to go though a bunch of manual config changes to get it working below is the KB and worth checking
https://kb.vmware.com/s/article/2073849
I was afraid that it's the same case again, but to my surprise it was not the case here.

If you are experiencing slimier issues, follow below steps

1) Open vSphere Web Client
2) Click on vCenter and Click on Configure at the right side
3) Under Settings -> General Click Edit
4) Click on Mail
5) Cut the Mail Server (same in a notepad if required) and type something else there (may be check@email.com) and click OK to same
6) Edit again and replace the Mail server with the actual SMTP FQDN that you cut earlier.

Boom!!

All the emails started flowing to my inbox again and there were a lot of them. Looks like they were queued.

Give it a try... All the best....

vCenter 6.5 Upgrade - The managing vCenter Server instance is the same as the source vCenter Server Appliance that you want to upgrade......

The managing vCenter Server instance is the same as the source vCenter Server Appliance that you want to upgrade.

But then the issue is that the stage 2 of the installer wont let you edit the config there. 

Fix

1) Close the Setup and you will get a message that you can continue the install by going to https://<vcenter>:5480

2) Copy the link

3) Open a new browser and paste the link

4) Click on Upgrade 

5) Type in root password of the new 6.5 appliance and continue Stage 2 of the upgrade, by entering host name of the ESXi host where VM reside and credentials. 

ESXi 6.0 to 6.5 upgrade - Conflicting VIBs in HP image

Upgrading HP 6.0 Image to 6.5 is listed as incompatible in VUM due to below VIB Conflicts

The upgrade contains the following set of conflicting VIBs:
Emulex_bootbank_scsi-lpfc820_8.2.4.151.65-1OEM.500.0.0.472560
Emulex_bootbank_scsi-lpfc820_8.2.4.151.65-1OEM.500.0.0.472560

Remove the conflicting VIBs or use Image Builder to create a custom upgrade ISO image that contains the newer versions of the conflicting VIBs, and try to upgrade again.

Fix

1) Look for the drivers first

[root@esx01:~] esxcli software vib list | grep  lpfc
lpfc                           11.4.199.0-1OEM.600.0.0.2768847        EMU              VMwareCertified   2018-06-01
scsi-lpfc820                   8.2.4.151.65-1OEM.500.0.0.472560       Emulex           VMwareCertified   2017-09-25

So I have ipfc and scsi-lpfc820 drivers installed. 

2) Check if it is in use

[root@esx01:~] esxcli network nic list

Name    PCI Device    Driver  Admin Status  Link Status  Speed  Duplex  MAC Address         MTU  Description
------  ------------  ------  ------------  -----------  -----  ------  -----------------  ----  -------------------------------------------------------------
vmnic0  0000:03:00.0  tg3     Up            Down             0  Half    a0:d3:c1:fb:fc:e0  1500  Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet
vmnic1  0000:03:00.1  tg3     Up            Down             0  Half    a0:d3:c1:fb:fc:e1  1500  Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet
vmnic2  0000:03:00.2  tg3     Up            Down             0  Half    a0:d3:c1:fb:fc:e2  1500  Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet
vmnic3  0000:03:00.3  tg3     Up            Down             0  Half    a0:d3:c1:fb:fc:e3  1500  Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet
vmnic4  0000:24:00.0  elxnet  Up            Up           10000  Full    c4:34:6b:1f:05:a8  1500  Emulex Corporation HP NC552SFP Dual Port 10GbE Server Adapter
vmnic5  0000:24:00.1  elxnet  Up            Up           10000  Full    c4:34:6b:1f:05:ac  1500  Emulex Corporation HP NC552SFP Dual Port 10GbE Server Adapter

[root@esx01:~] esxcfg-scsidevs -a
vmhba0  ata_piix          link-n/a  sata.vmhba0                             (0000:00:1f.2) Intel Corporation Patsburg 4 port SATA IDE Controller
vmhba1  hpsa              link-n/a  sas.5001438031025c30                    (0000:02:00.0) Hewlett Packard Enterprise Smart Array P420i
vmhba4  lpfc              link-up   fc.20000090fa561e40:10000090fa561e40    (0000:0a:00.0) Emulex Corporation LPe12000 8Gb Fibre Channel Host Adapter
vmhba5  lpfc              link-up   fc.20000090fa561e41:10000090fa561e41    (0000:0a:00.1) Emulex Corporation LPe12000 8Gb Fibre Channel Host Adapter
vmhba32 usb-storage       link-n/a  usb.vmhba32                             () USB
vmhba33 ata_piix          link-n/a  sata.vmhba33                            (0000:00:1f.2) Intel Corporation Patsburg 4 port SATA IDE Controller

Highlighted are the drivers in use and that confirmed scsi-lpfc820 is not in use. 

3) Remove the conflicting VIB
[root@esx01:~] esxcli software vib remove -n scsi-lpfc820

Removal Result
   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
   Reboot Required: true
   VIBs Installed:
   VIBs Removed: Emulex_bootbank_scsi-lpfc820_8.2.4.151.65-1OEM.500.0.0.472560
   VIBs Skipped:

4) Reboot ESXi host

5) Scan again and check status on VUM status

This time it shows Non-Compliant! 

So I decided to try removing Mellenox driver, which used to give issues with ESX5x to 6 upgrade! 

[root@esx01:~] esxcli software vib list |grep Mellanox

net-mlx4-core                  1.9.9.4-1OEM.550.0.0.1331820           Mellanox         VMwareCertified   2017-09-25

net-mlx4-en                    1.9.9.4-1OEM.550.0.0.1331820           Mellanox         VMwareCertified   2017-09-25

[root@esx01:~] esxcli software vib remove -n net-mlx4-en

Removal Result

   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.

   Reboot Required: true

   VIBs Installed:

   VIBs Removed: Mellanox_bootbank_net-mlx4-en_1.9.9.4-1OEM.550.0.0.1331820

   VIBs Skipped:

[root@esx01:~] esxcli software vib remove -n net-mlx4-core

Removal Result

   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.

   Reboot Required: true

   VIBs Installed:

   VIBs Removed: Mellanox_bootbank_net-mlx4-core_1.9.9.4-1OEM.550.0.0.1331820

   VIBs Skipped:

Rebooted ESXi host again. 

Once it came back, scanned again with VUM and checked status. It was happy this time and I could update it without issues.

Thank you! 

vCSA 6.0 to 6.5 Upgrade Fail – Migration Assistant on VUM Server

Upgrading PSC 6.0 appliance to 6.5 worked like a charm.

Next step was vCenter appliance upgrade,which would now consolidate Windows based VUM server to the vCenter appliance. So this requires you to run VMware Migration Assistant on the Windows VUM server.

Every time I start the Migration Assistant it failed with an below error

Error: A problem occurred during authentication to the legacy vCenter Server using the provided credentials. Resolution make sure the vCenter Server is up and running. Verify you have entered the correct credentials.

I was sure that the SSO password is correct. 

Looking at the logs

UpgradeRunner log mentions

2018-06-20T21:48:04.483Z INFO utils.featureState_utils Found FSS AUTOMATED_VUM_UPGRADE=True

2018-06-20T21:48:04.513Z WARNING bindings.vc_binding Cannot get access to VC Inventory http://<VCIP>:80, because of Error: Server has wrong SHA1 thumbprint: 7aa2612f090cdd1212sqw216943489ada70019 (required) != b3de7868330f12511b767a122abb245448d5d (server).

2018-06-20T21:48:04.515Z ERROR UpgradeRunner Cannot log in to legacy vCenter Server

2018-06-20T21:48:04.516Z INFO output.requirements_result_producer Repository is not set, not adding SSO information.

2018-06-20T21:48:04.529Z INFO output.requirements_result_producer Persisting preupgrade result :

FIX

1) Navigate to C:\Program Files (x86)\VMware\Infrastructure\Update Manager

2) Open "VMwareUpdateManagerUtility"

3) Provide vCenter FQDN or IP and Login with "administrator@vsphere.local" & SSO Password

4) Under the left side options list, click on "Re-register to vCenter server" and provide SSO credentials again (administrator@vsphere.local and password)

5) Once done, restart "VMware vSphere Update Manager Service"

6) Try Migration Assistant again

It's happy this time!!