Sunday, June 24, 2018

ESXi 6.0 to 6.5 upgrade - Conflicting VIBs in HP image

Upgrading HP 6.0 Image to 6.5 is listed as incompatible in VUM due to below VIB Conflicts

The upgrade contains the following set of conflicting VIBs:
Emulex_bootbank_scsi-lpfc820_8.2.4.151.65-1OEM.500.0.0.472560
Emulex_bootbank_scsi-lpfc820_8.2.4.151.65-1OEM.500.0.0.472560

Remove the conflicting VIBs or use Image Builder to create a custom upgrade ISO image that contains the newer versions of the conflicting VIBs, and try to upgrade again.

Fix

1) Look for the drivers first

[root@esx01:~] esxcli software vib list | grep  lpfc
lpfc                           11.4.199.0-1OEM.600.0.0.2768847        EMU              VMwareCertified   2018-06-01
scsi-lpfc820                   8.2.4.151.65-1OEM.500.0.0.472560       Emulex           VMwareCertified   2017-09-25

So I have ipfc and scsi-lpfc820 drivers installed. 

2) Check if it is in use

[root@esx01:~] esxcli network nic list
Name    PCI Device    Driver  Admin Status  Link Status  Speed  Duplex  MAC Address         MTU  Description
------  ------------  ------  ------------  -----------  -----  ------  -----------------  ----  -------------------------------------------------------------
vmnic0  0000:03:00.0  tg3     Up            Down             0  Half    a0:d3:c1:fb:fc:e0  1500  Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet
vmnic1  0000:03:00.1  tg3     Up            Down             0  Half    a0:d3:c1:fb:fc:e1  1500  Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet
vmnic2  0000:03:00.2  tg3     Up            Down             0  Half    a0:d3:c1:fb:fc:e2  1500  Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet
vmnic3  0000:03:00.3  tg3     Up            Down             0  Half    a0:d3:c1:fb:fc:e3  1500  Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet
vmnic4  0000:24:00.0  elxnet  Up            Up           10000  Full    c4:34:6b:1f:05:a8  1500  Emulex Corporation HP NC552SFP Dual Port 10GbE Server Adapter
vmnic5  0000:24:00.1  elxnet  Up            Up           10000  Full    c4:34:6b:1f:05:ac  1500  Emulex Corporation HP NC552SFP Dual Port 10GbE Server Adapter

[root@esx01:~] esxcfg-scsidevs -a
vmhba0  ata_piix          link-n/a  sata.vmhba0                             (0000:00:1f.2) Intel Corporation Patsburg 4 port SATA IDE Controller
vmhba1  hpsa              link-n/a  sas.5001438031025c30                    (0000:02:00.0) Hewlett Packard Enterprise Smart Array P420i
vmhba4  lpfc              link-up   fc.20000090fa561e40:10000090fa561e40    (0000:0a:00.0) Emulex Corporation LPe12000 8Gb Fibre Channel Host Adapter
vmhba5  lpfc              link-up   fc.20000090fa561e41:10000090fa561e41    (0000:0a:00.1) Emulex Corporation LPe12000 8Gb Fibre Channel Host Adapter
vmhba32 usb-storage       link-n/a  usb.vmhba32                             () USB
vmhba33 ata_piix          link-n/a  sata.vmhba33                            (0000:00:1f.2) Intel Corporation Patsburg 4 port SATA IDE Controller


Highlighted are the drivers in use and that confirmed scsi-lpfc820 is not in use. 

3) Remove the conflicting VIB
[root@esx01:~] esxcli software vib remove -n scsi-lpfc820

Removal Result
   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
   Reboot Required: true
   VIBs Installed:
   VIBs Removed: Emulex_bootbank_scsi-lpfc820_8.2.4.151.65-1OEM.500.0.0.472560
   VIBs Skipped:

4) Reboot ESXi host

5) Scan again and check status on VUM status
This time it shows Non-Compliant! 

So I decided to try removing Mellenox driver, which used to give issues with ESX5x to 6 upgrade! 

[root@esx01:~] esxcli software vib list |grep Mellanox
net-mlx4-core                  1.9.9.4-1OEM.550.0.0.1331820           Mellanox         VMwareCertified   2017-09-25
net-mlx4-en                    1.9.9.4-1OEM.550.0.0.1331820           Mellanox         VMwareCertified   2017-09-25

[root@esx01:~] esxcli software vib remove -n net-mlx4-en
Removal Result
   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
   Reboot Required: true
   VIBs Installed:
   VIBs Removed: Mellanox_bootbank_net-mlx4-en_1.9.9.4-1OEM.550.0.0.1331820
   VIBs Skipped:

[root@esx01:~] esxcli software vib remove -n net-mlx4-core
Removal Result
   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
   Reboot Required: true
   VIBs Installed:
   VIBs Removed: Mellanox_bootbank_net-mlx4-core_1.9.9.4-1OEM.550.0.0.1331820
   VIBs Skipped:

Rebooted ESXi host again. 
Once it came back, scanned again with VUM and checked status. It was happy this time and I could update it without issues.

Thank you! 













Wednesday, June 20, 2018

vCSA 6.0 to 6.5 Upgrade Fail – Migration Assistant on VUM Server

Upgrading PSC 6.0 appliance to 6.5 worked like a charm.
Next step was vCenter appliance upgrade,which would now consolidate Windows based VUM server to the vCenter appliance. So this requires you to run VMware Migration Assistant on the Windows VUM server.

Every time I start the Migration Assistant it failed with an below error

Error: A problem occurred during authentication to the legacy vCenter Server using the provided credentials. Resolution make sure the vCenter Server is up and running. Verify you have entered the correct credentials.


I was sure that the SSO password is correct. 

Looking at the logs
UpgradeRunner log mentions

2018-06-20T21:48:04.483Z INFO utils.featureState_utils Found FSS AUTOMATED_VUM_UPGRADE=True
2018-06-20T21:48:04.513Z WARNING bindings.vc_binding Cannot get access to VC Inventory http://<VCIP>:80, because of Error: Server has wrong SHA1 thumbprint: 7aa2612f090cdd1212sqw216943489ada70019 (required) != b3de7868330f12511b767a122abb245448d5d (server).
2018-06-20T21:48:04.515Z ERROR UpgradeRunner Cannot log in to legacy vCenter Server
2018-06-20T21:48:04.516Z INFO output.requirements_result_producer Repository is not set, not adding SSO information.
2018-06-20T21:48:04.529Z INFO output.requirements_result_producer Persisting preupgrade result :

FIX
1) Navigate to C:\Program Files (x86)\VMware\Infrastructure\Update Manager
2) Open "VMwareUpdateManagerUtility"
3) Provide vCenter FQDN or IP and Login with "administrator@vsphere.local" & SSO Password
4) Under the left side options list, click on "Re-register to vCenter server" and provide SSO credentials again (administrator@vsphere.local and password)
5) Once done, restart "VMware vSphere Update Manager Service"
6) Try Migration Assistant again

It's happy this time!!