Email alerts stopped after vCenter 6 to 6.5 upgrade

Upgraded out vCenter from 6.0 Appliance to 6.5 Appliance last week.
It was one of my smoothest experience with vCenters all these time. Having said that, after two days I realised that the email alerts stopped working. We had same issues when we upgraded 5.5 Windows to 6.0 Appliance and same thing happened. We had to go though a bunch of manual config changes to get it working below is the KB and worth checking
https://kb.vmware.com/s/article/2073849
I was afraid that it's the same case again, but to my surprise it was not the case here.

If you are experiencing slimier issues, follow below steps

1) Open vSphere Web Client
2) Click on vCenter and Click on Configure at the right side
3) Under Settings -> General Click Edit
4) Click on Mail
5) Cut the Mail Server (same in a notepad if required) and type something else there (may be check@email.com) and click OK to same
6) Edit again and replace the Mail server with the actual SMTP FQDN that you cut earlier.

Boom!!

All the emails started flowing to my inbox again and there were a lot of them. Looks like they were queued.

Give it a try... All the best....

vCenter 6.5 Upgrade - The managing vCenter Server instance is the same as the source vCenter Server Appliance that you want to upgrade......

The managing vCenter Server instance is the same as the source vCenter Server Appliance that you want to upgrade.

But then the issue is that the stage 2 of the installer wont let you edit the config there. 

Fix

1) Close the Setup and you will get a message that you can continue the install by going to https://<vcenter>:5480

2) Copy the link

3) Open a new browser and paste the link

4) Click on Upgrade 

5) Type in root password of the new 6.5 appliance and continue Stage 2 of the upgrade, by entering host name of the ESXi host where VM reside and credentials. 

ESXi 6.0 to 6.5 upgrade - Conflicting VIBs in HP image

Upgrading HP 6.0 Image to 6.5 is listed as incompatible in VUM due to below VIB Conflicts

The upgrade contains the following set of conflicting VIBs:
Emulex_bootbank_scsi-lpfc820_8.2.4.151.65-1OEM.500.0.0.472560
Emulex_bootbank_scsi-lpfc820_8.2.4.151.65-1OEM.500.0.0.472560

Remove the conflicting VIBs or use Image Builder to create a custom upgrade ISO image that contains the newer versions of the conflicting VIBs, and try to upgrade again.

Fix

1) Look for the drivers first

[root@esx01:~] esxcli software vib list | grep  lpfc
lpfc                           11.4.199.0-1OEM.600.0.0.2768847        EMU              VMwareCertified   2018-06-01
scsi-lpfc820                   8.2.4.151.65-1OEM.500.0.0.472560       Emulex           VMwareCertified   2017-09-25

So I have ipfc and scsi-lpfc820 drivers installed. 

2) Check if it is in use

[root@esx01:~] esxcli network nic list

Name    PCI Device    Driver  Admin Status  Link Status  Speed  Duplex  MAC Address         MTU  Description
------  ------------  ------  ------------  -----------  -----  ------  -----------------  ----  -------------------------------------------------------------
vmnic0  0000:03:00.0  tg3     Up            Down             0  Half    a0:d3:c1:fb:fc:e0  1500  Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet
vmnic1  0000:03:00.1  tg3     Up            Down             0  Half    a0:d3:c1:fb:fc:e1  1500  Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet
vmnic2  0000:03:00.2  tg3     Up            Down             0  Half    a0:d3:c1:fb:fc:e2  1500  Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet
vmnic3  0000:03:00.3  tg3     Up            Down             0  Half    a0:d3:c1:fb:fc:e3  1500  Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet
vmnic4  0000:24:00.0  elxnet  Up            Up           10000  Full    c4:34:6b:1f:05:a8  1500  Emulex Corporation HP NC552SFP Dual Port 10GbE Server Adapter
vmnic5  0000:24:00.1  elxnet  Up            Up           10000  Full    c4:34:6b:1f:05:ac  1500  Emulex Corporation HP NC552SFP Dual Port 10GbE Server Adapter

[root@esx01:~] esxcfg-scsidevs -a
vmhba0  ata_piix          link-n/a  sata.vmhba0                             (0000:00:1f.2) Intel Corporation Patsburg 4 port SATA IDE Controller
vmhba1  hpsa              link-n/a  sas.5001438031025c30                    (0000:02:00.0) Hewlett Packard Enterprise Smart Array P420i
vmhba4  lpfc              link-up   fc.20000090fa561e40:10000090fa561e40    (0000:0a:00.0) Emulex Corporation LPe12000 8Gb Fibre Channel Host Adapter
vmhba5  lpfc              link-up   fc.20000090fa561e41:10000090fa561e41    (0000:0a:00.1) Emulex Corporation LPe12000 8Gb Fibre Channel Host Adapter
vmhba32 usb-storage       link-n/a  usb.vmhba32                             () USB
vmhba33 ata_piix          link-n/a  sata.vmhba33                            (0000:00:1f.2) Intel Corporation Patsburg 4 port SATA IDE Controller

Highlighted are the drivers in use and that confirmed scsi-lpfc820 is not in use. 

3) Remove the conflicting VIB
[root@esx01:~] esxcli software vib remove -n scsi-lpfc820

Removal Result
   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
   Reboot Required: true
   VIBs Installed:
   VIBs Removed: Emulex_bootbank_scsi-lpfc820_8.2.4.151.65-1OEM.500.0.0.472560
   VIBs Skipped:

4) Reboot ESXi host

5) Scan again and check status on VUM status

This time it shows Non-Compliant! 

So I decided to try removing Mellenox driver, which used to give issues with ESX5x to 6 upgrade! 

[root@esx01:~] esxcli software vib list |grep Mellanox

net-mlx4-core                  1.9.9.4-1OEM.550.0.0.1331820           Mellanox         VMwareCertified   2017-09-25

net-mlx4-en                    1.9.9.4-1OEM.550.0.0.1331820           Mellanox         VMwareCertified   2017-09-25

[root@esx01:~] esxcli software vib remove -n net-mlx4-en

Removal Result

   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.

   Reboot Required: true

   VIBs Installed:

   VIBs Removed: Mellanox_bootbank_net-mlx4-en_1.9.9.4-1OEM.550.0.0.1331820

   VIBs Skipped:

[root@esx01:~] esxcli software vib remove -n net-mlx4-core

Removal Result

   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.

   Reboot Required: true

   VIBs Installed:

   VIBs Removed: Mellanox_bootbank_net-mlx4-core_1.9.9.4-1OEM.550.0.0.1331820

   VIBs Skipped:

Rebooted ESXi host again. 

Once it came back, scanned again with VUM and checked status. It was happy this time and I could update it without issues.

Thank you!