CISCO ACI + vCenter Integration Error "Create a vSphere Distributed Switch" "Status: The operation is not supported on the object"

"Create a vSphere Distributed Switch"  

"Status: The operation is not supported on the object"

We were using CISCO ACI + vSphere for a while. We now have a new datacentre coming up and there were conversations on ACI vs NSX. But finally decided to give ACI a second chance :(

Current datacenters are on vSphere 6.0 and were linked to ACI with no issues.

We decided to go with latest 6.7 vSphere and Built them. ESXi hosts were added on to vCenter and a management DVS was created manually (we always had management DVS out side ACI!) and ESXi VMK0 was moved there, basically all went as per plan and well.

Then came the ACI integration part... We had 3 vCenters and they were all under same VMM domain. So we decided to add the new VC also there.

ACI Role was created in vCenter for permissions, service account is configured there mapping it to the new role. Time came for the integration and it wasn't happy about something.. After adding the new VC, ACI could see ESXi inventory. But the DVS creation was failing with below error in vCenter

Operation is not supported! Come on VMware what operation! 

Anyways I had to gig though logs to find out. Finally below is what I found in VPXA log

2019-09-06T01:33:50.907Z error vpxd[04740] [Originator@6876 sub=DvsUtils opID=7a4ad61f] Non-VMware DVS [Cisco Systems Inc.: ] is not supported
2019-09-06T01:33:51.003Z warning vpxd[04740] [Originator@6876 sub=dvsKeeper opID=7a4ad61f] DVS name [virtual] not in reserved map of DvsManager instance
2019-09-06T01:33:51.003Z info vpxd[04740] [Originator@6876 sub=vpxLro opID=7a4ad61f] [VpxLRO] -- FINISH task-3903
2019-09-06T01:33:51.003Z info vpxd[04740] [Originator@6876 sub=Default opID=7a4ad61f] [VpxLRO] -- ERROR task-3903 -- group-n41 -- vim.Folder.createDistributedVirtualSwitch: vmodl.fault.NotSupported:


So, the issue was.....
Old VM domains were created log time back and were set to use Cisco Systems Inc. as the vendor. VMware 6.0 just dosent care (for now! But upgrade to 6.5 will fail and if you look got KBs there is a way to modify it with a SQL command. Atelast we havent go that far and we will be migrating all VMs to the new DC once built and old once will be decommessioned!). But starting from 6.5 U1 VMware stopped supporting third parity DVS switches. Instead they've opned up APIs and said these vendors can now use APIs create and consume VDS (VMware Distributed Switches). 

Now getting back to how we fixed it. Our ACI was maintained well was updated to the latest. So we could just create a new VMM domain and could specify VMware there.  All worked well!

PowerCLI Install error on powershell

Today I was trying to install PowerCli on my Windows 10 machine using below command

Install-Module -Name VMware.PowerCLI -RequiredVersion 11.1.0.11289667

Ended up with an error "PackageManagement\Install-Package : The following commands are already available on this system:'Get-Cluster,New-Cluster,Remove-Cluster'. This module 'VMware.VimAutomation.Core' may override the existing commands. If you still want to install this module"

Looked like some of the modules are conflicting.
As per MS document here "If the module being installed has the same name or version, or contains commands in an existing module, warning messages are displayed. After you confirm that you want to install the module and override the warnings, use the -Force and -AllowClobber" 

So decided to use -AllowClobber and it worked! 






vCenter Appliance WebClient down - Error: Service name is invalid

vCenter Appliance - WebClient down with "Error: Service name is invalid"

Last week I had to shutdown my lab vCenter 6.7. As it's just a lab I just powered it off instead of shutting down. After a few days I powered it back on when I wanted it back, and it booted without issues. But then WebClient wouldn't come up. Just says page cannot be displayed.

I logged in to the appliance via SSH and tried to list service status

root@vcenter [ ~ ]#  service-control --status --all
2019-04-04T10:43:41.796Z  Error: Service name "applmgmt" is invalid.
2019-04-04T10:43:41.816Z  Error: Service name "content-library" is invalid.
2019-04-04T10:43:41.837Z  Error: Service name "vcha" is invalid.
2019-04-04T10:43:41.857Z  Error: Service name "pschealth" is invalid.
2019-04-04T10:43:41.878Z  Error: Service name "sps" is invalid.
2019-04-04T10:43:41.898Z  Error: Service name "sca" is invalid.
2019-04-04T10:43:41.936Z  Error: Service name "vmcam" is invalid.
2019-04-04T10:43:41.956Z  Error: Service name "vmware-vpostgres" is invalid.
2019-04-04T10:43:41.976Z  Error: Service name "vsphere-client" is invalid.
2019-04-04T10:43:41.997Z  Error: Service name "vapi-endpoint" is invalid.
2019-04-04T10:43:42.017Z  Error: Service name "vmware-postgres-archiver" is invalid.
2019-04-04T10:43:42.037Z  Error: Service name "vsm" is invalid.
2019-04-04T10:43:42.057Z  Error: Service name "updatemgr" is invalid.
2019-04-04T10:43:42.078Z  Error: Service name "vmonapi" is invalid.
2019-04-04T10:43:42.098Z  Error: Service name "vsan-health" is invalid.
2019-04-04T10:43:42.119Z  Error: Service name "rbd" is invalid.
2019-04-04T10:43:42.156Z  Error: Service name "vpxd-svcs" is invalid.
2019-04-04T10:43:42.176Z  Error: Service name "statsmonitor" is invalid.
2019-04-04T10:43:42.264Z  Error: Service name "imagebuilder" is invalid.
2019-04-04T10:43:42.284Z  Error: Service name "vsan-dps" is invalid.
2019-04-04T10:43:42.304Z  Error: Service name "cm" is invalid.
2019-04-04T10:43:42.342Z  Error: Service name "mbcs" is invalid.
2019-04-04T10:43:42.362Z  Error: Service name "vpxd" is invalid.
2019-04-04T10:43:42.382Z  Error: Service name "netdumper" is invalid.
2019-04-04T10:43:42.403Z  Error: Service name "rhttpproxy" is invalid.
2019-04-04T10:43:42.423Z  Error: Service name "vsphere-ui" is invalid.
2019-04-04T10:43:42.444Z  Error: Service name "eam" is invalid.
2019-04-04T10:43:42.464Z  Error: Service name "perfcharts" is invalid.
2019-04-04T10:43:42.519Z  Error: Service name "analytics" is invalid.
2019-04-04T10:43:42.539Z  Error: Service name "cis-license" is invalid.
Running:
lwsmd vmafdd vmcad vmdird vmdnsd vmware-pod vmware-sts-idmd vmware-stsd vmware-vmon

Tried to start them using "service-control --start --all", but it didn't work. 

I followed below steps to get it back running.. 

Make sure that you take a backup of your vCenter before trying this (Snapshot or Clone)

1) Stop all services

root@vcenter [ ~ ]# service-control --stop --all

2) Cd to the below dir

root@vcenter [ ~ ]# cd /storage/vmware-vmon/.svcStats

3) Create a new directory and move all .Jason files there 

root@vcenter [ /storage/vmware-vmon/.svcStats ]# ls
stats_analytics.json    stats_cm.json            stats_netdumper.json   stats_sps.json            stats_vcha.json     stats_vmware-postgres-archiver.json  stats_vpxd-svcs.json    stats_vsphere-client.json
stats_applmgmt.json     stats_eam.json           stats_perfcharts.json  stats_statsmonitor.json   stats_vdcs.json     stats_vmware-sca.json                stats_vsan-dps.json     stats_vsphere-ui.json
stats_autodeploy.json   stats_imagebuilder.json  stats_pschealth.json   stats_updatemgr.json      stats_vmcam.json    stats_vmware-vpostgres.json          stats_vsan-health.json
stats_cis-license.json  stats_mbcs.json          stats_rhttpproxy.json  stats_vapi-endpoint.json  stats_vmonapi.json  stats_vpxd.json                      stats_vsm.json

root@vcenter [ /storage/vmware-vmon/.svcStats ]# mkdir StatsjsonBKP

root@vcenter [ /storage/vmware-vmon/.svcStats ]# mv *.json /storage/vmware-vmon/.svcStats/mkdir StatsjsonBKP/

4) Do another ls to confirm

root@vcenter [ /storage/vmware-vmon/.svcStats ]# ls

mkdir StatsjsonBKP

5) Now it's time to start all services

root@vcenter [ ~ ]# service-control --start --all

Operation not cancellable. Please wait for it to finish...
Performing start operation on service lwsmd...
Successfully started service lwsmd
Performing start operation on service vmafdd...
Successfully started service vmafdd
Performing start operation on service vmdird...
Successfully started service vmdird
Performing start operation on service vmcad...
Successfully started service vmcad
Performing start operation on service vmware-sts-idmd...
Successfully started service vmware-sts-idmd
Performing start operation on service vmware-stsd...
Successfully started service vmware-stsd
Performing start operation on service vmdnsd...
Successfully started service vmdnsd
Performing start operation on profile: ALL...
Successfully started profile: ALL.
Performing start operation on service vmware-pod...

Successfully started service vmware-pod

Bang!! 

Gave few mins to start web client and all good!

Converting P2V of a Windows 2000 Server to vSphere 6.X

It's a shame but I recently had to P2V a Windows 2000 server, part of our legacy services, in order to free up and decomm an old physical server. It was not a pleasure at all :(

Planning

1) Download and keep VMware Converter 4.0.1 (VMware-converter-4.0.1-161434.exe) – To P2V Windows 2000

P2V Converter 4.0.1 doesn’t support any vCenter/ ESXi later than 4.x. So I ended up in installing ESXi 4.0 on a VM running on top of my vSphere 6.5 infra!

So 2) ESXi 4.0 ISO

I read articles of people using VMware WorkStation for this, but in my case I didn’t want the export job to run outside datacentre, due to security reasons.

My Windows 2000 Server that I need to virtualize is 200 Gig. So if you look at the disk size of ESXi VM, I made sure that It’s bigger than that. 

 

 

3) Check Patch Level of Windows 2000 server and make sure

> It’s updated to SP4 Rollup 1 (KB891861)

If not, install from here and reboot the server before running P2V job

4) Download and keep VMware Converter 6.0.0 (VMware-converter-en-6.0.0-2716716.exe) – To move Virtual machine from ESX 4.0 to vCenter 6.X

5) vmscsi-1.2.0.4.flp For troubleshooting - https://kb.vmware.com/s/article/100520

Steps I followed

1. 1)      Installed ESX4.0 on my VM that I created, configured it with an IP that is accessible from Windows 2000 server and also from my PROD 6.x vCenter
   2)      Installed Converter 4.0.1 on the Windows 2000 machine
   3)      Started P2V Conversion of the Windows 2000 machine to the virtual ESX 4.1
   Make sure you select  “SCSI” as Disk Controller and not “Preserve Source”
   
   
   4)      Once the conversion is complete, we now have the VM inside ESX 4. However as this is a nested setup, we can’t power on and test it.
   5)      Install VMware Converter 6.0.0 on a machine that has access to the ESXi 4 VM and your PROD vCenter. We will use V2V on this converter version to move the VM to our PROD 6.5
   Not tested with 6.7, but worst case, we can also download the files from the datastore and move to PROD vCenter!
   6)      Once done, power on and test.
   
   
   
   

You can then take a backup and do R&D stuff to keep VMware tools and Hardware up to date J

Issues that I’ve seen

BSOD with “KMODE_EXCEPTION_NOT_HANDLED” error while loading windows.

This usually happens if you haven’t installed the SP4 Rollup 1 (KB891861) on the source VM.

I could fix it, without having to re-run P2V. Steps below

1. 1)      Detach the C:\ Disk from the Windows 2000 VM where you see this issue after P2V
   2)      Connect this disk to any of your existing Windows VMs.
   3)      Go to disk management and bring the disk online and assign a drive letter
   4)      Use 7 Zip and Extract KB891861 and vmscsi-1.2.0.4.flp that we’ve downloaded.
   Look for scsiport.sys under KB891861 extract – version 5.00.2195.7059
         And for vmscsi.sys under vmscsi-1.2.0.4.flp extract – version 1.2.0.4 
   
   
          Make sure the versions are correct. 
   
   
   5)  Copy these two files to the VM where we have the disk attached.

1. 6)      Navigate to WINNT\System32\drivers and rename existing scsiport.sys and vmscsi.sys to “–old”
   7)      Copy the two new files.
   8)      Bring disk offline and move it back to Windows 2000 VM
   9)      Double check and make sure that the SCSI controller type is “BusLogic Parallel”
   10)   Power on the VM to test!