vCenter Appliance WebClient down - Error: Service name is invalid

vCenter Appliance - WebClient down with "Error: Service name is invalid"

Last week I had to shutdown my lab vCenter 6.7. As it's just a lab I just powered it off instead of shutting down. After a few days I powered it back on when I wanted it back, and it booted without issues. But then WebClient wouldn't come up. Just says page cannot be displayed.

I logged in to the appliance via SSH and tried to list service status

root@vcenter [ ~ ]#  service-control --status --all
2019-04-04T10:43:41.796Z  Error: Service name "applmgmt" is invalid.
2019-04-04T10:43:41.816Z  Error: Service name "content-library" is invalid.
2019-04-04T10:43:41.837Z  Error: Service name "vcha" is invalid.
2019-04-04T10:43:41.857Z  Error: Service name "pschealth" is invalid.
2019-04-04T10:43:41.878Z  Error: Service name "sps" is invalid.
2019-04-04T10:43:41.898Z  Error: Service name "sca" is invalid.
2019-04-04T10:43:41.936Z  Error: Service name "vmcam" is invalid.
2019-04-04T10:43:41.956Z  Error: Service name "vmware-vpostgres" is invalid.
2019-04-04T10:43:41.976Z  Error: Service name "vsphere-client" is invalid.
2019-04-04T10:43:41.997Z  Error: Service name "vapi-endpoint" is invalid.
2019-04-04T10:43:42.017Z  Error: Service name "vmware-postgres-archiver" is invalid.
2019-04-04T10:43:42.037Z  Error: Service name "vsm" is invalid.
2019-04-04T10:43:42.057Z  Error: Service name "updatemgr" is invalid.
2019-04-04T10:43:42.078Z  Error: Service name "vmonapi" is invalid.
2019-04-04T10:43:42.098Z  Error: Service name "vsan-health" is invalid.
2019-04-04T10:43:42.119Z  Error: Service name "rbd" is invalid.
2019-04-04T10:43:42.156Z  Error: Service name "vpxd-svcs" is invalid.
2019-04-04T10:43:42.176Z  Error: Service name "statsmonitor" is invalid.
2019-04-04T10:43:42.264Z  Error: Service name "imagebuilder" is invalid.
2019-04-04T10:43:42.284Z  Error: Service name "vsan-dps" is invalid.
2019-04-04T10:43:42.304Z  Error: Service name "cm" is invalid.
2019-04-04T10:43:42.342Z  Error: Service name "mbcs" is invalid.
2019-04-04T10:43:42.362Z  Error: Service name "vpxd" is invalid.
2019-04-04T10:43:42.382Z  Error: Service name "netdumper" is invalid.
2019-04-04T10:43:42.403Z  Error: Service name "rhttpproxy" is invalid.
2019-04-04T10:43:42.423Z  Error: Service name "vsphere-ui" is invalid.
2019-04-04T10:43:42.444Z  Error: Service name "eam" is invalid.
2019-04-04T10:43:42.464Z  Error: Service name "perfcharts" is invalid.
2019-04-04T10:43:42.519Z  Error: Service name "analytics" is invalid.
2019-04-04T10:43:42.539Z  Error: Service name "cis-license" is invalid.
Running:
lwsmd vmafdd vmcad vmdird vmdnsd vmware-pod vmware-sts-idmd vmware-stsd vmware-vmon

Tried to start them using "service-control --start --all", but it didn't work. 

I followed below steps to get it back running.. 

Make sure that you take a backup of your vCenter before trying this (Snapshot or Clone)

1) Stop all services

root@vcenter [ ~ ]# service-control --stop --all

2) Cd to the below dir

root@vcenter [ ~ ]# cd /storage/vmware-vmon/.svcStats

3) Create a new directory and move all .Jason files there 

root@vcenter [ /storage/vmware-vmon/.svcStats ]# ls
stats_analytics.json    stats_cm.json            stats_netdumper.json   stats_sps.json            stats_vcha.json     stats_vmware-postgres-archiver.json  stats_vpxd-svcs.json    stats_vsphere-client.json
stats_applmgmt.json     stats_eam.json           stats_perfcharts.json  stats_statsmonitor.json   stats_vdcs.json     stats_vmware-sca.json                stats_vsan-dps.json     stats_vsphere-ui.json
stats_autodeploy.json   stats_imagebuilder.json  stats_pschealth.json   stats_updatemgr.json      stats_vmcam.json    stats_vmware-vpostgres.json          stats_vsan-health.json
stats_cis-license.json  stats_mbcs.json          stats_rhttpproxy.json  stats_vapi-endpoint.json  stats_vmonapi.json  stats_vpxd.json                      stats_vsm.json

root@vcenter [ /storage/vmware-vmon/.svcStats ]# mkdir StatsjsonBKP

root@vcenter [ /storage/vmware-vmon/.svcStats ]# mv *.json /storage/vmware-vmon/.svcStats/mkdir StatsjsonBKP/

4) Do another ls to confirm

root@vcenter [ /storage/vmware-vmon/.svcStats ]# ls

mkdir StatsjsonBKP

5) Now it's time to start all services

root@vcenter [ ~ ]# service-control --start --all

Operation not cancellable. Please wait for it to finish...
Performing start operation on service lwsmd...
Successfully started service lwsmd
Performing start operation on service vmafdd...
Successfully started service vmafdd
Performing start operation on service vmdird...
Successfully started service vmdird
Performing start operation on service vmcad...
Successfully started service vmcad
Performing start operation on service vmware-sts-idmd...
Successfully started service vmware-sts-idmd
Performing start operation on service vmware-stsd...
Successfully started service vmware-stsd
Performing start operation on service vmdnsd...
Successfully started service vmdnsd
Performing start operation on profile: ALL...
Successfully started profile: ALL.
Performing start operation on service vmware-pod...

Successfully started service vmware-pod

Bang!! 

Gave few mins to start web client and all good!

Converting P2V of a Windows 2000 Server to vSphere 6.X

It's a shame but I recently had to P2V a Windows 2000 server, part of our legacy services, in order to free up and decomm an old physical server. It was not a pleasure at all :(

Planning

1) Download and keep VMware Converter 4.0.1 (VMware-converter-4.0.1-161434.exe) – To P2V Windows 2000

P2V Converter 4.0.1 doesn’t support any vCenter/ ESXi later than 4.x. So I ended up in installing ESXi 4.0 on a VM running on top of my vSphere 6.5 infra!

So 2) ESXi 4.0 ISO

I read articles of people using VMware WorkStation for this, but in my case I didn’t want the export job to run outside datacentre, due to security reasons.

My Windows 2000 Server that I need to virtualize is 200 Gig. So if you look at the disk size of ESXi VM, I made sure that It’s bigger than that. 

 

 

3) Check Patch Level of Windows 2000 server and make sure

> It’s updated to SP4 Rollup 1 (KB891861)

If not, install from here and reboot the server before running P2V job

4) Download and keep VMware Converter 6.0.0 (VMware-converter-en-6.0.0-2716716.exe) – To move Virtual machine from ESX 4.0 to vCenter 6.X

5) vmscsi-1.2.0.4.flp For troubleshooting - https://kb.vmware.com/s/article/100520

Steps I followed

1. 1)      Installed ESX4.0 on my VM that I created, configured it with an IP that is accessible from Windows 2000 server and also from my PROD 6.x vCenter
   2)      Installed Converter 4.0.1 on the Windows 2000 machine
   3)      Started P2V Conversion of the Windows 2000 machine to the virtual ESX 4.1
   Make sure you select  “SCSI” as Disk Controller and not “Preserve Source”
   
   
   4)      Once the conversion is complete, we now have the VM inside ESX 4. However as this is a nested setup, we can’t power on and test it.
   5)      Install VMware Converter 6.0.0 on a machine that has access to the ESXi 4 VM and your PROD vCenter. We will use V2V on this converter version to move the VM to our PROD 6.5
   Not tested with 6.7, but worst case, we can also download the files from the datastore and move to PROD vCenter!
   6)      Once done, power on and test.
   
   
   
   

You can then take a backup and do R&D stuff to keep VMware tools and Hardware up to date J

Issues that I’ve seen

BSOD with “KMODE_EXCEPTION_NOT_HANDLED” error while loading windows.

This usually happens if you haven’t installed the SP4 Rollup 1 (KB891861) on the source VM.

I could fix it, without having to re-run P2V. Steps below

1. 1)      Detach the C:\ Disk from the Windows 2000 VM where you see this issue after P2V
   2)      Connect this disk to any of your existing Windows VMs.
   3)      Go to disk management and bring the disk online and assign a drive letter
   4)      Use 7 Zip and Extract KB891861 and vmscsi-1.2.0.4.flp that we’ve downloaded.
   Look for scsiport.sys under KB891861 extract – version 5.00.2195.7059
         And for vmscsi.sys under vmscsi-1.2.0.4.flp extract – version 1.2.0.4 
   
   
          Make sure the versions are correct. 
   
   
   5)  Copy these two files to the VM where we have the disk attached.

1. 6)      Navigate to WINNT\System32\drivers and rename existing scsiport.sys and vmscsi.sys to “–old”
   7)      Copy the two new files.
   8)      Bring disk offline and move it back to Windows 2000 VM
   9)      Double check and make sure that the SCSI controller type is “BusLogic Parallel”
   10)   Power on the VM to test!