vCenter Appliance WebClient down - Error: Service name is invalid

vCenter Appliance - WebClient down with "Error: Service name is invalid"

Last week I had to shutdown my lab vCenter 6.7. As it's just a lab I just powered it off instead of shutting down. After a few days I powered it back on when I wanted it back, and it booted without issues. But then WebClient wouldn't come up. Just says page cannot be displayed.

I logged in to the appliance via SSH and tried to list service status

root@vcenter [ ~ ]#  service-control --status --all
2019-04-04T10:43:41.796Z  Error: Service name "applmgmt" is invalid.
2019-04-04T10:43:41.816Z  Error: Service name "content-library" is invalid.
2019-04-04T10:43:41.837Z  Error: Service name "vcha" is invalid.
2019-04-04T10:43:41.857Z  Error: Service name "pschealth" is invalid.
2019-04-04T10:43:41.878Z  Error: Service name "sps" is invalid.
2019-04-04T10:43:41.898Z  Error: Service name "sca" is invalid.
2019-04-04T10:43:41.936Z  Error: Service name "vmcam" is invalid.
2019-04-04T10:43:41.956Z  Error: Service name "vmware-vpostgres" is invalid.
2019-04-04T10:43:41.976Z  Error: Service name "vsphere-client" is invalid.
2019-04-04T10:43:41.997Z  Error: Service name "vapi-endpoint" is invalid.
2019-04-04T10:43:42.017Z  Error: Service name "vmware-postgres-archiver" is invalid.
2019-04-04T10:43:42.037Z  Error: Service name "vsm" is invalid.
2019-04-04T10:43:42.057Z  Error: Service name "updatemgr" is invalid.
2019-04-04T10:43:42.078Z  Error: Service name "vmonapi" is invalid.
2019-04-04T10:43:42.098Z  Error: Service name "vsan-health" is invalid.
2019-04-04T10:43:42.119Z  Error: Service name "rbd" is invalid.
2019-04-04T10:43:42.156Z  Error: Service name "vpxd-svcs" is invalid.
2019-04-04T10:43:42.176Z  Error: Service name "statsmonitor" is invalid.
2019-04-04T10:43:42.264Z  Error: Service name "imagebuilder" is invalid.
2019-04-04T10:43:42.284Z  Error: Service name "vsan-dps" is invalid.
2019-04-04T10:43:42.304Z  Error: Service name "cm" is invalid.
2019-04-04T10:43:42.342Z  Error: Service name "mbcs" is invalid.
2019-04-04T10:43:42.362Z  Error: Service name "vpxd" is invalid.
2019-04-04T10:43:42.382Z  Error: Service name "netdumper" is invalid.
2019-04-04T10:43:42.403Z  Error: Service name "rhttpproxy" is invalid.
2019-04-04T10:43:42.423Z  Error: Service name "vsphere-ui" is invalid.
2019-04-04T10:43:42.444Z  Error: Service name "eam" is invalid.
2019-04-04T10:43:42.464Z  Error: Service name "perfcharts" is invalid.
2019-04-04T10:43:42.519Z  Error: Service name "analytics" is invalid.
2019-04-04T10:43:42.539Z  Error: Service name "cis-license" is invalid.
Running:
lwsmd vmafdd vmcad vmdird vmdnsd vmware-pod vmware-sts-idmd vmware-stsd vmware-vmon

Tried to start them using "service-control --start --all", but it didn't work. 

I followed below steps to get it back running.. 

Make sure that you take a backup of your vCenter before trying this (Snapshot or Clone)

1) Stop all services

root@vcenter [ ~ ]# service-control --stop --all

2) Cd to the below dir

root@vcenter [ ~ ]# cd /storage/vmware-vmon/.svcStats

3) Create a new directory and move all .Jason files there 

root@vcenter [ /storage/vmware-vmon/.svcStats ]# ls
stats_analytics.json    stats_cm.json            stats_netdumper.json   stats_sps.json            stats_vcha.json     stats_vmware-postgres-archiver.json  stats_vpxd-svcs.json    stats_vsphere-client.json
stats_applmgmt.json     stats_eam.json           stats_perfcharts.json  stats_statsmonitor.json   stats_vdcs.json     stats_vmware-sca.json                stats_vsan-dps.json     stats_vsphere-ui.json
stats_autodeploy.json   stats_imagebuilder.json  stats_pschealth.json   stats_updatemgr.json      stats_vmcam.json    stats_vmware-vpostgres.json          stats_vsan-health.json
stats_cis-license.json  stats_mbcs.json          stats_rhttpproxy.json  stats_vapi-endpoint.json  stats_vmonapi.json  stats_vpxd.json                      stats_vsm.json

root@vcenter [ /storage/vmware-vmon/.svcStats ]# mkdir StatsjsonBKP

root@vcenter [ /storage/vmware-vmon/.svcStats ]# mv *.json /storage/vmware-vmon/.svcStats/mkdir StatsjsonBKP/

4) Do another ls to confirm

root@vcenter [ /storage/vmware-vmon/.svcStats ]# ls

mkdir StatsjsonBKP

5) Now it's time to start all services

root@vcenter [ ~ ]# service-control --start --all

Operation not cancellable. Please wait for it to finish...
Performing start operation on service lwsmd...
Successfully started service lwsmd
Performing start operation on service vmafdd...
Successfully started service vmafdd
Performing start operation on service vmdird...
Successfully started service vmdird
Performing start operation on service vmcad...
Successfully started service vmcad
Performing start operation on service vmware-sts-idmd...
Successfully started service vmware-sts-idmd
Performing start operation on service vmware-stsd...
Successfully started service vmware-stsd
Performing start operation on service vmdnsd...
Successfully started service vmdnsd
Performing start operation on profile: ALL...
Successfully started profile: ALL.
Performing start operation on service vmware-pod...

Successfully started service vmware-pod

Bang!! 

Gave few mins to start web client and all good!