[Guide] VMware vCenter Troubleshooting Guide

calendar Aug 23, 2024 · 6 min read · vCenter VCF  ·
Share on: linkedin copy

Overview

vCenter Troubleshooting Guide

The following Troubleshooting Guide applies to vCenter 8 and above.

Caution!

Create a VM Snapshot of your vCenter before proceeding with any step!

General Troubleshooting

Log files can be accesses trough ssh or bash. SSH login works with the root user or your SSO Admi User (commonly [email protected])

SSH not working

If SSH does not work (eg. due to a networki misconfiguration) you can access the vCenter via bash (VM Remote Console).

At the VM Remote Console press F2 to access your vCenter or press ALT + F1 to access the bash shell.

Services

If you have trouble starting services and the GUI is not working, use service-control with ssh to get all the running services:

1service-control --status

To stop all services

1service-control --stop --all

To start all services:

1service-control --start --all

Log Files

All vCenter Log Files are stored under:

1/var/log/vmware/
2/var/log/vmware/<service_name>

vCenter logs are grouped by component and purpose:

  • vpxd.log - Main vCenter Server log for client connections, tasks, and host communication5
  • vpxd-profiler.log - Profiled metrics for vCenter operations
  • eam.log - ESX Agent Manager logs
  • sms.log - Storage Monitoring Service logs
  • ls.log - Licensing Services logs

Some key log files and directories include:

  • /var/log/vmware/vpxd/vpxd.log - The main vCenter Server log
  • /var/log/vmware/vsphere-ui/ - vSphere UI logs
  • /var/log/vmware/eam/eam.log - ESX Agent Manager log
  • /var/log/vmware/eam/applmgmt.log - Appliance Management Service

Additional Important Logs

  • /var/log/vmware/vpostgres/ - VMware Postgres service logs
  • /var/log/vmware/vcha/ - vCenter High Availability service logs
  • /var/log/vmware/rhttpproxy/ - VMware HTTP Reverse Proxy service logs
  • /var/log/vmware/content-library/ - VMware Content Library Service logs
  • /var/log/vmware/applmgmt/backup.log - VCSA Backup Log

GUI

vCenter Server logs can be viewed from:

  • Home > Administration > System Logs.

Ways to access your vcenter

GUI (VAMI)

'https:// vcenterfqdn :5480'

SSH

Login via SSH (root user or SSO User)

DCUI

Open the VM Remote Console of the vCenter VM (via ESXi) and Press F2. After a successfull login you will be able to edit the config via DCUI.

DCUI can also be opened from ssh.

Bash shell

To Access the Bash Shell: Open the VM Remote Console of the vCenter VM (via ESXi) and Press ALT+F1.

If you need to enable bash shell, do the following first:

  • Open the VM Remote Console of the vCenter VM (via ESXi) and Press F2. After a successfull login go to troubleshooting options and enable bash or ssh

Bash shell (Emergency Boot)

Look at here

Networking

Manual Network Config

Run the following command to change Networking Settings from BASH

1/opt/vmware/share/vami/vami_config_net

Alternatively edit the configuration files directly, to change network settings like IP address and netmask

1/etc/systemd/network/10-eth0.network

Example:

 1[Match]
 2Name=eth0
 3
 4[Network]
 5Address=192.168.1.100/24
 6Gateway=192.168.1.1
 7DNS=8.8.8.8
 8DNS=8.8.4.4
 9
10[Route]
11Destination=10.0.0.0/24
12Gateway=192.168.1.254

to modify routing information

1/etc/sysconfig/network/routes

Example

1default 192.168.1.1
210.0.0.0/24 192.168.1.254
3172.16.0.0/16 192.168.1.253

Each line follows this format: ' '

or use the builtin DCUI from the bash shell

1dcui

DNS

Manual DNS Config

1vi etc/systemd/resolved.conf

1[Resolve] 
2DNS=10.10.10.125
3FallbackDNS=10.10.10.126
4Domains=yourdomain.local

or use the builtin DCUI from the bash shell

1dcui

Flush DNS

1
2systemctl restart systemd–resolved.service
3
4systemctl restart dnsmasq

HTTP Proxy

Log File: /var/log/vmware/rhttpproxy/

To set a HTTP Proxy run:

1/opt/vmware/share/vami/vami_config_net

It is also possible to edit the file directly:

1vi /etc/sysconfig/proxy

Example:

1PROXY_ENABLED="yes"
2
3HTTP_PROXY="http://proxy.example.com:8080"
4HTTPS_PROXY="http://proxy.example.com:8080"
5
6NO_PROXY="localhost, 127.0.0.1, .*.example.com, 192.168.0.0/24, 192.168.1.55"

Certificate Management

Log File: /var/log/vmware/vmcad/certificate-manager.log

Run this command to check certificate expiration dates:

1for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do 
2  echo "[*] Store :" $store
3  /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | grep -ie "Alias" -ie "Not After"
4done

Renew all vCenter Machine Certificates

Generally this should be done of all VMCA Certificates are self-signed and are expired. Later you can change the certificates to named ones from your Enterprise PKI.

Run vCenter Certificate Manager:

1/usr/lib/vmware-vmca/bin/certificate-manager

Choose your desired option. Most commonly option 4 or 8 are used.

This step automatically restarts the vCenter Server services. Additionally, the Name, Hostname, and VMCA values should match the Primary Network Identifier (PNID). The PNID should always match the Hostname.

To get the vCenter PNID:

1/usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost

VMware Security Token Service (STS) certificate

https://knowledge.broadcom.com/external/article?legacyId=79248

Fix trust issues - lsdoctor

Use the lsdoctor tool from VMware: https://knowledge.broadcom.com/external/article?legacyId=80469

Upload and unzip lsdoctor on vCenter

Check for issues

1python lsdoctor.py -l

to fix trust mismatches

1python lsdoctor.py -t

vSphere HA

Log File: /var/log/fdm.log

vSphere Lifecycle Manager (vLCM)

Log File: /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log

1cat /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log | grep "error"

Error cannot sync depot

A usual error is that vCenter is unable to reach vmwaredepot. Check DNS, Firewall and HTTP Proxy Settings. The log file indicated it the following:

 1-->     "error_type": "ERROR",
 2-->     "messages": [
 3-->         {
 4-->             "args": [],
 5-->             "default_message": "A depot is inaccessible or has invalid contents. Make sure an official depot source is used and verify connection to the depot.",
 6-->             "id": "com.vmware.vcIntegrity.lifecycle.depotContent.ValidationError"
 7-->         }
 8-->     ]
 9--> }"
10--> }

Check if you can access VMware Online Depots, from vCenter run:

1 curl -vvv https://hostupdate.vmware.com

You should see a DigiCert Certificate printed:

1*  issuer: C=US; O=DigiCert Inc; CN=DigiCert TLS RSA SHA256 2020 CA1

If not, check with your Firewall Team if they do TLS Intercepts.

Also check your connected depots. Maybe there is a old depot still configured (like a old HPE OneView Instance).

HPE OneView for vCenter

Check

1cat /var/log/vmware/envoy/envoy-access.log | grep "hpe"

If you see some SSL Errors like:

1failed: cURL Error: SSL peer certificate or SSH remote key was not OK, SSL certificate problem: certificate has expired

Verify if the required certs for HPE OneView for vCenter are valid.

Further Troubleshooting Information

Backup and Restore

Backup

Log files:

  • /var/log/vmware/applmgmt/backup.log
  • /var/log/vmware/applmgmt/backupscheduler.log
  • /var/log/vmware/applmgmt/backupschedulercron.log

vCenter Appliance File-Restore

See: VMware Docs

If during a File Level Restore something went run, check your logs at:

/var/log/vmware/applmgmt/restore.log

Restore fails - unable to ssh into machine

If your vCenter Restore has failed and you aren't able to ssh into the machine. Try all the ways to access your vCenter as described in here

vCenter and ESXi

Disconnect ESXi if vCener no longer exists

1cmsso-util unregister --node-pnid vcenter.domain.com --username [[email protected]](mailto:[email protected]) --passwd pw

Forgot Root Password

Simple Method: Login with SSO User

If you forgot your root password or it is expired you can still access the vCenter via GUI or SSH with the SSO Admin User (usually [email protected])

1sudo passwd root

Advanced root password restore

  • Emergency boot the vCenter described below
  • unlock the root account
1/usr/sbin/faillock --user root --reset

After that you can set a new password

1passwd

Then unmount the filesystem and reboot

1umount / 
2reboot -f

Emergency Boot Edit GRUB Bootloader

If you cannot SSH into or open DCUI/Bash Shell (via VM Remote Console) to your vCenter you can access the vCenter without a password the following:

  • Access VM Remote Console of the vCenter VM
  • Reboot the VM
  • press 'e' for emergency mode
  • GRUB Bootloader will appear, edit the boot loader the following:
    • Append these entries to the end of the line of '...consoleblank=0' with: - 'rw init=/bin/bash'
  • Proceed to boot the vCenter VM with pressing F10
  • After booting you should be presented with a shell
  • run:
1mount -o remount,rw /

After that you can edit configuartion files as needed (in /etc/sysconfig for example)

Make sure you unmount the filesystem and reboot after you have done your configuration

1umount / 
2reboot -f