Issue Description

• EAM Status is showing in “Starting” mode.
• NSX Version is showing as “Not Running”.
• It is showing error “Error occurred while communicating with EsxAgentManager.”

Troubleshooting Performed:

Vmware-eam service is already in running status, showing in VCSA putty console.

• Check checked event logs of NSX, it is showing that Certificate ssoserverSign has expired.
  • Code: 360001
  • Event Source: Certificate Monitoring Service

When checked certificate status in NSX Manager console, it is showing that certificate is Valid till 18^th Jan 2029.

STS Certificates are also not expired.

Solution

Checking into the logs I found the following output:

2021-05-11T19:01:50.121Z | INFO | vim-monitor | VcConnection.java | 635 | Connecting to https://vcenter.pjlab.local:8089/sdk/vimService via vCenter proxy http://localhost:80

2021-05-11T19:01:54.151Z | ERROR | vim-monitor | VcConnection.java | 210 | Failed to login to vCenter as extension.

vCenter has probably not loaded the EAM extension.xml yet.: Cannot complete login due to an incorrect user name or password.

2021-05-11T19:01:54.151Z | WARN | vim-monitor | VcListener.java | 110 | Trying to recover from error

2021-05-11T19:02:04.157Z | INFO | vim-monitor | VcConnection.java | 635 | Connecting to https://vcenter.pjlab.local:8089/sdk/vimService via vCenter proxy http://localhost:80

2021-05-11T19:02:08.196Z | ERROR | vim-monitor | VcConnection.java | 210 | Failed to login to vCenter as extension.

vCenter has probably not loaded the EAM extension.xml yet.: Cannot complete login due to an incorrect user name or password.

2021-05-11T19:02:08.197Z | WARN | vim-monitor | VcListener.java | 110 | Trying to recover from error

Run below commands to fix this issue:

Command> shell.set --enabled true

Command> shell

root@vcenter [ ~ ]# mkdir /certificate

root@vcenter [ ~ ]# /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.crt

root@vcenter [ ~ ]# /usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.key

root@vcenter [ ~ ]# python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.vim.eam -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s vcenter.pjlab.local -u Administrator@vsphere.local

root@vcenter [ ~ ]# service-control --stop vmware-eam

root@vcenter [ ~ ]# service-control --start vmware-eam

Step by Step:

Login to vCenter Server Appliance through Putty console, and run above commands as showing below.

------------------------

login as: root

Command> shell.set --enabled true

Command> shell

Shell access is granted to root

root@vcenter [ ~ ]# mkdir /certificate

root@vcenter [ ~ ]# /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.crt

root@vcenter [ ~ ]# /usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.key

root@vcenter [ ~ ]# python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.vim.eam -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s vcenter.pjlab.local -u Administrator@vsphere.local

Password to connect to VC server for user="Administrator@vsphere.local":

2021-05-14T05:55:15.464Z   Updating certificate for "com.vmware.vim.eam" extension

2021-05-14T05:55:15.542Z   Successfully updated certificate for "com.vmware.vim.eam" extension

Traceback (most recent call last):

  File "/usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py", line 173, in <module>

    update_extension_cert_in_VC()

  File "/usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py", line 161, in update_extension_cert_in_VC

    sessionMgr = si.content.sessionManager

  File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 551, in __call__

    return self.f(*args, **kwargs)

  File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 371, in _InvokeAccessor

    return self._stub.InvokeAccessor(self, info)

  File "/usr/lib/vmware/site-packages/pyVmomi/StubAdapterAccessorImpl.py", line 23, in InvokeAccessor

    return self.InvokeMethod(mo, info, (prop,))

  File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1409, in InvokeMethod

    conn.request('POST', self.path, req, headers)

  File "/usr/lib/python2.7/httplib.py", line 1082, in request

    self._send_request(method, url, body, headers)

  File "/usr/lib/python2.7/httplib.py", line 1122, in _send_request

    self.endheaders(body)

  File "/usr/lib/python2.7/httplib.py", line 1078, in endheaders

    self._send_output(message_body)

  File "/usr/lib/python2.7/httplib.py", line 894, in _send_output

    self.send(msg)

  File "/usr/lib/python2.7/httplib.py", line 856, in send

    self.connect()

  File "/usr/lib/python2.7/httplib.py", line 1303, in connect

    server_hostname=server_hostname)

  File "/usr/lib/python2.7/ssl.py", line 393, in wrap_socket

    _context=self)

  File "/usr/lib/python2.7/ssl.py", line 641, in __init__

    self.do_handshake()

  File "/usr/lib/python2.7/ssl.py", line 878, in do_handshake

    match_hostname(self.getpeercert(), self.server_hostname)

  File "/usr/lib/python2.7/ssl.py", line 316, in match_hostname

    % (hostname, dnsnames[0]))

ssl.CertificateError: hostname 'sdkTunnel' doesn't match u'vcenter.pjlab.local'

root@vcenter [ ~ ]# service-control --stop vmware-eam

Perform stop operation. vmon_profile=None, svc_names=['vmware-eam'], include_coreossvcs=False, include_leafossvcs=False

Successfully stopped service eam

root@vcenter [ ~ ]# service-control --start vmware-eam

Perform start operation. vmon_profile=None, svc_names=['vmware-eam'], include_coreossvcs=False, include_leafossvcs=False

2021-05-14T06:05:44.363Z   Service eam state STOPPED

Successfully started service eam

root@vcenter [ ~ ]#

 -------------------------------

After running these commands, EAM service should be showing in Running state in NSX Dashboard.