Vsphere Client 6.5
March 13, 2015 by: in: vSphere 6.x Architecture vSphere Certificate replacement and implementation is much easier than Center Server 5.1 or 5.5. In the past, you would have to replace each out of the endpoint certificates, for example vCenter Server, Single Sign On, Inventory Service, Web Client, and so forth. To simplify the process, VMware now uses a Reverse HTTP Proxy which will route traffic accordingly, meaning we only need to replace one certificate, instead of replacing all them in the previous version. There are 4 Solution Users in vSphere 6.x – vpxd, vpxd-extention, vsphere-webclient, and machine and you can replace each solution user certificate if you would like, however it’s no longer necessary thanks to the reverse proxy. As you may know, there have been architectural changes to vSphere 6.x as well. There are now two different deployment options, an Embedded Platform Services Controller (PSC), or an external PSC). With embedded nodes, you will have one Reverse HTTP proxy endpoint to replace, and with an external PSC you will have two endpoint certificates to replace.
VSphere 6.x also ships with its own internal certificate Authority called the VMCA – VMware Certificate Authority. The VMCA will issue or validate certificates and has two different implementation methods. You can either use it as your Root CA, which is the default configuration, or it can be used as a Subordinate CA which will be signed by an Enterprise CA. To manage the VMCA, you will use the certool.exe located in the following directories:.
Program Files VMware vCenter Server vmcad. /usr/lib/vmware-vmca/bin/certool Trusts are handled by the VMware Endpoint Certificate Store (VECS). This is great news because we no longer have to update the trusts between the endpoint when we replace the certificate, the VECS will do it all for us! VECS holds stores that contain certificates and their keys.
Managing datastore files in VMware vSphere 6.5 Managing files in VMware vSphere 6.5 officially kills support for the Windows vSphere client as now you can no longer connect to vSphere vCenter 6.5 with the vSphere client.
By default there are three stores, as shown above, each store has an entry for a Certificate + Key. To manage the VECS we will use VECS-CLI which is located in the following directories. To learn more about vecs-cli, please click here:. VMware vCenter Server vmafdd. /usr/lib/vmware-vmafd/bin/vecs-cli With these changes you have three different types of certificates which can be replaced. Machine SSL / Reverse Proxy Certificate.
CA Certificates (Trusted Root Certificates). Solution User Certificates Now we get to the good stuff.
The certificate replacement. Certificate Replacement with ‘Certificate-Manager’ the new SSL Automation tool The new certificate automation tool is called “ Certificate-Manager.bat” and is installed with vCenter by default. It’s located in Program Files VMware vCenter Server vmcad If you are using this tool, you do not have to interact with vecs-cli.exe or the certool.exe; when you run the Certificate-Manager tool you will see the options pictured below. The most widely used will be the first two options, which I will go through step by step.
Replace Machine (Reverse HTTP Proxy) Certificate with Custom Certificate. Replace VMCA Root certificate with custom signing certificate and replace all Certificates.
Replace Machine (Reverse HTTP Proxy) Certificate with Custom Certificate Step 1. Edit the certool.cfg file – template file for CSR The file is located here: Program Files VMware vCenter Server vmcad certool.cfg I would leave the IP address blank since VMware will be dropping support for this soon. IPs are supposed to change, so you really don’t want this in your certificate. Run the Certificate-Manager.bat tool. Program Files VMware vCenter Server vmcad certificate-manager.bat. Select Option 1 to “Replace Machine SSL certificate with Custom Certificates”. Enter your SSO password.
Select Option 1 to “Generate Certificate Signing Request(s) and key(s) for Machine SSL certificate”. Choose the path to write your CSR and Key Step 3. Sign your CSR. Your new CSR is in the folder you specified titled “machinessl.csr” with it’s corresponding key file. You then want to go get your CSR signed by your CA.
In my case, I am signing it with an internal Microsoft Certificate Authority. I will not provide these steps as they are the same for any previous version, but I will provide a KB article that outlines this process. Please click and go to the section titled “ Obtaining the certificate” steps 1-10. You will also need to download your root certificate or certificate chain which is outlined in the same KB above section “Obtaining the certificate” steps 14-20. Import Custom Certificate in place of Machine SSL certificate. Provide the path to the new certificate. Provide the path to the key.
Provide the path to your Root certificate That’s it! All end points will communicate through the Reverse HTTP proxy which uses this certificate. This is much more simple than before right!?
Replace VMCA Root certificate with Custom Signing Certificate and replace all Certificates (Using VMCA as a subordinate CA) Step 1. Edit the certool.cfg file – template file for CSR The file is located here: Program Files VMware vCenter Server vmcad certool.cfg I would leave the IP address blank since VMware will be dropping support for this soon. IPs are supposed to change, so you really don’t want this in your certificate. Run the Certificate-Manager.bat tool. Program Files VMware vCenter Server vmcad certificate-manager.bat.
Select Option 2 to “Replace VMCA Root certificate with Custom Signing Certificate and replace all Certificates”. Enter your SSO Password. Select Option 1 to “Generate Certificate Signing Request(s) and Key(s) for VMCA Root Signing certificate”. Choose the path to write your CSR and Key Step 3. Sign your CSR Note: Please see to create a template for the signing certificate. Your new CSR is in the folder you specified titled “rootsigningcert.csr” with it’s corresponding key file.
When you sign this certificate, make sure you select the template “Subordinate Certificate Authority” or an altered copy of this. Select Base 64 encoded and “Download certificate chain”. Open the new.p7b certificate and export both certs as base 64.
Click Next, Select Base-64 encoded, give the certificate a name, click Finish. Create a chain file called chain.cer by running the following command to concatenate the new leaf (vmca) certificate, and the root certificate. Copy vmca.cer+root64.cer chain.cer Step 4. Import Custom certificate(s) and key(s) for VMCA Root Signing certificate.
Go back to the certificate-manager tool and select option 2 to “Import Custom certificate(s) and key(s) for VMCA Root Signing certificate”. Provide the chain.cer. Provide the rootsigningcert.key. Select ‘ Y‘ Note: If you are using an embedded PSC then you are done, otherwise, please proceed to the steps below. (Only necessary if External PSC) Recycle your vCenter Server services by running the following commands: c: Program Files VMware vCenter Server bin service-control -stop -all c: Program Files VMware vCenter Server bin service-control -start -all Step 6. Go to your vCenter Server and run the certificate manager tool (C: Program Files VMware vCenter Server vmcad) Select Option 3 – Replace Machine SSL certificate with VMCA Certificate Step 7.
(Only necessary if External PSC) Go to your vCenter Server and run the certificate manager tool (C: Program Files VMware vCenter Server vmcad) Select Option 6 – Replace solution user certificates with VMCA Certificate Note: You may need to add your VMCA signing certificate to Trusted Publishers as shown below; I didn’t have to run through this step, but a few customer’s have had to do this. Once this is done your VMCA will act like a subordinate CA and provide CA signed certificates for your services. During the replacement, it will also regenerate all other certificates. There were a few more steps to this option, but it is still much easier than in vSphere 5.1 or vSphere 5.5. If you want to as a subordinate CA, please check out my other post!
Please feel free to leave questions, comments, or suggestions. Great guide, great job. I’m using the linux appliance and almost exactly the same.
I’m stuck though on importing the new certificates, seems to fail on importing the root certificate. I wonder if my windows encryption is too high (which follow current best practice). The offline root ca is RSA 4096 and sha256. The online issuing is not as high. Followed the advice on the templates and copying the offline root and online issuing into a combined file, but no joy. Everything was exported to base64. ——————————————————– You are going to replace Machine SSL cert using custom cert Continue operation: OptionY/N?: y Status: 0% Completed Publishing Root cert Status: 0% Completed Operation failed, performing automatic rollback Error while replacing Machine SSL Cert, please see /var/log/vmware/vmcad/certificate-manager.log for more information.
——————————————————– Log file shows nothing useful that i can see, except it dies on the first command after creating the backup store. Hi Sean, I tried a few things to get vCenter appliance to accept the certificates. Similar error posted seems to indicate a possible problem with chains. I did his patch whether had any effect I don’t know.
Finally, this a big one. VCenter accepts certificates into the store that use dhe-rsa (PKCS#1 v2.1) instead of plain rsa. It gives no error but services fail to start. I think they should check the certificate on entry into the system for compatible algorithms. Unfortunately dhe-rsa is the default in Windows PKI, so i had to downgrade my security slightly by making AlternateSignatureAlgorithm=0 in capolicy.inf.
Then reissue root and subordinate CAs and eventually – voila!!! I have green padlock on webclient. Many many thanks for this excellent guide. This answer really helped me! THANKS “AlternateSignatureAlgorithm=0 in capolicy.inf.” I think by default the capolicy.inf file does not natively exist because in a lab on a fresh install of AD CA Enterprise I didn’t find it, but on the environment where I was having the issues with this failure getting certificate-manager to install the machine cert (fought this for 2 weeks), I did find the file there. Funny how one simple switch can stop everything. Want to say thanks again as I would’ve never found that on my own.
VMware script gives no indication of the cause for failure. We have a two tiered Enterprise PKI in our AD environment with our root CA offline and essentially locked away. Our CAPolicy for AlternateSignatureAlgorithm is AlternateSignatureAlgorithm=1 so we’re running into the situation described above.
Vsphere Client 6.5 Windows
We’re a little gun-shy about firing up the root CA and changing the policy. Has anyone found an alternative to this? What are the risks to my existing production PKI? Also, when we created the new template for vSphere 6 in our existing CA, I see there under the “Cryptography” tab there are multiple choices for “Providers”, some are checked and others not.
Has anyone tried these as a solution? Thanks in advance. Fantastic guide!
One question – I have a certificate from Go Daddy I want to install. How do I get the certificate onto the vCenter Server Appliance (VCSA)? I have a nice.pfx file with the private key embedded, or I can certainly convert it to any format needed. But I can’t seem to figure out how to actually get the file onto the appliance! WinSCP used to work with the 5.5 appliance, but VCSA 6.0 hates it. I think it has to do with the new frontend VMWare set up, that makes you type “shell.set –enabled true” before you can type shell, and get access to bash.
Seems kind of silly to me, having the directions on how to get into the shell right there, but whatevs. Ftp is not in the bash on the VCSA.
Sftpd is also not listening. SCP doesn’t work either. The VCSA doesn’t have links to the datastores that I can find (no /vmfs/ directory). Thanks again! The Machine SSL updates went very easy but the VMCA replacement of ssl certs is not going at all. I keep getting error 70037 VMCAAddRootCertificatePrivate failed error message cert/key pair does not match I have remade them 10 times they are the only one in the driectroy as I clear all others out each time and still get this error was getting the Certificate Chain is not completebut.
Add the certificate to the VMware Endpoint Certificate Store any suggestions Like your writeup got me a lot further then trying to read vmware garbage. Hi James, I haven’t attempted it yet, but you may be able to edit the certool.cfg file to reflect the short name as well: Program Files VMware vCenter Server vmcad certool.cfg In previous version the template we used was below: I am not sure if certool.exe will like the changes, but you can attempt and let me know, the subject alt name contains the shortname in the example below. req defaultbits = 2048 defaultkeyfile = rui.key distinguishedname = reqdistinguishedname encryptkey = no prompt = no stringmask = nombstr reqextensions = v3req v3req basicConstraints = CA:FALSE keyUsage = digitalSignature, keyEncipherment, dataEncipherment extendedKeyUsage = serverAuth, clientAuth subjectAltName = DNS: vc55-1, IP:10.0.0.10, DNS:vc55-1.vmware.com reqdistinguishedname countryName = US stateOrProvinceName = NY localityName = New York 0.organizationName = VMWare organizationalUnitName = vCenterUniqueServer commonName = vc55-1.vmware.com. Hi James, Hi Jean, in my case the variables IPAddress and Hostname of the certool.cfg were interpreted as SubjectAltName. Our CA generated a certficate file with: altnames = DNS:, DNS:, DNS:. However, i had to run the certool directly with: /usr/lib/vmware-vmca/bin/certool –genkey –privkey /root/machinessl.key –pubkey /root/machinessl.pub /usr/lib/vmware-vmca/bin/certool –gencsr –privkey /root/machinessl.key –pubkey /root/machinessl.pub –csrfile /root/machinessl.csr –config /root/certool.cfg In my case the certificate-manager command did not take the default certool.cfg into account. Well, something broke overnight.
VMotion is dead. VMs are stuck in their folders, and can’t be migrated. Is it possible that I use a UCC Cert that has up to 30 Subject Alternative Names? I got one from GoDaddy – and I generated my CSR from an Exchange Server – added the SAN’s for all my servers. I imported the certs provide by GoDaddy – and then exported a cert with private key from the Exchange Server. I’ve used this certificate on all the servers listed in the Subject Alternative Name line with no issues, but I’m confused on how to use it on the vCenter server.
This is a new cluster we are firing up for VDI – and it seems Horizon View requires vCenter server to have a non self-signed cert. In the past I only used the GoDaddy cert on my View Connection/Security servers. My confusion comes from where do I get the key file if I’m not generating the CSR from the vcenter appliance? Hi John, I’m using the same thing from GoDaddy, but with only 9 SANs (the actual name makes 10 devices).
I have it working on 10 devices, including the green padlock on vCenter, thanks to Sean’s brilliant thread here. I should say, I.had. it working, but it blew up. I had to roll back, and I haven’t had time to play with it since. I don’t think the problems were related to the certificate, but until I get it working again, I can’t be sure. I give it a 95% probability though – certainly enough to give it a shot.
Just be sure the DNS (public and private) FQDN of your vCenter server is one of the SANs. The physical hosts need not be on the list. One more thing, I chose option 2 (Replace VMCA Root certificate with Custom Signing Certificate and replace all Certificates (Using VMCA as a subordinate CA) on primary psc node, I believe it will replace the vmca and all certificates with custom certs, Do i need to replace the Machine SSL separately on the same node or option will take care of certs replacement. And do i need to run option 2 on secondary psc node as well (using external psc) to replace certs there as well.
I acknowledge the option 3 to 6 will be used on external vCenter. Thanks beforehand,. Sean, I have a situation where I have deployed a brand new VCSA 6.0U1. Chosen to use IP address instead of FQDN. But then post build, notice ‘localhost.localdom’ applied as a hostname of the appliance.
If I wanted to change that, I can change it via ‘/opt/vmware/share/vami/vamisethostname’ and then need to re-issue all certs with cert manager. Problem is, OVF deployments fail at point of validating storage location during OVF deployment wizard. And I notice vpxd.log still references the old cert for the STS.
(Where the ‘Retrieved trusted STS certificate: O=localhost.localdom,C=US,DC=local,DC=.my SSO site name.,CN=CA,’) One can change the hostname of an VCSA deployed via IP? This is for lab purposes and I have no DNS. I just want to change the ugly presence and DNS propagation of ‘localhost.localdom’ of the OS. Hello, very nice article i reinstallted VMCA certificate however after that i’m receving error in my vcenter 6 U1 web client: Error occurred while processing request. Check vSphere WebClient logs for details. Vmware tells: “To resolve this issue, ensure that you have updated the SSL Trust anchors of the vCenter Server and the Platform Services Controller stored in the Lookup Service after you have updated your SSL certificates” Question: do replacement VMCA Root certificate with Custom Signing Certificate and replace all Certificates also replaces Lookup Service SSL?
Hi Sean, Many, many thanks for the great articles (also replacing certificates on ESXi servers). For me everything run fine except the certificate from VAMI (web console on port 5480). Do you know if there is an issue or if there is a separate procedure for this particular component? My environement: – vSphere server appliance 6 update 2 – Custom vmware CA certificates then (from AD CS) – ESXi hosts: vSphere 6 update 2 as well That is my only problem., everything else run smoothly 🙂 Thanks a lot PS: I’ve noted with a test appliance that if you take the same certificate name/description for the intermediate VMCA and all other generated certificates, it might give the problem of non trusted certificate than Brian mentionned (April 4, 2015 – 7:59 pm). You have to give a different name for the VMCA and the generated certificates. Might be useful for some others 😉. Hi Sean, Excellent post, thank you.
I recently upgraded my existing vCenter from 5.5 to 6.0 U2 in an embedded PSC environment on Windows. I would like to know which steps must be performed in order to make the VMCA to act as a subordinate CA for the VMware infrastructure. I followed the procedure indicated above to “Replace VMCA Root certificate with custom signing” but when finished and all services restarted, I have experienced several issues on my clusters being disconnected The same happened with the Veeam integration; it looks like nobody was able to communicate correctly. Finally, I have had to roll back the action.
Just a side note, in my implementation of Certificate Manager I found more questions that are not posted in neither blog, for example running option 2 the app asks me if I wish to generate all certificates using the configuration file, and later (after SSO) it asks for reconfiguring all CFG files individually (MACHINESSLCERT.cfg file exists and so on). Is it normal? Thanks again.
I updated the machinessl certificate using certificate manager which replaced the certificate. I verified using the following 2 commands: 1. Vecs-cli.exe entry getcert –store machinesslcert –alias MACHINECERT 2.
Openssl.exe sclient -servername servername.company.com -connect localhost:443 -showcerts However, only the host cert was listed and not the full chain (intermediate and root certs). I had to manually replace the certificate using the following commands: Delete certificate: vecs-cli.exe entry delete –store MACHINESSLCERT –alias MACHINECERT Create certificate: vecs-cli entry create –store MACHINESSLCERT –alias MACHINECERT –cert c: certs cachain.crt –key c: certs machinesslpriv.key Restart virtualcenter. This correct the chain issue for port 443. However, I’m having the same problem with port 9443. It appears to be successfully replaced using the certificate manager, however, only the host cert is listed, and not the full chain. This is evident when I run the following command: openssl.exe sclient -servername servername.company.com -connect localhost:9443 -showcerts Do you know how I can manually replace the certificate for port 9443 to insure that it uses the cert with the full chain? Thanks in advance.
Sean, this is an excellent article. Good work on that. Have a question related to the design. I am working on designing vmca for our vsphere 6 stuff. We have 14 environments with 2 psc’s and 1 vcenter for each environment that belong to same domain at the moment. In above scenario, if i say all 28 psc’s combined acts as a subordinate CA for their own environment, it might worry our security teams. Just wondering if we can have 2 dedicated appliances that does the subordinate ca job for all the environments.
Is that even possible? Hey, great writeup! So I sucessfully did the first portion of your article “Replace Machine (Reverse HTTP Proxy) Certificate with Custom Certificate”. Now I can log into the vSphere web client without any certificate errors, great! However now if I check vSphere update manager, or use the C# client to my 6.0 vCenter server, I get an error about vSphere update manager certificate untrusted, and now VUM does not work. Also vSphere replication now says error – configuration required, and I fear SRM is hosed too. I did this first step with our Microsoft CA on both our vcenter and DR vcenter which we replicate to.
Do I have to also do this portion “Replace VMCA Root certificate with Custom Signing Certificate and replace all Certificates (Using VMCA as a subordinate CA)”? I’m stuck here because when creating the request it seems to repeat itself over and over again (asking for country, name, organization, etc) and I answer each question exactly the same (though everything is populated from that cfg file except for the FQDN at the end. Then it says vpxd.cfg file exisits, do you with so reconfigure? I don’t know if I should answer yes to that at this point? Hi guys, I have problem with deleting expired certificate in TRUSTEDROOTS keystore. (which issued in my CA) Thaed expired certificate trigers critical alarm in vCenter. Example from xpxd.log there: 2016-10-14T17:22:28.570+02:00 warning vpxd05988 Originator@6876 sub=Main opID=CheckCertificateExpiry-427c3c55 Vpxd::VecsUtil::CheckCertificatesFromStore Certificate Subject:xxx from store TRUSTEDROOTS will expire on 2016-07-31 19:37:46.000.
I found that this certificate is stored in TRUSTEDROOTS with commands: vecs-cli store list vecs-cli entry list –store TRUSTEDROOTS –text Then when I try to remove the certificate from store: vecs-cli entry delete –store TRUSTEDROOTS –alias a9a6c9e157d65f176c8d83c616d61e certificate is deleted but it comes back after a moment Do you know how to remove it permanently? Regards Kris.
Big problem with the vcsaU1 right now. I’ve created csr’s for all services and uploaded certs back to vcsa. Cert-manager will not install any of them.
They all fail and rollback fails both on machine certs as well as ALL solutions certs. Also NOW it fails to recreate VMCA cert and also fails on a “reset all certificates” attempt. There must be a manual way to clean up files and regenerate self signed certs or upload those signed by our CA.
Vsphere Client 6.5
We are currently unable to connect to the vcenter with our vdi and this is now our #1 priority. Any immediate help is appreciated.
Dear Sean, thank you for all your work and effort you put into this. You seem to have good knowledge and understanding of this topic, which is very helpful. I have been working with vSphere for almost three years now, and unfortunately I’m new to certificates as unexceptionally all clients I’ve been working with have been using self-signed certificates and – as far as I can tell – they had no intention to change this.
Until now On Monday I have upgraded a client’s environment with 80 hosts and roughly 800 VMs to v6.0 U3 (latest). Now, due to security concerns, the client needs to have all self-signed certificates replaced by certificates issued by their inhouse root CA. So far, so good. I understand that much has changed since 5.1 or 5.5 and that by the use of the tool “certificate manager” many things have become easier. My problem is, that I don’t understand the difference between all this.
The wizard offering 1., 2., 3. And so on is not as inuitive to me as it probably is to you. Here we use the VCSA and the VMCA is not supposed to be used as a SubCA. So I understand from KB article 2097936 that in this case the internal CA is bypassed as all certificates will have to be issued directly by the inhouse root CA and there are no cert chains. Now when I need to replace all certificates, which I’m afraid I’ll have to do, then what? I really tried to understand what I need to do, but I just couldn’t. In this context I can’t even tell what the Machine SSL certificate is, or the Solution user certificate.
Everything I did btw. Produced.csr files that only had 2048 bit encryption, but our inhouse root CA guy says he needs 4096 bit and I have failed to find a solution for that, too. Have been looking for this but couldn’t find an answer so maybe we’ll have to go with 2048 bit – it’s becoming more and more annoying:-/ so I was hoping and thus would kindly ask you what needs to be done in order to have all certificates replaced in terms of selecting what number in the wizard exactly. Sorry for this long post, and thank you in advance. Hello Sean, excellent article, thanks, I have an issue in my environment that I hope you can help me in someway to figure it out, environment: 1 windows server vcenter server 6.0u2 with the PSC embedded.
Overview: The vSphere Command-Line Interface (vSphere CLI) command set allows you to run common system administration commands against ESXi systems from any machine with network access to those systems. You can also run most vSphere CLI commands against a vCenter Server system and target any ESXi system that vCenter Server system manages. VSphere CLI also includes a set of host management commands: the ESXCLI command set, vicfg- commands, and some other commands. VSphere CLI also includes DCLI commands, which allow you to manage services in the vSphere REST API and are presented by the vSphere Automation SDK interfaces.
The vSphere Command-Line Interface (vSphere CLI) command set allows you to run common system administration commands against ESXi systems from any machine with network access to those systems. You can also run most vSphere CLI commands against a vCenter Server system and target any ESXi system that vCenter Server system manages. VSphere CLI also includes a set of host management commands: the ESXCLI command set, vicfg- commands, and some other commands. VSphere CLI also includes DCLI commands, which allow you to manage services in the vSphere REST API and are presented by the vSphere Automation SDK interfaces.
