Chef Manage Signed SSL certificate

I have signed an open SSL certificate request and uploaded it to the chef server using the following documentation and signed it via an Active Directory Certificate Services Machine on the domain where the Chef server lives.

Chef documentation found here

Assisting interwebs documentation found here.

I can then connect to the chef manage server using a web browser and get what I am expecting, a trusted certificate (within the browser) using both Internet Explorer or using Google Chrome.

This is the output of the knife ssl check from the machine using ChefDK on Windows

knife ssl check

Configuration Info:

OpenSSL Configuration:
* Version: OpenSSL 1.0.1t  3 May 2016
* Certificate file: C:/opscode/chefdk/embedded/ssl/cert.pem
* Certificate directory: C:/opscode/chefdk/embedded/ssl/certs
Chef SSL Configuration:
* ssl_ca_path: nil
* ssl_ca_file: "C:/opscode/chefdk/embedded/ssl/certs/cacert.pem"
* trusted_certs_dir: "c:\\users\\svucich\\chef-repo-sv.local\\.chef\\trusted_certs"
WARNING: There are invalid certificates in your trusted_certs_dir.
OpenSSL will not use the following certificates when verifying SSL connections:

c:/Users/svucich/chef-repo-sv.local/.chef/trusted_certs/chef_sv_local.crt: unable to get local issuer certificate


TO FIX THESE WARNINGS:

We are working on documentation for resolving common issues uncovered here.

* If the certificate is generated by the server, you may try redownloading the
server's certificate. By default, the certificate is stored in the following
location on the host where your chef-server runs:

   /var/opt/opscode/nginx/ca/SERVER_HOSTNAME.crt

Copy that file to your trusted_certs_dir (currently: c:\users\svucich\chef-repo-sv.local\.chef\trusted_certs)
using SSH/SCP or some other secure method, then re-run this command to confirm
that the server's certificate is now trusted.

Connecting to host chef.sv.local:443
Successfully verified certificates from `chef.sv.local'`

So I am a little confused about why, when I run a bootstrap to a windows machine, I get the following error which prevents bootstrap.

================================================================================
Chef encountered an error attempting to load the node data for "windows-node4.sv.local"
================================================================================

Unexpected Error:
-----------------
OpenSSL::SSL::SSLError: SSL Error connecting to https://chef.sv.local/organizations/bravura/nodes/windows-node4.sv.local - SSL_connect returned=1 errno=0 state=error: certificate verify failed

Platform:
---------
x64-mingw32

[2017-02-01T14:42:07+13:00] ERROR: Running exception handlers
[2017-02-01T14:42:07+13:00] ERROR: Exception handlers complete
[2017-02-01T14:42:07+13:00] FATAL: Stacktrace dumped to c:/chef/cache/chef-stacktrace.out
[2017-02-01T14:42:07+13:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2017-02-01T14:42:07+13:00] FATAL: OpenSSL::SSL::SSLError: SSL Error connecting to https://chef.sv.local/organizations/bravura/nodes/windows-node4.sv.local - SSL_connect returned=1 errno=0 state=error: certificate verify failed
ERROR: Failed to execute command on node4.sv.local return code 1

So, I understand that the error is saying I have an SSL cert issue, but this is contrary to what I am seeing in the web browser. I would appreciate some assistance as this is the last hurdle before I can really get this working within our environmentā€¦

I get the feeling that openssl canā€™t verify that the certificate is correct, so I would assume that it is looking in the wrong place for the CA cert/file for trust verification - Which is incidentally aggregated into the main SSL cert as per documentationā€¦ So how do I ensure that the client machine has the correct root CA cert?

So - Further this, I can add the following to the end of my bootstrap command --node-ssl-verify-mode none and complete a bootstrap. But this is a work around and does not ACTUALLY resolve my issue.

So how do I ensure that the client/node machines accept the ROOT CA Cert and signed certificate as trusted as it seems they donā€™t use the same Certificate store as the web browsers doā€¦

The chef client needs to know about your CA certificate - it does not use the same CA store as your Web browser. Generally, the chef client finds the CA certificate from the entry

ssl_ca_file "/etc/somewhere/on/your/disk"

in your /etc/chef/client.rb file (or in your case the equivalent on Windows, of course) During the bootstrap process, it doesn't have that knowledge yet. Thus your error.

BTW, your chefdk machine seems to have the same problem (except that it may use a different directory, I'm not sure off the top of my head):

WARNING: There are invalid certificates in your trusted_certs_dir.
OpenSSL will not use the following certificates when verifying SSL connections:
c:/Users/svucich/chef-repo-sv.local/.chef/trusted_certs/chef_sv_local.crt: unable to get local issuer certificate

There are several different ways to solve this:

  • You can manually preinstall the CA certificate, the client.rb file and the chef client before bootstrapping. Actually, if you do that, you really don't need bootstrapping any more - just add the validator and run the chef-client.
  • Run the bootstrap with node-ssl-verify-mode=none. I see that you already did that, and understandably find it unsatisfactory.
  • Create a bootstrap cookbook that will use chef-zero and sets up the client the way you need it. In that case, you will need to find a way to install the chef client and copy the bootstrap cookbook.

Kevin Keane
Whom the IT Pros Call
The NetTech
http://www.4nettech.com
Our values: Privacy, Liberty, Justice
See https://www.4nettech.com/corp/the-nettech-values.html

1 Like

Thanks allot for the info! Seems the chef documentation is, once again, lacking all this information or does not have all of this info in one place. I will give the ssl_ca_file property/value a go.

So where is the value for ssl_ca_path: nil set? I have tried adding a direct setting in my knife.rb file but this has changed nothing.

Yes, knife.rb on your chef workstation is both the wrong file, and also the wrong computer. It needs to be in client.rb on the system you are trying to bootstrap. Thatā€™s actually where the chicken-and-egg problem is: the client.rb file is usually only created during or after the bootstrap process.

Kevin Keane
Whom the IT Pros Call
The NetTech
http://www.4nettech.com
Our values: Privacy, Liberty, Justice
See https://www.4nettech.com/corp/the-nettech-values.html

Well, it kind defeats part of the SSL security and also the ā€˜AUTOMATEDā€™ side of things.Chef needs a way I can customize the files that are placed onto the client machine via a bootstrap - And I donā€™t mean writing a cookbook and chicken and egging it. :smiley:

I have it working as expected on the node/client side, but I still get an error on the local chefdk machine when I run knife ssl fetch I know where the issue is, now to fix it.

Thanks again.

In the contrary, this is actually essential for SSL security. Remember, a bare-metal system, the node that isnā€™t bootstrapped yet, doesnā€™t know anything about the chef server, so initially, there is no way at all for it to trust the chef server.

So there are really only two ways around this:

You can tell the node to start the process without validating the authenticity of the server.

Or you can enable the node to trust the server before the install process starts.

The first option actually is generally secure. --node-ssl-verify-mode none does not turn off security. It does theoretically allow a man-in-the-middle attack, but during the bootstrap process, that risk is fairly low in the first place.All an attacker can see is the cookbooks, data bags etc. that are transferred - there are easier ways to see those Other than that, there is nothing interesting on the server yet. Or an attacker could redirect the chef client to a malicious chef server - but you would know this immediately when the node fails to show up in the chef server. Subsequent chef runs would authenticate the server.

Granted, the risk is non-zero, and for some applications, you do need a better guarantee. In that case, you have to tell the new node how to validate the server. Fortunately, that, too can be automated, and itā€™s not even very hard. The bootstrap process is actually very straightforward. All you need to do is re-implement it in a script that also includes configuring the CA. You can even include this script with an unattended install (I have it as part of my kickstart for CentOS).

Step by step, what this script needs to do is:

  • create the chef configuration directory.
  • set up a basic version of the client.rb file
  • add the validation.pem and the CA certificate.
  • install the chef-client
  • run chef-cllient, specifying whatever runlist you need for bootstrapping (mine simply is ā€œchef-client -r mainā€)

Kevin Keane
Whom the IT Pros Call
The NetTech
http://www.4nettech.com
Our values: Privacy, Liberty, Justice
See https://www.4nettech.com/corp/the-nettech-values.html

I donā€™t disagree about security and SSL certs - But you have made my point for me.There are to many steps to get this working out of the box - It should be easier to do - And much like the Starter kit for the Chef repo - To many little nuances that are either poorly documented or the documentation is not linked correctly/obviously that making this easy to roll out to other users is not easy. I should be able to customize the default client.rb that is deployed via the bootstrap without having to jump through this many hoops or effectively reinventing the wheel that should already be suitable for the job.

It reminds me of 'The IT Crowdā€™ - So easy, so effectiveā€¦

So, to automate this deployment, I need to automate the deployment of this software using cmd/bash scripts which we know donā€™t scale very well - Hence using a tool like Chef to ā€˜Automateā€™ thingsā€¦

Steps to get this going prior to being able to use chef to ā€˜automateā€™

  • create the chef configuration directory.
  • set up a basic version of the client.rb file
  • add the validation.pem and the CA certificate.
  • install the chef-client
  • run chef-cllient, specifying whatever runlist you need for bootstrapping (mine simply is ā€œchef-client -r mainā€)

As this is something every client would do assuming they have a signed certificate - Then why isnā€™t this documented or customization options available out of the box without these hoopsā€¦

And thatā€™s where you start looking at writing your own custom knife bootstrapper or using something like chef-provisioning, to mass register nodes with Chef.