Struggling mightily with SSL on Windows

I’m struggling mightily getting SSL certs working for our new PROD Chef Server and would appreciate any assistance.

I’ve got a cert with CN = CHEF1.mydomain.com, and SANs of CHEF1.mydomain.com and CHEF.mydomain.com. It’s from a private, internal CA, and there are an intermediate cert and a Root cert that form the chain.

I concatenated all of the certs into a single file that I placed out at /var/opt/opscode/nginx/ca/CHEF.crt and restarted my Chef Server.

I’m able to successfully browse to https://chef.mydomain.com, and there are no security errors in the browser.

On my workstation, I do a knife ssl fetch, and it looks like knife is taking my concatenated certs and expanding them out into individual crt files for the component parts:

PS C:\repo> knife ssl fetch
WARNING: Certificates from chef1.mydomain.com will be fetched and placed in your trusted_cert
directory (c:\repo\.chef\trusted_certs).

Knife has no means to verify these are the correct certificates. You should
verify the authenticity of these certificates after downloading.

Adding certificate for CHEF1.mydomain.com in c:\repo\.chef\trusted_certs/CHEF1_mydomain_com.crt
Adding certificate for RootCA in c:\repo\.chef\trusted_certs/RootCA.crt
Adding certificate for IssuingCA01 in c:\repo\.chef\trusted_certs/IssuingCA01.crt

Doing a knife ssl check, I do see some warnings still:

PS C:\pop-repo> knife ssl check

Configuration Info:

OpenSSL Configuration:
* Version: OpenSSL 1.0.1t  3 May 2016
* Certificate file: C:/opscode/chefdk/embedded/ssl/cert.pem
* Certificate directory: C:/opscode/chefdk/embedded/ssl/certs
Chef SSL Configuration:
* ssl_ca_path: nil
* ssl_ca_file: "C:/opscode/chefdk/embedded/ssl/certs/cacert.pem"
* trusted_certs_dir: "c:\\repo\\.chef\\trusted_certs"
WARNING: There are invalid certificates in your trusted_certs_dir.
OpenSSL will not use the following certificates when verifying SSL connections:

c:/repo/.chef/trusted_certs/CHEF1_mydomain_com.crt: unable to get local issuer certificate
c:/repo/.chef/trusted_certs/IssuingCA01.crt: unable to get local issuer certificate

TO FIX THESE WARNINGS:

We are working on documentation for resolving common issues uncovered here.

* If the certificate is generated by the server, you may try redownloading the server's certificate. By default, the certificate is stored in the following location on the host where your chef-server runs:

  /var/opt/opscode/nginx/ca/SERVER_HOSTNAME.crt

Copy that file to your trusted_certs_dir (currently: c:\repo\.chef\trusted_certs)
using SSH/SCP or some other secure method, then re-run this command to confirm that the server's certificate is now trusted.

Connecting to host chef1.mydomain.com:443
Successfully verified certificates from `chef1.mydomain.com'

When I try to bootstrap a node, however, it errors out. It does not look like knife is creating the trusted_certs directory on the node or copying the certs from the workstation or server to the node.

knife bootstrap windows winrm 10.210.1.112 --winrm-user 'username' --winrm-password 'password' --node-name mymachine.mydomain.com --bootstrap-no-proxy '*.mydomain.com' --msi-url 'https://repo.mydomain.com/chef/chefclient/chef-client-12.16.42-1-x64.msi'

10.210.1.112 [2016-11-15T14:54:48-08:00] INFO: *** Chef 12.16.42 ***
10.210.1.112 [2016-11-15T14:54:48-08:00] INFO: Platform: x64-mingw32
10.210.1.112 [2016-11-15T14:54:48-08:00] INFO: Chef-client pid: 208
10.210.1.112 [2016-11-15T14:54:53-08:00] ERROR: SSL Validation failure connecting to host: popchef1.pop.portptld.com - SSL_connect returned=1 errno=0 state=error: certificate verify failed
10.210.1.112
10.210.1.112 ==================================================================
==============
10.210.1.112 Chef encountered an error attempting to load the node data for "mymachine.mydomain.com"
10.210.1.112 ==================================================================
==============
10.210.1.112
10.210.1.112 Unexpected Error:
10.210.1.112 -----------------
10.210.1.112 OpenSSL::SSL::SSLError: SSL Error connecting to https://chef1.mydomain.com/organizations/pop/nodes/mymachine.mydomain.com - SSL_connect returned=1 errno=0 state=error: certificate verify failed
10.210.1.112
10.210.1.112 Platform:
10.210.1.112 ---------
10.210.1.112 x64-mingw32
10.210.1.112
10.210.1.112 [2016-11-15T14:54:53-08:00] ERROR: Running exception handlers
10.210.1.112 [2016-11-15T14:54:53-08:00] ERROR: Exception handlers complete
10.210.1.112 [2016-11-15T14:54:53-08:00] FATAL: Stacktrace dumped to c:/chef/cache/chef-stacktrace.out
10.210.1.112 [2016-11-15T14:54:53-08:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
10.210.1.112 [2016-11-15T14:54:53-08:00] FATAL: OpenSSL::SSL::SSLError: SSL Error connecting to https://chef1.mydomain.com/organizations/pop/nodes/mymachine.mydomain.com - SSL_connect returned=1 errno=0 state=error: certificate verify failed
ERROR: Failed to execute command on 10.210.1.112 return code 1

Stacktrace:

Generated at 2016-11-15 14:54:53 -0800
OpenSSL::SSL::SSLError: SSL Error connecting to https://chef1.mydomain.com/organizations/pop/nodes/mymachine.mydomain.com - SSL_connect returned=1 errno=0 state=error: certificate verify failed
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/lib/chef/http.rb:412:in `rescue in retrying_http_errors'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/lib/chef/http.rb:369:in `retrying_http_errors'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/lib/chef/http.rb:333:in `send_http_request'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/lib/chef/http.rb:148:in `request'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/lib/chef/http.rb:115:in `get'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/lib/chef/node.rb:581:in `load'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/lib/chef/node.rb:565:in `find_or_create'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/lib/chef/policy_builder/dynamic.rb:72:in `load_node'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/lib/chef/client.rb:467:in `load_node'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/lib/chef/client.rb:269:in `run'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/lib/chef/application.rb:276:in `run_with_graceful_exit_option'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/lib/chef/application.rb:252:in `block in run_chef_client'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/lib/chef/local_mode.rb:44:in `with_server_connectivity'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/lib/chef/application.rb:235:in `run_chef_client'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/lib/chef/application/client.rb:464:in `sleep_then_run_chef_client'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/lib/chef/application/client.rb:451:in `block in interval_run_chef_client'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/lib/chef/application/client.rb:450:in `loop'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/lib/chef/application/client.rb:450:in `interval_run_chef_client'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/lib/chef/application/client.rb:434:in `run_application'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/lib/chef/application.rb:59:in `run'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/bin/chef-client:26:in `<top (required)>'
C:/opscode/chef/bin/chef-client:65:in `load'
C:/opscode/chef/bin/chef-client:65:in `<main>'

>>>> Caused by OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=error: certificate verify failed
C:/opscode/chef/embedded/lib/ruby/2.3.0/net/http.rb:933:in `connect_nonblock'
C:/opscode/chef/embedded/lib/ruby/2.3.0/net/http.rb:933:in `connect'
C:/opscode/chef/embedded/lib/ruby/2.3.0/net/http.rb:863:in `do_start'
C:/opscode/chef/embedded/lib/ruby/2.3.0/net/http.rb:852:in `start'
C:/opscode/chef/embedded/lib/ruby/2.3.0/net/http.rb:1398:in `request'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/lib/chef/http/basic_client.rb:70:in `request'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/lib/chef/http.rb:340:in `block in send_http_request'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/lib/chef/http.rb:372:in `block in retrying_http_errors'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/lib/chef/http.rb:370:in `loop'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/lib/chef/http.rb:370:in `retrying_http_errors'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/lib/chef/http.rb:333:in `send_http_request'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/lib/chef/http.rb:148:in `request'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/lib/chef/http.rb:115:in `get'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/lib/chef/node.rb:581:in `load'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/lib/chef/node.rb:565:in `find_or_create'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/lib/chef/policy_builder/dynamic.rb:72:in `load_node'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/lib/chef/client.rb:467:in `load_node'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/lib/chef/client.rb:269:in `run'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/lib/chef/application.rb:276:in `run_with_graceful_exit_option'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/lib/chef/application.rb:252:in `block in run_chef_client'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/lib/chef/local_mode.rb:44:in `with_server_connectivity'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/lib/chef/application.rb:235:in `run_chef_client'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/lib/chef/application/client.rb:464:in `sleep_then_run_chef_client'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/lib/chef/application/client.rb:451:in `block in interval_run_chef_client'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/lib/chef/application/client.rb:450:in `loop'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/lib/chef/application/client.rb:450:in `interval_run_chef_client'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/lib/chef/application/client.rb:434:in `run_application'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/lib/chef/application.rb:59:in `run'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.16.42-universal-mingw32/bin/chef-client:26:in `<top (required)>'
C:/opscode/chef/bin/chef-client:65:in `load'
C:/opscode/chef/bin/chef-client:65:in `<main>'

This sounds very much like this issue: https://github.com/chef/knife-windows/issues/404. You might either try applying the suggested “hack” or downgrading your dk to determine if that fixes the issue.

1 Like

Confirmed - I added the workaround to windows_bootstrap_context.rb as suggested, and I’m now able to successfully bootstrap nodes without error. The trusted_certs folder is getting created as well.

1 Like

The fix has just been released in knife-windows 1.7.1!

1 Like