Unable to use SSL cert from in-house Chef Server w/ knife

Ohai Chefs!

We have a Chef 12 server set up with a self-signed cert. I’m trying to get knife to communicate using peer verification with the server.

First, I run the check:
$ knife ssl check https://chef.owicloud.org/organizations/cida
Connecting to host chef.owicloud.org:443
ERROR: The SSL certificate of chef.owicloud.org could not be verified
Certificate issuer data: /C=us/L=Middleton, WI/O=OWI USGS/CN=OWI USGS VPN CA/emailAddress=isuftin@usgs.gov

Configuration Info:

OpenSSL Configuration:

  • Version: OpenSSL 1.0.1k 8 Jan 2015
  • Certificate file: /opt/chefdk/embedded/ssl/cert.pem
  • Certificate directory: /opt/chefdk/embedded/ssl/certs
    Chef SSL Configuration:
  • ssl_ca_path: nil
  • ssl_ca_file: nil
  • trusted_certs_dir: “/Users/isuftin/.chef/trusted_certs”

TO FIX THIS ERROR:

[ … the usual text we know and love …]

Ok, so obviously that doesn’t work. I then try to run a knife ssl fetch:
$ knife ssl fetch https://chef.owicloud.org/
WARNING: Certificates from chef.owicloud.org will be fetched and placed in your trusted_cert
directory (/Users/isuftin/.chef/trusted_certs).

Knife has no means to verify these are the correct certificates. You should
verify the authenticity of these certificates after downloading.

ERROR: knife encountered an unexpected error
This may be a bug in the ‘ssl fetch’ knife command or plugin
Please collect the output of this command with the -VV option before filing a bug report.
Exception: NoMethodError: undefined method `[]’ for nil:NilClass

I’ve always had this issue so I’ve never used knife ssl fetch to grab the SSL cert. So I scope the cert into /Users/isuftin/.chef/trusted_certs and run a hash check on them on the server and local:

Server:
$ sudo sha256sum /var/opt/opscode/nginx/ca/chef2a.crt
7a876dad9a3f6e59e169d5cb25d2ad64bd362515bbc7f9af2baec5936505ca09 /var/opt/opscode/nginx/ca/chef2a.crt

Local (mac os):
$ shasum -a256 /Users/isuftin/.chef/trusted_certs/chef2a.crt
7a876dad9a3f6e59e169d5cb25d2ad64bd362515bbc7f9af2baec5936505ca09 /Users/isuftin/.chef/trusted_certs/chef2a.crt

Initial thoughts of where I should look?

__________________________ (╯°□°)╯︵ ┻━┻
Ivan Suftin - Applications Developer - isuftin@usgs.gov
Office: (608) 821-3825 - Cell : (608) 345-8963
Center for Integrated Data Analytics - http://cida.usgs.gov/
United States Geological Survey
8505 Research Way, Middleton, WI 53562

On Monday, February 16, 2015 at 10:14 AM, Ivan Suftin wrote:

Ohai Chefs!

We have a Chef 12 server set up with a self-signed cert. I’m trying to get knife to communicate using peer verification with the server.

First, I run the check:

$ knife ssl check https://chef.owicloud.org/organizations/cida
Connecting to host chef.owicloud.org (http://chef.owicloud.org):443
ERROR: The SSL certificate of chef.owicloud.org (http://chef.owicloud.org) could not be verified
Certificate issuer data: /C=us/L=Middleton, WI/O=OWI USGS/CN=OWI USGS VPN CA/emailAddress=isuftin@usgs.gov (mailto:CA/emailAddress=isuftin@usgs.gov)

Configuration Info:

OpenSSL Configuration:

  • Version: OpenSSL 1.0.1k 8 Jan 2015
  • Certificate file: /opt/chefdk/embedded/ssl/cert.pem
  • Certificate directory: /opt/chefdk/embedded/ssl/certs
    Chef SSL Configuration:
  • ssl_ca_path: nil
  • ssl_ca_file: nil
  • trusted_certs_dir: "/Users/isuftin/.chef/trusted_certs"

TO FIX THIS ERROR:

[ … the usual text we know and love …]
Did you get a message about "There are invalid certificates in your trusted_certs_dir.” from knife ssl check? (Your certs would have to exist in your trusted_certs_dir before you run that command for knife ssl check to detect this issue). There are certain properties on certificates, which if they are not set, will make OpenSSL refuse to verify the certificate against itself. In the past we’ve seen this when users follow an outdated blog post for setting "Subject Alternative Names,” but you could have stumbled upon this issue as well.

Ok, so obviously that doesn’t work. I then try to run a knife ssl fetch:

$ knife ssl fetch https://chef.owicloud.org/
WARNING: Certificates from chef.owicloud.org (http://chef.owicloud.org) will be fetched and placed in your trusted_cert
directory (/Users/isuftin/.chef/trusted_certs).

Knife has no means to verify these are the correct certificates. You should
verify the authenticity of these certificates after downloading.

ERROR: knife encountered an unexpected error
This may be a bug in the 'ssl fetch' knife command or plugin
Please collect the output of this command with the -VV option before filing a bug report.
Exception: NoMethodError: undefined method `' for nil:NilClass

I’ve always had this issue so I’ve never used knife ssl fetch to grab the SSL cert. So I scope the cert into /Users/isuftin/.chef/trusted_certs and run a hash check on them on the server and local:

This looks like an actual bug with knife ssl fetch. If you run knife ssl fetch URL -VV, it will show the backtrace. You should then copy all of that into a bug report at Issues · chef/chef · GitHub so we can fix the command.

Server:

$ sudo sha256sum /var/opt/opscode/nginx/ca/chef2a.crt
7a876dad9a3f6e59e169d5cb25d2ad64bd362515bbc7f9af2baec5936505ca09 /var/opt/opscode/nginx/ca/chef2a.crt

Local (mac os):

$ shasum -a256 /Users/isuftin/.chef/trusted_certs/chef2a.crt
7a876dad9a3f6e59e169d5cb25d2ad64bd362515bbc7f9af2baec5936505ca09 /Users/isuftin/.chef/trusted_certs/chef2a.crt

Initial thoughts of where I should look?

__________________________ (╯°□°)╯︵ ┻━┻
Ivan Suftin - Applications Developer - isuftin@usgs.gov (mailto:isuftin@usgs.gov)
Office: (608) 821-3825 - Cell : (608) 345-8963
Center for Integrated Data Analytics - http://cida.usgs.gov/
United States Geological Survey
8505 Research Way, Middleton, WI 53562

--
Daniel DeLeo

Hi Daniel,

I did not see any messages regarding "There are invalid certificates in your trusted_certs_dir”. Do you have more information regarding the "Subject Alternative Names” issue? I could get more information for you if needed in order to better analyze the issue.

In regards to the knife issue, I’ve opened this ticket: knife ssl fetch needs better cert validation logic and should not throw NoMethodError on nil · Issue #2919 · chef/chef · GitHub https://github.com/chef/chef/issues/2919

__________________________ (╯°□°)╯︵ ┻━┻
Ivan Suftin - Applications Developer - isuftin@usgs.gov
Office: (608) 821-3825 - Cell : (608) 345-8963
Center for Integrated Data Analytics - http://cida.usgs.gov/
United States Geological Survey
8505 Research Way, Middleton, WI 53562

On Feb 16, 2015, at 2:16 PM, Daniel DeLeo dan@kallistec.com wrote:

On Monday, February 16, 2015 at 10:14 AM, Ivan Suftin wrote:

Ohai Chefs!

We have a Chef 12 server set up with a self-signed cert. I’m trying to get knife to communicate using peer verification with the server.

First, I run the check:

$ knife ssl check https://chef.owicloud.org/organizations/cida
Connecting to host chef.owicloud.org (http://chef.owicloud.org):443
ERROR: The SSL certificate of chef.owicloud.org (http://chef.owicloud.org) could not be verified
Certificate issuer data: /C=us/L=Middleton, WI/O=OWI USGS/CN=OWI USGS VPN CA/emailAddress=isuftin@usgs.gov (mailto:CA/emailAddress=isuftin@usgs.gov)

Configuration Info:

OpenSSL Configuration:

  • Version: OpenSSL 1.0.1k 8 Jan 2015
  • Certificate file: /opt/chefdk/embedded/ssl/cert.pem
  • Certificate directory: /opt/chefdk/embedded/ssl/certs
    Chef SSL Configuration:
  • ssl_ca_path: nil
  • ssl_ca_file: nil
  • trusted_certs_dir: "/Users/isuftin/.chef/trusted_certs"

TO FIX THIS ERROR:

[ … the usual text we know and love …]
Did you get a message about "There are invalid certificates in your trusted_certs_dir.” from knife ssl check? (Your certs would have to exist in your trusted_certs_dir before you run that command for knife ssl check to detect this issue). There are certain properties on certificates, which if they are not set, will make OpenSSL refuse to verify the certificate against itself. In the past we’ve seen this when users follow an outdated blog post for setting "Subject Alternative Names,” but you could have stumbled upon this issue as well.

Ok, so obviously that doesn’t work. I then try to run a knife ssl fetch:

$ knife ssl fetch https://chef.owicloud.org/
WARNING: Certificates from chef.owicloud.org (http://chef.owicloud.org) will be fetched and placed in your trusted_cert
directory (/Users/isuftin/.chef/trusted_certs).

Knife has no means to verify these are the correct certificates. You should
verify the authenticity of these certificates after downloading.

ERROR: knife encountered an unexpected error
This may be a bug in the 'ssl fetch' knife command or plugin
Please collect the output of this command with the -VV option before filing a bug report.
Exception: NoMethodError: undefined method `' for nil:NilClass

I’ve always had this issue so I’ve never used knife ssl fetch to grab the SSL cert. So I scope the cert into /Users/isuftin/.chef/trusted_certs and run a hash check on them on the server and local:

This looks like an actual bug with knife ssl fetch. If you run knife ssl fetch URL -VV, it will show the backtrace. You should then copy all of that into a bug report at Issues · chef/chef · GitHub so we can fix the command.

Server:

$ sudo sha256sum /var/opt/opscode/nginx/ca/chef2a.crt
7a876dad9a3f6e59e169d5cb25d2ad64bd362515bbc7f9af2baec5936505ca09 /var/opt/opscode/nginx/ca/chef2a.crt

Local (mac os):

$ shasum -a256 /Users/isuftin/.chef/trusted_certs/chef2a.crt
7a876dad9a3f6e59e169d5cb25d2ad64bd362515bbc7f9af2baec5936505ca09 /Users/isuftin/.chef/trusted_certs/chef2a.crt

Initial thoughts of where I should look?

__________________________ (╯°□°)╯︵ ┻━┻
Ivan Suftin - Applications Developer - isuftin@usgs.gov (mailto:isuftin@usgs.gov)
Office: (608) 821-3825 - Cell : (608) 345-8963
Center for Integrated Data Analytics - http://cida.usgs.gov/
United States Geological Survey
8505 Research Way, Middleton, WI 53562

--
Daniel DeLeo

I'll try to resume what I know with SSL validation.

I had issues with self-signed certificates for the chef-server and our
proxy certificate wich do ssl interception.

What solved nearly all for me is to add the certificates in the
cacert.pem file.

For the self signed certificate of the chef server:

openssl s_client -showcerts -connect <YOUR_CHEF_SERVER>:443 </dev/null
2>/dev/null|openssl x509 -outform PEM >>
/opt/chef/embedded/ssl/certs/cacert.pem

For my proxy I had to add our internal root Certificate authority cert
(took from a browser and exporting certificate to disk)

for some other tools I had to do export SSL_CERT_FILE=/opt/chef/embedded/ssl/certs/cacert.pem to enforce
openssl lib to use my updated certificate list.

After that, one cookbook to distribute it and update chef conf after the
file is in place (for nodes).

Hope it may help.

Le 2015-02-16 23:25, Ivan Suftin a écrit :

Hi Daniel,

I did not see any messages regarding "There are invalid certificates in your trusted_certs_dir". Do you have more information regarding the "Subject Alternative Names" issue? I could get more information for you if needed in order to better analyze the issue.

In regards to the knife issue, I've opened this ticket: knife ssl fetch needs better cert validation logic and should not throw NoMethodError on nil · Issue #2919 · chef/chef · GitHub [1]

__________________________ (╯°□°)╯︵ ┻━┻
Ivan Suftin - Applications Developer - isuftin@usgs.gov
Office: (608) 821-3825 - Cell : (608) 345-8963
Center for Integrated Data Analytics - http://cida.usgs.gov/ [2]
United States Geological Survey
8505 Research Way, Middleton, WI 53562

On Feb 16, 2015, at 2:16 PM, Daniel DeLeo dan@kallistec.com wrote:

On Monday, February 16, 2015 at 10:14 AM, Ivan Suftin wrote:
Ohai Chefs!

We have a Chef 12 server set up with a self-signed cert. I'm trying to get knife to communicate using peer verification with the server.

First, I run the check:
$ knife ssl check https://chef.owicloud.org/organizations/cida [3]
Connecting to host chef.owicloud.org [4] (http://chef.owicloud.org [4]):443
ERROR: The SSL certificate of chef.owicloud.org [4] (http://chef.owicloud.org [4]) could not be verified
Certificate issuer data: /C=us/L=Middleton, WI/O=OWI USGS/CN=OWI USGS VPN CA/emailAddress=isuftin@usgs.gov (mailto:CA/emailAddress=isuftin@usgs.gov)

Configuration Info:

OpenSSL Configuration:

  • Version: OpenSSL 1.0.1k 8 Jan 2015
  • Certificate file: /opt/chefdk/embedded/ssl/cert.pem
  • Certificate directory: /opt/chefdk/embedded/ssl/certs
    Chef SSL Configuration:
  • ssl_ca_path: nil
  • ssl_ca_file: nil
  • trusted_certs_dir: "/Users/isuftin/.chef/trusted_certs"

TO FIX THIS ERROR:

[ ... the usual text we know and love ...]
Did you get a message about "There are invalid certificates in your
trusted_certs_dir." from knife ssl check? (Your certs would have to
exist in your trusted_certs_dir before you run that command for knife ssl check to detect this issue). There are certain properties on
certificates, which if they are not set, will make OpenSSL refuse to
verify the certificate against itself. In the past we've seen this when
users follow an outdated blog post for setting "Subject Alternative
Names," but you could have stumbled upon this issue as well.

Ok, so obviously that doesn't work. I then try to run a knife ssl fetch:

$ knife ssl fetch https://chef.owicloud.org/ [5]
WARNING: Certificates from chef.owicloud.org [4] (http://chef.owicloud.org [4]) will be fetched and placed in your trusted_cert
directory (/Users/isuftin/.chef/trusted_certs).

Knife has no means to verify these are the correct certificates. You should
verify the authenticity of these certificates after downloading.

ERROR: knife encountered an unexpected error
This may be a bug in the 'ssl fetch' knife command or plugin
Please collect the output of this command with the -VV option before filing a bug report.
Exception: NoMethodError: undefined method `' for nil:NilClass

I've always had this issue so I've never used knife ssl fetch to grab the SSL cert. So I scope the cert into /Users/isuftin/.chef/trusted_certs and run a hash check on them on the server and local:

This looks like an actual bug with knife ssl fetch. If you run knife ssl fetch URL -VV, it will show the backtrace. You should then copy all
of that into a bug report at Issues · chef/chef · GitHub [6] so
we can fix the command.

Server:

$ sudo sha256sum /var/opt/opscode/nginx/ca/chef2a.crt
7a876dad9a3f6e59e169d5cb25d2ad64bd362515bbc7f9af2baec5936505ca09 /var/opt/opscode/nginx/ca/chef2a.crt

Local (mac os):

$ shasum -a256 /Users/isuftin/.chef/trusted_certs/chef2a.crt
7a876dad9a3f6e59e169d5cb25d2ad64bd362515bbc7f9af2baec5936505ca09 /Users/isuftin/.chef/trusted_certs/chef2a.crt

Initial thoughts of where I should look?

__________________________ (╯°□°)╯︵ ┻━┻
Ivan Suftin - Applications Developer - isuftin@usgs.gov (mailto:isuftin@usgs.gov)
Office: (608) 821-3825 - Cell : (608) 345-8963
Center for Integrated Data Analytics - http://cida.usgs.gov/ [2]
United States Geological Survey
8505 Research Way, Middleton, WI 53562

--
Daniel DeLeo

Links:

[1] knife ssl fetch needs better cert validation logic and should not throw NoMethodError on nil · Issue #2919 · chef/chef · GitHub
[2] http://cida.usgs.gov/
[3] https://chef.owicloud.org/organizations/cida
[4] http://chef.owicloud.org
[5] https://chef.owicloud.org/
[6] Issues · chef/chef · GitHub

So this actually turned out to be an issue with our VPC perimeter firewall we use at AWS. When VPN’d into the VPC, past the firewall and using the direct IP of the machine, the knife command acts as expected.

We will be switching to a dedicated VPN soon to connect our datacenter into the VPC in a bidirectional way. At that point this will not be an issue, so for now I will continue turning off peer verification.

Thank you for your help!
__________________________ (╯°□°)╯︵ ┻━┻
Ivan Suftin - Applications Developer - isuftin@usgs.gov
Office: (608) 821-3825 - Cell : (608) 345-8963
Center for Integrated Data Analytics - http://cida.usgs.gov/
United States Geological Survey
8505 Research Way, Middleton, WI 53562

On Feb 17, 2015, at 3:06 AM, Tensibai tensibai@iabis.net wrote:

I'll try to resume what I know with SSL validation.

I had issues with self-signed certificates for the chef-server and our proxy certificate wich do ssl interception.

What solved nearly all for me is to add the certificates in the cacert.pem file.

For the self signed certificate of the chef server:

openssl s_client -showcerts -connect <YOUR_CHEF_SERVER>:443 </dev/null 2>/dev/null|openssl x509 -outform PEM >> /opt/chef/embedded/ssl/certs/cacert.pem

For my proxy I had to add our internal root Certificate authority cert (took from a browser and exporting certificate to disk)

for some other tools I had to do export SSL_CERT_FILE=/opt/chef/embedded/ssl/certs/cacert.pem to enforce openssl lib to use my updated certificate list.

After that, one cookbook to distribute it and update chef conf after the file is in place (for nodes).

Hope it may help.

Le 2015-02-16 23:25, Ivan Suftin a écrit :

Hi Daniel,

I did not see any messages regarding "There are invalid certificates in your trusted_certs_dir". Do you have more information regarding the "Subject Alternative Names" issue? I could get more information for you if needed in order to better analyze the issue.

In regards to the knife issue, I've opened this ticket: knife ssl fetch needs better cert validation logic and should not throw NoMethodError on nil · Issue #2919 · chef/chef · GitHub https://github.com/chef/chef/issues/2919

__________________________ (╯°□°)╯︵ ┻━┻
Ivan Suftin - Applications Developer - isuftin@usgs.gov mailto:isuftin@usgs.gov
Office: (608) 821-3825 - Cell : (608) 345-8963
Center for Integrated Data Analytics - http://cida.usgs.gov/ http://cida.usgs.gov/
United States Geological Survey
8505 Research Way, Middleton, WI 53562

On Feb 16, 2015, at 2:16 PM, Daniel DeLeo <dan@kallistec.com mailto:dan@kallistec.com> wrote:

On Monday, February 16, 2015 at 10:14 AM, Ivan Suftin wrote:

Ohai Chefs!

We have a Chef 12 server set up with a self-signed cert. I'm trying to get knife to communicate using peer verification with the server.

First, I run the check:

$ knife ssl check https://chef.owicloud.org/organizations/cida https://chef.owicloud.org/organizations/cida
Connecting to host chef.owicloud.org http://chef.owicloud.org/ (http://chef.owicloud.org http://chef.owicloud.org/):443
ERROR: The SSL certificate of chef.owicloud.org http://chef.owicloud.org/ (http://chef.owicloud.org http://chef.owicloud.org/) could not be verified
Certificate issuer data: /C=us/L=Middleton, WI/O=OWI USGS/CN=OWI USGS VPN CA/emailAddress=isuftin@usgs.gov mailto:CA/emailAddress=isuftin@usgs.gov (mailto:CA/emailAddress=isuftin@usgs.gov mailto:CA/emailAddress=isuftin@usgs.gov)

Configuration Info:

OpenSSL Configuration:

  • Version: OpenSSL 1.0.1k 8 Jan 2015
  • Certificate file: /opt/chefdk/embedded/ssl/cert.pem
  • Certificate directory: /opt/chefdk/embedded/ssl/certs
    Chef SSL Configuration:
  • ssl_ca_path: nil
  • ssl_ca_file: nil
  • trusted_certs_dir: "/Users/isuftin/.chef/trusted_certs"

TO FIX THIS ERROR:

[ ... the usual text we know and love ...]
Did you get a message about "There are invalid certificates in your trusted_certs_dir." from knife ssl check? (Your certs would have to exist in your trusted_certs_dir before you run that command for knife ssl check to detect this issue). There are certain properties on certificates, which if they are not set, will make OpenSSL refuse to verify the certificate against itself. In the past we've seen this when users follow an outdated blog post for setting "Subject Alternative Names," but you could have stumbled upon this issue as well.

Ok, so obviously that doesn't work. I then try to run a knife ssl fetch:

$ knife ssl fetch https://chef.owicloud.org/ https://chef.owicloud.org/
WARNING: Certificates from chef.owicloud.org http://chef.owicloud.org/ (http://chef.owicloud.org http://chef.owicloud.org/) will be fetched and placed in your trusted_cert
directory (/Users/isuftin/.chef/trusted_certs).

Knife has no means to verify these are the correct certificates. You should
verify the authenticity of these certificates after downloading.

ERROR: knife encountered an unexpected error
This may be a bug in the 'ssl fetch' knife command or plugin
Please collect the output of this command with the -VV option before filing a bug report.
Exception: NoMethodError: undefined method `' for nil:NilClass

I've always had this issue so I've never used knife ssl fetch to grab the SSL cert. So I scope the cert into /Users/isuftin/.chef/trusted_certs and run a hash check on them on the server and local:

This looks like an actual bug with knife ssl fetch. If you run knife ssl fetch URL -VV, it will show the backtrace. You should then copy all of that into a bug report at Issues · chef/chef · GitHub https://github.com/chef/chef/issues so we can fix the command.

Server:

$ sudo sha256sum /var/opt/opscode/nginx/ca/chef2a.crt
7a876dad9a3f6e59e169d5cb25d2ad64bd362515bbc7f9af2baec5936505ca09 /var/opt/opscode/nginx/ca/chef2a.crt

Local (mac os):

$ shasum -a256 /Users/isuftin/.chef/trusted_certs/chef2a.crt
7a876dad9a3f6e59e169d5cb25d2ad64bd362515bbc7f9af2baec5936505ca09 /Users/isuftin/.chef/trusted_certs/chef2a.crt

Initial thoughts of where I should look?

__________________________ (╯°□°)╯︵ ┻━┻
Ivan Suftin - Applications Developer - isuftin@usgs.gov mailto:isuftin@usgs.gov (mailto:isuftin@usgs.gov mailto:isuftin@usgs.gov)
Office: (608) 821-3825 - Cell : (608) 345-8963
Center for Integrated Data Analytics - http://cida.usgs.gov/ http://cida.usgs.gov/
United States Geological Survey
8505 Research Way, Middleton, WI 53562

--
Daniel DeLeo