Upgrade Chef from 11.6 to 12.3


#1

When I upgrade Chef (on the clients) from 11.6 to 12.3 the connection to
server (Chef 11.6) broken.

root@test:~# chef-client

Starting Chef Client, version 12.3.0

Creating a new client identity for test.domain.com using the validator key.

[2015-05-07T16:46:17-04:00] ERROR: SSL Validation failure connecting to
host: xxxx.domain.com - SSL_connect returned=1 errno=0 state=SSLv3 read
server certificate B: certificate verify failed

================================================================================

Chef encountered an error attempting to create the client " test.domain.com
"

================================================================================

[2015-05-07T16:46:17-04:00] FATAL: Stacktrace dumped to
/var/chef/cache/chef-stacktrace.out

Chef Client failed. 0 resources updated in 1.306760691 seconds

[2015-05-07T16:46:17-04:00] ERROR: SSL_connect returned=1 errno=0
state=SSLv3 read server certificate B: certificate verify failed

[2015-05-07T16:46:17-04:00] FATAL: Chef::Exceptions::ChildConvergeError:
Chef run process exited unsuccessfully (exit code 1)

This is what I do to fix it:

  1.   delete the node / client from Chef WEB Interface
    
  2.   add “ssl_verify_mode :verify_peer” into client.rb (client side)
    
  3.   delete client.pm (client side)
    
  4.   mkdir /etc/chef/trusted_certs/
    
  5.   root@Cefh-server:/var/opt/chef-server/nginx/ca# scp server.crt
    

root@CLIENT:/etc/chef/trusted_certs/
<root@172.16.8.40:/etc/chef/trusted_certs/>

  1.   knife bootstrap CLIENT  --sudo -x toor
    
  2.   To check the SSL configuration : knife ssl check -c
    

/etc/chef/client.rb

Any other workaround?

Mohammad


#2

On Monday, May 11, 2015 at 10:36 AM, Mohammad Fattahian wrote:

When I upgrade Chef (on the clients) from 11.6 to 12.3 the connection to server (Chef 11.6) broken.

root@test:~# chef-client
Starting Chef Client, version 12.3.0
Creating a new client identity for test.domain.com (http://test.domain.com) using the validator key.
[2015-05-07T16:46:17-04:00] ERROR: SSL Validation failure connecting to host: xxxx.domain.com (http://xxxx.domain.com) - SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed

================================================================================
Chef encountered an error attempting to create the client " test.domain.com (http://test.domain.com) "

[2015-05-07T16:46:17-04:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
Chef Client failed. 0 resources updated in 1.306760691 seconds
[2015-05-07T16:46:17-04:00] ERROR: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed
[2015-05-07T16:46:17-04:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

This isn’t a bug. In the past, chef-client did not verify the SSL certificate of the server, which leaves you vulnerable to a MITM attack. If you are sure you don’t need this protection (I would argue that defense-in-depth means you should not turn this off, even if you’re only connecting on a private network), then you can set the old default with ssl_verify_mode :verify_none

This is what I do to fix it:

  1. delete the node / client from Chef WEB Interface

This isn’t necessary, the client.pem is used for application-level authentication, which is not relevant to the transport layer encryption.

  1. add “ssl_verify_mode :verify_peer” into client.rb (client side)

This isn’t necessary, ssl_verify_mode :verify_peer is the default in Chef Client 12+

  1. delete client.pm (http://client.pm) (client side)

Again, this isn’t required because the client.pem is only relevant to application layer authentication.

  1. mkdir /etc/chef/trusted_certs/
  2. root@Cefh-server:/var/opt/chef-server/nginx/ca# scp server.crt root@CLIENT:/etc/chef/trusted_certs/

These two steps are all you need to do.

  1. knife bootstrap CLIENT --sudo -x toor

You only need this step because you’ve deleted the client/node on the server, if you skip steps 1 and 3, then you don’t need this. FWIW, if you have downloaded your Chef Server’s cert to your workstation and you’re running knife 12+ on your workstation, it will copy your server’s cert to the bootstrap node as part of the bootstrap process.

  1. To check the SSL configuration : knife ssl check -c /etc/chef/client.rb

Any other workaround?

You can make knife download the certificates from the server with knife ssl fetch. HOWEVER, this is only as good as clicking the “trust this certificate for this server” from the security warning pop-up on a browser. If you’re already MITM’d then you’re just trusting the MITM’s certificate. If you use this command then you should take a SHA-256 of the cert on the server and compare to what you downloaded.

Mohammad

I’ll look into having the error messages on the client provide some of this information so it’s easier to resolve when this comes up.

HTH,


Daniel DeLeo


#3

It helps,

Thanks for reply.

Mohammad

-----Original Message-----
From: Daniel DeLeo [mailto:ddeleo@kallistec.com] On Behalf Of Daniel DeLeo
Sent: Monday, May 11, 2015 1:53 PM
To: chef@lists.opscode.com
Subject: [chef] Re: Upgrade Chef from 11.6 to 12.3

On Monday, May 11, 2015 at 10:36 AM, Mohammad Fattahian wrote:

When I upgrade Chef (on the clients) from 11.6 to 12.3 the connection to
server (Chef 11.6) broken.

root@test:~# chef-client
Starting Chef Client, version 12.3.0
Creating a new client identity for test.domain.com
(http://test.domain.com) using the validator key.
[2015-05-07T16:46:17-04:00] ERROR: SSL Validation failure connecting
to host: xxxx.domain.com (http://xxxx.domain.com) - SSL_connect
returned=1 errno=0 state=SSLv3 read server certificate B: certificate
verify failed

======================================================================
========== Chef encountered an error attempting to create the client "
test.domain.com (http://test.domain.com) "

==========

[2015-05-07T16:46:17-04:00] FATAL: Stacktrace dumped to
/var/chef/cache/chef-stacktrace.out
Chef Client failed. 0 resources updated in 1.306760691 seconds
[2015-05-07T16:46:17-04:00] ERROR: SSL_connect returned=1 errno=0
state=SSLv3 read server certificate B: certificate verify failed
[2015-05-07T16:46:17-04:00] FATAL:
Chef::Exceptions::ChildConvergeError: Chef run process exited
unsuccessfully (exit code 1)

This isn’t a bug. In the past, chef-client did not verify the SSL
certificate of the server, which leaves you vulnerable to a MITM attack. If
you are sure you don’t need this protection (I would argue that
defense-in-depth means you should not turn this off, even if you’re only
connecting on a private network), then you can set the old default with
ssl_verify_mode :verify_none

This is what I do to fix it:

  1. delete the node / client from Chef WEB Interface

This isn’t necessary, the client.pem is used for application-level
authentication, which is not relevant to the transport layer encryption.

  1. add “ssl_verify_mode :verify_peer” into client.rb (client side)

This isn’t necessary, ssl_verify_mode :verify_peer is the default in Chef
Client 12+

  1. delete client.pm (http://client.pm) (client side)

Again, this isn’t required because the client.pem is only relevant to
application layer authentication.

  1. mkdir /etc/chef/trusted_certs/
  2. root@Cefh-server:/var/opt/chef-server/nginx/ca# scp server.crt
    root@CLIENT:/etc/chef/trusted_certs/

These two steps are all you need to do.

  1. knife bootstrap CLIENT --sudo -x toor

You only need this step because you’ve deleted the client/node on the
server, if you skip steps 1 and 3, then you don’t need this. FWIW, if you
have downloaded your Chef Server’s cert to your workstation and you’re
running knife 12+ on your workstation, it will copy your server’s cert to
the bootstrap node as part of the bootstrap process.

  1. To check the SSL configuration : knife ssl check -c
    /etc/chef/client.rb

Any other workaround?

You can make knife download the certificates from the server with knife ssl fetch. HOWEVER, this is only as good as clicking the “trust this
certificate for this server” from the security warning pop-up on a browser.
If you’re already MITM’d then you’re just trusting the MITM’s certificate.
If you use this command then you should take a SHA-256 of the cert on the
server and compare to what you downloaded.

Mohammad

I’ll look into having the error messages on the client provide some of this
information so it’s easier to resolve when this comes up.

HTH,


Daniel DeLeo


#4

If I’m not wrong, all you need to do after you upgrade your node is to run
this command:

knife ssl fetch -c /etc/chef/client.rb

This will import the chef server certificate into your /etc/chef/ dir and
add that to your client.rb file

Sebastian Garcia

System Network Engineer

Smart Trade Technologies420 Lexington Ave. Ste. 2800-02
New York, NY 10170
212-867-4567 x90
http://www.smart-trade.net

On Tue, May 12, 2015 at 9:36 AM, Mohammad Fattahian <
mfattahian@masterfile.com> wrote:

It helps,

Thanks for reply.

Mohammad

-----Original Message-----
From: Daniel DeLeo [mailto:ddeleo@kallistec.com] On Behalf Of Daniel DeLeo
Sent: Monday, May 11, 2015 1:53 PM
To: chef@lists.opscode.com
Subject: [chef] Re: Upgrade Chef from 11.6 to 12.3

On Monday, May 11, 2015 at 10:36 AM, Mohammad Fattahian wrote:

When I upgrade Chef (on the clients) from 11.6 to 12.3 the connection to
server (Chef 11.6) broken.

root@test:~# chef-client
Starting Chef Client, version 12.3.0
Creating a new client identity for test.domain.com
(http://test.domain.com) using the validator key.
[2015-05-07T16:46:17-04:00] ERROR: SSL Validation failure connecting
to host: xxxx.domain.com (http://xxxx.domain.com) - SSL_connect
returned=1 errno=0 state=SSLv3 read server certificate B: certificate
verify failed

======================================================================
========== Chef encountered an error attempting to create the client "
test.domain.com (http://test.domain.com) "

==========

[2015-05-07T16:46:17-04:00] FATAL: Stacktrace dumped to
/var/chef/cache/chef-stacktrace.out
Chef Client failed. 0 resources updated in 1.306760691 seconds
[2015-05-07T16:46:17-04:00] ERROR: SSL_connect returned=1 errno=0
state=SSLv3 read server certificate B: certificate verify failed
[2015-05-07T16:46:17-04:00] FATAL:
Chef::Exceptions::ChildConvergeError: Chef run process exited
unsuccessfully (exit code 1)

This isn’t a bug. In the past, chef-client did not verify the SSL
certificate of the server, which leaves you vulnerable to a MITM attack. If
you are sure you don’t need this protection (I would argue that
defense-in-depth means you should not turn this off, even if you’re only
connecting on a private network), then you can set the old default with
ssl_verify_mode :verify_none

This is what I do to fix it:

  1. delete the node / client from Chef WEB Interface

This isn’t necessary, the client.pem is used for application-level
authentication, which is not relevant to the transport layer encryption.

  1. add “ssl_verify_mode :verify_peer” into client.rb (client side)

This isn’t necessary, ssl_verify_mode :verify_peer is the default in Chef
Client 12+

  1. delete client.pm (http://client.pm) (client side)

Again, this isn’t required because the client.pem is only relevant to
application layer authentication.

  1. mkdir /etc/chef/trusted_certs/
  2. root@Cefh-server:/var/opt/chef-server/nginx/ca# scp server.crt
    root@CLIENT:/etc/chef/trusted_certs/

These two steps are all you need to do.

  1. knife bootstrap CLIENT --sudo -x toor

You only need this step because you’ve deleted the client/node on the
server, if you skip steps 1 and 3, then you don’t need this. FWIW, if you
have downloaded your Chef Server’s cert to your workstation and you’re
running knife 12+ on your workstation, it will copy your server’s cert to
the bootstrap node as part of the bootstrap process.

  1. To check the SSL configuration : knife ssl check -c
    /etc/chef/client.rb

Any other workaround?

You can make knife download the certificates from the server with knife ssl fetch. HOWEVER, this is only as good as clicking the “trust this
certificate for this server” from the security warning pop-up on a browser.
If you’re already MITM’d then you’re just trusting the MITM’s certificate.
If you use this command then you should take a SHA-256 of the cert on the
server and compare to what you downloaded.

Mohammad

I’ll look into having the error messages on the client provide some of this
information so it’s easier to resolve when this comes up.

HTH,


Daniel DeLeo