Hi Everyone,
I am trying to setup a chef environment with three nodes. Setup the chef server and workstation fine but getting issues while trying to bootstrap a node/chef-client from the workstation.
Details below:
[root@iyyrpbxxxxx chef-repo]# knife ssh ‘name:*’ --attribute ipaddress 'sudo knife ssl fetch -c /etc/chef/client.rb’
FATAL: 1 node found, but does not have the required attribute to establish the connection. Try setting another attribute to open the connection using --attribute.
[root@iyyrpbxxxxx chef-repo]# knife node show node1
Node Name: node1
Environment: _default
FQDN:
IP:
Run List:
Roles:
Recipes:
Platform:
Tags:
------------- on node / chef client ------------
[root@bcefgh090 ~]# hostname --fqdn
bcefgh090.xxxx.com
[root@bcefgh090 ~]# hostname
bcefgh090.xxxx.com
[root@bcefgh090 ~]# hostname --ip
7.105.40.15
I am able to see the ip and fqdn when logged in the chef node/client but I understand those are automatically retrieved by OHAI component and set in Chef config in the server. Also these get set only after a successful run of chef-client run in the node, but in this case, chef-client fails on the node with SSL Validation Error:
[root@bcdefg150 ~]# chef-client
Starting Chef Client, version 12.9.38
Creating a new client identity for node1 using the validator key.
[2016-05-10T12:53:41+05:30] ERROR: SSL Validation failure connecting to host: bcmtpc065.xxxx.com - SSL_connect returned=1 errno=0 state=SSLv2/v3 read server hello A: sslv3 alert handshake failure
================================================================================
Chef encountered an error attempting to create the client “node1”
Platform:
x86_64-linux
[2016-05-10T12:54:07+05:30] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
Stacktrace:
[root@bcdefg150 ~]# cat /var/chef/cache/chef-stacktrace.out
Generated at 2016-05-10 12:54:07 +0530
OpenSSL::SSL::SSLError: SSL Error connecting to https://bcmtpc065.xxxx.com/organizations/xxxx/clients - SSL_connect returned=1 errno=0 state=SSLv2/v3 read server hello A: sslv3 alert handshake failure
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.9.38/lib/chef/http.rb:376:in rescue in retrying_http_errors' /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.9.38/lib/chef/http.rb:333:inretrying_http_errors’
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.9.38/lib/chef/http.rb:299:in send_http_request' /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.9.38/lib/chef/http.rb:144:inrequest’
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.9.38/lib/chef/http.rb:127:in post' /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.9.38/lib/chef/api_client/registration.rb:96:increate’
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.9.38/lib/chef/api_client/registration.rb:87:in create_or_update' /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.9.38/lib/chef/api_client/registration.rb:58:inrun’
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.9.38/lib/chef/client.rb:618:in register' /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.9.38/lib/chef/client.rb:265:inrun’
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.9.38/lib/chef/application.rb:285:in block in fork_chef_client' /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.9.38/lib/chef/application.rb:273:infork’
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.9.38/lib/chef/application.rb:273:in fork_chef_client' /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.9.38/lib/chef/application.rb:238:inblock in run_chef_client’
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.9.38/lib/chef/local_mode.rb:44:in with_server_connectivity' /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.9.38/lib/chef/application.rb:226:inrun_chef_client’
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.9.38/lib/chef/application/client.rb:456:in sleep_then_run_chef_client' /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.9.38/lib/chef/application/client.rb:443:inblock in interval_run_chef_client’
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.9.38/lib/chef/application/client.rb:442:in loop' /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.9.38/lib/chef/application/client.rb:442:ininterval_run_chef_client’
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.9.38/lib/chef/application/client.rb:426:in run_application' /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.9.38/lib/chef/application.rb:58:inrun’
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.9.38/bin/chef-client:26:in <top (required)>' /usr/bin/chef-client:51:inload’
/usr/bin/chef-client:51:in `’[root@bcdefg150 ~]#
Note: knife ssl check from chef workstation to chef server works fine.
[root@iyyrpbxxxxx chef-repo]# knife ssl check
Connecting to host bcmtpc065.xxxx.com:443
Successfully verified certificates from `bcmtpc065.xxxx.com’
Is there a proxy server in your environment that handles SSL traffic? We have to explicitly set the no_proxy setting in the client.rb and in the shell environment to bypass our proxy for all Chef servers.
You can also try adding the following lines to your client.rb:
ssl_verify_mode :verify_none
verify_api_cert false
I have tried the above options but still it is not resolved, getting same error. Also, I have checked that no proxy is set anywhere - chef server, workstation or client/node.
It sounds like you need to manually run knife ssl fetch on the node, then rerun chef-client. At that point, the node attributes should be pushed up to the chef server and further knife ssh attempts should work. Right now it doesn’t know the IP of the node because chef-client hasn’t run successfully.
Nathan Clemons
DevOps Engineer
Moxie Cloud Services (MCS)
O +1.425.467.5075
M +1.360.861.6291
E nclemons@gomoxie.com
W www.gomoxie.comhttp://www.gomoxie.com/
I have taken another machine to try to bootstrap as a chef node, this time it proceeds more but still gives an error at the end:
knife bootstrap <> --ssh-user <> --ssh-password <> --sudo --use-sudo-password --node-name node1
xx.xx.xx.xx Setting up chef (12.9.41-1) …
xx.xx.xx.xx Thank you for installing Chef!
xx.xx.xx.xx Starting the first Chef Client run…
xx.xx.xx.xx Starting Chef Client, version 12.9.41
xx.xx.xx.xx Creating a new client identity for node1 using the validator key.
xx.xx.xx.xx resolving cookbooks for run list: []
xx.xx.xx.xx Synchronizing Cookbooks:
xx.xx.xx.xx Installing Cookbook Gems:
xx.xx.xx.xx Compiling Cookbooks…
xx.xx.xx.xx [2016-05-10T22:46:55+05:30] WARN: Node node1 has an empty run list.
xx.xx.xx.xx Converging 0 resources
xx.xx.xx.xx
xx.xx.xx.xx Running handlers:
xx.xx.xx.xx [2016-05-10T22:46:55+05:30] ERROR: Running exception handlers
xx.xx.xx.xx Running handlers complete
xx.xx.xx.xx [2016-05-10T22:46:55+05:30] ERROR: Exception handlers complete
xx.xx.xx.xx Chef Client failed. 0 resources updated in 18 seconds
xx.xx.xx.xx [2016-05-10T22:46:56+05:30] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
xx.xx.xx.xx [2016-05-10T22:46:56+05:30] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
xx.xx.xx.xx [2016-05-10T22:46:56+05:30] ERROR: 403 "Forbidden"
xx.xx.xx.xx [2016-05-10T22:46:56+05:30] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
The stacktrace:
cat /var/chef/cache/chef-stacktrace.out
Generated at 2016-05-10 23:14:15 +0530
Net::HTTPServerException: 403 “Forbidden”
/opt/chef/embedded/lib/ruby/2.1.0/net/http/response.rb:119:in error!' /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.9.41/lib/chef/http.rb:146:inrequest’
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.9.41/lib/chef/http.rb:119:in put' /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.9.41/lib/chef/node.rb:620:insave’
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.9.41/lib/chef/client.rb:542:in save_updated_node' /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.9.41/lib/chef/client.rb:704:inconverge_and_save’
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.9.41/lib/chef/client.rb:281:in run' /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.9.41/lib/chef/application.rb:285:inblock in fork_chef_client’
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.9.41/lib/chef/application.rb:273:in fork' /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.9.41/lib/chef/application.rb:273:infork_chef_client’
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.9.41/lib/chef/application.rb:238:in block in run_chef_client' /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.9.41/lib/chef/local_mode.rb:44:inwith_server_connectivity’
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.9.41/lib/chef/application.rb:226:in run_chef_client' /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.9.41/lib/chef/application/client.rb:456:insleep_then_run_chef_client’
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.9.41/lib/chef/application/client.rb:443:in block in interval_run_chef_client' /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.9.41/lib/chef/application/client.rb:442:inloop’
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.9.41/lib/chef/application/client.rb:442:in interval_run_chef_client' /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.9.41/lib/chef/application/client.rb:426:inrun_application’
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.9.41/lib/chef/application.rb:58:in run' /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.9.41/bin/chef-client:26:in<top (required)>’
/usr/bin/chef-client:51:in load' /usr/bin/chef-client:51:in’
That sounds like it was unable to save the data back to the chef server, likely due to an ACL issue. Did the node create itself with the validator, or did you create it ahead of time during the bootstrap command?
Nathan Clemons
DevOps Engineer
Moxie Cloud Services (MCS)
O +1.425.467.5075
M +1.360.861.6291
E nclemons@gomoxie.com
W www.gomoxie.comhttp://www.gomoxie.com/
I created the node with the command - knife node create node1
After that I ran the knife bootstrap command.
You don’t need to create the node first…maybe that is part of the issue?
Delete the node and the client from the chef server, then /etc/chef/ on the client, then bootstrap again.
Chris Cureau
DevOps and Continuous Delivery Engineer
Industrial Internet
GE Oil & Gas
T: +1 504 595 1339
M: +1 504 444 8668
christopher.cureau@ge.com
201 St. Charles Avenue, Suite 3000
New Orleans, LA 70170, US
GE - imagination at work
Yeah, by creating the node first you’re setting yourself as the “owner” for ACL purposes, and then the client isn’t able to update its data because of permission issues. If the client creates the node, it will be the owner and can update it, and if you’re an admin, you’ll also be able to do so still.
Nathan Clemons
DevOps Engineer
Moxie Cloud Services (MCS)
O +1.425.467.5075
M +1.360.861.6291
E nclemons@gomoxie.com
W www.gomoxie.comhttp://www.gomoxie.com/
You could also use something like knife-acl to correct the permissions on the node object, but deleting the node, client, server client config and having the client reregister is the fastest way in my opinion.
Nathan Clemons
DevOps Engineer
Moxie Cloud Services (MCS)
O +1.425.467.5075
M +1.360.861.6291
E nclemons@gomoxie.com
W www.gomoxie.comhttp://www.gomoxie.com/
Thanks a lot Chris and Nathan, that solved the issue. I deleted the node and the client, removed /etc/chef on the client and reran bootstrap again, it finished successfully.
knife bootstrap xx.xx.xx.xx --ssh-user <> --ssh-password <> --sudo --use-sudo-password --node-name node1
Doing old-style registration with the validation key at /root/learn-chef/chef-repo/.chef/xxx-validator.pem…
Delete your validation key in order to use your user credentials instead
Connecting to xx.xx.xx.xx
xx.xx.xx.xx /etc/bash.bashrc: line 6: /usr/share/lsf/conf/profile.lsf: No such file or directory
xx.xx.xx.xx -----> Existing Chef installation detected
xx.xx.xx.xx Starting the first Chef Client run…
xx.xx.xx.xx Starting Chef Client, version 12.9.41
xx.xx.xx.xx Creating a new client identity for node1 using the validator key.
xx.xx.xx.xx resolving cookbooks for run list: []
xx.xx.xx.xx Synchronizing Cookbooks:
xx.xx.xx.xx Installing Cookbook Gems:
xx.xx.xx.xx Compiling Cookbooks…
xx.xx.xx.xx [2016-05-11T00:03:43+05:30] WARN: Node node1 has an empty run list.
xx.xx.xx.xx Converging 0 resources
xx.xx.xx.xx
xx.xx.xx.xx Running handlers:
xx.xx.xx.xx Running handlers complete
xx.xx.xx.xx Chef Client finished, 0/0 resources updated in 02 seconds
chef-client run also went fine on the node:
root@abcd120:~# chef-client
Starting Chef Client, version 12.9.41
resolving cookbooks for run list: []
Synchronizing Cookbooks:
Installing Cookbook Gems:
Compiling Cookbooks…
[2016-05-11T00:04:03+05:30] WARN: Node node1 has an empty run list.
Converging 0 resources
Running handlers:
Running handlers complete
Chef Client finished, 0/0 resources updated in 02 seconds