I have recently upgraded from Chef Server 11 to 12 and have come across a
very frustrating issue.
I have a set of nodes that are converging fine and then poof they
spontaneously can not set their node state back up to the chef server at
the end of the run.
Can anyone point out some things I might do to troubleshoot this issue. I
am including what I think are helpful logs but I am sure there is more than
I can submit.
The only way that I can fix this is by deleting the node and client and
reestablish them.
Thanks!
Server = CentOS 6.6
########################
This on the client
########################
Running handlers:
[2015-01-02T10:26:00-08:00] ERROR: Running exception handlers
Running handlers complete
[2015-01-02T10:26:00-08:00] ERROR: Exception handlers complete
[2015-01-02T10:26:00-08:00] FATAL: Stacktrace dumped to
/var/chef/cache/chef-stacktrace.out
Chef Client failed. 0 resources updated in 4.740723394 seconds
[2015-01-02T10:26:00-08:00] ERROR: 403 “Forbidden”
[2015-01-02T10:26:00-08:00] FATAL: Chef::Exceptions::ChildConvergeError:
Chef run process exited unsuccessfully (exit code 1)
cat /var/chef/cache/chef-stacktrace.out
Generated at 2015-01-02 10:26:00 -0800
Net::HTTPServerException: 403 “Forbidden”
/opt/chef/embedded/lib/ruby/2.1.0/net/http/response.rb:119:in error!' /opt/chef/embedded/apps/chef/lib/chef/http.rb:145:inrequest’
/opt/chef/embedded/apps/chef/lib/chef/http.rb:118:in put' /opt/chef/embedded/apps/chef/lib/chef/node.rb:512:insave’
/opt/chef/embedded/apps/chef/lib/chef/client.rb:257:in save_updated_node' /opt/chef/embedded/apps/chef/lib/chef/client.rb:403:inrun’
/opt/chef/embedded/apps/chef/lib/chef/application.rb:261:in block in fork_chef_client' /opt/chef/embedded/apps/chef/lib/chef/application.rb:249:infork’
/opt/chef/embedded/apps/chef/lib/chef/application.rb:249:in
fork_chef_client' /opt/chef/embedded/apps/chef/lib/chef/application.rb:215:inblock in
run_chef_client’
/opt/chef/embedded/apps/chef/lib/chef/local_mode.rb:38:in
with_server_connectivity' /opt/chef/embedded/apps/chef/lib/chef/application.rb:201:inrun_chef_client’
/opt/chef/embedded/apps/chef/lib/chef/application/client.rb:355:in block in interval_run_chef_client' /opt/chef/embedded/apps/chef/lib/chef/application/client.rb:345:inloop’
/opt/chef/embedded/apps/chef/lib/chef/application/client.rb:345:in
interval_run_chef_client' /opt/chef/embedded/apps/chef/lib/chef/application/client.rb:335:inrun_application’
/opt/chef/embedded/apps/chef/lib/chef/application.rb:58:in run' /opt/chef/embedded/apps/chef/bin/chef-client:26:in<top (required)>’
/usr/bin/chef-client:40:in load' /usr/bin/chef-client:40:in’
########################
This on the server
########################
from /var/log/opscode/opscode-erchef/requests.log.1
2015-01-01T14:19:15Z erchef@127.0.0.1 method=PUT;
path=/organizations/example/nodes/serv.example.com; status=200;
req_id=g3IAA2QAEGVyY2hlZkAxMjcuMC4wLjEBAAA4PQAAAAgAAAAA; org_name=example;
couchdb_groups=false; couchdb_organizations=false;
couchdb_containers=false; couchdb_acls=false; 503_mode=false;
couchdb_associations=false; couchdb_association_requests=false;
req_time=18; rdbms_time=5; rdbms_count=4; authz_time=3; authz_count=1; user=
serv.example.com;
2015-01-01T14:52:34Z erchef@127.0.0.1 method=PUT;
path=/organizations/example/nodes/serv.example.com; status=200;
req_id=g3IAA2QAEGVyY2hlZkAxMjcuMC4wLjEBAAC2vQAAAAgAAAAA; org_name=example;
couchdb_groups=false; couchdb_organizations=false;
couchdb_containers=false; couchdb_acls=false; 503_mode=false;
couchdb_associations=false; couchdb_association_requests=false;
req_time=17; rdbms_time=5; rdbms_count=4; authz_time=3; authz_count=1; user=
serv.example.com;
2015-01-01T15:22:44Z erchef@127.0.0.1 method=PUT;
path=/organizations/example/nodes/serv.example.com; status=200;
req_id=g3IAA2QAEGVyY2hlZkAxMjcuMC4wLjEBAAEgewAAAAgAAAAA; org_name=example;
couchdb_groups=false; couchdb_organizations=false;
couchdb_containers=false; couchdb_acls=false; 503_mode=false;
couchdb_associations=false; couchdb_association_requests=false;
req_time=17; rdbms_time=6; rdbms_count=4; authz_time=3; authz_count=1; user=
serv.example.com;
2015-01-01T15:56:29Z erchef@127.0.0.1 method=PUT;
path=/organizations/example/nodes/serv.example.com; status=200;
req_id=g3IAA2QAEGVyY2hlZkAxMjcuMC4wLjEBAAGgNwAAAAgAAAAA; org_name=example;
couchdb_groups=false; couchdb_organizations=false;
couchdb_containers=false; couchdb_acls=false; 503_mode=false;
couchdb_associations=false; couchdb_association_requests=false;
req_time=17; rdbms_time=5; rdbms_count=4; authz_time=3; authz_count=1; user=
serv.example.com;
2015-01-01T16:39:35Z erchef@127.0.0.1 method=PUT;
path=/organizations/example/nodes/serv.example.com; status=403;
req_id=g3IAA2QAEGVyY2hlZkAxMjcuMC4wLjEBAAJMIQAAAAgAAAAA; org_name=example;
msg={update,forbidden}; couchdb_groups=false; couchdb_organizations=false;
couchdb_containers=false; couchdb_acls=false; 503_mode=false;
couchdb_associations=false; couchdb_association_requests=false; req_time=7;
rdbms_time=0; rdbms_count=3; authz_time=4; authz_count=1; user=
serv.example.com;
2015-01-01T17:02:59Z erchef@127.0.0.1 method=PUT;
path=/organizations/example/nodes/serv.example.com; status=403;
req_id=g3IAA2QAEGVyY2hlZkAxMjcuMC4wLjEBAAKU/gAAAAgAAAAA; org_name=example;
msg={update,forbidden}; couchdb_groups=false; couchdb_organizations=false;
couchdb_containers=false; couchdb_acls=false; 503_mode=false;
couchdb_associations=false; couchdb_association_requests=false; req_time=7;
rdbms_time=0; rdbms_count=3; authz_time=3; authz_count=1; user=
serv.example.com;
2015-01-01T17:34:07Z erchef@127.0.0.1 method=PUT;
path=/organizations/example/nodes/serv.example.com; status=403;
req_id=g3IAA2QAEGVyY2hlZkAxMjcuMC4wLjEBAAMKLAAAAAgAAAAA; org_name=example;
msg={update,forbidden}; couchdb_groups=false; couchdb_organizations=false;
couchdb_containers=false; couchdb_acls=false; 503_mode=false;
couchdb_associations=false; couchdb_association_requests=false; req_time=7;
rdbms_time=0; rdbms_count=3; authz_time=3; authz_count=1; user=
serv.example.com;
2015-01-01T18:06:20Z erchef@127.0.0.1 method=PUT;
path=/organizations/example/nodes/serv.example.com; status=403;
req_id=g3IAA2QAEGVyY2hlZkAxMjcuMC4wLjEBAAN/AwAAAAgAAAAA; org_name=example;
msg={update,forbidden}; couchdb_groups=false; couchdb_organizations=false;
couchdb_containers=false; couchdb_acls=false; 503_mode=false;
couchdb_associations=false; couchdb_association_requests=false; req_time=7;
rdbms_time=0; rdbms_count=3; authz_time=3; authz_count=1; user=
serv.example.com;
2015-01-01T18:36:35Z erchef@127.0.0.1 method=PUT;
path=/organizations/example/nodes/serv.example.com; status=403;
req_id=g3IAA2QAEGVyY2hlZkAxMjcuMC4wLjEBAAPs6gAAAAgAAAAA; org_name=example;
msg={update,forbidden}; couchdb_groups=false; couchdb_organizations=false;
couchdb_containers=false; couchdb_acls=false; 503_mode=false;
couchdb_associations=false; couchdb_association_requests=false; req_time=7;
rdbms_time=0; rdbms_count=3; authz_time=3; authz_count=1; user=
serv.example.com;