Randomly a chef run will fail - right now I have 4 nodes, all built by
chef, all working like champs, but usually overnight there will be 1 or 2
that fail a chef run, so when I check in the morning they haven’t been
chef’d for 10-12 hours. It says it’s trying again in 1800 seconds, but if
it’s running once an hour it should have succeeded by now… when I login
and run chef-client manually, it just works without me having to restart
chef or kill any processes. Looking at the stacktrace it looks like it
stopped at yum-dump.py - could this be what’s holding it up? And if so, why
doesn’t it clear itself on the next run, since it did when I manually ran
it. Thanks
[…LOGS…]
[2013-11-15T08:05:32-06:00] INFO: Processing yum_key[RPM-GPG-KEY-EPEL-6]
action add (yum::epel line 22)
[2013-11-15T08:05:32-06:00] INFO: Processing yum_repository[epel] action
add (yum::epel line 27)
[2013-11-15T08:05:32-06:00] INFO: Processing package[ntp] action install
(ntp::default line 25)
[2013-11-15T08:20:43-06:00] ERROR:
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py
exceeded timeout 900
[2013-11-15T08:20:43-06:00] INFO: Running queued delayed notifications
before re-raising exception
[2013-11-15T08:20:43-06:00] ERROR: Running exception handlers
[2013-11-15T08:20:43-06:00] ERROR: Exception handlers complete
[2013-11-15T08:20:43-06:00] FATAL: Stacktrace dumped to
/var/chef/cache/chef-stacktrace.out
[2013-11-15T08:20:43-06:00] ERROR: Chef::Exceptions::ChildConvergeError:
Chef run process exited unsuccessfully (exit code 1)
[2013-11-15T08:20:43-06:00] ERROR: Sleeping for 1800 seconds before trying
again
[…STACKTRACE…]
Generated at 2013-11-15 08:20:43 -0600
Mixlib::ShellOut::CommandTimeout: package[ntp] (ntp::default line 25) had
an error: Mixlib::ShellOut::CommandTimeout: command timed out:
---- Begin output of /usr/bin/python
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py
–options --installed-provides ----
STDOUT:
STDERR:
---- End output of /usr/bin/python
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py
–options --installed-provides ----
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout/unix.rb:64:in
run_command' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout.rb:225:inrun_command’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/mixin/shell_out.rb:30:in
shell_out' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/mixin/shell_out.rb:35:inshell_out!’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:714:in
refresh' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:806:inpackage_available?’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:1055:in
load_current_resource' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider.rb:97:inrun_action’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource.rb:625:in
run_action' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:49:inrun_action’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:in
block (2 levels) in converge' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:ineach’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:in
block in converge' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection.rb:98:inblock in execute_each_resource’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:116:in
call' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:116:incall_iterator_block’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:85:in
step' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:104:initerate’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:55:in
each_with_index' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection.rb:96:inexecute_each_resource’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:80:in
converge' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:429:inconverge’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:494:in
do_run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:199:inblock in run’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:193:in
fork' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:193:inrun’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application.rb:183:in
run_chef_client' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:302:inblock in run_application’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:294:in
loop' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:294:inrun_application’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application.rb:66:in
run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/bin/chef-client:26:in<top (required)>’
/usr/bin/chef-client:23:in load' /usr/bin/chef-client:23:in’
–
http://philcryer.com/
Looks like it is a timeout connecting to a package repository to check to
make sure it has the correct version of ntp. These are fairly common in my
experience. The simple solution that catches most issues is to add retries
and retry_delay to all of your package install blocks to get around minor
hiccups.
Old:
package "ntp" do
action :install
end
New:
package "ntp" do
action :install
retry_delay 5
retries 2
end
Additionally, I would recommend setting up a failure handler than emails
you(or dumps somewhere off machine) the Chef run log so you don't have to
do so much digging. The error handler Opscode has is a decent start:
On Fri, Nov 15, 2013 at 9:52 AM, Phil Cryer phil@philcryer.com wrote:
Randomly a chef run will fail - right now I have 4 nodes, all built by
chef, all working like champs, but usually overnight there will be 1 or 2
that fail a chef run, so when I check in the morning they haven't been
chef'd for 10-12 hours. It says it's trying again in 1800 seconds, but if
it's running once an hour it should have succeeded by now... when I login
and run chef-client manually, it just works without me having to restart
chef or kill any processes. Looking at the stacktrace it looks like it
stopped at yum-dump.py - could this be what's holding it up? And if so, why
doesn't it clear itself on the next run, since it did when I manually ran
it. Thanks
[...LOGS...]
[2013-11-15T08:05:32-06:00] INFO: Processing yum_key[RPM-GPG-KEY-EPEL-6]
action add (yum::epel line 22)
[2013-11-15T08:05:32-06:00] INFO: Processing yum_repository[epel] action
add (yum::epel line 27)
[2013-11-15T08:05:32-06:00] INFO: Processing package[ntp] action install
(ntp::default line 25)
[2013-11-15T08:20:43-06:00] ERROR:
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py
exceeded timeout 900
[2013-11-15T08:20:43-06:00] INFO: Running queued delayed notifications
before re-raising exception
[2013-11-15T08:20:43-06:00] ERROR: Running exception handlers
[2013-11-15T08:20:43-06:00] ERROR: Exception handlers complete
[2013-11-15T08:20:43-06:00] FATAL: Stacktrace dumped to
/var/chef/cache/chef-stacktrace.out
[2013-11-15T08:20:43-06:00] ERROR: Chef::Exceptions::ChildConvergeError:
Chef run process exited unsuccessfully (exit code 1)
[2013-11-15T08:20:43-06:00] ERROR: Sleeping for 1800 seconds before trying
again
[...STACKTRACE...]
Generated at 2013-11-15 08:20:43 -0600
Mixlib::ShellOut::CommandTimeout: package[ntp] (ntp::default line 25) had
an error: Mixlib::ShellOut::CommandTimeout: command timed out:
---- Begin output of /usr/bin/python
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py
--options --installed-provides ----
STDOUT:
STDERR:
---- End output of /usr/bin/python
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py
--options --installed-provides ----
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout/unix.rb:64:in
run_command' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout.rb:225:in run_command'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/mixin/shell_out.rb:30:in
shell_out' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/mixin/shell_out.rb:35:in shell_out!'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:714:in
refresh' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:806:in package_available?'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:1055:in
load_current_resource' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider.rb:97:in run_action'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource.rb:625:in
run_action' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:49:in run_action'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:in
block (2 levels) in converge' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:in each'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:in
block in converge' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection.rb:98:in block in execute_each_resource'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:116:in
call' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:116:in call_iterator_block'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:85:in
step' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:104:in iterate'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:55:in
each_with_index' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection.rb:96:in execute_each_resource'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:80:in
converge' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:429:in converge'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:494:in
do_run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:199:in block in run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:193:in
fork' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:193:in run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application.rb:183:in
run_chef_client' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:302:in block in run_application'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:294:in
loop' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:294:in run_application'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application.rb:66:in
run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/bin/chef-client:26:in <top (required)>'
/usr/bin/chef-client:23:in load' /usr/bin/chef-client:23:in '
--
http://philcryer.com/
Thanks, this should really help - is there a way to make this a global
option?
On Fri, Nov 15, 2013 at 9:22 AM, Andrew Gross andrew@yipit.com wrote:
Looks like it is a timeout connecting to a package repository to check to
make sure it has the correct version of ntp. These are fairly common in my
experience. The simple solution that catches most issues is to add retries
and retry_delay to all of your package install blocks to get around minor
hiccups.
Old:
package "ntp" do
action :install
end
New:
package "ntp" do
action :install
retry_delay 5
retries 2
end
Additionally, I would recommend setting up a failure handler than emails
you(or dumps somewhere off machine) the Chef run log so you don't have to
do so much digging. The error handler Opscode has is a decent start:
About Handlers
On Fri, Nov 15, 2013 at 9:52 AM, Phil Cryer phil@philcryer.com wrote:
Randomly a chef run will fail - right now I have 4 nodes, all built by
chef, all working like champs, but usually overnight there will be 1 or 2
that fail a chef run, so when I check in the morning they haven't been
chef'd for 10-12 hours. It says it's trying again in 1800 seconds, but if
it's running once an hour it should have succeeded by now... when I login
and run chef-client manually, it just works without me having to restart
chef or kill any processes. Looking at the stacktrace it looks like it
stopped at yum-dump.py - could this be what's holding it up? And if so, why
doesn't it clear itself on the next run, since it did when I manually ran
it. Thanks
[...LOGS...]
[2013-11-15T08:05:32-06:00] INFO: Processing yum_key[RPM-GPG-KEY-EPEL-6]
action add (yum::epel line 22)
[2013-11-15T08:05:32-06:00] INFO: Processing yum_repository[epel] action
add (yum::epel line 27)
[2013-11-15T08:05:32-06:00] INFO: Processing package[ntp] action install
(ntp::default line 25)
[2013-11-15T08:20:43-06:00] ERROR:
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py
exceeded timeout 900
[2013-11-15T08:20:43-06:00] INFO: Running queued delayed notifications
before re-raising exception
[2013-11-15T08:20:43-06:00] ERROR: Running exception handlers
[2013-11-15T08:20:43-06:00] ERROR: Exception handlers complete
[2013-11-15T08:20:43-06:00] FATAL: Stacktrace dumped to
/var/chef/cache/chef-stacktrace.out
[2013-11-15T08:20:43-06:00] ERROR: Chef::Exceptions::ChildConvergeError:
Chef run process exited unsuccessfully (exit code 1)
[2013-11-15T08:20:43-06:00] ERROR: Sleeping for 1800 seconds before
trying again
[...STACKTRACE...]
Generated at 2013-11-15 08:20:43 -0600
Mixlib::ShellOut::CommandTimeout: package[ntp] (ntp::default line 25) had
an error: Mixlib::ShellOut::CommandTimeout: command timed out:
---- Begin output of /usr/bin/python
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py
--options --installed-provides ----
STDOUT:
STDERR:
---- End output of /usr/bin/python
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py
--options --installed-provides ----
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout/unix.rb:64:in
run_command' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout.rb:225:in run_command'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/mixin/shell_out.rb:30:in
shell_out' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/mixin/shell_out.rb:35:in shell_out!'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:714:in
refresh' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:806:in package_available?'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:1055:in
load_current_resource' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider.rb:97:in run_action'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource.rb:625:in
run_action' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:49:in run_action'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:in
block (2 levels) in converge' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:in each'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:in
block in converge' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection.rb:98:in block in execute_each_resource'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:116:in
call' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:116:in call_iterator_block'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:85:in
step' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:104:in iterate'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:55:in
each_with_index' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection.rb:96:in execute_each_resource'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:80:in
converge' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:429:in converge'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:494:in
do_run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:199:in block in run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:193:in
fork' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:193:in run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application.rb:183:in
run_chef_client' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:302:in block in run_application'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:294:in
loop' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:294:in run_application'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application.rb:66:in
run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/bin/chef-client:26:in <top (required)>'
/usr/bin/chef-client:23:in load' /usr/bin/chef-client:23:in '
--
http://philcryer.com/
--
http://philcryer.com/
Not yet, I believe there will be some support for this sort of thing in the
next major Chef version, but I am not 100% sure.
On Fri, Nov 15, 2013 at 11:05 AM, Phil Cryer phil@philcryer.com wrote:
Thanks, this should really help - is there a way to make this a global
option?
On Fri, Nov 15, 2013 at 9:22 AM, Andrew Gross andrew@yipit.com wrote:
Looks like it is a timeout connecting to a package repository to check to
make sure it has the correct version of ntp. These are fairly common in my
experience. The simple solution that catches most issues is to add retries
and retry_delay to all of your package install blocks to get around minor
hiccups.
Old:
package "ntp" do
action :install
end
New:
package "ntp" do
action :install
retry_delay 5
retries 2
end
Additionally, I would recommend setting up a failure handler than emails
you(or dumps somewhere off machine) the Chef run log so you don't have to
do so much digging. The error handler Opscode has is a decent start:
About Handlers
On Fri, Nov 15, 2013 at 9:52 AM, Phil Cryer phil@philcryer.com wrote:
Randomly a chef run will fail - right now I have 4 nodes, all built by
chef, all working like champs, but usually overnight there will be 1 or 2
that fail a chef run, so when I check in the morning they haven't been
chef'd for 10-12 hours. It says it's trying again in 1800 seconds, but if
it's running once an hour it should have succeeded by now... when I login
and run chef-client manually, it just works without me having to restart
chef or kill any processes. Looking at the stacktrace it looks like it
stopped at yum-dump.py - could this be what's holding it up? And if so, why
doesn't it clear itself on the next run, since it did when I manually ran
it. Thanks
[...LOGS...]
[2013-11-15T08:05:32-06:00] INFO: Processing yum_key[RPM-GPG-KEY-EPEL-6]
action add (yum::epel line 22)
[2013-11-15T08:05:32-06:00] INFO: Processing yum_repository[epel] action
add (yum::epel line 27)
[2013-11-15T08:05:32-06:00] INFO: Processing package[ntp] action install
(ntp::default line 25)
[2013-11-15T08:20:43-06:00] ERROR:
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py
exceeded timeout 900
[2013-11-15T08:20:43-06:00] INFO: Running queued delayed notifications
before re-raising exception
[2013-11-15T08:20:43-06:00] ERROR: Running exception handlers
[2013-11-15T08:20:43-06:00] ERROR: Exception handlers complete
[2013-11-15T08:20:43-06:00] FATAL: Stacktrace dumped to
/var/chef/cache/chef-stacktrace.out
[2013-11-15T08:20:43-06:00] ERROR: Chef::Exceptions::ChildConvergeError:
Chef run process exited unsuccessfully (exit code 1)
[2013-11-15T08:20:43-06:00] ERROR: Sleeping for 1800 seconds before
trying again
[...STACKTRACE...]
Generated at 2013-11-15 08:20:43 -0600
Mixlib::ShellOut::CommandTimeout: package[ntp] (ntp::default line 25)
had an error: Mixlib::ShellOut::CommandTimeout: command timed out:
---- Begin output of /usr/bin/python
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py
--options --installed-provides ----
STDOUT:
STDERR:
---- End output of /usr/bin/python
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py
--options --installed-provides ----
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout/unix.rb:64:in
run_command' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout.rb:225:in run_command'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/mixin/shell_out.rb:30:in
shell_out' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/mixin/shell_out.rb:35:in shell_out!'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:714:in
refresh' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:806:in package_available?'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:1055:in
load_current_resource' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider.rb:97:in run_action'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource.rb:625:in
run_action' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:49:in run_action'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:in
block (2 levels) in converge' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:in each'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:in
block in converge' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection.rb:98:in block in execute_each_resource'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:116:in
call' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:116:in call_iterator_block'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:85:in
step' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:104:in iterate'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:55:in
each_with_index' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection.rb:96:in execute_each_resource'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:80:in
converge' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:429:in converge'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:494:in
do_run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:199:in block in run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:193:in
fork' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:193:in run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application.rb:183:in
run_chef_client' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:302:in block in run_application'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:294:in
loop' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:294:in run_application'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application.rb:66:in
run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/bin/chef-client:26:in <top (required)>'
/usr/bin/chef-client:23:in load' /usr/bin/chef-client:23:in '
--
http://philcryer.com/
--
http://philcryer.com/