# Unattended chef-client runs fail

**URL:** <https://discourse.chef.io/t/unattended-chef-client-runs-fail/4683>\
**Category:** Chef Infra (archive)\
**Created:** [November 15, 2013, 2:52pm UTC](https://discourse.chef.io/t/unattended-chef-client-runs-fail/4683 "2013-11-15T14:52:47Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![philcryer](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/philcryer/32/404_2.png) [@philcryer](https://discourse.chef.io/u/philcryer)\
**Post date:** [November 15, 2013, 2:52pm UTC](https://discourse.chef.io/t/unattended-chef-client-runs-fail/4683/1 "2013-11-15T14:52:47Z")

</div>

Randomly a chef run will fail - right now I have 4 nodes, all built by  
chef, all working like champs, but usually overnight there will be 1 or 2  
that fail a chef run, so when I check in the morning they haven’t been  
chef’d for 10-12 hours. It says it’s trying again in 1800 seconds, but if  
it’s running once an hour it should have succeeded by now… when I login  
and run chef-client manually, it just works without me having to restart  
chef or kill any processes. Looking at the stacktrace it looks like it  
stopped at [yum-dump.py](http://yum-dump.py) - could this be what’s holding it up? And if so, why  
doesn’t it clear itself on the next run, since it did when I manually ran  
it. Thanks

[…LOGS…]  
[2013-11-15T08:05:32-06:00] INFO: Processing yum\_key[RPM-GPG-KEY-EPEL-6]  
action add (yum::epel line 22)  
[2013-11-15T08:05:32-06:00] INFO: Processing yum\_repository[epel] action  
add (yum::epel line 27)  
[2013-11-15T08:05:32-06:00] INFO: Processing package[ntp] action install  
(ntp::default line 25)  
[2013-11-15T08:20:43-06:00] ERROR:  
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py  
exceeded timeout 900  
[2013-11-15T08:20:43-06:00] INFO: Running queued delayed notifications  
before re-raising exception  
[2013-11-15T08:20:43-06:00] ERROR: Running exception handlers  
[2013-11-15T08:20:43-06:00] ERROR: Exception handlers complete  
[2013-11-15T08:20:43-06:00] FATAL: Stacktrace dumped to  
/var/chef/cache/chef-stacktrace.out  
[2013-11-15T08:20:43-06:00] ERROR: Chef::Exceptions::ChildConvergeError:  
Chef run process exited unsuccessfully (exit code 1)  
[2013-11-15T08:20:43-06:00] ERROR: Sleeping for 1800 seconds before trying  
again

[…STACKTRACE…]  
Generated at 2013-11-15 08:20:43 -0600  
Mixlib::ShellOut::CommandTimeout: package[ntp] (ntp::default line 25) had  
an error: Mixlib::ShellOut::CommandTimeout: command timed out:  
---- Begin output of /usr/bin/python  
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py  
–options --installed-provides ----  
STDOUT:  
STDERR:  
---- End output of /usr/bin/python  
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py  
–options --installed-provides ----

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout/unix.rb:64:in  
`run_command' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout.rb:225:in`run\_command’  
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/mixin/shell\_out.rb:30:in  
`shell_out' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/mixin/shell_out.rb:35:in`shell\_out!’  
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:714:in  
`refresh' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:806:in`package\_available?’  
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:1055:in  
`load_current_resource' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider.rb:97:in`run\_action’  
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource.rb:625:in  
`run_action' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:49:in`run\_action’  
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:in  
`block (2 levels) in converge' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:in`each’  
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:in  
`block in converge' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection.rb:98:in`block in execute\_each\_resource’  
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource\_collection/stepable\_iterator.rb:116:in  
`call' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:116:in`call\_iterator\_block’  
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource\_collection/stepable\_iterator.rb:85:in  
`step' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:104:in`iterate’  
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource\_collection/stepable\_iterator.rb:55:in  
`each_with_index' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection.rb:96:in`execute\_each\_resource’  
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:80:in  
`converge' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:429:in`converge’  
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:494:in  
`do_run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:199:in`block in run’  
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:193:in  
`fork' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:193:in`run’  
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application.rb:183:in  
`run_chef_client' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:302:in`block in run\_application’  
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:294:in  
`loop' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:294:in`run\_application’  
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application.rb:66:in  
`run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/bin/chef-client:26:in`\<top (required)\>’  
/usr/bin/chef-client:23:in `load' /usr/bin/chef-client:23:in`’

–  
[http://philcryer.com/](http://philcryer.com/)

---

<div class="post-metadata">

**Author:** ![Andrew\_Gross](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/andrew_gross/32/320_2.png) [@Andrew\_Gross](https://discourse.chef.io/u/Andrew_Gross)\
**Post date:** [November 15, 2013, 3:22pm UTC](https://discourse.chef.io/t/unattended-chef-client-runs-fail/4683/2 "2013-11-15T15:22:14Z")

</div>

Looks like it is a timeout connecting to a package repository to check to  
make sure it has the correct version of ntp. These are fairly common in my  
experience. The simple solution that catches most issues is to add retries  
and retry\_delay to all of your package install blocks to get around minor  
hiccups.

Old:

package "ntp" do  
action :install  
end

New:

package "ntp" do  
action :install  
retry\_delay 5  
retries 2  
end

Additionally, I would recommend setting up a failure handler than emails  
you(or dumps somewhere off machine) the Chef run log so you don't have to  
do so much digging. The error handler Opscode has is a decent start:

> **[About Handlers](https://docs.chef.io/handlers/)**
>
> Use a handler to identify situations that arise during a Chef Infra Client run, and then tell Chef Infra Client how to handle these situations when they occur.

On Fri, Nov 15, 2013 at 9:52 AM, Phil Cryer [phil@philcryer.com](mailto:phil@philcryer.com) wrote:

> Randomly a chef run will fail - right now I have 4 nodes, all built by  
> chef, all working like champs, but usually overnight there will be 1 or 2  
> that fail a chef run, so when I check in the morning they haven't been  
> chef'd for 10-12 hours. It says it's trying again in 1800 seconds, but if  
> it's running once an hour it should have succeeded by now... when I login  
> and run chef-client manually, it just works without me having to restart  
> chef or kill any processes. Looking at the stacktrace it looks like it  
> stopped at yum-dump.py - could this be what's holding it up? And if so, why  
> doesn't it clear itself on the next run, since it did when I manually ran  
> it. Thanks
> 
> [...LOGS...]  
> [2013-11-15T08:05:32-06:00] INFO: Processing yum\_key[RPM-GPG-KEY-EPEL-6]  
> action add (yum::epel line 22)  
> [2013-11-15T08:05:32-06:00] INFO: Processing yum\_repository[epel] action  
> add (yum::epel line 27)  
> [2013-11-15T08:05:32-06:00] INFO: Processing package[ntp] action install  
> (ntp::default line 25)  
> [2013-11-15T08:20:43-06:00] ERROR:  
> /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py  
> exceeded timeout 900  
> [2013-11-15T08:20:43-06:00] INFO: Running queued delayed notifications  
> before re-raising exception  
> [2013-11-15T08:20:43-06:00] ERROR: Running exception handlers  
> [2013-11-15T08:20:43-06:00] ERROR: Exception handlers complete  
> [2013-11-15T08:20:43-06:00] FATAL: Stacktrace dumped to  
> /var/chef/cache/chef-stacktrace.out  
> [2013-11-15T08:20:43-06:00] ERROR: Chef::Exceptions::ChildConvergeError:  
> Chef run process exited unsuccessfully (exit code 1)  
> [2013-11-15T08:20:43-06:00] ERROR: Sleeping for 1800 seconds before trying  
> again
> 
> [...STACKTRACE...]  
> Generated at 2013-11-15 08:20:43 -0600  
> Mixlib::ShellOut::CommandTimeout: package[ntp] (ntp::default line 25) had  
> an error: Mixlib::ShellOut::CommandTimeout: command timed out:  
> ---- Begin output of /usr/bin/python  
> /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py  
> --options --installed-provides ----  
> STDOUT:  
> STDERR:  
> ---- End output of /usr/bin/python  
> /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py  
> --options --installed-provides ----
> 
> /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout/unix.rb:64:in  
> `run_command' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout.rb:225:in `run\_command'  
> /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/mixin/shell\_out.rb:30:in  
> `shell_out' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/mixin/shell_out.rb:35:in `shell\_out!'  
> /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:714:in  
> `refresh' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:806:in `package\_available?'  
> /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:1055:in  
> `load_current_resource' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider.rb:97:in `run\_action'  
> /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource.rb:625:in  
> `run_action' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:49:in `run\_action'  
> /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:in  
> `block (2 levels) in converge' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:in `each'  
> /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:in  
> `block in converge' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection.rb:98:in `block in execute\_each\_resource'  
> /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource\_collection/stepable\_iterator.rb:116:in  
> `call' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:116:in `call\_iterator\_block'  
> /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource\_collection/stepable\_iterator.rb:85:in  
> `step' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:104:in `iterate'  
> /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource\_collection/stepable\_iterator.rb:55:in  
> `each_with_index' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection.rb:96:in `execute\_each\_resource'  
> /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:80:in  
> `converge' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:429:in `converge'  
> /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:494:in  
> `do_run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:199:in `block in run'  
> /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:193:in  
> `fork' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:193:in `run'  
> /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application.rb:183:in  
> `run_chef_client' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:302:in `block in run\_application'  
> /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:294:in  
> `loop' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:294:in `run\_application'  
> /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application.rb:66:in  
> `run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/bin/chef-client:26:in `\<top (required)\>'  
> /usr/bin/chef-client:23:in `load' /usr/bin/chef-client:23:in `'
> 
> --  
> [http://philcryer.com/](http://philcryer.com/)

---

<div class="post-metadata">

**Author:** ![philcryer](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/philcryer/32/404_2.png) [@philcryer](https://discourse.chef.io/u/philcryer)\
**Post date:** [November 15, 2013, 4:05pm UTC](https://discourse.chef.io/t/unattended-chef-client-runs-fail/4683/3 "2013-11-15T16:05:03Z")

</div>

Thanks, this should really help - is there a way to make this a global  
option?

On Fri, Nov 15, 2013 at 9:22 AM, Andrew Gross [andrew@yipit.com](mailto:andrew@yipit.com) wrote:

> Looks like it is a timeout connecting to a package repository to check to  
> make sure it has the correct version of ntp. These are fairly common in my  
> experience. The simple solution that catches most issues is to add retries  
> and retry\_delay to all of your package install blocks to get around minor  
> hiccups.
> 
> Old:
> 
> package "ntp" do  
> action :install  
> end
> 
> New:
> 
> package "ntp" do  
> action :install  
> retry\_delay 5  
> retries 2  
> end
> 
> Additionally, I would recommend setting up a failure handler than emails  
> you(or dumps somewhere off machine) the Chef run log so you don't have to  
> do so much digging. The error handler Opscode has is a decent start:  
> [About Handlers](http://docs.opscode.com/essentials_handlers.html)
> 
> On Fri, Nov 15, 2013 at 9:52 AM, Phil Cryer [phil@philcryer.com](mailto:phil@philcryer.com) wrote:
> 
> > Randomly a chef run will fail - right now I have 4 nodes, all built by  
> > chef, all working like champs, but usually overnight there will be 1 or 2  
> > that fail a chef run, so when I check in the morning they haven't been  
> > chef'd for 10-12 hours. It says it's trying again in 1800 seconds, but if  
> > it's running once an hour it should have succeeded by now... when I login  
> > and run chef-client manually, it just works without me having to restart  
> > chef or kill any processes. Looking at the stacktrace it looks like it  
> > stopped at yum-dump.py - could this be what's holding it up? And if so, why  
> > doesn't it clear itself on the next run, since it did when I manually ran  
> > it. Thanks
> > 
> > [...LOGS...]  
> > [2013-11-15T08:05:32-06:00] INFO: Processing yum\_key[RPM-GPG-KEY-EPEL-6]  
> > action add (yum::epel line 22)  
> > [2013-11-15T08:05:32-06:00] INFO: Processing yum\_repository[epel] action  
> > add (yum::epel line 27)  
> > [2013-11-15T08:05:32-06:00] INFO: Processing package[ntp] action install  
> > (ntp::default line 25)  
> > [2013-11-15T08:20:43-06:00] ERROR:  
> > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py  
> > exceeded timeout 900  
> > [2013-11-15T08:20:43-06:00] INFO: Running queued delayed notifications  
> > before re-raising exception  
> > [2013-11-15T08:20:43-06:00] ERROR: Running exception handlers  
> > [2013-11-15T08:20:43-06:00] ERROR: Exception handlers complete  
> > [2013-11-15T08:20:43-06:00] FATAL: Stacktrace dumped to  
> > /var/chef/cache/chef-stacktrace.out  
> > [2013-11-15T08:20:43-06:00] ERROR: Chef::Exceptions::ChildConvergeError:  
> > Chef run process exited unsuccessfully (exit code 1)  
> > [2013-11-15T08:20:43-06:00] ERROR: Sleeping for 1800 seconds before  
> > trying again
> > 
> > [...STACKTRACE...]  
> > Generated at 2013-11-15 08:20:43 -0600  
> > Mixlib::ShellOut::CommandTimeout: package[ntp] (ntp::default line 25) had  
> > an error: Mixlib::ShellOut::CommandTimeout: command timed out:  
> > ---- Begin output of /usr/bin/python  
> > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py  
> > --options --installed-provides ----  
> > STDOUT:  
> > STDERR:  
> > ---- End output of /usr/bin/python  
> > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py  
> > --options --installed-provides ----
> > 
> > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout/unix.rb:64:in  
> > `run_command' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout.rb:225:in `run\_command'  
> > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/mixin/shell\_out.rb:30:in  
> > `shell_out' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/mixin/shell_out.rb:35:in `shell\_out!'  
> > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:714:in  
> > `refresh' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:806:in `package\_available?'  
> > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:1055:in  
> > `load_current_resource' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider.rb:97:in `run\_action'  
> > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource.rb:625:in  
> > `run_action' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:49:in `run\_action'  
> > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:in  
> > `block (2 levels) in converge' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:in `each'  
> > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:in  
> > `block in converge' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection.rb:98:in `block in execute\_each\_resource'  
> > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource\_collection/stepable\_iterator.rb:116:in  
> > `call' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:116:in `call\_iterator\_block'  
> > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource\_collection/stepable\_iterator.rb:85:in  
> > `step' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:104:in `iterate'  
> > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource\_collection/stepable\_iterator.rb:55:in  
> > `each_with_index' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection.rb:96:in `execute\_each\_resource'  
> > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:80:in  
> > `converge' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:429:in `converge'  
> > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:494:in  
> > `do_run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:199:in `block in run'  
> > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:193:in  
> > `fork' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:193:in `run'  
> > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application.rb:183:in  
> > `run_chef_client' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:302:in `block in run\_application'  
> > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:294:in  
> > `loop' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:294:in `run\_application'  
> > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application.rb:66:in  
> > `run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/bin/chef-client:26:in `\<top (required)\>'  
> > /usr/bin/chef-client:23:in `load' /usr/bin/chef-client:23:in `'
> > 
> > --  
> > [http://philcryer.com/](http://philcryer.com/)

--  
[http://philcryer.com/](http://philcryer.com/)

---

<div class="post-metadata">

**Author:** ![Andrew\_Gross](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/andrew_gross/32/320_2.png) [@Andrew\_Gross](https://discourse.chef.io/u/Andrew_Gross)\
**Post date:** [November 15, 2013, 4:21pm UTC](https://discourse.chef.io/t/unattended-chef-client-runs-fail/4683/4 "2013-11-15T16:21:10Z")

</div>

Not yet, I believe there will be some support for this sort of thing in the  
next major Chef version, but I am not 100% sure.

On Fri, Nov 15, 2013 at 11:05 AM, Phil Cryer [phil@philcryer.com](mailto:phil@philcryer.com) wrote:

> Thanks, this should really help - is there a way to make this a global  
> option?
> 
> On Fri, Nov 15, 2013 at 9:22 AM, Andrew Gross [andrew@yipit.com](mailto:andrew@yipit.com) wrote:
> 
> > Looks like it is a timeout connecting to a package repository to check to  
> > make sure it has the correct version of ntp. These are fairly common in my  
> > experience. The simple solution that catches most issues is to add retries  
> > and retry\_delay to all of your package install blocks to get around minor  
> > hiccups.
> > 
> > Old:
> > 
> > package "ntp" do  
> > action :install  
> > end
> > 
> > New:
> > 
> > package "ntp" do  
> > action :install  
> > retry\_delay 5  
> > retries 2  
> > end
> > 
> > Additionally, I would recommend setting up a failure handler than emails  
> > you(or dumps somewhere off machine) the Chef run log so you don't have to  
> > do so much digging. The error handler Opscode has is a decent start:  
> > [About Handlers](http://docs.opscode.com/essentials_handlers.html)
> > 
> > On Fri, Nov 15, 2013 at 9:52 AM, Phil Cryer [phil@philcryer.com](mailto:phil@philcryer.com) wrote:
> > 
> > > Randomly a chef run will fail - right now I have 4 nodes, all built by  
> > > chef, all working like champs, but usually overnight there will be 1 or 2  
> > > that fail a chef run, so when I check in the morning they haven't been  
> > > chef'd for 10-12 hours. It says it's trying again in 1800 seconds, but if  
> > > it's running once an hour it should have succeeded by now... when I login  
> > > and run chef-client manually, it just works without me having to restart  
> > > chef or kill any processes. Looking at the stacktrace it looks like it  
> > > stopped at yum-dump.py - could this be what's holding it up? And if so, why  
> > > doesn't it clear itself on the next run, since it did when I manually ran  
> > > it. Thanks
> > > 
> > > [...LOGS...]  
> > > [2013-11-15T08:05:32-06:00] INFO: Processing yum\_key[RPM-GPG-KEY-EPEL-6]  
> > > action add (yum::epel line 22)  
> > > [2013-11-15T08:05:32-06:00] INFO: Processing yum\_repository[epel] action  
> > > add (yum::epel line 27)  
> > > [2013-11-15T08:05:32-06:00] INFO: Processing package[ntp] action install  
> > > (ntp::default line 25)  
> > > [2013-11-15T08:20:43-06:00] ERROR:  
> > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py  
> > > exceeded timeout 900  
> > > [2013-11-15T08:20:43-06:00] INFO: Running queued delayed notifications  
> > > before re-raising exception  
> > > [2013-11-15T08:20:43-06:00] ERROR: Running exception handlers  
> > > [2013-11-15T08:20:43-06:00] ERROR: Exception handlers complete  
> > > [2013-11-15T08:20:43-06:00] FATAL: Stacktrace dumped to  
> > > /var/chef/cache/chef-stacktrace.out  
> > > [2013-11-15T08:20:43-06:00] ERROR: Chef::Exceptions::ChildConvergeError:  
> > > Chef run process exited unsuccessfully (exit code 1)  
> > > [2013-11-15T08:20:43-06:00] ERROR: Sleeping for 1800 seconds before  
> > > trying again
> > > 
> > > [...STACKTRACE...]  
> > > Generated at 2013-11-15 08:20:43 -0600  
> > > Mixlib::ShellOut::CommandTimeout: package[ntp] (ntp::default line 25)  
> > > had an error: Mixlib::ShellOut::CommandTimeout: command timed out:  
> > > ---- Begin output of /usr/bin/python  
> > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py  
> > > --options --installed-provides ----  
> > > STDOUT:  
> > > STDERR:  
> > > ---- End output of /usr/bin/python  
> > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py  
> > > --options --installed-provides ----
> > > 
> > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout/unix.rb:64:in  
> > > `run_command' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout.rb:225:in `run\_command'  
> > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/mixin/shell\_out.rb:30:in  
> > > `shell_out' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/mixin/shell_out.rb:35:in `shell\_out!'  
> > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:714:in  
> > > `refresh' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:806:in `package\_available?'  
> > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:1055:in  
> > > `load_current_resource' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider.rb:97:in `run\_action'  
> > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource.rb:625:in  
> > > `run_action' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:49:in `run\_action'  
> > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:in  
> > > `block (2 levels) in converge' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:in `each'  
> > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:in  
> > > `block in converge' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection.rb:98:in `block in execute\_each\_resource'  
> > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource\_collection/stepable\_iterator.rb:116:in  
> > > `call' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:116:in `call\_iterator\_block'  
> > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource\_collection/stepable\_iterator.rb:85:in  
> > > `step' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:104:in `iterate'  
> > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource\_collection/stepable\_iterator.rb:55:in  
> > > `each_with_index' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection.rb:96:in `execute\_each\_resource'  
> > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:80:in  
> > > `converge' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:429:in `converge'  
> > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:494:in  
> > > `do_run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:199:in `block in run'  
> > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:193:in  
> > > `fork' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:193:in `run'  
> > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application.rb:183:in  
> > > `run_chef_client' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:302:in `block in run\_application'  
> > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:294:in  
> > > `loop' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:294:in `run\_application'  
> > > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application.rb:66:in  
> > > `run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/bin/chef-client:26:in `\<top (required)\>'  
> > > /usr/bin/chef-client:23:in `load' /usr/bin/chef-client:23:in `'
> > > 
> > > --  
> > > [http://philcryer.com/](http://philcryer.com/)
> 
> --  
> [http://philcryer.com/](http://philcryer.com/)