# 76 running chef-client processes exhausting system memory

**URL:** <https://discourse.chef.io/t/76-running-chef-client-processes-exhausting-system-memory/4577>\
**Category:** Chef Infra (archive)\
**Created:** [October 3, 2013, 2:19pm UTC](https://discourse.chef.io/t/76-running-chef-client-processes-exhausting-system-memory/4577 "2013-10-03T14:19:28Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![philcryer](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/philcryer/32/404_2.png) [@philcryer](https://discourse.chef.io/u/philcryer)\
**Post date:** [October 3, 2013, 2:19pm UTC](https://discourse.chef.io/t/76-running-chef-client-processes-exhausting-system-memory/4577/1 "2013-10-03T14:19:28Z")

</div>

# ISSUE

76 running chef-client processes exhausting system memory, causing  
other apps to crash (in this case zabbix), chef-client is set to run  
once an hour, currently showing 76 running processes.

[root@dc2mgmtsavpd01 ~]# ps -fe | grep chef-client | wc -l  
76

Also, when I have failed chef-client runs like this I always find this  
in the ps; a chef-client worker process that is old (hung?), and a  
yum-dump.py that always needs to be killed before a chef-client run  
will be successful.

# ps -fe|grep chef

root 7082 13613 23 09:12 ? 00:00:00 chef-client worker:  
ppid=13613;start=09:12:03;  
root 7221 7082 26 09:12 ? 00:00:00 /usr/bin/python  
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py  
--options --installed-provides

After killing the 76 processes, and a kill -9 on the chef-client work  
and yum-dump.py, a chef-client run will be successful, as will others  
that are run automatically after that - it's just at some point down  
the line one will get hung up and cause this cascade of processes.

# SPECS

# chef-client -v

Chef: 11.6.0

# cat /etc/issue.net

CentOS release 6.4 (Final)  
Kernel \r on an \m

Vmware instance with one core and 3G RAM

# LOGS

[...]  
[2013-10-03T09:06:37-05:00] INFO: Forking chef instance to converge...  
[2013-10-03T09:06:37-05:00] INFO: \*\*\* Chef 11.6.0 \*\*\*  
[2013-10-03T09:06:37-05:00] WARN: unable to detect ipaddress  
[2013-10-03T09:06:37-05:00] WARN: unable to detect macaddress  
[2013-10-03T09:06:37-05:00] WARN: unable to detect ip6address  
[2013-10-03T09:06:37-05:00] INFO: Run List is [recipe[base],  
recipe[pd-web], recipe[diptables]]  
[2013-10-03T09:06:37-05:00] INFO: Run List expands to [base, pd-web, diptables]  
[2013-10-03T09:06:37-05:00] INFO: Starting Chef Run for  
dc2mgmtsavpd01.mgmt.sdirect  
[2013-10-03T09:06:37-05:00] INFO: Running start handlers  
[2013-10-03T09:06:37-05:00] INFO: Start handlers complete.  
[2013-10-03T09:06:38-05:00] INFO: Loading cookbooks [apache2, apt,  
aws, base, build-essential, chef-client, chef-varnish, chef\_handler,  
couchbase, cron, database, diptables, dmg, drush, firewall, git, line,  
mysql, nfs, ntp, openssl, php, postfix, postgresql, rsyslog, runit,  
savviscom-pd-web, splunk\_handler, timezone-ii, ufw, windows, xfs, xml,  
yum, zabbix]  
[2013-10-03T09:06:39-05:00] WARN: Cloning resource attributes for  
template[/etc/php.ini] from prior resource (CHEF-3694)  
[2013-10-03T09:06:39-05:00] WARN: Previous template[/etc/php.ini]:  
/var/chef/cache/cookbooks/pd-web/recipes/default.rb:4:in `from_file' [2013-10-03T09:06:39-05:00] WARN: Current template[/etc/php.ini]: /var/chef/cache/cookbooks/php/recipes/package.rb:27:in `from\_file'  
[2013-10-03T09:06:39-05:00] WARN: Cloning resource attributes for  
cookbook\_file[/etc/httpd/conf/httpd.conf] from prior resource  
(CHEF-3694)  
[2013-10-03T09:06:39-05:00] WARN: Previous  
cookbook\_file[/etc/httpd/conf/httpd.conf]:  
/var/chef/cache/cookbooks/savviscom-pd-web/recipes/default.rb:55:in  
`from_file' [2013-10-03T09:06:39-05:00] WARN: Current cookbook_file[/etc/httpd/conf/httpd.conf]: /var/chef/cache/cookbooks/apache2/recipes/default.rb:221:in `from\_file'  
[2013-10-03T09:06:39-05:00] WARN: Cloning resource attributes for  
service[apache2] from prior resource (CHEF-3694)  
[2013-10-03T09:06:39-05:00] WARN: Previous service[apache2]:  
/var/chef/cache/cookbooks/apache2/recipes/default.rb:24:in `from_file' [2013-10-03T09:06:39-05:00] WARN: Current service[apache2]: /var/chef/cache/cookbooks/apache2/recipes/default.rb:228:in `from\_file'  
[2013-10-03T09:06:39-05:00] WARN: Cloning resource attributes for  
service[moxi-server] from prior resource (CHEF-3694)  
[2013-10-03T09:06:39-05:00] WARN: Previous service[moxi-server]:  
/var/chef/cache/cookbooks/couchbase/recipes/moxi.rb:42:in `from_file' [2013-10-03T09:06:39-05:00] WARN: Current service[moxi-server]: /var/chef/cache/cookbooks/couchbase/recipes/moxi.rb:56:in `from\_file'  
[2013-10-03T09:06:39-05:00] WARN: Cloning resource attributes for  
template[/etc/varnish/default.vcl] from prior resource (CHEF-3694)  
[2013-10-03T09:06:39-05:00] WARN: Previous  
template[/etc/varnish/default.vcl]:  
/var/chef/cache/cookbooks/chef-varnish/recipes/default.rb:56:in  
`from_file' [2013-10-03T09:06:39-05:00] WARN: Current template[/etc/varnish/default.vcl]: /var/chef/cache/cookbooks/savviscom-pd-web/recipes/default.rb:113:in `from\_file'  
[2013-10-03T09:06:39-05:00] WARN: Cloning resource attributes for  
service[varnishlog] from prior resource (CHEF-3694)  
[2013-10-03T09:06:39-05:00] WARN: Previous service[varnishlog]:  
/var/chef/cache/cookbooks/chef-varnish/recipes/default.rb:81:in  
`from_file' [2013-10-03T09:06:39-05:00] WARN: Current service[varnishlog]: /var/chef/cache/cookbooks/savviscom-pd-web/recipes/default.rb:120:in `from\_file'  
[2013-10-03T09:06:39-05:00] WARN: Don't know how to set up automatic  
iptables on your distribution, sorry. Please submit a bug ticket at  
[https://github.com/wk8/cookbook-iptables/issues](https://github.com/wk8/cookbook-iptables/issues)  
[2013-10-03T09:06:39-05:00] INFO: Processing template[/root/.profile]  
action create (base::default line 1)  
[2013-10-03T09:06:39-05:00] INFO: Processing  
template[/root/.bash\_profile] action create (base::default line 8)  
[2013-10-03T09:06:39-05:00] INFO: Processing template[/root/.curlrc]  
action create (base::default line 15)  
[2013-10-03T09:06:39-05:00] INFO: Processing  
yum\_key[RPM-GPG-KEY-EPEL-6] action add (yum::epel line 22)  
[2013-10-03T09:06:39-05:00] INFO: Processing yum\_repository[epel]  
action add (yum::epel line 27)  
[2013-10-03T09:06:39-05:00] INFO: Processing package[tzdata] action  
install (timezone-ii::default line 16)  
[2013-10-03T09:06:39-05:00] INFO: Running queued delayed notifications  
before re-raising exception  
[2013-10-03T09:06:39-05:00] ERROR: Running exception handlers  
[2013-10-03T09:06:39-05:00] ERROR: Exception handlers complete  
[2013-10-03T09:06:39-05:00] FATAL: Stacktrace dumped to  
/var/chef/cache/chef-stacktrace.out  
[2013-10-03T09:06:39-05:00] ERROR:  
Chef::Exceptions::ChildConvergeError: Chef run process exited  
unsuccessfully (exit code 1)  
[2013-10-03T09:06:39-05:00] ERROR: Sleeping for 3600 seconds before trying again

# STACKTRACE

# cat /var/chef/cache/chef-stacktrace.out

Generated at 2013-10-03 09:06:39 -0500  
Errno::ENOMEM: package[tzdata] (timezone-ii::default line 16) had an  
error: Errno::ENOMEM: Cannot allocate memory - fork(2)  
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout/unix.rb:256:in  
`fork' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout/unix.rb:256:in `fork\_subprocess'  
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout/unix.rb:40:in  
`run_command' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout.rb:225:in `run\_command'  
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/mixin/shell\_out.rb:30:in  
`shell_out' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/mixin/shell_out.rb:35:in `shell\_out!'  
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:714:in  
`refresh' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:806:in `package\_available?'  
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:1055:in  
`load_current_resource' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider.rb:97:in `run\_action'  
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource.rb:625:in  
`run_action' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:49:in `run\_action'  
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:in  
`block (2 levels) in converge' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:in `each'  
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:in  
`block in converge' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection.rb:98:in `block in execute\_each\_resource'  
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource\_collection/stepable\_iterator.rb:116:in  
`call' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:116:in `call\_iterator\_block'  
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource\_collection/stepable\_iterator.rb:85:in  
`step' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:104:in `iterate'  
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource\_collection/stepable\_iterator.rb:55:in  
`each_with_index' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection.rb:96:in `execute\_each\_resource'  
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:80:in  
`converge' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:429:in `converge'  
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:494:in  
`do_run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:199:in `block in run'  
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:193:in  
`fork' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:193:in `run'  
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application.rb:183:in  
`run_chef_client' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:302:in `block in run\_application'  
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:294:in  
`loop' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:294:in `run\_application'  
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application.rb:66:in  
`run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/bin/chef-client:26:in `\<top (required)\>'  
/usr/bin/chef-client:23:in `load' /usr/bin/chef-client:23:in `'

## [https://gist.github.com/philcryer/6810475](https://gist.github.com/philcryer/6810475)

[http://philcryer.com](http://philcryer.com)

---

<div class="post-metadata">

**Author:** ![philcryer](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/philcryer/32/404_2.png) [@philcryer](https://discourse.chef.io/u/philcryer)\
**Post date:** [October 3, 2013, 2:30pm UTC](https://discourse.chef.io/t/76-running-chef-client-processes-exhausting-system-memory/4577/2 "2013-10-03T14:30:09Z")

</div>

Additionally, I'm currently running 6 nodes, and the 4 that this  
always happens to are the web servers (which have 15+ recipes/packages  
assigned to them), the 2 db servers (which only have 2 recipes  
assigned) never have this issue, which makes me think something is  
happening with yum to gum things up.

On Thu, Oct 3, 2013 at 9:19 AM, Phil Cryer [phil@philcryer.com](mailto:phil@philcryer.com) wrote:

> # ISSUE
> 
> 76 running chef-client processes exhausting system memory, causing  
> other apps to crash (in this case zabbix), chef-client is set to run  
> once an hour, currently showing 76 running processes.
> 
> [root@dc2mgmtsavpd01 ~]# ps -fe | grep chef-client | wc -l  
> 76
> 
> Also, when I have failed chef-client runs like this I always find this  
> in the ps; a chef-client worker process that is old (hung?), and a  
> yum-dump.py that always needs to be killed before a chef-client run  
> will be successful.
> 
> # ps -fe|grep chef
> 
> root 7082 13613 23 09:12 ? 00:00:00 chef-client worker:  
> ppid=13613;start=09:12:03;  
> root 7221 7082 26 09:12 ? 00:00:00 /usr/bin/python  
> /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum-dump.py  
> --options --installed-provides
> 
> After killing the 76 processes, and a kill -9 on the chef-client work  
> and yum-dump.py, a chef-client run will be successful, as will others  
> that are run automatically after that - it's just at some point down  
> the line one will get hung up and cause this cascade of processes.
> 
> # SPECS
> 
> # chef-client -v
> 
> Chef: 11.6.0
> 
> # cat /etc/issue.net
> 
> CentOS release 6.4 (Final)  
> Kernel \r on an \m
> 
> Vmware instance with one core and 3G RAM
> 
> # LOGS
> 
> [...]  
> [2013-10-03T09:06:37-05:00] INFO: Forking chef instance to converge...  
> [2013-10-03T09:06:37-05:00] INFO: \*\*\* Chef 11.6.0 \*\*\*  
> [2013-10-03T09:06:37-05:00] WARN: unable to detect ipaddress  
> [2013-10-03T09:06:37-05:00] WARN: unable to detect macaddress  
> [2013-10-03T09:06:37-05:00] WARN: unable to detect ip6address  
> [2013-10-03T09:06:37-05:00] INFO: Run List is [recipe[base],  
> recipe[pd-web], recipe[diptables]]  
> [2013-10-03T09:06:37-05:00] INFO: Run List expands to [base, pd-web, diptables]  
> [2013-10-03T09:06:37-05:00] INFO: Starting Chef Run for  
> dc2mgmtsavpd01.mgmt.sdirect  
> [2013-10-03T09:06:37-05:00] INFO: Running start handlers  
> [2013-10-03T09:06:37-05:00] INFO: Start handlers complete.  
> [2013-10-03T09:06:38-05:00] INFO: Loading cookbooks [apache2, apt,  
> aws, base, build-essential, chef-client, chef-varnish, chef\_handler,  
> couchbase, cron, database, diptables, dmg, drush, firewall, git, line,  
> mysql, nfs, ntp, openssl, php, postfix, postgresql, rsyslog, runit,  
> savviscom-pd-web, splunk\_handler, timezone-ii, ufw, windows, xfs, xml,  
> yum, zabbix]  
> [2013-10-03T09:06:39-05:00] WARN: Cloning resource attributes for  
> template[/etc/php.ini] from prior resource (CHEF-3694)  
> [2013-10-03T09:06:39-05:00] WARN: Previous template[/etc/php.ini]:  
> /var/chef/cache/cookbooks/pd-web/recipes/default.rb:4:in `from_file' [2013-10-03T09:06:39-05:00] WARN: Current template[/etc/php.ini]: /var/chef/cache/cookbooks/php/recipes/package.rb:27:in `from\_file'  
> [2013-10-03T09:06:39-05:00] WARN: Cloning resource attributes for  
> cookbook\_file[/etc/httpd/conf/httpd.conf] from prior resource  
> (CHEF-3694)  
> [2013-10-03T09:06:39-05:00] WARN: Previous  
> cookbook\_file[/etc/httpd/conf/httpd.conf]:  
> /var/chef/cache/cookbooks/savviscom-pd-web/recipes/default.rb:55:in  
> `from_file' [2013-10-03T09:06:39-05:00] WARN: Current cookbook_file[/etc/httpd/conf/httpd.conf]: /var/chef/cache/cookbooks/apache2/recipes/default.rb:221:in `from\_file'  
> [2013-10-03T09:06:39-05:00] WARN: Cloning resource attributes for  
> service[apache2] from prior resource (CHEF-3694)  
> [2013-10-03T09:06:39-05:00] WARN: Previous service[apache2]:  
> /var/chef/cache/cookbooks/apache2/recipes/default.rb:24:in `from_file' [2013-10-03T09:06:39-05:00] WARN: Current service[apache2]: /var/chef/cache/cookbooks/apache2/recipes/default.rb:228:in `from\_file'  
> [2013-10-03T09:06:39-05:00] WARN: Cloning resource attributes for  
> service[moxi-server] from prior resource (CHEF-3694)  
> [2013-10-03T09:06:39-05:00] WARN: Previous service[moxi-server]:  
> /var/chef/cache/cookbooks/couchbase/recipes/moxi.rb:42:in `from_file' [2013-10-03T09:06:39-05:00] WARN: Current service[moxi-server]: /var/chef/cache/cookbooks/couchbase/recipes/moxi.rb:56:in `from\_file'  
> [2013-10-03T09:06:39-05:00] WARN: Cloning resource attributes for  
> template[/etc/varnish/default.vcl] from prior resource (CHEF-3694)  
> [2013-10-03T09:06:39-05:00] WARN: Previous  
> template[/etc/varnish/default.vcl]:  
> /var/chef/cache/cookbooks/chef-varnish/recipes/default.rb:56:in  
> `from_file' [2013-10-03T09:06:39-05:00] WARN: Current template[/etc/varnish/default.vcl]: /var/chef/cache/cookbooks/savviscom-pd-web/recipes/default.rb:113:in `from\_file'  
> [2013-10-03T09:06:39-05:00] WARN: Cloning resource attributes for  
> service[varnishlog] from prior resource (CHEF-3694)  
> [2013-10-03T09:06:39-05:00] WARN: Previous service[varnishlog]:  
> /var/chef/cache/cookbooks/chef-varnish/recipes/default.rb:81:in  
> `from_file' [2013-10-03T09:06:39-05:00] WARN: Current service[varnishlog]: /var/chef/cache/cookbooks/savviscom-pd-web/recipes/default.rb:120:in `from\_file'  
> [2013-10-03T09:06:39-05:00] WARN: Don't know how to set up automatic  
> iptables on your distribution, sorry. Please submit a bug ticket at  
> [Issues · wk8/cookbook-diptables · GitHub](https://github.com/wk8/cookbook-iptables/issues)  
> [2013-10-03T09:06:39-05:00] INFO: Processing template[/root/.profile]  
> action create (base::default line 1)  
> [2013-10-03T09:06:39-05:00] INFO: Processing  
> template[/root/.bash\_profile] action create (base::default line 8)  
> [2013-10-03T09:06:39-05:00] INFO: Processing template[/root/.curlrc]  
> action create (base::default line 15)  
> [2013-10-03T09:06:39-05:00] INFO: Processing  
> yum\_key[RPM-GPG-KEY-EPEL-6] action add (yum::epel line 22)  
> [2013-10-03T09:06:39-05:00] INFO: Processing yum\_repository[epel]  
> action add (yum::epel line 27)  
> [2013-10-03T09:06:39-05:00] INFO: Processing package[tzdata] action  
> install (timezone-ii::default line 16)  
> [2013-10-03T09:06:39-05:00] INFO: Running queued delayed notifications  
> before re-raising exception  
> [2013-10-03T09:06:39-05:00] ERROR: Running exception handlers  
> [2013-10-03T09:06:39-05:00] ERROR: Exception handlers complete  
> [2013-10-03T09:06:39-05:00] FATAL: Stacktrace dumped to  
> /var/chef/cache/chef-stacktrace.out  
> [2013-10-03T09:06:39-05:00] ERROR:  
> Chef::Exceptions::ChildConvergeError: Chef run process exited  
> unsuccessfully (exit code 1)  
> [2013-10-03T09:06:39-05:00] ERROR: Sleeping for 3600 seconds before trying again
> 
> # STACKTRACE
> 
> # cat /var/chef/cache/chef-stacktrace.out
> 
> Generated at 2013-10-03 09:06:39 -0500  
> Errno::ENOMEM: package[tzdata] (timezone-ii::default line 16) had an  
> error: Errno::ENOMEM: Cannot allocate memory - fork(2)  
> /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout/unix.rb:256:in  
> `fork' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout/unix.rb:256:in `fork\_subprocess'  
> /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout/unix.rb:40:in  
> `run_command' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/mixlib-shellout-1.2.0/lib/mixlib/shellout.rb:225:in `run\_command'  
> /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/mixin/shell\_out.rb:30:in  
> `shell_out' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/mixin/shell_out.rb:35:in `shell\_out!'  
> /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:714:in  
> `refresh' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:806:in `package\_available?'  
> /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider/package/yum.rb:1055:in  
> `load_current_resource' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/provider.rb:97:in `run\_action'  
> /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource.rb:625:in  
> `run_action' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:49:in `run\_action'  
> /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:in  
> `block (2 levels) in converge' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:in `each'  
> /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:81:in  
> `block in converge' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection.rb:98:in `block in execute\_each\_resource'  
> /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource\_collection/stepable\_iterator.rb:116:in  
> `call' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:116:in `call\_iterator\_block'  
> /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource\_collection/stepable\_iterator.rb:85:in  
> `step' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection/stepable_iterator.rb:104:in `iterate'  
> /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource\_collection/stepable\_iterator.rb:55:in  
> `each_with_index' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/resource_collection.rb:96:in `execute\_each\_resource'  
> /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/runner.rb:80:in  
> `converge' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:429:in `converge'  
> /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:494:in  
> `do_run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:199:in `block in run'  
> /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:193:in  
> `fork' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:193:in `run'  
> /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application.rb:183:in  
> `run_chef_client' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:302:in `block in run\_application'  
> /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:294:in  
> `loop' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:294:in `run\_application'  
> /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application.rb:66:in  
> `run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/bin/chef-client:26:in `\<top (required)\>'  
> /usr/bin/chef-client:23:in `load' /usr/bin/chef-client:23:in `'
> 
> ## [20131003 - 76 running chef-client processes exhausting system memory · GitHub](https://gist.github.com/philcryer/6810475)
> 
> [http://philcryer.com](http://philcryer.com)

--  
[http://philcryer.com](http://philcryer.com)

---

<div class="post-metadata">

**Author:** ![kallistec](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/kallistec/32/23_2.png) [@kallistec](https://discourse.chef.io/u/kallistec)\
**Post date:** [October 3, 2013, 3:41pm UTC](https://discourse.chef.io/t/76-running-chef-client-processes-exhausting-system-memory/4577/3 "2013-10-03T15:41:43Z")

</div>

On Thursday, October 3, 2013 at 7:30 AM, Phil Cryer wrote:

> Additionally, I'm currently running 6 nodes, and the 4 that this  
> always happens to are the web servers (which have 15+ recipes/packages  
> assigned to them), the 2 db servers (which only have 2 recipes  
> assigned) never have this issue, which makes me think something is  
> happening with yum to gum things up.
> 
> On Thu, Oct 3, 2013 at 9:19 AM, Phil Cryer \<[phil@philcryer.com](mailto:phil@philcryer.com) ([mailto:phil@philcryer.com](mailto:phil@philcryer.com))\> wrote:
> 
> > # ISSUE
> > 
> > 76 running chef-client processes exhausting system memory, causing  
> > other apps to crash (in this case zabbix), chef-client is set to run  
> > once an hour, currently showing 76 running processes.

Are you running chef out of cron?

--  
Daniel DeLeo

---

<div class="post-metadata">

**Author:** ![philcryer](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/philcryer/32/404_2.png) [@philcryer](https://discourse.chef.io/u/philcryer)\
**Post date:** [October 3, 2013, 4:56pm UTC](https://discourse.chef.io/t/76-running-chef-client-processes-exhausting-system-memory/4577/4 "2013-10-03T16:56:52Z")

</div>

On Thu, Oct 3, 2013 at 10:41 AM, Daniel DeLeo [dan@kallistec.com](mailto:dan@kallistec.com) wrote:

> On Thursday, October 3, 2013 at 7:30 AM, Phil Cryer wrote:
> 
> Additionally, I'm currently running 6 nodes, and the 4 that this  
> always happens to are the web servers (which have 15+ recipes/packages  
> assigned to them), the 2 db servers (which only have 2 recipes  
> assigned) never have this issue, which makes me think something is  
> happening with yum to gum things up.
> 
> On Thu, Oct 3, 2013 at 9:19 AM, Phil Cryer [phil@philcryer.com](mailto:phil@philcryer.com) wrote:
> 
> # ISSUE
> 
> 76 running chef-client processes exhausting system memory, causing  
> other apps to crash (in this case zabbix), chef-client is set to run  
> once an hour, currently showing 76 running processes.
> 
> Are you running chef out of cron?

No, it's installed as a service, so it's 'running' all the time, here  
you can see it's set to only run once an hour.

root 1663 1 0 00:15 ? 00:00:00  
/opt/chef/embedded/bin/ruby /usr/bin/chef-client -d -c  
/etc/chef/client.rb -L /var/log/chef/client.log -P  
/var/run/chef/client.pid -i 3600 -s 300

Also, while scrolling the logs I did see this error, that I hadn't  
caught before - seem to be related to the yum-dump.py script, looks  
like I have something wrong with my yum config - "Problem parsing line  
'Freeing read locks for locker 0xca9:" will look for that error online

- package[tzdata] action install[2013-10-03T10:00:39-05:00] WARN:  
Problem parsing line 'Freeing read locks for locker 0xca7:  
32758/140342062917376' from yum-dump.py! Please check your yum  
configuration.  
[2013-10-03T10:00:39-05:00] WARN: Problem parsing line 'Freeing read  
locks for locker 0xca9: 32758/140342062917376' from yum-dump.py!  
Please check your yum configuration.  
(up to date)

FWIW, I use Debian almost exclusively (sometimes Ubuntu) so this  
Centos stuff is new to me 🙂

---

<div class="post-metadata">

**Author:** ![kallistec](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/kallistec/32/23_2.png) [@kallistec](https://discourse.chef.io/u/kallistec)\
**Post date:** [October 3, 2013, 5:34pm UTC](https://discourse.chef.io/t/76-running-chef-client-processes-exhausting-system-memory/4577/5 "2013-10-03T17:34:47Z")

</div>

On Thursday, October 3, 2013 at 9:56 AM, Phil Cryer wrote:

> On Thu, Oct 3, 2013 at 10:41 AM, Daniel DeLeo \<[dan@kallistec.com](mailto:dan@kallistec.com) ([mailto:dan@kallistec.com](mailto:dan@kallistec.com))\> wrote:
> 
> > On Thursday, October 3, 2013 at 7:30 AM, Phil Cryer wrote:
> > 
> > Additionally, I'm currently running 6 nodes, and the 4 that this  
> > always happens to are the web servers (which have 15+ recipes/packages  
> > assigned to them), the 2 db servers (which only have 2 recipes  
> > assigned) never have this issue, which makes me think something is  
> > happening with yum to gum things up.
> > 
> > On Thu, Oct 3, 2013 at 9:19 AM, Phil Cryer \<[phil@philcryer.com](mailto:phil@philcryer.com) ([mailto:phil@philcryer.com](mailto:phil@philcryer.com))\> wrote:
> > 
> > # ISSUE
> > 
> > 76 running chef-client processes exhausting system memory, causing  
> > other apps to crash (in this case zabbix), chef-client is set to run  
> > once an hour, currently showing 76 running processes.
> > 
> > Are you running chef out of cron?
> 
> No, it's installed as a service, so it's 'running' all the time, here  
> you can see it's set to only run once an hour.
> 
> root 1663 1 0 00:15 ? 00:00:00  
> /opt/chef/embedded/bin/ruby /usr/bin/chef-client -d -c  
> /etc/chef/client.rb -L /var/log/chef/client.log -P  
> /var/run/chef/client.pid -i 3600 -s 300
> 
> Also, while scrolling the logs I did see this error, that I hadn't  
> caught before - seem to be related to the yum-dump.py script, looks  
> like I have something wrong with my yum config - "Problem parsing line  
> 'Freeing read locks for locker 0xca9:" will look for that error online
> 
> - package[tzdata] action install[2013-10-03T10:00:39-05:00] WARN:  
> Problem parsing line 'Freeing read locks for locker 0xca7:  
> 32758/140342062917376' from yum-dump.py! Please check your yum  
> configuration.  
> [2013-10-03T10:00:39-05:00] WARN: Problem parsing line 'Freeing read  
> locks for locker 0xca9: 32758/140342062917376' from yum-dump.py!  
> Please check your yum configuration.  
> (up to date)
> 
> FWIW, I use Debian almost exclusively (sometimes Ubuntu) so this  
> Centos stuff is new to me 🙂

Seems like you have 2 problems here. One looks a lot like [https://tickets.opscode.com/browse/CHEF-4556](https://tickets.opscode.com/browse/CHEF-4556)

This issue applies when:

- You're running chef daemonized
- You're using the chef-client cookbook to manage the daemon
- You're using using a System V R4 type init system (i.e., plain-ol-init scripts and not daemontools/runit/upstart/systemd/etc.)
- You get into a state where the pid file used by chef's daemonization code is corrupt or stale.

You can generally rectify this by killing all chef processes, removing the pid file, and then starting chef via the init script. We narrowed the range of conditions in which this problem can occur in Chef 11.6, and are going to rewrite that portion of the daemonization code to fix it entirely in 11.8

The yum-dump.py bug is over my head, not being too familiar with yum/CentOS myself. Maybe take a look on the bug tracker to see if there's anything relevant, and if not, create a new issue?

--  
Daniel DeLeo

---

<div class="post-metadata">

**Author:** ![Lamont\_Granquist\_OLD](https://avatars.discourse-cdn.com/v4/letter/l/7feea3/32.png) [@Lamont\_Granquist\_OLD](https://discourse.chef.io/u/Lamont_Granquist_OLD)\
**Post date:** [October 3, 2013, 7:58pm UTC](https://discourse.chef.io/t/76-running-chef-client-processes-exhausting-system-memory/4577/6 "2013-10-03T19:58:00Z")

</div>

> Also, while scrolling the logs I did see this error, that I hadn't  
> caught before - seem to be related to the yum-dump.py script, looks  
> like I have something wrong with my yum config - "Problem parsing line  
> 'Freeing read locks for locker 0xca9:" will look for that error online  
> if you haven't found it already:

[https://bugzilla.redhat.com/show\_bug.cgi?id=918184](https://bugzilla.redhat.com/show_bug.cgi?id=918184)

as an ex-redhat/centos sysadmin for probably 15-ish years, i'd like to  
welcome you to the wonderful world of rpm database lock issues =)

you'll probably want to bookmark this page as well:

[http://www.oldrpm.org/hintskinks/repairdb/](http://www.oldrpm.org/hintskinks/repairdb/)