I have a cookbook where we generate a particular execute resource in a loop. Recently, the number of items in the loop has been growing so large that the chef-client runs out of memory trying to hold all of the execute resources in memory.
I have tried adding execute_resource.run_action(:run) in the hopes that it would simply execute the resource at compile time and then not have to hold them all but to no avail.
Any thoughts on how we could prevent chef-client from running out of memory while attempting to do this??
That would mean something like millions or billions of resources unless this is on a RasPi. At that point it’s probably time for a dedicated tool that Chef just runs.
The loop is on a knife search command for all nodes using a particular recipe, and we certainly dont have millions or billions of nodes - probably something more like 700~1000.
Most likely your issue is loading all the node data from all those into memory. Have a look at search filtering to have the Chef Server pre-filter the results for you: About Search
That was exactly the issue. We only use the attribute [“ipadress”] in our code, so I was able to change
search( :node, search_string ) do |n|
in to
search(:node, search_string, :filter_result => { "ipaddress" => [ "ipaddress" ] } ) do |n|
and now our chef-client runs using that cookbook still take a super long time as it processes a 700-something item loop, but they complete smoothly with pretty much 0 memory footprint.