Chef and find a json file generated from a previous recipe

Chef Infra (archive)
1

Hi,

I have two recipes.

The job of the first recipe is the run a python script and to save to disk
a json file.

The job of the second recipe is to read the the json file into a hash and
pass to a template.

Below is the second recipe

if File.exists?("#{Chef::Config[:file_cache_path]}/zookeeper_hosts")
zookeeper_hosts =
File.read("#{Chef::Config[:file_cache_path]}/zookeeper_hosts")
end

template “/etc/zookeeper/conf/zoo.cfg” do
path "/etc/zookeeper/conf/zoo.cfg"
source "cloudera.zoo.cfg.erb"
owner "root"
group "root"
mode "0644"
variables :zookeeper => zookeeper_hosts
notifies :restart, resources(:service => “zookeeper-server”)
end

On the first run of chef, all works but the zookeeper_hosts is Null. If I
rerun chef after the intial run then zookeeper_hosts has the proper hash
even though the previous run generated the hash.

How can I load the json file generated from json created from a previous
recipe?

Thanks

2

David - I can't do the details of a chef run justice in an email, but it is important to know the a chef run has multiple phases. A compilation phase and a converge phase. Ruby code that you write in your recipe is evaluated in the compilation phase and resources that make changes to system are generally executed in the converge phase. You can see http://docs.chef.io/chef_client.html for more information. The reason your host file is null on the first run is because your resource in the other recipe hasn't yet written the file to disk. One method to tackle this is to read the file content in using a lazy block. Your code would become: template "/etc/zookeeper/conf/zoo.cfg" do

path "/etc/zookeeper/conf/zoo.cfg"

source "cloudera.zoo.cfg.erb"

owner "root"

group "root"

mode "0644"

variables lazy { :zookeeper => File.read("#{Chef::Config[:file_cache_path]}/zookeeper_hosts" }

notifies :restart, resources(:service => "zookeeper-server")

end Depending on your use case, you may want to reconsider using elements like if File.exists? as checks. Should the recipe continue if a file is missing or should the chef run fail to indicate that a problem has occurred? It seems like that element being nil was a problem, perhaps the chef run should abort if that file is missing? I don't know if its an anti-pattern, but I try to avoid using the file_cache_path for elements my system needs for configuration. I assume files in that path may be wiped out at any time and even if another recipe has a job of putting a file in there during subsequent chef runs I like to see as few resources as possible change without an external change being initiated.

Brian O'Connell, STSM

IBM - Continuous Availability Services

----- Original message -----

From: David Montgomery <davidmontgomery@gmail.com>

To: chef@lists.opscode.com

Cc:

Subject: [chef] Chef and find a json file generated from a previous recipe

Date: Tue, Jun 2, 2015 6:45 AM

Hi,

I have two recipes.

The job of the first recipe is the run a python script and to save to disk a json file.

The job of the second recipe is to read the the json file into a hash and pass to a template.

Below is the second recipe

if File.exists?("#{Chef::Config[:file_cache_path]}/zookeeper_hosts")

zookeeper_hosts = File.read("#{Chef::Config[:file_cache_path]}/zookeeper_hosts")

end

template "/etc/zookeeper/conf/zoo.cfg" do

path "/etc/zookeeper/conf/zoo.cfg"

source "cloudera.zoo.cfg.erb"

owner "root"

group "root"

mode "0644"

variables :zookeeper => zookeeper_hosts

notifies :restart, resources(:service => "zookeeper-server")

end

On the first run of chef, all works but the zookeeper_hosts is Null. If I rerun chef after the intial run then zookeeper_hosts has the proper hash even though the previous run generated the hash.

How can I load the json file generated from json created from a previous recipe?

Thanks

3

Thanks,

But is there a solution? I have to use my python to generate the json
file. I leave if
File.exists?("#{Chef::Config[:file_cache_path]}/zookeeper_hosts") I get a
chef error and fails and not very pretty to have to run chef twice. Is this
the best chef can do?

Thanks

On Tue, Jun 2, 2015 at 8:35 PM, Brian O'Connell boc@us.ibm.com wrote:

David -
I can't do the details of a chef run justice in an email, but it is
important to know the a chef run has multiple phases. A compilation phase
and a converge phase. Ruby code that you write in your recipe is evaluated
in the compilation phase and resources that make changes to system are
generally executed in the converge phase. You can see
Chef Infra Client Overview for more information.

The reason your host file is null on the first run is because your
resource in the other recipe hasn't yet written the file to disk. One
method to tackle this is to read the file content in using a lazy block.
Your code would become:

template "/etc/zookeeper/conf/zoo.cfg" do
path "/etc/zookeeper/conf/zoo.cfg"
source "cloudera.zoo.cfg.erb"
owner "root"
group "root"
mode "0644"
variables lazy { :zookeeper =>
File.read("#{Chef::Config[:file_cache_path]}/zookeeper_hosts" }
notifies :restart, resources(:service => "zookeeper-server")
end

Depending on your use case, you may want to reconsider using elements like
if File.exists? as checks. Should the recipe continue if a file is missing
or should the chef run fail to indicate that a problem has occurred? It
seems like that element being nil was a problem, perhaps the chef run
should abort if that file is missing?

I don't know if its an anti-pattern, but I try to avoid using the
file_cache_path for elements my system needs for configuration. I assume
files in that path may be wiped out at any time and even if another recipe
has a job of putting a file in there during subsequent chef runs I like to
see as few resources as possible change without an external change being
initiated.

Brian O'Connell, STSM
IBM - Continuous Availability Services

----- Original message -----
From: David Montgomery davidmontgomery@gmail.com
To: chef@lists.opscode.com
Cc:
Subject: [chef] Chef and find a json file generated from a previous recipe
Date: Tue, Jun 2, 2015 6:45 AM

Hi,

I have two recipes.

The job of the first recipe is the run a python script and to save to disk
a json file.

The job of the second recipe is to read the the json file into a hash and
pass to a template.

Below is the second recipe

if File.exists?("#{Chef::Config[:file_cache_path]}/zookeeper_hosts")
zookeeper_hosts =
File.read("#{Chef::Config[:file_cache_path]}/zookeeper_hosts")
end

template "/etc/zookeeper/conf/zoo.cfg" do
path "/etc/zookeeper/conf/zoo.cfg"
source "cloudera.zoo.cfg.erb"
owner "root"
group "root"
mode "0644"
variables :zookeeper => zookeeper_hosts
notifies :restart, resources(:service => "zookeeper-server")
end

On the first run of chef, all works but the zookeeper_hosts is Null. If I
rerun chef after the intial run then zookeeper_hosts has the proper hash
even though the previous run generated the hash.

How can I load the json file generated from json created from a previous
recipe?

Thanks

4

On Jun 2, 2015, at 4:33 PM, David Montgomery davidmontgomery@gmail.com wrote:

Thanks,

But is there a solution? I have to use my python to generate the json file. I leave if File.exists?("#{Chef::Config[:file_cache_path]}/zookeeper_hosts") I get a chef error and fails and not very pretty to have to run chef twice. Is this the best chef can do?

Delay the file read until converge time:

template "/etc/zookeeper/conf/zoo.cfg" do
path "/etc/zookeeper/conf/zoo.cfg"
source "cloudera.zoo.cfg.erb"
owner "root"
group "root"
mode "0644"
variables lazy { { :zookeeper => File.read("#{Chef::Config[:file_cache_path]}/zookeeper_hosts") } }
notifies :restart, resources(:service => "zookeeper-server")
end

The lazy {} helper puts off evaluating a resource property until the first time it is used, which in this case should be after the file is written. You could also do a similar thing using the node.run_state hash to avoid the time cost or writing/reading a file. Overall this kind of coupling should be avoided though, cookbooks should be self-contained.

--Noah

5

wow..I see now..I now have a better understanding.....thanks for help

On Wed, Jun 3, 2015 at 7:36 AM, Noah Kantrowitz noah@coderanger.net wrote:

On Jun 2, 2015, at 4:33 PM, David Montgomery davidmontgomery@gmail.com
wrote:

Thanks,

But is there a solution? I have to use my python to generate the json
file. I leave if
File.exists?("#{Chef::Config[:file_cache_path]}/zookeeper_hosts") I get a
chef error and fails and not very pretty to have to run chef twice. Is this
the best chef can do?

Delay the file read until converge time:

template "/etc/zookeeper/conf/zoo.cfg" do
path "/etc/zookeeper/conf/zoo.cfg"
source "cloudera.zoo.cfg.erb"
owner "root"
group "root"
mode "0644"
variables lazy { { :zookeeper =>
File.read("#{Chef::Config[:file_cache_path]}/zookeeper_hosts") } }
notifies :restart, resources(:service => "zookeeper-server")
end

The lazy {} helper puts off evaluating a resource property until the first
time it is used, which in this case should be after the file is written.
You could also do a similar thing using the node.run_state hash to avoid
the time cost or writing/reading a file. Overall this kind of coupling
should be avoided though, cookbooks should be self-contained.

--Noah