Yum::epel problems in wrapped cookbook


#1

Hello all.

I am using Vagrant with chef and berkshelf (A killer combo!) and have
run into some problems with the yum cookbook.

I have a “base” cookbook which installs a whole range of basic linux
utilities and software. It depends on the latest yum cookbook and
includes yum::yum and yum::epel. This works without trouble.

After that I have other cookbooks which depends on the “base” cookbook
and then add more specific software depending on what the server is
going to be doing, I believe this is commonly referred to as a
"wrapper" cookbook. The first line of the recipes/default.rb file in
this wrapper cookbook is: include_recipe “base::default”.

Now when I run the base cookbook on its own the provisioning works
without trouble.

However when I run the wrapper cookbook the epel installation appears to
fail with the following: https://gist.github.com/rurounijones/5771208

It appears that it is not finding the epel key etc etc. but I cannot
understand why.

I don’t suppose anyone could enlighten me and explain what would be the
right way to fix this?

Cheers

Jeff


#2

Hi Jeff,

On 6/12/13 9:22 PM, Jeffrey Jones wrote:

However when I run the wrapper cookbook the epel installation appears to
fail with the following: https://gist.github.com/rurounijones/5771208

Can you post the output with an increased log level? I believe that you
can do this in your vagrant file by setting chef.log_level = :debug.
Alternatively, you can ssh into the VM and run

chef-solo -ldebug -c /tmp/vagrant-chef-1/solo.rb -j
/tmp/vagrant-chef-1/dna.json

Cheers,

Steven


Steven Danna
Systems Engineer, Opscode, Inc
GPG Key: http://stevendanna.github.com/downloads/code/public.key


#3

Hello Steven,

I have added the extra debug output to the gist:

Thanks

Jeff

On 13/06/13 15:50, Steven Danna wrote:

Hi Jeff,

On 6/12/13 9:22 PM, Jeffrey Jones wrote:

However when I run the wrapper cookbook the epel installation appears to
fail with the following: https://gist.github.com/rurounijones/5771208
Can you post the output with an increased log level? I believe that you
can do this in your vagrant file by setting chef.log_level = :debug.
Alternatively, you can ssh into the VM and run

chef-solo -ldebug -c /tmp/vagrant-chef-1/solo.rb -j
/tmp/vagrant-chef-1/dna.json

Cheers,

Steven


#4

Hey Jeffrey: I don’t see your recipe code anywhere but judging from the
gist you have something like this:

connector::default has include_recipe "torquebox-web-server"
torquebox-web-server::default has include_recipe “yum::epel”

Is “yum” anywhere in your metadata for any of those cookbooks?

Even I’m not entirely sure whether include_recipe also loads attributes
from the dependent cookbook…

  • Julian

On Thu, Jun 13, 2013 at 8:26 PM, Jeffrey Jones jjones@toppan-f.co.jpwrote:

Hello Steven,

I have added the extra debug output to the gist: https://gist.github.com/*
*rurounijones/5771208 https://gist.github.com/rurounijones/5771208

Thanks

Jeff

On 13/06/13 15:50, Steven Danna wrote:

Hi Jeff,

On 6/12/13 9:22 PM, Jeffrey Jones wrote:

However when I run the wrapper cookbook the epel installation appears to

fail with the following: https://gist.github.com/**rurounijones/5771208https://gist.github.com/rurounijones/5771208

Can you post the output with an increased log level? I believe that you
can do this in your vagrant file by setting chef.log_level = :debug.
Alternatively, you can ssh into the VM and run

chef-solo -ldebug -c /tmp/vagrant-chef-1/solo.rb -j
/tmp/vagrant-chef-1/dna.json

Cheers,

Steven


[ Julian C. Dunn jdunn@aquezada.com * Sorry, I’m ]
[ WWW: http://www.aquezada.com/staff/julian * only Web 1.0 ]
[ gopher://sdf.org/1/users/keymaker/ * compliant! ]
[ PGP: 91B3 7A9D 683C 7C16 715F 442C 6065 D533 FDC2 05B9 ]


#5

On 6/14/13 7:57 AM, Julian C. Dunn wrote:

Hey Jeffrey: I don’t see your recipe code anywhere but judging from the
gist you have something like this:

connector::default has include_recipe "torquebox-web-server"
torquebox-web-server::default has include_recipe “yum::epel”

Is “yum” anywhere in your metadata for any of those cookbooks?

Even I’m not entirely sure whether include_recipe also loads attributes
from the dependent cookbook…

The attribute files of all cookbooks in the run list and their
dependencies get loaded during the converge phase. What is interesting
here is that a breaking change for Chef 11 was that chef-solo won’t be
able to include_recipe unless you’ve also specified “depends” in your
metadata. Thus, I believe this should be working correctly.

If you use

include_attribute "yum::epel"

in your default attribute file for the connector cookbook, does that
resolve the issue?

Cheers,

Steven

  • Julian

On Thu, Jun 13, 2013 at 8:26 PM, Jeffrey Jones <jjones@toppan-f.co.jp
mailto:jjones@toppan-f.co.jp> wrote:

Hello Steven,

I have added the extra debug output to the gist:
https://gist.github.com/__rurounijones/5771208
<https://gist.github.com/rurounijones/5771208>

Thanks

Jeff


On 13/06/13 15:50, Steven Danna wrote:

    Hi Jeff,

    On 6/12/13 9:22 PM, Jeffrey Jones wrote:

        However when I run the wrapper cookbook the epel
        installation appears to
        fail with the following:
        https://gist.github.com/__rurounijones/5771208
        <https://gist.github.com/rurounijones/5771208>

    Can you post the output with an increased log level?  I believe
    that you
    can do this in your vagrant file by setting chef.log_level = :debug.
    Alternatively, you can ssh into the VM and run

       chef-solo -ldebug -c /tmp/vagrant-chef-1/solo.rb -j
    /tmp/vagrant-chef-1/dna.json

    Cheers,

    Steven


[ Julian C. Dunn <jdunn@aquezada.com mailto:jdunn@aquezada.com>


Steven Danna
Systems Engineer, Opscode, Inc
GPG Key: http://stevendanna.github.com/downloads/code/public.key


#6

Hello Steven

On 15/06/13 06:22, Steven Danna wrote:

On 6/14/13 7:57 AM, Julian C. Dunn wrote:

Hey Jeffrey: I don’t see your recipe code anywhere but judging from the
gist you have something like this:

connector::default has include_recipe "torquebox-web-server"
torquebox-web-server::default has include_recipe “yum::epel”

Is “yum” anywhere in your metadata for any of those cookbooks?

Even I’m not entirely sure whether include_recipe also loads attributes
from the dependent cookbook…
The attribute files of all cookbooks in the run list and their
dependencies get loaded during the converge phase. What is interesting
here is that a breaking change for Chef 11 was that chef-solo won’t be
able to include_recipe unless you’ve also specified “depends” in your
metadata. Thus, I believe this should be working correctly.

If you use

 include_attribute "yum::epel"

in your default attribute file for the connector cookbook, does that
resolve the issue?

Cheers,

Steven

I have added the debug output after adding the “include_attribute” section.
It looks like the old error has been fixed but a new error has appeared.

“[2013-06-17T02:12:46+00:00] FATAL: NameError: Cannot find a resource
for yum_key on centos version 6.4”

I am guessing that the attributes/epel.rb file is being executed by the
selection logic inside the file is failing for some reason?

https://gist.github.com/rurounijones/5771208 (“Add attribute” section
for full debug output)

Cheers

Jeff


#7

On Sunday, June 16, 2013 at 7:25 PM, Jeffrey Jones wrote:

Hello Steven

On 15/06/13 06:22, Steven Danna wrote:

On 6/14/13 7:57 AM, Julian C. Dunn wrote:

Hey Jeffrey: I don’t see your recipe code anywhere but judging from the
gist you have something like this:

connector::default has include_recipe "torquebox-web-server"
torquebox-web-server::default has include_recipe “yum::epel”

Is “yum” anywhere in your metadata for any of those cookbooks?

Even I’m not entirely sure whether include_recipe also loads attributes
from the dependent cookbook…

I have added the debug output after adding the “include_attribute” section.
It looks like the old error has been fixed but a new error has appeared.

“[2013-06-17T02:12:46+00:00] FATAL: NameError: Cannot find a resource
for yum_key on centos version 6.4”

I am guessing that the attributes/epel.rb file is being executed by the
selection logic inside the file is failing for some reason?

https://gist.github.com/rurounijones/5771208 (“Add attribute” section
for full debug output)

Cheers

Jeff
As Julian said, the key here is that you need to specify the dependency between your cookbook and the yum cookbook in the cookbook’s metadata:

http://docs.opscode.com/essentials_cookbook_metadata.html

Chef 11 uses the metadata to figure out a sane order for cookbook loading. If a cookbook doesn’t appear in either the run list or dependencies of a cookbook in the run_list, it will not get loaded correctly. As you’ve seen, you can work around this a little bit with include_recipe and include_attribute, but you won’t load libraries, resource definitions, or LWRPs. Adding the dependency in the metadata is the only way to make those work.


Daniel DeLeo


#8

On 18/06/13 01:24, Daniel DeLeo wrote:

On Sunday, June 16, 2013 at 7:25 PM, Jeffrey Jones wrote:

Hello Steven

On 15/06/13 06:22, Steven Danna wrote:

On 6/14/13 7:57 AM, Julian C. Dunn wrote:

Hey Jeffrey: I don’t see your recipe code anywhere but judging from the
gist you have something like this:

connector::default has include_recipe "torquebox-web-server"
torquebox-web-server::default has include_recipe “yum::epel”

Is “yum” anywhere in your metadata for any of those cookbooks?

Even I’m not entirely sure whether include_recipe also loads attributes
from the dependent cookbook…
I have added the debug output after adding the "include_attribute"
section.
It looks like the old error has been fixed but a new error has appeared.

“[2013-06-17T02:12:46+00:00] FATAL: NameError: Cannot find a resource
for yum_key on centos version 6.4”

I am guessing that the attributes/epel.rb file is being executed by the
selection logic inside the file is failing for some reason?

https://gist.github.com/rurounijones/5771208 (“Add attribute” section
for full debug output)

Cheers

Jeff
As Julian said, the key here is that you need to specify the
dependency between your cookbook and the yum cookbook in the
cookbook’s metadata:

http://docs.opscode.com/essentials_cookbook_metadata.html

Chef 11 uses the metadata to figure out a sane order for cookbook
loading. If a cookbook doesn’t appear in either the run list or
dependencies of a cookbook in the run_list, it will not get loaded
correctly. As you’ve seen, you can work around this a little bit with
include_recipe and include_attribute, but you won’t load libraries,
resource definitions, or LWRPs. Adding the dependency in the metadata
is the only way to make those work.


Daniel DeLeo

Hello Daniel.

I have the the “torquebox-web-server” cookbook which is wrapped and I
have the “connector” cookbook which is the wrapper.

In the “torquebox-web-sever” cookbook’s “metadata.rb” file I have the
line: depends ‘yum’, '2.2.2’
So the dependency is specified in the wrapped cookbook and using the
torquebox-web-server cookbook directly works fine.

In the “connector” cookbook I depend on the "torquebox-web-server"
cookbook using a Berksfile entry (Which, it has just occurred to be, may
be a factor in this issue).

My understanding is that Chef should pick up the dependencies from the
wrapped cookbooks automatically during convergence so that they do not
need to be re-declared in the wrapper cookbook. Is my understanding correct?

I am not sure how exactly it does the dependency checking. Does
specifying a dependency is the Berksfile means it won’t be properly
picked up by chef in this scenario?

Thanks for the information.

Jeff


#9

On 18/06/13 10:09, Jeffrey Jones wrote:

On 18/06/13 01:24, Daniel DeLeo wrote:

On Sunday, June 16, 2013 at 7:25 PM, Jeffrey Jones wrote:

Hello Steven

On 15/06/13 06:22, Steven Danna wrote:

On 6/14/13 7:57 AM, Julian C. Dunn wrote:

Hey Jeffrey: I don’t see your recipe code anywhere but judging
from the
gist you have something like this:

connector::default has include_recipe "torquebox-web-server"
torquebox-web-server::default has include_recipe “yum::epel”

Is “yum” anywhere in your metadata for any of those cookbooks?

Even I’m not entirely sure whether include_recipe also loads
attributes
from the dependent cookbook…
I have added the debug output after adding the "include_attribute"
section.
It looks like the old error has been fixed but a new error has
appeared.

“[2013-06-17T02:12:46+00:00] FATAL: NameError: Cannot find a resource
for yum_key on centos version 6.4”

I am guessing that the attributes/epel.rb file is being executed by the
selection logic inside the file is failing for some reason?

https://gist.github.com/rurounijones/5771208 (“Add attribute” section
for full debug output)

Cheers

Jeff
As Julian said, the key here is that you need to specify the
dependency between your cookbook and the yum cookbook in the
cookbook’s metadata:

http://docs.opscode.com/essentials_cookbook_metadata.html

Chef 11 uses the metadata to figure out a sane order for cookbook
loading. If a cookbook doesn’t appear in either the run list or
dependencies of a cookbook in the run_list, it will not get loaded
correctly. As you’ve seen, you can work around this a little bit with
include_recipe and include_attribute, but you won’t load libraries,
resource definitions, or LWRPs. Adding the dependency in the metadata
is the only way to make those work.


Daniel DeLeo

Hello Daniel.

I have the the “torquebox-web-server” cookbook which is wrapped and I
have the “connector” cookbook which is the wrapper.

In the “torquebox-web-sever” cookbook’s “metadata.rb” file I have the
line: depends ‘yum’, '2.2.2’
So the dependency is specified in the wrapped cookbook and using the
torquebox-web-server cookbook directly works fine.

In the “connector” cookbook I depend on the "torquebox-web-server"
cookbook using a Berksfile entry (Which, it has just occurred to be,
may be a factor in this issue).

My understanding is that Chef should pick up the dependencies from the
wrapped cookbooks automatically during convergence so that they do not
need to be re-declared in the wrapper cookbook. Is my understanding
correct?

I am not sure how exactly it does the dependency checking. Does
specifying a dependency is the Berksfile means it won’t be properly
picked up by chef in this scenario?

Thanks for the information.

Jeff

Further to last email (After I realised it may be due to using
berkshelf) I did some experimentation. Using berkshelf I installed the
"torquebox-web-server" cookbook (and others) into the site-cookbooks
directory in the “connector” cookbook then added a "depends
’torquebox-web-server’ "dependency into the “connector” metadata.rb file

This appears to have worked. It is not the most elegant solution but
until we can use chef-solo with entries in the metadata.rb file that can
point to arbitrary remote git repositories I cannot think of a better
one (Any suggestions?).

Thanks for helping with my understanding.

Jeff


#10

Hi Jeff,

Chef does not know anything about Berksfile, thus it only checks
metadata.rb, thus you have to keep both in sync.

And yes, in metadata.rb you can not specify an alternative cookbook soure
(like a git location).

The best way is to use declare all dependencies in metadata.rb, then in the
Berksfile use the “metadata” keyword. For any custom-located cookbooks
repeat them in the Berksfile (with a git location for example) - this will
override their location.

Then you can do “berks install” (followed by “berks upload” if you are
using chef server) and it will respect the git location.

HTH, Torben
On Jun 18, 2013 3:45 AM, “Jeffrey Jones” jjones@toppan-f.co.jp wrote:

On 18/06/13 10:09, Jeffrey Jones wrote:

On 18/06/13 01:24, Daniel DeLeo wrote:

On Sunday, June 16, 2013 at 7:25 PM, Jeffrey Jones wrote:

Hello Steven

On 15/06/13 06:22, Steven Danna wrote:

On 6/14/13 7:57 AM, Julian C. Dunn wrote:

Hey Jeffrey: I don’t see your recipe code anywhere but judging from
the
gist you have something like this:

connector::default has include_recipe "torquebox-web-server"
torquebox-web-server::default has include_recipe “yum::epel”

Is “yum” anywhere in your metadata for any of those cookbooks?

Even I’m not entirely sure whether include_recipe also loads
attributes
from the dependent cookbook…

I have added the debug output after adding the "include_attribute"
section.
It looks like the old error has been fixed but a new error has appeared.

“[2013-06-17T02:12:46+00:00] FATAL: NameError: Cannot find a resource
for yum_key on centos version 6.4”

I am guessing that the attributes/epel.rb file is being executed by the
selection logic inside the file is failing for some reason?

https://gist.github.com/**rurounijones/5771208https://gist.github.com/rurounijones/5771208(“Add attribute” section
for full debug output)

Cheers

Jeff

As Julian said, the key here is that you need to specify the dependency
between your cookbook and the yum cookbook in the cookbook’s metadata:

http://docs.opscode.com/**essentials_cookbook_metadata.**htmlhttp://docs.opscode.com/essentials_cookbook_metadata.html

Chef 11 uses the metadata to figure out a sane order for cookbook
loading. If a cookbook doesn’t appear in either the run list or
dependencies of a cookbook in the run_list, it will not get loaded
correctly. As you’ve seen, you can work around this a little bit with
include_recipe and include_attribute, but you won’t load libraries,
resource definitions, or LWRPs. Adding the dependency in the metadata is
the only way to make those work.


Daniel DeLeo

Hello Daniel.

I have the the “torquebox-web-server” cookbook which is wrapped and I
have the “connector” cookbook which is the wrapper.

In the “torquebox-web-sever” cookbook’s “metadata.rb” file I have the
line: depends ‘yum’, '2.2.2’
So the dependency is specified in the wrapped cookbook and using the
torquebox-web-server cookbook directly works fine.

In the “connector” cookbook I depend on the "torquebox-web-server"
cookbook using a Berksfile entry (Which, it has just occurred to be, may be
a factor in this issue).

My understanding is that Chef should pick up the dependencies from the
wrapped cookbooks automatically during convergence so that they do not need
to be re-declared in the wrapper cookbook. Is my understanding correct?

I am not sure how exactly it does the dependency checking. Does
specifying a dependency is the Berksfile means it won’t be properly picked
up by chef in this scenario?

Thanks for the information.

Jeff

Further to last email (After I realised it may be due to using berkshelf)
I did some experimentation. Using berkshelf I installed the
"torquebox-web-server" cookbook (and others) into the site-cookbooks
directory in the “connector” cookbook then added a "depends
’torquebox-web-server’ "dependency into the “connector” metadata.rb file

This appears to have worked. It is not the most elegant solution but until
we can use chef-solo with entries in the metadata.rb file that can point to
arbitrary remote git repositories I cannot think of a better one (Any
suggestions?).

Thanks for helping with my understanding.

Jeff