Role Versioning?

Hey guys,

there was a recent discussion about “role cookbooks” in IRC where
@threescoops posted a link to the wiki where versioning for roles is
mentioned as a proposal:

http://wiki.opscode.com/display/chef/Infrastructure+Proposal#InfrastructureProposal-Roles

Just wanted to throw in my 5 cents and continue discussion here:

[12:56] zts, threescoops: i would be happy as well if:
[12:56] zts, threescoops: roles could be versioned
[12:56] zts, threescoops: and i could define cookbook version
constraints in the roles (want to lock down transitive cookbook
dependencies as well)
[12:56] zts, threescoops: and in the environments i would specify
the role versions rather than cookbook versions

Does that make any sense, or am I on the wrong path here?

Cheers, Torben

We accomplish this with what we call "top level" cookbooks (TLC) (I know,
lame name). I was against them at first but it's working out. We lock
down cookbook dependencies with "depends" in metadata.rb. The cookbooks
themselves use the "pessimistic" approach to versioning so it's not so
restrictive when developing.

So our tlc_tomcat cookbook would have the following in metadata.rb:

name "tlc_tomcat"
version "1.1.0"
depends "base", "= 1.0.6"
depends "os_patch, "= 1.1.2"
depends "auditing", "= 0.3.3"
depends "security", "= 0.1.1"
depends "backups", "= 0.2.5"
depends "splunkagent", "= 0.4.5"
depends "java", "= 1.0.4"
depends "newrelic", "= 1.0.5"
depends "tomcat", "= 1.0.3"

and then the default recipe of the TLC installs them in the order we want
with "include_recipe" statements. This way, tlc_tomcat@1.1.0 will always
install the exact same versions of cookbooks. I mentioned the pessimistic
versioning above because we made the mistake of being rigid with versioning
within the lower level cookbooks as well and it made development a PITA.
As an example, here's the metadata.rb of the tomcat@1.0.3 cookbook in the
TLC above:

name "tomcat"
version "1.0.3"
depends "java", "~> 1.0.3"
depends "os_patch", "~> 1.1.2"
depends "newrelic", "~> 1.0.2"

We haven't used roles since we started with this approach and we haven't
missed them.

MG

On Thu, Feb 14, 2013 at 5:09 AM, Torben Knerr ukio@gmx.de wrote:

Hey guys,

there was a recent discussion about "role cookbooks" in IRC where
@threescoops posted a link to the wiki where versioning for roles is
mentioned as a proposal:

http://wiki.opscode.com/display/chef/Infrastructure+Proposal#InfrastructureProposal-Roles

Just wanted to throw in my 5 cents and continue discussion here:

[12:56] zts, threescoops: i would be happy as well if:
[12:56] zts, threescoops: roles could be versioned
[12:56] zts, threescoops: and i could define cookbook version
constraints in the roles (want to lock down transitive cookbook
dependencies as well)
[12:56] zts, threescoops: and in the environments i would
specify the role versions rather than cookbook versions

Does that make any sense, or am I on the wrong path here?

Cheers, Torben

It'd be nice if all metadata was versioned. Until then, I have git.

On Thu, Feb 14, 2013 at 7:09 AM, Torben Knerr ukio@gmx.de wrote:

Hey guys,

there was a recent discussion about "role cookbooks" in IRC where
@threescoops posted a link to the wiki where versioning for roles is
mentioned as a proposal:

http://wiki.opscode.com/display/chef/Infrastructure+Proposal#InfrastructureProposal-Roles

Just wanted to throw in my 5 cents and continue discussion here:

[12:56] zts, threescoops: i would be happy as well if:
[12:56] zts, threescoops: roles could be versioned
[12:56] zts, threescoops: and i could define cookbook version
constraints in the roles (want to lock down transitive cookbook
dependencies as well)
[12:56] zts, threescoops: and in the environments i would
specify the role versions rather than cookbook versions

Does that make any sense, or am I on the wrong path here?

Cheers, Torben

Can we do that in chef 10.x or is it only possible in chef 11.x.

On Thu, Feb 14, 2013 at 5:40 AM, Michael Glenney mike.glenney@gmail.comwrote:

We accomplish this with what we call "top level" cookbooks (TLC) (I know,
lame name). I was against them at first but it's working out. We lock
down cookbook dependencies with "depends" in metadata.rb. The cookbooks
themselves use the "pessimistic" approach to versioning so it's not so
restrictive when developing.

So our tlc_tomcat cookbook would have the following in metadata.rb:

name "tlc_tomcat"
version "1.1.0"
depends "base", "= 1.0.6"
depends "os_patch, "= 1.1.2"
depends "auditing", "= 0.3.3"
depends "security", "= 0.1.1"
depends "backups", "= 0.2.5"
depends "splunkagent", "= 0.4.5"
depends "java", "= 1.0.4"
depends "newrelic", "= 1.0.5"
depends "tomcat", "= 1.0.3"

and then the default recipe of the TLC installs them in the order we want
with "include_recipe" statements. This way, tlc_tomcat@1.1.0 will always
install the exact same versions of cookbooks. I mentioned the pessimistic
versioning above because we made the mistake of being rigid with versioning
within the lower level cookbooks as well and it made development a PITA.
As an example, here's the metadata.rb of the tomcat@1.0.3 cookbook in the
TLC above:

name "tomcat"
version "1.0.3"
depends "java", "~> 1.0.3"
depends "os_patch", "~> 1.1.2"
depends "newrelic", "~> 1.0.2"

We haven't used roles since we started with this approach and we haven't
missed them.

MG

On Thu, Feb 14, 2013 at 5:09 AM, Torben Knerr ukio@gmx.de wrote:

Hey guys,

there was a recent discussion about "role cookbooks" in IRC where
@threescoops posted a link to the wiki where versioning for roles is
mentioned as a proposal:

http://wiki.opscode.com/display/chef/Infrastructure+Proposal#InfrastructureProposal-Roles

Just wanted to throw in my 5 cents and continue discussion here:

[12:56] zts, threescoops: i would be happy as well if:
[12:56] zts, threescoops: roles could be versioned
[12:56] zts, threescoops: and i could define cookbook version
constraints in the roles (want to lock down transitive cookbook
dependencies as well)
[12:56] zts, threescoops: and in the environments i would
specify the role versions rather than cookbook versions

Does that make any sense, or am I on the wrong path here?

Cheers, Torben

--
..Senthil

"If there's anything more important than my ego around, I want it
caught and shot now."
- Douglas Adams.

"Can we do that in chef 10.x or is it only possible in chef 11.x."

if you're asking me then yes, we do it with 10.x currently

On Thu, Feb 14, 2013 at 9:54 AM, Senthilvel Rangaswamy <senthilvel@gmail.com

wrote:

Can we do that in chef 10.x or is it only possible in chef 11.x.

On Thu, Feb 14, 2013 at 5:40 AM, Michael Glenney mike.glenney@gmail.comwrote:

We accomplish this with what we call "top level" cookbooks (TLC) (I know,
lame name). I was against them at first but it's working out. We lock
down cookbook dependencies with "depends" in metadata.rb. The cookbooks
themselves use the "pessimistic" approach to versioning so it's not so
restrictive when developing.

So our tlc_tomcat cookbook would have the following in metadata.rb:

name "tlc_tomcat"
version "1.1.0"
depends "base", "= 1.0.6"
depends "os_patch, "= 1.1.2"
depends "auditing", "= 0.3.3"
depends "security", "= 0.1.1"
depends "backups", "= 0.2.5"
depends "splunkagent", "= 0.4.5"
depends "java", "= 1.0.4"
depends "newrelic", "= 1.0.5"
depends "tomcat", "= 1.0.3"

and then the default recipe of the TLC installs them in the order we want
with "include_recipe" statements. This way, tlc_tomcat@1.1.0 will
always install the exact same versions of cookbooks. I mentioned the
pessimistic versioning above because we made the mistake of being rigid
with versioning within the lower level cookbooks as well and it made
development a PITA. As an example, here's the metadata.rb of the
tomcat@1.0.3 cookbook in the TLC above:

name "tomcat"
version "1.0.3"
depends "java", "~> 1.0.3"
depends "os_patch", "~> 1.1.2"
depends "newrelic", "~> 1.0.2"

We haven't used roles since we started with this approach and we haven't
missed them.

MG

On Thu, Feb 14, 2013 at 5:09 AM, Torben Knerr ukio@gmx.de wrote:

Hey guys,

there was a recent discussion about "role cookbooks" in IRC where
@threescoops posted a link to the wiki where versioning for roles is
mentioned as a proposal:

http://wiki.opscode.com/display/chef/Infrastructure+Proposal#InfrastructureProposal-Roles

Just wanted to throw in my 5 cents and continue discussion here:

[12:56] zts, threescoops: i would be happy as well if:
[12:56] zts, threescoops: roles could be versioned
[12:56] zts, threescoops: and i could define cookbook version
constraints in the roles (want to lock down transitive cookbook
dependencies as well)
[12:56] zts, threescoops: and in the environments i would
specify the role versions rather than cookbook versions

Does that make any sense, or am I on the wrong path here?

Cheers, Torben

--
..Senthil

"If there's anything more important than my ego around, I want it
caught and shot now."
- Douglas Adams.

Chef 11.x has ordered evaluation of attributes files which does make
this easier, otherwise you need to be careful about using
include_attribute correctly

(

)

On 2/14/13 10:28 AM, Michael Glenney wrote:

"Can we do that in chef 10.x or is it only possible in chef 11.x."

if you're asking me then yes, we do it with 10.x currently

On Thu, Feb 14, 2013 at 9:54 AM, Senthilvel Rangaswamy
<senthilvel@gmail.com mailto:senthilvel@gmail.com> wrote:

Can we do that in chef 10.x or is it only possible in chef 11.x.


On Thu, Feb 14, 2013 at 5:40 AM, Michael Glenney
<mike.glenney@gmail.com <mailto:mike.glenney@gmail.com>> wrote:

    We accomplish this with what we call "top level" cookbooks
    (TLC) (I know, lame name).  I was against them at first but
    it's working out.  We lock down cookbook dependencies with
    "depends" in metadata.rb.  The cookbooks themselves use the
    "pessimistic" approach to versioning so it's not so
    restrictive when developing.

    So our tlc_tomcat cookbook would have the following in
    metadata.rb:

    name "tlc_tomcat"
    version "1.1.0"
    depends "base", "= 1.0.6"
    depends "os_patch, "= 1.1.2"
    depends "auditing", "= 0.3.3"
    depends "security", "= 0.1.1"
    depends "backups", "= 0.2.5"
    depends "splunkagent", "= 0.4.5"
    depends "java", "= 1.0.4"
    depends "newrelic", "= 1.0.5"
    depends "tomcat", "= 1.0.3"

    and then the default recipe of the TLC installs them in the
    order we want with "include_recipe" statements.  This way,
    tlc_tomcat@1.1.0 will always install the exact same versions
    of cookbooks.  I mentioned the pessimistic versioning above
    because we made the mistake of being rigid with versioning
    within the lower level cookbooks as well and it made
    development a PITA.  As an example, here's the metadata.rb of
    the tomcat@1.0.3 cookbook in the TLC above:

    name "tomcat"
    version "1.0.3"
    depends  "java", "~> 1.0.3"
    depends  "os_patch", "~> 1.1.2"
    depends  "newrelic", "~> 1.0.2"

    We haven't used roles since we started with this approach and
    we haven't missed them.

    MG


    On Thu, Feb 14, 2013 at 5:09 AM, Torben Knerr <ukio@gmx.de
    <mailto:ukio@gmx.de>> wrote:

        Hey guys,

        there was a recent discussion about "role cookbooks" in
        IRC where @threescoops posted a link to the wiki where
        versioning for roles is mentioned as a proposal:

        http://wiki.opscode.com/display/chef/Infrastructure+Proposal#InfrastructureProposal-Roles

        Just wanted to throw in my 5 cents and continue discussion
        here:

        [12:56] <tknerr> zts, threescoops: i would be happy as
        well if:
        [12:56] <tknerr> zts, threescoops: roles could be versioned
        [12:56] <tknerr> zts, threescoops: and i could define
        cookbook version constraints in the roles (want to lock
        down transitive cookbook dependencies as well)
        [12:56] <tknerr> zts, threescoops: and in the environments
        i would specify the role versions rather than cookbook
        versions

        Does that make any sense, or am I on the wrong path here?

        Cheers, Torben






-- 
..Senthil

"If there's anything more important than my ego around, I want it
 caught and shot now."
- Douglas Adams.

@Michael: what you describe is exactly what I'm doing now, but naming it
"application cookbook" or "role cookbook" rather than "top level cookbook"
(which is a quite precise name though).

Fully agree that the top level cookbook (or whatever you call it) is the
place for pessimistically locking dependency versions, and the other
cookbooks being included (most of us would call them "library cookbooks" I
guess) should use optimistic dependency versioning. Actually, what I see in
most of the community cookbooks is no version constraint at all, which is
fairly optimistic but ok imho (see also Ruby Libraries vs Ruby
Applicationshttp://yehudakatz.com/2010/12/16/clarifying-the-roles-of-the-gemspec-and-gemfile/,
it's the same thing on the gem level and the cookbook level).

The only but very very big limitation of the "application cookbook" / "role
cookbook" / "top level cookbook" pattern is that due to the
include_recipes the included recipes don't show up in the run_list
anymore and thus you can't search for them (which breaks many of the
community cookbooks, e.g. apt).

That's why the discussion in IRC came up I guess, and where @threescoops posted
the above mentioned link which proposes versioning of roles. If roles could
be versioned already from the beginning, the idea of "application cookbook"
/ "role cookbook" / "top level cookbook" probably never came up as we could
just use roles instead.

But if role versioning would be added in the next Chef version, I would
argue that:

  • roles are the place to lock down cookbook dependency versions
  • environments are the place to lock down role versions

This would allow pessimistic versioning on a role level and keep the
environments free from cookbook version clutter.

I think that the environment is ok for locking "top level cookbook
versions" (or role versions if that would be possible someday), but it's
the totally the wrong place for locking "library cookbook" versions (why?
because you might have two different nodes in the same environment
requiring different versions of the postgresql cookbook for example).

On Thu, Feb 14, 2013 at 8:48 PM, Lamont Granquist lamont@opscode.comwrote:

Chef 11.x has ordered evaluation of attributes files which does make this
easier, otherwise you need to be careful about using include_attribute
correctly

(
Chef 11 In-Depth: Attributes Changes - Chef Blog | Chef)

On 2/14/13 10:28 AM, Michael Glenney wrote:

"Can we do that in chef 10.x or is it only possible in chef 11.x."

if you're asking me then yes, we do it with 10.x currently

On Thu, Feb 14, 2013 at 9:54 AM, Senthilvel Rangaswamy <
senthilvel@gmail.com> wrote:

Can we do that in chef 10.x or is it only possible in chef 11.x.

On Thu, Feb 14, 2013 at 5:40 AM, Michael Glenney mike.glenney@gmail.comwrote:

We accomplish this with what we call "top level" cookbooks (TLC) (I
know, lame name). I was against them at first but it's working out. We
lock down cookbook dependencies with "depends" in metadata.rb. The
cookbooks themselves use the "pessimistic" approach to versioning so it's
not so restrictive when developing.

So our tlc_tomcat cookbook would have the following in metadata.rb:

name "tlc_tomcat"
version "1.1.0"
depends "base", "= 1.0.6"
depends "os_patch, "= 1.1.2"
depends "auditing", "= 0.3.3"
depends "security", "= 0.1.1"
depends "backups", "= 0.2.5"
depends "splunkagent", "= 0.4.5"
depends "java", "= 1.0.4"
depends "newrelic", "= 1.0.5"
depends "tomcat", "= 1.0.3"

and then the default recipe of the TLC installs them in the order we
want with "include_recipe" statements. This way, tlc_tomcat@1.1.0 will
always install the exact same versions of cookbooks. I mentioned the
pessimistic versioning above because we made the mistake of being rigid
with versioning within the lower level cookbooks as well and it made
development a PITA. As an example, here's the metadata.rb of the
tomcat@1.0.3 cookbook in the TLC above:

name "tomcat"
version "1.0.3"
depends "java", "~> 1.0.3"
depends "os_patch", "~> 1.1.2"
depends "newrelic", "~> 1.0.2"

We haven't used roles since we started with this approach and we
haven't missed them.

MG

On Thu, Feb 14, 2013 at 5:09 AM, Torben Knerr ukio@gmx.de wrote:

Hey guys,

there was a recent discussion about "role cookbooks" in IRC where
@threescoops posted a link to the wiki where versioning for roles is
mentioned as a proposal:

http://wiki.opscode.com/display/chef/Infrastructure+Proposal#InfrastructureProposal-Roles

Just wanted to throw in my 5 cents and continue discussion here:

[12:56] zts, threescoops: i would be happy as well if:
[12:56] zts, threescoops: roles could be versioned
[12:56] zts, threescoops: and i could define cookbook
version constraints in the roles (want to lock down transitive cookbook
dependencies as well)
[12:56] zts, threescoops: and in the environments i would
specify the role versions rather than cookbook versions

Does that make any sense, or am I on the wrong path here?

Cheers, Torben

--
..Senthil

"If there's anything more important than my ego around, I want it
caught and shot now."
- Douglas Adams.

We don't use any community cookbooks so it hasn't been an issue for us yet. Maybe I should be careful about throwing that out there. Though I'm surprised to hear there isn't a way to see all cookbooks that are applied to a node unless they're in the run list. That would mean you can't see any if the dependency cookbooks unless they're explicitly listed as well. Sounds like a gap or opscode trying to force a "best practice".

And to be clear, in our TLC's we use explicit dependencies ("java" "= 1.0.0"). To ensure we lock down the version. In the std cookbooks we use pessimistic versioning ("~> 1.0.0"). We may switch to optimistic in the future to further ease development as we open it to more teams.

Michael Glenney
Sent from my iPhone

On Feb 14, 2013, at 3:20 PM, Torben Knerr ukio@gmx.de wrote:

@Michael: what you describe is exactly what I'm doing now, but naming it "application cookbook" or "role cookbook" rather than "top level cookbook" (which is a quite precise name though).

Fully agree that the top level cookbook (or whatever you call it) is the place for pessimistically locking dependency versions, and the other cookbooks being included (most of us would call them "library cookbooks" I guess) should use optimistic dependency versioning. Actually, what I see in most of the community cookbooks is no version constraint at all, which is fairly optimistic but ok imho (see also Ruby Libraries vs Ruby Applications, it's the same thing on the gem level and the cookbook level).

The only but very very big limitation of the "application cookbook" / "role cookbook" / "top level cookbook" pattern is that due to the include_recipes the included recipes don't show up in the run_list anymore and thus you can't search for them (which breaks many of the community cookbooks, e.g. apt).

That's why the discussion in IRC came up I guess, and where @threescoops posted the above mentioned link which proposes versioning of roles. If roles could be versioned already from the beginning, the idea of "application cookbook" / "role cookbook" / "top level cookbook" probably never came up as we could just use roles instead.

But if role versioning would be added in the next Chef version, I would argue that:

  • roles are the place to lock down cookbook dependency versions
  • environments are the place to lock down role versions

This would allow pessimistic versioning on a role level and keep the environments free from cookbook version clutter.

I think that the environment is ok for locking "top level cookbook versions" (or role versions if that would be possible someday), but it's the totally the wrong place for locking "library cookbook" versions (why? because you might have two different nodes in the same environment requiring different versions of the postgresql cookbook for example).

On Thu, Feb 14, 2013 at 8:48 PM, Lamont Granquist lamont@opscode.com wrote:

Chef 11.x has ordered evaluation of attributes files which does make this easier, otherwise you need to be careful about using include_attribute correctly

( Chef 11 In-Depth: Attributes Changes - Chef Blog | Chef )

On 2/14/13 10:28 AM, Michael Glenney wrote:

"Can we do that in chef 10.x or is it only possible in chef 11.x."

if you're asking me then yes, we do it with 10.x currently

On Thu, Feb 14, 2013 at 9:54 AM, Senthilvel Rangaswamy senthilvel@gmail.com wrote:

Can we do that in chef 10.x or is it only possible in chef 11.x.

On Thu, Feb 14, 2013 at 5:40 AM, Michael Glenney mike.glenney@gmail.com wrote:

We accomplish this with what we call "top level" cookbooks (TLC) (I know, lame name). I was against them at first but it's working out. We lock down cookbook dependencies with "depends" in metadata.rb. The cookbooks themselves use the "pessimistic" approach to versioning so it's not so restrictive when developing.

So our tlc_tomcat cookbook would have the following in metadata.rb:

name "tlc_tomcat"
version "1.1.0"
depends "base", "= 1.0.6"
depends "os_patch, "= 1.1.2"
depends "auditing", "= 0.3.3"
depends "security", "= 0.1.1"
depends "backups", "= 0.2.5"
depends "splunkagent", "= 0.4.5"
depends "java", "= 1.0.4"
depends "newrelic", "= 1.0.5"
depends "tomcat", "= 1.0.3"

and then the default recipe of the TLC installs them in the order we want with "include_recipe" statements. This way, tlc_tomcat@1.1.0 will always install the exact same versions of cookbooks. I mentioned the pessimistic versioning above because we made the mistake of being rigid with versioning within the lower level cookbooks as well and it made development a PITA. As an example, here's the metadata.rb of the tomcat@1.0.3 cookbook in the TLC above:

name "tomcat"
version "1.0.3"
depends "java", "~> 1.0.3"
depends "os_patch", "~> 1.1.2"
depends "newrelic", "~> 1.0.2"

We haven't used roles since we started with this approach and we haven't missed them.

MG

On Thu, Feb 14, 2013 at 5:09 AM, Torben Knerr ukio@gmx.de wrote:

Hey guys,

there was a recent discussion about "role cookbooks" in IRC where @threescoops posted a link to the wiki where versioning for roles is mentioned as a proposal:

http://wiki.opscode.com/display/chef/Infrastructure+Proposal#InfrastructureProposal-Roles

Just wanted to throw in my 5 cents and continue discussion here:

[12:56] zts, threescoops: i would be happy as well if:
[12:56] zts, threescoops: roles could be versioned
[12:56] zts, threescoops: and i could define cookbook version constraints in the roles (want to lock down transitive cookbook dependencies as well)
[12:56] zts, threescoops: and in the environments i would specify the role versions rather than cookbook versions

Does that make any sense, or am I on the wrong path here?

Cheers, Torben

--
..Senthil

"If there's anything more important than my ego around, I want it
caught and shot now."
- Douglas Adams.

Hi,

On Fri, Feb 15, 2013 at 9:48 AM, Michael Glenney mike.glenney@gmail.com wrote:

We don't use any community cookbooks so it hasn't been an issue for us yet.
Maybe I should be careful about throwing that out there. Though I'm
surprised to hear there isn't a way to see all cookbooks that are applied to
a node unless they're in the run list. That would mean you can't see any if
the dependency cookbooks unless they're explicitly listed as well. Sounds
like a gap or opscode trying to force a "best practice".

It is a gap IMHO. We have the following snippet in a cookbook that
records the final expanded set of recipes executed. We then have a
whole suite of custom scripts that mine this data at a later time.

ruby_block "get_seen_recipes" do
block do
node.override["knife_audit"]["seen_recipes"] = node.run_state[:seen_recipes]
end
action :create
end

And to be clear, in our TLC's we use explicit dependencies ("java" "=
1.0.0"). To ensure we lock down the version. In the std cookbooks we use
pessimistic versioning ("~> 1.0.0"). We may switch to optimistic in the
future to further ease development as we open it to more teams.

Interesting to hear you say that. As we start to move from a smaller
team to a larger one. We are moving towards having pessimistic
dependencies in certain TLCs cookbooks and having pessimistic
constraints in our environments for just the TLCs. Right now we do all
of our version constraints in the environment config files.

--
Cheers,

Peter Donald

We are in the “use cookbooks instead of roles/environments” camp and
lock down versions in our TLC’s as discussed. We never use search
directly in cookbooks (well almost never). We wrote a search wrapper
similar to https://github.com/infochimps-labs/ironfan-pantry/tree/master/cookbooks/silverware
that works in chef-solo as well and that has served us well.

–Brian

The constraints on versions just seem to be an annoyance when developing. If we want to test changes on a full run list you end up having to create new versions of other cookbooks with the same dependency just so you can run your test. Even if nothing had changed in that cookbook. We recently changed to the pessimistic versioning in the lab and its been nice.

We use our environment files to gate what can be run in that environment based on a standard QA cycle. So all the versions in there are "<= ver"

Michael Glenney
Sent from my iPhone

On Feb 14, 2013, at 4:24 PM, Peter Donald peter@realityforge.org wrote:

Hi,

On Fri, Feb 15, 2013 at 9:48 AM, Michael Glenney mike.glenney@gmail.com wrote:

We don't use any community cookbooks so it hasn't been an issue for us yet.
Maybe I should be careful about throwing that out there. Though I'm
surprised to hear there isn't a way to see all cookbooks that are applied to
a node unless they're in the run list. That would mean you can't see any if
the dependency cookbooks unless they're explicitly listed as well. Sounds
like a gap or opscode trying to force a "best practice".

It is a gap IMHO. We have the following snippet in a cookbook that
records the final expanded set of recipes executed. We then have a
whole suite of custom scripts that mine this data at a later time.

ruby_block "get_seen_recipes" do
block do
node.override["knife_audit"]["seen_recipes"] = node.run_state[:seen_recipes]
end
action :create
end

And to be clear, in our TLC's we use explicit dependencies ("java" "=
1.0.0"). To ensure we lock down the version. In the std cookbooks we use
pessimistic versioning ("~> 1.0.0"). We may switch to optimistic in the
future to further ease development as we open it to more teams.

Interesting to hear you say that. As we start to move from a smaller
team to a larger one. We are moving towards having pessimistic
dependencies in certain TLCs cookbooks and having pessimistic
constraints in our environments for just the TLCs. Right now we do all
of our version constraints in the environment config files.

--
Cheers,

Peter Donald

Cool ideas. Now I want to be able to know that data. I'll try these approaches. Thx

Michael Glenney
Sent from my iPhone

On Feb 14, 2013, at 4:33 PM, Brian Akins brian@akins.org wrote:

We are in the "use cookbooks instead of roles/environments" camp and
lock down versions in our TLC's as discussed. We never use search
directly in cookbooks (well almost never). We wrote a search wrapper
similar to ironfan-pantry/cookbooks/silverware at master · infochimps-labs/ironfan-pantry · GitHub
that works in chef-solo as well and that has served us well.

--Brian

For this exact reason, in my own top-level cookbook, I am not using include_recipe, but rather node.run_list.add (actually, I’m using a mix of the two).

The effect is almost the same, but because the recipes are added to the runlist, they are searchable.

One difference is that these recipes will not take effect until the next chef run.

The second difference is that with include_recipe, you have more control over order-of-execution. In my mind, that’s actually a benefit; include_recipe tempts you to think procedurally instead of declaratively.

A third difference is that I don’t need to declare all the cookbooks as dependency in my top-level cookbook. With include_recipe, I may have dependencies on many cookbooks that aren’t even relevant to a particular server. It makes it easier to have a single top-level cookbook for all my servers and use a big case statement to distinguish the servers.

Of course you can still declare a dependency in the metadata if you need it to lock down a version number. You just don’t have to do it for all your recipes.
-----Original message-----
From: Torben Knerr ukio@gmx.de
Sent: Thursday 14th February 2013 14:21
To: chef@lists.opscode.com
Subject: [chef] Re: Re: Re: Re: Re: Role Versioning?

The only but very very big limitation of the “application cookbook” / “role cookbook” / “top level cookbook” pattern is that due to the include_recipes the included recipes don’t show up in the run_list anymore and thus you can’t search for them (which breaks many of the community cookbooks, e.g. apt).

Am 15.02.2013 00:24 schrieb "Peter Donald" peter@realityforge.org:

Hi,

On Fri, Feb 15, 2013 at 9:48 AM, Michael Glenney mike.glenney@gmail.com
wrote:

We don't use any community cookbooks so it hasn't been an issue for us
yet.
Maybe I should be careful about throwing that out there. Though I'm
surprised to hear there isn't a way to see all cookbooks that are
applied to
a node unless they're in the run list. That would mean you can't see
any if
the dependency cookbooks unless they're explicitly listed as well.
Sounds
like a gap or opscode trying to force a "best practice".

It is a gap IMHO. We have the following snippet in a cookbook that
records the final expanded set of recipes executed. We then have a
whole suite of custom scripts that mine this data at a later time.

ruby_block "get_seen_recipes" do
block do
node.override["knife_audit"]["seen_recipes"] =
node.run_state[:seen_recipes]
end
action :create
end

I would even go that far and state that using search "recipe:*" is an
antipattern and should not be used in community cookbooks (due to the
include_recipe flaws).

Thanks for the pointer to knife-audit (
GitHub - jbz/knife-audit: Plugin for Opscode Chef's 'knife' utility to gather cookbook use info) btw, looks like a nice add-on that
will be added soon to my standard
knife plugins :wink:

And to be clear, in our TLC's we use explicit dependencies ("java" "=
1.0.0"). To ensure we lock down the version. In the std cookbooks we
use
pessimistic versioning ("~> 1.0.0"). We may switch to optimistic in the
future to further ease development as we open it to more teams.

Interesting to hear you say that. As we start to move from a smaller
team to a larger one. We are moving towards having pessimistic
dependencies in certain TLCs cookbooks and having pessimistic
constraints in our environments for just the TLCs. Right now we do all
of our version constraints in the environment config files.

--
Cheers,

Peter Donald

Am 15.02.2013 04:42 schrieb "Kevin Keane Subscription" <
subscription@kkeane.com>:

For this exact reason, in my own top-level cookbook, I am not using
include_recipe, but rather node.run_list.add (actually, I'm using a mix of
the two).

Nice, I didn't think of this possibility yet

The effect is almost the same, but because the recipes are added to the
runlist, they are searchable.

One difference is that these recipes will not take effect until the
next chef run.

And that would be the dealbreaker for me.

But: you could probably combine both approaches, i.e. using
'include_recipe' but then do something like this at the end of the chef run:

ruby_block "get_seen_recipes" do
block do
node.override["knife_audit"]["seen_recipes"] =
node.run_state[:seen_recipes]
node.run_list.clear
node.run_state[:seen_recipes].each do { ¦r¦ node.run_list.add r }
end
action :create
end

Are there any reasons against such an approach? For sure, searches for
"recipe:*" would only work after the first chef run of the node, but that
would not be a big deal.

The second difference is that with include_recipe, you have more control
over order-of-execution. In my mind, that's actually a benefit;
include_recipe tempts you to think procedurally instead of declaratively.

A third difference is that I don't need to declare all the cookbooks as
dependency in my top-level cookbook. With include_recipe, I may have
dependencies on many cookbooks that aren't even relevant to a particular
server. It makes it easier to have a single top-level cookbook for all my
servers and use a big case statement to distinguish the servers.

Of course you can still declare a dependency in the metadata if you need
it to lock down a version number. You just don't have to do it for all
your recipes.

-----Original message-----
From: Torben Knerr ukio@gmx.de
Sent: Thursday 14th February 2013 14:21
To: chef@lists.opscode.com
Subject: [chef] Re: Re: Re: Re: Re: Role Versioning?

The only but very very big limitation of the "application cookbook" /
"role cookbook" / "top level cookbook" pattern is that due to the
include_recipes the included recipes don't show up in the run_list
anymore and thus you can't search for them (which breaks many of the
community cookbooks, e.g. apt).

Am 15.02.2013 07:09 schrieb "Torben Knerr" ukio@gmx.de:

Am 15.02.2013 04:42 schrieb "Kevin Keane Subscription" <
subscription@kkeane.com>:

For this exact reason, in my own top-level cookbook, I am not using
include_recipe, but rather node.run_list.add (actually, I'm using a mix of
the two).

Nice, I didn't think of this possibility yet

The effect is almost the same, but because the recipes are added to
the runlist, they are searchable.

One difference is that these recipes will not take effect until the
next chef run.

And that would be the dealbreaker for me.

But: you could probably combine both approaches, i.e. using
'include_recipe' but then do something like this at the end of the chef run:

ruby_block "get_seen_recipes" do
block do
node.override["knife_audit"]["seen_recipes"] =
node.run_state[:seen_recipes]
node.run_list.clear
node.run_state[:seen_recipes].each do { ¦r¦ node.run_list.add r }
end
action :create
end

Are there any reasons against such an approach? For sure, searches for
"recipe:*" would only work after the first chef run of the node, but that
would not be a big deal.

Oh wait - this is borken :frowning:

=> if you remove an include_recipe or if that include_recipe is skipped due
to a conditional, it would require two chef runs to become effective - ugh
:-/

The second difference is that with include_recipe, you have more
control over order-of-execution. In my mind, that's actually a benefit;
include_recipe tempts you to think procedurally instead of declaratively.

A third difference is that I don't need to declare all the cookbooks as
dependency in my top-level cookbook. With include_recipe, I may have
dependencies on many cookbooks that aren't even relevant to a particular
server. It makes it easier to have a single top-level cookbook for all my
servers and use a big case statement to distinguish the servers.

Of course you can still declare a dependency in the metadata if you
need it to lock down a version number. You just don't have to do it for
all your recipes.

-----Original message-----
From: Torben Knerr ukio@gmx.de
Sent: Thursday 14th February 2013 14:21
To: chef@lists.opscode.com
Subject: [chef] Re: Re: Re: Re: Re: Role Versioning?

The only but very very big limitation of the "application cookbook" /
"role cookbook" / "top level cookbook" pattern is that due to the
include_recipes the included recipes don't show up in the run_list
anymore and thus you can't search for them (which breaks many of the
community cookbooks, e.g. apt).

We should just fix it, so that recipes included are searchable.

Adam

From: Torben Knerr <ukio@gmx.demailto:ukio@gmx.de>
Reply-To: "chef@lists.opscode.commailto:chef@lists.opscode.com" <chef@lists.opscode.commailto:chef@lists.opscode.com>
Date: Thursday, February 14, 2013 10:33 PM
To: "chef@lists.opscode.commailto:chef@lists.opscode.com" <chef@lists.opscode.commailto:chef@lists.opscode.com>
Subject: [chef] Re: RE: Re: Re: Re: Re: Re: Role Versioning?

Am 15.02.2013 07:09 schrieb "Torben Knerr" <ukio@gmx.demailto:ukio@gmx.de>:

Am 15.02.2013 04:42 schrieb "Kevin Keane Subscription" <subscription@kkeane.commailto:subscription@kkeane.com>:

For this exact reason, in my own top-level cookbook, I am not using include_recipe, but rather node.run_list.add (actually, I'm using a mix of the two).

Nice, I didn't think of this possibility yet

The effect is almost the same, but because the recipes are added to the runlist, they are searchable.

One difference is that these recipes will not take effect until the next chef run.

And that would be the dealbreaker for me.

But: you could probably combine both approaches, i.e. using 'include_recipe' but then do something like this at the end of the chef run:

ruby_block "get_seen_recipes" do
block do
node.override["knife_audit"]["seen_recipes"] = node.run_state[:seen_recipes]
node.run_list.clear
node.run_state[:seen_recipes].each do { ¦r¦ node.run_list.add r }
end
action :create
end

Are there any reasons against such an approach? For sure, searches for "recipe:*" would only work after the first chef run of the node, but that would not be a big deal.

Oh wait - this is borken :frowning:

=> if you remove an include_recipe or if that include_recipe is skipped due to a conditional, it would require two chef runs to become effective - ugh :-/

The second difference is that with include_recipe, you have more control over order-of-execution. In my mind, that's actually a benefit; include_recipe tempts you to think procedurally instead of declaratively.

A third difference is that I don't need to declare all the cookbooks as dependency in my top-level cookbook. With include_recipe, I may have dependencies on many cookbooks that aren't even relevant to a particular server. It makes it easier to have a single top-level cookbook for all my servers and use a big case statement to distinguish the servers.

Of course you can still declare a dependency in the metadata if you need it to lock down a version number. You just don't have to do it for all your recipes.

-----Original message-----
From: Torben Knerr <ukio@gmx.demailto:ukio@gmx.de>
Sent: Thursday 14th February 2013 14:21
To: chef@lists.opscode.commailto:chef@lists.opscode.com
Subject: [chef] Re: Re: Re: Re: Re: Role Versioning?

The only but very very big limitation of the "application cookbook" / "role cookbook" / "top level cookbook" pattern is that due to the include_recipes the included recipes don't show up in the run_list anymore and thus you can't search for them (which breaks many of the community cookbooks, e.g. apt).

.Your mileage may vary. I have no problem with recipes only taking effect the next time, but that's just me. In fact, I actually like it because this way, on the first run on a new node, for the most part only the top-level cookbook will run. That makes for a quick and reliable bootstrapping mechanism.

If you want to get the best of both worlds, you could always do both:

include_recipe "mycookbook"
node.run_list.add("mycookbook")

The recipe will run right away, AND be searchable as soon as the chef run completes (well, OK, after the updated node information has been indexed). On the second run, the cookbook would be called twice, but since chef is declaratory, that shouldn't matter.

As for your concern about removing recipes - good point. In fact, my pattern does have a somewhat unexpected behavior: if you remove a recipe from the runlist using knife, it will come back on the next chef run. My philosophy on that: don't do it. If you use a TLC to manage your runlist, then stick with that.

Another option - conceptually very clean, in my mind: think of the "main" cookbook not as a TLC, but as a complete chef run that sets up a different system (namely, the runlist, as well as any attributes, as needed).

If you want this approach, don't include the TLC in your runlist at all. Instead, write a script that gets called by cron:

chef-client --override-runlist=tlc_cookbook
chef-client

The reason I consider this clean is that in effect, you are simply adding a third phase to a chef run. Instead of compile-execute, we now have setup-runlist/compile/execute.

-----Original message-----
From: Torben Knerr ukio@gmx.de
Sent: Thursday 14th February 2013 22:33
To: chef@lists.opscode.com
Subject: [chef] Re: RE: Re: Re: Re: Re: Re: Role Versioning?

Am 15.02.2013 07:09 schrieb "Torben Knerr" <ukio@gmx.de mailto:ukio@gmx.de >:

Am 15.02.2013 04:42 schrieb "Kevin Keane Subscription" <subscription@kkeane.com mailto:subscription@kkeane.com >:

For this exact reason, in my own top-level cookbook, I am not using include_recipe, but rather node.run_list.add (actually, I'm using a mix of the two).

Nice, I didn't think of this possibility yet

The effect is almost the same, but because the recipes are added to the runlist, they are searchable.

One difference is that these recipes will not take effect until the next chef run.

And that would be the dealbreaker for me.

But: you could probably combine both approaches, i.e. using 'include_recipe' but then do something like this at the end of the chef run:

ruby_block "get_seen_recipes" do
block do
node.override["knife_audit"]["seen_recipes"] = node.run_state[:seen_recipes]
node.run_list.clear
node.run_state[:seen_recipes].each do { ¦r¦ node.run_list.add r }
end
action :create
end

Are there any reasons against such an approach? For sure, searches for "recipe:*" would only work after the first chef run of the node, but that would not be a big deal.

Oh wait - this is borken :frowning:

=> if you remove an include_recipe or if that include_recipe is skipped due to a conditional, it would require two chef runs to become effective - ugh :-/

The second difference is that with include_recipe, you have more control over order-of-execution. In my mind, that's actually a benefit; include_recipe tempts you to think procedurally instead of declaratively.

A third difference is that I don't need to declare all the cookbooks as dependency in my top-level cookbook. With include_recipe, I may have dependencies on many cookbooks that aren't even relevant to a particular server. It makes it easier to have a single top-level cookbook for all my servers and use a big case statement to distinguish the servers.

Of course you can still declare a dependency in the metadata if you need it to lock down a version number. You just don't have to do it for all your recipes.

-----Original message-----
From: Torben Knerr <ukio@gmx.de mailto:ukio@gmx.de >
Sent: Thursday 14th February 2013 14:21
To: chef@lists.opscode.com mailto:chef@lists.opscode.com
Subject: [chef] Re: Re: Re: Re: Re: Role Versioning?

The only but very very big limitation of the "application cookbook" / "role cookbook" / "top level cookbook" pattern is that due to the include_recipes the included recipes don't show up in the run_list anymore and thus you can't search for them (which breaks many of the community cookbooks, e.g. apt).

+1 for this

Actually I don't know how the search index for "recipe:*" is built.

Is it built directly from the expanded run_list?

Or is there a special node attribute like node[:recipes] that could be
populated with node.run_state[:seen_recipes] at the very end of the Chef
run?

Hey,I just found the corresponding ticket to up-vote :wink:
http://tickets.opscode.com/browse/CHEF-1977

Cheers, Torben

On Fri, Feb 15, 2013 at 8:41 AM, Adam Jacob adam@opscode.com wrote:

We should just fix it, so that recipes included are searchable.

Adam

From: Torben Knerr ukio@gmx.de
Reply-To: "chef@lists.opscode.com" chef@lists.opscode.com
Date: Thursday, February 14, 2013 10:33 PM
To: "chef@lists.opscode.com" chef@lists.opscode.com
Subject: [chef] Re: RE: Re: Re: Re: Re: Re: Role Versioning?

Am 15.02.2013 07:09 schrieb "Torben Knerr" ukio@gmx.de:

Am 15.02.2013 04:42 schrieb "Kevin Keane Subscription" <
subscription@kkeane.com>:

For this exact reason, in my own top-level cookbook, I am not using
include_recipe, but rather node.run_list.add (actually, I'm using a mix of
the two).

Nice, I didn't think of this possibility yet

The effect is almost the same, but because the recipes are added to
the runlist, they are searchable.

One difference is that these recipes will not take effect until the
next chef run.

And that would be the dealbreaker for me.

But: you could probably combine both approaches, i.e. using
'include_recipe' but then do something like this at the end of the chef run:

ruby_block "get_seen_recipes" do
block do
node.override["knife_audit"]["seen_recipes"] =
node.run_state[:seen_recipes]
node.run_list.clear
node.run_state[:seen_recipes].each do { ¦r¦ node.run_list.add r }
end
action :create
end

Are there any reasons against such an approach? For sure, searches for
"recipe:*" would only work after the first chef run of the node, but that
would not be a big deal.

Oh wait - this is borken :frowning:

=> if you remove an include_recipe or if that include_recipe is skipped
due to a conditional, it would require two chef runs to become effective -
ugh :-/

The second difference is that with include_recipe, you have more
control over order-of-execution. In my mind, that's actually a benefit;
include_recipe tempts you to think procedurally instead of declaratively.

A third difference is that I don't need to declare all the cookbooks
as dependency in my top-level cookbook. With include_recipe, I may have
dependencies on many cookbooks that aren't even relevant to a particular
server. It makes it easier to have a single top-level cookbook for all my
servers and use a big case statement to distinguish the servers.

Of course you can still declare a dependency in the metadata if you
need it to lock down a version number. You just don't have to do it for
all your recipes.

-----Original message-----
From: Torben Knerr ukio@gmx.de
Sent: Thursday 14th February 2013 14:21
To: chef@lists.opscode.com
Subject: [chef] Re: Re: Re: Re: Re: Role Versioning?

The only but very very big limitation of the "application cookbook" /
"role cookbook" / "top level cookbook" pattern is that due to the
include_recipes the included recipes don't show up in the run_list
anymore and thus you can't search for them (which breaks many of the
community cookbooks, e.g. apt).

On Thu, Feb 14, 2013 at 10:42 PM, Kevin Keane Subscription
subscription@kkeane.com wrote:

A third difference is that I don't need to declare all the cookbooks as
dependency in my top-level cookbook.

And that's a good thing? Interested to hear more about your thoughts
about this. I personally like having to declare those dependencies.

With include_recipe, I may have
dependencies on many cookbooks that aren't even relevant to a particular
server.

Like yum dependencies on an ubuntu box. That always bugged me as
well, but I finally just started ignoring it.

Of course you can still declare a dependency in the metadata if you need it
to lock down a version number. You just don't have to do it for all your
recipes.

The run_list.add doesn't trigger the foodcritic test, which is also a
negative I think.

Always interesting to hear approaches that are different from mine.

--Brian