Demonstrating the Power of Chef, Librarian and Vagrant

I created a chef-repo https://github.com/jeremiahsnapp/chef-repo and
documentation https://github.com/jeremiahsnapp/chef-repo/wiki that should
make it very easy to demonstrate the power of using Chef, Librarian and
Vagrant together for iterative infrastructure building and testing.

I hope this is helpful even to those who aren’t necessarily new to Chef
since one of the primary reasons I created this is to demonstrate how I
manage cookbooks using Librarian.

I’ve made some choices that are a little different than the wikis recommend
but it has really helped me be able to decouple cookbooks from my chef-repo
and more easily be able to contribute back to the community.

I describe my Librarian usage
herehttps://github.com/jeremiahsnapp/chef-repo/wiki/Chef-Cookbook-Management
.

Have fun cooking.

  • Jeremiah

Oops, the inline links didn't show up real well. So here they are again.

Documentation: https://github.com/jeremiahsnapp/chef-repo/wiki

Chef-Repo: https://github.com/jeremiahsnapp/chef-repo

Chef Cookbook Management with Librarian:

nice writeup Jeremiah!

When you add a new cookbook to your repository, do you just git clone
it into site-cookbooks/ ?

I found that if I modified a cookbook in cookbooks/ the change would
not be reflected on upload as librarian actually uploaded the same
cookbook stored under chef-repo/temp/librarian/.../
rather than the one stored in chef-repo/cookbooks which I presume is
put there for easy referencing.

Since you source your site-cookbooks/ in your Cheffile using :path,
does librarian also keep an additional copy of each cookbook in
site-cookbooks/ or only one?

On Fri, Apr 20, 2012 at 10:09 PM, Jeremiah Snapp
jeremiah.snapp@gmail.com wrote:

Oops, the inline links didn't show up real well. So here they are again.

Documentation: Home · jeremiahsnapp/chef-repo Wiki · GitHub

Chef-Repo: GitHub - jeremiahsnapp/chef-repo: A Chef repository - Designed to demonstrate the power of Chef, Librarian and Vagrant

Chef Cookbook Management with
Librarian: Chef Cookbook Management · jeremiahsnapp/chef-repo Wiki · GitHub

Hi Jeremiah!

On Fri, Apr 20, 2012 at 2:05 PM, Jeremiah Snapp
jeremiah.snapp@gmail.com wrote:

I created a chef-repo and documentation that should make it very easy to
demonstrate the power of using Chef, Librarian and Vagrant together for
iterative infrastructure building and testing.

I hope this is helpful even to those who aren't necessarily new to Chef
since one of the primary reasons I created this is to demonstrate how I
manage cookbooks using Librarian.

I've made some choices that are a little different than the wikis recommend
but it has really helped me be able to decouple cookbooks from my chef-repo
and more easily be able to contribute back to the community.

This is really cool, I was just tooling around with a similar approach
for my own personal Chef Repository! Jay is doing a great job with
Librarian.

Regarding this in the "Chef Cookbook Management" page:

"It used to be recommended that we customize cookbooks' templates,
files, recipes, etc by expecting knife cookbook upload to merge
multiple cookbook paths set in knife.rb cookbook_path (e.g. cookbooks,
site-cookbooks) but that is being deprecated."

I'd like to clarify what is being deprecated here.

The ability to have both a "cookbooks" and "site-cookbooks" (or
vendor-cookbooks, or whatever you want to name it) is not being
deprecated[0]. The ability to override a specific cookbook by
"shadowing" components in a second directory is deprecated[1]. To
illustrate with an example, suppose you had this your cookbook_path:

cookbook_path ["./cookbooks", "./site-cookbooks"]

And then you downloaded Opscode's chef-client cookbook:

> ls cookbooks/chef-client
cookbooks/chef-client/* contents printed

But you want to modify the client.rb config file and put it in site-cookbooks:

> tree site-cookbooks/chef-client
site-cookbooks/chef-client
└── templates
    └── default
        └── client.rb.erb

When uploading the chef-client cookbook, you'll get a deprecation
warning, which will be printed for each cookbook that exists in both
directories:

> knife cookbook upload chef-client
WARNING: * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

WARNING: The cookbooks: chef-client exist in multiple places in

your cookbook_path.
A composite version of these cookbooks has been compiled for uploading.

IMPORTANT: In a future version of Chef, this behavior will be

removed and you will no longer
be able to have the same version of a cookbook in multiple places
in your cookbook_path.
WARNING: The affected cookbooks are located:
chef-client:
/home/jtimberman/tmp/c/cookbooks/chef-client
/home/jtimberman/tmp/c/site-cookbooks/chef-client
WARNING: * * * * * * * * * * * * * * * * * * * * * * * * * * * * *


Uploading chef-client             [1.1.2]
Uploaded 1 cookbook.

It does upload fine because this hasn't been completely deprecated
(i.e., removed).

However, if you simply wanted to make your own "site specific"
cookbook in the site-cookbooks directory and it didn't overlap with
one in cookbooks, then it will be as normal:

> mkdir -p site-cookbooks/normal-things/recipes
> touch site-cookbooks/normal-things/recipes/default.rb
> knife cookbook upload normal-things
Uploading normal-things             [0.0.1]
Uploaded 1 cookbook.

TL;DR: you can have multiple directories in your cookbook_path and
that's fine as long as you don't duplicate a cookbook between them,
then you'll see the warning.

[0]: unless someone decided that and didn't tell me :-).
[1]: http://tickets.opscode.com/browse/CHEF-2308

--
Opscode, Inc
Joshua Timberman, Technical Program Manager
IRC, Skype, Twitter, Github: jtimberman

Thanks Joshua for the feedback. I’m sorry if my description confused
things. I’ll try to make some changes to my document to make it more clear.

My primary reason for choosing to only have a single "authoritative"
directory for cookbooks in the cookbook_path knife.rb parameter comes from
my use of Librarian for cookbook management. Rather than tell knife to
look in multiple locations for cookbooks my knife knows only one directory
chef-repo/cookbooks.

My Librarian Cheffile however knows where all the cookbooks’ source code
are whether they are on the community site, github, someone’s git server,
or on my local disk. I let Librarian pull all those cookbooks (and
dependencies) together into the untracked chef-repo/cookbooks directory.
Then I can just ‘knife cookbook upload’ the cookbooks from there.

It seems like using multiple paths in knife.rb cookbook_path while also
using Librarian could potentially be problematic even after identical
cookbook merging gets completely deprecated. For example, let’s say you
have modified the Perl cookbook in chef-repo/site-cookbooks and you add
Logwatch cookbook to the Librarian Cheffile. Then Librarian will put
Logwatch and it’s dependency Perl into chef-repo/cookbooks because
Librarian isn’t aware of your modified Perl cookbook in
chef-repo/site-cookbooks. Now knife’s cookbook_path will be aware of two
Perl cookbooks. I don’t know what would happen when you try to upload
these cookbooks after cookbook merging is completely deprecated. I suppose
knife would have an order of precedence based on the order of paths in
cookbook_path.

I would avoid the problem by using ‘:path’ to source the
site-cookbooks/perl cookbook in Librarian so it is aware of it. But then
why not do that for all the cookbooks in chef-repo/site-cookbooks which
makes having site-cookbooks unnecessary in knife’s cookbook_path.

I’d rather avoid the complexity by relying on Librarian to get all the
cookbooks and put them in one location and make knife aware of that one
location.

On Sat, Apr 21, 2012 at 6:06 AM, Bryan Berry bryan.berry@gmail.com wrote:

nice writeup Jeremiah!

Thanks Bryan. I'm glad to have something to contribute to this community.
Working with Chef has really brought some joy back into my work which is a
nice change.

When you add a new cookbook to your repository, do you just git clone
it into site-cookbooks/ ?

I prefer not to have git repos inside my chef-repo.
Either I'm just not daring enough to mess with nested git repos or I'm
smart enough to avoid them. :slight_smile:

If I'm going to clone any cookbooks then I clone them to a different
cookbooks directory just outside my chef-repo.

So it looks like this.
~/chef
└── cookbooks <--- git cloned vendor or personal
cookbooks I want to modify
use ':path' in
Librarian Cheffile so these get put into chef-repo/cookbooks
└── chef-repo
└── cookbooks <--- Librarian managed directory;
never touch or modify the cookbooks here
└── site-cookbooks <--- rare personal cookbooks that I
don't want to track in individual git repos

I rarely use my site-cookbooks directory. It's reserved for cookbooks that
more than likely no one else would benefit from. Cookbooks that are
specific to my infrastructure that I don't want/need to track in individual
git repos.

I found that if I modified a cookbook in cookbooks/ the change would

not be reflected on upload as librarian actually uploaded the same
cookbook stored under chef-repo/temp/librarian/.../
rather than the one stored in chef-repo/cookbooks which I presume is
put there for easy referencing.

It sounds like you chose to integrate knife and librarian as is
recommended in librarian's documentation. So what you are experiencing is
expected. When they are integrated knife actually doesn't upload cookbooks
from the cookbooks path but instead uses librarian to directly provide
whatever you defined in librarian's Cheffile. So by design you will get
what you asked librarian to provide even if you modify what librarian put
in your chef-repo/cookbooks directory.

I chose not to integrate knife and librarian simply because I didn't like
how knife commands were completely broken unless they were run in the same
folder that contains the librarian Cheffile. I don't personally see any
compelling reason for integration as long as I am willing to be responsible
for making sure my chef-repo/cookbooks directory has what is defined in
Cheffile by running 'librarian-chef clean && librarian-chef install'.

Since you source your site-cookbooks/ in your Cheffile using :path,
does librarian also keep an additional copy of each cookbook in
site-cookbooks/ or only one?

No. I hope my use of the word "source" isn't confusing things. I was
simply using it because that's what Librarian's documentation uses and it
seems to make sense. Source just refers to the home location of a
cookbook. Use ':git' if the cookbook lives in a git repo. Use ':path' if
the cookbook lives on your local disk. Librarian retrieves the cookbooks
from their "source" and puts all of them in Librarian's tmp cache directory
which is transparent during normal Librarian use. Then Librarian puts only
one "copy" of each cookbook into the chef-repo/cookbooks directory.

It occurs to me that I may be confusing things by using the name
site-cookbooks for the directory that actually only holds rare,
infrastructure specific cookbooks rather than vendor cookbooks. Would you
suggest I use a different name? Any ideas?

I will update my write up soon to try to clear these things up. It might be
a couple of days from now though since my wife is five days over due with
our second son. :slight_smile:

Hi Jeremiah,

thanks for writing this up! I was basically considering the same approach,
i.e. moving my cookbooks outside of the chef-repo so I can track them in
their individual git repos without having nested git repos in my chef-repo.
I also came across librarian and had something similar in my head, but I
didn't put it into practice yet...

Without ever having tried it out, how does librarian handle dependency
conflicts? Consider the following cookbook dependencies:
foo v1.0 -> baz v1.0
bar v1.0 -> baz 2.0

I believe chef-server can host multiple versions of the same cookbook, but
how does librarian handle this situation? Will it add the version number to
the cookbooks it creates inside chef-repo/cookbooks?

Another question that maybe someone from OpsCode can answer: will knife cookbook site install be deprecated or remain as-is now that the OpsCode
cookbooks are organized in individual git repos? As Jeremiah already
mentioned in his writeup there seems to be an impedance mismatch as you
"lose" the source / origin of the cookbooks with the knife cookbook site install approach.

Cheers,
Torben

On Sun, Apr 22, 2012 at 2:25 PM, Jeremiah Snapp jeremiah.snapp@gmail.comwrote:

It occurs to me that I may be confusing things by using the name
site-cookbooks for the directory that actually only holds rare,
infrastructure specific cookbooks rather than vendor cookbooks. Would you
suggest I use a different name? Any ideas?

I will update my write up soon to try to clear these things up. It might
be a couple of days from now though since my wife is five days over due
with our second son. :slight_smile:

Torben,

Librarian-Chef has a stricter set of constraints than Chef.

  • Librarian-Chef does not let you have two versions of one cookbook at
    the same time in your repository.
  • Librarian-Chef does not let you have dependency conflicts among
    cookbooks in your repository.

Cheers,
Jay

On Mon, Apr 23, 2012 at 2:23 AM, Torben Knerr ukio@gmx.de wrote:

Hi Jeremiah,

thanks for writing this up! I was basically considering the same approach,
i.e. moving my cookbooks outside of the chef-repo so I can track them in
their individual git repos without having nested git repos in my chef-repo.
I also came across librarian and had something similar in my head, but I
didn't put it into practice yet...

Without ever having tried it out, how does librarian handle dependency
conflicts? Consider the following cookbook dependencies:
foo v1.0 -> baz v1.0
bar v1.0 -> baz 2.0

I believe chef-server can host multiple versions of the same cookbook, but
how does librarian handle this situation? Will it add the version number to
the cookbooks it creates inside chef-repo/cookbooks?

Another question that maybe someone from OpsCode can answer: will knife cookbook site install be deprecated or remain as-is now that the OpsCode
cookbooks are organized in individual git repos? As Jeremiah already
mentioned in his writeup there seems to be an impedance mismatch as you
"lose" the source / origin of the cookbooks with the knife cookbook site install approach.

Cheers,
Torben

On Sun, Apr 22, 2012 at 2:25 PM, Jeremiah Snapp jeremiah.snapp@gmail.comwrote:

It occurs to me that I may be confusing things by using the name
site-cookbooks for the directory that actually only holds rare,
infrastructure specific cookbooks rather than vendor cookbooks. Would you
suggest I use a different name? Any ideas?

I will update my write up soon to try to clear these things up. It might
be a couple of days from now though since my wife is five days over due
with our second son. :slight_smile:

Hi Jay,

thanks for the quick reply!

I see the value in not having the same cookbook in different versions, but
it is also very restricting and I also see some cases where you might need
that.

In the example above, consider that “foo” and “bar” are well tested
cookbooks not owned by me. As long as I use them independently (i.e. no
node has both of the cookbooks) that dependency to different versions of
"baz" is totally fine, and would become a conflict only when I used both
cookbooks on the same node, isn’t it so?

Now to resolve the conflict, I would have to patch “foo” to be compatible
with “baz” 2.0, which might not be an easy task and would potentially break
it, so I’d rather not touch that thing!

I’m wondering: shouldn’t librarian-chef be more liberal wrt to defining
dependencies to different versions of the same cookbook, and on the other
hand the chef server (or knife) be more strict wrt to configuring a node
with conflicting dependencies?

Cheers,
Torben

Torben,

Fantastic question. You're describing the standard way people might use
Chef, which is "Infrastructure as Code." This is what Opscode documents in
their tutorials and in their Wiki.

But the purpose of Librarian-Chef is to help people use Chef in a subtly
different, stricter way, which I call "Infrastructure as Application." The
difference is that, in an application, you don't just have a
loosely-related set of modules in source control. You have a single
integrated codebase in source control, which requires that you have a
single integrated set of dependencies, with their exact versions and
sources, documented in source control. This results in every node getting
the exact same cookbooks as are listed in source control each and every run.

Librarian-Chef is not a tool to help fetch cookbooks. It is a tool to help
you automatically enforce a policy of having a single integrated codebase
in source control with a single integrated set of dependencies. As part of
that enforcing that policy, it provides useful services: it can fetch
cookbooks, it can tell you which cookbooks are outdated, it can update
cookbooks one-by-one and all-at-once, and it can hook into knife. There are
some things it does not do, such as enforce that the resolved cookbooks are
synced to your chef-server (although boundary-knife-plugins can help
there). This is what Bundler does for the gems used by any Ruby application.

This policy will probably be acceptable to most people most of the time,
since most people most of the time will not have a strong need to have two
versions of the same cookbook in their infrastructure repository at the
same time. Some people may even agree with the policy, and Librarian-Chef
can help them enforce it. If you are faced with a particular
incompatibility, then you can certainly fork one of the cookbooks in
question and change it to be compatible, so that you can end up with a
single integrated codebase with a single integrated set of dependencies.

If what you need is simply a tool to help fetch cookbooks, without
enforcing the "Infrastructure as Application" policy, there are a number of
others tools available that do that.

On Mon, Apr 23, 2012 at 12:45 PM, Torben Knerr ukio@gmx.de wrote:

Hi Jay,

thanks for the quick reply!

I see the value in not having the same cookbook in different versions, but
it is also very restricting and I also see some cases where you might need
that.

In the example above, consider that "foo" and "bar" are well tested
cookbooks not owned by me. As long as I use them independently (i.e. no
node has both of the cookbooks) that dependency to different versions of
"baz" is totally fine, and would become a conflict only when I used both
cookbooks on the same node, isn't it so?

Now to resolve the conflict, I would have to patch "foo" to be compatible
with "baz" 2.0, which might not be an easy task and would potentially break
it, so I'd rather not touch that thing!

I'm wondering: shouldn't librarian-chef be more liberal wrt to defining
dependencies to different versions of the same cookbook, and on the other
hand the chef server (or knife) be more strict wrt to configuring a node
with conflicting dependencies?

Cheers,
Torben

Am 23.04.2012 19:40 schrieb "Jay Feldblum" y_feldblum@yahoo.com:

Torben,

Fantastic question. You're describing the standard way people might use
Chef, which is "Infrastructure as Code." This is what Opscode documents in
their tutorials and in their Wiki.

But the purpose of Librarian-Chef is to help people use Chef in a subtly
different, stricter way, which I call "Infrastructure as Application." The
difference is that, in an application, you don't just have a
loosely-related set of modules in source control. You have a single
integrated codebase in source control, which requires that you have a
single integrated set of dependencies, with their exact versions and
sources, documented in source control. This results in every node getting
the exact same cookbooks as are listed in source control each and every run.

Librarian-Chef is not a tool to help fetch cookbooks. It is a tool to
help you automatically enforce a policy of having a single integrated
codebase in source control with a single integrated set of dependencies. As
part of that enforcing that policy, it provides useful services: it can
fetch cookbooks, it can tell you which cookbooks are outdated, it can
update cookbooks one-by-one and all-at-once, and it can hook into knife.
There are some things it does not do, such as enforce that the resolved
cookbooks are synced to your chef-server (although boundary-knife-plugins
can help there). This is what Bundler does for the gems used by any Ruby
application.

This policy will probably be acceptable to most people most of the time,
since most people most of the time will not have a strong need to have two
versions of the same cookbook in their infrastructure repository at the
same time. Some people may even agree with the policy, and Librarian-Chef
can help them enforce it. If you are faced with a particular
incompatibility, then you can certainly fork one of the cookbooks in
question and change it to be compatible, so that you can end up with a
single integrated codebase with a single integrated set of dependencies.

If what you need is simply a tool to help fetch cookbooks, without
enforcing the "Infrastructure as Application" policy, there are a number of
others tools available that do that.

Thanks for the explanation, the "single integrated set of dependencies"
policy was not clear to me, I saw it rather as a generic dependency
management solution for cookbooks (like maven for cookbooks if you will).

My requirements would be:

  • keep my cookbooks in their own git repos outside the chef-repo (don't
    like nested git repos)
  • pull in 3rd party cookbooks as dependencies (via git, opscode api,
    etc...) in a declarative manner (like Cheffile)
  • allow multiple versions of the same cookbook in the chef-repo
  • detect and handle dependency version conflicts when putting together the
    run_list (does chef do this already???)

Except for the third point librarian-chef supports this already, and it
would probably be easy to fork librarian for that.

But the more I am thinking about it the more I believe that either
a) I am doing something wrong (against the intended use of chef), or
b) this should be an integral part of chef itself

Help, I'm confused... :slight_smile:

Torben

On Mon, Apr 23, 2012 at 12:45 PM, Torben Knerr ukio@gmx.de wrote:

Hi Jay,

thanks for the quick reply!

I see the value in not having the same cookbook in different versions,
but it is also very restricting and I also see some cases where you might
need that.

In the example above, consider that "foo" and "bar" are well tested
cookbooks not owned by me. As long as I use them independently (i.e. no
node has both of the cookbooks) that dependency to different versions of
"baz" is totally fine, and would become a conflict only when I used both
cookbooks on the same node, isn't it so?

Now to resolve the conflict, I would have to patch "foo" to be
compatible with "baz" 2.0, which might not be an easy task and would
potentially break it, so I'd rather not touch that thing!

I'm wondering: shouldn't librarian-chef be more liberal wrt to defining
dependencies to different versions of the same cookbook, and on the other
hand the chef server (or knife) be more strict wrt to configuring a node
with conflicting dependencies?

Cheers,
Torben

Torben,

  • I'm not sure how to handle multiple versions of the same cookbook in the
    same git repository.

  • The chef-server has a dependency resolver which will put together
    consistent run-lists based on the cookbooks, including multiple versions of
    the same cookbook, that have been uploaded to the chef-server.

Chef supports a wide variety of ways to use it. Uploading multiple versions
of the same cookbook seems to be one of the intended ways to use it. In
that case, you would be relying on the chef-server to deliver a consistent
run-list to each node.

Cheers,
Jay

On Tue, Apr 24, 2012 at 2:11 AM, Torben Knerr ukio@gmx.de wrote:

Am 23.04.2012 19:40 schrieb "Jay Feldblum" y_feldblum@yahoo.com:

Torben,

Fantastic question. You're describing the standard way people might use
Chef, which is "Infrastructure as Code." This is what Opscode documents in
their tutorials and in their Wiki.

But the purpose of Librarian-Chef is to help people use Chef in a subtly
different, stricter way, which I call "Infrastructure as Application." The
difference is that, in an application, you don't just have a
loosely-related set of modules in source control. You have a single
integrated codebase in source control, which requires that you have a
single integrated set of dependencies, with their exact versions and
sources, documented in source control. This results in every node getting
the exact same cookbooks as are listed in source control each and every run.

Librarian-Chef is not a tool to help fetch cookbooks. It is a tool to
help you automatically enforce a policy of having a single integrated
codebase in source control with a single integrated set of dependencies. As
part of that enforcing that policy, it provides useful services: it can
fetch cookbooks, it can tell you which cookbooks are outdated, it can
update cookbooks one-by-one and all-at-once, and it can hook into knife.
There are some things it does not do, such as enforce that the resolved
cookbooks are synced to your chef-server (although boundary-knife-plugins
can help there). This is what Bundler does for the gems used by any Ruby
application.

This policy will probably be acceptable to most people most of the time,
since most people most of the time will not have a strong need to have two
versions of the same cookbook in their infrastructure repository at the
same time. Some people may even agree with the policy, and Librarian-Chef
can help them enforce it. If you are faced with a particular
incompatibility, then you can certainly fork one of the cookbooks in
question and change it to be compatible, so that you can end up with a
single integrated codebase with a single integrated set of dependencies.

If what you need is simply a tool to help fetch cookbooks, without
enforcing the "Infrastructure as Application" policy, there are a number of
others tools available that do that.

Thanks for the explanation, the "single integrated set of dependencies"
policy was not clear to me, I saw it rather as a generic dependency
management solution for cookbooks (like maven for cookbooks if you will).

My requirements would be:

  • keep my cookbooks in their own git repos outside the chef-repo (don't
    like nested git repos)
  • pull in 3rd party cookbooks as dependencies (via git, opscode api,
    etc...) in a declarative manner (like Cheffile)
  • allow multiple versions of the same cookbook in the chef-repo
  • detect and handle dependency version conflicts when putting together the
    run_list (does chef do this already???)

Except for the third point librarian-chef supports this already, and it
would probably be easy to fork librarian for that.

But the more I am thinking about it the more I believe that either
a) I am doing something wrong (against the intended use of chef), or
b) this should be an integral part of chef itself

Help, I'm confused... :slight_smile:

Torben

On Mon, Apr 23, 2012 at 12:45 PM, Torben Knerr ukio@gmx.de wrote:

Hi Jay,

thanks for the quick reply!

I see the value in not having the same cookbook in different versions,
but it is also very restricting and I also see some cases where you might
need that.

In the example above, consider that "foo" and "bar" are well tested
cookbooks not owned by me. As long as I use them independently (i.e. no
node has both of the cookbooks) that dependency to different versions of
"baz" is totally fine, and would become a conflict only when I used both
cookbooks on the same node, isn't it so?

Now to resolve the conflict, I would have to patch "foo" to be
compatible with "baz" 2.0, which might not be an easy task and would
potentially break it, so I'd rather not touch that thing!

I'm wondering: shouldn't librarian-chef be more liberal wrt to defining
dependencies to different versions of the same cookbook, and on the other
hand the chef server (or knife) be more strict wrt to configuring a node
with conflicting dependencies?

Cheers,
Torben

On Tue, Apr 24, 2012 at 12:22 PM, Jay Feldblum y_feldblum@yahoo.com wrote:

Torben,

  • I'm not sure how to handle multiple versions of the same cookbook in the
    same git repository.

My idea was to use tags, as in:

cookbook "rvm",
:git => "GitHub - sous-chefs/rvm: Development repository for the rvm cookbook",
:ref => "v0.7.1"

  • The chef-server has a dependency resolver which will put together
    consistent run-lists based on the cookbooks, including multiple versions of
    the same cookbook, that have been uploaded to the chef-server.

I assumed that, but didn't know for sure. Can you point me to any
documentation about it? I would be especially interested if you could make
it fail if it detects a dependency version conflict in a run_list...

Cheers,
Torben

Chef supports a wide variety of ways to use it. Uploading multiple
versions of the same cookbook seems to be one of the intended ways to use
it. In that case, you would be relying on the chef-server to deliver a
consistent run-list to each node.

Cheers,
Jay

On Tue, Apr 24, 2012 at 2:11 AM, Torben Knerr ukio@gmx.de wrote:

Am 23.04.2012 19:40 schrieb "Jay Feldblum" y_feldblum@yahoo.com:

Torben,

Fantastic question. You're describing the standard way people might use
Chef, which is "Infrastructure as Code." This is what Opscode documents in
their tutorials and in their Wiki.

But the purpose of Librarian-Chef is to help people use Chef in a
subtly different, stricter way, which I call "Infrastructure as
Application." The difference is that, in an application, you don't just
have a loosely-related set of modules in source control. You have a single
integrated codebase in source control, which requires that you have a
single integrated set of dependencies, with their exact versions and
sources, documented in source control. This results in every node getting
the exact same cookbooks as are listed in source control each and every run.

Librarian-Chef is not a tool to help fetch cookbooks. It is a tool to
help you automatically enforce a policy of having a single integrated
codebase in source control with a single integrated set of dependencies. As
part of that enforcing that policy, it provides useful services: it can
fetch cookbooks, it can tell you which cookbooks are outdated, it can
update cookbooks one-by-one and all-at-once, and it can hook into knife.
There are some things it does not do, such as enforce that the resolved
cookbooks are synced to your chef-server (although boundary-knife-plugins
can help there). This is what Bundler does for the gems used by any Ruby
application.

This policy will probably be acceptable to most people most of the
time, since most people most of the time will not have a strong need to
have two versions of the same cookbook in their infrastructure repository
at the same time. Some people may even agree with the policy, and
Librarian-Chef can help them enforce it. If you are faced with a particular
incompatibility, then you can certainly fork one of the cookbooks in
question and change it to be compatible, so that you can end up with a
single integrated codebase with a single integrated set of dependencies.

If what you need is simply a tool to help fetch cookbooks, without
enforcing the "Infrastructure as Application" policy, there are a number of
others tools available that do that.

Thanks for the explanation, the "single integrated set of dependencies"
policy was not clear to me, I saw it rather as a generic dependency
management solution for cookbooks (like maven for cookbooks if you will).

My requirements would be:

  • keep my cookbooks in their own git repos outside the chef-repo (don't
    like nested git repos)
  • pull in 3rd party cookbooks as dependencies (via git, opscode api,
    etc...) in a declarative manner (like Cheffile)
  • allow multiple versions of the same cookbook in the chef-repo
  • detect and handle dependency version conflicts when putting together
    the run_list (does chef do this already???)

Except for the third point librarian-chef supports this already, and it
would probably be easy to fork librarian for that.

But the more I am thinking about it the more I believe that either
a) I am doing something wrong (against the intended use of chef), or
b) this should be an integral part of chef itself

Help, I'm confused... :slight_smile:

Torben

On Mon, Apr 23, 2012 at 12:45 PM, Torben Knerr ukio@gmx.de wrote:

Hi Jay,

thanks for the quick reply!

I see the value in not having the same cookbook in different versions,
but it is also very restricting and I also see some cases where you might
need that.

In the example above, consider that "foo" and "bar" are well tested
cookbooks not owned by me. As long as I use them independently (i.e. no
node has both of the cookbooks) that dependency to different versions of
"baz" is totally fine, and would become a conflict only when I used both
cookbooks on the same node, isn't it so?

Now to resolve the conflict, I would have to patch "foo" to be
compatible with "baz" 2.0, which might not be an easy task and would
potentially break it, so I'd rather not touch that thing!

I'm wondering: shouldn't librarian-chef be more liberal wrt to
defining dependencies to different versions of the same cookbook, and on
the other hand the chef server (or knife) be more strict wrt to configuring
a node with conflicting dependencies?

Cheers,
Torben

Torben,

When a chef-client running on a node starts a run, it asks the chef-server
for the cookbooks applicable to that node and downloads them. The
chef-server looks at the node's run-list, expands the node's run-list, and
does a full dependency-resolution for the cookbooks listed in the expanded
run-list against all the cookbooks that have been uploaded to the
chef-server. The chef-client tries only to download the specific cookbooks
it actually will need, and only those files from those cookbooks that it
will actually need.

Cheers,
Jay

On Tue, Apr 24, 2012 at 6:03 PM, Torben Knerr ukio@gmx.de wrote:

On Tue, Apr 24, 2012 at 12:22 PM, Jay Feldblum y_feldblum@yahoo.comwrote:

Torben,

  • I'm not sure how to handle multiple versions of the same cookbook in
    the same git repository.

My idea was to use tags, as in:

cookbook "rvm",
:git => "GitHub - sous-chefs/rvm: Development repository for the rvm cookbook",
:ref => "v0.7.1"

  • The chef-server has a dependency resolver which will put together
    consistent run-lists based on the cookbooks, including multiple versions of
    the same cookbook, that have been uploaded to the chef-server.

I assumed that, but didn't know for sure. Can you point me to any
documentation about it? I would be especially interested if you could make
it fail if it detects a dependency version conflict in a run_list...

Cheers,
Torben

Chef supports a wide variety of ways to use it. Uploading multiple
versions of the same cookbook seems to be one of the intended ways to use
it. In that case, you would be relying on the chef-server to deliver a
consistent run-list to each node.

Cheers,
Jay

On Tue, Apr 24, 2012 at 2:11 AM, Torben Knerr ukio@gmx.de wrote:

Am 23.04.2012 19:40 schrieb "Jay Feldblum" y_feldblum@yahoo.com:

Torben,

Fantastic question. You're describing the standard way people might
use Chef, which is "Infrastructure as Code." This is what Opscode documents
in their tutorials and in their Wiki.

But the purpose of Librarian-Chef is to help people use Chef in a
subtly different, stricter way, which I call "Infrastructure as
Application." The difference is that, in an application, you don't just
have a loosely-related set of modules in source control. You have a single
integrated codebase in source control, which requires that you have a
single integrated set of dependencies, with their exact versions and
sources, documented in source control. This results in every node getting
the exact same cookbooks as are listed in source control each and every run.

Librarian-Chef is not a tool to help fetch cookbooks. It is a tool to
help you automatically enforce a policy of having a single integrated
codebase in source control with a single integrated set of dependencies. As
part of that enforcing that policy, it provides useful services: it can
fetch cookbooks, it can tell you which cookbooks are outdated, it can
update cookbooks one-by-one and all-at-once, and it can hook into knife.
There are some things it does not do, such as enforce that the resolved
cookbooks are synced to your chef-server (although boundary-knife-plugins
can help there). This is what Bundler does for the gems used by any Ruby
application.

This policy will probably be acceptable to most people most of the
time, since most people most of the time will not have a strong need to
have two versions of the same cookbook in their infrastructure repository
at the same time. Some people may even agree with the policy, and
Librarian-Chef can help them enforce it. If you are faced with a particular
incompatibility, then you can certainly fork one of the cookbooks in
question and change it to be compatible, so that you can end up with a
single integrated codebase with a single integrated set of dependencies.

If what you need is simply a tool to help fetch cookbooks, without
enforcing the "Infrastructure as Application" policy, there are a number of
others tools available that do that.

Thanks for the explanation, the "single integrated set of dependencies"
policy was not clear to me, I saw it rather as a generic dependency
management solution for cookbooks (like maven for cookbooks if you will).

My requirements would be:

  • keep my cookbooks in their own git repos outside the chef-repo (don't
    like nested git repos)
  • pull in 3rd party cookbooks as dependencies (via git, opscode api,
    etc...) in a declarative manner (like Cheffile)
  • allow multiple versions of the same cookbook in the chef-repo
  • detect and handle dependency version conflicts when putting together
    the run_list (does chef do this already???)

Except for the third point librarian-chef supports this already, and it
would probably be easy to fork librarian for that.

But the more I am thinking about it the more I believe that either
a) I am doing something wrong (against the intended use of chef), or
b) this should be an integral part of chef itself

Help, I'm confused... :slight_smile:

Torben

On Mon, Apr 23, 2012 at 12:45 PM, Torben Knerr ukio@gmx.de wrote:

Hi Jay,

thanks for the quick reply!

I see the value in not having the same cookbook in different
versions, but it is also very restricting and I also see some cases where
you might need that.

In the example above, consider that "foo" and "bar" are well tested
cookbooks not owned by me. As long as I use them independently (i.e. no
node has both of the cookbooks) that dependency to different versions of
"baz" is totally fine, and would become a conflict only when I used both
cookbooks on the same node, isn't it so?

Now to resolve the conflict, I would have to patch "foo" to be
compatible with "baz" 2.0, which might not be an easy task and would
potentially break it, so I'd rather not touch that thing!

I'm wondering: shouldn't librarian-chef be more liberal wrt to
defining dependencies to different versions of the same cookbook, and on
the other hand the chef server (or knife) be more strict wrt to configuring
a node with conflicting dependencies?

Cheers,
Torben