# Demonstrating the Power of Chef, Librarian and Vagrant

**URL:** <https://discourse.chef.io/t/demonstrating-the-power-of-chef-librarian-and-vagrant/2703>\
**Category:** Chef Infra (archive)\
**Created:** [April 20, 2012, 8:05pm UTC](https://discourse.chef.io/t/demonstrating-the-power-of-chef-librarian-and-vagrant/2703 "2012-04-20T20:05:59Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jeremiah\_Snapp](https://avatars.discourse-cdn.com/v4/letter/j/3bc359/32.png) [@Jeremiah\_Snapp](https://discourse.chef.io/u/Jeremiah_Snapp)\
**Post date:** [April 20, 2012, 8:05pm UTC](https://discourse.chef.io/t/demonstrating-the-power-of-chef-librarian-and-vagrant/2703/1 "2012-04-20T20:05:59Z")

</div>

I created a chef-repo [https://github.com/jeremiahsnapp/chef-repo](https://github.com/jeremiahsnapp/chef-repo) and  
documentation [https://github.com/jeremiahsnapp/chef-repo/wiki](https://github.com/jeremiahsnapp/chef-repo/wiki) that should  
make it very easy to demonstrate the power of using Chef, Librarian and  
Vagrant together for iterative infrastructure building and testing.

I hope this is helpful even to those who aren’t necessarily new to Chef  
since one of the primary reasons I created this is to demonstrate how I  
manage cookbooks using Librarian.

I’ve made some choices that are a little different than the wikis recommend  
but it has really helped me be able to decouple cookbooks from my chef-repo  
and more easily be able to contribute back to the community.

I describe my Librarian usage  
here[https://github.com/jeremiahsnapp/chef-repo/wiki/Chef-Cookbook-Management](https://github.com/jeremiahsnapp/chef-repo/wiki/Chef-Cookbook-Management)  
.

Have fun cooking.

- Jeremiah

---

<div class="post-metadata">

**Author:** ![Jeremiah\_Snapp](https://avatars.discourse-cdn.com/v4/letter/j/3bc359/32.png) [@Jeremiah\_Snapp](https://discourse.chef.io/u/Jeremiah_Snapp)\
**Post date:** [April 20, 2012, 8:09pm UTC](https://discourse.chef.io/t/demonstrating-the-power-of-chef-librarian-and-vagrant/2703/2 "2012-04-20T20:09:38Z")

</div>

Oops, the inline links didn't show up real well. So here they are again.

Documentation: [https://github.com/jeremiahsnapp/chef-repo/wiki](https://github.com/jeremiahsnapp/chef-repo/wiki)

Chef-Repo: [https://github.com/jeremiahsnapp/chef-repo](https://github.com/jeremiahsnapp/chef-repo)

Chef Cookbook Management with Librarian:

> **[jeremiahsnapp/chef-repo](https://github.com/jeremiahsnapp/chef-repo/wiki/Chef-Cookbook-Management)**
>
> A Chef repository - Designed to demonstrate the power of Chef, Librarian and Vagrant - jeremiahsnapp/chef-repo

---

<div class="post-metadata">

**Author:** ![Bryan\_Berry](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/bryan_berry/32/213_2.png) [@Bryan\_Berry](https://discourse.chef.io/u/Bryan_Berry)\
**Post date:** [April 21, 2012, 10:06am UTC](https://discourse.chef.io/t/demonstrating-the-power-of-chef-librarian-and-vagrant/2703/3 "2012-04-21T10:06:40Z")

</div>

nice writeup Jeremiah!

When you add a new cookbook to your repository, do you just git clone  
it into site-cookbooks/ ?

I found that if I modified a cookbook in cookbooks/ the change would  
not be reflected on upload as librarian actually uploaded the same  
cookbook stored under chef-repo/temp/librarian/.../  
rather than the one stored in chef-repo/cookbooks which I presume is  
put there for easy referencing.

Since you source your site-cookbooks/ in your Cheffile using :path,  
does librarian also keep an additional copy of each cookbook in  
site-cookbooks/ or only one?

On Fri, Apr 20, 2012 at 10:09 PM, Jeremiah Snapp  
[jeremiah.snapp@gmail.com](mailto:jeremiah.snapp@gmail.com) wrote:

> Oops, the inline links didn't show up real well. So here they are again.
> 
> Documentation: [Home · jeremiahsnapp/chef-repo Wiki · GitHub](https://github.com/jeremiahsnapp/chef-repo/wiki)
> 
> Chef-Repo: [GitHub - jeremiahsnapp/chef-repo: A Chef repository - Designed to demonstrate the power of Chef, Librarian and Vagrant](https://github.com/jeremiahsnapp/chef-repo)
> 
> Chef Cookbook Management with  
> Librarian: [Chef Cookbook Management · jeremiahsnapp/chef-repo Wiki · GitHub](https://github.com/jeremiahsnapp/chef-repo/wiki/Chef-Cookbook-Management)

---

<div class="post-metadata">

**Author:** ![Joshua\_Timberman](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/joshua_timberman/32/424_2.png) [@Joshua\_Timberman](https://discourse.chef.io/u/Joshua_Timberman)\
**Post date:** [April 21, 2012, 7:36pm UTC](https://discourse.chef.io/t/demonstrating-the-power-of-chef-librarian-and-vagrant/2703/4 "2012-04-21T19:36:09Z")

</div>

Hi Jeremiah!

On Fri, Apr 20, 2012 at 2:05 PM, Jeremiah Snapp  
[jeremiah.snapp@gmail.com](mailto:jeremiah.snapp@gmail.com) wrote:

> I created a chef-repo and documentation that should make it very easy to  
> demonstrate the power of using Chef, Librarian and Vagrant together for  
> iterative infrastructure building and testing.
> 
> I hope this is helpful even to those who aren't necessarily new to Chef  
> since one of the primary reasons I created this is to demonstrate how I  
> manage cookbooks using Librarian.
> 
> I've made some choices that are a little different than the wikis recommend  
> but it has really helped me be able to decouple cookbooks from my chef-repo  
> and more easily be able to contribute back to the community.

This is really cool, I was just tooling around with a similar approach  
for my own personal Chef Repository! Jay is doing a great job with  
Librarian.

Regarding this in the "Chef Cookbook Management" page:

"It used to be recommended that we customize cookbooks' templates,  
files, recipes, etc by expecting knife cookbook upload to merge  
multiple cookbook paths set in knife.rb cookbook\_path (e.g. cookbooks,  
site-cookbooks) but that is being deprecated."

I'd like to clarify what is being deprecated here.

The ability to have both a "cookbooks" and "site-cookbooks" (or  
vendor-cookbooks, or whatever you want to name it) _is not_ being  
deprecated[0]. The ability to override a specific cookbook by  
"shadowing" components in a second directory _is_ deprecated[1]. To  
illustrate with an example, suppose you had this your cookbook\_path:

```
cookbook_path ["./cookbooks", "./site-cookbooks"]

```

And then you downloaded Opscode's chef-client cookbook:

```
> ls cookbooks/chef-client
cookbooks/chef-client/* contents printed

```

But you want to modify the client.rb config file and put it in site-cookbooks:

```
> tree site-cookbooks/chef-client
site-cookbooks/chef-client
└── templates
    └── default
        └── client.rb.erb

```

When uploading the chef-client cookbook, you'll get a deprecation  
warning, which will be printed for each cookbook that exists in both  
directories:

```
> knife cookbook upload chef-client
WARNING: * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

```

* * *

```
WARNING: The cookbooks: chef-client exist in multiple places in

```

your cookbook\_path.  
A composite version of these cookbooks has been compiled for uploading.

```
IMPORTANT: In a future version of Chef, this behavior will be

```

removed and you will no longer  
be able to have the same version of a cookbook in multiple places  
in your cookbook\_path.  
WARNING: The affected cookbooks are located:  
chef-client:  
/home/jtimberman/tmp/c/cookbooks/chef-client  
/home/jtimberman/tmp/c/site-cookbooks/chef-client  
WARNING: \* \* \* \* \* \* \* \* \* \* \* \* \* \* \* \* \* \* \* \* \* \* \* \* \* \* \* \* \*

* * *

```
Uploading chef-client [1.1.2]
Uploaded 1 cookbook.

```

It does upload fine because this hasn't been completely deprecated  
(i.e., removed).

However, if you simply wanted to make your own "site specific"  
cookbook in the site-cookbooks directory and it didn't overlap with  
one in cookbooks, then it will be as normal:

```
> mkdir -p site-cookbooks/normal-things/recipes
> touch site-cookbooks/normal-things/recipes/default.rb
> knife cookbook upload normal-things
Uploading normal-things [0.0.1]
Uploaded 1 cookbook.

```

TL;DR: you can have multiple directories in your cookbook\_path and  
that's fine as long as you don't duplicate a cookbook between them,  
then you'll see the warning.

[0]: unless someone decided that and didn't tell me :-).  
[1]: [http://tickets.opscode.com/browse/CHEF-2308](http://tickets.opscode.com/browse/CHEF-2308)

--  
Opscode, Inc  
Joshua Timberman, Technical Program Manager  
IRC, Skype, Twitter, Github: jtimberman

---

<div class="post-metadata">

**Author:** ![Jeremiah\_Snapp](https://avatars.discourse-cdn.com/v4/letter/j/3bc359/32.png) [@Jeremiah\_Snapp](https://discourse.chef.io/u/Jeremiah_Snapp)\
**Post date:** [April 22, 2012, 4:04am UTC](https://discourse.chef.io/t/demonstrating-the-power-of-chef-librarian-and-vagrant/2703/5 "2012-04-22T04:04:24Z")

</div>

Thanks Joshua for the feedback. I’m sorry if my description confused  
things. I’ll try to make some changes to my document to make it more clear.

My primary reason for choosing to only have a single "authoritative"  
directory for cookbooks in the cookbook\_path knife.rb parameter comes from  
my use of Librarian for cookbook management. Rather than tell knife to  
look in multiple locations for cookbooks my knife knows only one directory  
chef-repo/cookbooks.

My Librarian Cheffile however knows where all the cookbooks’ source code  
are whether they are on the community site, github, someone’s git server,  
or on my local disk. I let Librarian pull all those cookbooks (and  
dependencies) together into the untracked chef-repo/cookbooks directory.  
Then I can just ‘knife cookbook upload’ the cookbooks from there.

It seems like using multiple paths in knife.rb cookbook\_path while also  
using Librarian could potentially be problematic even after identical  
cookbook merging gets completely deprecated. For example, let’s say you  
have modified the Perl cookbook in chef-repo/site-cookbooks and you add  
Logwatch cookbook to the Librarian Cheffile. Then Librarian will put  
Logwatch and it’s dependency Perl into chef-repo/cookbooks because  
Librarian isn’t aware of your modified Perl cookbook in  
chef-repo/site-cookbooks. Now knife’s cookbook\_path will be aware of two  
Perl cookbooks. I don’t know what would happen when you try to upload  
these cookbooks after cookbook merging is completely deprecated. I suppose  
knife would have an order of precedence based on the order of paths in  
cookbook\_path.

I would avoid the problem by using ‘:path’ to source the  
site-cookbooks/perl cookbook in Librarian so it is aware of it. But then  
why not do that for all the cookbooks in chef-repo/site-cookbooks which  
makes having site-cookbooks unnecessary in knife’s cookbook\_path.

I’d rather avoid the complexity by relying on Librarian to get all the  
cookbooks and put them in one location and make knife aware of that one  
location.

---

<div class="post-metadata">

**Author:** ![Jeremiah\_Snapp](https://avatars.discourse-cdn.com/v4/letter/j/3bc359/32.png) [@Jeremiah\_Snapp](https://discourse.chef.io/u/Jeremiah_Snapp)\
**Post date:** [April 22, 2012, 4:43am UTC](https://discourse.chef.io/t/demonstrating-the-power-of-chef-librarian-and-vagrant/2703/6 "2012-04-22T04:43:00Z")

</div>

On Sat, Apr 21, 2012 at 6:06 AM, Bryan Berry [bryan.berry@gmail.com](mailto:bryan.berry@gmail.com) wrote:

> nice writeup Jeremiah!
> 
> Thanks Bryan. I'm glad to have something to contribute to this community.  
> Working with Chef has really brought some joy back into my work which is a  
> nice change.

> When you add a new cookbook to your repository, do you just git clone  
> it into site-cookbooks/ ?
> 
> I prefer not to have git repos inside my chef-repo.  
> Either I'm just not daring enough to mess with nested git repos or I'm  
> smart enough to avoid them. 🙂

If I'm going to clone any cookbooks then I clone them to a different  
cookbooks directory just outside my chef-repo.

So it looks like this.  
~/chef  
└── cookbooks \<--- git cloned vendor or personal  
cookbooks I want to modify  
use ':path' in  
Librarian Cheffile so these get put into chef-repo/cookbooks  
└── chef-repo  
└── cookbooks \<--- Librarian managed directory;  
never touch or modify the cookbooks here  
└── site-cookbooks \<--- rare personal cookbooks that I  
don't want to track in individual git repos

I rarely use my site-cookbooks directory. It's reserved for cookbooks that  
more than likely no one else would benefit from. Cookbooks that are  
specific to my infrastructure that I don't want/need to track in individual  
git repos.

I found that if I modified a cookbook in cookbooks/ the change would

> not be reflected on upload as librarian actually uploaded the same  
> cookbook stored under chef-repo/temp/librarian/.../  
> rather than the one stored in chef-repo/cookbooks which I presume is  
> put there for easy referencing.
> 
> It sounds like you chose to integrate knife and librarian as is  
> recommended in librarian's documentation. So what you are experiencing is  
> expected. When they are integrated knife actually doesn't upload cookbooks  
> from the cookbooks path but instead uses librarian to directly provide  
> whatever you defined in librarian's Cheffile. So by design you will get  
> what you asked librarian to provide even if you modify what librarian put  
> in your chef-repo/cookbooks directory.

I chose not to integrate knife and librarian simply because I didn't like  
how knife commands were completely broken unless they were run in the same  
folder that contains the librarian Cheffile. I don't personally see any  
compelling reason for integration as long as I am willing to be responsible  
for making sure my chef-repo/cookbooks directory has what is defined in  
Cheffile by running 'librarian-chef clean && librarian-chef install'.

> Since you source your site-cookbooks/ in your Cheffile using :path,  
> does librarian also keep an additional copy of each cookbook in  
> site-cookbooks/ or only one?
> 
> No. I hope my use of the word "source" isn't confusing things. I was  
> simply using it because that's what Librarian's documentation uses and it  
> seems to make sense. Source just refers to the home location of a  
> cookbook. Use ':git' if the cookbook lives in a git repo. Use ':path' if  
> the cookbook lives on your local disk. Librarian retrieves the cookbooks  
> from their "source" and puts all of them in Librarian's tmp cache directory  
> which is transparent during normal Librarian use. Then Librarian puts only  
> one "copy" of each cookbook into the chef-repo/cookbooks directory.

---

<div class="post-metadata">

**Author:** ![Jeremiah\_Snapp](https://avatars.discourse-cdn.com/v4/letter/j/3bc359/32.png) [@Jeremiah\_Snapp](https://discourse.chef.io/u/Jeremiah_Snapp)\
**Post date:** [April 22, 2012, 12:25pm UTC](https://discourse.chef.io/t/demonstrating-the-power-of-chef-librarian-and-vagrant/2703/7 "2012-04-22T12:25:53Z")

</div>

It occurs to me that I may be confusing things by using the name  
site-cookbooks for the directory that actually only holds rare,  
infrastructure specific cookbooks rather than vendor cookbooks. Would you  
suggest I use a different name? Any ideas?

I will update my write up soon to try to clear these things up. It might be  
a couple of days from now though since my wife is five days over due with  
our second son. 🙂

---

<div class="post-metadata">

**Author:** ![Torben\_Knerr](https://avatars.discourse-cdn.com/v4/letter/t/ba8739/32.png) [@Torben\_Knerr](https://discourse.chef.io/u/Torben_Knerr)\
**Post date:** [April 23, 2012, 6:23am UTC](https://discourse.chef.io/t/demonstrating-the-power-of-chef-librarian-and-vagrant/2703/8 "2012-04-23T06:23:51Z")

</div>

Hi Jeremiah,

thanks for writing this up! I was basically considering the same approach,  
i.e. moving my cookbooks outside of the chef-repo so I can track them in  
their individual git repos without having nested git repos in my chef-repo.  
I also came across librarian and had something similar in my head, but I  
didn't put it into practice yet...

Without ever having tried it out, how does librarian handle dependency  
conflicts? Consider the following cookbook dependencies:  
foo v1.0 -\> baz v1.0  
bar v1.0 -\> baz 2.0

I believe chef-server can host multiple versions of the same cookbook, but  
how does librarian handle this situation? Will it add the version number to  
the cookbooks it creates inside chef-repo/cookbooks?

Another question that maybe someone from OpsCode can answer: will `knife cookbook site install` be deprecated or remain as-is now that the OpsCode  
cookbooks are organized in individual git repos? As Jeremiah already  
mentioned in his writeup there seems to be an impedance mismatch as you  
"lose" the source / origin of the cookbooks with the `knife cookbook site install` approach.

Cheers,  
Torben

On Sun, Apr 22, 2012 at 2:25 PM, Jeremiah Snapp [jeremiah.snapp@gmail.com](mailto:jeremiah.snapp@gmail.com)wrote:

> It occurs to me that I may be confusing things by using the name  
> site-cookbooks for the directory that actually only holds rare,  
> infrastructure specific cookbooks rather than vendor cookbooks. Would you  
> suggest I use a different name? Any ideas?
> 
> I will update my write up soon to try to clear these things up. It might  
> be a couple of days from now though since my wife is five days over due  
> with our second son. 🙂

---

<div class="post-metadata">

**Author:** ![Jay\_Feldblum](https://avatars.discourse-cdn.com/v4/letter/j/9fc29f/32.png) [@Jay\_Feldblum](https://discourse.chef.io/u/Jay_Feldblum)\
**Post date:** [April 23, 2012, 10:28am UTC](https://discourse.chef.io/t/demonstrating-the-power-of-chef-librarian-and-vagrant/2703/9 "2012-04-23T10:28:27Z")

</div>

Torben,

Librarian-Chef has a stricter set of constraints than Chef.

- Librarian-Chef does not let you have two versions of one cookbook at  
the same time in your repository.
- Librarian-Chef does not let you have dependency conflicts among  
cookbooks in your repository.

Cheers,  
Jay

On Mon, Apr 23, 2012 at 2:23 AM, Torben Knerr [ukio@gmx.de](mailto:ukio@gmx.de) wrote:

> Hi Jeremiah,
> 
> thanks for writing this up! I was basically considering the same approach,  
> i.e. moving my cookbooks outside of the chef-repo so I can track them in  
> their individual git repos without having nested git repos in my chef-repo.  
> I also came across librarian and had something similar in my head, but I  
> didn't put it into practice yet...
> 
> Without ever having tried it out, how does librarian handle dependency  
> conflicts? Consider the following cookbook dependencies:  
> foo v1.0 -\> baz v1.0  
> bar v1.0 -\> baz 2.0
> 
> I believe chef-server can host multiple versions of the same cookbook, but  
> how does librarian handle this situation? Will it add the version number to  
> the cookbooks it creates inside chef-repo/cookbooks?
> 
> Another question that maybe someone from OpsCode can answer: will `knife cookbook site install` be deprecated or remain as-is now that the OpsCode  
> cookbooks are organized in individual git repos? As Jeremiah already  
> mentioned in his writeup there seems to be an impedance mismatch as you  
> "lose" the source / origin of the cookbooks with the `knife cookbook site install` approach.
> 
> Cheers,  
> Torben
> 
> On Sun, Apr 22, 2012 at 2:25 PM, Jeremiah Snapp [jeremiah.snapp@gmail.com](mailto:jeremiah.snapp@gmail.com)wrote:
> 
> > It occurs to me that I may be confusing things by using the name  
> > site-cookbooks for the directory that actually only holds rare,  
> > infrastructure specific cookbooks rather than vendor cookbooks. Would you  
> > suggest I use a different name? Any ideas?
> > 
> > I will update my write up soon to try to clear these things up. It might  
> > be a couple of days from now though since my wife is five days over due  
> > with our second son. 🙂

---

<div class="post-metadata">

**Author:** ![Torben\_Knerr](https://avatars.discourse-cdn.com/v4/letter/t/ba8739/32.png) [@Torben\_Knerr](https://discourse.chef.io/u/Torben_Knerr)\
**Post date:** [April 23, 2012, 4:45pm UTC](https://discourse.chef.io/t/demonstrating-the-power-of-chef-librarian-and-vagrant/2703/10 "2012-04-23T16:45:07Z")

</div>

Hi Jay,

thanks for the quick reply!

I see the value in not having the same cookbook in different versions, but  
it is also very restricting and I also see some cases where you might need  
that.

In the example above, consider that “foo” and “bar” are well tested  
cookbooks not owned by me. As long as I use them independently (i.e. no  
node has both of the cookbooks) that dependency to different versions of  
"baz" is totally fine, and would become a conflict only when I used both  
cookbooks on the same node, isn’t it so?

Now to resolve the conflict, I would have to patch “foo” to be compatible  
with “baz” 2.0, which might not be an easy task and would potentially break  
it, so I’d rather not touch that thing!

I’m wondering: shouldn’t librarian-chef be more liberal wrt to defining  
dependencies to different versions of the same cookbook, and on the other  
hand the chef server (or knife) be more strict wrt to configuring a node  
with conflicting dependencies?

Cheers,  
Torben

---

<div class="post-metadata">

**Author:** ![Jay\_Feldblum](https://avatars.discourse-cdn.com/v4/letter/j/9fc29f/32.png) [@Jay\_Feldblum](https://discourse.chef.io/u/Jay_Feldblum)\
**Post date:** [April 23, 2012, 5:40pm UTC](https://discourse.chef.io/t/demonstrating-the-power-of-chef-librarian-and-vagrant/2703/11 "2012-04-23T17:40:03Z")

</div>

Torben,

Fantastic question. You're describing the standard way people might use  
Chef, which is "Infrastructure as Code." This is what Opscode documents in  
their tutorials and in their Wiki.

But the purpose of Librarian-Chef is to help people use Chef in a subtly  
different, stricter way, which I call "Infrastructure as Application." The  
difference is that, in an application, you don't just have a  
loosely-related set of modules in source control. You have a single  
integrated codebase in source control, which requires that you have a  
single integrated set of dependencies, with their exact versions and  
sources, documented in source control. This results in every node getting  
the exact same cookbooks as are listed in source control each and every run.

Librarian-Chef is not a tool to help fetch cookbooks. It is a tool to help  
you automatically enforce a policy of having a single integrated codebase  
in source control with a single integrated set of dependencies. As part of  
that enforcing that policy, it provides useful services: it can fetch  
cookbooks, it can tell you which cookbooks are outdated, it can update  
cookbooks one-by-one and all-at-once, and it can hook into knife. There are  
some things it does not do, such as enforce that the resolved cookbooks are  
synced to your chef-server (although boundary-knife-plugins can help  
there). This is what Bundler does for the gems used by any Ruby application.

This policy will probably be acceptable to most people most of the time,  
since most people most of the time will not have a strong need to have two  
versions of the same cookbook in their infrastructure repository at the  
same time. Some people may even agree with the policy, and Librarian-Chef  
can help them enforce it. If you are faced with a particular  
incompatibility, then you can certainly fork one of the cookbooks in  
question and change it to be compatible, so that you can end up with a  
single integrated codebase with a single integrated set of dependencies.

If what you need is simply a tool to help fetch cookbooks, without  
enforcing the "Infrastructure as Application" policy, there are a number of  
others tools available that do that.

On Mon, Apr 23, 2012 at 12:45 PM, Torben Knerr [ukio@gmx.de](mailto:ukio@gmx.de) wrote:

> Hi Jay,
> 
> thanks for the quick reply!
> 
> I see the value in not having the same cookbook in different versions, but  
> it is also very restricting and I also see some cases where you might need  
> that.
> 
> In the example above, consider that "foo" and "bar" are well tested  
> cookbooks not owned by me. As long as I use them independently (i.e. no  
> node has both of the cookbooks) that dependency to different versions of  
> "baz" is totally fine, and would become a conflict only when I used both  
> cookbooks on the same node, isn't it so?
> 
> Now to resolve the conflict, I would have to patch "foo" to be compatible  
> with "baz" 2.0, which might not be an easy task and would potentially break  
> it, so I'd rather not touch that thing!
> 
> I'm wondering: shouldn't librarian-chef be more liberal wrt to defining  
> dependencies to different versions of the same cookbook, and on the other  
> hand the chef server (or knife) be more strict wrt to configuring a node  
> with conflicting dependencies?
> 
> Cheers,  
> Torben

---

<div class="post-metadata">

**Author:** ![Torben\_Knerr](https://avatars.discourse-cdn.com/v4/letter/t/ba8739/32.png) [@Torben\_Knerr](https://discourse.chef.io/u/Torben_Knerr)\
**Post date:** [April 24, 2012, 6:11am UTC](https://discourse.chef.io/t/demonstrating-the-power-of-chef-librarian-and-vagrant/2703/12 "2012-04-24T06:11:08Z")

</div>

Am 23.04.2012 19:40 schrieb "Jay Feldblum" [y\_feldblum@yahoo.com](mailto:y_feldblum@yahoo.com):

> Torben,
> 
> Fantastic question. You're describing the standard way people might use  
> Chef, which is "Infrastructure as Code." This is what Opscode documents in  
> their tutorials and in their Wiki.
> 
> But the purpose of Librarian-Chef is to help people use Chef in a subtly  
> different, stricter way, which I call "Infrastructure as Application." The  
> difference is that, in an application, you don't just have a  
> loosely-related set of modules in source control. You have a single  
> integrated codebase in source control, which requires that you have a  
> single integrated set of dependencies, with their exact versions and  
> sources, documented in source control. This results in every node getting  
> the exact same cookbooks as are listed in source control each and every run.
> 
> Librarian-Chef is not a tool to help fetch cookbooks. It is a tool to  
> help you automatically enforce a policy of having a single integrated  
> codebase in source control with a single integrated set of dependencies. As  
> part of that enforcing that policy, it provides useful services: it can  
> fetch cookbooks, it can tell you which cookbooks are outdated, it can  
> update cookbooks one-by-one and all-at-once, and it can hook into knife.  
> There are some things it does not do, such as enforce that the resolved  
> cookbooks are synced to your chef-server (although boundary-knife-plugins  
> can help there). This is what Bundler does for the gems used by any Ruby  
> application.
> 
> This policy will probably be acceptable to most people most of the time,  
> since most people most of the time will not have a strong need to have two  
> versions of the same cookbook in their infrastructure repository at the  
> same time. Some people may even agree with the policy, and Librarian-Chef  
> can help them enforce it. If you are faced with a particular  
> incompatibility, then you can certainly fork one of the cookbooks in  
> question and change it to be compatible, so that you can end up with a  
> single integrated codebase with a single integrated set of dependencies.
> 
> If what you need is simply a tool to help fetch cookbooks, without  
> enforcing the "Infrastructure as Application" policy, there are a number of  
> others tools available that do that.

Thanks for the explanation, the "single integrated set of dependencies"  
policy was not clear to me, I saw it rather as a generic dependency  
management solution for cookbooks (like maven for cookbooks if you will).

My requirements would be:

- keep my cookbooks in their own git repos outside the chef-repo (don't  
like nested git repos)
- pull in 3rd party cookbooks as dependencies (via git, opscode api,  
etc...) in a declarative manner (like Cheffile)
- allow multiple versions of the same cookbook in the chef-repo
- detect and handle dependency version conflicts when putting together the  
run\_list (does chef do this already???)

Except for the third point librarian-chef supports this already, and it  
would probably be easy to fork librarian for that.

But the more I am thinking about it the more I believe that either  
a) I am doing something wrong (against the intended use of chef), or  
b) this should be an integral part of chef itself

Help, I'm confused... 🙂

Torben

> On Mon, Apr 23, 2012 at 12:45 PM, Torben Knerr [ukio@gmx.de](mailto:ukio@gmx.de) wrote:
> 
> > Hi Jay,
> > 
> > thanks for the quick reply!
> > 
> > I see the value in not having the same cookbook in different versions,  
> > but it is also very restricting and I also see some cases where you might  
> > need that.
> > 
> > In the example above, consider that "foo" and "bar" are well tested  
> > cookbooks not owned by me. As long as I use them independently (i.e. no  
> > node has both of the cookbooks) that dependency to different versions of  
> > "baz" is totally fine, and would become a conflict only when I used both  
> > cookbooks on the same node, isn't it so?
> > 
> > Now to resolve the conflict, I would have to patch "foo" to be  
> > compatible with "baz" 2.0, which might not be an easy task and would  
> > potentially break it, so I'd rather not touch that thing!
> > 
> > I'm wondering: shouldn't librarian-chef be more liberal wrt to defining  
> > dependencies to different versions of the same cookbook, and on the other  
> > hand the chef server (or knife) be more strict wrt to configuring a node  
> > with conflicting dependencies?
> > 
> > Cheers,  
> > Torben

---

<div class="post-metadata">

**Author:** ![Jay\_Feldblum](https://avatars.discourse-cdn.com/v4/letter/j/9fc29f/32.png) [@Jay\_Feldblum](https://discourse.chef.io/u/Jay_Feldblum)\
**Post date:** [April 24, 2012, 10:22am UTC](https://discourse.chef.io/t/demonstrating-the-power-of-chef-librarian-and-vagrant/2703/13 "2012-04-24T10:22:07Z")

</div>

Torben,

- I'm not sure how to handle multiple versions of the same cookbook in the  
same git repository.

- The chef-server has a dependency resolver which will put together  
consistent run-lists based on the cookbooks, including multiple versions of  
the same cookbook, that have been uploaded to the chef-server.

Chef supports a wide variety of ways to use it. Uploading multiple versions  
of the same cookbook seems to be one of the intended ways to use it. In  
that case, you would be relying on the chef-server to deliver a consistent  
run-list to each node.

Cheers,  
Jay

On Tue, Apr 24, 2012 at 2:11 AM, Torben Knerr [ukio@gmx.de](mailto:ukio@gmx.de) wrote:

> Am 23.04.2012 19:40 schrieb "Jay Feldblum" [y\_feldblum@yahoo.com](mailto:y_feldblum@yahoo.com):
> 
> > Torben,
> > 
> > Fantastic question. You're describing the standard way people might use  
> > Chef, which is "Infrastructure as Code." This is what Opscode documents in  
> > their tutorials and in their Wiki.
> > 
> > But the purpose of Librarian-Chef is to help people use Chef in a subtly  
> > different, stricter way, which I call "Infrastructure as Application." The  
> > difference is that, in an application, you don't just have a  
> > loosely-related set of modules in source control. You have a single  
> > integrated codebase in source control, which requires that you have a  
> > single integrated set of dependencies, with their exact versions and  
> > sources, documented in source control. This results in every node getting  
> > the exact same cookbooks as are listed in source control each and every run.
> > 
> > Librarian-Chef is not a tool to help fetch cookbooks. It is a tool to  
> > help you automatically enforce a policy of having a single integrated  
> > codebase in source control with a single integrated set of dependencies. As  
> > part of that enforcing that policy, it provides useful services: it can  
> > fetch cookbooks, it can tell you which cookbooks are outdated, it can  
> > update cookbooks one-by-one and all-at-once, and it can hook into knife.  
> > There are some things it does not do, such as enforce that the resolved  
> > cookbooks are synced to your chef-server (although boundary-knife-plugins  
> > can help there). This is what Bundler does for the gems used by any Ruby  
> > application.
> > 
> > This policy will probably be acceptable to most people most of the time,  
> > since most people most of the time will not have a strong need to have two  
> > versions of the same cookbook in their infrastructure repository at the  
> > same time. Some people may even agree with the policy, and Librarian-Chef  
> > can help them enforce it. If you are faced with a particular  
> > incompatibility, then you can certainly fork one of the cookbooks in  
> > question and change it to be compatible, so that you can end up with a  
> > single integrated codebase with a single integrated set of dependencies.
> > 
> > If what you need is simply a tool to help fetch cookbooks, without  
> > enforcing the "Infrastructure as Application" policy, there are a number of  
> > others tools available that do that.
> 
> Thanks for the explanation, the "single integrated set of dependencies"  
> policy was not clear to me, I saw it rather as a generic dependency  
> management solution for cookbooks (like maven for cookbooks if you will).
> 
> My requirements would be:
> 
> - keep my cookbooks in their own git repos outside the chef-repo (don't  
> like nested git repos)
> - pull in 3rd party cookbooks as dependencies (via git, opscode api,  
> etc...) in a declarative manner (like Cheffile)
> - allow multiple versions of the same cookbook in the chef-repo
> - detect and handle dependency version conflicts when putting together the  
> run\_list (does chef do this already???)
> 
> Except for the third point librarian-chef supports this already, and it  
> would probably be easy to fork librarian for that.
> 
> But the more I am thinking about it the more I believe that either  
> a) I am doing something wrong (against the intended use of chef), or  
> b) this should be an integral part of chef itself
> 
> Help, I'm confused... 🙂
> 
> Torben
> 
> > On Mon, Apr 23, 2012 at 12:45 PM, Torben Knerr [ukio@gmx.de](mailto:ukio@gmx.de) wrote:
> > 
> > > Hi Jay,
> > > 
> > > thanks for the quick reply!
> > > 
> > > I see the value in not having the same cookbook in different versions,  
> > > but it is also very restricting and I also see some cases where you might  
> > > need that.
> > > 
> > > In the example above, consider that "foo" and "bar" are well tested  
> > > cookbooks not owned by me. As long as I use them independently (i.e. no  
> > > node has both of the cookbooks) that dependency to different versions of  
> > > "baz" is totally fine, and would become a conflict only when I used both  
> > > cookbooks on the same node, isn't it so?
> > > 
> > > Now to resolve the conflict, I would have to patch "foo" to be  
> > > compatible with "baz" 2.0, which might not be an easy task and would  
> > > potentially break it, so I'd rather not touch that thing!
> > > 
> > > I'm wondering: shouldn't librarian-chef be more liberal wrt to defining  
> > > dependencies to different versions of the same cookbook, and on the other  
> > > hand the chef server (or knife) be more strict wrt to configuring a node  
> > > with conflicting dependencies?
> > > 
> > > Cheers,  
> > > Torben

---

<div class="post-metadata">

**Author:** ![Torben\_Knerr](https://avatars.discourse-cdn.com/v4/letter/t/ba8739/32.png) [@Torben\_Knerr](https://discourse.chef.io/u/Torben_Knerr)\
**Post date:** [April 24, 2012, 10:03pm UTC](https://discourse.chef.io/t/demonstrating-the-power-of-chef-librarian-and-vagrant/2703/14 "2012-04-24T22:03:55Z")

</div>

On Tue, Apr 24, 2012 at 12:22 PM, Jay Feldblum [y\_feldblum@yahoo.com](mailto:y_feldblum@yahoo.com) wrote:

> Torben,
> 
> - I'm not sure how to handle multiple versions of the same cookbook in the  
> same git repository.

My idea was to use tags, as in:

cookbook "rvm",  
:git =\> "[GitHub - sous-chefs/rvm: Development repository for the rvm cookbook](https://github.com/fnichol/chef-rvm)",  
:ref =\> "v0.7.1"

> - The chef-server has a dependency resolver which will put together  
> consistent run-lists based on the cookbooks, including multiple versions of  
> the same cookbook, that have been uploaded to the chef-server.

I assumed that, but didn't know for sure. Can you point me to any  
documentation about it? I would be especially interested if you could make  
it fail if it detects a dependency version conflict in a run\_list...

Cheers,  
Torben

> Chef supports a wide variety of ways to use it. Uploading multiple  
> versions of the same cookbook seems to be one of the intended ways to use  
> it. In that case, you would be relying on the chef-server to deliver a  
> consistent run-list to each node.
> 
> Cheers,  
> Jay
> 
> On Tue, Apr 24, 2012 at 2:11 AM, Torben Knerr [ukio@gmx.de](mailto:ukio@gmx.de) wrote:
> 
> > Am 23.04.2012 19:40 schrieb "Jay Feldblum" [y\_feldblum@yahoo.com](mailto:y_feldblum@yahoo.com):
> > 
> > > Torben,
> > > 
> > > Fantastic question. You're describing the standard way people might use  
> > > Chef, which is "Infrastructure as Code." This is what Opscode documents in  
> > > their tutorials and in their Wiki.
> > > 
> > > But the purpose of Librarian-Chef is to help people use Chef in a  
> > > subtly different, stricter way, which I call "Infrastructure as  
> > > Application." The difference is that, in an application, you don't just  
> > > have a loosely-related set of modules in source control. You have a single  
> > > integrated codebase in source control, which requires that you have a  
> > > single integrated set of dependencies, with their exact versions and  
> > > sources, documented in source control. This results in every node getting  
> > > the exact same cookbooks as are listed in source control each and every run.
> > > 
> > > Librarian-Chef is not a tool to help fetch cookbooks. It is a tool to  
> > > help you automatically enforce a policy of having a single integrated  
> > > codebase in source control with a single integrated set of dependencies. As  
> > > part of that enforcing that policy, it provides useful services: it can  
> > > fetch cookbooks, it can tell you which cookbooks are outdated, it can  
> > > update cookbooks one-by-one and all-at-once, and it can hook into knife.  
> > > There are some things it does not do, such as enforce that the resolved  
> > > cookbooks are synced to your chef-server (although boundary-knife-plugins  
> > > can help there). This is what Bundler does for the gems used by any Ruby  
> > > application.
> > > 
> > > This policy will probably be acceptable to most people most of the  
> > > time, since most people most of the time will not have a strong need to  
> > > have two versions of the same cookbook in their infrastructure repository  
> > > at the same time. Some people may even agree with the policy, and  
> > > Librarian-Chef can help them enforce it. If you are faced with a particular  
> > > incompatibility, then you can certainly fork one of the cookbooks in  
> > > question and change it to be compatible, so that you can end up with a  
> > > single integrated codebase with a single integrated set of dependencies.
> > > 
> > > If what you need is simply a tool to help fetch cookbooks, without  
> > > enforcing the "Infrastructure as Application" policy, there are a number of  
> > > others tools available that do that.
> > 
> > Thanks for the explanation, the "single integrated set of dependencies"  
> > policy was not clear to me, I saw it rather as a generic dependency  
> > management solution for cookbooks (like maven for cookbooks if you will).
> > 
> > My requirements would be:
> > 
> > - keep my cookbooks in their own git repos outside the chef-repo (don't  
> > like nested git repos)
> > - pull in 3rd party cookbooks as dependencies (via git, opscode api,  
> > etc...) in a declarative manner (like Cheffile)
> > - allow multiple versions of the same cookbook in the chef-repo
> > - detect and handle dependency version conflicts when putting together  
> > the run\_list (does chef do this already???)
> > 
> > Except for the third point librarian-chef supports this already, and it  
> > would probably be easy to fork librarian for that.
> > 
> > But the more I am thinking about it the more I believe that either  
> > a) I am doing something wrong (against the intended use of chef), or  
> > b) this should be an integral part of chef itself
> > 
> > Help, I'm confused... 🙂
> > 
> > Torben
> > 
> > > On Mon, Apr 23, 2012 at 12:45 PM, Torben Knerr [ukio@gmx.de](mailto:ukio@gmx.de) wrote:
> > > 
> > > > Hi Jay,
> > > > 
> > > > thanks for the quick reply!
> > > > 
> > > > I see the value in not having the same cookbook in different versions,  
> > > > but it is also very restricting and I also see some cases where you might  
> > > > need that.
> > > > 
> > > > In the example above, consider that "foo" and "bar" are well tested  
> > > > cookbooks not owned by me. As long as I use them independently (i.e. no  
> > > > node has both of the cookbooks) that dependency to different versions of  
> > > > "baz" is totally fine, and would become a conflict only when I used both  
> > > > cookbooks on the same node, isn't it so?
> > > > 
> > > > Now to resolve the conflict, I would have to patch "foo" to be  
> > > > compatible with "baz" 2.0, which might not be an easy task and would  
> > > > potentially break it, so I'd rather not touch that thing!
> > > > 
> > > > I'm wondering: shouldn't librarian-chef be more liberal wrt to  
> > > > defining dependencies to different versions of the same cookbook, and on  
> > > > the other hand the chef server (or knife) be more strict wrt to configuring  
> > > > a node with conflicting dependencies?
> > > > 
> > > > Cheers,  
> > > > Torben

---

<div class="post-metadata">

**Author:** ![Jay\_Feldblum](https://avatars.discourse-cdn.com/v4/letter/j/9fc29f/32.png) [@Jay\_Feldblum](https://discourse.chef.io/u/Jay_Feldblum)\
**Post date:** [April 24, 2012, 10:36pm UTC](https://discourse.chef.io/t/demonstrating-the-power-of-chef-librarian-and-vagrant/2703/15 "2012-04-24T22:36:55Z")

</div>

Torben,

When a chef-client running on a node starts a run, it asks the chef-server  
for the cookbooks applicable to that node and downloads them. The  
chef-server looks at the node's run-list, expands the node's run-list, and  
does a full dependency-resolution for the cookbooks listed in the expanded  
run-list against all the cookbooks that have been uploaded to the  
chef-server. The chef-client tries only to download the specific cookbooks  
it actually will need, and only those files from those cookbooks that it  
will actually need.

Cheers,  
Jay

On Tue, Apr 24, 2012 at 6:03 PM, Torben Knerr [ukio@gmx.de](mailto:ukio@gmx.de) wrote:

> On Tue, Apr 24, 2012 at 12:22 PM, Jay Feldblum [y\_feldblum@yahoo.com](mailto:y_feldblum@yahoo.com)wrote:
> 
> > Torben,
> > 
> > - I'm not sure how to handle multiple versions of the same cookbook in  
> > the same git repository.
> 
> My idea was to use tags, as in:
> 
> cookbook "rvm",  
> :git =\> "[GitHub - sous-chefs/rvm: Development repository for the rvm cookbook](https://github.com/fnichol/chef-rvm)",  
> :ref =\> "v0.7.1"
> 
> > - The chef-server has a dependency resolver which will put together  
> > consistent run-lists based on the cookbooks, including multiple versions of  
> > the same cookbook, that have been uploaded to the chef-server.
> 
> I assumed that, but didn't know for sure. Can you point me to any  
> documentation about it? I would be especially interested if you could make  
> it fail if it detects a dependency version conflict in a run\_list...
> 
> Cheers,  
> Torben
> 
> > Chef supports a wide variety of ways to use it. Uploading multiple  
> > versions of the same cookbook seems to be one of the intended ways to use  
> > it. In that case, you would be relying on the chef-server to deliver a  
> > consistent run-list to each node.
> > 
> > Cheers,  
> > Jay
> > 
> > On Tue, Apr 24, 2012 at 2:11 AM, Torben Knerr [ukio@gmx.de](mailto:ukio@gmx.de) wrote:
> > 
> > > Am 23.04.2012 19:40 schrieb "Jay Feldblum" [y\_feldblum@yahoo.com](mailto:y_feldblum@yahoo.com):
> > > 
> > > > Torben,
> > > > 
> > > > Fantastic question. You're describing the standard way people might  
> > > > use Chef, which is "Infrastructure as Code." This is what Opscode documents  
> > > > in their tutorials and in their Wiki.
> > > > 
> > > > But the purpose of Librarian-Chef is to help people use Chef in a  
> > > > subtly different, stricter way, which I call "Infrastructure as  
> > > > Application." The difference is that, in an application, you don't just  
> > > > have a loosely-related set of modules in source control. You have a single  
> > > > integrated codebase in source control, which requires that you have a  
> > > > single integrated set of dependencies, with their exact versions and  
> > > > sources, documented in source control. This results in every node getting  
> > > > the exact same cookbooks as are listed in source control each and every run.
> > > > 
> > > > Librarian-Chef is not a tool to help fetch cookbooks. It is a tool to  
> > > > help you automatically enforce a policy of having a single integrated  
> > > > codebase in source control with a single integrated set of dependencies. As  
> > > > part of that enforcing that policy, it provides useful services: it can  
> > > > fetch cookbooks, it can tell you which cookbooks are outdated, it can  
> > > > update cookbooks one-by-one and all-at-once, and it can hook into knife.  
> > > > There are some things it does not do, such as enforce that the resolved  
> > > > cookbooks are synced to your chef-server (although boundary-knife-plugins  
> > > > can help there). This is what Bundler does for the gems used by any Ruby  
> > > > application.
> > > > 
> > > > This policy will probably be acceptable to most people most of the  
> > > > time, since most people most of the time will not have a strong need to  
> > > > have two versions of the same cookbook in their infrastructure repository  
> > > > at the same time. Some people may even agree with the policy, and  
> > > > Librarian-Chef can help them enforce it. If you are faced with a particular  
> > > > incompatibility, then you can certainly fork one of the cookbooks in  
> > > > question and change it to be compatible, so that you can end up with a  
> > > > single integrated codebase with a single integrated set of dependencies.
> > > > 
> > > > If what you need is simply a tool to help fetch cookbooks, without  
> > > > enforcing the "Infrastructure as Application" policy, there are a number of  
> > > > others tools available that do that.
> > > 
> > > Thanks for the explanation, the "single integrated set of dependencies"  
> > > policy was not clear to me, I saw it rather as a generic dependency  
> > > management solution for cookbooks (like maven for cookbooks if you will).
> > > 
> > > My requirements would be:
> > > 
> > > - keep my cookbooks in their own git repos outside the chef-repo (don't  
> > > like nested git repos)
> > > - pull in 3rd party cookbooks as dependencies (via git, opscode api,  
> > > etc...) in a declarative manner (like Cheffile)
> > > - allow multiple versions of the same cookbook in the chef-repo
> > > - detect and handle dependency version conflicts when putting together  
> > > the run\_list (does chef do this already???)
> > > 
> > > Except for the third point librarian-chef supports this already, and it  
> > > would probably be easy to fork librarian for that.
> > > 
> > > But the more I am thinking about it the more I believe that either  
> > > a) I am doing something wrong (against the intended use of chef), or  
> > > b) this should be an integral part of chef itself
> > > 
> > > Help, I'm confused... 🙂
> > > 
> > > Torben
> > > 
> > > > On Mon, Apr 23, 2012 at 12:45 PM, Torben Knerr [ukio@gmx.de](mailto:ukio@gmx.de) wrote:
> > > > 
> > > > > Hi Jay,
> > > > > 
> > > > > thanks for the quick reply!
> > > > > 
> > > > > I see the value in not having the same cookbook in different  
> > > > > versions, but it is also very restricting and I also see some cases where  
> > > > > you might need that.
> > > > > 
> > > > > In the example above, consider that "foo" and "bar" are well tested  
> > > > > cookbooks not owned by me. As long as I use them independently (i.e. no  
> > > > > node has both of the cookbooks) that dependency to different versions of  
> > > > > "baz" is totally fine, and would become a conflict only when I used both  
> > > > > cookbooks on the same node, isn't it so?
> > > > > 
> > > > > Now to resolve the conflict, I would have to patch "foo" to be  
> > > > > compatible with "baz" 2.0, which might not be an easy task and would  
> > > > > potentially break it, so I'd rather not touch that thing!
> > > > > 
> > > > > I'm wondering: shouldn't librarian-chef be more liberal wrt to  
> > > > > defining dependencies to different versions of the same cookbook, and on  
> > > > > the other hand the chef server (or knife) be more strict wrt to configuring  
> > > > > a node with conflicting dependencies?
> > > > > 
> > > > > Cheers,  
> > > > > Torben