# What to pin in environments?

**URL:** <https://discourse.chef.io/t/what-to-pin-in-environments/4920>\
**Category:** Chef Infra (archive)\
**Created:** [January 19, 2014, 7:24pm UTC](https://discourse.chef.io/t/what-to-pin-in-environments/4920 "2014-01-19T19:24:54Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Steffen\_Gebert1](https://avatars.discourse-cdn.com/v4/letter/s/e56c9b/32.png) [@Steffen\_Gebert1](https://discourse.chef.io/u/Steffen_Gebert1)\
**Post date:** [January 19, 2014, 7:24pm UTC](https://discourse.chef.io/t/what-to-pin-in-environments/4920/1 "2014-01-19T19:24:54Z")

</div>

Hi,

we’re pinning cookbook versions in environments. I think ~80% of all  
cookbooks in use are pinned to a specific version.

Do you consider that a good idea?

Or do you only pin the version of your application or wrapper cookbooks  
there in the environment (while these cookbooks then depend on other  
cookbooks in certain versions)?

I’m motivated to drop all the constraints for community cookbooks in  
environments and moving them to the particular cookbooks of our  
wrapper/application cookbooks, as this will (I hope) ease upgrades of  
particular cookbooks (e.g. the jenkins cookbook now).

Got my point?

I mean a server running the new jenkins cookbook now will also use the  
newer apache cookbook for instance. However, I cannot easly see then,  
which old versions of cookbooks are still running, can’t I?

Thanks for giving me an impression, how you deal with that!

Yours  
Steffen

---

<div class="post-metadata">

**Author:** ![Torben\_Knerr](https://avatars.discourse-cdn.com/v4/letter/t/ba8739/32.png) [@Torben\_Knerr](https://discourse.chef.io/u/Torben_Knerr)\
**Post date:** [January 26, 2014, 2:44pm UTC](https://discourse.chef.io/t/what-to-pin-in-environments/4920/2 "2014-01-26T14:44:29Z")

</div>

Hi Steffen,

the approach I tend to prefer is pinning only the "top-level" cookbooks in  
your environment file.

Top-level just means the top-level per node, i.e. we have one cookbook which  
a) reprents the "stuff" to be installed on that node  
b) locks all transitive dependencies (and has the Berksfile.lock checked in)  
c) documents the "public interface" i.e. attributes, usage, etc...

All the transitive cookbook dependencies are defined in the top-level  
cookbook's metadata.rb. Since I want to lock the whole dependency graph  
here I usually do a `berks list` and transform that output to something I  
can paste into metadata.rb

It's probably exactly the same as you meant, but there is some confusion  
about the definition of application / library / wrapper cookbooks so I tend  
to name them "top-level".

Yes, this makes upgrading a cookbook in a specific environment as easy as  
flipping a single version bit in your environment file.

Yes, it lets you use different versions of the apache cookbook in the same  
environment, which is good imho because it's just an implementation detail  
of your top-level cookbook. Upgrading to a newer apache version for that  
cookbook should not affect all the other cookbooks in your environment!

And no, it's not that easy anymore to see which versions of the apache  
cookbook are in use anymore (well it might be searchable via the chef  
server API?)

To me it seems that many people take the route of pinning everything in the  
environment (e.g. via `berks apply`). To me it seems wrong (or at least as  
a smell) to use environments for that, but that's just my view and there  
might be some good reasons to do it the other way around. This is just what  
works well for me...

I'd be happy to hear what others say.

Cheers,  
Torben

On Sun, Jan 19, 2014 at 8:24 PM, Steffen Gebert [st+gmane@st-g.de](mailto:st+gmane@st-g.de) wrote:

> Hi,
> 
> we're pinning cookbook versions in environments. I think ~80% of all  
> cookbooks in use are pinned to a specific version.
> 
> Do you consider that a good idea?
> 
> Or do you only pin the version of your application or wrapper cookbooks  
> there in the environment (while these cookbooks then depend on other  
> cookbooks in certain versions)?
> 
> I'm motivated to drop all the constraints for community cookbooks in  
> environments and moving them to the particular cookbooks of our  
> wrapper/application cookbooks, as this will (I hope) ease upgrades of  
> particular cookbooks (e.g. the jenkins cookbook now).
> 
> Got my point?
> 
> I mean a server running the new jenkins cookbook now will also use the  
> newer apache cookbook for instance. However, I cannot easly see then,  
> which old versions of cookbooks are still running, can't I?
> 
> Thanks for giving me an impression, how you deal with that!
> 
> Yours  
> Steffen

---

<div class="post-metadata">

**Author:** ![realityforge](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/realityforge/32/395_2.png) [@realityforge](https://discourse.chef.io/u/realityforge)\
**Post date:** [January 29, 2014, 5:02am UTC](https://discourse.chef.io/t/what-to-pin-in-environments/4920/3 "2014-01-29T05:02:11Z")

</div>

> And no, it's not that easy anymore to see which versions of the apache  
> cookbook are in use anymore (well it might be searchable via the chef server  
> API?)

We use a variant of knife-audit [1] to do this for us and it works well.

> To me it seems that many people take the route of pinning everything in the  
> environment (e.g. via `berks apply`). To me it seems wrong (or at least as a  
> smell) to use environments for that, but that's just my view and there might  
> be some good reasons to do it the other way around. This is just what works  
> well for me...

I would really like to use the technique you use and we did actually  
migrate towards the approach of only locking down top-level cookbooks  
in the environment files and then lock down supporting cookbooks in  
the top-level cookbooks metadata.

Unfortunately we repeatedly ran in to the dependency resolver bug in  
Chef 11 (CHEF-3921) which would kill our chef server. So we were  
forced back to locking down everything in environments.

[1] [GitHub - jbz/knife-audit: Plugin for Opscode Chef's 'knife' utility to gather cookbook use info](https://github.com/jbz/knife-audit)

--  
Cheers,

Peter Donald

---

<div class="post-metadata">

**Author:** ![Steffen\_Gebert1](https://avatars.discourse-cdn.com/v4/letter/s/e56c9b/32.png) [@Steffen\_Gebert1](https://discourse.chef.io/u/Steffen_Gebert1)\
**Post date:** [January 29, 2014, 7:30am UTC](https://discourse.chef.io/t/what-to-pin-in-environments/4920/4 "2014-01-29T07:30:28Z")

</div>

Hi Torben, Hi Peter,

thanks for your replies!

This knife-audit only returns a list of cookbooks in use, or also the  
used versions? (couldn't find any information about the latter one in  
the README).

What you pointed out is what I had in mind and I guess it's atm the  
easiest way to to. Having knife-audit to see, which nodes still use old  
versions would be really helpful to e.g. make sure that some "bad"  
configuration for one software is not isn't in use anywhere anymore.

What still bothers me is the duplicate definition of dependencies in  
metadata plus Berksfile. We haven't adopted Berkshelf completely, yet,  
but to specify all deps twice smells a bit to me (but there's no other  
way, not even with Berks 3, isn't it?).

Thanks for your input!

Yours  
Steffen

On 29/01/14 06:02, Peter Donald wrote:

> > And no, it's not that easy anymore to see which versions of the apache  
> > cookbook are in use anymore (well it might be searchable via the chef server  
> > API?)
> 
> We use a variant of knife-audit [1] to do this for us and it works well.
> 
> > To me it seems that many people take the route of pinning everything in the  
> > environment (e.g. via `berks apply`). To me it seems wrong (or at least as a  
> > smell) to use environments for that, but that's just my view and there might  
> > be some good reasons to do it the other way around. This is just what works  
> > well for me...
> 
> I would really like to use the technique you use and we did actually  
> migrate towards the approach of only locking down top-level cookbooks  
> in the environment files and then lock down supporting cookbooks in  
> the top-level cookbooks metadata.
> 
> Unfortunately we repeatedly ran in to the dependency resolver bug in  
> Chef 11 (CHEF-3921) which would kill our chef server. So we were  
> forced back to locking down everything in environments.
> 
> [1] [GitHub - jbz/knife-audit: Plugin for Opscode Chef's 'knife' utility to gather cookbook use info](https://github.com/jbz/knife-audit)

---

<div class="post-metadata">

**Author:** ![Torben\_Knerr](https://avatars.discourse-cdn.com/v4/letter/t/ba8739/32.png) [@Torben\_Knerr](https://discourse.chef.io/u/Torben_Knerr)\
**Post date:** [January 29, 2014, 5:49pm UTC](https://discourse.chef.io/t/what-to-pin-in-environments/4920/5 "2014-01-29T17:49:23Z")

</div>

Hi Steffen, Peter,

Yes, there is some kind of duplication involved, but there should be always  
a single source of truth. Here's how we handle it:

1. in the "top-level" cookbook we define the dependencies in the Berksfile,  
and the metadata.rb contains a protected region for the depends statements.  
You can fill this protected region either manually or via a rake task or  
similar during the build / test, which would ensure it is always consistent  
with what you specified in the Berksfile. You can use 'berks list' and  
transform that output to depends statements to get the fully  
Berksfile.lock'ed dependency graph.

2. in any non-"top-level" cookbook we are much more lax with constraining  
dependencies so they can be easily mixed together. Here we define the  
direct dependencies (not the transitive ones) in metadata.rb, and the  
Berksfile only contains the 'metadata' keyword (plus git locations etc if  
any, but they had to be repeated in the top-level cookbook's Berksfile  
anyway...)

3. is fine in terms of duplication, because there is essentially none 🙂

4. is far from ideal, but the best I could get so far. It's definitely ugly  
that a part of the metadata.rb is generated. I would rather use includes  
than protected regions, but haven't found a clever way how to do so yet. Or  
can we just use IO.read(transform("Berksfile.lock")) in metadata.rb and it  
gets properly rendered when the metadata.json is generated?!

Ideally I'd like some first-class Berkshelf integration for "top-level"  
cookbooks (e.g. 'berks apply\_metadata' rather than 'berks apply  
'), but that's not a feature they'd like to support in  
Berkshelf. It probably can be done by extending / hooking in at the right  
place with a side-by-side installed gem, but my Ruby-Fu is too weak yet to  
do hack that up in a reasonable amount of time 😉

P.S.: didn't hit CHEF-3921 yet because I'm mostly using chef-solo, but good  
to know...

Cheers,  
Torben  
On Jan 29, 2014 8:30 AM, "Steffen Gebert" [st+gmane@st-g.de](mailto:st+gmane@st-g.de) wrote:

> Hi Torben, Hi Peter,
> 
> thanks for your replies!
> 
> This knife-audit only returns a list of cookbooks in use, or also the  
> used versions? (couldn't find any information about the latter one in  
> the README).
> 
> What you pointed out is what I had in mind and I guess it's atm the  
> easiest way to to. Having knife-audit to see, which nodes still use old  
> versions would be really helpful to e.g. make sure that some "bad"  
> configuration for one software is not isn't in use anywhere anymore.
> 
> What still bothers me is the duplicate definition of dependencies in  
> metadata plus Berksfile. We haven't adopted Berkshelf completely, yet,  
> but to specify all deps twice smells a bit to me (but there's no other  
> way, not even with Berks 3, isn't it?).
> 
> Thanks for your input!
> 
> Yours  
> Steffen
> 
> On 29/01/14 06:02, Peter Donald wrote:
> 
> > > And no, it's not that easy anymore to see which versions of the apache  
> > > cookbook are in use anymore (well it might be searchable via the chef  
> > > server  
> > > API?)
> > 
> > We use a variant of knife-audit [1] to do this for us and it works well.
> > 
> > > To me it seems that many people take the route of pinning everything in  
> > > the  
> > > environment (e.g. via `berks apply`). To me it seems wrong (or at least  
> > > as a  
> > > smell) to use environments for that, but that's just my view and there  
> > > might  
> > > be some good reasons to do it the other way around. This is just what  
> > > works  
> > > well for me...
> > 
> > I would really like to use the technique you use and we did actually  
> > migrate towards the approach of only locking down top-level cookbooks  
> > in the environment files and then lock down supporting cookbooks in  
> > the top-level cookbooks metadata.
> > 
> > Unfortunately we repeatedly ran in to the dependency resolver bug in  
> > Chef 11 (CHEF-3921) which would kill our chef server. So we were  
> > forced back to locking down everything in environments.
> > 
> > [1] [GitHub - jbz/knife-audit: Plugin for Opscode Chef's 'knife' utility to gather cookbook use info](https://github.com/jbz/knife-audit)

---

<div class="post-metadata">

**Author:** ![Steffen\_Gebert1](https://avatars.discourse-cdn.com/v4/letter/s/e56c9b/32.png) [@Steffen\_Gebert1](https://discourse.chef.io/u/Steffen_Gebert1)\
**Post date:** [January 29, 2014, 7:16pm UTC](https://discourse.chef.io/t/what-to-pin-in-environments/4920/6 "2014-01-29T19:16:36Z")

</div>

Hi Torben,

> 1. is far from ideal, but the best I could get so far. It's definitely  
> ugly  
> that a part of the metadata.rb is generated. I would rather use includes  
> than protected regions, but haven't found a clever way how to do so  
> yet. Or  
> can we just use IO.read(transform("Berksfile.lock")) in metadata.rb and it  
> gets properly rendered when the metadata.json is generated?!

That should be possible. The only thing to solve would be parsing the  
Berksfile and transforming it into the "depends" statements.  
Do you remember the old cookbook template that contained such a line?

> long\_description IO.read(File.join(File.dirname( **FILE** ), 'README.rdoc'))  
> That gives me the impression that any ruby code can be used there.

Yours  
Steffen

On 29/01/14 18:49, Torben Knerr wrote:

> Hi Steffen, Peter,
> 
> Yes, there is some kind of duplication involved, but there should be always  
> a single source of truth. Here's how we handle it:
> 
> 1. in the "top-level" cookbook we define the dependencies in the Berksfile,  
> and the metadata.rb contains a protected region for the depends statements.  
> You can fill this protected region either manually or via a rake task or  
> similar during the build / test, which would ensure it is always consistent  
> with what you specified in the Berksfile. You can use 'berks list' and  
> transform that output to depends statements to get the fully  
> Berksfile.lock'ed dependency graph.
> 
> 2. in any non-"top-level" cookbook we are much more lax with constraining  
> dependencies so they can be easily mixed together. Here we define the  
> direct dependencies (not the transitive ones) in metadata.rb, and the  
> Berksfile only contains the 'metadata' keyword (plus git locations etc if  
> any, but they had to be repeated in the top-level cookbook's Berksfile  
> anyway...)
> 
> 3. is fine in terms of duplication, because there is essentially none 🙂
> 
> 4. is far from ideal, but the best I could get so far. It's definitely ugly  
> that a part of the metadata.rb is generated. I would rather use includes  
> than protected regions, but haven't found a clever way how to do so yet. Or  
> can we just use IO.read(transform("Berksfile.lock")) in metadata.rb and it  
> gets properly rendered when the metadata.json is generated?!
> 
> Ideally I'd like some first-class Berkshelf integration for "top-level"  
> cookbooks (e.g. 'berks apply\_metadata' rather than 'berks apply  
> '), but that's not a feature they'd like to support in  
> Berkshelf. It probably can be done by extending / hooking in at the right  
> place with a side-by-side installed gem, but my Ruby-Fu is too weak yet to  
> do hack that up in a reasonable amount of time 😉
> 
> P.S.: didn't hit CHEF-3921 yet because I'm mostly using chef-solo, but good  
> to know...
> 
> Cheers,  
> Torben  
> On Jan 29, 2014 8:30 AM, "Steffen Gebert" [st+gmane@st-g.de](mailto:st+gmane@st-g.de) wrote:
> 
> > Hi Torben, Hi Peter,
> > 
> > thanks for your replies!
> > 
> > This knife-audit only returns a list of cookbooks in use, or also the  
> > used versions? (couldn't find any information about the latter one in  
> > the README).
> > 
> > What you pointed out is what I had in mind and I guess it's atm the  
> > easiest way to to. Having knife-audit to see, which nodes still use old  
> > versions would be really helpful to e.g. make sure that some "bad"  
> > configuration for one software is not isn't in use anywhere anymore.
> > 
> > What still bothers me is the duplicate definition of dependencies in  
> > metadata plus Berksfile. We haven't adopted Berkshelf completely, yet,  
> > but to specify all deps twice smells a bit to me (but there's no other  
> > way, not even with Berks 3, isn't it?).
> > 
> > Thanks for your input!
> > 
> > Yours  
> > Steffen
> > 
> > On 29/01/14 06:02, Peter Donald wrote:
> > 
> > > > And no, it's not that easy anymore to see which versions of the apache  
> > > > cookbook are in use anymore (well it might be searchable via the chef  
> > > > server  
> > > > API?)
> > > 
> > > We use a variant of knife-audit [1] to do this for us and it works well.
> > > 
> > > > To me it seems that many people take the route of pinning everything in  
> > > > the  
> > > > environment (e.g. via `berks apply`). To me it seems wrong (or at least  
> > > > as a  
> > > > smell) to use environments for that, but that's just my view and there  
> > > > might  
> > > > be some good reasons to do it the other way around. This is just what  
> > > > works  
> > > > well for me...
> > > 
> > > I would really like to use the technique you use and we did actually  
> > > migrate towards the approach of only locking down top-level cookbooks  
> > > in the environment files and then lock down supporting cookbooks in  
> > > the top-level cookbooks metadata.
> > > 
> > > Unfortunately we repeatedly ran in to the dependency resolver bug in  
> > > Chef 11 (CHEF-3921) which would kill our chef server. So we were  
> > > forced back to locking down everything in environments.
> > > 
> > > [1] [GitHub - jbz/knife-audit: Plugin for Opscode Chef's 'knife' utility to gather cookbook use info](https://github.com/jbz/knife-audit)