What to pin in environments?

[Steffen_Gebert1]

Hi,

we’re pinning cookbook versions in environments. I think ~80% of all
cookbooks in use are pinned to a specific version.

Do you consider that a good idea?

Or do you only pin the version of your application or wrapper cookbooks
there in the environment (while these cookbooks then depend on other
cookbooks in certain versions)?

I’m motivated to drop all the constraints for community cookbooks in
environments and moving them to the particular cookbooks of our
wrapper/application cookbooks, as this will (I hope) ease upgrades of
particular cookbooks (e.g. the jenkins cookbook now).

Got my point?

I mean a server running the new jenkins cookbook now will also use the
newer apache cookbook for instance. However, I cannot easly see then,
which old versions of cookbooks are still running, can’t I?

Thanks for giving me an impression, how you deal with that!

Yours
Steffen

[Torben_Knerr]

Hi Steffen,

the approach I tend to prefer is pinning only the "top-level" cookbooks in
your environment file.

Top-level just means the top-level per node, i.e. we have one cookbook which
a) reprents the "stuff" to be installed on that node
b) locks all transitive dependencies (and has the Berksfile.lock checked in)
c) documents the "public interface" i.e. attributes, usage, etc...

All the transitive cookbook dependencies are defined in the top-level
cookbook's metadata.rb. Since I want to lock the whole dependency graph
here I usually do a berks list and transform that output to something I
can paste into metadata.rb

It's probably exactly the same as you meant, but there is some confusion
about the definition of application / library / wrapper cookbooks so I tend
to name them "top-level".

Yes, this makes upgrading a cookbook in a specific environment as easy as
flipping a single version bit in your environment file.

Yes, it lets you use different versions of the apache cookbook in the same
environment, which is good imho because it's just an implementation detail
of your top-level cookbook. Upgrading to a newer apache version for that
cookbook should not affect all the other cookbooks in your environment!

And no, it's not that easy anymore to see which versions of the apache
cookbook are in use anymore (well it might be searchable via the chef
server API?)

To me it seems that many people take the route of pinning everything in the
environment (e.g. via berks apply). To me it seems wrong (or at least as
a smell) to use environments for that, but that's just my view and there
might be some good reasons to do it the other way around. This is just what
works well for me...

I'd be happy to hear what others say.

Cheers,
Torben

On Sun, Jan 19, 2014 at 8:24 PM, Steffen Gebert st+gmane@st-g.de wrote:

Hi,

we're pinning cookbook versions in environments. I think ~80% of all
cookbooks in use are pinned to a specific version.

Do you consider that a good idea?

Or do you only pin the version of your application or wrapper cookbooks
there in the environment (while these cookbooks then depend on other
cookbooks in certain versions)?

I'm motivated to drop all the constraints for community cookbooks in
environments and moving them to the particular cookbooks of our
wrapper/application cookbooks, as this will (I hope) ease upgrades of
particular cookbooks (e.g. the jenkins cookbook now).

Got my point?

I mean a server running the new jenkins cookbook now will also use the
newer apache cookbook for instance. However, I cannot easly see then,
which old versions of cookbooks are still running, can't I?

Thanks for giving me an impression, how you deal with that!

Yours
Steffen

[realityforge]

And no, it's not that easy anymore to see which versions of the apache
cookbook are in use anymore (well it might be searchable via the chef server
API?)

We use a variant of knife-audit [1] to do this for us and it works well.

To me it seems that many people take the route of pinning everything in the
environment (e.g. via berks apply). To me it seems wrong (or at least as a
smell) to use environments for that, but that's just my view and there might
be some good reasons to do it the other way around. This is just what works
well for me...

I would really like to use the technique you use and we did actually
migrate towards the approach of only locking down top-level cookbooks
in the environment files and then lock down supporting cookbooks in
the top-level cookbooks metadata.

Unfortunately we repeatedly ran in to the dependency resolver bug in
Chef 11 (CHEF-3921) which would kill our chef server. So we were
forced back to locking down everything in environments.

[1] GitHub - jbz/knife-audit: Plugin for Opscode Chef's 'knife' utility to gather cookbook use info

--
Cheers,

Peter Donald

[Steffen_Gebert1]

Hi Torben, Hi Peter,

thanks for your replies!

This knife-audit only returns a list of cookbooks in use, or also the
used versions? (couldn't find any information about the latter one in
the README).

What you pointed out is what I had in mind and I guess it's atm the
easiest way to to. Having knife-audit to see, which nodes still use old
versions would be really helpful to e.g. make sure that some "bad"
configuration for one software is not isn't in use anywhere anymore.

What still bothers me is the duplicate definition of dependencies in
metadata plus Berksfile. We haven't adopted Berkshelf completely, yet,
but to specify all deps twice smells a bit to me (but there's no other
way, not even with Berks 3, isn't it?).

Thanks for your input!

Yours
Steffen

On 29/01/14 06:02, Peter Donald wrote:

And no, it's not that easy anymore to see which versions of the apache
cookbook are in use anymore (well it might be searchable via the chef server
API?)

We use a variant of knife-audit [1] to do this for us and it works well.

To me it seems that many people take the route of pinning everything in the
environment (e.g. via berks apply). To me it seems wrong (or at least as a
smell) to use environments for that, but that's just my view and there might
be some good reasons to do it the other way around. This is just what works
well for me...

I would really like to use the technique you use and we did actually
migrate towards the approach of only locking down top-level cookbooks
in the environment files and then lock down supporting cookbooks in
the top-level cookbooks metadata.

Unfortunately we repeatedly ran in to the dependency resolver bug in
Chef 11 (CHEF-3921) which would kill our chef server. So we were
forced back to locking down everything in environments.

[1] GitHub - jbz/knife-audit: Plugin for Opscode Chef's 'knife' utility to gather cookbook use info

[Torben_Knerr]

Hi Steffen, Peter,

Yes, there is some kind of duplication involved, but there should be always
a single source of truth. Here's how we handle it:

1. in the "top-level" cookbook we define the dependencies in the Berksfile,
   and the metadata.rb contains a protected region for the depends statements.
   You can fill this protected region either manually or via a rake task or
   similar during the build / test, which would ensure it is always consistent
   with what you specified in the Berksfile. You can use 'berks list' and
   transform that output to depends statements to get the fully
   Berksfile.lock'ed dependency graph.

2. in any non-"top-level" cookbook we are much more lax with constraining
   dependencies so they can be easily mixed together. Here we define the
   direct dependencies (not the transitive ones) in metadata.rb, and the
   Berksfile only contains the 'metadata' keyword (plus git locations etc if
   any, but they had to be repeated in the top-level cookbook's Berksfile
   anyway...)

3. is fine in terms of duplication, because there is essentially none

4. is far from ideal, but the best I could get so far. It's definitely ugly
   that a part of the metadata.rb is generated. I would rather use includes
   than protected regions, but haven't found a clever way how to do so yet. Or
   can we just use IO.read(transform("Berksfile.lock")) in metadata.rb and it
   gets properly rendered when the metadata.json is generated?!

Ideally I'd like some first-class Berkshelf integration for "top-level"
cookbooks (e.g. 'berks apply_metadata' rather than 'berks apply
'), but that's not a feature they'd like to support in
Berkshelf. It probably can be done by extending / hooking in at the right
place with a side-by-side installed gem, but my Ruby-Fu is too weak yet to
do hack that up in a reasonable amount of time

P.S.: didn't hit CHEF-3921 yet because I'm mostly using chef-solo, but good
to know...

Cheers,
Torben
On Jan 29, 2014 8:30 AM, "Steffen Gebert" st+gmane@st-g.de wrote:

Hi Torben, Hi Peter,

thanks for your replies!

This knife-audit only returns a list of cookbooks in use, or also the
used versions? (couldn't find any information about the latter one in
the README).

What you pointed out is what I had in mind and I guess it's atm the
easiest way to to. Having knife-audit to see, which nodes still use old
versions would be really helpful to e.g. make sure that some "bad"
configuration for one software is not isn't in use anywhere anymore.

What still bothers me is the duplicate definition of dependencies in
metadata plus Berksfile. We haven't adopted Berkshelf completely, yet,
but to specify all deps twice smells a bit to me (but there's no other
way, not even with Berks 3, isn't it?).

Thanks for your input!

Yours
Steffen

On 29/01/14 06:02, Peter Donald wrote:

And no, it's not that easy anymore to see which versions of the apache
cookbook are in use anymore (well it might be searchable via the chef
server
API?)

We use a variant of knife-audit [1] to do this for us and it works well.

To me it seems that many people take the route of pinning everything in
the
environment (e.g. via berks apply). To me it seems wrong (or at least
as a
smell) to use environments for that, but that's just my view and there
might
be some good reasons to do it the other way around. This is just what
works
well for me...

I would really like to use the technique you use and we did actually
migrate towards the approach of only locking down top-level cookbooks
in the environment files and then lock down supporting cookbooks in
the top-level cookbooks metadata.

Unfortunately we repeatedly ran in to the dependency resolver bug in
Chef 11 (CHEF-3921) which would kill our chef server. So we were
forced back to locking down everything in environments.

[1] GitHub - jbz/knife-audit: Plugin for Opscode Chef's 'knife' utility to gather cookbook use info

[Steffen_Gebert1]

Hi Torben,

1. is far from ideal, but the best I could get so far. It's definitely
   ugly
   that a part of the metadata.rb is generated. I would rather use includes
   than protected regions, but haven't found a clever way how to do so
   yet. Or
   can we just use IO.read(transform("Berksfile.lock")) in metadata.rb and it
   gets properly rendered when the metadata.json is generated?!

That should be possible. The only thing to solve would be parsing the
Berksfile and transforming it into the "depends" statements.
Do you remember the old cookbook template that contained such a line?

long_description IO.read(File.join(File.dirname(FILE), 'README.rdoc'))
That gives me the impression that any ruby code can be used there.

Yours
Steffen

On 29/01/14 18:49, Torben Knerr wrote:

Hi Steffen, Peter,

Yes, there is some kind of duplication involved, but there should be always
a single source of truth. Here's how we handle it:

1. in the "top-level" cookbook we define the dependencies in the Berksfile,
   and the metadata.rb contains a protected region for the depends statements.
   You can fill this protected region either manually or via a rake task or
   similar during the build / test, which would ensure it is always consistent
   with what you specified in the Berksfile. You can use 'berks list' and
   transform that output to depends statements to get the fully
   Berksfile.lock'ed dependency graph.

2. in any non-"top-level" cookbook we are much more lax with constraining
   dependencies so they can be easily mixed together. Here we define the
   direct dependencies (not the transitive ones) in metadata.rb, and the
   Berksfile only contains the 'metadata' keyword (plus git locations etc if
   any, but they had to be repeated in the top-level cookbook's Berksfile
   anyway...)

3. is fine in terms of duplication, because there is essentially none

4. is far from ideal, but the best I could get so far. It's definitely ugly
   that a part of the metadata.rb is generated. I would rather use includes
   than protected regions, but haven't found a clever way how to do so yet. Or
   can we just use IO.read(transform("Berksfile.lock")) in metadata.rb and it
   gets properly rendered when the metadata.json is generated?!

Ideally I'd like some first-class Berkshelf integration for "top-level"
cookbooks (e.g. 'berks apply_metadata' rather than 'berks apply
'), but that's not a feature they'd like to support in
Berkshelf. It probably can be done by extending / hooking in at the right
place with a side-by-side installed gem, but my Ruby-Fu is too weak yet to
do hack that up in a reasonable amount of time

P.S.: didn't hit CHEF-3921 yet because I'm mostly using chef-solo, but good
to know...

Cheers,
Torben
On Jan 29, 2014 8:30 AM, "Steffen Gebert" st+gmane@st-g.de wrote:

Hi Torben, Hi Peter,

thanks for your replies!

This knife-audit only returns a list of cookbooks in use, or also the
used versions? (couldn't find any information about the latter one in
the README).

What you pointed out is what I had in mind and I guess it's atm the
easiest way to to. Having knife-audit to see, which nodes still use old
versions would be really helpful to e.g. make sure that some "bad"
configuration for one software is not isn't in use anywhere anymore.

What still bothers me is the duplicate definition of dependencies in
metadata plus Berksfile. We haven't adopted Berkshelf completely, yet,
but to specify all deps twice smells a bit to me (but there's no other
way, not even with Berks 3, isn't it?).

Thanks for your input!

Yours
Steffen

On 29/01/14 06:02, Peter Donald wrote:

And no, it's not that easy anymore to see which versions of the apache
cookbook are in use anymore (well it might be searchable via the chef
server
API?)

We use a variant of knife-audit [1] to do this for us and it works well.

To me it seems that many people take the route of pinning everything in
the
environment (e.g. via berks apply). To me it seems wrong (or at least
as a
smell) to use environments for that, but that's just my view and there
might
be some good reasons to do it the other way around. This is just what
works
well for me...

I would really like to use the technique you use and we did actually
migrate towards the approach of only locking down top-level cookbooks
in the environment files and then lock down supporting cookbooks in
the top-level cookbooks metadata.

Unfortunately we repeatedly ran in to the dependency resolver bug in
Chef 11 (CHEF-3921) which would kill our chef server. So we were
forced back to locking down everything in environments.

[1] GitHub - jbz/knife-audit: Plugin for Opscode Chef's 'knife' utility to gather cookbook use info