Environment attributes in Cookbooks

If I was going to put move my environmental attributes into cookbooks…

When deployed to an instance, the top level run list would need to include
the environmental run list. That’s ok.

However, when testing, this would mean that each cookbook would need to
either include the environment cookbook (ie #include_recipe “env-dev”) and
refer to it in the metadata.rb and Berksfile files, OR I’d have to
specifically put each attribute into the Vagrantfile’s json data.

This might work for simple cookbooks, but it would mean the cookbook is not
being effectively tested. The attributes I’m testing against aren’t what
really gets deployed to an instance. I also have to maintain two copies of
the attributes and the worst case scenario is where the attribute is set in
both places but the value in the Vagranrfile is incorrect and therefore
leads to the incorrect assumption that the cookbook is working correctly.

For my base cookbook it gets even worse. It would need to have every single
attribute from every cookbook it includes put into the Vagrantfile’s json
data. This just does not scale.

It doesn’t seem like the benefits of gaining some revision control outweigh
the maintenance disadvantages here. For all those that espouse the use of
putting environment attributes into a cookbook how do you get around this?

Doug

Hey Doug,

I believe you first have to decide for what purpose you actually want to
use Chef environments, and this decision then limits the options for
technically implementing this.

For me the key decision is:
a) if you are going to use the berkflow / the environment cookbook pattern
with berks apply environment, then this means you will end up with one
environment per node
b) if you are going to use environments for dev / test / staging / prod
stages, you probably want to share environmental attributes across many
nodes

Personally, I prefer b) and try to stick to the following principles:

  • use environments only for supplying environment specific data
  • don’t use environment for composing a node’s run list or other node
    specific internals
  • instead I use a single “top-level” wrapper cookbook per node, which
    • glues together the run list
    • sets attributes for configuring the wrapped cookbooks
    • sometimes adds addional glue code
    • locks the whole cookbook dependency graph via metadata.rb (so I can do
      it per node here and not need to use an environment for that)
    • defines the interface for the user (i.e. README, attributes, etc…)
  • the environments should contain only environment specific attribute
    overrides + the versions of the “top-level” cookbooks

Since I’m wrapping other cookbooks and setting their attributes in (the
"top-level" cookbook’s) recipe, I’m sometimes faced with the “computed
attributes problem”. I tackle that by reloading the wrapped attributes file
between setting the attributes and including the recipe, e.g. here:
https://docs.chef.io/chef/essentials_cookbook_recipes.html#reload-attributes

Also, my environments often share the same data, i.e. common attributes
that are environmental data but still the same for many or even all of my
environments. Ranjib recently posted a nice way on how to deal with that:

Hope that helps or at least gives you some more ideas. There are many ways
and even more opinions :wink:

Cheers,
Torben

On Fri, Jun 19, 2015 at 4:22 PM, Douglas Garstang doug.garstang@gmail.com
wrote:

If I was going to put move my environmental attributes into cookbooks…

When deployed to an instance, the top level run list would need to include
the environmental run list. That’s ok.

However, when testing, this would mean that each cookbook would need to
either include the environment cookbook (ie #include_recipe “env-dev”) and
refer to it in the metadata.rb and Berksfile files, OR I’d have to
specifically put each attribute into the Vagrantfile’s json data.

This might work for simple cookbooks, but it would mean the cookbook is
not being effectively tested. The attributes I’m testing against aren’t
what really gets deployed to an instance. I also have to maintain two
copies of the attributes and the worst case scenario is where the attribute
is set in both places but the value in the Vagranrfile is incorrect and
therefore leads to the incorrect assumption that the cookbook is working
correctly.

For my base cookbook it gets even worse. It would need to have every
single attribute from every cookbook it includes put into the Vagrantfile’s
json data. This just does not scale.

It doesn’t seem like the benefits of gaining some revision control
outweigh the maintenance disadvantages here. For all those that espouse the
use of putting environment attributes into a cookbook how do you get around
this?

Doug

Hi Torben,

Not sure if you know, but the A/B choice you stated above aren’t actually
mutually exclusive. If you want to use berkflow/berks apply to apply
cookbook version locks to an environment that doesn’t stop you supplying
normal environment variables to the environment. Berkflow/berks apply
doesn’t change any attributes other than the version locks.

Cheers,
Yoshi

On Sun, Jun 21, 2015 at 6:39 AM, Torben Knerr mail@tknerr.de wrote:

Hey Doug,

I believe you first have to decide for what purpose you actually want to
use Chef environments, and this decision then limits the options for
technically implementing this.

For me the key decision is:
a) if you are going to use the berkflow / the environment cookbook pattern
with berks apply environment, then this means you will end up with one
environment per node
b) if you are going to use environments for dev / test / staging / prod
stages, you probably want to share environmental attributes across many
nodes

Personally, I prefer b) and try to stick to the following principles:

  • use environments only for supplying environment specific data
  • don’t use environment for composing a node’s run list or other node
    specific internals
  • instead I use a single “top-level” wrapper cookbook per node, which
    • glues together the run list
    • sets attributes for configuring the wrapped cookbooks
    • sometimes adds addional glue code
    • locks the whole cookbook dependency graph via metadata.rb (so I can
      do it per node here and not need to use an environment for that)
    • defines the interface for the user (i.e. README, attributes, etc…)
  • the environments should contain only environment specific attribute
    overrides + the versions of the “top-level” cookbooks

Since I’m wrapping other cookbooks and setting their attributes in (the
"top-level" cookbook’s) recipe, I’m sometimes faced with the “computed
attributes problem”. I tackle that by reloading the wrapped attributes file
between setting the attributes and including the recipe, e.g. here:

https://docs.chef.io/chef/essentials_cookbook_recipes.html#reload-attributes

Also, my environments often share the same data, i.e. common attributes
that are environmental data but still the same for many or even all of my
environments. Ranjib recently posted a nice way on how to deal with that:
https://gist.github.com/ranjib/2f7e1482a3ac8f6e5bfc

Hope that helps or at least gives you some more ideas. There are many ways
and even more opinions :wink:

Cheers,
Torben

On Fri, Jun 19, 2015 at 4:22 PM, Douglas Garstang <doug.garstang@gmail.com

wrote:

If I was going to put move my environmental attributes into cookbooks…

When deployed to an instance, the top level run list would need to
include the environmental run list. That’s ok.

However, when testing, this would mean that each cookbook would need to
either include the environment cookbook (ie #include_recipe “env-dev”) and
refer to it in the metadata.rb and Berksfile files, OR I’d have to
specifically put each attribute into the Vagrantfile’s json data.

This might work for simple cookbooks, but it would mean the cookbook is
not being effectively tested. The attributes I’m testing against aren’t
what really gets deployed to an instance. I also have to maintain two
copies of the attributes and the worst case scenario is where the attribute
is set in both places but the value in the Vagranrfile is incorrect and
therefore leads to the incorrect assumption that the cookbook is working
correctly.

For my base cookbook it gets even worse. It would need to have every
single attribute from every cookbook it includes put into the Vagrantfile’s
json data. This just does not scale.

It doesn’t seem like the benefits of gaining some revision control
outweigh the maintenance disadvantages here. For all those that espouse the
use of putting environment attributes into a cookbook how do you get around
this?

Doug


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com

Torben,

We’re definitely going along the path of using environments for dev, qa,
stage, prod etc. What you posted above was really useful, except that it
doesn’t help with the testing issue which is kinda the blocker here. If I
can’t test it, well I can’t really use it. :frowning:

Doug.

On Sun, Jun 21, 2015 at 6:39 AM, Torben Knerr mail@tknerr.de wrote:

Hey Doug,

I believe you first have to decide for what purpose you actually want to
use Chef environments, and this decision then limits the options for
technically implementing this.

For me the key decision is:
a) if you are going to use the berkflow / the environment cookbook pattern
with berks apply environment, then this means you will end up with one
environment per node
b) if you are going to use environments for dev / test / staging / prod
stages, you probably want to share environmental attributes across many
nodes

Personally, I prefer b) and try to stick to the following principles:

  • use environments only for supplying environment specific data
  • don’t use environment for composing a node’s run list or other node
    specific internals
  • instead I use a single “top-level” wrapper cookbook per node, which
    • glues together the run list
    • sets attributes for configuring the wrapped cookbooks
    • sometimes adds addional glue code
    • locks the whole cookbook dependency graph via metadata.rb (so I can
      do it per node here and not need to use an environment for that)
    • defines the interface for the user (i.e. README, attributes, etc…)
  • the environments should contain only environment specific attribute
    overrides + the versions of the “top-level” cookbooks

Since I’m wrapping other cookbooks and setting their attributes in (the
"top-level" cookbook’s) recipe, I’m sometimes faced with the “computed
attributes problem”. I tackle that by reloading the wrapped attributes file
between setting the attributes and including the recipe, e.g. here:

https://docs.chef.io/chef/essentials_cookbook_recipes.html#reload-attributes

Also, my environments often share the same data, i.e. common attributes
that are environmental data but still the same for many or even all of my
environments. Ranjib recently posted a nice way on how to deal with that:
https://gist.github.com/ranjib/2f7e1482a3ac8f6e5bfc

Hope that helps or at least gives you some more ideas. There are many ways
and even more opinions :wink:

Cheers,
Torben

On Fri, Jun 19, 2015 at 4:22 PM, Douglas Garstang <doug.garstang@gmail.com

wrote:

If I was going to put move my environmental attributes into cookbooks…

When deployed to an instance, the top level run list would need to
include the environmental run list. That’s ok.

However, when testing, this would mean that each cookbook would need to
either include the environment cookbook (ie #include_recipe “env-dev”) and
refer to it in the metadata.rb and Berksfile files, OR I’d have to
specifically put each attribute into the Vagrantfile’s json data.

This might work for simple cookbooks, but it would mean the cookbook is
not being effectively tested. The attributes I’m testing against aren’t
what really gets deployed to an instance. I also have to maintain two
copies of the attributes and the worst case scenario is where the attribute
is set in both places but the value in the Vagranrfile is incorrect and
therefore leads to the incorrect assumption that the cookbook is working
correctly.

For my base cookbook it gets even worse. It would need to have every
single attribute from every cookbook it includes put into the Vagrantfile’s
json data. This just does not scale.

It doesn’t seem like the benefits of gaining some revision control
outweigh the maintenance disadvantages here. For all those that espouse the
use of putting environment attributes into a cookbook how do you get around
this?

Doug


Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

May you explain why you can’t have multiple chef.add_recipe in your
vagrant file ?

I really don’t understand where you’re stuck.

Le 2015-06-22 17:51, Douglas Garstang a écrit :

Torben,

We’re definitely going along the path of using environments for dev, qa, stage, prod etc. What you posted above was really useful, except that it doesn’t help with the testing issue which is kinda the blocker here. If I can’t test it, well I can’t really use it. :frowning:

Doug.

On Sun, Jun 21, 2015 at 6:39 AM, Torben Knerr mail@tknerr.de wrote:

Hey Doug,

I believe you first have to decide for what purpose you actually want to use Chef environments, and this decision then limits the options for technically implementing this.

For me the key decision is:
a) if you are going to use the berkflow / the environment cookbook pattern with berks apply environment, then this means you will end up with one environment per node
b) if you are going to use environments for dev / test / staging / prod stages, you probably want to share environmental attributes across many nodes

Personally, I prefer b) and try to stick to the following principles: - use environments only for supplying environment specific data

  • don’t use environment for composing a node’s run list or other node specific internals
  • instead I use a single “top-level” wrapper cookbook per node, which
  • glues together the run list
  • sets attributes for configuring the wrapped cookbooks
  • sometimes adds addional glue code
  • locks the whole cookbook dependency graph via metadata.rb (so I can do it per node here and not need to use an environment for that)
  • defines the interface for the user (i.e. README, attributes, etc…)
  • the environments should contain only environment specific attribute overrides + the versions of the “top-level” cookbooks

Since I’m wrapping other cookbooks and setting their attributes in (the “top-level” cookbook’s) recipe, I’m sometimes faced with the “computed attributes problem”. I tackle that by reloading the wrapped attributes file between setting the attributes and including the recipe, e.g. here:
https://docs.chef.io/chef/essentials_cookbook_recipes.html#reload-attributes [1]

Also, my environments often share the same data, i.e. common attributes that are environmental data but still the same for many or even all of my environments. Ranjib recently posted a nice way on how to deal with that:
https://gist.github.com/ranjib/2f7e1482a3ac8f6e5bfc [2]

Hope that helps or at least gives you some more ideas. There are many ways and even more opinions :wink:

Cheers,
Torben

On Fri, Jun 19, 2015 at 4:22 PM, Douglas Garstang doug.garstang@gmail.com wrote:

If I was going to put move my environmental attributes into cookbooks…

When deployed to an instance, the top level run list would need to include the environmental run list. That’s ok.

However, when testing, this would mean that each cookbook would need to either include the environment cookbook (ie #include_recipe “env-dev”) and refer to it in the metadata.rb and Berksfile files, OR I’d have to specifically put each attribute into the Vagrantfile’s json data.

This might work for simple cookbooks, but it would mean the cookbook is not being effectively tested. The attributes I’m testing against aren’t what really gets deployed to an instance. I also have to maintain two copies of the attributes and the worst case scenario is where the attribute is set in both places but the value in the Vagranrfile is incorrect and therefore leads to the incorrect assumption that the cookbook is working correctly.

For my base cookbook it gets even worse. It would need to have every single attribute from every cookbook it includes put into the Vagrantfile’s json data. This just does not scale.

It doesn’t seem like the benefits of gaining some revision control outweigh the maintenance disadvantages here. For all those that espouse the use of putting environment attributes into a cookbook how do you get around this?

Doug

Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang [3]
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

Links:

[1]
https://docs.chef.io/chef/essentials_cookbook_recipes.html#reload-attributes
[2] https://gist.github.com/ranjib/2f7e1482a3ac8f6e5bfc
[3] http://www.linkedin.com/in/garstang

Hey Doug,

as for the testing thing: if you’d specify environmental attributes in an
environment json / rb file, you could also refer to this in your
Vagrantfile instead of using an env recipe or json attributes directly (see
"environment" property):

That way you wouldn’t need to duplicate them.

HTH,
Torben

On Mon, Jun 22, 2015 at 5:51 PM, Douglas Garstang doug.garstang@gmail.com
wrote:

Torben,

We’re definitely going along the path of using environments for dev, qa,
stage, prod etc. What you posted above was really useful, except that it
doesn’t help with the testing issue which is kinda the blocker here. If I
can’t test it, well I can’t really use it. :frowning:

Doug.

On Sun, Jun 21, 2015 at 6:39 AM, Torben Knerr mail@tknerr.de wrote:

Hey Doug,

I believe you first have to decide for what purpose you actually want to
use Chef environments, and this decision then limits the options for
technically implementing this.

For me the key decision is:
a) if you are going to use the berkflow / the environment cookbook
pattern with berks apply environment, then this means you will end up
with one environment per node
b) if you are going to use environments for dev / test / staging / prod
stages, you probably want to share environmental attributes across many
nodes

Personally, I prefer b) and try to stick to the following principles:

  • use environments only for supplying environment specific data
  • don’t use environment for composing a node’s run list or other node
    specific internals
  • instead I use a single “top-level” wrapper cookbook per node, which
    • glues together the run list
    • sets attributes for configuring the wrapped cookbooks
    • sometimes adds addional glue code
    • locks the whole cookbook dependency graph via metadata.rb (so I can
      do it per node here and not need to use an environment for that)
    • defines the interface for the user (i.e. README, attributes, etc…)
  • the environments should contain only environment specific attribute
    overrides + the versions of the “top-level” cookbooks

Since I’m wrapping other cookbooks and setting their attributes in (the
"top-level" cookbook’s) recipe, I’m sometimes faced with the “computed
attributes problem”. I tackle that by reloading the wrapped attributes file
between setting the attributes and including the recipe, e.g. here:

https://docs.chef.io/chef/essentials_cookbook_recipes.html#reload-attributes

Also, my environments often share the same data, i.e. common attributes
that are environmental data but still the same for many or even all of my
environments. Ranjib recently posted a nice way on how to deal with that:
https://gist.github.com/ranjib/2f7e1482a3ac8f6e5bfc

Hope that helps or at least gives you some more ideas. There are many
ways and even more opinions :wink:

Cheers,
Torben

On Fri, Jun 19, 2015 at 4:22 PM, Douglas Garstang <
doug.garstang@gmail.com> wrote:

If I was going to put move my environmental attributes into cookbooks…

When deployed to an instance, the top level run list would need to
include the environmental run list. That’s ok.

However, when testing, this would mean that each cookbook would need to
either include the environment cookbook (ie #include_recipe “env-dev”) and
refer to it in the metadata.rb and Berksfile files, OR I’d have to
specifically put each attribute into the Vagrantfile’s json data.

This might work for simple cookbooks, but it would mean the cookbook is
not being effectively tested. The attributes I’m testing against aren’t
what really gets deployed to an instance. I also have to maintain two
copies of the attributes and the worst case scenario is where the attribute
is set in both places but the value in the Vagranrfile is incorrect and
therefore leads to the incorrect assumption that the cookbook is working
correctly.

For my base cookbook it gets even worse. It would need to have every
single attribute from every cookbook it includes put into the Vagrantfile’s
json data. This just does not scale.

It doesn’t seem like the benefits of gaining some revision control
outweigh the maintenance disadvantages here. For all those that espouse the
use of putting environment attributes into a cookbook how do you get around
this?

Doug


Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

Hi Yoshi,

ah, interesting, was not aware of the fact that it would apply the cookbook
locks to an existing environment while keeping the env attributes. That
gets me some new thoughts.

However, that would still mean I have one environment per node (or "role"
at least), e.g.: “prod_lb”, “prod_web”, “prod_db” etc…, right?

If so, that would also mean I have to duplicate the prod-specific
attributes here (3x times in the example above), right?

I realize I may could get around this using the YAML include approach
Ranjib shared, which would keep it single-sourced, but still I’d need to
update each of the “prod_*” environments from that single source.

Last but not Ieast might have an impact on searches or environment checks
as well, but that could be dealt with easily I guess, at least if it’s
happening in your own recipes where you are in control of it.

If there’s a better way to work with berksflow / berks apply while keeping
the intuitive environment semantics, I’d be happy to learn more about it.

Cheers,
Torben

On Mon, Jun 22, 2015 at 5:33 PM, Yoshi Spendiff <
yoshi.spendiff@indochino.com> wrote:

Hi Torben,

Not sure if you know, but the A/B choice you stated above aren’t actually
mutually exclusive. If you want to use berkflow/berks apply to apply
cookbook version locks to an environment that doesn’t stop you supplying
normal environment variables to the environment. Berkflow/berks apply
doesn’t change any attributes other than the version locks.

Cheers,
Yoshi

On Sun, Jun 21, 2015 at 6:39 AM, Torben Knerr mail@tknerr.de wrote:

Hey Doug,

I believe you first have to decide for what purpose you actually want to
use Chef environments, and this decision then limits the options for
technically implementing this.

For me the key decision is:
a) if you are going to use the berkflow / the environment cookbook
pattern with berks apply environment, then this means you will end up
with one environment per node
b) if you are going to use environments for dev / test / staging / prod
stages, you probably want to share environmental attributes across many
nodes

Personally, I prefer b) and try to stick to the following principles:

  • use environments only for supplying environment specific data
  • don’t use environment for composing a node’s run list or other node
    specific internals
  • instead I use a single “top-level” wrapper cookbook per node, which
    • glues together the run list
    • sets attributes for configuring the wrapped cookbooks
    • sometimes adds addional glue code
    • locks the whole cookbook dependency graph via metadata.rb (so I can
      do it per node here and not need to use an environment for that)
    • defines the interface for the user (i.e. README, attributes, etc…)
  • the environments should contain only environment specific attribute
    overrides + the versions of the “top-level” cookbooks

Since I’m wrapping other cookbooks and setting their attributes in (the
"top-level" cookbook’s) recipe, I’m sometimes faced with the “computed
attributes problem”. I tackle that by reloading the wrapped attributes file
between setting the attributes and including the recipe, e.g. here:

https://docs.chef.io/chef/essentials_cookbook_recipes.html#reload-attributes

Also, my environments often share the same data, i.e. common attributes
that are environmental data but still the same for many or even all of my
environments. Ranjib recently posted a nice way on how to deal with that:
https://gist.github.com/ranjib/2f7e1482a3ac8f6e5bfc

Hope that helps or at least gives you some more ideas. There are many
ways and even more opinions :wink:

Cheers,
Torben

On Fri, Jun 19, 2015 at 4:22 PM, Douglas Garstang <
doug.garstang@gmail.com> wrote:

If I was going to put move my environmental attributes into cookbooks…

When deployed to an instance, the top level run list would need to
include the environmental run list. That’s ok.

However, when testing, this would mean that each cookbook would need to
either include the environment cookbook (ie #include_recipe “env-dev”) and
refer to it in the metadata.rb and Berksfile files, OR I’d have to
specifically put each attribute into the Vagrantfile’s json data.

This might work for simple cookbooks, but it would mean the cookbook is
not being effectively tested. The attributes I’m testing against aren’t
what really gets deployed to an instance. I also have to maintain two
copies of the attributes and the worst case scenario is where the attribute
is set in both places but the value in the Vagranrfile is incorrect and
therefore leads to the incorrect assumption that the cookbook is working
correctly.

For my base cookbook it gets even worse. It would need to have every
single attribute from every cookbook it includes put into the Vagrantfile’s
json data. This just does not scale.

It doesn’t seem like the benefits of gaining some revision control
outweigh the maintenance disadvantages here. For all those that espouse the
use of putting environment attributes into a cookbook how do you get around
this?

Doug


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com

In the demonstration that Jamie Windsor did for berkflow at Chefcon he used
a role/environment cookbook with multiple roles rather than single role
cookbooks.

I think in your case if you have a cookbook per role then the only way you
can make it work that way is if cookbooks common to your role cookbooks all
have the same version requirement (or no specific version requirement).
I.e. if your lb cookbook needs nginx 1.2.3 but your web cookbook needs
1.2.4, and those versions are locked, then there’s not much you can do as
you’ll have conflicting version locks in your environment.

If you don’t have one role cookbook per role or you don’t have these
specific version requirements then it’s pretty easy. You’d make a top level
cookbook to wrap your role cookbooks which would be your environment
cookbook (or if you have one role cookbook with multiple roles then this
could be your environment cookbook) then just add the Berksfile.lock from
the cookbook to source control, make sure it’s not in chefignore and that’s
your one source of version locks which you can apply to all environments.
You then add the recipes from the environment cookbooks to your nodes/roles.

As there’s no actual logic in the environment cookbook whenever you want to
update your environments you bump the metadata on the environment cookbook,
run berks update and then upload it to your chef server. You can then use
berkflow to apply the cookbook from your chef server to an environment,
which will apply the locks only.

You can now use your environments for whatever attributes you want.

On Mon, Jun 22, 2015 at 10:43 PM, Torben Knerr mail@tknerr.de wrote:

Hi Yoshi,

ah, interesting, was not aware of the fact that it would apply the
cookbook locks to an existing environment while keeping the env attributes.
That gets me some new thoughts.

However, that would still mean I have one environment per node (or "role"
at least), e.g.: “prod_lb”, “prod_web”, “prod_db” etc…, right?

If so, that would also mean I have to duplicate the prod-specific
attributes here (3x times in the example above), right?

I realize I may could get around this using the YAML include approach
Ranjib shared, which would keep it single-sourced, but still I’d need to
update each of the “prod_*” environments from that single source.

Last but not Ieast might have an impact on searches or environment checks
as well, but that could be dealt with easily I guess, at least if it’s
happening in your own recipes where you are in control of it.

If there’s a better way to work with berksflow / berks apply while keeping
the intuitive environment semantics, I’d be happy to learn more about it.

Cheers,
Torben

On Mon, Jun 22, 2015 at 5:33 PM, Yoshi Spendiff <
yoshi.spendiff@indochino.com> wrote:

Hi Torben,

Not sure if you know, but the A/B choice you stated above aren’t actually
mutually exclusive. If you want to use berkflow/berks apply to apply
cookbook version locks to an environment that doesn’t stop you supplying
normal environment variables to the environment. Berkflow/berks apply
doesn’t change any attributes other than the version locks.

Cheers,
Yoshi

On Sun, Jun 21, 2015 at 6:39 AM, Torben Knerr mail@tknerr.de wrote:

Hey Doug,

I believe you first have to decide for what purpose you actually want to
use Chef environments, and this decision then limits the options for
technically implementing this.

For me the key decision is:
a) if you are going to use the berkflow / the environment cookbook
pattern with berks apply environment, then this means you will end up
with one environment per node
b) if you are going to use environments for dev / test / staging / prod
stages, you probably want to share environmental attributes across many
nodes

Personally, I prefer b) and try to stick to the following principles:

  • use environments only for supplying environment specific data
  • don’t use environment for composing a node’s run list or other node
    specific internals
  • instead I use a single “top-level” wrapper cookbook per node, which
    • glues together the run list
    • sets attributes for configuring the wrapped cookbooks
    • sometimes adds addional glue code
    • locks the whole cookbook dependency graph via metadata.rb (so I can
      do it per node here and not need to use an environment for that)
    • defines the interface for the user (i.e. README, attributes, etc…)
  • the environments should contain only environment specific attribute
    overrides + the versions of the “top-level” cookbooks

Since I’m wrapping other cookbooks and setting their attributes in (the
"top-level" cookbook’s) recipe, I’m sometimes faced with the “computed
attributes problem”. I tackle that by reloading the wrapped attributes file
between setting the attributes and including the recipe, e.g. here:

https://docs.chef.io/chef/essentials_cookbook_recipes.html#reload-attributes

Also, my environments often share the same data, i.e. common attributes
that are environmental data but still the same for many or even all of my
environments. Ranjib recently posted a nice way on how to deal with that:
https://gist.github.com/ranjib/2f7e1482a3ac8f6e5bfc

Hope that helps or at least gives you some more ideas. There are many
ways and even more opinions :wink:

Cheers,
Torben

On Fri, Jun 19, 2015 at 4:22 PM, Douglas Garstang <
doug.garstang@gmail.com> wrote:

If I was going to put move my environmental attributes into cookbooks…

When deployed to an instance, the top level run list would need to
include the environmental run list. That’s ok.

However, when testing, this would mean that each cookbook would need to
either include the environment cookbook (ie #include_recipe “env-dev”) and
refer to it in the metadata.rb and Berksfile files, OR I’d have to
specifically put each attribute into the Vagrantfile’s json data.

This might work for simple cookbooks, but it would mean the cookbook is
not being effectively tested. The attributes I’m testing against aren’t
what really gets deployed to an instance. I also have to maintain two
copies of the attributes and the worst case scenario is where the attribute
is set in both places but the value in the Vagranrfile is incorrect and
therefore leads to the incorrect assumption that the cookbook is working
correctly.

For my base cookbook it gets even worse. It would need to have every
single attribute from every cookbook it includes put into the Vagrantfile’s
json data. This just does not scale.

It doesn’t seem like the benefits of gaining some revision control
outweigh the maintenance disadvantages here. For all those that espouse the
use of putting environment attributes into a cookbook how do you get around
this?

Doug


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com

I.e. if your lb cookbook needs nginx 1.2.3 but your web cookbook needs
1.2.4, and those versions are locked, then there’s not much you can do as
you’ll have conflicting version locks in your environment.

That’s exactly the freedom I want, and the reason why I settled with one
"top-level" cookbook per role (guess I should call that "role cookbook"
then ;-)) which locks it’s downstream cookbooks on its own via its
metadata.rb

Thanks for the summary though! Even though berkflow doesn’t fit for my
case, I still learned something new :slight_smile:
Am 23.06.2015 17:20 schrieb “Yoshi Spendiff” yoshi.spendiff@indochino.com:

In the demonstration that Jamie Windsor did for berkflow at Chefcon he
used a role/environment cookbook with multiple roles rather than single
role cookbooks.

https://www.youtube.com/watch?v=Dq_vGxd-jps

I think in your case if you have a cookbook per role then the only way you
can make it work that way is if cookbooks common to your role cookbooks all
have the same version requirement (or no specific version requirement).
I.e. if your lb cookbook needs nginx 1.2.3 but your web cookbook needs
1.2.4, and those versions are locked, then there’s not much you can do as
you’ll have conflicting version locks in your environment.

If you don’t have one role cookbook per role or you don’t have these
specific version requirements then it’s pretty easy. You’d make a top level
cookbook to wrap your role cookbooks which would be your environment
cookbook (or if you have one role cookbook with multiple roles then this
could be your environment cookbook) then just add the Berksfile.lock from
the cookbook to source control, make sure it’s not in chefignore and that’s
your one source of version locks which you can apply to all environments.
You then add the recipes from the environment cookbooks to your nodes/roles.

As there’s no actual logic in the environment cookbook whenever you want
to update your environments you bump the metadata on the environment
cookbook, run berks update and then upload it to your chef server. You can
then use berkflow to apply the cookbook from your chef server to an
environment, which will apply the locks only.

You can now use your environments for whatever attributes you want.

On Mon, Jun 22, 2015 at 10:43 PM, Torben Knerr mail@tknerr.de wrote:

Hi Yoshi,

ah, interesting, was not aware of the fact that it would apply the
cookbook locks to an existing environment while keeping the env attributes.
That gets me some new thoughts.

However, that would still mean I have one environment per node (or "role"
at least), e.g.: “prod_lb”, “prod_web”, “prod_db” etc…, right?

If so, that would also mean I have to duplicate the prod-specific
attributes here (3x times in the example above), right?

I realize I may could get around this using the YAML include approach
Ranjib shared, which would keep it single-sourced, but still I’d need to
update each of the “prod_*” environments from that single source.

Last but not Ieast might have an impact on searches or environment checks
as well, but that could be dealt with easily I guess, at least if it’s
happening in your own recipes where you are in control of it.

If there’s a better way to work with berksflow / berks apply while
keeping the intuitive environment semantics, I’d be happy to learn more
about it.

Cheers,
Torben

On Mon, Jun 22, 2015 at 5:33 PM, Yoshi Spendiff <
yoshi.spendiff@indochino.com> wrote:

Hi Torben,

Not sure if you know, but the A/B choice you stated above aren’t
actually mutually exclusive. If you want to use berkflow/berks apply to
apply cookbook version locks to an environment that doesn’t stop you
supplying normal environment variables to the environment. Berkflow/berks
apply doesn’t change any attributes other than the version locks.

Cheers,
Yoshi

On Sun, Jun 21, 2015 at 6:39 AM, Torben Knerr mail@tknerr.de wrote:

Hey Doug,

I believe you first have to decide for what purpose you actually want
to use Chef environments, and this decision then limits the options for
technically implementing this.

For me the key decision is:
a) if you are going to use the berkflow / the environment cookbook
pattern with berks apply environment, then this means you will end up
with one environment per node
b) if you are going to use environments for dev / test / staging / prod
stages, you probably want to share environmental attributes across many
nodes

Personally, I prefer b) and try to stick to the following principles:

  • use environments only for supplying environment specific data
  • don’t use environment for composing a node’s run list or other node
    specific internals
  • instead I use a single “top-level” wrapper cookbook per node, which
    • glues together the run list
    • sets attributes for configuring the wrapped cookbooks
    • sometimes adds addional glue code
    • locks the whole cookbook dependency graph via metadata.rb (so I
      can do it per node here and not need to use an environment for that)
    • defines the interface for the user (i.e. README, attributes, etc…)
  • the environments should contain only environment specific attribute
    overrides + the versions of the “top-level” cookbooks

Since I’m wrapping other cookbooks and setting their attributes in (the
"top-level" cookbook’s) recipe, I’m sometimes faced with the “computed
attributes problem”. I tackle that by reloading the wrapped attributes file
between setting the attributes and including the recipe, e.g. here:

https://docs.chef.io/chef/essentials_cookbook_recipes.html#reload-attributes

Also, my environments often share the same data, i.e. common attributes
that are environmental data but still the same for many or even all of my
environments. Ranjib recently posted a nice way on how to deal with that:
https://gist.github.com/ranjib/2f7e1482a3ac8f6e5bfc

Hope that helps or at least gives you some more ideas. There are many
ways and even more opinions :wink:

Cheers,
Torben

On Fri, Jun 19, 2015 at 4:22 PM, Douglas Garstang <
doug.garstang@gmail.com> wrote:

If I was going to put move my environmental attributes into
cookbooks…

When deployed to an instance, the top level run list would need to
include the environmental run list. That’s ok.

However, when testing, this would mean that each cookbook would need
to either include the environment cookbook (ie #include_recipe “env-dev”)
and refer to it in the metadata.rb and Berksfile files, OR I’d have to
specifically put each attribute into the Vagrantfile’s json data.

This might work for simple cookbooks, but it would mean the cookbook
is not being effectively tested. The attributes I’m testing against aren’t
what really gets deployed to an instance. I also have to maintain two
copies of the attributes and the worst case scenario is where the attribute
is set in both places but the value in the Vagranrfile is incorrect and
therefore leads to the incorrect assumption that the cookbook is working
correctly.

For my base cookbook it gets even worse. It would need to have every
single attribute from every cookbook it includes put into the Vagrantfile’s
json data. This just does not scale.

It doesn’t seem like the benefits of gaining some revision control
outweigh the maintenance disadvantages here. For all those that espouse the
use of putting environment attributes into a cookbook how do you get around
this?

Doug


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com

No worries mate, good to have the cards in your back pocket if you need
them!

On Tue, Jun 23, 2015 at 8:44 AM, Torben Knerr mail@tknerr.de wrote:

I.e. if your lb cookbook needs nginx 1.2.3 but your web cookbook needs
1.2.4, and those versions are locked, then there’s not much you can do as
you’ll have conflicting version locks in your environment.

That’s exactly the freedom I want, and the reason why I settled with one
"top-level" cookbook per role (guess I should call that "role cookbook"
then ;-)) which locks it’s downstream cookbooks on its own via its
metadata.rb

Thanks for the summary though! Even though berkflow doesn’t fit for my
case, I still learned something new :slight_smile:
Am 23.06.2015 17:20 schrieb “Yoshi Spendiff” <yoshi.spendiff@indochino.com

:

In the demonstration that Jamie Windsor did for berkflow at Chefcon he
used a role/environment cookbook with multiple roles rather than single
role cookbooks.

https://www.youtube.com/watch?v=Dq_vGxd-jps

I think in your case if you have a cookbook per role then the only way
you can make it work that way is if cookbooks common to your role cookbooks
all have the same version requirement (or no specific version requirement).
I.e. if your lb cookbook needs nginx 1.2.3 but your web cookbook needs
1.2.4, and those versions are locked, then there’s not much you can do as
you’ll have conflicting version locks in your environment.

If you don’t have one role cookbook per role or you don’t have these
specific version requirements then it’s pretty easy. You’d make a top level
cookbook to wrap your role cookbooks which would be your environment
cookbook (or if you have one role cookbook with multiple roles then this
could be your environment cookbook) then just add the Berksfile.lock from
the cookbook to source control, make sure it’s not in chefignore and that’s
your one source of version locks which you can apply to all environments.
You then add the recipes from the environment cookbooks to your nodes/roles.

As there’s no actual logic in the environment cookbook whenever you want
to update your environments you bump the metadata on the environment
cookbook, run berks update and then upload it to your chef server. You can
then use berkflow to apply the cookbook from your chef server to an
environment, which will apply the locks only.

You can now use your environments for whatever attributes you want.

On Mon, Jun 22, 2015 at 10:43 PM, Torben Knerr mail@tknerr.de wrote:

Hi Yoshi,

ah, interesting, was not aware of the fact that it would apply the
cookbook locks to an existing environment while keeping the env attributes.
That gets me some new thoughts.

However, that would still mean I have one environment per node (or
"role" at least), e.g.: “prod_lb”, “prod_web”, “prod_db” etc…, right?

If so, that would also mean I have to duplicate the prod-specific
attributes here (3x times in the example above), right?

I realize I may could get around this using the YAML include approach
Ranjib shared, which would keep it single-sourced, but still I’d need to
update each of the “prod_*” environments from that single source.

Last but not Ieast might have an impact on searches or environment
checks as well, but that could be dealt with easily I guess, at least if
it’s happening in your own recipes where you are in control of it.

If there’s a better way to work with berksflow / berks apply while
keeping the intuitive environment semantics, I’d be happy to learn more
about it.

Cheers,
Torben

On Mon, Jun 22, 2015 at 5:33 PM, Yoshi Spendiff <
yoshi.spendiff@indochino.com> wrote:

Hi Torben,

Not sure if you know, but the A/B choice you stated above aren’t
actually mutually exclusive. If you want to use berkflow/berks apply to
apply cookbook version locks to an environment that doesn’t stop you
supplying normal environment variables to the environment. Berkflow/berks
apply doesn’t change any attributes other than the version locks.

Cheers,
Yoshi

On Sun, Jun 21, 2015 at 6:39 AM, Torben Knerr mail@tknerr.de wrote:

Hey Doug,

I believe you first have to decide for what purpose you actually want
to use Chef environments, and this decision then limits the options for
technically implementing this.

For me the key decision is:
a) if you are going to use the berkflow / the environment cookbook
pattern with berks apply environment, then this means you will end up
with one environment per node
b) if you are going to use environments for dev / test / staging /
prod stages, you probably want to share environmental attributes across
many nodes

Personally, I prefer b) and try to stick to the following principles:

  • use environments only for supplying environment specific data
  • don’t use environment for composing a node’s run list or other node
    specific internals
  • instead I use a single “top-level” wrapper cookbook per node, which
    • glues together the run list
    • sets attributes for configuring the wrapped cookbooks
    • sometimes adds addional glue code
    • locks the whole cookbook dependency graph via metadata.rb (so I
      can do it per node here and not need to use an environment for that)
    • defines the interface for the user (i.e. README, attributes,
      etc…)
  • the environments should contain only environment specific attribute
    overrides + the versions of the “top-level” cookbooks

Since I’m wrapping other cookbooks and setting their attributes in
(the “top-level” cookbook’s) recipe, I’m sometimes faced with the “computed
attributes problem”. I tackle that by reloading the wrapped attributes file
between setting the attributes and including the recipe, e.g. here:

https://docs.chef.io/chef/essentials_cookbook_recipes.html#reload-attributes

Also, my environments often share the same data, i.e. common
attributes that are environmental data but still the same for many or even
all of my environments. Ranjib recently posted a nice way on how to deal
with that:
https://gist.github.com/ranjib/2f7e1482a3ac8f6e5bfc

Hope that helps or at least gives you some more ideas. There are many
ways and even more opinions :wink:

Cheers,
Torben

On Fri, Jun 19, 2015 at 4:22 PM, Douglas Garstang <
doug.garstang@gmail.com> wrote:

If I was going to put move my environmental attributes into
cookbooks…

When deployed to an instance, the top level run list would need to
include the environmental run list. That’s ok.

However, when testing, this would mean that each cookbook would need
to either include the environment cookbook (ie #include_recipe “env-dev”)
and refer to it in the metadata.rb and Berksfile files, OR I’d have to
specifically put each attribute into the Vagrantfile’s json data.

This might work for simple cookbooks, but it would mean the cookbook
is not being effectively tested. The attributes I’m testing against aren’t
what really gets deployed to an instance. I also have to maintain two
copies of the attributes and the worst case scenario is where the attribute
is set in both places but the value in the Vagranrfile is incorrect and
therefore leads to the incorrect assumption that the cookbook is working
correctly.

For my base cookbook it gets even worse. It would need to have every
single attribute from every cookbook it includes put into the Vagrantfile’s
json data. This just does not scale.

It doesn’t seem like the benefits of gaining some revision control
outweigh the maintenance disadvantages here. For all those that espouse the
use of putting environment attributes into a cookbook how do you get around
this?

Doug


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com