Best practices


#1

Hi all,

I am looking for some tips on chef best practices and having a hard time finding anything other than highly focused docs and tutorials on the web. Here are some of the issues I am struggling with:

  •     What granularity is best for recipes?  When does it make sense to use some of the more complex ideas like depends, replaces, recommends/suggest, etc
    
  •     How to handle uninstallation? As different cookbooks, or as a different recipe, or as something that will be done under the covers by a replacement recipe...
    
  •     How to structure roles in a generic and useful way for workstations and servers.
    
  •     I find that there is a certain amount of overlap in definition between node, client, and user. In fact bootstrapping creates a node and a client for each machine. Is there a good way to think about how these 3 relate to each other.
    
  •     How to best manage recipe versioning using versions, environments, version constraints, etc. I am specifically wondering about being able to upload recipes that are being developed and tested without the fear that they will end up in production. I would like to stay away from having to manage 2 instances of chef.
    

I realise that this is a whole lot of stuff in one email… this is a list of questions that has been growing in the last few weeks as I explore and plan for chef in our environment.

Cheers,
Flo

LEGAL DISCLAIMER
This communication and any attached documents are strictly confidential and/or legally privileged and they may not be used or disclosed by someone who is not a named recipient. If you have received this electronic communication in error please notify the sender by replying to this electronic communication inserting the word “misdirected” as the subject and delete this communication from your system.


#2

I’ll take a stab at a few of these. Answers inline.

On Tue, Jul 16, 2013 at 5:02 AM, Florian Hehlen <
Florian.Hehlen@mri-group.com> wrote:

Hi all,

I am looking for some tips on chef best practices and having a hard time
finding anything other than highly focused docs and tutorials on the web.
Here are some of the issues I am struggling with:

· What granularity is best for recipes? When does it make sense
to use some of the more complex ideas like depends, replaces,
recommends/suggest, etc

Granularity is really up to you, but I think it’s better to not have a
given cookbook or recipe do too much. As far as the various cookbook
metadata fields, I don’t think I’ve ever seen any of them other than
"depends" used. Do declare your dependencies, though.

· How to handle uninstallation? As different cookbooks, or as a
different recipe, or as something that will be done under the covers by a
replacement recipe…

It kind of depends on what your end goal is. I’ve never had the need to
specifically remove a package in order to get to by desired end-state. If
you do need that, I’d put

· How to structure roles in a generic and useful way for
workstations and servers.

Again, this depends on your needs. Roles are generally used to group a set
of related recipes, but some folks don’t use them at all.

· I find that there is a certain amount of overlap in definition
between node, client, and user. In fact bootstrapping creates a node and a
client for each machine. Is there a good way to think about how these 3
relate to each other.

So, nodes are “the thing you’re configuring” - they generally correspond to
"servers" or “ec2 instances”. Clients are “how you authenticate to the
Chef Server”, and you’re right: if you’re using a Chef server, they’ll
pretty much always be one-to-one (i think it’s technically possible to do
otherwise, but I wouldn’t). Users are really just a special kind of client
that also has a password for the WebUI - they generally correspond to
people (like you).

· How to best manage recipe versioning using versions,
environments, version constraints, etc. I am specifically wondering about
being able to upload recipes that are being developed and tested without
the fear that they will end up in production. I would like to stay away
from having to manage 2 instances of chef.

I realise that this is a whole lot of stuff in one email… this is a list
of questions that has been growing in the last few weeks as I explore and
plan for chef in our environment.

Cheers,

Flo

LEGAL DISCLAIMER
This communication and any attached documents are strictly confidential
and/or legally privileged and they may not be used or disclosed by someone
who is not a named recipient. If you have received this electronic
communication in error please notify the sender by replying to this
electronic communication inserting the word “misdirected” as the subject
and delete this communication from your system.

Hope that helps,

Matt Moretti


#3

For uninstallation, I like the idea of an “undo” recipe. For example,
RHEL flavors tend to install NFS as part of the base installation. When
I set out to do the nfs cookbook, I was sure to create an "nfs::undo"
run-once recipe http://community.opscode.com/cookbooks/nfs/source
which will clean up that annoyance. The annoyances cookbook is another
good example of reversing an undesired state on a managed node.

I think the idea of an undo recipe should be to step backwards logically
through the installed/running pattern. Run-once capability is a nice
thing to have for a “kickstart” type of role. For example, I may want
to undo nfs on every fresh installation. However, I don’t want the
"nfs::undo" recipe to permanently exist in the run_list and reverse the
state of an actual NFS server either.

Eric G. Wolfe
Senior Linux Administrator,
IT Infrastructure Systems

Marshall University Computing Services
Drinko Library 428-K
One John Marshall Dr.
Huntington, WV 25755
Phone: 304.942.3970
Email: eric.wolfe@marshall.edu

You will be recognized and honored as a community leader.

On 07/16/2013 12:50 PM, Matthew Moretti wrote:

I’ll take a stab at a few of these. Answers inline.

On Tue, Jul 16, 2013 at 5:02 AM, Florian Hehlen
<Florian.Hehlen@mri-group.com mailto:Florian.Hehlen@mri-group.com>
wrote:

Hi all,

I am looking for some tips on chef best practices and having a
hard time finding anything other than highly focused docs and
 tutorials on the web. Here are some of the issues I am struggling
with:

·What granularity is best for recipes?  When does it make sense to
use some of the more complex ideas like depends, replaces,
recommends/suggest, etc

Granularity is really up to you, but I think it’s better to not have a
given cookbook or recipe do too much. As far as the various cookbook
metadata fields, I don’t think I’ve ever seen any of them other than
"depends" used. Do declare your dependencies, though.

·How to handle uninstallation? As different cookbooks, or as a
different recipe, or as something that will be done under the
covers by a replacement recipe…

It kind of depends on what your end goal is. I’ve never had the need
to specifically remove a package in order to get to by desired
end-state. If you do need that, I’d put

·How to structure roles in a generic and useful way for
workstations and servers.

Again, this depends on your needs. Roles are generally used to group
a set of related recipes, but some folks don’t use them at all.

·I find that there is a certain amount of overlap in definition
between node, client, and user. In fact bootstrapping creates a
node and a client for each machine. Is there a good way to think
about how these 3 relate to each other.

So, nodes are “the thing you’re configuring” - they generally
correspond to “servers” or “ec2 instances”. Clients are “how you
authenticate to the Chef Server”, and you’re right: if you’re using a
Chef server, they’ll pretty much always be one-to-one (i think it’s
technically possible to do otherwise, but I wouldn’t). Users are
really just a special kind of client that also has a password for the
WebUI - they generally correspond to people (like you).

·How to best manage recipe versioning using versions,
environments, version constraints, etc. I am specifically
wondering about being able to upload recipes that are being
developed and tested without the fear that they will end up in
production. I would like to stay away from having to manage 2
instances of chef.

I realise that this is a whole lot of stuff in one email… this is
a list of questions that has been growing in the last few weeks as
I explore and plan for chef in our environment.

Cheers,

Flo



LEGAL DISCLAIMER
This communication and any attached documents are strictly
confidential and/or legally privileged and they may not be used or
disclosed by someone who is not a named recipient. If you have
received this electronic communication in error please notify the
sender by replying to this electronic communication inserting the
word "misdirected" as the subject and delete this communication
from your system.

Hope that helps,

Matt Moretti


#4

Hi,

On Tue, 16 Jul 2013 13:33:19 -0400 "Eric G. Wolfe"
eric.wolfe@marshall.edu wrote:

For uninstallation, I like the idea of an “undo” recipe. For
example, RHEL flavors tend to install NFS as part of the base
installation. When I set out to do the nfs cookbook, I was sure to
create an “nfs::undo” run-once recipe
http://community.opscode.com/cookbooks/nfs/source which will clean
up that annoyance. The annoyances cookbook is another good example
of reversing an undesired state on a managed node.

I think the idea of an undo recipe should be to step backwards
logically through the installed/running pattern. Run-once capability
is a nice thing to have for a “kickstart” type of role. For example,
I may want to undo nfs on every fresh installation. However, I don’t
want the “nfs::undo” recipe to permanently exist in the run_list and
reverse the state of an actual NFS server either.

Hm, but all recipe should be “run once” recipes. Regardless what the
state before is, the end state should always be the desired one. So as
it shouldn’t hurt to “install” nfs-server on each chef-client run, it
also shouldn’t hurt to uninstall nfs on each chef-client run.
Only when you apply your “nfs::uninstall” and “nfs::server” recipes on
the same node at the same time…

So as I don’t care when chef-client makes sure vim-nox is installed on
each system (to make sure my co-workers don’t leave me an unusable
machine), I also don’t care when chef-client makes sure that emacs
isn’t installed even when some co-worker decided to do a quick “apt-get
install” just because.

So I actually fail to see a special need for making sure recipes only
run once. No matter what they do, they should always lead to the
desired outcome, regardless whether that means doing something or not.

Have fun,

Arnold


#5

The crux of the matter is the conflicting state of two, or more,
opposing recipes. If the removal of the undo/uninstall recipe is
guarded so that it defers to the conflicting recipe only when there is a
conflict in the run_list, there should be no other issue with
idempotence within that recipe.

I would tend to disagree that every recipe should be "run once"
complete. While every recipe running only once and producing the
correct state would be certainly be optimal. The important point is to
eventually reach the correct state. If that takes two subsequent
runs, rather than only once (in certain cases) it should not be a deal
breaker so to speak.

Anyhow, thank you for the suggestion. I have taken action to guard
self-removal of the recipe, only when there is certain state conflict in
the run_list.

Eric G. Wolfe
Senior Linux Administrator,
IT Infrastructure Systems

Marshall University Computing Services
Drinko Library 428-K
One John Marshall Dr.
Huntington, WV 25755
Phone: 304.942.3970
Email: eric.wolfe@marshall.edu

You will be recognized and honored as a community leader.

On 07/16/2013 05:05 PM, Arnold Krille wrote:

Hi,

On Tue, 16 Jul 2013 13:33:19 -0400 "Eric G. Wolfe"
eric.wolfe@marshall.edu wrote:

For uninstallation, I like the idea of an “undo” recipe. For
example, RHEL flavors tend to install NFS as part of the base
installation. When I set out to do the nfs cookbook, I was sure to
create an “nfs::undo” run-once recipe
http://community.opscode.com/cookbooks/nfs/source which will clean
up that annoyance. The annoyances cookbook is another good example
of reversing an undesired state on a managed node.

I think the idea of an undo recipe should be to step backwards
logically through the installed/running pattern. Run-once capability
is a nice thing to have for a “kickstart” type of role. For example,
I may want to undo nfs on every fresh installation. However, I don’t
want the “nfs::undo” recipe to permanently exist in the run_list and
reverse the state of an actual NFS server either.
Hm, but all recipe should be “run once” recipes. Regardless what the
state before is, the end state should always be the desired one. So as
it shouldn’t hurt to “install” nfs-server on each chef-client run, it
also shouldn’t hurt to uninstall nfs on each chef-client run.
Only when you apply your “nfs::uninstall” and “nfs::server” recipes on
the same node at the same time…

So as I don’t care when chef-client makes sure vim-nox is installed on
each system (to make sure my co-workers don’t leave me an unusable
machine), I also don’t care when chef-client makes sure that emacs
isn’t installed even when some co-worker decided to do a quick “apt-get
install” just because.

So I actually fail to see a special need for making sure recipes only
run once. No matter what they do, they should always lead to the
desired outcome, regardless whether that means doing something or not.

Have fun,

Arnold


#6

Adding my 5 cents below… :slight_smile:

On Jul 16, 2013 6:50 PM, “Matthew Moretti” werebus@gmail.com wrote:

I’ll take a stab at a few of these. Answers inline.

On Tue, Jul 16, 2013 at 5:02 AM, Florian Hehlen <
Florian.Hehlen@mri-group.com> wrote:

Hi all,

I am looking for some tips on chef best practices and having a hard time
finding anything other than highly focused docs and tutorials on the web.
Here are some of the issues I am struggling with:

· What granularity is best for recipes? When does it make sense
to use some of the more complex ideas like depends, replaces,
recommends/suggest, etc

Granularity is really up to you, but I think it’s better to not have a
given cookbook or recipe do too much. As far as the various cookbook
metadata fields, I don’t think I’ve ever seen any of them other than
"depends" used. Do declare your dependencies, though.

As of Chef 11 you really must declare your dependencies via ‘depends’ in
metadata, otherwise recipes will not be loaded (this was different in Chef
10).

+1 for fine-grained cookbooks, i.e. on the granularity level of apache,
mysql etc… - pretty much what you find on the community side. Having this
level of granularity makes them reusable and composable.

When it comes to assembling your specific application by reusing /
composing and configuring a set of these cookbooks (which I tend to call
library cookbooks) then I would create a socalled "application cookbook"
rather than using roles (the benefit of an “application cookbook” is that
it can be properly versioned and dependency managed). My rule of thumb is
that one application cookbook represents a whole node. These kinds of
cookbooks are at a different level of granularity because they describe
your whole application rather than the single parts it is composed of. This
also makes it much less reusable than library cookbooks because they are
so specific to your use case / application (i.e. they tend to live in a
corporate git repo rather than on the community site)

· How to handle uninstallation? As different cookbooks, or as a
different recipe, or as something that will be done under the covers by a
replacement recipe…

It kind of depends on what your end goal is. I’ve never had the need to
specifically remove a package in order to get to by desired end-state. If
you do need that, I’d put

· How to structure roles in a generic and useful way for
workstations and servers.

Again, this depends on your needs. Roles are generally used to group a
set of related recipes, but some folks don’t use them at all.

Some folks = proponents of the application cookbook pattern (see above) :slight_smile:

See this blog post for a nice summary:

· I find that there is a certain amount of overlap in definition
between node, client, and user. In fact bootstrapping creates a node and a
client for each machine. Is there a good way to think about how these 3
relate to each other.

So, nodes are “the thing you’re configuring” - they generally correspond
to “servers” or “ec2 instances”. Clients are “how you authenticate to the
Chef Server”, and you’re right: if you’re using a Chef server, they’ll
pretty much always be one-to-one (i think it’s technically possible to do
otherwise, but I wouldn’t). Users are really just a special kind of client
that also has a password for the WebUI - they generally correspond to
people (like you).

· How to best manage recipe versioning using versions,
environments, version constraints, etc. I am specifically wondering about
being able to upload recipes that are being developed and tested without
the fear that they will end up in production. I would like to stay away
from having to manage 2 instances of chef.

I have a very specific opinion on that as I believe that you are doing it
wrong if you are using environments to lock your complete cookbook
dependency graph.

So what should you do instead?

  1. use application cookbooks in favor of roles (one app cookbook represents
    one node)
  2. lock your dependency versions in your application cookbook’s metadata.rb
    (e.g. “depends ‘apache2’, ‘=1.0.0’”). When you release a version of your
    application cookbook lock the transitive dependencies as well so the
    complete graph of cookbook dependencies for that app cookbook / node is
    locked.
  3. in your environment files, only lock your application cookbook versions
    (their dependencies are locked in their metadata)

This approach gives you a clean dependency management, does not clutter
your environments with implementation details (locking transitive cookbook
dependencies), and makes it really easy to promote applications from one
environment to another.

I’m aware that many folks (in my opinion) misuse environments for the sole
purpose of locking the complete cookbook dependency graphs, and some tools
are even encouraging this style of work (e.g. ‘berks apply ENVIRONMENT’).

Well, if you use roles, then you don’t have a choice, but if you use
application cookbooks, then this is really an antipattern (and I’m very
opinionated about this :-))

P.S.: I was assuming you are using some kind of cookbook dependency
management. I would recommend to use Berkshelf. You can also look for “The
Berkshelf Way” - you should find some slides from ChefConf 2013…

HTH,
Torben

I realise that this is a whole lot of stuff in one email… this is a list
of questions that has been growing in the last few weeks as I explore and
plan for chef in our environment.

Cheers,

Flo

LEGAL DISCLAIMER
This communication and any attached documents are strictly confidential
and/or legally privileged and they may not be used or disclosed by someone
who is not a named recipient. If you have received this electronic
communication in error please notify the sender by replying to this
electronic communication inserting the word “misdirected” as the subject
and delete this communication from your system.

Hope that helps,

Matt Moretti


#7

Wow! Thanks for all the input. There is a lot to digest(pun intended) in all the responses, not just this one.

Cheers,
Florian

-----Original Message-----
From: Eric G. Wolfe [mailto:eric.wolfe@marshall.edu]
Sent: 17 July 2013 01:34
To: chef@lists.opscode.com
Subject: [chef] Re: Re: Re: Re: best practices

The crux of the matter is the conflicting state of two, or more, opposing recipes. If the removal of the undo/uninstall recipe is guarded so that it defers to the conflicting recipe only when there is a conflict in the run_list, there should be no other issue with idempotence within that recipe.

I would tend to disagree that every recipe should be "run once"
complete. While every recipe running only once and producing the correct state would be certainly be optimal. The important point is to
eventually reach the correct state. If that takes two subsequent runs, rather than only once (in certain cases) it should not be a deal breaker so to speak.

Anyhow, thank you for the suggestion. I have taken action to guard self-removal of the recipe, only when there is certain state conflict in the run_list.

Eric G. Wolfe
Senior Linux Administrator,
IT Infrastructure Systems

Marshall University Computing Services
Drinko Library 428-K
One John Marshall Dr.
Huntington, WV 25755
Phone: 304.942.3970
Email: eric.wolfe@marshall.edu

You will be recognized and honored as a community leader.

On 07/16/2013 05:05 PM, Arnold Krille wrote:

Hi,

On Tue, 16 Jul 2013 13:33:19 -0400 "Eric G. Wolfe"
eric.wolfe@marshall.edu wrote:

For uninstallation, I like the idea of an “undo” recipe. For
example, RHEL flavors tend to install NFS as part of the base
installation. When I set out to do the nfs cookbook, I was sure to
create an “nfs::undo” run-once recipe
http://community.opscode.com/cookbooks/nfs/source which will clean
up that annoyance. The annoyances cookbook is another good example
of reversing an undesired state on a managed node.

I think the idea of an undo recipe should be to step backwards
logically through the installed/running pattern. Run-once capability
is a nice thing to have for a “kickstart” type of role. For example,
I may want to undo nfs on every fresh installation. However, I don’t
want the “nfs::undo” recipe to permanently exist in the run_list and
reverse the state of an actual NFS server either.
Hm, but all recipe should be “run once” recipes. Regardless what the
state before is, the end state should always be the desired one. So as
it shouldn’t hurt to “install” nfs-server on each chef-client run, it
also shouldn’t hurt to uninstall nfs on each chef-client run.
Only when you apply your “nfs::uninstall” and “nfs::server” recipes on
the same node at the same time…

So as I don’t care when chef-client makes sure vim-nox is installed on
each system (to make sure my co-workers don’t leave me an unusable
machine), I also don’t care when chef-client makes sure that emacs
isn’t installed even when some co-worker decided to do a quick
"apt-get install" just because.

So I actually fail to see a special need for making sure recipes only
run once. No matter what they do, they should always lead to the
desired outcome, regardless whether that means doing something or not.

Have fun,

Arnold

LEGAL DISCLAIMER
This communication and any attached documents are strictly confidential and/or legally privileged and they may not be used or disclosed by someone who is not a named recipient. If you have received this electronic communication in error please notify the sender by replying to this electronic communication inserting the word “misdirected” as the subject and delete this communication from your system.


#8

Torben,

Berkshelf is interesting but I still need to get to grips with more of the fundamentals before adding this to the mix. It will be a while in my environment before we have a hard to manage amount of nodes. We are mostly interested in how chef can help us automate/standardize test and productive environments(configuration is code) since we are developer/tester/integrator all rolled up into one.

That said I like your point of view on application cookbooks. Where I previously worked we had a bespoke automated deployment system and had this concept in the system.

Cheer,
Florian

From: Torben Knerr [mailto:ukio@gmx.de]
Sent: 17 July 2013 08:17
To: chef@lists.opscode.com
Subject: [chef] Re: Re: best practices

Adding my 5 cents below… :slight_smile:

On Jul 16, 2013 6:50 PM, “Matthew Moretti” <werebus@gmail.commailto:werebus@gmail.com> wrote:

I’ll take a stab at a few of these. Answers inline.

On Tue, Jul 16, 2013 at 5:02 AM, Florian Hehlen <Florian.Hehlen@mri-group.commailto:Florian.Hehlen@mri-group.com> wrote:

Hi all,

I am looking for some tips on chef best practices and having a hard time finding anything other than highly focused docs and tutorials on the web. Here are some of the issues I am struggling with:

  •     What granularity is best for recipes?  When does it make sense to use some of the more complex ideas like depends, replaces, recommends/suggest, etc
    

Granularity is really up to you, but I think it’s better to not have a given cookbook or recipe do too much. As far as the various cookbook metadata fields, I don’t think I’ve ever seen any of them other than “depends” used. Do declare your dependencies, though.

As of Chef 11 you really must declare your dependencies via ‘depends’ in metadata, otherwise recipes will not be loaded (this was different in Chef 10).

+1 for fine-grained cookbooks, i.e. on the granularity level of apache, mysql etc… - pretty much what you find on the community side. Having this level of granularity makes them reusable and composable.

When it comes to assembling your specific application by reusing / composing and configuring a set of these cookbooks (which I tend to call library cookbooks) then I would create a socalled “application cookbook” rather than using roles (the benefit of an “application cookbook” is that it can be properly versioned and dependency managed). My rule of thumb is that one application cookbook represents a whole node. These kinds of cookbooks are at a different level of granularity because they describe your whole application rather than the single parts it is composed of. This also makes it much less reusable than library cookbooks because they are so specific to your use case / application (i.e. they tend to live in a corporate git repo rather than on the community site)

  •     How to handle uninstallation? As different cookbooks, or as a different recipe, or as something that will be done under the covers by a replacement recipe...
    

It kind of depends on what your end goal is. I’ve never had the need to specifically remove a package in order to get to by desired end-state. If you do need that, I’d put

  •     How to structure roles in a generic and useful way for workstations and servers.
    

Again, this depends on your needs. Roles are generally used to group a set of related recipes, but some folks don’t use them at all.

Some folks = proponents of the application cookbook pattern (see above) :slight_smile:

See this blog post for a nice summary:

  •     I find that there is a certain amount of overlap in definition between node, client, and user. In fact bootstrapping creates a node and a client for each machine. Is there a good way to think about how these 3 relate to each other.
    

So, nodes are “the thing you’re configuring” - they generally correspond to “servers” or “ec2 instances”. Clients are “how you authenticate to the Chef Server”, and you’re right: if you’re using a Chef server, they’ll pretty much always be one-to-one (i think it’s technically possible to do otherwise, but I wouldn’t). Users are really just a special kind of client that also has a password for the WebUI - they generally correspond to people (like you).

  •     How to best manage recipe versioning using versions, environments, version constraints, etc. I am specifically wondering about being able to upload recipes that are being developed and tested without the fear that they will end up in production. I would like to stay away from having to manage 2 instances of chef.
    

I have a very specific opinion on that as I believe that you are doing it wrong if you are using environments to lock your complete cookbook dependency graph.

So what should you do instead?

  1. use application cookbooks in favor of roles (one app cookbook represents one node)
  2. lock your dependency versions in your application cookbook’s metadata.rb (e.g. “depends ‘apache2’, ‘=1.0.0’”). When you release a version of your application cookbook lock the transitive dependencies as well so the complete graph of cookbook dependencies for that app cookbook / node is locked.
  3. in your environment files, only lock your application cookbook versions (their dependencies are locked in their metadata)

This approach gives you a clean dependency management, does not clutter your environments with implementation details (locking transitive cookbook dependencies), and makes it really easy to promote applications from one environment to another.

I’m aware that many folks (in my opinion) misuse environments for the sole purpose of locking the complete cookbook dependency graphs, and some tools are even encouraging this style of work (e.g. ‘berks apply ENVIRONMENT’).

Well, if you use roles, then you don’t have a choice, but if you use application cookbooks, then this is really an antipattern (and I’m very opinionated about this :-))

P.S.: I was assuming you are using some kind of cookbook dependency management. I would recommend to use Berkshelf. You can also look for “The Berkshelf Way” - you should find some slides from ChefConf 2013…

HTH,
Torben

I realise that this is a whole lot of stuff in one email… this is a list of questions that has been growing in the last few weeks as I explore and plan for chef in our environment.

Cheers,

Flo

LEGAL DISCLAIMER
This communication and any attached documents are strictly confidential and/or legally privileged and they may not be used or disclosed by someone who is not a named recipient. If you have received this electronic communication in error please notify the sender by replying to this electronic communication inserting the word “misdirected” as the subject and delete this communication from your system.

Hope that helps,

Matt Moretti

LEGAL DISCLAIMER
This communication and any attached documents are strictly confidential and/or legally privileged and they may not be used or disclosed by someone who is not a named recipient. If you have received this electronic communication in error please notify the sender by replying to this electronic communication inserting the word “misdirected” as the subject and delete this communication from your system.


#9

We’re doing exactly what Torben suggests not to do, use environments to lock down the cookbook versions. We’ve gone a few steps further than that, let me explain the workflow:

-we’ve decided our chef artifacts should be treated the same as our code, and are essentially part of our product. they even live in the same git repository.
-workflow: developers push to git, jenkins publishes to chef.
-when jenkins runs a build, it takes the build number (i.e.: 1234) and updates the cookbook minor version, such that chef-client cookbook version is 2.0.1234.
-environments are named as ‘production-0_1234’, and have the corresponding cookbook versions with the 1234 minor number set to the same number.
-roles are also named -0_1234.
-we have plans to do something similar with data bags but aren’t there yet.
-our environments have nothing in them except for the cookbook versions, the software (debian package) version that we just built, and four attributes that are global.
-jenkins uploads all the cookbooks, roles and environments, moves the test servers from to the new environment and roles, and runs the tests.
-jenkins has a daily job to clean out old unused cookbooks, roles and environments.

When we decide to activate/publish/deploy a build, we use a custom tool to switch each node to the new environment and roles (since from chef’s perspective the roles change) plus add some logging about who made the change.

Future plans:
-do versioning around some of the data bags
-publish builds daily and eventually publish every successful build.

On our dev laptops, we generate a somewhat random build number (based on a hash of the username etc) which is tied to the developer, and we use chef-client on our dev systems. That way if I need to do some testing on a specific node i can switch the node to my build number, test it, and the switch it back again. I want to eventually get to what Facebook mentioned at ChefConf, where you essentially ‘checkout’ a node and if you don’t touch anything after an hour it gets automatically put back to the production version, but we’re a bit away from that yet.

Why do it this way? We have some tough requirements to fill around versioning and testing, partially because we’re managing remote appliances which. If we lose the ability to manage those appliances the only way to recover is to have the customer ship the appliance back to us, resulting in cost and embarrassment. This seemed the sanest way to ensure that we always get the exact version we expect to get.

I’d love to get some feedback on what we’ve done.

cheers
mike


Michael Hart
Arctic Wolf Networks
M: 226.388.4773

On 2013-07-17, at 2:16 AM, Torben Knerr <ukio@gmx.demailto:ukio@gmx.de> wrote:

Adding my 5 cents below… :slight_smile:

On Jul 16, 2013 6:50 PM, “Matthew Moretti” <werebus@gmail.commailto:werebus@gmail.com> wrote:

I’ll take a stab at a few of these. Answers inline.

On Tue, Jul 16, 2013 at 5:02 AM, Florian Hehlen <Florian.Hehlen@mri-group.commailto:Florian.Hehlen@mri-group.com> wrote:

Hi all,

I am looking for some tips on chef best practices and having a hard time finding anything other than highly focused docs and tutorials on the web. Here are some of the issues I am struggling with:

· What granularity is best for recipes? When does it make sense to use some of the more complex ideas like depends, replaces, recommends/suggest, etc

Granularity is really up to you, but I think it’s better to not have a given cookbook or recipe do too much. As far as the various cookbook metadata fields, I don’t think I’ve ever seen any of them other than “depends” used. Do declare your dependencies, though.

As of Chef 11 you really must declare your dependencies via ‘depends’ in metadata, otherwise recipes will not be loaded (this was different in Chef 10).

+1 for fine-grained cookbooks, i.e. on the granularity level of apache, mysql etc… - pretty much what you find on the community side. Having this level of granularity makes them reusable and composable.

When it comes to assembling your specific application by reusing / composing and configuring a set of these cookbooks (which I tend to call library cookbooks) then I would create a socalled “application cookbook” rather than using roles (the benefit of an “application cookbook” is that it can be properly versioned and dependency managed). My rule of thumb is that one application cookbook represents a whole node. These kinds of cookbooks are at a different level of granularity because they describe your whole application rather than the single parts it is composed of. This also makes it much less reusable than library cookbooks because they are so specific to your use case / application (i.e. they tend to live in a corporate git repo rather than on the community site)

· How to handle uninstallation? As different cookbooks, or as a different recipe, or as something that will be done under the covers by a replacement recipe…

It kind of depends on what your end goal is. I’ve never had the need to specifically remove a package in order to get to by desired end-state. If you do need that, I’d put

· How to structure roles in a generic and useful way for workstations and servers.

Again, this depends on your needs. Roles are generally used to group a set of related recipes, but some folks don’t use them at all.

Some folks = proponents of the application cookbook pattern (see above) :slight_smile:

See this blog post for a nice summary:

· I find that there is a certain amount of overlap in definition between node, client, and user. In fact bootstrapping creates a node and a client for each machine. Is there a good way to think about how these 3 relate to each other.

So, nodes are “the thing you’re configuring” - they generally correspond to “servers” or “ec2 instances”. Clients are “how you authenticate to the Chef Server”, and you’re right: if you’re using a Chef server, they’ll pretty much always be one-to-one (i think it’s technically possible to do otherwise, but I wouldn’t). Users are really just a special kind of client that also has a password for the WebUI - they generally correspond to people (like you).

· How to best manage recipe versioning using versions, environments, version constraints, etc. I am specifically wondering about being able to upload recipes that are being developed and tested without the fear that they will end up in production. I would like to stay away from having to manage 2 instances of chef.

I have a very specific opinion on that as I believe that you are doing it wrong if you are using environments to lock your complete cookbook dependency graph.

So what should you do instead?

  1. use application cookbooks in favor of roles (one app cookbook represents one node)
  2. lock your dependency versions in your application cookbook’s metadata.rb (e.g. “depends ‘apache2’, ‘=1.0.0’”). When you release a version of your application cookbook lock the transitive dependencies as well so the complete graph of cookbook dependencies for that app cookbook / node is locked.
  3. in your environment files, only lock your application cookbook versions (their dependencies are locked in their metadata)

This approach gives you a clean dependency management, does not clutter your environments with implementation details (locking transitive cookbook dependencies), and makes it really easy to promote applications from one environment to another.

I’m aware that many folks (in my opinion) misuse environments for the sole purpose of locking the complete cookbook dependency graphs, and some tools are even encouraging this style of work (e.g. ‘berks apply ENVIRONMENT’).

Well, if you use roles, then you don’t have a choice, but if you use application cookbooks, then this is really an antipattern (and I’m very opinionated about this :-))

P.S.: I was assuming you are using some kind of cookbook dependency management. I would recommend to use Berkshelf. You can also look for “The Berkshelf Way” - you should find some slides from ChefConf 2013…

HTH,
Torben

I realise that this is a whole lot of stuff in one email… this is a list of questions that has been growing in the last few weeks as I explore and plan for chef in our environment.

Cheers,

Flo

LEGAL DISCLAIMER
This communication and any attached documents are strictly confidential and/or legally privileged and they may not be used or disclosed by someone who is not a named recipient. If you have received this electronic communication in error please notify the sender by replying to this electronic communication inserting the word “misdirected” as the subject and delete this communication from your system.

Hope that helps,

Matt Moretti


#10

Hi Florian,

I would definitely consider Berkshelf (or librarian-chef as an alternative)
as one of the fundamentals, namely dependency management for cookbooks!

You will have to deal with that and both Berkshelf and librarian-chef make
cookbook dependency management dead easy.

Forget about all the other features that berkshelf provides for now - you
will probably not need them in the beginning - but dependency management
(i.e. “berks install”) I would recommend to use right from the start.

It’s just my experience that it makes your life so much easier (especially
in combination with the vagrant-berkshelf plugin) - but decide for
yourself… :wink:

Cheers, Torben
On Jul 17, 2013 9:02 AM, “Florian Hehlen” Florian.Hehlen@mri-group.com
wrote:

Torben,

Berkshelf is interesting but I still need to get to grips with more of the
fundamentals before adding this to the mix. It will be a while in my
environment before we have a hard to manage amount of nodes. We are mostly
interested in how chef can help us automate/standardize test and productive
environments(configuration is code) since we are
developer/tester/integrator all rolled up into one.

That said I like your point of view on application cookbooks. Where I
previously worked we had a bespoke automated deployment system and had this
concept in the system.

Cheer,

Florian

From: Torben Knerr [mailto:ukio@gmx.de]
Sent: 17 July 2013 08:17
To: chef@lists.opscode.com
Subject: [chef] Re: Re: best practices

Adding my 5 cents below… :slight_smile:

On Jul 16, 2013 6:50 PM, “Matthew Moretti” werebus@gmail.com wrote:

I’ll take a stab at a few of these. Answers inline.

On Tue, Jul 16, 2013 at 5:02 AM, Florian Hehlen <
Florian.Hehlen@mri-group.com> wrote:

Hi all,

I am looking for some tips on chef best practices and having a hard
time finding anything other than highly focused docs and tutorials on the
web. Here are some of the issues I am struggling with:

· What granularity is best for recipes? When does it make
sense to use some of the more complex ideas like depends, replaces,
recommends/suggest, etc

Granularity is really up to you, but I think it’s better to not have a
given cookbook or recipe do too much. As far as the various cookbook
metadata fields, I don’t think I’ve ever seen any of them other than
"depends" used. Do declare your dependencies, though.

As of Chef 11 you really must declare your dependencies via ‘depends’ in
metadata, otherwise recipes will not be loaded (this was different in Chef
10).

+1 for fine-grained cookbooks, i.e. on the granularity level of apache,
mysql etc… - pretty much what you find on the community side. Having this
level of granularity makes them reusable and composable.

When it comes to assembling your specific application by reusing /
composing and configuring a set of these cookbooks (which I tend to call
library cookbooks) then I would create a socalled "application cookbook"
rather than using roles (the benefit of an “application cookbook” is that
it can be properly versioned and dependency managed). My rule of thumb is
that one application cookbook represents a whole node. These kinds of
cookbooks are at a different level of granularity because they describe
your whole application rather than the single parts it is composed of. This
also makes it much less reusable than library cookbooks because they are
so specific to your use case / application (i.e. they tend to live in a
corporate git repo rather than on the community site)

· How to handle uninstallation? As different cookbooks, or as a
different recipe, or as something that will be done under the covers by a
replacement recipe…

It kind of depends on what your end goal is. I’ve never had the need to
specifically remove a package in order to get to by desired end-state. If
you do need that, I’d put

· How to structure roles in a generic and useful way for
workstations and servers.

Again, this depends on your needs. Roles are generally used to group a
set of related recipes, but some folks don’t use them at all.

Some folks = proponents of the application cookbook pattern (see above) :slight_smile:

See this blog post for a nice summary:

http://alluvium.com/blog/2013/05/03/the-application-cookbook-pattern-berkshelf-and-team-chef-workflow/

· I find that there is a certain amount of overlap in
definition between node, client, and user. In fact bootstrapping creates a
node and a client for each machine. Is there a good way to think about how
these 3 relate to each other.

So, nodes are “the thing you’re configuring” - they generally correspond
to “servers” or “ec2 instances”. Clients are “how you authenticate to the
Chef Server”, and you’re right: if you’re using a Chef server, they’ll
pretty much always be one-to-one (i think it’s technically possible to do
otherwise, but I wouldn’t). Users are really just a special kind of client
that also has a password for the WebUI - they generally correspond to
people (like you).

· How to best manage recipe versioning using versions,
environments, version constraints, etc. I am specifically wondering about
being able to upload recipes that are being developed and tested without
the fear that they will end up in production. I would like to stay away
from having to manage 2 instances of chef.

I have a very specific opinion on that as I believe that you are doing it
wrong if you are using environments to lock your complete cookbook
dependency graph.

So what should you do instead?

  1. use application cookbooks in favor of roles (one app cookbook
    represents one node)
  2. lock your dependency versions in your application cookbook’s
    metadata.rb (e.g. “depends ‘apache2’, ‘=1.0.0’”). When you release a
    version of your application cookbook lock the transitive dependencies as
    well so the complete graph of cookbook dependencies for that app cookbook /
    node is locked.
  3. in your environment files, only lock your application cookbook versions
    (their dependencies are locked in their metadata)

This approach gives you a clean dependency management, does not clutter
your environments with implementation details (locking transitive cookbook
dependencies), and makes it really easy to promote applications from one
environment to another.

I’m aware that many folks (in my opinion) misuse environments for the sole
purpose of locking the complete cookbook dependency graphs, and some tools
are even encouraging this style of work (e.g. ‘berks apply ENVIRONMENT’).

Well, if you use roles, then you don’t have a choice, but if you use
application cookbooks, then this is really an antipattern (and I’m very
opinionated about this :-))

P.S.: I was assuming you are using some kind of cookbook dependency
management. I would recommend to use Berkshelf. You can also look for “The
Berkshelf Way” - you should find some slides from ChefConf 2013…

HTH,
Torben

I realise that this is a whole lot of stuff in one email… this is a
list of questions that has been growing in the last few weeks as I explore
and plan for chef in our environment.

Cheers,

Flo

LEGAL DISCLAIMER
This communication and any attached documents are strictly confidential
and/or legally privileged and they may not be used or disclosed by someone
who is not a named recipient. If you have received this electronic
communication in error please notify the sender by replying to this
electronic communication inserting the word “misdirected” as the subject
and delete this communication from your system.

Hope that helps,

Matt Moretti

LEGAL DISCLAIMER
This communication and any attached documents are strictly confidential
and/or legally privileged and they may not be used or disclosed by someone
who is not a named recipient. If you have received this electronic
communication in error please notify the sender by replying to this
electronic communication inserting the word “misdirected” as the subject
and delete this communication from your system.


#11

Hi Michael,

That sounds very cool what you are doing!

I think there are two camps in Chef land:

  • the folks who are using roles
  • the folks who are using application cookbooks

If you are in the roles camp, then your only choice is to use environments
for locking cookbook versions, and maybe this is the sole purpose of
environments then. This is what most people do I guess because before the
idea of application cookbooks came up this was the only way to go.

If you are in the application cookbooks camp, then I would strongly suggest
to lock your versions in metadata.rb, because it’s actually an
implementation detail of the application cookbook not the environment. If
you do this, the environments become simple and you can use them for their
originally intended purpose (or at least my understanding of it) of
modeling the different environments / stages in a deployment pipeline.

Only in the latter case, i.e. using application cookbooks but still locking
dependencies in the environments, I would call it an anti-pattern.

But that’s just my point of view and I’m waiting for someone of the
application cookbook + locking in environments camp to jump in with some
arguments… :slight_smile:

Cheers,
Torben
On Jul 17, 2013 3:43 PM, “Michael Hart” michael.hart@arcticwolf.com
wrote:

We’re doing exactly what Torben suggests not to do, use environments to
lock down the cookbook versions. We’ve gone a few steps further than that,
let me explain the workflow:

-we’ve decided our chef artifacts should be treated the same as our
code, and are essentially part of our product. they even live in the same
git repository.
-workflow: developers push to git, jenkins publishes to chef.
-when jenkins runs a build, it takes the build number (i.e.: 1234) and
updates the cookbook minor version, such that chef-client cookbook version
is 2.0.1234.
-environments are named as ‘production-0_1234’, and have the corresponding
cookbook versions with the 1234 minor number set to the same number.
-roles are also named -0_1234.
-we have plans to do something similar with data bags but aren’t there yet.
-our environments have nothing in them except for the cookbook versions,
the software (debian package) version that we just built, and four
attributes that are global.
-jenkins uploads all the cookbooks, roles and environments, moves the test
servers from to the new environment and roles, and runs the tests.
-jenkins has a daily job to clean out old unused cookbooks, roles and
environments.

When we decide to activate/publish/deploy a build, we use a custom tool
to switch each node to the new environment and roles (since from chef’s
perspective the roles change) plus add some logging about who made the
change.

Future plans:
-do versioning around some of the data bags
-publish builds daily and eventually publish every successful build.

On our dev laptops, we generate a somewhat random build number (based on
a hash of the username etc) which is tied to the developer, and we use
chef-client on our dev systems. That way if I need to do some testing on a
specific node i can switch the node to my build number, test it, and the
switch it back again. I want to eventually get to what Facebook mentioned
at ChefConf, where you essentially ‘checkout’ a node and if you don’t touch
anything after an hour it gets automatically put back to the production
version, but we’re a bit away from that yet.

Why do it this way? We have some tough requirements to fill around
versioning and testing, partially because we’re managing remote appliances
which. If we lose the ability to manage those appliances the only way to
recover is to have the customer ship the appliance back to us, resulting in
cost and embarrassment. This seemed the sanest way to ensure that we always
get the exact version we expect to get.

I’d love to get some feedback on what we’ve done.

cheers
mike


Michael Hart
Arctic Wolf Networks
M: 226.388.4773

On 2013-07-17, at 2:16 AM, Torben Knerr ukio@gmx.de wrote:

Adding my 5 cents below… :slight_smile:

On Jul 16, 2013 6:50 PM, “Matthew Moretti” werebus@gmail.com wrote:

I’ll take a stab at a few of these. Answers inline.

On Tue, Jul 16, 2013 at 5:02 AM, Florian Hehlen <
Florian.Hehlen@mri-group.com> wrote:

Hi all,

I am looking for some tips on chef best practices and having a hard
time finding anything other than highly focused docs and tutorials on the
web. Here are some of the issues I am struggling with:

· What granularity is best for recipes? When does it make
sense to use some of the more complex ideas like depends, replaces,
recommends/suggest, etc

Granularity is really up to you, but I think it’s better to not have a
given cookbook or recipe do too much. As far as the various cookbook
metadata fields, I don’t think I’ve ever seen any of them other than
"depends" used. Do declare your dependencies, though.

As of Chef 11 you really must declare your dependencies via ‘depends’ in
metadata, otherwise recipes will not be loaded (this was different in Chef
10).

+1 for fine-grained cookbooks, i.e. on the granularity level of apache,
mysql etc… - pretty much what you find on the community side. Having this
level of granularity makes them reusable and composable.

When it comes to assembling your specific application by reusing /
composing and configuring a set of these cookbooks (which I tend to call
library cookbooks) then I would create a socalled "application cookbook"
rather than using roles (the benefit of an “application cookbook” is that
it can be properly versioned and dependency managed). My rule of thumb is
that one application cookbook represents a whole node. These kinds of
cookbooks are at a different level of granularity because they describe
your whole application rather than the single parts it is composed of. This
also makes it much less reusable than library cookbooks because they are
so specific to your use case / application (i.e. they tend to live in a
corporate git repo rather than on the community site)

· How to handle uninstallation? As different cookbooks, or as a
different recipe, or as something that will be done under the covers by a
replacement recipe…

It kind of depends on what your end goal is. I’ve never had the need to
specifically remove a package in order to get to by desired end-state. If
you do need that, I’d put

· How to structure roles in a generic and useful way for
workstations and servers.

Again, this depends on your needs. Roles are generally used to group a
set of related recipes, but some folks don’t use them at all.

Some folks = proponents of the application cookbook pattern (see above) :slight_smile:

See this blog post for a nice summary:

http://alluvium.com/blog/2013/05/03/the-application-cookbook-pattern-berkshelf-and-team-chef-workflow/

· I find that there is a certain amount of overlap in
definition between node, client, and user. In fact bootstrapping creates a
node and a client for each machine. Is there a good way to think about how
these 3 relate to each other.

So, nodes are “the thing you’re configuring” - they generally correspond
to “servers” or “ec2 instances”. Clients are “how you authenticate to the
Chef Server”, and you’re right: if you’re using a Chef server, they’ll
pretty much always be one-to-one (i think it’s technically possible to do
otherwise, but I wouldn’t). Users are really just a special kind of client
that also has a password for the WebUI - they generally correspond to
people (like you).

· How to best manage recipe versioning using versions,
environments, version constraints, etc. I am specifically wondering about
being able to upload recipes that are being developed and tested without
the fear that they will end up in production. I would like to stay away
from having to manage 2 instances of chef.

I have a very specific opinion on that as I believe that you are doing it
wrong if you are using environments to lock your complete cookbook
dependency graph.

So what should you do instead?

  1. use application cookbooks in favor of roles (one app cookbook
    represents one node)
  2. lock your dependency versions in your application cookbook’s
    metadata.rb (e.g. “depends ‘apache2’, ‘=1.0.0’”). When you release a
    version of your application cookbook lock the transitive dependencies as
    well so the complete graph of cookbook dependencies for that app cookbook /
    node is locked.
  3. in your environment files, only lock your application cookbook versions
    (their dependencies are locked in their metadata)

This approach gives you a clean dependency management, does not clutter
your environments with implementation details (locking transitive cookbook
dependencies), and makes it really easy to promote applications from one
environment to another.

I’m aware that many folks (in my opinion) misuse environments for the sole
purpose of locking the complete cookbook dependency graphs, and some tools
are even encouraging this style of work (e.g. ‘berks apply ENVIRONMENT’).

Well, if you use roles, then you don’t have a choice, but if you use
application cookbooks, then this is really an antipattern (and I’m very
opinionated about this :-))

P.S.: I was assuming you are using some kind of cookbook dependency
management. I would recommend to use Berkshelf. You can also look for “The
Berkshelf Way” - you should find some slides from ChefConf 2013…

HTH,
Torben

I realise that this is a whole lot of stuff in one email… this is a
list of questions that has been growing in the last few weeks as I explore
and plan for chef in our environment.

Cheers,

Flo

LEGAL DISCLAIMER
This communication and any attached documents are strictly confidential
and/or legally privileged and they may not be used or disclosed by someone
who is not a named recipient. If you have received this electronic
communication in error please notify the sender by replying to this
electronic communication inserting the word “misdirected” as the subject
and delete this communication from your system.

Hope that helps,

Matt Moretti


#12

Ohai,

On Jul 16, 2013, at 3:02 AM, Florian Hehlen Florian.Hehlen@mri-group.com wrote:

· What granularity is best for recipes? When does it make sense to use some of the more complex ideas like depends, replaces, recommends/suggest, etc

Only depends is implemented at this time. The other “dependency” metadata elements, suggests, recommends, conflicts, provides, replaces, do not actually have an effect on Chef.

For more information, see this ticket:

http://tickets.opscode.com/browse/CHEF-3870

· How to handle uninstallation? As different cookbooks, or as a different recipe, or as something that will be done under the covers by a replacement recipe…

It depends on what you mean by uninstallation. Others have touched on this, but if the policy of your environment is that certain things should be removed, all the core resources in Chef support “negative” actions, where the “positive” action is [generally] the default. For example, package has :remove, template (or any of the “file” resources) has :delete, service has :stop and :disable, etc.

· How to structure roles in a generic and useful way for workstations and servers.

I would have a workstation role that has the “workstation things” in the run list.

· I find that there is a certain amount of overlap in definition between node, client, and user. In fact bootstrapping creates a node and a client for each machine. Is there a good way to think about how these 3 relate to each other.

I think others have covered this, but to be clear:

  • “node” - a system that is managed by chef, contains a run list, attributes, and environment.
  • “client” - an authenticating API client, contains an RSA key pair (server stores the public key), and has permissions. The permissions vary based on the Chef Server (open source has only “admin” or “non-admin”, Opscode’s Private/Hosted Chef servers have a full role-based access control system).
  • “user” - a new feature of Chef 11, users have key pairs like clients, but they also have login in the webui, and are representative of human users.

· How to best manage recipe versioning using versions, environments, version constraints, etc. I am specifically wondering about being able to upload recipes that are being developed and tested without the fear that they will end up in production. I would like to stay away from having to manage 2 instances of chef.

I like pinning versions of all cookbooks in a “stable” or “production” type of environment using the equality constraint, e.g. “= 2.1.3”. I prefer to reduce the per-cookbook version constraints as much as possible.

Hope this helps,
Joshua


#13

On Wed, 17 Jul 2013 18:04:12 +0200 Torben Knerr ukio@gmx.de wrote:

Hi Michael,

That sounds very cool what you are doing!

I think there are two camps in Chef land:

  • the folks who are using roles
  • the folks who are using application cookbooks

If you are in the roles camp, then your only choice is to use
environments for locking cookbook versions, and maybe this is the
sole purpose of environments then.

Environments are usefull for a lot of other stuff too:
For example the question “all non-virtual machine with ntp in my
environment” (because that is the same subnet by coincidence) is a good
choice to select peers (for non-virtual machine) or server (for virtual
machines).
Another example: “All machines with the recipe munin::client in the
’dev’-environment” is also a nice way to configure the munin-server.
Doing that with cookbooks or roles would certainly be possible: a role
"dev-munin-client" that inherits the role “munin-client” and then
search for all “dev-munin-client” to get the same as with the previous
example. Buts its not as nice I think.

Yes, I think I am part of the roles/environments-camp…

Have fun,

Arnold


#14

On Jul 17, 2013 9:06 PM, “Arnold Krille” arnold@arnoldarts.de wrote:

On Wed, 17 Jul 2013 18:04:12 +0200 Torben Knerr ukio@gmx.de wrote:

Hi Michael,

That sounds very cool what you are doing!

I think there are two camps in Chef land:

  • the folks who are using roles
  • the folks who are using application cookbooks

If you are in the roles camp, then your only choice is to use
environments for locking cookbook versions, and maybe this is the
sole purpose of environments then.

Environments are usefull for a lot of other stuff too:
For example the question “all non-virtual machine with ntp in my
environment” (because that is the same subnet by coincidence) is a good
choice to select peers (for non-virtual machine) or server (for virtual
machines).
Another example: “All machines with the recipe munin::client in the
’dev’-environment” is also a nice way to configure the munin-server.

Oh, I did not mean to say that environments are used for locking cookbook
dependencies only – that was just a reference to an earlier statement in
this thread.

100% agree that environments are totally useful and should be used for
other things like search or setting environment specific attributes etc.!

Doing that with cookbooks or roles would certainly be possible: a role
"dev-munin-client" that inherits the role “munin-client” and then
search for all “dev-munin-client” to get the same as with the previous
example. Buts its not as nice I think.

Me too, that would be abusing roles/cookbooks for environments. Definitely
Not Like! -1. :slight_smile:

Yes, I think I am part of the roles/environments-camp…

Have fun,

Arnold


#15

Okay Torben, I’ll bite. We use application cookbooks instead of roles, but we pin versions in the environments; which we use in the very traditional way of gating promotions through the release pipeline: Dev->Test->QA->Staging->Prod.

We briefly experimented with the idea of pinning versions in the application cookbook’s metadata.rb, but found that when we upgraded a cookbook that was part of our “base”, we then had to modify the pins in many places (each cookbook that eventually depends on the changed one) instead of just one or two (the environments). We’re thinking that using “pessimistic version constraints” (which actually seem optimistic to me…) would help alleviate some of this maintenance at the cost of not being quite as sure exactly what version will run.

Have I mis-understood something about what you are suggesting? From here it really seems like a case of DRY violation. Perhaps your requirements are such that once a node has been deployed with a particular application cookbook it is relatively static? In our environment things are constantly evolving, so maintaining all these extra pins is a PITA.

From: Torben Knerr [mailto:ukio@gmx.de]
Sent: Wednesday, July 17, 2013 12:04 PM
To: chef@lists.opscode.com
Subject: [chef] Re: Re: best practices

Hi Michael,

That sounds very cool what you are doing!

I think there are two camps in Chef land:

  • the folks who are using roles
  • the folks who are using application cookbooks

If you are in the roles camp, then your only choice is to use environments for locking cookbook versions, and maybe this is the sole purpose of environments then. This is what most people do I guess because before the idea of application cookbooks came up this was the only way to go.

If you are in the application cookbooks camp, then I would strongly suggest to lock your versions in metadata.rb, because it’s actually an implementation detail of the application cookbook not the environment. If you do this, the environments become simple and you can use them for their originally intended purpose (or at least my understanding of it) of modeling the different environments / stages in a deployment pipeline.

Only in the latter case, i.e. using application cookbooks but still locking dependencies in the environments, I would call it an anti-pattern.

But that’s just my point of view and I’m waiting for someone of the application cookbook + locking in environments camp to jump in with some arguments… :slight_smile:

Cheers,


#16

Hi,

On Saturday, July 20, 2013, Chris Sibbitt wrote:

Okay Torben, I’ll bite. We use application cookbooks instead of roles,
but we pin versions in the environments; which we use in the very
traditional way of gating promotions through the release pipeline:
Dev->Test->QA->Staging->Prod.

That is pretty much what we do today.

We briefly experimented with the idea of pinning versions in the
application cookbook’s metadata.rb, but found that when we upgraded a
cookbook that was part of our “base”, we then had to modify the pins in
many places (each cookbook that eventually depends on the changed one)
instead of just one or two (the environments). We’re thinking that using
“pessimistic version constraints” (which actually seem optimistic to me…)
would help alleviate some of this maintenance at the cost of not being
quite as sure exactly what version will run.

So we started to move towards all the versions being declared in the
application cookbook but turned back due to bugs in the current chef server
(CHEF-3921).

However we went a slightly different path that may help for this. We had
our top level role cookbooks (i.e. mybiz-myapplication) have pessimistic
version constraints on most of it’s dependencies except for the "base"
cookbook in which case we used unconstrained depends. The “base” cookbook
then had pessimistic version constraints on it’s dependencies. Our
environment file locked down both the “base” cookbook version and the
application cookbook version. In general what we tried to do was to
identify cookbooks that we wanted to independently version and use
pessimistic constraints everywhere else. I think that would have been
successful in our environment and not too much work.

It should be noted that we have decoupled application releases from
cookbook releases and it is very rare that an application release requires
anything but an update of a value in a data bag. Thus the period of time
during which a cookbook has different versions across our environment files
is relatively short.

However with the small scale of our group (1-5 people working in chef, <
200 cookbooks, < 100 nodes) we have decided to go with all versions in
environment files for the time being.


#17

Hi Chris, Peter,

Thanks for jumping in!

On Jul 20, 2013 12:02 AM, “Peter Donald” peter@realityforge.org wrote:

Hi,

On Saturday, July 20, 2013, Chris Sibbitt wrote:

Okay Torben, I’ll bite. We use application cookbooks instead of roles,
but we pin versions in the environments; which we use in the very
traditional way of gating promotions through the release pipeline:
Dev->Test->QA->Staging->Prod.

That is pretty much what we do today.

We briefly experimented with the idea of pinning versions in the
application cookbook’s metadata.rb, but found that when we upgraded a
cookbook that was part of our “base”, we then had to modify the pins in
many places (each cookbook that eventually depends on the changed one)
instead of just one or two (the environments). We’re thinking that using
“pessimistic version constraints” (which actually seem optimistic to me…)
would help alleviate some of this maintenance at the cost of not being
quite as sure exactly what version will run.

We might have different goals here. What you describe here as a pain / DRY
violation I really see as a strength of this approach: it actually allows
me to update the “base” of one application cookbook without affecting the
other application cookbooks in that environment.

If you use environments to lock versions, you have no choice but update the
"base" for all of your application cookbooks which are in that environment
(and be sure they are compatible with that “base”).

“base” might be a special case / contrived example though, and I like
Peter’s approach to dealing with this - sounds like a good compromise.

If your goal is to have only a single version of any cookbook in an
environment --> use environments for locking versions.

If your goal is to keep the environments independent of the cookbooks (i.e.
being able to freely add/remove an application cookbook without affecting
anything else in that environment) --> use application cookbook’s metadata
for locking versions.

So we started to move towards all the versions being declared in the
application cookbook but turned back due to bugs in the current chef server
(CHEF-3921).

However we went a slightly different path that may help for this. We had
our top level role cookbooks (i.e. mybiz-myapplication) have pessimistic
version constraints on most of it’s dependencies except for the "base"
cookbook in which case we used unconstrained depends. The “base” cookbook
then had pessimistic version constraints on it’s dependencies. Our
environment file locked down both the “base” cookbook version and the
application cookbook version. In general what we tried to do was to
identify cookbooks that we wanted to independently version and use
pessimistic constraints everywhere else. I think that would have been
successful in our environment and not too much work.

It should be noted that we have decoupled application releases from
cookbook releases and it is very rare that an application release requires
anything but an update of a value in a data bag. Thus the period of time
during which a cookbook has different versions across our environment files
is relatively short.

+1 - (application) cookbook releases and application releases are two
separate things imo

However with the small scale of our group (1-5 people working in chef, <
200 cookbooks, < 100 nodes) we have decided to go with all versions in
environment files for the time being.