Help on best practices for roles and wrappers


#1

Hi,

We have been using Chef for a while at my company and have been very happy with it.

We’ve come to a point where we are making a few changes to the way we use Chef.

I’d like to use wrapper cookbooks to wrap generic cookbooks that we can contribute back to upstream. I’d also like to use role cookbooks so that we can version our roles and make incremental changes to the runlists of our machines without risking blowing up prod and without having to do the role-renaming dance. At least that is my thought as of right now.

We’ve actually had a good deal of debate on the subject and are working on coming to a consensus, which brings me to the point. Besides the Gangnam-style one (http://devopsanywhere.blogspot.com/2012/11/how-to-write-reusable-chef-cookbooks.html) and a few other high level articles, I haven’t been able to find anything on how people are really doing this (i.e. versioning roles by using role cookbooks and / or using wrapper cookbooks) and what the best practices are.

Any help or direction to info sources would be much appreciated.

Thanks!
Sean


#2

I have always tried to use role to compose recipes together, to alter their
behaviour i have used overridden attributes. This enforces two basic design
principles,

  1. cookbooks are like object, and follows OCP. Open for extension (via
    attributes) and closed for modifications.
  2. different version requirements are enforced via different environments.

In many cases i fail to do so. If i need decision making (if , else etc)
and i need them on the fly, i’ll fall back to wrapper recipes/cookbooks.
But this also means i am loosing readability , i feel a role or a set of
role should give an unique type of instances in my infrastructure, and
when applied against an environment (which might have cookbook version
constrains) , will always give identical instances.

I also use this principle(or role as an abstraction) to build other
tooling, like firewalls(think of security groups in aws), monitoring etc. I
always develop a complementary firewall rule having same name as the role
(and similar things for network). So a role in short help me understand
configuration (via chef), network requirements (like database should
allow ingress tcp traffic from web at port 3306).

But in general, i think you can choose any of the two approaches, but
whatever you choose, try to follow them strictly, and exploit that
convention to build/scaffold higher order system.

best,
ranjib

On Fri, Jan 18, 2013 at 3:37 PM, Sean McKenna sean.mckenna@mosaik.comwrote:

Hi,****


We have been using Chef for a while at my company and have been very happy
with it.****


We’ve come to a point where we are making a few changes to the way we use
Chef.****


I’d like to use wrapper cookbooks to wrap generic cookbooks that we can
contribute back to upstream. I’d also like to use role cookbooks so that we
can version our roles and make incremental changes to the runlists of our
machines without risking blowing up prod and without having to do the
role-renaming dance. At least that is my thought as of right now.****


We’ve actually had a good deal of debate on the subject and are working on
coming to a consensus, which brings me to the point. Besides the
Gangnam-style one (
http://devopsanywhere.blogspot.com/2012/11/how-to-write-reusable-chef-cookbooks.html)
and a few other high level articles, I haven’t been able to find anything
on how people are really doing this (i.e. versioning roles by using role
cookbooks and / or using wrapper cookbooks) and what the best practices are.



Any help or direction to info sources would be much appreciated.****


Thanks!****

Sean****



#3

Hi Sean, Reality Forge wrote a blog
post a few months ago that I’ve been using to model some
experiments.

http://realityforge.org/code/2012/11/19/role-cookbooks-and-wrapper-cookbooks.html

  We're working through a lot of your same issues at the moment and
  I'm not sure I have any comments yet on how we are experimenting
  with the overall wrapping structure, but I can tell you how I've
  been writing the "library" cookbooks.  My general approach right
  now is, if I'm engaged in writing a cookbook I think will become a
  general use cookbook, I iterate like this:




  Write a recipe to define optimal flow, test the recipe to make
  sure it's working like I think it should.  


  If I have time, I refactor it back into a definition or LWRP which
  is a good way to abstract attributes and hide code from devs,
  creating a clean interface for them.


  If I don't have time, I parametrize with default attributes and
  write up notes in the README about how to override the attrs from
  a wrapper implementation .




  LIke I said, we're still experimenting with wrapper
  implementations, so I don't have many comments on that yet.




  Hopefully the blog link above helps. If you are looking for an
  example, I pasted an innocuous set of snippets showing a template
  block from a recipe associated attributes file in both a library
  cookbook and the wrapper.  I'm no authority and YMMV and all that
  stuff, but these are certainly good questions to ask as many
  people are discovering.




  Good Luck on your adventures.




  Sascha

Sascha Bates
| sascha.bates@gmail.com | 612 850 0444 | sascha.bates@skype |
sascha_bates@yahoo |
On 1/18/13 5:37 PM, Sean McKenna wrote:

Hi,

We have been using Chef for a while at my company and have been very happy with it.

We’ve come to a point where we are making a few changes to the way we use Chef.

I’d like to use wrapper cookbooks to wrap generic cookbooks that we can contribute back to upstream. I’d also like to use role cookbooks so that we can version our roles and make incremental changes to the runlists of our machines without risking blowing up prod and without having to do the role-renaming dance. At least that is my thought as of right now.

We’ve actually had a good deal of debate on the subject and are working on coming to a consensus, which brings me to the point. Besides the Gangnam-style one (http://devopsanywhere.blogspot.com/2012/11/how-to-write-reusable-chef-cookbooks.html) and a few other high level articles, I haven’t been able to find anything on how people are really doing this (i.e. versioning roles by using role cookbooks and / or using wrapper cookbooks) and what the best practices are.

Any help or direction to info sources would be much appreciated.

Thanks!

Sean


#4

Those of you using role-cookbooks and wrapper cookbooks, how do you use
your environment files? I work with Sean, and we’ve been considering using
role-cookbooks to help us specify versions of cookbooks in the runlist.

So do you guys manually maintain your versions in a role cookbook metadata
file for each role? Or do you still use your environment files because you
don’t have a reason to have more than one version of a service in
production at the same time? Did you just do a massive switchover to role
cookbooks? Or have you slowly transitioned, having some basic roles as well
as role cookbooks for a time?

I appreciate your help!

Kayla

On Fri, Jan 18, 2013 at 7:46 PM, Sascha Bates sascha.bates@gmail.comwrote:

Hi Sean, Reality Forge wrote a blog post a few months ago that I’ve
been using to model some experiments.

http://realityforge.org/code/2012/11/19/role-cookbooks-and-wrapper-cookbooks.html

We’re working through a lot of your same issues at the moment and I’m not
sure I have any comments yet on how we are experimenting with the overall
wrapping structure, but I can tell you how I’ve been writing the "library"
cookbooks. My general approach right now is, if I’m engaged in writing a
cookbook I think will become a general use cookbook, I iterate like this:

Write a recipe to define optimal flow, test the recipe to make sure it’s
working like I think it should.
If I have time, I refactor it back into a definition or LWRP which is a
good way to abstract attributes and hide code from devs, creating a clean
interface for them.
If I don’t have time, I parametrize with default attributes and write up
notes in the README about how to override the attrs from a wrapper
implementation .

LIke I said, we’re still experimenting with wrapper implementations, so I
don’t have many comments on that yet.

Hopefully the blog link above helps. If you are looking for an example, I
pasted an innocuous set of snippets showing a template block from a recipe
associated attributes file in both a library cookbook and the wrapper. I’m
no authority and YMMV and all that stuff, but these are certainly good
questions to ask as many people are discovering.

Good Luck on your adventures.

Sascha

Sascha Bates | sascha.bates@gmail.com | 612 850 0444 | sascha.bates@skype| sascha_bates@yahoo|
On 1/18/13 5:37 PM, Sean McKenna wrote:

Hi,****


We have been using Chef for a while at my company and have been very happy
with it.****


We’ve come to a point where we are making a few changes to the way we use
Chef.****


I’d like to use wrapper cookbooks to wrap generic cookbooks that we can
contribute back to upstream. I’d also like to use role cookbooks so that we
can version our roles and make incremental changes to the runlists of our
machines without risking blowing up prod and without having to do the
role-renaming dance. At least that is my thought as of right now.****


We’ve actually had a good deal of debate on the subject and are working on
coming to a consensus, which brings me to the point. Besides the
Gangnam-style one (
http://devopsanywhere.blogspot.com/2012/11/how-to-write-reusable-chef-cookbooks.html)
and a few other high level articles, I haven’t been able to find anything
on how people are really doing this (i.e. versioning roles by using role
cookbooks and / or using wrapper cookbooks) and what the best practices are.



Any help or direction to info sources would be much appreciated.****


Thanks!****

Sean****



#5

On Tue, Jan 22, 2013 at 8:26 AM, Kayla Townsend kamabe@gmail.com wrote:

So do you guys manually maintain your versions in a role cookbook metadata
file for each role? Or do you still use your environment files because you
don’t have a reason to have more than one version of a service in
production at the same time? Did you just do a massive switchover to role
cookbooks? Or have you slowly transitioned, having some basic roles as well
as role cookbooks for a time?

We use role cookbooks but mostly keep the versions of the cookbooks in the
environment. We have contemplated moving some version information into the
role cookbook itself but as yet we have not found a scenario that justifies
the extra cost of this variation.


Cheers,

Peter Donald


#6

Hey Kayla,

I suggest that you think of your cookbooks in two different types: Application cookbooks and Library cookbooks.

An Application cookbook is what you develop for your company. Generally it contains a resource block which does the job of deploying a piece of software for your company. This cookbook should just do the job of deploying and configuring that software and it’s components. It’s metadata file probably has a long list of cookbooks it depends on and these dependencies should have their versions locked. We lock the versions here so when we pass our application cookbook to another developer to work on we know he gets the right dependencies. We developed (and use) Berkshelf to achieve this portableness. You can see a perfect example of this in the Chef 11 server cookbook https://github.com/opscode-cookbooks/chef-server. Notice the Berksfile and Vagrantfile at the root of the cookbook.

A Library cookbook is something that you include into your Application cookbook. It’s generally something you need to have but it isn’t something that the customers of your organization are really asking for. For example: If you’re Facebook then people want to share pictures of their stupid cats and cups of Starbucks that they are about to drink. They don’t care if your web application is running on NGINX or Apache, they just want to interact with your application. These Library cookbooks can often be obtained from the Opscode community site and should be re-usable from project to project. They should be highly customizable through the setting of various attributes which will change the way the cookbook functions. This will allow you to put something like this into a recipe in your Application cookbook:

Cookbook Name:: my_face

Recipe:: web_proxy

node.set[:nginx][:install_method] = "source"
include_recipe “nginx::default”

This recipe above may actually be all there is to it. You would have successfully installed NGINX from source and that’s all you care about for setting up a web_proxy for your application “my_face”. Since we want to make sure the way we install NGINX from source with the NGINX cookbook doesn’t change, we’ll go into our metadata.rb and lock the version.

Now on your environment file you don’t want to lock NGINX down because we’ve already locked it in the “my_face” cookbook. Instead you want to lock the version of the “my_face” cookbook. If you recall a bit ago in this insanely TLDR email I mentioned that NGINX should be invisible to your users so this should make sense to why we wouldn’t lock NGINX here. In the end you should have an environment file which locks down the versions of all of your application cookbooks and completely ignores the version of your library cookbooks.

One minor note: A role does nothing but set a run_list and some attributes. A recipe can do these things, too. The biggest difference is that we have put a mental label on what a Role is and it’s really easy to explain that to people. The naming conventions in Chef are cute, but they really hurt us when we try to teach these concepts to people or when we try to begin to think outside the box. In my experience it has been difficult to explain to people why Roles are not necessary because they can’t seem to reify a Recipe the same they have a Role.

I hope this clears things up for you!


Jamie Winsor
@resetexistence

On Monday, January 21, 2013 at 1:41 PM, Peter Donald wrote:

On Tue, Jan 22, 2013 at 8:26 AM, Kayla Townsend <kamabe@gmail.com (mailto:kamabe@gmail.com)> wrote:

So do you guys manually maintain your versions in a role cookbook metadata file for each role? Or do you still use your environment files because you don’t have a reason to have more than one version of a service in production at the same time? Did you just do a massive switchover to role cookbooks? Or have you slowly transitioned, having some basic roles as well as role cookbooks for a time?

We use role cookbooks but mostly keep the versions of the cookbooks in the environment. We have contemplated moving some version information into the role cookbook itself but as yet we have not found a scenario that justifies the extra cost of this variation.


Cheers,

Peter Donald


#7

Thanks! That helps a lot! I think I was stuck on the all or nothing
approach when it came to environments.

On Mon, Jan 21, 2013 at 4:16 PM, Jamie Winsor jamie@vialstudios.com wrote:

Hey Kayla,

I suggest that you think of your cookbooks in two different types:
Application cookbooks and Library cookbooks.

An Application cookbook is what you develop for your company. Generally it
contains a resource block which does the job of deploying a piece of
software for your company. This cookbook should just do the job of
deploying and configuring that software and it’s components. It’s metadata
file probably has a long list of cookbooks it depends on and these
dependencies should have their versions locked. We lock the versions here
so when we pass our application cookbook to another developer to work on we
know he gets the right dependencies. We developed (and use) Berkshelf to
achieve this portableness. You can see a perfect example of this in the
Chef 11 server cookbook https://github.com/opscode-cookbooks/chef-server.
Notice the Berksfile and Vagrantfile at the root of the cookbook.

A Library cookbook is something that you include into your Application
cookbook. It’s generally something you need to have but it isn’t something
that the customers of your organization are really asking for. For example:
If you’re Facebook then people want to share pictures of their stupid cats
and cups of Starbucks that they are about to drink. They don’t care if your
web application is running on NGINX or Apache, they just want to interact
with your application. These Library cookbooks can often be obtained from
the Opscode community site and should be re-usable from project to project.
They should be highly customizable through the setting of various
attributes which will change the way the cookbook functions. This will
allow you to put something like this into a recipe in your Application
cookbook:

Cookbook Name:: my_face

Recipe:: web_proxy

node.set[:nginx][:install_method] = "source"
include_recipe “nginx::default”

This recipe above may actually be all there is to it. You would have
successfully installed NGINX from source and that’s all you care about for
setting up a web_proxy for your application “my_face”. Since we want to
make sure the way we install NGINX from source with the NGINX cookbook
doesn’t change, we’ll go into our metadata.rb and lock the version.

Now on your environment file you don’t want to lock NGINX down because
we’ve already locked it in the “my_face” cookbook. Instead you want to lock
the version of the “my_face” cookbook. If you recall a bit ago in this
insanely TLDR email I mentioned that NGINX should be invisible to your
users so this should make sense to why we wouldn’t lock NGINX here. In the
end you should have an environment file which locks down the versions of
all of your application cookbooks and completely ignores the version of
your library cookbooks.

One minor note: A role does nothing but set a run_list and some
attributes. A recipe can do these things, too. The biggest difference is
that we have put a mental label on what a Role is and it’s really easy to
explain that to people. The naming conventions in Chef are cute, but they
really hurt us when we try to teach these concepts to people or when we try
to begin to think outside the box. In my experience it has been difficult
to explain to people why Roles are not necessary because they can’t seem to
reify a Recipe the same they have a Role.

I hope this clears things up for you!


Jamie Winsor
@resetexistence
https://github.com/reset

On Monday, January 21, 2013 at 1:41 PM, Peter Donald wrote:

On Tue, Jan 22, 2013 at 8:26 AM, Kayla Townsend kamabe@gmail.com wrote:

So do you guys manually maintain your versions in a role cookbook
metadata file for each role? Or do you still use your environment files
because you don’t have a reason to have more than one version of a service
in production at the same time? Did you just do a massive switchover to
role cookbooks? Or have you slowly transitioned, having some basic roles as
well as role cookbooks for a time?

We use role cookbooks but mostly keep the versions of the cookbooks in the
environment. We have contemplated moving some version information into the
role cookbook itself but as yet we have not found a scenario that justifies
the extra cost of this variation.


Cheers,

Peter Donald