Understanding the Chef philosophy


#1

I’m a (relative) beginner with Chef. One of the things I’ve been struggling with is understanding how to use community cookbooks, and whether there is an underlying philosophy behind how to use it. At first glance, it seems to me that many cookbooks seem to reinvent the wheel. Obviously there are many smart minds at work here, so there probably is a fundamental reason that I’m just not getting.

Case in point - and something I just ran into: the yum cookbook to manage repositories.

This cookbook creates .repo files; as far as I can tell, it does not support downloading epel-release*.rpm and installing that RPM.

The reason this tripped me up was that I tried to install the remi-collet repository (by RPM), which depends on the EPEL RPM.

In other cases, cookbooks put files into non-standard files or directory structures (which tends to break SELinux and native tools; SELinux is an absolute must-have for me).

Sure, this particular example is easy to solve.

But I hope to better understand the underlying idea so I can better leverage all the community cookbooks without giving up on the features of the platform.


#2

Hi Kevin,

The yum cookbook located on the Community Site http://ckbk.it/yum has
a recipe named yum::epel - so including that recipe in your roles
prior to installing other packages would ensure that the correct EPEL
repo is created.

In the past, EPEL have broken quite a few of their installation rpms
and this has become the best way to maintain the code.

In practice for your problem of remi-collect, I suspect that adding
the yum::epel recipe to your run_list prior to installing
remi-colelct will work, as the epel-release*.rpm is inside the EPEL
repository - installing the package that depends on this should pull
down the epel-release package as a dependency.

Best,
-Mike

On Fri, Nov 16, 2012 at 9:19 PM, Kevin Keane (subscriptions)
subscription@kkeane.com wrote:

I’m a (relative) beginner with Chef. One of the things I’ve been struggling
with is understanding how to use community cookbooks, and whether there is
an underlying philosophy behind how to use it. At first glance, it seems to
me that many cookbooks seem to reinvent the wheel. Obviously there are many
smart minds at work here, so there probably is a fundamental reason that I’m
just not getting.

Case in point - and something I just ran into: the yum cookbook to manage
repositories.

This cookbook creates .repo files; as far as I can tell, it does not support
downloading epel-release*.rpm and installing that RPM.

The reason this tripped me up was that I tried to install the remi-collet
repository (by RPM), which depends on the EPEL RPM.

In other cases, cookbooks put files into non-standard files or directory
structures (which tends to break SELinux and native tools; SELinux is an
absolute must-have for me).

Sure, this particular example is easy to solve.

But I hope to better understand the underlying idea so I can better leverage
all the community cookbooks without giving up on the features of the
platform.


#3

Ah, thanks!

You are right about the specific solution. The other way to solve the remi issue is to also use the yum repository to create it

But my question was more about the philosophy, because I’ve seen similar things elsewhere. So it seems to be a case-by-case decision, instead of an overriding philosophy. Good to know!

I think I’ll fork the yum recipe and add support for installing from RPM. Might be a good exercise for me, if nothing else.

-----Original message-----
From:Mike miketheman@gmail.com
Sent:Sat 11-17-2012 08:22 am
Subject:[chef] Re: Understanding the Chef philosophy
To:chef@lists.opscode.com;
Hi Kevin,

The yum cookbook located on the Community Site http://ckbk.it/yum has
a recipe named yum::epel - so including that recipe in your roles
prior to installing other packages would ensure that the correct EPEL
repo is created.

In the past, EPEL have broken quite a few of their installation rpms
and this has become the best way to maintain the code.

In practice for your problem of remi-collect, I suspect that adding
the yum::epel recipe to your run_list prior to installing
remi-colelct will work, as the epel-release*.rpm is inside the EPEL
repository - installing the package that depends on this should pull
down the epel-release package as a dependency.

Best,
-Mike

On Fri, Nov 16, 2012 at 9:19 PM, Kevin Keane (subscriptions)
subscription@kkeane.com wrote:

I’m a (relative) beginner with Chef. One of the things I’ve been struggling
with is understanding how to use community cookbooks, and whether there is
an underlying philosophy behind how to use it. At first glance, it seems to
me that many cookbooks seem to reinvent the wheel. Obviously there are many
smart minds at work here, so there probably is a fundamental reason that I’m
just not getting.

Case in point - and something I just ran into: the yum cookbook to manage
repositories.

This cookbook creates .repo files; as far as I can tell, it does not support
downloading epel-release*.rpm and installing that RPM.

The reason this tripped me up was that I tried to install the remi-collet
repository (by RPM), which depends on the EPEL RPM.

In other cases, cookbooks put files into non-standard files or directory
structures (which tends to break SELinux and native tools; SELinux is an
absolute must-have for me).

Sure, this particular example is easy to solve.

But I hope to better understand the underlying idea so I can better leverage
all the community cookbooks without giving up on the features of the
platform.


#4

I can’t speak to the over-arching philosophy, but my approach to
writing reusable chef cookbooks is:

  • Do no harm
  • Do the thing that you would typically expect to happen had you not used Chef

This means that setting defaults for configuration files based on what
one would normally expect to be there, and making them attributes that
can be configured/overridden by the user.

Adhering to platform-specific behaviors is also a good way to go, one
major example of where this isn’t the case is the apache2 cookbook,
where the author(s) have determined that the Debian/Ubuntu style of
configuration made more sense than the RHEL/CentOS one, and adopted
the same configuration style for both platforms.

Regarding the yum cookbook, I believe the installable RPM was actually
included in version 1.0.0 1. Post-1.0.0, Eric submitted COOK1772 2
which was accepted and merged into the master branch.
Are you working off master, or version 1.0.0 from the community site? 3

You can read Eric’s reasoning behind the change on his pull request
4 - I think he does a great job of explaining the why’s and
wherefore’s.

-Mike

On Sat, Nov 17, 2012 at 4:16 PM, Kevin Keane (subscriptions)
subscription@kkeane.com wrote:

Ah, thanks!

You are right about the specific solution. The other way to solve the remi
issue is to also use the yum repository to create it

But my question was more about the philosophy, because I’ve seen similar
things elsewhere. So it seems to be a case-by-case decision, instead of an
overriding philosophy. Good to know!

I think I’ll fork the yum recipe and add support for installing from RPM.
Might be a good exercise for me, if nothing else.

-----Original message-----
From: Mike miketheman@gmail.com
Sent: Sat 11-17-2012 08:22 am
Subject: [chef] Re: Understanding the Chef philosophy
To: chef@lists.opscode.com;
Hi Kevin,

The yum cookbook located on the Community Site http://ckbk.it/yum has
a recipe named yum::epel - so including that recipe in your roles
prior to installing other packages would ensure that the correct EPEL
repo is created.

In the past, EPEL have broken quite a few of their installation rpms
and this has become the best way to maintain the code.

In practice for your problem of remi-collect, I suspect that adding
the yum::epel recipe to your run_list prior to installing
remi-colelct will work, as the epel-release*.rpm is inside the EPEL
repository - installing the package that depends on this should pull
down the epel-release package as a dependency.

Best,
-Mike

On Fri, Nov 16, 2012 at 9:19 PM, Kevin Keane (subscriptions)
subscription@kkeane.com wrote:

I’m a (relative) beginner with Chef. One of the things I’ve been
struggling
with is understanding how to use community cookbooks, and whether there is
an underlying philosophy behind how to use it. At first glance, it seems
to
me that many cookbooks seem to reinvent the wheel. Obviously there are
many
smart minds at work here, so there probably is a fundamental reason that
I’m
just not getting.

Case in point - and something I just ran into: the yum cookbook to manage
repositories.

This cookbook creates .repo files; as far as I can tell, it does not
support
downloading epel-release*.rpm and installing that RPM.

The reason this tripped me up was that I tried to install the remi-collet
repository (by RPM), which depends on the EPEL RPM.

In other cases, cookbooks put files into non-standard files or directory
structures (which tends to break SELinux and native tools; SELinux is an
absolute must-have for me).

Sure, this particular example is easy to solve.

But I hope to better understand the underlying idea so I can better
leverage
all the community cookbooks without giving up on the features of the
platform.


#5

Kevin:

Here is a stab (knife pun) at your question:

I use the apache, redis, rabbitmq and python community cookbooks. I use
them to get a sane and “cheffy” install of these services and set the basic
defaults.

Then, in my application cookbooks, I have a recipe that configures each
server the specific way I want it for my application. For example a recipe
that checks available memory on a server and the role of the machine then
sets the apache worker.mpm settings accordingly.

So my pattern is community delivers the goods and I bake them to order.
(oh take-n-bake pun score!).

I would also like to call you attention to the efforts of Noah Kantrowitz
(sp?) who is looking at better abstracting such things. (Can someone help
me with a link?) His talk at the community summit was really intriguing
and the thinking behind the idea is really excellent!

Hope that gives you a start
Boyd

On Sun, Nov 18, 2012 at 8:33 AM, Mike miketheman@gmail.com wrote:

I can’t speak to the over-arching philosophy, but my approach to
writing reusable chef cookbooks is:

  • Do no harm
  • Do the thing that you would typically expect to happen had you not used
    Chef

This means that setting defaults for configuration files based on what
one would normally expect to be there, and making them attributes that
can be configured/overridden by the user.

Adhering to platform-specific behaviors is also a good way to go, one
major example of where this isn’t the case is the apache2 cookbook,
where the author(s) have determined that the Debian/Ubuntu style of
configuration made more sense than the RHEL/CentOS one, and adopted
the same configuration style for both platforms.

Regarding the yum cookbook, I believe the installable RPM was actually
included in version 1.0.0 1. Post-1.0.0, Eric submitted COOK1772 2
which was accepted and merged into the master branch.
Are you working off master, or version 1.0.0 from the community site? 3

You can read Eric’s reasoning behind the change on his pull request
4 - I think he does a great job of explaining the why’s and
wherefore’s.

-Mike

On Sat, Nov 17, 2012 at 4:16 PM, Kevin Keane (subscriptions)
subscription@kkeane.com wrote:

Ah, thanks!

You are right about the specific solution. The other way to solve the
remi
issue is to also use the yum repository to create it

But my question was more about the philosophy, because I’ve seen similar
things elsewhere. So it seems to be a case-by-case decision, instead of
an
overriding philosophy. Good to know!

I think I’ll fork the yum recipe and add support for installing from RPM.
Might be a good exercise for me, if nothing else.

-----Original message-----
From: Mike miketheman@gmail.com
Sent: Sat 11-17-2012 08:22 am
Subject: [chef] Re: Understanding the Chef philosophy
To: chef@lists.opscode.com;
Hi Kevin,

The yum cookbook located on the Community Site http://ckbk.it/yum has
a recipe named yum::epel - so including that recipe in your roles
prior to installing other packages would ensure that the correct EPEL
repo is created.

In the past, EPEL have broken quite a few of their installation rpms
and this has become the best way to maintain the code.

In practice for your problem of remi-collect, I suspect that adding
the yum::epel recipe to your run_list prior to installing
remi-colelct will work, as the epel-release*.rpm is inside the EPEL
repository - installing the package that depends on this should pull
down the epel-release package as a dependency.

Best,
-Mike

On Fri, Nov 16, 2012 at 9:19 PM, Kevin Keane (subscriptions)
subscription@kkeane.com wrote:

I’m a (relative) beginner with Chef. One of the things I’ve been
struggling
with is understanding how to use community cookbooks, and whether there
is

an underlying philosophy behind how to use it. At first glance, it seems
to
me that many cookbooks seem to reinvent the wheel. Obviously there are
many
smart minds at work here, so there probably is a fundamental reason that
I’m
just not getting.

Case in point - and something I just ran into: the yum cookbook to
manage

repositories.

This cookbook creates .repo files; as far as I can tell, it does not
support
downloading epel-release*.rpm and installing that RPM.

The reason this tripped me up was that I tried to install the
remi-collet

repository (by RPM), which depends on the EPEL RPM.

In other cases, cookbooks put files into non-standard files or directory
structures (which tends to break SELinux and native tools; SELinux is an
absolute must-have for me).

Sure, this particular example is easy to solve.

But I hope to better understand the underlying idea so I can better
leverage
all the community cookbooks without giving up on the features of the
platform.


#6

Hi,

On Tue, Nov 20, 2012 at 10:15 AM, Boyd Hemphill behemphi@gmail.com wrote:

I would also like to call you attention to the efforts of Noah Kantrowitz
(sp?) who is looking at better abstracting such things. (Can someone help
me with a link?) His talk at the community summit was really intriguing and
the thinking behind the idea is really excellent!

There was a recent food fight hangout about this topic at
http://foodfightshow.org/2012/11/google-plus-hangouts.html

There is also some notes from the community summit at
http://wiki.opscode.com/pages/viewpage.action?pageId=26510041

Bryan Berry wrote “How to Write Reusable Chef Cookbooks, Gangnam
Style” at http://devopsanywhere.blogspot.it/2012/11/how-to-write-reusable-chef-cookbooks.html

Jamie Winsor wrote “Authoring a Chef Cookbook as a Developer” at
http://vialstudios.com/guide-authoring-cookbooks.html

I have written a couple of blog posts “Reusable Cookbooks Revisited”
(http://realityforge.org/code/2012/11/12/reusable-cookbooks-revisited.html),
“Evolving Towards Cookbook Reusability”
(http://realityforge.org/code/2012/05/12/evolving-towards-cookbook-reusability-in-chef.html)
and “Role Cookbooks and Wrapper Cookbooks”
(http://realityforge.org/code/2012/11/19/role-cookbooks-and-wrapper-cookbooks.html)

and lastly listen to the foodfight show (http://foodfightshow.org/) as
the topic intermittently comes up there. I forget which episodes but I
am sure there is people around here that could give you pointers …


Cheers,

Peter Donald


#7

Boyd,

Thank you for explaining your method! And thanks to everybody else who responded, too. Based on what I’ve been reading here, I’m glad to hear that this is not based on a general philosophical principle, but rather decisions applicable to specific packages. I can confirm that; the postfix cookbook seems to work wonderfully for me.

It seems that with the apache2 cookbook, I may simply have been trying to use the wrong tool for the job.

What it comes down to is that, at least in some cases, you can’t really mix the community way and the native way very well.

For you, that isn’t a problem, because, within Apache, you seem to have decided to go all-cookbook, no-native. Right tool for the job.

For me, it isn’t working, because while I would love to use a cookbook to manage Apache itself, much of the software I’m running on Apache comes as RPMs, with the RedHat assumptions built in: Zarafa, Z-Push, Icinga, Nagios, Cacti, Pnp4Nagios, OTRS. Some of these are easily available only as RPMs, making it very difficult to adjust paths.

The other major thing that this cookbook breaks on RedHat is SELinux - a major factor for me. Chef itself has a problem with SELinux (apparently because it first creates files in /tmp and then moves them, rather than creating the file in place), but that is fairly easily resolved with the restorecon command - as long as the configuration files adhere to the RedHat conventions; the SELinux policies have paths baked in.

Now to be clear: I understand just how difficult it is to maintain a multi-platform cookbook in the first place, and I admire the people who created the apache cookbook. A cookbook that, depending on the platform, creates a completely different structure is dramatically more complicated. On top of that, Apache itself is highly complex. Maybe one factor is that it’s simply a herculaean task to create such an Apache cookbook?

-----Original message-----
From:Boyd Hemphill behemphi@gmail.com
Sent:Mon 11-19-2012 03:16 pm
Subject:[chef] Re: Re: RE: Re: Understanding the Chef philosophy
To:chef@lists.opscode.com;
Kevin:
Here is a stab (knife pun) at your question:
I use the apache, redis, rabbitmq and python community cookbooks. I use them to get a sane and “cheffy” install of these services and set the basic defaults.

Then, in my application cookbooks, I have a recipe that configures each server the specific way I want it for my application. For example a recipe that checks available memory on a server and the role of the machine then sets the apache worker.mpm settings accordingly.

So my pattern is community delivers the goods and I bake them to order. (oh take-n-bake pun score!).
I would also like to call you attention to the efforts of Noah Kantrowitz (sp?) who is looking at better abstracting such things. (Can someone help me with a link?) His talk at the community summit was really intriguing and the thinking behind the idea is really excellent!

Hope that gives you a start
Boyd

On Sun, Nov 18, 2012 at 8:33 AM, Mike miketheman@gmail.com wrote:
I can’t speak to the over-arching philosophy, but my approach to
writing reusable chef cookbooks is:

  • Do no harm
  • Do the thing that you would typically expect to happen had you not used Chef

This means that setting defaults for configuration files based on what
one would normally expect to be there, and making them attributes that
can be configured/overridden by the user.

Adhering to platform-specific behaviors is also a good way to go, one
major example of where this isn’t the case is the apache2 cookbook,
where the author(s) have determined that the Debian/Ubuntu style of
configuration made more sense than the RHEL/CentOS one, and adopted
the same configuration style for both platforms.

Regarding the yum cookbook, I believe the installable RPM was actually
included in version 1.0.0 1. Post-1.0.0, Eric submitted COOK1772 2
which was accepted and merged into the master branch.
Are you working off master, or version 1.0.0 from the community site? 3

You can read Eric’s reasoning behind the change on his pull request
4 - I think he does a great job of explaining the why’s and
wherefore’s.

-Mike

On Sat, Nov 17, 2012 at 4:16 PM, Kevin Keane (subscriptions)

subscription@kkeane.com wrote:

Ah, thanks!

You are right about the specific solution. The other way to solve the remi
issue is to also use the yum repository to create it

But my question was more about the philosophy, because I’ve seen similar
things elsewhere. So it seems to be a case-by-case decision, instead of an
overriding philosophy. Good to know!

I think I’ll fork the yum recipe and add support for installing from RPM.
Might be a good exercise for me, if nothing else.

-----Original message-----
From: Mike miketheman@gmail.com
Sent: Sat 11-17-2012 08:22 am
Subject: [chef] Re: Understanding the Chef philosophy
To: chef@lists.opscode.com;
Hi Kevin,

The yum cookbook located on the Community Site http://ckbk.it/yum has
a recipe named yum::epel - so including that recipe in your roles
prior to installing other packages would ensure that the correct EPEL
repo is created.

In the past, EPEL have broken quite a few of their installation rpms
and this has become the best way to maintain the code.

In practice for your problem of remi-collect, I suspect that adding
the yum::epel recipe to your run_list prior to installing
remi-colelct will work, as the epel-release*.rpm is inside the EPEL
repository - installing the package that depends on this should pull
down the epel-release package as a dependency.

Best,
-Mike

On Fri, Nov 16, 2012 at 9:19 PM, Kevin Keane (subscriptions)
subscription@kkeane.com wrote:

I’m a (relative) beginner with Chef. One of the things I’ve been
struggling
with is understanding how to use community cookbooks, and whether there is
an underlying philosophy behind how to use it. At first glance, it seems
to
me that many cookbooks seem to reinvent the wheel. Obviously there are
many
smart minds at work here, so there probably is a fundamental reason that
I’m
just not getting.

Case in point - and something I just ran into: the yum cookbook to manage
repositories.

This cookbook creates .repo files; as far as I can tell, it does not
support
downloading epel-release*.rpm and installing that RPM.

The reason this tripped me up was that I tried to install the remi-collet
repository (by RPM), which depends on the EPEL RPM.

In other cases, cookbooks put files into non-standard files or directory
structures (which tends to break SELinux and native tools; SELinux is an
absolute must-have for me).

Sure, this particular example is easy to solve.

But I hope to better understand the underlying idea so I can better
leverage
all the community cookbooks without giving up on the features of the
platform.