Best practice for cross platform cookbook dependencies?


#1

I’m working on extending the existing nodejs cookbook to support installs on Windows. This brings up a bit of a conundrum…if I do this, I’m going to take a dependency hit on the “windows” cookbook. This will cause non-windows shops to now have another cookbook they don’t need. On the flip side, windows-only shops will have the useless to them “apt” cookbook.

Is there a solution that the community prefers? I could roll this out as a separate cookbook, but I would prefer to keep it all in one. Just eat the extra dependency overhead?

Thanks.

-Pete


#2

The dependency manifests as files located on the filesystem.
I don’t think there’s any other problems, are there?

On Mon, Feb 11, 2013 at 1:53 PM, Peter Loron peterl@standingwave.org wrote:

I’m working on extending the existing nodejs cookbook to support installs on Windows. This brings up a bit of a conundrum…if I do this, I’m going to take a dependency hit on the “windows” cookbook. This will cause non-windows shops to now have another cookbook they don’t need. On the flip side, windows-only shops will have the useless to them “apt” cookbook.

Is there a solution that the community prefers? I could roll this out as a separate cookbook, but I would prefer to keep it all in one. Just eat the extra dependency overhead?

Thanks.

-Pete


#3

I’d have to say yes.

My workflow involves running Berkshelf and Vagrant VMs to develop and test cookbooks.

The more dependencies a cookbook I use have, the longer it takes for me to iterate. Everytime I “vagrant up” or “vagrant provision”, every single cookbook that’ll be used for that session gets loaded on the system, uploaded to the Chef Server and downloaded in the VM.

I like the way the apache2 cookbook [0] deals with that: no direct dependencies, default recipe doesn’t need any. If you load a recipe with an external dependency, make sure to include whatever recipe you need from that on the node (like it does with logrotate, for example).

I’m not sure that applies to the nodejs cookbook, but it surely is an annoyance that should be fixed somehow.

I feel the same about having to load up apt cookbooks onto CentOS/RHEL or Windows nodes.

Even though the inclusion of unused cookbooks brings no harm in terms of cuntionality, it does slow down both cookbook development and node conversion.

  • cassiano

On Monday, February 11, 2013 at 16:57, Mike wrote:

The dependency manifests as files located on the filesystem.
I don’t think there’s any other problems, are there?

On Mon, Feb 11, 2013 at 1:53 PM, Peter Loron <peterl@standingwave.org (mailto:peterl@standingwave.org)> wrote:

I’m working on extending the existing nodejs cookbook to support installs on Windows. This brings up a bit of a conundrum…if I do this, I’m going to take a dependency hit on the “windows” cookbook. This will cause non-windows shops to now have another cookbook they don’t need. On the flip side, windows-only shops will have the useless to them “apt” cookbook.

Is there a solution that the community prefers? I could roll this out as a separate cookbook, but I would prefer to keep it all in one. Just eat the extra dependency overhead?

Thanks.

-Pete


#4

https://github.com/erikh/knife_cookbook_sync can help you with the upload
performance (which in my experience is the slowest part of this).

It probably needs some touchups for Chef 11.

On Mon, Feb 11, 2013 at 1:00 PM, Cassiano Leal cassianoleal@gmail.comwrote:

I’d have to say yes.

My workflow involves running Berkshelf and Vagrant VMs to develop and test
cookbooks.

The more dependencies a cookbook I use have, the longer it takes for me to
iterate. Everytime I “vagrant up” or “vagrant provision”, every single
cookbook that’ll be used for that session gets loaded on the system,
uploaded to the Chef Server and downloaded in the VM.

I like the way the apache2 cookbook [0] deals with that: no direct
dependencies, default recipe doesn’t need any. If you load a recipe with an
external dependency, make sure to include whatever recipe you need from
that on the node (like it does with logrotate, for example).

I’m not sure that applies to the nodejs cookbook, but it surely is an
annoyance that should be fixed somehow.

I feel the same about having to load up apt cookbooks onto CentOS/RHEL or
Windows nodes.

Even though the inclusion of unused cookbooks brings no harm in terms of
cuntionality, it does slow down both cookbook development and node
conversion.

  • cassiano

On Monday, February 11, 2013 at 16:57, Mike wrote:

The dependency manifests as files located on the filesystem.
I don’t think there’s any other problems, are there?

On Mon, Feb 11, 2013 at 1:53 PM, Peter Loron peterl@standingwave.org
wrote:

I’m working on extending the existing nodejs cookbook to support installs
on Windows. This brings up a bit of a conundrum…if I do this, I’m going to
take a dependency hit on the “windows” cookbook. This will cause
non-windows shops to now have another cookbook they don’t need. On the flip
side, windows-only shops will have the useless to them “apt” cookbook.

Is there a solution that the community prefers? I could roll this out as a
separate cookbook, but I would prefer to keep it all in one. Just eat the
extra dependency overhead?

Thanks.

-Pete


#5

Ohai,

I agree that it is annoying, and frustrating for the feedback cycle to take longer than is really necessary. However, I think having cookbooks’ default use pattern broken is even more annoying.

In the ease of use department, Chef already has a fairly steep learning curve. New users to Chef often have to learn:

  • A new paradigm for managing systems (declarative, convergent resources).
  • New languages for defining their infrastructure (Ruby, JSON).
  • Manage things through an API instead of text files.
  • A new authentication system that is similar, but not exactly the same as one(s) they know.
  • Several new command line tools with their own configuration files.
  • A dependency management system to distribute automation components to their systems.

The simple example or demonstration is often magical and “just works.” But a single change can cause things to break in spectacular ways. Sometimes you get a nice clear error, like “this resource wasn’t found (because you typed tumplate instead of template).” But errors like, “Undefined method `[]’ on NilClass” because you didn’t have a dependency on a cookbook that had an attribute is really confusing.

By adding stronger dependencies, we can often remove these confusing errors for the default use case. Documentation on cookbook usage should be up to date with how you actually use the thing, but let’s face it, in our busy schedules not everyone is going to have the time to read 5 paragraphs of new jargon to get something working.

For Opscode’s cookbooks in particular, we try to remove as many dependencies as possible for the default use case for a particular cookbook. This means for example, we have statements like this:

package "perl"

In a recipe, rather than incuding another cookbook as a dependency. We also try to document dependencies when the default use case doesn’t require them. For example, if you’re using chef-client::service, it can set up services using a variety of init styles. You have to know which ones it supports (like bluepill or runit), and then change the appropriate attribute. The reasoning to not have these as dependencies is because you’d want those cookbooks elsewhere in your node’s run list, such as in a “base”/“common” role or similar.

Unfortunately we haven’t been consistent across all our own cookbooks. I know it’s annoying and I’m sorry about that.

This is an unsolved problem until one or all of the following are solved:

  • suggests/recommends have a behavior (configurable, ideally) which allows them to be downloaded. Right now they do nothing and only serve as loose documentation.
  • per-platform dependency mapping, so CentOS doesn’t get “windows” or Ubuntu doesn’t get “yum”, for example.

I was unable to find a ticket on these topics, so I opened these:

Cheers,
Joshua

On Feb 11, 2013, at 2:00 PM, Cassiano Leal cassianoleal@gmail.com wrote:

I’d have to say yes.

My workflow involves running Berkshelf and Vagrant VMs to develop and test cookbooks.

The more dependencies a cookbook I use have, the longer it takes for me to iterate. Everytime I “vagrant up” or “vagrant provision”, every single cookbook that’ll be used for that session gets loaded on the system, uploaded to the Chef Server and downloaded in the VM.

I like the way the apache2 cookbook [0] deals with that: no direct dependencies, default recipe doesn’t need any. If you load a recipe with an external dependency, make sure to include whatever recipe you need from that on the node (like it does with logrotate, for example).

I’m not sure that applies to the nodejs cookbook, but it surely is an annoyance that should be fixed somehow.

I feel the same about having to load up apt cookbooks onto CentOS/RHEL or Windows nodes.

Even though the inclusion of unused cookbooks brings no harm in terms of cuntionality, it does slow down both cookbook development and node conversion.

  • cassiano

On Monday, February 11, 2013 at 16:57, Mike wrote:

The dependency manifests as files located on the filesystem.
I don’t think there’s any other problems, are there?

On Mon, Feb 11, 2013 at 1:53 PM, Peter Loron peterl@standingwave.org wrote:

I’m working on extending the existing nodejs cookbook to support installs on Windows. This brings up a bit of a conundrum…if I do this, I’m going to take a dependency hit on the “windows” cookbook. This will cause non-windows shops to now have another cookbook they don’t need. On the flip side, windows-only shops will have the useless to them “apt” cookbook.

Is there a solution that the community prefers? I could roll this out as a separate cookbook, but I would prefer to keep it all in one. Just eat the extra dependency overhead?

Thanks.

-Pete


Joshua Timberman
Technical Community Manager, Opscode, Inc
IRC, Skype, Twitter, Github: jtimberman


#6

That’s nice. Shortly after I wrote the e-mail I wondered about the possibility of implementing per-platform dependencies, so thanks for the tickets!

  • cassiano

On Tuesday, February 12, 2013 at 18:56, Joshua Timberman wrote:

Ohai,

I agree that it is annoying, and frustrating for the feedback cycle to take longer than is really necessary. However, I think having cookbooks’ default use pattern broken is even more annoying.

In the ease of use department, Chef already has a fairly steep learning curve. New users to Chef often have to learn:

  • A new paradigm for managing systems (declarative, convergent resources).
  • New languages for defining their infrastructure (Ruby, JSON).
  • Manage things through an API instead of text files.
  • A new authentication system that is similar, but not exactly the same as one(s) they know.
  • Several new command line tools with their own configuration files.
  • A dependency management system to distribute automation components to their systems.

The simple example or demonstration is often magical and “just works.” But a single change can cause things to break in spectacular ways. Sometimes you get a nice clear error, like “this resource wasn’t found (because you typed tumplate instead of template).” But errors like, “Undefined method `[]’ on NilClass” because you didn’t have a dependency on a cookbook that had an attribute is really confusing.

By adding stronger dependencies, we can often remove these confusing errors for the default use case. Documentation on cookbook usage should be up to date with how you actually use the thing, but let’s face it, in our busy schedules not everyone is going to have the time to read 5 paragraphs of new jargon to get something working.

For Opscode’s cookbooks in particular, we try to remove as many dependencies as possible for the default use case for a particular cookbook. This means for example, we have statements like this:

package “perl”

In a recipe, rather than incuding another cookbook as a dependency. We also try to document dependencies when the default use case doesn’t require them. For example, if you’re using chef-client::service, it can set up services using a variety of init styles. You have to know which ones it supports (like bluepill or runit), and then change the appropriate attribute. The reasoning to not have these as dependencies is because you’d want those cookbooks elsewhere in your node’s run list, such as in a “base”/“common” role or similar.

Unfortunately we haven’t been consistent across all our own cookbooks. I know it’s annoying and I’m sorry about that.

This is an unsolved problem until one or all of the following are solved:

  • suggests/recommends have a behavior (configurable, ideally) which allows them to be downloaded. Right now they do nothing and only serve as loose documentation.
  • per-platform dependency mapping, so CentOS doesn’t get “windows” or Ubuntu doesn’t get “yum”, for example.

I was unable to find a ticket on these topics, so I opened these:

Cheers,
Joshua

On Feb 11, 2013, at 2:00 PM, Cassiano Leal <cassianoleal@gmail.com (mailto:cassianoleal@gmail.com)> wrote:

I’d have to say yes.

My workflow involves running Berkshelf and Vagrant VMs to develop and test cookbooks.

The more dependencies a cookbook I use have, the longer it takes for me to iterate. Everytime I “vagrant up” or “vagrant provision”, every single cookbook that’ll be used for that session gets loaded on the system, uploaded to the Chef Server and downloaded in the VM.

I like the way the apache2 cookbook [0] deals with that: no direct dependencies, default recipe doesn’t need any. If you load a recipe with an external dependency, make sure to include whatever recipe you need from that on the node (like it does with logrotate, for example).

I’m not sure that applies to the nodejs cookbook, but it surely is an annoyance that should be fixed somehow.

I feel the same about having to load up apt cookbooks onto CentOS/RHEL or Windows nodes.

Even though the inclusion of unused cookbooks brings no harm in terms of cuntionality, it does slow down both cookbook development and node conversion.

  • cassiano

On Monday, February 11, 2013 at 16:57, Mike wrote:

The dependency manifests as files located on the filesystem.
I don’t think there’s any other problems, are there?

On Mon, Feb 11, 2013 at 1:53 PM, Peter Loron <peterl@standingwave.org (mailto:peterl@standingwave.org)> wrote:

I’m working on extending the existing nodejs cookbook to support installs on Windows. This brings up a bit of a conundrum…if I do this, I’m going to take a dependency hit on the “windows” cookbook. This will cause non-windows shops to now have another cookbook they don’t need. On the flip side, windows-only shops will have the useless to them “apt” cookbook.

Is there a solution that the community prefers? I could roll this out as a separate cookbook, but I would prefer to keep it all in one. Just eat the extra dependency overhead?

Thanks.

-Pete


Joshua Timberman
Technical Community Manager, Opscode, Inc
IRC, Skype, Twitter, Github: jtimberman


#7

Will that speedup Berkshelf’s upload? Judging by the readme, it’s a separate knife subcommand that has to be explicitly invoked.

  • cassiano

On Monday, February 11, 2013 at 19:19, Erik Hollensbe wrote:

https://github.com/erikh/knife_cookbook_sync can help you with the upload performance (which in my experience is the slowest part of this).

It probably needs some touchups for Chef 11.

On Mon, Feb 11, 2013 at 1:00 PM, Cassiano Leal <cassianoleal@gmail.com (mailto:cassianoleal@gmail.com)> wrote:

I’d have to say yes.

My workflow involves running Berkshelf and Vagrant VMs to develop and test cookbooks.

The more dependencies a cookbook I use have, the longer it takes for me to iterate. Everytime I “vagrant up” or “vagrant provision”, every single cookbook that’ll be used for that session gets loaded on the system, uploaded to the Chef Server and downloaded in the VM.

I like the way the apache2 cookbook [0] deals with that: no direct dependencies, default recipe doesn’t need any. If you load a recipe with an external dependency, make sure to include whatever recipe you need from that on the node (like it does with logrotate, for example).

I’m not sure that applies to the nodejs cookbook, but it surely is an annoyance that should be fixed somehow.

I feel the same about having to load up apt cookbooks onto CentOS/RHEL or Windows nodes.

Even though the inclusion of unused cookbooks brings no harm in terms of cuntionality, it does slow down both cookbook development and node conversion.

  • cassiano

On Monday, February 11, 2013 at 16:57, Mike wrote:

The dependency manifests as files located on the filesystem.
I don’t think there’s any other problems, are there?

On Mon, Feb 11, 2013 at 1:53 PM, Peter Loron <peterl@standingwave.org (mailto:peterl@standingwave.org)> wrote:

I’m working on extending the existing nodejs cookbook to support installs on Windows. This brings up a bit of a conundrum…if I do this, I’m going to take a dependency hit on the “windows” cookbook. This will cause non-windows shops to now have another cookbook they don’t need. On the flip side, windows-only shops will have the useless to them “apt” cookbook.

Is there a solution that the community prefers? I could roll this out as a separate cookbook, but I would prefer to keep it all in one. Just eat the extra dependency overhead?

Thanks.

-Pete


#8

It’s a separate knife command, so no, unless you want to use a very early version of berkshelf that used knife plugins.


Erik Hollensbe
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)

On Wednesday, February 13, 2013 at 6:31 AM, Cassiano Leal wrote:

Will that speedup Berkshelf’s upload? Judging by the readme, it’s a separate knife subcommand that has to be explicitly invoked.

  • cassiano

On Monday, February 11, 2013 at 19:19, Erik Hollensbe wrote:

https://github.com/erikh/knife_cookbook_sync can help you with the upload performance (which in my experience is the slowest part of this).

It probably needs some touchups for Chef 11.

On Mon, Feb 11, 2013 at 1:00 PM, Cassiano Leal <cassianoleal@gmail.com (mailto:cassianoleal@gmail.com)> wrote:

I’d have to say yes.

My workflow involves running Berkshelf and Vagrant VMs to develop and test cookbooks.

The more dependencies a cookbook I use have, the longer it takes for me to iterate. Everytime I “vagrant up” or “vagrant provision”, every single cookbook that’ll be used for that session gets loaded on the system, uploaded to the Chef Server and downloaded in the VM.

I like the way the apache2 cookbook [0] deals with that: no direct dependencies, default recipe doesn’t need any. If you load a recipe with an external dependency, make sure to include whatever recipe you need from that on the node (like it does with logrotate, for example).

I’m not sure that applies to the nodejs cookbook, but it surely is an annoyance that should be fixed somehow.

I feel the same about having to load up apt cookbooks onto CentOS/RHEL or Windows nodes.

Even though the inclusion of unused cookbooks brings no harm in terms of cuntionality, it does slow down both cookbook development and node conversion.

  • cassiano

On Monday, February 11, 2013 at 16:57, Mike wrote:

The dependency manifests as files located on the filesystem.
I don’t think there’s any other problems, are there?

On Mon, Feb 11, 2013 at 1:53 PM, Peter Loron <peterl@standingwave.org (mailto:peterl@standingwave.org)> wrote:

I’m working on extending the existing nodejs cookbook to support installs on Windows. This brings up a bit of a conundrum…if I do this, I’m going to take a dependency hit on the “windows” cookbook. This will cause non-windows shops to now have another cookbook they don’t need. On the flip side, windows-only shops will have the useless to them “apt” cookbook.

Is there a solution that the community prefers? I could roll this out as a separate cookbook, but I would prefer to keep it all in one. Just eat the extra dependency overhead?

Thanks.

-Pete