Dependency spaghetti


#1

(Please redirect me if this is the wrong list)

We seem to be in a place as a Chef community where the ‘depends’ feature of
cookbook metadata is getting used a little bit too freely.

For example, the ‘java’ cookbook includes a ‘depends’ for the 'windows’
cookbook - this means that I’m forced to download the ‘windows’ cookbook to
all of my *nix machines. Messy at best.

Shouldn’t we be using ‘suggests’? Or better yet, shouldn’t we reserve
dependency for cases where a cookbook really must have another cookbook
available to it to compile (and, shouldn’t we avoid tightly-coupled
dependencies whenever possible because we don’t wan to be brittle)?

This is especially painful for us because we are currently seeing very slow
download rates from hosted Chef (about 60kbps), so downloading addt’l
cookbooks makes things noticeably slower.

(Another great example of dependency hell is installing the 'nagios’
cookbook, which requires ‘apache’, which requires ‘php’, which in turn
requires ‘mysql’ - yet it’s not really true that all of these things are
intrinsically tied together - we are losing the things we’ve learned from
the Unix model and service-oriented architectures when we staple things to
each other this way.)

Okay, so that got a little bit ranty, but that’s because I really like Chef
and I hate to see a slim, sexy, powerful tool get bloated and brittle for
no good reason.

Discuss!

Blake


#2

I agree. I’d like to see a notion of “conditional dependence”, so that java depends on windows OR me setting an opt-out flag that says I won’t ever windows. I sometimes go in and edit these dependencies out of the metadata before I load it to my chef server. It happens a lot with yum or apt as well.

From: Blake Irvin <blake@wanelo.commailto:blake@wanelo.com>
Date: Wednesday, October 2, 2013 6:17 PM
To: "chef-dev@lists.opscode.commailto:chef-dev@lists.opscode.com Dev" <chef-dev@lists.opscode.commailto:chef-dev@lists.opscode.com>
Subject: [chef-dev] dependency spaghetti

(Please redirect me if this is the wrong list)

We seem to be in a place as a Chef community where the ‘depends’ feature of cookbook metadata is getting used a little bit too freely.

For example, the ‘java’ cookbook includes a ‘depends’ for the ‘windows’ cookbook - this means that I’m forced to download the ‘windows’ cookbook to all of my *nix machines. Messy at best.

Shouldn’t we be using ‘suggests’? Or better yet, shouldn’t we reserve dependency for cases where a cookbook really must have another cookbook available to it to compile (and, shouldn’t we avoid tightly-coupled dependencies whenever possible because we don’t wan to be brittle)?

This is especially painful for us because we are currently seeing very slow download rates from hosted Chef (about 60kbps), so downloading addt’l cookbooks makes things noticeably slower.

(Another great example of dependency hell is installing the ‘nagios’ cookbook, which requires ‘apache’, which requires ‘php’, which in turn requires ‘mysql’ - yet it’s not really true that all of these things are intrinsically tied together - we are losing the things we’ve learned from the Unix model and service-oriented architectures when we staple things to each other this way.)

Okay, so that got a little bit ranty, but that’s because I really like Chef and I hate to see a slim, sexy, powerful tool get bloated and brittle for no good reason.

Discuss!

Blake


#3

The ‘suggests’ and ‘recommends’ features of metadata are NOPs, they
don’t do anything. And the java cookbook is broken on windows without
the windows cookbook – so in fixing your pain you would break all the
windows people. The problem is that the list of cookbooks to download
is computed server-side by the dep-solver and it doesn’t have visibility
into the platform the client is running on. The client could send that
information to the server and the server could use it to filter the
results, or server could return enough metadata in response to
cookbook_versions that the client could filter the cookbooks itself, or
the client could do additional requests for the cookbook metadata and do
the filtering (but that means the additional round trips that you don’t
want), etc. But the problem is deeper than just switching the metadata
field.

We should also be doing downloads in parallel
(https://tickets.opscode.com/browse/CHEF-4423). That patch badly needs
some careful tests written for it, though, because we need to prove that
typhoeus works across our whole test matrix - and if it fails somewhere
like ruby 1.8 or AIX then it would need to get rewritten with net-http
and threads like dan suggested…

We also have issues where due to eventual consistency issues between S3
regions we force everything to talk to the US-East S3 region which
causes large delays in Europe and Asia.

On 10/2/13 4:17 PM, Blake Irvin wrote:

(Please redirect me if this is the wrong list)

We seem to be in a place as a Chef community where the 'depends’
feature of cookbook metadata is getting used a little bit too freely.

For example, the ‘java’ cookbook includes a ‘depends’ for the
’windows’ cookbook - this means that I’m forced to download the
’windows’ cookbook to all of my *nix machines. Messy at best.

Shouldn’t we be using ‘suggests’? Or better yet, shouldn’t we reserve
dependency for cases where a cookbook really must have another
cookbook available to it to compile (and, shouldn’t we avoid
tightly-coupled dependencies whenever possible because we don’t wan to
be brittle)?

This is especially painful for us because we are currently seeing very
slow download rates from hosted Chef (about 60kbps), so downloading
addt’l cookbooks makes things noticeably slower.

(Another great example of dependency hell is installing the 'nagios’
cookbook, which requires ‘apache’, which requires ‘php’, which in turn
requires ‘mysql’ - yet it’s not really true that all of these things
are intrinsically tied together - we are losing the things we’ve
learned from the Unix model and service-oriented architectures when we
staple things to each other this way.)

Okay, so that got a little bit ranty, but that’s because I really like
Chef and I hate to see a slim, sexy, powerful tool get bloated and
brittle for no good reason.

Discuss!

Blake


#4

There has been a long running concept to turn #suggests into an optional form of #depends, so it would have the same effect as depends if the cookbook is found in terms of triggering a download and loading in the right order, but wouldn’t cause a failure if that cookbook wasn’t found. Maybe after I get this dialect stuff squared away I’ll write up that patch :slight_smile:

–Noah

On Oct 2, 2013, at 5:26 PM, Bryan Taylor btaylor@rackspace.com wrote:

I agree. I’d like to see a notion of “conditional dependence”, so that java depends on windows OR me setting an opt-out flag that says I won’t ever windows. I sometimes go in and edit these dependencies out of the metadata before I load it to my chef server. It happens a lot with yum or apt as well.

From: Blake Irvin blake@wanelo.com
Date: Wednesday, October 2, 2013 6:17 PM
To: "chef-dev@lists.opscode.com Dev" chef-dev@lists.opscode.com
Subject: [chef-dev] dependency spaghetti

(Please redirect me if this is the wrong list)

We seem to be in a place as a Chef community where the ‘depends’ feature of cookbook metadata is getting used a little bit too freely.

For example, the ‘java’ cookbook includes a ‘depends’ for the ‘windows’ cookbook - this means that I’m forced to download the ‘windows’ cookbook to all of my *nix machines. Messy at best.

Shouldn’t we be using ‘suggests’? Or better yet, shouldn’t we reserve dependency for cases where a cookbook really must have another cookbook available to it to compile (and, shouldn’t we avoid tightly-coupled dependencies whenever possible because we don’t wan to be brittle)?

This is especially painful for us because we are currently seeing very slow download rates from hosted Chef (about 60kbps), so downloading addt’l cookbooks makes things noticeably slower.

(Another great example of dependency hell is installing the ‘nagios’ cookbook, which requires ‘apache’, which requires ‘php’, which in turn requires ‘mysql’ - yet it’s not really true that all of these things are intrinsically tied together - we are losing the things we’ve learned from the Unix model and service-oriented architectures when we staple things to each other this way.)

Okay, so that got a little bit ranty, but that’s because I really like Chef and I hate to see a slim, sexy, powerful tool get bloated and brittle for no good reason.

Discuss!

Blake


#5

Noah, your suggestion for ‘suggests’ sounds like the right compromise to me.

Lamont, parallel downloads would certainly be nice. My DC is in CA right
now, and cookbook sync takes upwards of 6 minutes. To make test cycles
fast enough to be worthwhile I think I’m going to have to roll my own chef
servers (I can’t ask engineers to wait 7 minutes just to begin an initial
converge).

IIRC, the biggest reason I’ve ever used ‘depends’ in an incorrect way was
that chef-client errors if a cookbook is missing are pretty cryptic, so I
just (out of blind laziness) ended up depending all over the place. About
a year later (now), I’ve found that doing this is sub-optimal (and that’s
being as nice about it as I know how).

Blake

On Wed, Oct 2, 2013 at 5:32 PM, Noah Kantrowitz noah@coderanger.net wrote:

There has been a long running concept to turn #suggests into an optional
form of #depends, so it would have the same effect as depends if the
cookbook is found in terms of triggering a download and loading in the
right order, but wouldn’t cause a failure if that cookbook wasn’t found.
Maybe after I get this dialect stuff squared away I’ll write up that patch
:slight_smile:

–Noah

On Oct 2, 2013, at 5:26 PM, Bryan Taylor btaylor@rackspace.com wrote:

I agree. I’d like to see a notion of “conditional dependence”, so that
java depends on windows OR me setting an opt-out flag that says I won’t
ever windows. I sometimes go in and edit these dependencies out of the
metadata before I load it to my chef server. It happens a lot with yum or
apt as well.

From: Blake Irvin blake@wanelo.com
Date: Wednesday, October 2, 2013 6:17 PM
To: "chef-dev@lists.opscode.com Dev" chef-dev@lists.opscode.com
Subject: [chef-dev] dependency spaghetti

(Please redirect me if this is the wrong list)

We seem to be in a place as a Chef community where the ‘depends’ feature
of cookbook metadata is getting used a little bit too freely.

For example, the ‘java’ cookbook includes a ‘depends’ for the 'windows’
cookbook - this means that I’m forced to download the ‘windows’ cookbook to
all of my *nix machines. Messy at best.

Shouldn’t we be using ‘suggests’? Or better yet, shouldn’t we reserve
dependency for cases where a cookbook really must have another cookbook
available to it to compile (and, shouldn’t we avoid tightly-coupled
dependencies whenever possible because we don’t wan to be brittle)?

This is especially painful for us because we are currently seeing very
slow download rates from hosted Chef (about 60kbps), so downloading addt’l
cookbooks makes things noticeably slower.

(Another great example of dependency hell is installing the 'nagios’
cookbook, which requires ‘apache’, which requires ‘php’, which in turn
requires ‘mysql’ - yet it’s not really true that all of these things are
intrinsically tied together - we are losing the things we’ve learned from
the Unix model and service-oriented architectures when we staple things to
each other this way.)

Okay, so that got a little bit ranty, but that’s because I really like
Chef and I hate to see a slim, sexy, powerful tool get bloated and brittle
for no good reason.

Discuss!

Blake


#6

Hi,

On Thu, Oct 3, 2013 at 9:17 AM, Blake Irvin blake@wanelo.com wrote:

For example, the ‘java’ cookbook includes a ‘depends’ for the 'windows’
cookbook - this means that I’m forced to download the ‘windows’ cookbook to
all of my *nix machines. Messy at best.

It also tends to pollute the attribute namespace, possibly add all sorts of
code via libraries etc. We hit problems with a dependency monkey patching
things to broken-ness. Not fun.

Thus we tend strip out all non essential cookbooks from depends flags and
add suggests where there is an optional context specific dependency. The
context is usually dependent on the environment or features used in the
cookbook. i.e. The cookbook when run on ubuntu needs ‘apt’, while on
windows it needs ‘windows’. The cookbook needs the authbind cookbook if you
are using authbind or the runit cookbook if the init system is set to
runit. It then becomes the responsibility of the wrapper cookbook to depend
on the required dependencies rather than the library cookbook.

The “add a depends for a any potential dependency” philosophy harks back to
earlier patterns of chef usage before wrapper/library cookbook distinctions
started to being used when cookbooks tended to be fairly tightly wedded to
the initial context of use.


Cheers,

Peter Donald


#7

On Oct 2, 2013, at 6:57 PM, Peter Donald peter@realityforge.org wrote:

Hi,

On Thu, Oct 3, 2013 at 9:17 AM, Blake Irvin blake@wanelo.com wrote:
For example, the ‘java’ cookbook includes a ‘depends’ for the ‘windows’ cookbook - this means that I’m forced to download the ‘windows’ cookbook to all of my *nix machines. Messy at best.

It also tends to pollute the attribute namespace, possibly add all sorts of code via libraries etc. We hit problems with a dependency monkey patching things to broken-ness. Not fun.

Thus we tend strip out all non essential cookbooks from depends flags and add suggests where there is an optional context specific dependency. The context is usually dependent on the environment or features used in the cookbook. i.e. The cookbook when run on ubuntu needs ‘apt’, while on windows it needs ‘windows’. The cookbook needs the authbind cookbook if you are using authbind or the runit cookbook if the init system is set to runit. It then becomes the responsibility of the wrapper cookbook to depend on the required dependencies rather than the library cookbook.

The “add a depends for a any potential dependency” philosophy harks back to earlier patterns of chef usage before wrapper/library cookbook distinctions started to being used when cookbooks tended to be fairly tightly wedded to the initial context of use.

The problem is as we move the community to Berkshelf and tools like it (which is totally the right direction) you can’t just edit the upstream cookbooks since you never see them directly. Optional dependencies would help, but the user experience if you are missing an optional cookbook is definitely not great and we’ll need some improvements there. The real dream is some kind of declarative way to state when a dependency is needed, but so far I’ve not seen any proposed way to do that which doesn’t veer off into crazytown.

–Noah


#8

On Thu, Oct 3, 2013 at 12:05 PM, Noah Kantrowitz noah@coderanger.netwrote:

On Oct 2, 2013, at 6:57 PM, Peter Donald peter@realityforge.org wrote:

On Thu, Oct 3, 2013 at 9:17 AM, Blake Irvin blake@wanelo.com wrote:
For example, the ‘java’ cookbook includes a ‘depends’ for the 'windows’
cookbook - this means that I’m forced to download the ‘windows’ cookbook to
all of my *nix machines. Messy at best.

It also tends to pollute the attribute namespace, possibly add all sorts
of code via libraries etc. We hit problems with a dependency monkey
patching things to broken-ness. Not fun.

Thus we tend strip out all non essential cookbooks from depends flags
and add suggests where there is an optional context specific dependency.
The context is usually dependent on the environment or features used in the
cookbook. i.e. The cookbook when run on ubuntu needs ‘apt’, while on
windows it needs ‘windows’. The cookbook needs the authbind cookbook if you
are using authbind or the runit cookbook if the init system is set to
runit. It then becomes the responsibility of the wrapper cookbook to depend
on the required dependencies rather than the library cookbook.

The “add a depends for a any potential dependency” philosophy harks back
to earlier patterns of chef usage before wrapper/library cookbook
distinctions started to being used when cookbooks tended to be fairly
tightly wedded to the initial context of use.

The problem is as we move the community to Berkshelf and tools like it
(which is totally the right direction) you can’t just edit the upstream
cookbooks since you never see them directly. Optional dependencies would
help, but the user experience if you are missing an optional cookbook is
definitely not great and we’ll need some improvements there. The real dream
is some kind of declarative way to state when a dependency is needed, but
so far I’ve not seen any proposed way to do that which doesn’t veer off
into crazytown.

We do enforcement within the cookbook itself. Roughly it is the
psychological equivalent of

raise “Must include cookbook ‘mydep’ for reason Y” unless
run_context.cookbook_collection.values.any? {|c| c.name.to_s == ‘mydep’ }

wrapped in a handy little helper like

ensure_cookbook(‘mydep’)

We also go a step further and allow passing in of a recipe that is also
included at the same time ala

ensure_cookbook(‘mydep’,‘mydep::some_recipe’)

A little ugly but seems to be relatively useful and it puts the requirement
into the code when needed. No need to create a whole capability model for
cookbooks to determine when/where features are enabled.


Cheers,

Peter Donald


#9

On Thu, Oct 3, 2013 at 2:32 AM, Noah Kantrowitz noah@coderanger.net wrote:

There has been a long running concept to turn #suggests into an optional form of #depends, so it would have the same effect as depends if the cookbook is found in terms of triggering a download and loading in the right order, but wouldn’t cause a failure if that cookbook wasn’t found. Maybe after I get this dialect stuff squared away I’ll write up that patch :slight_smile:

I like this. It’s better than ‘suggests’ just being a no-op. An
alternative may be to enable case statements in metadata.rb. The
individual depends could then properly be resolved based on platform
or recipe usage. The client can then filter based on the case in the
cookbook’s metadata, and only download those cookbooks it really
needs.

This would be a further step towards making metadata actually useful.
It remains to be seen which is more easily implemented/accepted.

  • Dimitri

#10

]] Noah Kantrowitz

There has been a long running concept to turn #suggests into an
optional form of #depends, so it would have the same effect as depends
if the cookbook is found in terms of triggering a download and loading
in the right order, but wouldn’t cause a failure if that cookbook
wasn’t found. Maybe after I get this dialect stuff squared away I’ll
write up that patch :slight_smile:

As a minor nit, it’d be nice to use recommends for this, rather than
suggests, to be more in line with what the terms mean in Debian
packaging. (A Suggests is a weaker Recommends which is a weaker
Depends.) The exact semantics won’t line up, but having the strength of
them line up would be less confusing.


Tollef Fog Heen
UNIX is user friendly, it’s just picky about who its friends are


#11

On Oct 2, 2013, at 11:04 PM, Dimitri Aivaliotis aglarond@gmail.com wrote:

On Thu, Oct 3, 2013 at 2:32 AM, Noah Kantrowitz noah@coderanger.net wrote:

There has been a long running concept to turn #suggests into an optional form of #depends, so it would have the same effect as depends if the cookbook is found in terms of triggering a download and loading in the right order, but wouldn’t cause a failure if that cookbook wasn’t found. Maybe after I get this dialect stuff squared away I’ll write up that patch :slight_smile:

I like this. It’s better than ‘suggests’ just being a no-op. An
alternative may be to enable case statements in metadata.rb. The
individual depends could then properly be resolved based on platform
or recipe usage. The client can then filter based on the case in the
cookbook’s metadata, and only download those cookbooks it really
needs.

The cookbook metadata isn’t actually code, the Ruby format is just convenience so you don’t have to write the (rather long and complex) JSON yourself. More specifically the cookbook dependency tree is resolved on the server, not the client, so whatever the solution is it has to be fully declarative.

–Noah


#12

On Thu, Oct 3, 2013 at 8:48 AM, Noah Kantrowitz noah@coderanger.net wrote:

The cookbook metadata isn’t actually code, the Ruby format is just convenience so you don’t have to write the (rather long and complex) JSON yourself. More specifically the cookbook dependency tree is resolved on the server, not the client, so whatever the solution is it has to be fully declarative.

I’m aware of that, but I find that a weakness in the current
implementation. The resolved dependency tree can be further filtered
on the client to eliminate cookbooks not supported on the running
platform.

Why is there a “platforms” object if it’s not currently used to
resolve dependencies? It doesn’t (any longer) restrict a specific
cookbook from being installed on a particular platform, e.g. the
windows cookbook “supports” only windows, but can be (and must be, due
to the dependencies the OP mentioned) installed on any platform. Is
this another appendix in Chef, in the sense that it originally had a
purpose, but now seems to be unused?

While we’re talking of dialects and alternative formats anyways, how
much more of a stretch would it be to extend the Ruby format to
generate an alternative JSON serialization, one in which the
"dependencies" object is nested under “platforms”? I realize that
this may actually have a greater impact than taking the two objects on
the client and filtering out the cookbooks not supported on that
platform, which may be the easiest near-term solution.

  • Dimitri

#13

Why can’t we just switch to client-side dependency solving instead? (ala
berks)
-s

On Thu, Oct 3, 2013 at 2:48 AM, Noah Kantrowitz noah@coderanger.net wrote:

On Oct 2, 2013, at 11:04 PM, Dimitri Aivaliotis aglarond@gmail.com
wrote:

On Thu, Oct 3, 2013 at 2:32 AM, Noah Kantrowitz noah@coderanger.net
wrote:

There has been a long running concept to turn #suggests into an
optional form of #depends, so it would have the same effect as depends if
the cookbook is found in terms of triggering a download and loading in the
right order, but wouldn’t cause a failure if that cookbook wasn’t found.
Maybe after I get this dialect stuff squared away I’ll write up that patch
:slight_smile:

I like this. It’s better than ‘suggests’ just being a no-op. An
alternative may be to enable case statements in metadata.rb. The
individual depends could then properly be resolved based on platform
or recipe usage. The client can then filter based on the case in the
cookbook’s metadata, and only download those cookbooks it really
needs.

The cookbook metadata isn’t actually code, the Ruby format is just
convenience so you don’t have to write the (rather long and complex) JSON
yourself. More specifically the cookbook dependency tree is resolved on the
server, not the client, so whatever the solution is it has to be fully
declarative.

–Noah


#14

If you factor in that depsolver feeds back into dependency resolution it would be a lot to move on to the client. Also re: per platform dependencies, that’s only one use case out of many, another big one is service frameworks like the runit cookbook only being used if initstyle is set to runit.

–Noah

Sean OMeara someara@gmail.com wrote:

Why can’t we just switch to client-side dependency solving instead?
(ala
berks)
-s

On Thu, Oct 3, 2013 at 2:48 AM, Noah Kantrowitz noah@coderanger.net
wrote:

On Oct 2, 2013, at 11:04 PM, Dimitri Aivaliotis aglarond@gmail.com
wrote:

On Thu, Oct 3, 2013 at 2:32 AM, Noah Kantrowitz
noah@coderanger.net
wrote:

There has been a long running concept to turn #suggests into an
optional form of #depends, so it would have the same effect as
depends if
the cookbook is found in terms of triggering a download and loading
in the
right order, but wouldn’t cause a failure if that cookbook wasn’t
found.
Maybe after I get this dialect stuff squared away I’ll write up that
patch
:slight_smile:

I like this. It’s better than ‘suggests’ just being a no-op. An
alternative may be to enable case statements in metadata.rb. The
individual depends could then properly be resolved based on
platform

or recipe usage. The client can then filter based on the case in
the

cookbook’s metadata, and only download those cookbooks it really
needs.

The cookbook metadata isn’t actually code, the Ruby format is just
convenience so you don’t have to write the (rather long and complex)
JSON
yourself. More specifically the cookbook dependency tree is resolved
on the
server, not the client, so whatever the solution is it has to be
fully
declarative.

–Noah


#15

i had suggested chef-berks already. Above everything this reduces the load
on chef server. and we get rid of storing community cookbooks in our chef
servers…

On Thu, Oct 3, 2013 at 1:09 AM, Sean OMeara someara@gmail.com wrote:

Why can’t we just switch to client-side dependency solving instead? (ala
berks)
-s

On Thu, Oct 3, 2013 at 2:48 AM, Noah Kantrowitz noah@coderanger.netwrote:

On Oct 2, 2013, at 11:04 PM, Dimitri Aivaliotis aglarond@gmail.com
wrote:

On Thu, Oct 3, 2013 at 2:32 AM, Noah Kantrowitz noah@coderanger.net
wrote:

There has been a long running concept to turn #suggests into an
optional form of #depends, so it would have the same effect as depends if
the cookbook is found in terms of triggering a download and loading in the
right order, but wouldn’t cause a failure if that cookbook wasn’t found.
Maybe after I get this dialect stuff squared away I’ll write up that patch
:slight_smile:

I like this. It’s better than ‘suggests’ just being a no-op. An
alternative may be to enable case statements in metadata.rb. The
individual depends could then properly be resolved based on platform
or recipe usage. The client can then filter based on the case in the
cookbook’s metadata, and only download those cookbooks it really
needs.

The cookbook metadata isn’t actually code, the Ruby format is just
convenience so you don’t have to write the (rather long and complex) JSON
yourself. More specifically the cookbook dependency tree is resolved on the
server, not the client, so whatever the solution is it has to be fully
declarative.

–Noah


#16

aglarond@gmail.com writes:

I like this. It’s better than ‘suggests’ just being a no-op. An
alternative may be to enable case statements in metadata.rb. The
individual depends could then properly be resolved based on platform
or recipe usage. The client can then filter based on the case in the
cookbook’s metadata, and only download those cookbooks it really
needs.

This would be a further step towards making metadata actually useful.
It remains to be seen which is more easily implemented/accepted.

One small note on making any of the cookbook version metadata fields
"active" rather than no-ops. Presently, there are many cookbooks that
have values in these fields. If the server or client starts to use these
fields, we’ll have many cookbooks in the wild with invalid values. So as
an implementation detail, we may want to consider a new name.

  • seth


Seth Falcon | Development Lead | Opscode | @sfalcon


#17

Shouldn’t berkshelf make it easier to fork a community cookbook and
modify the depends, and then merge upstream periodically when you need
to and point your berkshelf at your local copy? It certainly works
better than the previous knife cookbook site stuff, and merge conflicts
over the depends should be pretty easy to deal with…

On 10/2/13 7:05 PM, Noah Kantrowitz wrote:

The problem is as we move the community to Berkshelf and tools like it (which is totally the right direction) you can’t just edit the upstream cookbooks since you never see them directly. Optional dependencies would help, but the user experience if you are missing an optional cookbook is definitely not great and we’ll need some improvements there. The real dream is some kind of declarative way to state when a dependency is needed, but so far I’ve not seen any proposed way to do that which doesn’t veer off into crazytown.

–Noah


#18

No, because the whole idea of Berkshelf is to treat cookbooks like any other package manager. At best you could have a feature in Berkshelf to apply a patch blob from the Berksfile kind of like how Homebrew or Portage work, but down this path lies madness.

–Noah

On Oct 3, 2013, at 10:53 AM, Lamont Granquist lamont@opscode.com wrote:

Shouldn’t berkshelf make it easier to fork a community cookbook and modify the depends, and then merge upstream periodically when you need to and point your berkshelf at your local copy? It certainly works better than the previous knife cookbook site stuff, and merge conflicts over the depends should be pretty easy to deal with…

On 10/2/13 7:05 PM, Noah Kantrowitz wrote:

The problem is as we move the community to Berkshelf and tools like it (which is totally the right direction) you can’t just edit the upstream cookbooks since you never see them directly. Optional dependencies would help, but the user experience if you are missing an optional cookbook is definitely not great and we’ll need some improvements there. The real dream is some kind of declarative way to state when a dependency is needed, but so far I’ve not seen any proposed way to do that which doesn’t veer off into crazytown.

–Noah


#19

On Thu, Oct 3, 2013 at 1:09 AM, Sean OMeara someara@gmail.com wrote:

Why can’t we just switch to client-side dependency solving instead? (ala
berks)

Yes please.

-s

On Thu, Oct 3, 2013 at 2:48 AM, Noah Kantrowitz noah@coderanger.netwrote:

On Oct 2, 2013, at 11:04 PM, Dimitri Aivaliotis aglarond@gmail.com
wrote:

On Thu, Oct 3, 2013 at 2:32 AM, Noah Kantrowitz noah@coderanger.net
wrote:

There has been a long running concept to turn #suggests into an
optional form of #depends, so it would have the same effect as depends if
the cookbook is found in terms of triggering a download and loading in the
right order, but wouldn’t cause a failure if that cookbook wasn’t found.
Maybe after I get this dialect stuff squared away I’ll write up that patch
:slight_smile:

I like this. It’s better than ‘suggests’ just being a no-op. An
alternative may be to enable case statements in metadata.rb. The
individual depends could then properly be resolved based on platform
or recipe usage. The client can then filter based on the case in the
cookbook’s metadata, and only download those cookbooks it really
needs.

The cookbook metadata isn’t actually code, the Ruby format is just
convenience so you don’t have to write the (rather long and complex) JSON
yourself. More specifically the cookbook dependency tree is resolved on the
server, not the client, so whatever the solution is it has to be fully
declarative.

–Noah


#20

On Thursday, October 3, 2013 at 11:10 AM, Blake Irvin wrote:

On Thu, Oct 3, 2013 at 1:09 AM, Sean OMeara <someara@gmail.com (mailto:someara@gmail.com)> wrote:

Why can’t we just switch to client-side dependency solving instead? (ala berks)

Yes please.
I think this will happen on a Chef 12-ish timescale. There’s a lot of work required, and it will have pretty big impacts on workflow, which I think will be for the best.


Daniel DeLeo