Excessive dependency chaining?


#1

I’m only using a handful of cookbooks directly, but my berkshelf list is
large. Pulling in Apache2 and Nginx adds in a lot of dependencies, one of
which is perl, and one of the Java cookbooks ends up pulling in Gnome and
X-windows. One of the chef cookbooks (chef-client?) pulls in Windows. These
are all headless Linux instances without X installed (and I’d like to keep
it that way, heh).

When the run list is compiled and the cookbooks are sent to the client,
they end up transferring a lot of cookbooks that never end up actually
getting used. And if I touch any of the higher level cookbooks (like the
one for my org that configures some things via LWRPs and overrides some
cookbook resources), berkshelf then has to parse the list of all those
dependencies as well just to upload my local cookbook. I even tried using a
group to filter things down but dependencies are resolved even when using
groups. Berks also seems to re-upload every cookbook even if the version
number hasn’t changed, which is odd to me. (Fixing that would minimize at
least some of the annoyance factor of how long “berks upload” takes.)

Is this just a situation where “deal with it” is the most practical advice,
or is there a way to break some of these dependencies that I know aren’t
going to be reached in my use case (like the gnome and windows
cookbooks…)?

~~ StormeRider ~~

“Every world needs its heroes […] They inspire us to be better than we
are. And they protect from the darkness that’s just around the corner.”

(from Smallville Season 6x1: “Zod”)

On why I hate the phrase “that’s so lame”… http://bit.ly/Ps3uSS


#2

there are no clear solution right now. this issue have been discussed
previously , specifically in context of platform specific cookbooks.
Currently chef does not have support for conditional cookbook dependency.
So even if a parent cookbook conditionally uses a dependent cookbook , it
has to declare this in its metadata. iirc it was suggested in certain cases
the cookbook author can skip enlisting those conditional cookbooks as
dependency in metadata and it was also assumed that the user will have an
understanding of this, and will assign those conditional cookbooks
explicitly in the node’s run list when they need them. but i dont think we
can generalized these approach for all cookbooks.

On Wed, Jun 5, 2013 at 9:49 AM, Morgan Blackthorne stormerider@gmail.comwrote:

I’m only using a handful of cookbooks directly, but my berkshelf list is
large. Pulling in Apache2 and Nginx adds in a lot of dependencies, one of
which is perl, and one of the Java cookbooks ends up pulling in Gnome and
X-windows. One of the chef cookbooks (chef-client?) pulls in Windows. These
are all headless Linux instances without X installed (and I’d like to keep
it that way, heh).

When the run list is compiled and the cookbooks are sent to the client,
they end up transferring a lot of cookbooks that never end up actually
getting used. And if I touch any of the higher level cookbooks (like the
one for my org that configures some things via LWRPs and overrides some
cookbook resources), berkshelf then has to parse the list of all those
dependencies as well just to upload my local cookbook. I even tried using a
group to filter things down but dependencies are resolved even when using
groups. Berks also seems to re-upload every cookbook even if the version
number hasn’t changed, which is odd to me. (Fixing that would minimize at
least some of the annoyance factor of how long “berks upload” takes.)

Is this just a situation where “deal with it” is the most practical
advice, or is there a way to break some of these dependencies that I know
aren’t going to be reached in my use case (like the gnome and windows
cookbooks…)?

~~ StormeRider ~~

“Every world needs its heroes […] They inspire us to be better than we
are. And they protect from the darkness that’s just around the corner.”

(from Smallville Season 6x1: “Zod”)

On why I hate the phrase “that’s so lame”… http://bit.ly/Ps3uSS


#3

I’m assuming that the cookbook metadata can’t be overridden like resources
can?

On Wednesday, June 5, 2013, Ranjib Dey wrote:

there are no clear solution right now. this issue have been discussed
previously , specifically in context of platform specific cookbooks.
Currently chef does not have support for conditional cookbook dependency.
So even if a parent cookbook conditionally uses a dependent cookbook , it
has to declare this in its metadata. iirc it was suggested in certain cases
the cookbook author can skip enlisting those conditional cookbooks as
dependency in metadata and it was also assumed that the user will have an
understanding of this, and will assign those conditional cookbooks
explicitly in the node’s run list when they need them. but i dont think we
can generalized these approach for all cookbooks.

On Wed, Jun 5, 2013 at 9:49 AM, Morgan Blackthorne <stormerider@gmail.com<javascript:_e({}, ‘cvml’, ‘stormerider@gmail.com’);>

wrote:

I’m only using a handful of cookbooks directly, but my berkshelf list is
large. Pulling in Apache2 and Nginx adds in a lot of dependencies, one of
which is perl, and one of the Java cookbooks ends up pulling in Gnome and
X-windows. One of the chef cookbooks (chef-client?) pulls in Windows. These
are all headless Linux instances without X installed (and I’d like to keep
it that way, heh).

When the run list is compiled and the cookbooks are sent to the client,
they end up transferring a lot of cookbooks that never end up actually
getting used. And if I touch any of the higher level cookbooks (like the
one for my org that configures some things via LWRPs and overrides some
cookbook resources), berkshelf then has to parse the list of all those
dependencies as well just to upload my local cookbook. I even tried using a
group to filter things down but dependencies are resolved even when using
groups. Berks also seems to re-upload every cookbook even if the version
number hasn’t changed, which is odd to me. (Fixing that would minimize at
least some of the annoyance factor of how long “berks upload” takes.)

Is this just a situation where “deal with it” is the most practical
advice, or is there a way to break some of these dependencies that I know
aren’t going to be reached in my use case (like the gnome and windows
cookbooks…)?

~~ StormeRider ~~

“Every world needs its heroes […] They inspire us to be better than we
are. And they protect from the darkness that’s just around the corner.”

(from Smallville Season 6x1: “Zod”)

On why I hate the phrase “that’s so lame”… http://bit.ly/Ps3uSS

~~ StormeRider ~~

“Every world needs its heroes […] They inspire us to be better than we
are. And they protect from the darkness that’s just around the corner.”

(from Smallville Season 6x1: “Zod”)

On why I hate the phrase “that’s so lame”… http://bit.ly/Ps3uSS


#4

no. but its ruby so there must be a way to hack it (which will involve
tampering core chef’s classes/modules) to do what you want

On Wed, Jun 5, 2013 at 10:11 AM, Morgan Blackthorne
stormerider@gmail.comwrote:

I’m assuming that the cookbook metadata can’t be overridden like resources
can?

On Wednesday, June 5, 2013, Ranjib Dey wrote:

there are no clear solution right now. this issue have been discussed
previously , specifically in context of platform specific cookbooks.
Currently chef does not have support for conditional cookbook dependency.
So even if a parent cookbook conditionally uses a dependent cookbook , it
has to declare this in its metadata. iirc it was suggested in certain cases
the cookbook author can skip enlisting those conditional cookbooks as
dependency in metadata and it was also assumed that the user will have an
understanding of this, and will assign those conditional cookbooks
explicitly in the node’s run list when they need them. but i dont think we
can generalized these approach for all cookbooks.

On Wed, Jun 5, 2013 at 9:49 AM, Morgan Blackthorne <stormerider@gmail.com

wrote:

I’m only using a handful of cookbooks directly, but my berkshelf list is
large. Pulling in Apache2 and Nginx adds in a lot of dependencies, one of
which is perl, and one of the Java cookbooks ends up pulling in Gnome and
X-windows. One of the chef cookbooks (chef-client?) pulls in Windows. These
are all headless Linux instances without X installed (and I’d like to keep
it that way, heh).

When the run list is compiled and the cookbooks are sent to the client,
they end up transferring a lot of cookbooks that never end up actually
getting used. And if I touch any of the higher level cookbooks (like the
one for my org that configures some things via LWRPs and overrides some
cookbook resources), berkshelf then has to parse the list of all those
dependencies as well just to upload my local cookbook. I even tried using a
group to filter things down but dependencies are resolved even when using
groups. Berks also seems to re-upload every cookbook even if the version
number hasn’t changed, which is odd to me. (Fixing that would minimize at
least some of the annoyance factor of how long “berks upload” takes.)

Is this just a situation where “deal with it” is the most practical
advice, or is there a way to break some of these dependencies that I know
aren’t going to be reached in my use case (like the gnome and windows
cookbooks…)?

~~ StormeRider ~~

“Every world needs its heroes […] They inspire us to be better than we
are. And they protect from the darkness that’s just around the corner.”

(from Smallville Season 6x1: “Zod”)

On why I hate the phrase “that’s so lame”… http://bit.ly/Ps3uSS

~~ StormeRider ~~

“Every world needs its heroes […] They inspire us to be better than we
are. And they protect from the darkness that’s just around the corner.”

(from Smallville Season 6x1: “Zod”)

On why I hate the phrase “that’s so lame”… http://bit.ly/Ps3uSS


#5

Hacking internals isn’t what I want. :slight_smile:

Any ideas on the Berkshelf reuploading without a version change? Is there a
config setting to alter that behavior, perhaps?

On Wednesday, June 5, 2013, Ranjib Dey wrote:

no. but its ruby so there must be a way to hack it (which will involve
tampering core chef’s classes/modules) to do what you want

On Wed, Jun 5, 2013 at 10:11 AM, Morgan Blackthorne <stormerider@gmail.com<javascript:_e({}, ‘cvml’, ‘stormerider@gmail.com’);>

wrote:

I’m assuming that the cookbook metadata can’t be overridden like
resources can?

On Wednesday, June 5, 2013, Ranjib Dey wrote:

there are no clear solution right now. this issue have been discussed
previously , specifically in context of platform specific cookbooks.
Currently chef does not have support for conditional cookbook dependency.
So even if a parent cookbook conditionally uses a dependent cookbook , it
has to declare this in its metadata. iirc it was suggested in certain cases
the cookbook author can skip enlisting those conditional cookbooks as
dependency in metadata and it was also assumed that the user will have an
understanding of this, and will assign those conditional cookbooks
explicitly in the node’s run list when they need them. but i dont think we
can generalized these approach for all cookbooks.

On Wed, Jun 5, 2013 at 9:49 AM, Morgan Blackthorne <
stormerider@gmail.com> wrote:

I’m only using a handful of cookbooks directly, but my berkshelf list
is large. Pulling in Apache2 and Nginx adds in a lot of dependencies, one
of which is perl, and one of the Java cookbooks ends up pulling in Gnome
and X-windows. One of the chef cookbooks (chef-client?) pulls in
Windows. These are all headless Linux instances without X installed (and
I’d like to keep it that way, heh).

When the run list is compiled and the cookbooks are sent to the client,
they end up transferring a lot of cookbooks that never end up actually
getting used. And if I touch any of the higher level cookbooks (like the
one for my org that configures some things via LWRPs and overrides some
cookbook resources), berkshelf then has to parse the list of all those
dependencies as well just to upload my local cookbook. I even tried using a
group to filter things down but dependencies are resolved even when using
groups. Berks also seems to re-upload every cookbook even if the version
number hasn’t changed, which is odd to me. (Fixing that would minimize at
least some of the annoyance factor of how long “berks upload” takes.)

Is this just a situation where “deal with it” is the most practical
advice, or is there a way to break some of these dependencies that I know
aren’t going to be reached in my use case (like the gnome and windows
cookbooks…)?

~~ StormeRider ~~

“Every world needs its heroes […] They inspire us to be better than
we are. And they protect from the darkness that’s just around the corner.”

(from Smallville Season 6x1: “Zod”)

On why I hate the phrase “that’s so lame”… http://bit.ly/Ps3uSS

~~ StormeRider ~~

“Every world needs its heroes […] They inspire us to be better than we
are. And they protect from the darkness that’s just around the corner.”

(from Smallville Season 6x1: “Zod”)

On why I hate the phrase “that’s so lame”… http://bit.ly/Ps3uSS

~~ StormeRider ~~

“Every world needs its heroes […] They inspire us to be better than we
are. And they protect from the darkness that’s just around the corner.”

(from Smallville Season 6x1: “Zod”)

On why I hate the phrase “that’s so lame”… http://bit.ly/Ps3uSS


#6

Berkself has a nofreeze option iirc it does not bump the version…are u
using knife-spork?
On Jun 5, 2013 10:26 AM, “Morgan Blackthorne” stormerider@gmail.com wrote:

Hacking internals isn’t what I want. :slight_smile:

Any ideas on the Berkshelf reuploading without a version change? Is there
a config setting to alter that behavior, perhaps?

On Wednesday, June 5, 2013, Ranjib Dey wrote:

no. but its ruby so there must be a way to hack it (which will involve
tampering core chef’s classes/modules) to do what you want

On Wed, Jun 5, 2013 at 10:11 AM, Morgan Blackthorne <
stormerider@gmail.com> wrote:

I’m assuming that the cookbook metadata can’t be overridden like
resources can?

On Wednesday, June 5, 2013, Ranjib Dey wrote:

there are no clear solution right now. this issue have been discussed
previously , specifically in context of platform specific cookbooks.
Currently chef does not have support for conditional cookbook dependency.
So even if a parent cookbook conditionally uses a dependent cookbook , it
has to declare this in its metadata. iirc it was suggested in certain cases
the cookbook author can skip enlisting those conditional cookbooks as
dependency in metadata and it was also assumed that the user will have an
understanding of this, and will assign those conditional cookbooks
explicitly in the node’s run list when they need them. but i dont think we
can generalized these approach for all cookbooks.

On Wed, Jun 5, 2013 at 9:49 AM, Morgan Blackthorne <
stormerider@gmail.com> wrote:

I’m only using a handful of cookbooks directly, but my berkshelf list
is large. Pulling in Apache2 and Nginx adds in a lot of dependencies, one
of which is perl, and one of the Java cookbooks ends up pulling in Gnome
and X-windows. One of the chef cookbooks (chef-client?) pulls in
Windows. These are all headless Linux instances without X installed (and
I’d like to keep it that way, heh).

When the run list is compiled and the cookbooks are sent to the
client, they end up transferring a lot of cookbooks that never end up
actually getting used. And if I touch any of the higher level cookbooks
(like the one for my org that configures some things via LWRPs and
overrides some cookbook resources), berkshelf then has to parse the list of
all those dependencies as well just to upload my local cookbook. I even
tried using a group to filter things down but dependencies are resolved
even when using groups. Berks also seems to re-upload every cookbook even
if the version number hasn’t changed, which is odd to me. (Fixing that
would minimize at least some of the annoyance factor of how long “berks
upload” takes.)

Is this just a situation where “deal with it” is the most practical
advice, or is there a way to break some of these dependencies that I know
aren’t going to be reached in my use case (like the gnome and windows
cookbooks…)?

~~ StormeRider ~~

“Every world needs its heroes […] They inspire us to be better than
we are. And they protect from the darkness that’s just around the corner.”

(from Smallville Season 6x1: “Zod”)

On why I hate the phrase “that’s so lame”… http://bit.ly/Ps3uSS

~~ StormeRider ~~

“Every world needs its heroes […] They inspire us to be better than we
are. And they protect from the darkness that’s just around the corner.”

(from Smallville Season 6x1: “Zod”)

On why I hate the phrase “that’s so lame”… http://bit.ly/Ps3uSS

~~ StormeRider ~~

“Every world needs its heroes […] They inspire us to be better than we
are. And they protect from the darkness that’s just around the corner.”

(from Smallville Season 6x1: “Zod”)

On why I hate the phrase “that’s so lame”… http://bit.ly/Ps3uSS


#7

I have it installed but I bump my personal cookbook manually… Haven’t
gotten sport into my workflow yet.

To clarify, I think what I want (at least for cookbooks in the “site
:opscode” group) is for them to only be reuploaded to the Chef server if
the Chef server does not have that version already uploaded, instead of
having “berks upload” push all ~40 cookbooks each time when only 1 has
changed. Pushing all the opscode cookbooks each run is just wasted cycles.

On Wednesday, June 5, 2013, Ranjib Dey wrote:

Berkself has a nofreeze option iirc it does not bump the version…are u
using knife-spork?
On Jun 5, 2013 10:26 AM, “Morgan Blackthorne” stormerider@gmail.com
wrote:

Hacking internals isn’t what I want. :slight_smile:

Any ideas on the Berkshelf reuploading without a version change? Is there
a config setting to alter that behavior, perhaps?

On Wednesday, June 5, 2013, Ranjib Dey wrote:

no. but its ruby so there must be a way to hack it (which will involve
tampering core chef’s classes/modules) to do what you want

On Wed, Jun 5, 2013 at 10:11 AM, Morgan Blackthorne <stormerider@gmail.com

wrote:

I’m assuming that the cookbook metadata can’t be overridden like resources
can?

On Wednesday, June 5, 2013, Ranjib Dey wrote:

there are no clear solution right now. this issue have been discussed
previously , specifically in context of platform specific cookbooks.
Currently chef does not have support for conditional cookbook dependency.
So even if a parent cookbook conditionally uses a dependent cookbook , it
has to declare this in its metadata. iirc it was suggested in certain cases
the cookbook author can skip enlisting those conditional cookbooks as
dependency in metadata and it was also assumed that the user will have an
understanding of this, and will assign those conditional cookbooks
explicitly in the node’s run list when they need them. but i dont think we
can generalized these approach for all cookbooks.

On Wed, Jun 5, 2013 at 9:49 AM, Morgan Blackthorne stormerider@gmail.comwrote:

I’m only using a handful of cookbooks directly, but my berkshelf list is
large. Pulling in Apache2 and Nginx adds in a lot of dependencies, one of
which is perl, and one of the Java cookbooks ends up pulling in Gnome and
X-windows. One of the chef cookbooks (chef-client?) pulls in Windows. These
are all headless Linux instances without X installed (and I’d like to keep
it that way, heh).

When the run list is compiled and the cookbooks are sent to the client,
they end up transferring a lot of cookbooks that never end up actually
getting used. And if I touch any of the higher level cookbooks (like the
one for my org that configures some things via LWRPs and overrides some
cookbook resources), berkshelf then has to parse the list of all those
dependencies as well just to upload my local cookbook. I even tried using a
group to filter things down but dependencies are resolved even when using
groups. Berks also seems to re-upload every cookbook even if the version
number hasn’t changed, which is odd to me. (Fixing that would minimize at
least some of the annoyance factor of how long “berks upload” takes.)

Is this just a situation where “deal with it” is the most practical
advice, or is there a way to break some of these dependencies that I know
aren’t going to be reached in my use case (like the gnome and windows
cookbooks…)?

~~ StormeRider ~~

“Every world needs its heroes […] They inspire us to be better than we
are. And they protect from the darkness that’s just around the corner.”

(from Smallville Season 6x1: “Zod”)

On why I hate the phrase “that’s so lame”… http://bit.ly/Ps3uSS

~~ StormeRider ~~

“Every world needs its heroes […] They inspire us to be better than we
are. And they protect from the darkness that’s just around the corner.”

(from Smallville Season 6x1: “Zod”)

On why I hate the phrase “that’s so lame”… http://bit.ly/Ps3uSS

~~ StormeRider ~~

“Every world needs its heroes […] They inspire us to be better than we
are. And they protect from the darkness that’s just around the corner.”

(from Smallville Season 6x1: “Zod”)

On why I hate the phrase “that’s so lame”… http://bit.ly/Ps3uSS


#8

i dont know if you can do that.
the problem is chef server does not mandate version freeze for cookbooks.
You can upload the different content with same version, so for an external
tool like berkshelf its impossible to understand if the cookbook has
changed (since version is not the absolute indicator) , unless it does a
full download checksum based comparison (this is how chef-client works). So
its risky for a tool like berkshelf to do this, but definitely its a good
option to have (in that case you might also like to specify only a subset
of cookbooks for which you want to do this).

On Wed, Jun 5, 2013 at 11:30 AM, Morgan Blackthorne
stormerider@gmail.comwrote:

I have it installed but I bump my personal cookbook manually… Haven’t
gotten sport into my workflow yet.

To clarify, I think what I want (at least for cookbooks in the “site
:opscode” group) is for them to only be reuploaded to the Chef server if
the Chef server does not have that version already uploaded, instead of
having “berks upload” push all ~40 cookbooks each time when only 1 has
changed. Pushing all the opscode cookbooks each run is just wasted cycles.

On Wednesday, June 5, 2013, Ranjib Dey wrote:

Berkself has a nofreeze option iirc it does not bump the version…are u
using knife-spork?
On Jun 5, 2013 10:26 AM, “Morgan Blackthorne” stormerider@gmail.com
wrote:

Hacking internals isn’t what I want. :slight_smile:

Any ideas on the Berkshelf reuploading without a version change? Is there
a config setting to alter that behavior, perhaps?

On Wednesday, June 5, 2013, Ranjib Dey wrote:

no. but its ruby so there must be a way to hack it (which will involve
tampering core chef’s classes/modules) to do what you want

On Wed, Jun 5, 2013 at 10:11 AM, Morgan Blackthorne <
stormerider@gmail.com> wrote:

I’m assuming that the cookbook metadata can’t be overridden like
resources can?

On Wednesday, June 5, 2013, Ranjib Dey wrote:

there are no clear solution right now. this issue have been discussed
previously , specifically in context of platform specific cookbooks.
Currently chef does not have support for conditional cookbook dependency.
So even if a parent cookbook conditionally uses a dependent cookbook , it
has to declare this in its metadata. iirc it was suggested in certain cases
the cookbook author can skip enlisting those conditional cookbooks as
dependency in metadata and it was also assumed that the user will have an
understanding of this, and will assign those conditional cookbooks
explicitly in the node’s run list when they need them. but i dont think we
can generalized these approach for all cookbooks.

On Wed, Jun 5, 2013 at 9:49 AM, Morgan Blackthorne <stormerider@gmail.com

wrote:

I’m only using a handful of cookbooks directly, but my berkshelf list is
large. Pulling in Apache2 and Nginx adds in a lot of dependencies, one of
which is perl, and one of the Java cookbooks ends up pulling in Gnome and
X-windows. One of the chef cookbooks (chef-client?) pulls in Windows. These
are all headless Linux instances without X installed (and I’d like to keep
it that way, heh).

When the run list is compiled and the cookbooks are sent to the client,
they end up transferring a lot of cookbooks that never end up actually
getting used. And if I touch any of the higher level cookbooks (like the
one for my org that configures some things via LWRPs and overrides some
cookbook resources), berkshelf then has to parse the list of all those
dependencies as well just to upload my local cookbook. I even tried using a
group to filter things down but dependencies are resolved even when using
groups. Berks also seems to re-upload every cookbook even if the version
number hasn’t changed, which is odd to me. (Fixing that would minimize at
least some of the annoyance factor of how long “berks upload” takes.)

Is this just a situation where “deal with it” is the most practical
advice, or is there a way to break some of these dependencies that I know
aren’t going to be reached in my use case (like the gnome and windows
cookbooks…)?

~~ StormeRider ~~

“Every world needs its heroes […] They inspire us to be better than we
are. And they protect from the darkness that’s just around the corner.”

(from Smallville Season 6x1: “Zod”)

On why I hate the phrase “that’s so lame”… http://bit.ly/Ps3uSS

~~ StormeRider ~~

“Every world needs its heroes […] They inspire us to be better than we
are. And they protect from the darkness that’s just around the corner.”

(from Smallville Season 6x1: “Zod”)

On why I hate the phrase “that’s so lame”… http://bit.ly/Ps3uSS

~~ StormeRider ~~

“Every world needs its heroes […] They inspire us to be better than we
are. And they protect from the darkness that’s just around the corner.”

(from Smallville Season 6x1: “Zod”)

On why I hate the phrase “that’s so lame”… http://bit.ly/Ps3uSS


#9

Nod. I might try playing with the code at some point once I get my MacBook
repaired (I am so glad that I made myself sit down and configure it in
BackupPC right about now…).

On Wednesday, June 5, 2013, Ranjib Dey wrote:

i dont know if you can do that.
the problem is chef server does not mandate version freeze for cookbooks.
You can upload the different content with same version, so for an external
tool like berkshelf its impossible to understand if the cookbook has
changed (since version is not the absolute indicator) , unless it does a
full download checksum based comparison (this is how chef-client works). So
its risky for a tool like berkshelf to do this, but definitely its a good
option to have (in that case you might also like to specify only a subset
of cookbooks for which you want to do this).

On Wed, Jun 5, 2013 at 11:30 AM, Morgan Blackthorne <stormerider@gmail.com

wrote:

I have it installed but I bump my personal cookbook manually… Haven’t
gotten sport into my workflow yet.

To clarify, I think what I want (at least for cookbooks in the “site
:opscode” group) is for them to only be reuploaded to the Chef server if
the Chef server does not have that version already uploaded, instead of
having “berks upload” push all ~40 cookbooks each time when only 1 has
changed. Pushing all the opscode cookbooks each run is just wasted cycles.

On Wednesday, June 5, 2013, Ranjib Dey wrote:

Berkself has a nofreeze option iirc it does not bump the version…are u
using knife-spork?
On Jun 5, 2013 10:26 AM, “Morgan Blackthorne” stormerider@gmail.com
wrote:

Hacking internals isn’t what I want. :slight_smile:

Any ideas on the Berkshelf reuploading without a version change? Is there
a config setting to alter that behavior, perhaps?

On Wednesday, June 5, 2013, Ranjib Dey wrote:

no. but its ruby so there must be a way to hack it (which will involve
tampering core chef’s classes/modules) to do what you want

On Wed, Jun 5, 2013 at 10:11 AM, Morgan Blackthorne <stormerider@gmail.com

wrote:

I’m assuming that the cookbook metadata can’t be overridden like resources
can?

On Wednesday, June 5, 2013, Ranjib Dey wrote:

there are no clear solution right now. this issue have been discussed
previously , specifically in context of platform specific cookbooks.
Currently chef does not have support for conditional cookbook dependency.
So even if a parent cookbook conditionally uses a dependent cookbook , it
has to declare this in its metadata. iirc it was suggested in certain cases
the cookbook author can skip enlisting those conditional cookbooks as
dependency in metadata and it was also assumed that the user will have an
understanding of this, and will assign those conditional cookbooks
explicitly in the node’s run list when they need them. but i dont think we
can generalized these approach for all cookbooks.

On Wed, Jun 5, 2013 at 9:49 AM, Morgan Blackthorne stormerider@gmail.comwrote:

I’m only using a handful of cookbooks directly, but my berkshelf list is
large. Pulling in Apache2 and Nginx adds in a lot of dependencies, one of
which is perl, and one of the Java cookbooks ends up pulling in Gnome and
X-windows. One of the chef cookbooks (chef-client?) pulls in Windows. These
are all headless Linux instances without X installed (and I’d like to keep
it that way, heh).

When the run list is compiled and the cookbooks are sent to the client,
they end up transferring a lot of cookbooks that never end up actually
getting used. And if I touch any of the higher level cookbooks (like the
one for my org that configures some things via LWRPs and overrides some
cookbook resources), berkshelf then has to parse the list of all those
dependencies as well just to upload my local cookbook. I even tried using a
group to filter things down but dependencies are resolved even when using
groups. Berks also seems to re-upload every cookbook even if the version
number hasn’t changed, which is odd to me. (Fixing that would minimize at
least some of the annoyance factor of how long “berks upload” takes.)

Is this just a situation where “deal with it” is the most practical
advice, or is there a way to break some of these dependencies that I know
aren’t going to be reached in my use case (like the gnome and windows
cookbooks…)?

~~ StormeRider ~~

“Every world needs its heroes […] They inspire us to be better than we
are. And they protect from the darkness that’s just around the corner.”

(from Smallville Sea

~~ StormeRider ~~

“Every world needs its heroes […] They inspire us to be better than we
are. And they protect from the darkness that’s just around the corner.”

(from Smallville Season 6x1: “Zod”)

On why I hate the phrase “that’s so lame”… http://bit.ly/Ps3uSS


#10

What version of Berkshelf are you using?

In the 1.4.x series the default behavior is to freeze a cookbook on upload and to skip over frozen dependencies on subsequent uploads.

Jamie Winsor
@resetexistence

On Jun 5, 2013, at 11:36 AM, Ranjib Dey dey.ranjib@gmail.com wrote:

i dont know if you can do that.
the problem is chef server does not mandate version freeze for cookbooks. You can upload the different content with same version, so for an external tool like berkshelf its impossible to understand if the cookbook has changed (since version is not the absolute indicator) , unless it does a full download checksum based comparison (this is how chef-client works). So its risky for a tool like berkshelf to do this, but definitely its a good option to have (in that case you might also like to specify only a subset of cookbooks for which you want to do this).

On Wed, Jun 5, 2013 at 11:30 AM, Morgan Blackthorne stormerider@gmail.com wrote:

I have it installed but I bump my personal cookbook manually… Haven’t gotten sport into my workflow yet.

To clarify, I think what I want (at least for cookbooks in the “site :opscode” group) is for them to only be reuploaded to the Chef server if the Chef server does not have that version already uploaded, instead of having “berks upload” push all ~40 cookbooks each time when only 1 has changed. Pushing all the opscode cookbooks each run is just wasted cycles.

On Wednesday, June 5, 2013, Ranjib Dey wrote:

Berkself has a nofreeze option iirc it does not bump the version…are u using knife-spork?

On Jun 5, 2013 10:26 AM, “Morgan Blackthorne” stormerider@gmail.com wrote:
Hacking internals isn’t what I want. :slight_smile:

Any ideas on the Berkshelf reuploading without a version change? Is there a config setting to alter that behavior, perhaps?

On Wednesday, June 5, 2013, Ranjib Dey wrote:
no. but its ruby so there must be a way to hack it (which will involve tampering core chef’s classes/modules) to do what you want

On Wed, Jun 5, 2013 at 10:11 AM, Morgan Blackthorne stormerider@gmail.com wrote:
I’m assuming that the cookbook metadata can’t be overridden like resources can?

On Wednesday, June 5, 2013, Ranjib Dey wrote:
there are no clear solution right now. this issue have been discussed previously , specifically in context of platform specific cookbooks. Currently chef does not have support for conditional cookbook dependency. So even if a parent cookbook conditionally uses a dependent cookbook , it has to declare this in its metadata. iirc it was suggested in certain cases the cookbook author can skip enlisting those conditional cookbooks as dependency in metadata and it was also assumed that the user will have an understanding of this, and will assign those conditional cookbooks explicitly in the node’s run list when they need them. but i dont think we can generalized these approach for all cookbooks.

On Wed, Jun 5, 2013 at 9:49 AM, Morgan Blackthorne stormerider@gmail.com wrote:
I’m only using a handful of cookbooks directly, but my berkshelf list is large. Pulling in Apache2 and Nginx adds in a lot of dependencies, one of which is perl, and one of the Java cookbooks ends up pulling in Gnome and X-windows. One of the chef cookbooks (chef-client?) pulls in Windows. These are all headless Linux instances without X installed (and I’d like to keep it that way, heh).

When the run list is compiled and the cookbooks are sent to the client, they end up transferring a lot of cookbooks that never end up actually getting used. And if I touch any of the higher level cookbooks (like the one for my org that configures some things via LWRPs and overrides some cookbook resources), berkshelf then has to parse the list of all those dependencies as well just to upload my local cookbook. I even tried using a group to filter things down but dependencies are resolved even when using groups. Berks also seems to re-upload every cookbook even if the version number hasn’t changed, which is odd to me. (Fixing that would minimize at least some of the annoyance factor of how long “berks upload” takes.)

Is this just a situation where “deal with it” is the most practical advice, or is there a way to break some of these dependencies that I know aren’t going to be reached in my use case (like the gnome and windows cookbooks…)?

~~ StormeRider ~~

“Every world needs its heroes […] They inspire us to be better than we are. And they protect from the darkness that’s just around the corner.”

(from Smallville Season 6x1: “Zod”)

On why I hate the phrase “that’s so lame”… http://bit.ly/Ps3uSS

~~ StormeRider ~~

“Every world needs its heroes […] They inspire us to be better than we are. And they protect from the darkness that’s just around the corner.”

(from Smallville Season 6x1: “Zod”)

On why I hate the phrase “that’s so lame”… http://bit.ly/Ps3uSS

~~ StormeRider ~~

“Every world needs its heroes […] They inspire us to be better than we are. And they protect from the darkness that’s just around the corner.”

(from Smallville Season 6x1: “Zod”)

On why I hate the phrase “that’s so lame”… http://bit.ly/Ps3uSS


#11

On Jun 5, 2013, at 12:36 PM, Ranjib Dey dey.ranjib@gmail.com wrote:

the problem is chef server does not mandate version freeze for cookbooks. You can upload the different content with same version, so for an external tool like berkshelf its impossible to understand if the cookbook has changed (since version is not the absolute indicator) , unless it does a full download checksum based comparison (this is how chef-client works). So its risky for a tool like berkshelf to do this, but definitely its a good option to have (in that case you might also like to specify only a subset of cookbooks for which you want to do this).

OTOH, Berkshelf could be configured to automatically update the version number on every change and freeze the version after uploading. Since the version would have been incremented and frozen by Berkshelf itself, it should be safe to assume that the cookbook is unchanged if the version number hasn’t changed.

Yes, I know people could un-freeze and then manually upload a different version with the same version number, but if they want to shoot themselves in the foot like that, then we should be prepared to let them do so.


Brad Knowles brad@shub-internet.org
LinkedIn Profile: http://tinyurl.com/y8kpxu


#12

Using 2.x.

On Wednesday, June 5, 2013, Brad Knowles wrote:

On Jun 5, 2013, at 12:36 PM, Ranjib Dey <dey.ranjib@gmail.com<javascript:;>>
wrote:

the problem is chef server does not mandate version freeze for
cookbooks. You can upload the different content with same version, so for
an external tool like berkshelf its impossible to understand if the
cookbook has changed (since version is not the absolute indicator) , unless
it does a full download checksum based comparison (this is how chef-client
works). So its risky for a tool like berkshelf to do this, but definitely
its a good option to have (in that case you might also like to specify only
a subset of cookbooks for which you want to do this).

OTOH, Berkshelf could be configured to automatically update the version
number on every change and freeze the version after uploading. Since the
version would have been incremented and frozen by Berkshelf itself, it
should be safe to assume that the cookbook is unchanged if the version
number hasn’t changed.

Yes, I know people could un-freeze and then manually upload a different
version with the same version number, but if they want to shoot themselves
in the foot like that, then we should be prepared to let them do so.


Brad Knowles <brad@shub-internet.org <javascript:;>>
LinkedIn Profile: http://tinyurl.com/y8kpxu

~~ StormeRider ~~

“Every world needs its heroes […] They inspire us to be better than we
are. And they protect from the darkness that’s just around the corner.”

(from Smallville Season 6x1: “Zod”)

On why I hate the phrase “that’s so lame”… http://bit.ly/Ps3uSS


#13

I haven’t been able to figure out how to get freezing to actually work. Is
there any documentation on it anywhere? I haven’t been able to find any,
just some of the tickets and commits, but I’m not exactly a Ruby guru :slight_smile:


~~ StormeRider ~~

“Every world needs its heroes […] They inspire us to be better than we
are. And they protect from the darkness that’s just around the corner.”

(from Smallville Season 6x1: “Zod”)

On why I hate the phrase “that’s so lame”… http://bit.ly/Ps3uSS

On Wed, Jun 5, 2013 at 12:15 PM, Morgan Blackthorne
stormerider@gmail.comwrote:

Using 2.x.

On Wednesday, June 5, 2013, Brad Knowles wrote:

On Jun 5, 2013, at 12:36 PM, Ranjib Dey dey.ranjib@gmail.com wrote:

the problem is chef server does not mandate version freeze for
cookbooks. You can upload the different content with same version, so for
an external tool like berkshelf its impossible to understand if the
cookbook has changed (since version is not the absolute indicator) , unless
it does a full download checksum based comparison (this is how chef-client
works). So its risky for a tool like berkshelf to do this, but definitely
its a good option to have (in that case you might also like to specify only
a subset of cookbooks for which you want to do this).

OTOH, Berkshelf could be configured to automatically update the version
number on every change and freeze the version after uploading. Since the
version would have been incremented and frozen by Berkshelf itself, it
should be safe to assume that the cookbook is unchanged if the version
number hasn’t changed.

Yes, I know people could un-freeze and then manually upload a different
version with the same version number, but if they want to shoot themselves
in the foot like that, then we should be prepared to let them do so.


Brad Knowles brad@shub-internet.org
LinkedIn Profile: http://tinyurl.com/y8kpxu

~~ StormeRider ~~

“Every world needs its heroes […] They inspire us to be better than we
are. And they protect from the darkness that’s just around the corner.”

(from Smallville Season 6x1: “Zod”)

On why I hate the phrase “that’s so lame”… http://bit.ly/Ps3uSS


#14

On Jun 10, 2013, at 3:31 AM, Morgan Blackthorne stormerider@gmail.com wrote:

I haven’t been able to figure out how to get freezing to actually work. Is there any documentation on it anywhere? I haven’t been able to find any, just some of the tickets and commits, but I’m not exactly a Ruby guru :slight_smile:

I’ve always just used “knife cookbook upload example_cookbook --freeze” and that did it for me.

Does that not work for you?


Brad Knowles brad@shub-internet.org
LinkedIn Profile: http://tinyurl.com/y8kpxu