In-house berks-api cookbook naming conflicts


#1

i just set up an in-house berks-api. its pretty neat!

in Berksfile i now do:
source "http://chef.my.organization:26200"
source “https://supermarket.getchef.com

my only issue is that we have some inhouse cookbooks that have the same
name as the ones on the supermarket. for example we have a kafka cookbook.
i used to do:
cookbook “kafka”, path: "…/kafka"
but now that i have an inhouse berks-api i would like to just say:
cookbook "kafka"
and have it download from our chef server. unfortunately this means it
downloads another cookbook from supermarket which has the same name and a
newer version.

i tried doing:
cookbook “kafka”, source: "http://chef.my.organization:26200"
but that didn’t work.

any suggestions? i would like to avoid having to rename all our inhouse
cookbooks.
i also find this behaviour somewhat dangerous since i dont control
supermarket. so if i have cookbook x and a week later someone adds cookbook
x to supermarket without me being aware of it my cookbook could get
suddenly clobbered.

thanks! koert


#2

On Jan 2, 2015, at 11:00 AM, Koert Kuipers koert@tresata.com wrote:

i just set up an in-house berks-api. its pretty neat!

in Berksfile i now do:
source "http://chef.my.organization:26200"
source “https://supermarket.getchef.com

my only issue is that we have some inhouse cookbooks that have the same name as the ones on the supermarket. for example we have a kafka cookbook. i used to do:
cookbook “kafka”, path: "…/kafka"
but now that i have an inhouse berks-api i would like to just say:
cookbook "kafka"
and have it download from our chef server. unfortunately this means it downloads another cookbook from supermarket which has the same name and a newer version.

i tried doing:
cookbook “kafka”, source: "http://chef.my.organization:26200"
but that didn’t work.

any suggestions? i would like to avoid having to rename all our inhouse cookbooks.
i also find this behaviour somewhat dangerous since i dont control supermarket. so if i have cookbook x and a week later someone adds cookbook x to supermarket without me being aware of it my cookbook could get suddenly clobbered.

Always name things ‘mycompany-thing’, assume a globally flat namespace. The alternative is to run only off the internal berks API and the cross load stuff in to your chef server somewhat more manually when you want to pull in a community cookbook.

–Noah


#3

On Wed, Jan 7, 2015 at 12:52 AM, Noah Kantrowitz noah@coderanger.net wrote:

On Jan 6, 2015, at 3:46 PM, Torben Knerr mail@tknerr.de wrote:

There was https://tickets.opscode.com/browse/CHEF-4027 once, but it
seems it had not been migrated to the Github issues (at least I
couldn’t find it there).

And what’s so bad about referencing Git repos from a Berksfile?

You have to do it everywhere individually rather than relying on centralized release processes. Definitely doable on a small scale, but it can rapidly get unsustainable.

But isn’t that the same as updating a cookbook version? You have to do
that in every place you want the new version, too.

Maybe we are just talking about continous integration vs. locking deps
for a release here?

It’s much more explicit and transparent than overloading namespaces
globally by pointing to a different “source” location.

Having an explicitly isolated universe of cookbooks is just as explicit, but it may throw some people for a loop if they are used to existing in the community namespace. There has been much talk of how to handle namespace management in the community, but little motion other than myself and a few others waving arms. Until something changes there, you must treat the cookbook namespace as globally flat and thus if you want to control it you need to make your own bubble universe off the side.

It’s only as explicit if you look at the whole universe (and there
might be more than one afaiu), not if you look at individual projects.
And how can I tell that cookbook-xyz 1.0.0 in my private universe is
actually a fork of the community cookbook-xyz 1.0.0 in an internal
repo? Would this information get lost or does the /universe endpoint
know about that and tell you if you ask?

Cheers,
Torben

–Noah


#4

that’s an ticket interesting. having SNAPSHOT releases (with upload checks
accordingly that allow overwrites) would be a great thing to have as well.

On Tue, Jan 6, 2015 at 6:46 PM, Torben Knerr mail@tknerr.de wrote:

There was https://tickets.opscode.com/browse/CHEF-4027 once, but it
seems it had not been migrated to the Github issues (at least I
couldn’t find it there).

And what’s so bad about referencing Git repos from a Berksfile?

It’s much more explicit and transparent than overloading namespaces
globally by pointing to a different “source” location.

Cheers,
Torben

On Tue, Jan 6, 2015 at 9:38 PM, Noah Kantrowitz noah@coderanger.net
wrote:

On Jan 6, 2015, at 11:35 AM, Koert Kuipers koert@tresata.com wrote:

renaming internal cookbooks is not really an option, since it would
also involve renaming all attributes and it would all together create a
giant mess.

it seems odd that i am forced to rename cookbook “x” just because
someone managed to upload an “x” to the central supermarket, even if the
"x" on the central supermarket is barely used.

it is also unclear to me how we can go about modifying an existing
cookbook. typically we do this and then a forced to temporarily use an
in-house version (until our pullreq is accepted and incorporated in a
release). for this we do not want to rename the cookbook since that is way
too disruptive and the change is only temporary. again this problem would
be solved if our inhouse supermarket could take precendence. with that
option not available i looked at other alternatives. one would be to
reference a git repo, since a git repo can take precendence over
supermarket, but everyone says that is bad practice. ok then i thought,
what if we modify cookbook “x” version 1.0.0 and publish it inhouse as
cookbook “x” version 1.0.0-mycompany, and pin the version we depend on?
again no luck, since chef and berks dont support such pre-release versions.

Indeed, the minimal support for complex version numbers is a bit painful
here. There are a few major options:

The most robust solution is to stop using “source ‘
https://supermarket.chef.io’” in your Berksfiles in favor of your own
berks-api server pointed at your own Chef server and nothing else. This
allows you to create an isolated universe where you control the namespace
"entirely". The air quotes are because if you import any cookbooks from the
community site, you may run in to issues where things depend on the
community version of X instead of your incompatible version of X. This can
be worked around at times, but you’ll either have to not use community
cookbooks as much, or eat the renaming pain on the community cookbook side
eventually.

In cases where it is truly a temporary fork, I’ll try really hard to
keep the logic in a wrapper cookbook via chef-rewind or monkey patching but
that isn’t always going to be possible. If a fork has to happen, I’ll
generally set the version to 99.99.99 so it is unlikely to ever collide
with a real release version, and then lock that version in the wrapper
cookbook or environment (or both).

This in turn sometimes won’t work if a ton of other cookbooks outside my
control have version dependencies on the things I’m modifying. At this
point a lot of people might be tempted to just leave the version number the
same or bump it by 0.0.1 and upload to their Chef server, but here be
dragons. When using non-git sources, Berks stores cookbooks by
"name-version", not content hash, so you risk effectively a self-induced
cache poisoning attack where some people might get your modified version in
their cache and others might get the unmodified community version, and the
only way you can really tell is when things break mysteriously despite
using the same Berksfile.lock. If you must do a local fork with the same
version number, definitely keep it in the Berksfile as a git reference so
you don’t clobber your cache.

If you don’t like any of these options, sometimes discretion is the
better part of valor and you should just wait for a new release. For
cookbooks maintained by volunteers on their own time, offering to pay for
some development time can greatly accelerate things but tread lightly as
some devs might not like that idea. Good luck, you may need it :slight_smile:

–Noah


#5

“path:”, “git:” and “github:” always felt strange to me in Berks simply
because they point to only one release. this seems to be at odds with the
idea of berkshelf, which is to have berkshelf pick the most suitable
version based on metadata release constraints.

for example if i release a new version of inhouse cookbook x inhouse that
can handle yum >= 3, then i dont want that one to be applied (picked) until
all other cookbooks can also handle yum >= 3, which means i want to make it
available and let berkshelf pick it when the time is ready. i dont see how
i can do that with path:, github: or git: in Berkshelf file.

so to me an in-house supermarket is the way to go. however i certainly do
not want to exclude the public supermarket, because that seems way too
complex and tedious. it should not be that way.

so right now i am going through the process or renaming all inhouse
cookbooks, which is far from easy, since the name is all over the place
(comments, attributes, recipes, lwrps, etc.). i find it very frustrating.

it seems only a few very basic things are needed to not have to do this
futile renaming exercise: support for release tags such as 0.1.2-mycompany,
and maybe a simple blacklist to keep a few cookbooks from public
supermarket out if i need to, or a way to set precendence perhaps. however
i think just simple release tags would get us 95% of the way there.

note that in java we deal with artifacts and transitive dependencies all
the time, and we have in-house versions of public artifacts all the time
too, and it certainly does not involve renaming all artifacts and modules.

On Tue, Jan 6, 2015 at 10:45 PM, Koert Kuipers koert@tresata.com wrote:

that’s an ticket interesting. having SNAPSHOT releases (with upload checks
accordingly that allow overwrites) would be a great thing to have as well.

On Tue, Jan 6, 2015 at 6:46 PM, Torben Knerr mail@tknerr.de wrote:

There was https://tickets.opscode.com/browse/CHEF-4027 once, but it
seems it had not been migrated to the Github issues (at least I
couldn’t find it there).

And what’s so bad about referencing Git repos from a Berksfile?

It’s much more explicit and transparent than overloading namespaces
globally by pointing to a different “source” location.

Cheers,
Torben

On Tue, Jan 6, 2015 at 9:38 PM, Noah Kantrowitz noah@coderanger.net
wrote:

On Jan 6, 2015, at 11:35 AM, Koert Kuipers koert@tresata.com wrote:

renaming internal cookbooks is not really an option, since it would
also involve renaming all attributes and it would all together create a
giant mess.

it seems odd that i am forced to rename cookbook “x” just because
someone managed to upload an “x” to the central supermarket, even if the
"x" on the central supermarket is barely used.

it is also unclear to me how we can go about modifying an existing
cookbook. typically we do this and then a forced to temporarily use an
in-house version (until our pullreq is accepted and incorporated in a
release). for this we do not want to rename the cookbook since that is way
too disruptive and the change is only temporary. again this problem would
be solved if our inhouse supermarket could take precendence. with that
option not available i looked at other alternatives. one would be to
reference a git repo, since a git repo can take precendence over
supermarket, but everyone says that is bad practice. ok then i thought,
what if we modify cookbook “x” version 1.0.0 and publish it inhouse as
cookbook “x” version 1.0.0-mycompany, and pin the version we depend on?
again no luck, since chef and berks dont support such pre-release versions.

Indeed, the minimal support for complex version numbers is a bit
painful here. There are a few major options:

The most robust solution is to stop using “source ‘
https://supermarket.chef.io’” in your Berksfiles in favor of your own
berks-api server pointed at your own Chef server and nothing else. This
allows you to create an isolated universe where you control the namespace
"entirely". The air quotes are because if you import any cookbooks from the
community site, you may run in to issues where things depend on the
community version of X instead of your incompatible version of X. This can
be worked around at times, but you’ll either have to not use community
cookbooks as much, or eat the renaming pain on the community cookbook side
eventually.

In cases where it is truly a temporary fork, I’ll try really hard to
keep the logic in a wrapper cookbook via chef-rewind or monkey patching but
that isn’t always going to be possible. If a fork has to happen, I’ll
generally set the version to 99.99.99 so it is unlikely to ever collide
with a real release version, and then lock that version in the wrapper
cookbook or environment (or both).

This in turn sometimes won’t work if a ton of other cookbooks outside
my control have version dependencies on the things I’m modifying. At this
point a lot of people might be tempted to just leave the version number the
same or bump it by 0.0.1 and upload to their Chef server, but here be
dragons. When using non-git sources, Berks stores cookbooks by
"name-version", not content hash, so you risk effectively a self-induced
cache poisoning attack where some people might get your modified version in
their cache and others might get the unmodified community version, and the
only way you can really tell is when things break mysteriously despite
using the same Berksfile.lock. If you must do a local fork with the same
version number, definitely keep it in the Berksfile as a git reference so
you don’t clobber your cache.

If you don’t like any of these options, sometimes discretion is the
better part of valor and you should just wait for a new release. For
cookbooks maintained by volunteers on their own time, offering to pay for
some development time can greatly accelerate things but tread lightly as
some devs might not like that idea. Good luck, you may need it :slight_smile:

–Noah


#6

From a naive persective it would seem that adding a “priority” keyword to
the Berksfile format seems like it would be a simple way to address this
issue. Then, we could say:

source ‘https://supermarket.chef.io’, priority: 100
source ‘https://my-private-supermarket.example.com’, priority: 200

and then Berkshelf would choose the first matching cookbook in
my-private-supermarket, or, if no matches, look at supermarket.chef.io.

If you were to add such a feature, you open up a much more difficult
situation. The issue that comes up when you abandon the idea of a flat
cookbook namespace is dependency resolution on the Chef server. If
cookbook “foo” uses the “apache” cookbook from the community supermarket
and cookbook “bar” (in the same chef org) uses the “apache” cookbook from
the private supermarket, I don’t think there’s currently a mechanism to
prioritize one over the other in each context.

So ultimately, this isn’t an Berkshelf issue at all.

On Wed, Jan 7, 2015 at 7:57 AM, Koert Kuipers koert@tresata.com wrote:

“path:”, “git:” and “github:” always felt strange to me in Berks simply
because they point to only one release. this seems to be at odds with the
idea of berkshelf, which is to have berkshelf pick the most suitable
version based on metadata release constraints.

for example if i release a new version of inhouse cookbook x inhouse that
can handle yum >= 3, then i dont want that one to be applied (picked) until
all other cookbooks can also handle yum >= 3, which means i want to make it
available and let berkshelf pick it when the time is ready. i dont see how
i can do that with path:, github: or git: in Berkshelf file.

so to me an in-house supermarket is the way to go. however i certainly do
not want to exclude the public supermarket, because that seems way too
complex and tedious. it should not be that way.

so right now i am going through the process or renaming all inhouse
cookbooks, which is far from easy, since the name is all over the place
(comments, attributes, recipes, lwrps, etc.). i find it very frustrating.

it seems only a few very basic things are needed to not have to do this
futile renaming exercise: support for release tags such as 0.1.2-mycompany,
and maybe a simple blacklist to keep a few cookbooks from public
supermarket out if i need to, or a way to set precendence perhaps. however
i think just simple release tags would get us 95% of the way there.

note that in java we deal with artifacts and transitive dependencies all
the time, and we have in-house versions of public artifacts all the time
too, and it certainly does not involve renaming all artifacts and modules.

On Tue, Jan 6, 2015 at 10:45 PM, Koert Kuipers koert@tresata.com wrote:

that’s an ticket interesting. having SNAPSHOT releases (with upload
checks accordingly that allow overwrites) would be a great thing to have as
well.

On Tue, Jan 6, 2015 at 6:46 PM, Torben Knerr mail@tknerr.de wrote:

There was https://tickets.opscode.com/browse/CHEF-4027 once, but it
seems it had not been migrated to the Github issues (at least I
couldn’t find it there).

And what’s so bad about referencing Git repos from a Berksfile?

It’s much more explicit and transparent than overloading namespaces
globally by pointing to a different “source” location.

Cheers,
Torben

On Tue, Jan 6, 2015 at 9:38 PM, Noah Kantrowitz noah@coderanger.net
wrote:

On Jan 6, 2015, at 11:35 AM, Koert Kuipers koert@tresata.com wrote:

renaming internal cookbooks is not really an option, since it would
also involve renaming all attributes and it would all together create a
giant mess.

it seems odd that i am forced to rename cookbook “x” just because
someone managed to upload an “x” to the central supermarket, even if the
"x" on the central supermarket is barely used.

it is also unclear to me how we can go about modifying an existing
cookbook. typically we do this and then a forced to temporarily use an
in-house version (until our pullreq is accepted and incorporated in a
release). for this we do not want to rename the cookbook since that is way
too disruptive and the change is only temporary. again this problem would
be solved if our inhouse supermarket could take precendence. with that
option not available i looked at other alternatives. one would be to
reference a git repo, since a git repo can take precendence over
supermarket, but everyone says that is bad practice. ok then i thought,
what if we modify cookbook “x” version 1.0.0 and publish it inhouse as
cookbook “x” version 1.0.0-mycompany, and pin the version we depend on?
again no luck, since chef and berks dont support such pre-release versions.

Indeed, the minimal support for complex version numbers is a bit
painful here. There are a few major options:

The most robust solution is to stop using “source ‘
https://supermarket.chef.io’” in your Berksfiles in favor of your own
berks-api server pointed at your own Chef server and nothing else. This
allows you to create an isolated universe where you control the namespace
"entirely". The air quotes are because if you import any cookbooks from the
community site, you may run in to issues where things depend on the
community version of X instead of your incompatible version of X. This can
be worked around at times, but you’ll either have to not use community
cookbooks as much, or eat the renaming pain on the community cookbook side
eventually.

In cases where it is truly a temporary fork, I’ll try really hard to
keep the logic in a wrapper cookbook via chef-rewind or monkey patching but
that isn’t always going to be possible. If a fork has to happen, I’ll
generally set the version to 99.99.99 so it is unlikely to ever collide
with a real release version, and then lock that version in the wrapper
cookbook or environment (or both).

This in turn sometimes won’t work if a ton of other cookbooks outside
my control have version dependencies on the things I’m modifying. At this
point a lot of people might be tempted to just leave the version number the
same or bump it by 0.0.1 and upload to their Chef server, but here be
dragons. When using non-git sources, Berks stores cookbooks by
"name-version", not content hash, so you risk effectively a self-induced
cache poisoning attack where some people might get your modified version in
their cache and others might get the unmodified community version, and the
only way you can really tell is when things break mysteriously despite
using the same Berksfile.lock. If you must do a local fork with the same
version number, definitely keep it in the Berksfile as a git reference so
you don’t clobber your cache.

If you don’t like any of these options, sometimes discretion is the
better part of valor and you should just wait for a new release. For
cookbooks maintained by volunteers on their own time, offering to pay for
some development time can greatly accelerate things but tread lightly as
some devs might not like that idea. Good luck, you may need it :slight_smile:

–Noah


Justin Dossey
Practice Owner
New Context Services, Inc


#7

yeah i see the issues with priority, since we also upload general
supermarket cookbooks into our private supermarket as part of the process
(this is not the case with java, where the private maven repo only holds
private artifacts). so since say the yum cookcook is also sitting in our
private repo this would get messy quickly: we would never see updates for
the yum cookbook since the inhouse always has precedence!

priorities would work well if our private supermarket only held the
cookbooks we uploaded to it, not their dependencies. then the separation
would be clean and priorities would work. that seems doable.

On Wed, Jan 7, 2015 at 12:11 PM, Justin Dossey <justin.dossey@newcontext.com

wrote:

From a naive persective it would seem that adding a “priority” keyword to
the Berksfile format seems like it would be a simple way to address this
issue. Then, we could say:

source ‘https://supermarket.chef.io’, priority: 100
source ‘https://my-private-supermarket.example.com’, priority: 200

and then Berkshelf would choose the first matching cookbook in
my-private-supermarket, or, if no matches, look at supermarket.chef.io.

If you were to add such a feature, you open up a much more difficult
situation. The issue that comes up when you abandon the idea of a flat
cookbook namespace is dependency resolution on the Chef server. If
cookbook “foo” uses the “apache” cookbook from the community supermarket
and cookbook “bar” (in the same chef org) uses the “apache” cookbook from
the private supermarket, I don’t think there’s currently a mechanism to
prioritize one over the other in each context.

So ultimately, this isn’t an Berkshelf issue at all.

On Wed, Jan 7, 2015 at 7:57 AM, Koert Kuipers koert@tresata.com wrote:

“path:”, “git:” and “github:” always felt strange to me in Berks simply
because they point to only one release. this seems to be at odds with the
idea of berkshelf, which is to have berkshelf pick the most suitable
version based on metadata release constraints.

for example if i release a new version of inhouse cookbook x inhouse that
can handle yum >= 3, then i dont want that one to be applied (picked) until
all other cookbooks can also handle yum >= 3, which means i want to make it
available and let berkshelf pick it when the time is ready. i dont see how
i can do that with path:, github: or git: in Berkshelf file.

so to me an in-house supermarket is the way to go. however i certainly do
not want to exclude the public supermarket, because that seems way too
complex and tedious. it should not be that way.

so right now i am going through the process or renaming all inhouse
cookbooks, which is far from easy, since the name is all over the place
(comments, attributes, recipes, lwrps, etc.). i find it very frustrating.

it seems only a few very basic things are needed to not have to do this
futile renaming exercise: support for release tags such as 0.1.2-mycompany,
and maybe a simple blacklist to keep a few cookbooks from public
supermarket out if i need to, or a way to set precendence perhaps. however
i think just simple release tags would get us 95% of the way there.

note that in java we deal with artifacts and transitive dependencies all
the time, and we have in-house versions of public artifacts all the time
too, and it certainly does not involve renaming all artifacts and modules.

On Tue, Jan 6, 2015 at 10:45 PM, Koert Kuipers koert@tresata.com wrote:

that’s an ticket interesting. having SNAPSHOT releases (with upload
checks accordingly that allow overwrites) would be a great thing to have as
well.

On Tue, Jan 6, 2015 at 6:46 PM, Torben Knerr mail@tknerr.de wrote:

There was https://tickets.opscode.com/browse/CHEF-4027 once, but it
seems it had not been migrated to the Github issues (at least I
couldn’t find it there).

And what’s so bad about referencing Git repos from a Berksfile?

It’s much more explicit and transparent than overloading namespaces
globally by pointing to a different “source” location.

Cheers,
Torben

On Tue, Jan 6, 2015 at 9:38 PM, Noah Kantrowitz noah@coderanger.net
wrote:

On Jan 6, 2015, at 11:35 AM, Koert Kuipers koert@tresata.com wrote:

renaming internal cookbooks is not really an option, since it would
also involve renaming all attributes and it would all together create a
giant mess.

it seems odd that i am forced to rename cookbook “x” just because
someone managed to upload an “x” to the central supermarket, even if the
"x" on the central supermarket is barely used.

it is also unclear to me how we can go about modifying an existing
cookbook. typically we do this and then a forced to temporarily use an
in-house version (until our pullreq is accepted and incorporated in a
release). for this we do not want to rename the cookbook since that is way
too disruptive and the change is only temporary. again this problem would
be solved if our inhouse supermarket could take precendence. with that
option not available i looked at other alternatives. one would be to
reference a git repo, since a git repo can take precendence over
supermarket, but everyone says that is bad practice. ok then i thought,
what if we modify cookbook “x” version 1.0.0 and publish it inhouse as
cookbook “x” version 1.0.0-mycompany, and pin the version we depend on?
again no luck, since chef and berks dont support such pre-release versions.

Indeed, the minimal support for complex version numbers is a bit
painful here. There are a few major options:

The most robust solution is to stop using “source ‘
https://supermarket.chef.io’” in your Berksfiles in favor of your own
berks-api server pointed at your own Chef server and nothing else. This
allows you to create an isolated universe where you control the namespace
"entirely". The air quotes are because if you import any cookbooks from the
community site, you may run in to issues where things depend on the
community version of X instead of your incompatible version of X. This can
be worked around at times, but you’ll either have to not use community
cookbooks as much, or eat the renaming pain on the community cookbook side
eventually.

In cases where it is truly a temporary fork, I’ll try really hard to
keep the logic in a wrapper cookbook via chef-rewind or monkey patching but
that isn’t always going to be possible. If a fork has to happen, I’ll
generally set the version to 99.99.99 so it is unlikely to ever collide
with a real release version, and then lock that version in the wrapper
cookbook or environment (or both).

This in turn sometimes won’t work if a ton of other cookbooks outside
my control have version dependencies on the things I’m modifying. At this
point a lot of people might be tempted to just leave the version number the
same or bump it by 0.0.1 and upload to their Chef server, but here be
dragons. When using non-git sources, Berks stores cookbooks by
"name-version", not content hash, so you risk effectively a self-induced
cache poisoning attack where some people might get your modified version in
their cache and others might get the unmodified community version, and the
only way you can really tell is when things break mysteriously despite
using the same Berksfile.lock. If you must do a local fork with the same
version number, definitely keep it in the Berksfile as a git reference so
you don’t clobber your cache.

If you don’t like any of these options, sometimes discretion is the
better part of valor and you should just wait for a new release. For
cookbooks maintained by volunteers on their own time, offering to pay for
some development time can greatly accelerate things but tread lightly as
some devs might not like that idea. Good luck, you may need it :slight_smile:

–Noah


Justin Dossey
Practice Owner
New Context Services, Inc


#8

renaming internal cookbooks is not really an option, since it would also
involve renaming all attributes and it would all together create a giant
mess.

it seems odd that i am forced to rename cookbook “x” just because someone
managed to upload an “x” to the central supermarket, even if the “x” on the
central supermarket is barely used.

it is also unclear to me how we can go about modifying an existing
cookbook. typically we do this and then a forced to temporarily use an
in-house version (until our pullreq is accepted and incorporated in a
release). for this we do not want to rename the cookbook since that is way
too disruptive and the change is only temporary. again this problem would
be solved if our inhouse supermarket could take precendence. with that
option not available i looked at other alternatives. one would be to
reference a git repo, since a git repo can take precendence over
supermarket, but everyone says that is bad practice. ok then i thought,
what if we modify cookbook “x” version 1.0.0 and publish it inhouse as
cookbook “x” version 1.0.0-mycompany, and pin the version we depend on?
again no luck, since chef and berks dont support such pre-release versions.

On Fri, Jan 2, 2015 at 2:36 PM, Noah Kantrowitz noah@coderanger.net wrote:

On Jan 2, 2015, at 11:00 AM, Koert Kuipers koert@tresata.com wrote:

i just set up an in-house berks-api. its pretty neat!

in Berksfile i now do:
source "http://chef.my.organization:26200"
source “https://supermarket.getchef.com

my only issue is that we have some inhouse cookbooks that have the same
name as the ones on the supermarket. for example we have a kafka cookbook.
i used to do:
cookbook “kafka”, path: "…/kafka"
but now that i have an inhouse berks-api i would like to just say:
cookbook "kafka"
and have it download from our chef server. unfortunately this means it
downloads another cookbook from supermarket which has the same name and a
newer version.

i tried doing:
cookbook “kafka”, source: "http://chef.my.organization:26200"
but that didn’t work.

any suggestions? i would like to avoid having to rename all our inhouse
cookbooks.
i also find this behaviour somewhat dangerous since i dont control
supermarket. so if i have cookbook x and a week later someone adds cookbook
x to supermarket without me being aware of it my cookbook could get
suddenly clobbered.

Always name things ‘mycompany-thing’, assume a globally flat namespace.
The alternative is to run only off the internal berks API and the cross
load stuff in to your chef server somewhat more manually when you want to
pull in a community cookbook.

–Noah


#9

On Jan 6, 2015, at 11:35 AM, Koert Kuipers koert@tresata.com wrote:

renaming internal cookbooks is not really an option, since it would also involve renaming all attributes and it would all together create a giant mess.

it seems odd that i am forced to rename cookbook “x” just because someone managed to upload an “x” to the central supermarket, even if the “x” on the central supermarket is barely used.

it is also unclear to me how we can go about modifying an existing cookbook. typically we do this and then a forced to temporarily use an in-house version (until our pullreq is accepted and incorporated in a release). for this we do not want to rename the cookbook since that is way too disruptive and the change is only temporary. again this problem would be solved if our inhouse supermarket could take precendence. with that option not available i looked at other alternatives. one would be to reference a git repo, since a git repo can take precendence over supermarket, but everyone says that is bad practice. ok then i thought, what if we modify cookbook “x” version 1.0.0 and publish it inhouse as cookbook “x” version 1.0.0-mycompany, and pin the version we depend on? again no luck, since chef and berks dont support such pre-release versions.

Indeed, the minimal support for complex version numbers is a bit painful here. There are a few major options:

The most robust solution is to stop using “source ‘https://supermarket.chef.io’” in your Berksfiles in favor of your own berks-api server pointed at your own Chef server and nothing else. This allows you to create an isolated universe where you control the namespace “entirely”. The air quotes are because if you import any cookbooks from the community site, you may run in to issues where things depend on the community version of X instead of your incompatible version of X. This can be worked around at times, but you’ll either have to not use community cookbooks as much, or eat the renaming pain on the community cookbook side eventually.

In cases where it is truly a temporary fork, I’ll try really hard to keep the logic in a wrapper cookbook via chef-rewind or monkey patching but that isn’t always going to be possible. If a fork has to happen, I’ll generally set the version to 99.99.99 so it is unlikely to ever collide with a real release version, and then lock that version in the wrapper cookbook or environment (or both).

This in turn sometimes won’t work if a ton of other cookbooks outside my control have version dependencies on the things I’m modifying. At this point a lot of people might be tempted to just leave the version number the same or bump it by 0.0.1 and upload to their Chef server, but here be dragons. When using non-git sources, Berks stores cookbooks by “name-version”, not content hash, so you risk effectively a self-induced cache poisoning attack where some people might get your modified version in their cache and others might get the unmodified community version, and the only way you can really tell is when things break mysteriously despite using the same Berksfile.lock. If you must do a local fork with the same version number, definitely keep it in the Berksfile as a git reference so you don’t clobber your cache.

If you don’t like any of these options, sometimes discretion is the better part of valor and you should just wait for a new release. For cookbooks maintained by volunteers on their own time, offering to pay for some development time can greatly accelerate things but tread lightly as some devs might not like that idea. Good luck, you may need it :slight_smile:

–Noah


#10

There was https://tickets.opscode.com/browse/CHEF-4027 once, but it
seems it had not been migrated to the Github issues (at least I
couldn’t find it there).

And what’s so bad about referencing Git repos from a Berksfile?

It’s much more explicit and transparent than overloading namespaces
globally by pointing to a different “source” location.

Cheers,
Torben

On Tue, Jan 6, 2015 at 9:38 PM, Noah Kantrowitz noah@coderanger.net wrote:

On Jan 6, 2015, at 11:35 AM, Koert Kuipers koert@tresata.com wrote:

renaming internal cookbooks is not really an option, since it would also involve renaming all attributes and it would all together create a giant mess.

it seems odd that i am forced to rename cookbook “x” just because someone managed to upload an “x” to the central supermarket, even if the “x” on the central supermarket is barely used.

it is also unclear to me how we can go about modifying an existing cookbook. typically we do this and then a forced to temporarily use an in-house version (until our pullreq is accepted and incorporated in a release). for this we do not want to rename the cookbook since that is way too disruptive and the change is only temporary. again this problem would be solved if our inhouse supermarket could take precendence. with that option not available i looked at other alternatives. one would be to reference a git repo, since a git repo can take precendence over supermarket, but everyone says that is bad practice. ok then i thought, what if we modify cookbook “x” version 1.0.0 and publish it inhouse as cookbook “x” version 1.0.0-mycompany, and pin the version we depend on? again no luck, since chef and berks dont support such pre-release versions.

Indeed, the minimal support for complex version numbers is a bit painful here. There are a few major options:

The most robust solution is to stop using “source ‘https://supermarket.chef.io’” in your Berksfiles in favor of your own berks-api server pointed at your own Chef server and nothing else. This allows you to create an isolated universe where you control the namespace “entirely”. The air quotes are because if you import any cookbooks from the community site, you may run in to issues where things depend on the community version of X instead of your incompatible version of X. This can be worked around at times, but you’ll either have to not use community cookbooks as much, or eat the renaming pain on the community cookbook side eventually.

In cases where it is truly a temporary fork, I’ll try really hard to keep the logic in a wrapper cookbook via chef-rewind or monkey patching but that isn’t always going to be possible. If a fork has to happen, I’ll generally set the version to 99.99.99 so it is unlikely to ever collide with a real release version, and then lock that version in the wrapper cookbook or environment (or both).

This in turn sometimes won’t work if a ton of other cookbooks outside my control have version dependencies on the things I’m modifying. At this point a lot of people might be tempted to just leave the version number the same or bump it by 0.0.1 and upload to their Chef server, but here be dragons. When using non-git sources, Berks stores cookbooks by “name-version”, not content hash, so you risk effectively a self-induced cache poisoning attack where some people might get your modified version in their cache and others might get the unmodified community version, and the only way you can really tell is when things break mysteriously despite using the same Berksfile.lock. If you must do a local fork with the same version number, definitely keep it in the Berksfile as a git reference so you don’t clobber your cache.

If you don’t like any of these options, sometimes discretion is the better part of valor and you should just wait for a new release. For cookbooks maintained by volunteers on their own time, offering to pay for some development time can greatly accelerate things but tread lightly as some devs might not like that idea. Good luck, you may need it :slight_smile:

–Noah


#11

On Jan 6, 2015, at 3:46 PM, Torben Knerr mail@tknerr.de wrote:

There was https://tickets.opscode.com/browse/CHEF-4027 once, but it
seems it had not been migrated to the Github issues (at least I
couldn’t find it there).

And what’s so bad about referencing Git repos from a Berksfile?

You have to do it everywhere individually rather than relying on centralized release processes. Definitely doable on a small scale, but it can rapidly get unsustainable.

It’s much more explicit and transparent than overloading namespaces
globally by pointing to a different “source” location.

Having an explicitly isolated universe of cookbooks is just as explicit, but it may throw some people for a loop if they are used to existing in the community namespace. There has been much talk of how to handle namespace management in the community, but little motion other than myself and a few others waving arms. Until something changes there, you must treat the cookbook namespace as globally flat and thus if you want to control it you need to make your own bubble universe off the side.

–Noah