From a naive persective it would seem that adding a "priority" keyword to
the Berksfile format seems like it would be a simple way to address this
issue. Then, we could say:
source 'https://supermarket.chef.io', priority: 100
source 'https://my-private-supermarket.example.com', priority: 200
and then Berkshelf would choose the first matching cookbook in
my-private-supermarket, or, if no matches, look at supermarket.chef.io.
If you were to add such a feature, you open up a much more difficult
situation. The issue that comes up when you abandon the idea of a flat
cookbook namespace is dependency resolution on the Chef server. If
cookbook "foo" uses the "apache" cookbook from the community supermarket
and cookbook "bar" (in the same chef org) uses the "apache" cookbook from
the private supermarket, I don't think there's currently a mechanism to
prioritize one over the other in each context.
So ultimately, this isn't an Berkshelf issue at all.
On Wed, Jan 7, 2015 at 7:57 AM, Koert Kuipers koert@tresata.com wrote:
"path:", "git:" and "github:" always felt strange to me in Berks simply
because they point to only one release. this seems to be at odds with the
idea of berkshelf, which is to have berkshelf pick the most suitable
version based on metadata release constraints.
for example if i release a new version of inhouse cookbook x inhouse that
can handle yum >= 3, then i dont want that one to be applied (picked) until
all other cookbooks can also handle yum >= 3, which means i want to make it
available and let berkshelf pick it when the time is ready. i dont see how
i can do that with path:, github: or git: in Berkshelf file.
so to me an in-house supermarket is the way to go. however i certainly do
not want to exclude the public supermarket, because that seems way too
complex and tedious. it should not be that way.
so right now i am going through the process or renaming all inhouse
cookbooks, which is far from easy, since the name is all over the place
(comments, attributes, recipes, lwrps, etc.). i find it very frustrating.
it seems only a few very basic things are needed to not have to do this
futile renaming exercise: support for release tags such as 0.1.2-mycompany,
and maybe a simple blacklist to keep a few cookbooks from public
supermarket out if i need to, or a way to set precendence perhaps. however
i think just simple release tags would get us 95% of the way there.
note that in java we deal with artifacts and transitive dependencies all
the time, and we have in-house versions of public artifacts all the time
too, and it certainly does not involve renaming all artifacts and modules.
On Tue, Jan 6, 2015 at 10:45 PM, Koert Kuipers koert@tresata.com wrote:
that's an ticket interesting. having SNAPSHOT releases (with upload
checks accordingly that allow overwrites) would be a great thing to have as
well.
On Tue, Jan 6, 2015 at 6:46 PM, Torben Knerr mail@tknerr.de wrote:
There was https://tickets.opscode.com/browse/CHEF-4027 once, but it
seems it had not been migrated to the Github issues (at least I
couldn't find it there).
And what's so bad about referencing Git repos from a Berksfile?
It's much more explicit and transparent than overloading namespaces
globally by pointing to a different "source" location.
Cheers,
Torben
On Tue, Jan 6, 2015 at 9:38 PM, Noah Kantrowitz noah@coderanger.net
wrote:
On Jan 6, 2015, at 11:35 AM, Koert Kuipers koert@tresata.com wrote:
renaming internal cookbooks is not really an option, since it would
also involve renaming all attributes and it would all together create a
giant mess.
it seems odd that i am forced to rename cookbook "x" just because
someone managed to upload an "x" to the central supermarket, even if the
"x" on the central supermarket is barely used.
it is also unclear to me how we can go about modifying an existing
cookbook. typically we do this and then a forced to temporarily use an
in-house version (until our pullreq is accepted and incorporated in a
release). for this we do not want to rename the cookbook since that is way
too disruptive and the change is only temporary. again this problem would
be solved if our inhouse supermarket could take precendence. with that
option not available i looked at other alternatives. one would be to
reference a git repo, since a git repo can take precendence over
supermarket, but everyone says that is bad practice. ok then i thought,
what if we modify cookbook "x" version 1.0.0 and publish it inhouse as
cookbook "x" version 1.0.0-mycompany, and pin the version we depend on?
again no luck, since chef and berks dont support such pre-release versions.
Indeed, the minimal support for complex version numbers is a bit
painful here. There are a few major options:
The most robust solution is to stop using "source '
https://supermarket.chef.io'" in your Berksfiles in favor of your own
berks-api server pointed at your own Chef server and nothing else. This
allows you to create an isolated universe where you control the namespace
"entirely". The air quotes are because if you import any cookbooks from the
community site, you may run in to issues where things depend on the
community version of X instead of your incompatible version of X. This can
be worked around at times, but you'll either have to not use community
cookbooks as much, or eat the renaming pain on the community cookbook side
eventually.
In cases where it is truly a temporary fork, I'll try really hard to
keep the logic in a wrapper cookbook via chef-rewind or monkey patching but
that isn't always going to be possible. If a fork has to happen, I'll
generally set the version to 99.99.99 so it is unlikely to ever collide
with a real release version, and then lock that version in the wrapper
cookbook or environment (or both).
This in turn sometimes won't work if a ton of other cookbooks outside
my control have version dependencies on the things I'm modifying. At this
point a lot of people might be tempted to just leave the version number the
same or bump it by 0.0.1 and upload to their Chef server, but here be
dragons. When using non-git sources, Berks stores cookbooks by
"name-version", not content hash, so you risk effectively a self-induced
cache poisoning attack where some people might get your modified version in
their cache and others might get the unmodified community version, and the
only way you can really tell is when things break mysteriously despite
using the same Berksfile.lock. If you must do a local fork with the same
version number, definitely keep it in the Berksfile as a git reference so
you don't clobber your cache.
If you don't like any of these options, sometimes discretion is the
better part of valor and you should just wait for a new release. For
cookbooks maintained by volunteers on their own time, offering to pay for
some development time can greatly accelerate things but tread lightly as
some devs might not like that idea. Good luck, you may need it 
--Noah
--
Justin Dossey
Practice Owner
New Context Services, Inc