Centralized cookbook-library repos vs distributed cookbook repos

[hedgehog]

On Wed, Apr 14, 2010 at 3:30 PM, Adam Jacob adam@opscode.com wrote:

Okay, in my head, I see two different workflows here:

1. The Consumer Workflow

This is the workflow where you find a cookbook that works for you, and
you want to use it in your infrastructure, and track it in your own
source control (probably a chef-repo.) You may then want to make
changes to the upstream cookbook, and you then want to be able to
easily see the diff between the changes you have made and any new
versions the upstream has published.

The important thing here is that you care about the cookbook having
it's changes tracked in your local repository (since you use it to
build your infrastructure), but you don't care so much about having
the entirety of the upstream revision history (indeed, you might not
be able to, in the cases where you want to use a different source
control system from the upstream.) So you want to easily track the
upstream, and apply your changes.

2. The Developer/Collaborator Workflow

In this case, you want to collaborate with the upstream on development
of the cookbook directly - you want your changes to be situated in the
upstream. To me, this workflow is actually defined by whomever is
managing the upstream - for Opscode, it's tickets and pull requests to
the opscode/chef repo, for 37signals it's different, and for someone
else it may require using mercurial or svn.

I've been thinking about the consumer workflow for a while now, and I
spent some of today working on getting a good pass at getting it down
to essentially a single command. For the developer/collaborator
workflow, we may also want to make a best practice, but I'm less
concerned about it right now.

The pattern I'm following is called "vendor branching", and it's been
around forever. The basic method is that you are tracking an upstream
source release, and you want to keep a certain set of patches in sync,
and be able to move between local versions easily. It was pretty
popular in the CVS days, if any of you can get into the way-back
machine with me. Here is a good description of doing this with Git:

Vendor branches in git – Jabbering Giraffe

I've adapted this pattern to be integrated with knife and the
cookbooks.opscode.com site. If you already have knife configured to
work with your local chef repo, from the current opscode/chef, you can
do:

$ knife cookbook site vendor apache2

And you'll get the latest version of the apache2 cookbook in a local
vendor branch. You can then make changes directly to the cookbook in
your local repository - you don't need to build a site-cookbook, or do
anything else. Just make changes like normal. When the upstream
releases a new version, just repeat the command:

$ knife cookbook site vendor apache2

And the new version will be downloaded, the vendor branch updated, and
then merged into your master branch. The resulting diff can be easily
viewed, and if there are any merge conflicts, you get a chance to
review them. If you don't like the changes, you can just 'git reset
--hard HEAD~1'.

You can always get a diff between your current cookbook and the upstream with:

[adam@latte]% git diff chef-vendor-apache2-0.10.1 HEAD
diff --git a/cookbooks/apache2/attributes/apache.rb
b/cookbooks/apache2/attributes/apache.rb
index c733d5e..d6a8f76 100644
--- a/cookbooks/apache2/attributes/apache.rb
+++ b/cookbooks/apache2/attributes/apache.rb
@@ -75,4 +75,3 @@ set_unless[:apache][:worker][:minsparethreads] = 64
set_unless[:apache][:worker][:maxsparethreads] = 192
set_unless[:apache][:worker][:threadsperchild] = 64
set_unless[:apache][:worker][:maxrequestsperchild] = 0
-# whee, vendor merge.

Check this picture of my local git history out - it shows going from
the 0.9.1 version of the apache2 cookbook to 0.10.1, with a local
change in the middle:

Skitch | Evernote

Check out this gist for some in-action shots:

chef-vendor-branches.txt · GitHub

In order to make this complete, a few things still need to be done:

1. We need to enable the API for uploading to the cookbook site
   easily, to make it simple for cookbook authors to publish their
   cookbooks.
2. We need to enable tracking of the upstream source repository in the
   cookbook site, along with an optional path, which should enable you to
   both vendor HEAD and easily collaborate on development.

What do you think?

That is one way of carving cookbooks out of a cookbook-library repo.
Perhaps I have misunderstood:
If I want to organize several libraries I have to download any common
cookbooks to all of them - correct?
When working on cookbook A in library 1, I make some vital changes,
which upstream loves.
How do those changes propogate to cookbook A in library 2?

Still not sure why the aversion to submodules.

I prefer using existing tools and think git solves 1) and 2). Inter
cookbook repo dependencies is an issue.
My preferred workflow is (currently):
A) See if using Git submodules to track individual cookbook repos works
B) See if the Bundler can be "massaged" to resolve inter-cookbook dependencies.
C) See if I can deploy from cookbook-libraries contsructed from the above

Still thinking...

Again this doesn't change anything for anyone else, just exporing how
I can organize my cookbooks and libraries of them with maximum
flexibility.

Regards

Adam

--
Opscode, Inc.
Adam Jacob, CTO
T: (206) 508-7449 E: adam@opscode.com

--
πόλλ' οἶδ ἀλώπηξ, ἀλλ' ἐχῖνος ἓν μέγα
[The fox knows many things, but the hedgehog knows one big thing.]
Archilochus, Greek poet (c. 680 BC – c. 645 BC)
http://wiki.hedgehogshiatus.com

[ringods]

Adam,

I am with you on the 2 different workflows, but I would back out of
going hard on an SCM oriented path. Going only the Git way leads to
a lot of manual setup and a barrier for Chef adoption.

Let me elaborate on a manner where chef itself is used to manage the
cookbooks. What follows below is a description of a 10 minute thinking
path. There will definitely be loose ends…

Here it goes:

• a recipe is a Resource (yes, a Chef Resource )
• Chef already has a gem_package Provider
• add to this, a Git/Hg/Bazaar/… Provider

Now you have the basic building blocks to use Chef itself to install
cookbooks on demand for the 2 workflows you mentioned, e.g:

• I use knife to install a role description
• the role description refers to recipes
• when a recipe is not yet available, the correct provider (git/gem)
  is used to retrieve the recipe/cookbooks and install them on the Chef
  server

About the workflows:

1. The more Chef will attract, the more recipes will become available
   and the more stable each recipe will become, supporting more different
   usage scenarios. Once becoming stable, a lot of users probably just
   want to use them, so why not publish these as gems? Gems already
   support proper versioning and dependencies. A role definition could
   then refer to a specific version of a recipe (packaged as a gem).
   Making sure a new published version of recipe is used in your setup is
   as simple as updating the version number in your role definition.

2. For the people who want to delve into extending/patching the
   recipes, they can still use an SCM based provider to have the recipe
   retrieved from a repository. A role definition could then refer to a
   specific commit of a recipe. Making sure that the new commit is used
   on the server is as easy as updating the revision hash. The SCM based
   provider then fetches the revision from the remote repository and it
   is used.

Remark that I talk about a version of a recipe, whether it be a gem
version number or a revision id of an SCM. The result is that a change
is required in the specification of a recipe in a role definition, and
in the metadata of a recipe to refer to other recipes used. Together
with it, recipes to manage the same thing can be delivered by multiple
people. The apache recipe is a good example here. As an additional
aspect of a recipe specification, a domain should be added. To
complete a possible specification, this is e.g. what I would write in
a role definition:

recipes( “com.opsode::users::1.2.13”, “com.37signals::sudo::3f0643335ce2” )

Updating to a new version of the users recipe is then changing this line:

recipes( “com.opsode::users::1.2.15”, “com.37signals::sudo::3f0643335ce2” )

The new recipe version get’s installed via the recipe resource
provider and then any nodes having the role will run the new recipe.

Can we still go more meta?

Ringo

[Dreamcat4]

Hi,
This is all great discussion. I hope others can follow too.

My questions:

• Does compliment / conflict wither the Chef infrastructures feature?
• Should we re-adjust Chef-infrastructures to take into account these use cases?

Thanks

Dreamcat4
dreamcat4@gmail.com

[Adam_Jacob]

I want to make sure I'm really clear about what Opscode actually
thinks here: our job is to provide you guys the primitives you need to
be able to do things the way you want to do them. So I <3 whatever
workflow anyone wants to use that works for them, and that they enjoy
using - and I want to make it as easy as possible for as many of those
workflows to be as painless as they can be.

So don't mistake my answering questions about a workflow, or giving an
opinion about it, to be a final answer - it's just me trying to
suss-out where the primitives are, so we can make it easy to get the
data you need.

On Tue, Apr 13, 2010 at 11:34 PM, Hedge Hog hedgehogshiatus@gmail.com wrote:

That is one way of carving cookbooks out of a cookbook-library repo.
Perhaps I have misunderstood:
If I want to organize several libraries I have to download any common
cookbooks to all of them - correct?
When working on cookbook A in library 1, I make some vital changes,
which upstream loves.
How do those changes propogate to cookbook A in library 2?

That's going to be up to the developer of the cookbook to tell you how
they want to be collaborated with. I think it's smart to come up with
a common consensus about what the best practice is for that, and
encourage people to set up their repositories that way. But we need
to be flexible enough to let people go a different way, because in the
end, it's up to the developer how they want to work with others to
incorporate changes to the code they have written.

The assumption that everyone in the chain is using Git, for example,
is a false one. Most of us are - but not all of us, and if someone
wants to use Launchpad and Mercurial for source control, I want to
include them, not force them to use a different source control system.

For getting changes from your local vendor branch back upstream, in
the example I cooked up , you could do a couple of things. Assuming
you wanted all your changes from a specific version, you could easily
just do this:

adam@latte% git diff chef-vendor-apache2-0.10.1 HEAD
diff --git a/cookbooks/apache2/attributes/apache.rb
b/cookbooks/apache2/attributes/apache.rb
index d6a8f76..b889de2 100644
--- a/cookbooks/apache2/attributes/apache.rb
+++ b/cookbooks/apache2/attributes/apache.rb
@@ -17,6 +17,8 @@

limitations under the License.

+### Moar comments!
+

Where the various parts of apache are

case platform
when "redhat","centos","fedora","suse"

If you wanted that to apply directly to the upstream, which in this
case is Opscode, your workflow would look like this:

$ cd ~/opscode-cookbooks
$ (cd ~/local-chef-repo; git diff --no-color
chef-vendor-apache2-0.10.1 HEAD) | patch -p2

Then you could commit your changes, push to your fork on github, send
them to the upstream, whatever. Again, it would depend on what the
upstream wants - in our case, we would want it on a branch with a
ticket in the cookbooks project on tickets.opscode.com, but someone
else may want a pull request from github. YMMV.

Still not sure why the aversion to submodules.

My personal aversion to submodules is that, as an upstream maintainer,
it's easier for me to maintain a single project, rather than publish a
separate git repo for all 80+ cookbooks we maintain. I'm not saying
you can't use submodules - just that I don't want to, personally.

I prefer using existing tools and think git solves 1) and 2). Inter
cookbook repo dependencies is an issue.

It solves 1 and 2 if everyone is using git, which they won't be.

My preferred workflow is (currently):
A) See if using Git submodules to track individual cookbook repos works
B) See if the Bundler can be "massaged" to resolve inter-cookbook dependencies.
C) See if I can deploy from cookbook-libraries contsructed from the above

I see almost an identical workflow developing from the hacking I did
yesterday. The difference would be that A and B are both provided
through discovery of published cookbooks on cookbooks.opscode.com. We
add the functionality to let you publish a cookbook with references to
the upstream source repository, and extend things to allow you to get
the cookbook artifact essentially from HEAD of the upstream. We can
do dependency resolution easily from there, since the deps in the
cookbooks will reflect the names of the cookbooks on the cookbook
site.

C will then be trivial.

Still thinking...

Ditto.

Again this doesn't change anything for anyone else, just exporing how
I can organize my cookbooks and libraries of them with maximum
flexibility.

Me too, and to me, that also means not being tied to a particular
source control system or repository layout, if possible.

Adam

--
Opscode, Inc.
Adam Jacob, CTO
T: (206) 508-7449 E: adam@opscode.com

[Adam_Jacob]

On Wed, Apr 14, 2010 at 3:20 AM, dreamcat four dreamcat4@gmail.com wrote:

Hi,
This is all great discussion. I hope others can follow too.

My questions:

• Does compliment / conflict wither the Chef infrastructures feature?

The stuff I worked up yesterday compliments it. The version numbers
of the cookbook will be what you can set a given infrastructures
cookbook policy to ("use version 0.10.1 in production, use HEAD in
development"). So the idea of tracking your changes with upstream
version changes is pretty solid.

• Should we re-adjust Chef-infrastructures to take into account these use cases?

I think we're already there.

Adam

--
Opscode, Inc.
Adam Jacob, CTO
T: (206) 508-7449 E: adam@opscode.com

[Alex_Soto]

great discussion, definitely a pain I'm experiencing while trying to use the platform is how to have my repo, have recipes from other repos and still track upstream changes and hopefully contribute back.

I think using git is a fair assumption for most of us early adopters. For the few that don't, a tarball is what you get.

I don't want to sound as elitist of git vs other scm's. It's actually for efficiency. The chef community isn't large enough to give all this flexibility to everyone. Git is the chosen tool for the time being. Sooner or later, someone else will scratch their itch for svn, mercurial, etc or other better workflows will emerge. Like water flowing downhill, someone will find the path of least resistance, but in the meantime, don't put too many hurdles in front of the community at large for the few in the minority.

My 2 cents.

Alex

On Apr 14, 2010, at 11:20 AM, Adam Jacob wrote:

I want to make sure I'm really clear about what Opscode actually
thinks here: our job is to provide you guys the primitives you need to
be able to do things the way you want to do them. So I <3 whatever
workflow anyone wants to use that works for them, and that they enjoy
using - and I want to make it as easy as possible for as many of those
workflows to be as painless as they can be.

So don't mistake my answering questions about a workflow, or giving an
opinion about it, to be a final answer - it's just me trying to
suss-out where the primitives are, so we can make it easy to get the
data you need.

On Tue, Apr 13, 2010 at 11:34 PM, Hedge Hog hedgehogshiatus@gmail.com wrote:

That is one way of carving cookbooks out of a cookbook-library repo.
Perhaps I have misunderstood:
If I want to organize several libraries I have to download any common
cookbooks to all of them - correct?
When working on cookbook A in library 1, I make some vital changes,
which upstream loves.
How do those changes propogate to cookbook A in library 2?

That's going to be up to the developer of the cookbook to tell you how
they want to be collaborated with. I think it's smart to come up with
a common consensus about what the best practice is for that, and
encourage people to set up their repositories that way. But we need
to be flexible enough to let people go a different way, because in the
end, it's up to the developer how they want to work with others to
incorporate changes to the code they have written.

The assumption that everyone in the chain is using Git, for example,
is a false one. Most of us are - but not all of us, and if someone
wants to use Launchpad and Mercurial for source control, I want to
include them, not force them to use a different source control system.

For getting changes from your local vendor branch back upstream, in
the example I cooked up , you could do a couple of things. Assuming
you wanted all your changes from a specific version, you could easily
just do this:

adam@latte% git diff chef-vendor-apache2-0.10.1 HEAD
diff --git a/cookbooks/apache2/attributes/apache.rb
b/cookbooks/apache2/attributes/apache.rb
index d6a8f76..b889de2 100644
--- a/cookbooks/apache2/attributes/apache.rb
+++ b/cookbooks/apache2/attributes/apache.rb
@@ -17,6 +17,8 @@

limitations under the License.

+### Moar comments!
+

Where the various parts of apache are

case platform
when "redhat","centos","fedora","suse"

If you wanted that to apply directly to the upstream, which in this
case is Opscode, your workflow would look like this:

$ cd ~/opscode-cookbooks
$ (cd ~/local-chef-repo; git diff --no-color
chef-vendor-apache2-0.10.1 HEAD) | patch -p2

Then you could commit your changes, push to your fork on github, send
them to the upstream, whatever. Again, it would depend on what the
upstream wants - in our case, we would want it on a branch with a
ticket in the cookbooks project on tickets.opscode.com, but someone
else may want a pull request from github. YMMV.

Still not sure why the aversion to submodules.

My personal aversion to submodules is that, as an upstream maintainer,
it's easier for me to maintain a single project, rather than publish a
separate git repo for all 80+ cookbooks we maintain. I'm not saying
you can't use submodules - just that I don't want to, personally.

I prefer using existing tools and think git solves 1) and 2). Inter
cookbook repo dependencies is an issue.

It solves 1 and 2 if everyone is using git, which they won't be.

My preferred workflow is (currently):
A) See if using Git submodules to track individual cookbook repos works
B) See if the Bundler can be "massaged" to resolve inter-cookbook dependencies.
C) See if I can deploy from cookbook-libraries contsructed from the above

I see almost an identical workflow developing from the hacking I did
yesterday. The difference would be that A and B are both provided
through discovery of published cookbooks on cookbooks.opscode.com. We
add the functionality to let you publish a cookbook with references to
the upstream source repository, and extend things to allow you to get
the cookbook artifact essentially from HEAD of the upstream. We can
do dependency resolution easily from there, since the deps in the
cookbooks will reflect the names of the cookbooks on the cookbook
site.

C will then be trivial.

Still thinking...

Ditto.

Again this doesn't change anything for anyone else, just exporing how
I can organize my cookbooks and libraries of them with maximum
flexibility.

Me too, and to me, that also means not being tied to a particular
source control system or repository layout, if possible.

Adam

--
Opscode, Inc.
Adam Jacob, CTO
T: (206) 508-7449 E: adam@opscode.com

[Adam_Jacob]

On Wed, Apr 14, 2010 at 12:34 PM, Alex Soto apsoto@gmail.com wrote:

great discussion, definitely a pain I'm experiencing while trying to use the platform is how to have my repo, have recipes from other repos and still track upstream changes and hopefully contribute back.

I think using git is a fair assumption for most of us early adopters. For the few that don't, a tarball is what you get.

I don't want to sound as elitist of git vs other scm's. It's actually for efficiency. The chef community isn't large enough to give all this flexibility to everyone. Git is the chosen tool for the time being. Sooner or later, someone else will scratch their itch for svn, mercurial, etc or other better workflows will emerge. Like water flowing downhill, someone will find the path of least resistance, but in the meantime, don't put too many hurdles in front of the community at large for the few in the minority.

Is the workflow I outlined earlier not efficient enough still? (How
much smaller can I make it?

Adam

--
Opscode, Inc.
Adam Jacob, CTO
T: (206) 508-7449 E: adam@opscode.com

[Alex_Soto]

My comment was triggered by you mentioning that:

The assumption that everyone in the chain is using Git, for example,
is a false one. Most of us are - but not all of us, and if someone
wants to use Launchpad and Mercurial for source control, I want to
include them, not force them to use a different source control system.

I read too much into that to think you meant to support all these other scm's which is not necessarily what you meant. My bad, I should have re-read things. My off the cuff remark, was because it just raised the flag in my head to point out that we don't want to over engineer this to make things super flexible. keep it simple

With respect to the developer workflow:

Here's my specific problem.

My previous chef deployment used my own chef server, which was cool because it allowed me to have multiple cookbook repos that the server pulled from. I maintained an internal repo that had my specific cookbooks for my infrastructure and I had a fork of the opscode repo that I could declare dependencies on from my 'internal repo'. My fork of the opscode repo also allowed me to update those cookbooks when I found problems.

Now I'm trying to use the platform (I'm still just barely scratched the surface of it, so I may be mistaken with how I'm using it), but it looks like I have effectively one cookbook repo which is the 'platform' where I have to push cookbooks into. So if I want to use some opscode cookbooks, I have to copy them and push them up individually. Because the granularity of the repo is not at the cookbook level, I don't see how I could have just a copy of an individual opscode cookbook and still track upstream changes easily.

I 'think' I like the one repo per cookbook strategy because those are the logical modules I think of when developing cookbooks/recipes. Then I would be able to fork and submodule them into my repo. However there is a nagging feeling that makes me feel that's a bit too granular and may get cumbersome to manage.

Alex

On Apr 14, 2010, at 12:39 PM, Adam Jacob wrote:

On Wed, Apr 14, 2010 at 12:34 PM, Alex Soto apsoto@gmail.com wrote:

great discussion, definitely a pain I'm experiencing while trying to use the platform is how to have my repo, have recipes from other repos and still track upstream changes and hopefully contribute back.

I think using git is a fair assumption for most of us early adopters. For the few that don't, a tarball is what you get.

I don't want to sound as elitist of git vs other scm's. It's actually for efficiency. The chef community isn't large enough to give all this flexibility to everyone. Git is the chosen tool for the time being. Sooner or later, someone else will scratch their itch for svn, mercurial, etc or other better workflows will emerge. Like water flowing downhill, someone will find the path of least resistance, but in the meantime, don't put too many hurdles in front of the community at large for the few in the minority.

Is the workflow I outlined earlier not efficient enough still? (How
much smaller can I make it?

Adam

--
Opscode, Inc.
Adam Jacob, CTO
T: (206) 508-7449 E: adam@opscode.com

[Adam_Jacob]

On Wed, Apr 14, 2010 at 1:08 PM, Alex Soto apsoto@gmail.com wrote:

My comment was triggered by you mentioning that:

The assumption that everyone in the chain is using Git, for example,
is a false one. Most of us are - but not all of us, and if someone
wants to use Launchpad and Mercurial for source control, I want to
include them, not force them to use a different source control system.

I read too much into that to think you meant to support all these other scm's which is not necessarily what you meant. My bad, I should have re-read things. My off the cuff remark, was because it just raised the flag in my head to point out that we don't want to over engineer this to make things super flexible. keep it simple

Absolutely. The pattern I proposed in an earlier message was using
'vendor branches', which exist in every VCS since CVS, to deal with
tracking your local changes against an up-stream release. The
integration with various source code control systems is easy here,
since it's just 'implement that VCS systems vendor branch pattern'.

Check out the source here for an example:

http://github.com/adamhjk/chef/blob/master/chef/lib/chef/knife/cookbook_site_vendor.rb

You can imagine how that could get re-factored to support multiple VCS
systems without much pain.

With respect to the developer workflow:

Here's my specific problem.

My previous chef deployment used my own chef server, which was cool because it allowed me to have multiple cookbook repos that the server pulled from. I maintained an internal repo that had my specific cookbooks for my infrastructure and I had a fork of the opscode repo that I could declare dependencies on from my 'internal repo'. My fork of the opscode repo also allowed me to update those cookbooks when I found problems.

Now I'm trying to use the platform (I'm still just barely scratched the surface of it, so I may be mistaken with how I'm using it), but it looks like I have effectively one cookbook repo which is the 'platform' where I have to push cookbooks into. So if I want to use some opscode cookbooks, I have to copy them and push them up individually. Because the granularity of the repo is not at the cookbook level, I don't see how I could have just a copy of an individual opscode cookbook and still track upstream changes easily.

The message I posted earlier is my answer to this - check it out and
see if it works. The idea is that you would still have your own git
repository full of all your own cookbooks (you want it this way,
because it lets you rebuild everything from scratch with only a copy
of what you care about,) that you're publishing to the platform or a
chef server.

It lets you track the upstream easily, and provides a more flexible
way for you to have custom modifications that travel forward in your
repository. I think it's pretty close.

I 'think' I like the one repo per cookbook strategy because those are the logical modules I think of when developing cookbooks/recipes. Then I would be able to fork and submodule them into my repo. However there is a nagging feeling that makes me feel that's a bit too granular and may get cumbersome to manage.

It would be significantly harder to manage for the maintainers of a
collection of complex, interdependent cookbooks.

Adam

--
Opscode, Inc.
Adam Jacob, CTO
T: (206) 508-7449 E: adam@opscode.com

[hedgehog]

On Fri, Apr 16, 2010 at 8:35 AM, Hedge Hog hedgehogshiatus@gmail.com wrote:

On Fri, Apr 16, 2010 at 7:43 AM, Adam Jacob adam@opscode.com wrote:

On Wed, Apr 14, 2010 at 7:08 PM, Hedge Hog hedgehogshiatus@gmail.com wrote:

This approach still preserves different cookbook-libraries as separate silo's.
Simply becasue it still ties dependency management to repo organisation.

Yes - the places where a given cookbook lives as a unit of
collaboration is a separate silo.

There is also a dependency management problem, but we have the
ground-work already to solve that.

More on it in a sec.

OK, this is getting clearer to me

I think we are getting side tracked by the fact that cookbook library
repo organization is being used to proxy for a dependency management
solution.
However, library repos are being justified/defended as easy to
maintain in terms of VCS effort and not dependency management
effort!
In my mind the two activities should be separate but have become conflated
I don't think VCS used has anything to do with dependency management.

I agree.

In reality the real reason for prefering library repos is not that
library repos are easier to maintain, rather cookbook dependencies
are easier to enforce.
Changing to cookbook repos will raise the same issues in CVS, hg, etc.

• If you are not going to enforce cookbook-library silos, how do you
  manage cookbook dependencies?

The cost of cookbook library repos is giving up distributed cookbook
development and maintenance/admin burdens, having cookbook library
silos develop indepently, etc all outlined in earlier emails
The benefit of cookbook library repos is trivial dependency magement:
Solve dependencies by getting all cookbooks from this cookbook-library
and no others.

The proposal to elevate cookbooks to repos is to try and create a low
pain path for cookbook contributions to flow around the community,
rather than just up-and-down different cookbook-library silos.
To do this we'd need a dependency management approach that works for
Chef's users and use cases.

I think there is a third way, which is that we already have a
publishing platform for cookbooks, and the cookbooks themselves have
dependencies. I've extended the vendor stuff I wrote before, so that
it now downloads and vendors the first cookbook, and then reads the
metadata and repeats the process recursively for all the dependencies:

% ./knife cookbook site vendor wordpress 0.5.0 -d

Now grabs wordpress and all it's dependencies, giving you a tree like this:

% ls -la cookbooks
total 8
drwxr-xr-x 8 adam adam 272 Apr 15 14:32 ./
drwxr-xr-x 11 adam adam 374 Apr 15 14:32 ../
-rw-r--r-- 1 adam adam 151 Apr 15 14:32 README
drwxr-xr-x 10 adam adam 340 Mar 10 16:43 apache2/
drwxr-xr-x 11 adam adam 374 Apr 15 14:32 mysql/
drwxr-xr-x 7 adam adam 238 Feb 26 14:35 openssl/
drwxr-xr-x 8 adam adam 272 Apr 15 14:32 php/
drwxr-xr-x 8 adam adam 272 Apr 15 14:32 wordpress/

% git branch
chef-vendor-apache2
chef-vendor-mysql
chef-vendor-openssl
chef-vendor-php
chef-vendor-wordpress

• master

% git tag
chef-vendor-apache2-0.10.1
chef-vendor-mysql-0.15.0
chef-vendor-openssl-0.1.0
chef-vendor-php-0.7.0
chef-vendor-wordpress-0.5.0

If we start publishing cookbooks and specifying dependencies in
metadata, and extend the cookbooks site to allow the upstream
developers to put the source control end-point for development in,
we'll have versioned cookbooks, with dependency management, along with
the mechanisms required to live on the bleeding edge. And it'll be
discoverable.

Sound good?

So it seems that the debate over cookbook repo approach really has
little (nothing?) to do with the effort involved in managing repos and
everything to do with the effort required by the dependency magaement
solution adopted.
Which really should be discussed in a separte thread:
'Freeing cookbook dependency management from source repo organization'

Thoughts?

I think we agree. There are (at least) 3 things here:

1. How do you discover cookbooks you want to use
2. How do you track them over time, and potentially make site-specific
   changes, and track those over time.
3. How do you track and resolve the dependencies one cookbook has on another

I think I've got the 'knife cookbook site vendor' stuff far enough
that it does all 3 of these based on the cookbook site information
today, and we can focus on adding in the functionality we need to
cover the remaining use cases.

You?

Sounds good.
Regular work can be a real PITA - give me 24hrs and I should have an
example alternative to discuss/evaluate

Better late than never,,,

http://hedgehogshiatus.com/carving-chefs-cookbooks

HTH

Regards

Adam

--
Opscode, Inc.
Adam Jacob, CTO
T: (206) 508-7449 E: adam@opscode.com

--
πόλλ' οἶδ ἀλώπηξ, ἀλλ' ἐχῖνος ἓν μέγα
[The fox knows many things, but the hedgehog knows one big thing.]
Archilochus, Greek poet (c. 680 BC – c. 645 BC)
http://wiki.hedgehogshiatus.com

--
πόλλ' οἶδ ἀλώπηξ, ἀλλ' ἐχῖνος ἓν μέγα
[The fox knows many things, but the hedgehog knows one big thing.]
Archilochus, Greek poet (c. 680 BC – c. 645 BC)
http://wiki.hedgehogshiatus.com

[Adam_Jacob]

On Thu, Dec 9, 2010 at 1:54 AM, Hedge Hog hedgehogshiatus@gmail.com wrote:

Better late than never,,,

http://hedgehogshiatus.com/carving-chefs-cookbooks

That looks epic, sir. I'll read it over.

Adam

--
Opscode, Inc.
Adam Jacob, CTO
T: (206) 508-7449 E: adam@opscode.com

[hedgehog]

On Fri, Dec 10, 2010 at 4:18 AM, Adam Jacob adam@opscode.com wrote:

On Thu, Dec 9, 2010 at 1:54 AM, Hedge Hog hedgehogshiatus@gmail.com wrote:

Better late than never,,,

http://hedgehogshiatus.com/carving-chefs-cookbooks

That looks epic, sir. I'll read it over.

My Bundler fork's Chef branch is now at least usable without any
changes to Chef recipe names (resolving dependencies has been improved
too, still some Bundler bugs remain, and features missing that will be
useful in Chef's context):

http://hedgehogshiatus.com/carving-and-bundling-chef-cookbooks-alpha-2

Adam

--
Opscode, Inc.
Adam Jacob, CTO
T: (206) 508-7449 E: adam@opscode.com

--
πόλλ' οἶδ ἀλώπηξ, ἀλλ' ἐχῖνος ἓν μέγα
[The fox knows many things, but the hedgehog knows one big thing.]
Archilochus, Greek poet (c. 680 BC – c. 645 BC)
http://wiki.hedgehogshiatus.com

[Dreamcat4]

On Tue, Dec 14, 2010 at 7:02 AM, Hedge Hog hedgehogshiatus@gmail.com wrote:

On Fri, Dec 10, 2010 at 4:18 AM, Adam Jacob adam@opscode.com wrote:

On Thu, Dec 9, 2010 at 1:54 AM, Hedge Hog hedgehogshiatus@gmail.com wrote:

Better late than never,,,

http://hedgehogshiatus.com/carving-chefs-cookbooks

That looks epic, sir. I'll read it over.

My Bundler fork's Chef branch is now at least usable without any
changes to Chef recipe names (resolving dependencies has been improved
too, still some Bundler bugs remain, and features missing that will be
useful in Chef's context):

http://hedgehogshiatus.com/carving-and-bundling-chef-cookbooks-alpha-2

I couldn't find an issue for this. But it looks like there should be
one raised to the Bundler team.

Adam

--
Opscode, Inc.
Adam Jacob, CTO
T: (206) 508-7449 E: adam@opscode.com

--
πόλλ' οἶδ ἀλώπηξ, ἀλλ' ἐχῖνος ἓν μέγα
[The fox knows many things, but the hedgehog knows one big thing.]
Archilochus, Greek poet (c. 680 BC – c. 645 BC)
http://wiki.hedgehogshiatus.com

[hedgehog]

On Tue, Dec 14, 2010 at 9:53 PM, Dreamcat4 dreamcat4@gmail.com wrote:

On Tue, Dec 14, 2010 at 7:02 AM, Hedge Hog hedgehogshiatus@gmail.com wrote:

On Fri, Dec 10, 2010 at 4:18 AM, Adam Jacob adam@opscode.com wrote:

On Thu, Dec 9, 2010 at 1:54 AM, Hedge Hog hedgehogshiatus@gmail.com wrote:

Better late than never,,,

http://hedgehogshiatus.com/carving-chefs-cookbooks

That looks epic, sir. I'll read it over.

My Bundler fork's Chef branch is now at least usable without any
changes to Chef recipe names (resolving dependencies has been improved
too, still some Bundler bugs remain, and features missing that will be
useful in Chef's context):

http://hedgehogshiatus.com/carving-and-bundling-chef-cookbooks-alpha-2

I couldn't find an issue for this. But it looks like there should be
one raised to the Bundler team.

Issues · rubygems/bundler

The more voices speak up for this use case the more likely it is they
will consider it.
I think I've uncovered the cause of the error:
Could not find - in any of the sources

It may be related to this:

https://github.com/carlhuda/bundler/issuesearch?state=open&q=pack#issue/67

I'm not yet sure so I haven't opened a ticket. It is very difficult
to tell what is doing on with out unit tests, so far I know bundle
install passes around very different sets of data describing the repo,
depending on whether bundle install is called first or subsequently,
but nothing explicit to indicate this....
All by way of saying working in Bundler internals is a very unpleasant
experience and don't expect too rapid progress

Adam

--
Opscode, Inc.
Adam Jacob, CTO
T: (206) 508-7449 E: adam@opscode.com

--
πόλλ' οἶδ ἀλώπηξ, ἀλλ' ἐχῖνος ἓν μέγα
[The fox knows many things, but the hedgehog knows one big thing.]
Archilochus, Greek poet (c. 680 BC – c. 645 BC)
http://wiki.hedgehogshiatus.com

--
πόλλ' οἶδ ἀλώπηξ, ἀλλ' ἐχῖνος ἓν μέγα
[The fox knows many things, but the hedgehog knows one big thing.]
Archilochus, Greek poet (c. 680 BC – c. 645 BC)
http://wiki.hedgehogshiatus.com

[hedgehog]

On Thu, Apr 15, 2010 at 5:39 AM, Adam Jacob adam@opscode.com wrote:

On Wed, Apr 14, 2010 at 12:34 PM, Alex Soto apsoto@gmail.com wrote:

great discussion, definitely a pain I'm experiencing while trying to use the platform is how to have my repo, have recipes from other repos and still track upstream changes and hopefully contribute back.

I think using git is a fair assumption for most of us early adopters. For the few that don't, a tarball is what you get.

I don't want to sound as elitist of git vs other scm's. It's actually for efficiency. The chef community isn't large enough to give all this flexibility to everyone. Git is the chosen tool for the time being. Sooner or later, someone else will scratch their itch for svn, mercurial, etc or other better workflows will emerge. Like water flowing downhill, someone will find the path of least resistance, but in the meantime, don't put too many hurdles in front of the community at large for the few in the minority.

Is the workflow I outlined earlier not efficient enough still?

Depends.
How long do I have to wait for someone's cookbook change in the repo
to showup in:
/Users/adam/src/sandbox/opscode/chef-repo/cookbooks/apache2.tar.gz

True,

$ ./knife cookbook site vendor apache2 0.9.1

is as good as

$ (cd ; git pull)

I just ask:
Is pay the price of giving up distributed cookbook development worth
the effort saved in dependency management?
I think that depends on the dependency mangement alternatives.

My inital raction is no it is not worth it, and is worth looking into.

Regards

(How
much smaller can I make it?

Adam

--
Opscode, Inc.
Adam Jacob, CTO
T: (206) 508-7449 E: adam@opscode.com

--
πόλλ' οἶδ ἀλώπηξ, ἀλλ' ἐχῖνος ἓν μέγα
[The fox knows many things, but the hedgehog knows one big thing.]
Archilochus, Greek poet (c. 680 BC – c. 645 BC)
http://wiki.hedgehogshiatus.com

[Dreamcat4]

If I may, I would like to offer some discussion about dependency
resolution and cookbook sources.

Over on cookbooks.opscode.com, we make people register an account
username (or organisational account name, such as '37signals'). Isnt
the account username what we are already registering as the upstream
source / cookbooks namespace?

Then over on github.com, that same username string translates to our
github account name. Eg http://github.com/37signals/cookbooks. Other
SCMs like launchpad and bitbucket should be very easily compatible
with some similar naming convention too. If the chef tools can know to
check at cookbooks.opscode.com for the authority of the registered
account names.

Then it seems to follow that (in some version of chef) a local
cookbooks folder might look like this:

/srv/chef/cookbooks/opscode
/srv/chef/cookbooks/37signals
/srv/chef/cookbooks/dreamcat4
/srv/chef/site-cookbooks/opscode
/srv/chef/site-cookbooks/37signals
/srv/chef/site-cookbooks/dreamcat4

To explicitly reference the 37signals' apache cookbook, I might write:

include_recipe '37signals/apache:recipe_name'

And the same way for roles etc.

When no prefix is specified like for now, the implicit namespace could
default to "opscode". Or we might reference the locally bound
namespace (at some kind of locally bound level).

Im not trying to state that Opscode here can or should really do it
like that. But it seemed kindda worth mentioning here in the
discussion.

On Wed, Apr 14, 2010 at 10:04 PM, Adam Jacob adam@opscode.com wrote:

On Wed, Apr 14, 2010 at 1:08 PM, Alex Soto apsoto@gmail.com wrote:

My comment was triggered by you mentioning that:

The assumption that everyone in the chain is using Git, for example,
is a false one. Most of us are - but not all of us, and if someone
wants to use Launchpad and Mercurial for source control, I want to
include them, not force them to use a different source control system.

I read too much into that to think you meant to support all these other scm's which is not necessarily what you meant. My bad, I should have re-read things. My off the cuff remark, was because it just raised the flag in my head to point out that we don't want to over engineer this to make things super flexible. keep it simple

Absolutely. The pattern I proposed in an earlier message was using
'vendor branches', which exist in every VCS since CVS, to deal with
tracking your local changes against an up-stream release. The
integration with various source code control systems is easy here,
since it's just 'implement that VCS systems vendor branch pattern'.

Check out the source here for an example:

http://github.com/adamhjk/chef/blob/master/chef/lib/chef/knife/cookbook_site_vendor.rb

You can imagine how that could get re-factored to support multiple VCS
systems without much pain.

With respect to the developer workflow:

Here's my specific problem.

My previous chef deployment used my own chef server, which was cool because it allowed me to have multiple cookbook repos that the server pulled from. I maintained an internal repo that had my specific cookbooks for my infrastructure and I had a fork of the opscode repo that I could declare dependencies on from my 'internal repo'. My fork of the opscode repo also allowed me to update those cookbooks when I found problems.

Now I'm trying to use the platform (I'm still just barely scratched the surface of it, so I may be mistaken with how I'm using it), but it looks like I have effectively one cookbook repo which is the 'platform' where I have to push cookbooks into. So if I want to use some opscode cookbooks, I have to copy them and push them up individually. Because the granularity of the repo is not at the cookbook level, I don't see how I could have just a copy of an individual opscode cookbook and still track upstream changes easily.

The message I posted earlier is my answer to this - check it out and
see if it works. The idea is that you would still have your own git
repository full of all your own cookbooks (you want it this way,
because it lets you rebuild everything from scratch with only a copy
of what you care about,) that you're publishing to the platform or a
chef server.

It lets you track the upstream easily, and provides a more flexible
way for you to have custom modifications that travel forward in your
repository. I think it's pretty close.

I 'think' I like the one repo per cookbook strategy because those are the logical modules I think of when developing cookbooks/recipes. Then I would be able to fork and submodule them into my repo. However there is a nagging feeling that makes me feel that's a bit too granular and may get cumbersome to manage.

It would be significantly harder to manage for the maintainers of a
collection of complex, interdependent cookbooks.

Adam

--
Opscode, Inc.
Adam Jacob, CTO
T: (206) 508-7449 E: adam@opscode.com

[hedgehog]

On Thu, Apr 15, 2010 at 7:04 AM, Adam Jacob adam@opscode.com wrote:

On Wed, Apr 14, 2010 at 1:08 PM, Alex Soto apsoto@gmail.com wrote:

My comment was triggered by you mentioning that:

The assumption that everyone in the chain is using Git, for example,
is a false one. Most of us are - but not all of us, and if someone
wants to use Launchpad and Mercurial for source control, I want to
include them, not force them to use a different source control system.

I read too much into that to think you meant to support all these other scm's which is not necessarily what you meant. My bad, I should have re-read things. My off the cuff remark, was because it just raised the flag in my head to point out that we don't want to over engineer this to make things super flexible. keep it simple

Absolutely. The pattern I proposed in an earlier message was using
'vendor branches', which exist in every VCS since CVS, to deal with
tracking your local changes against an up-stream release.

This approach still preserves different cookbook-libraries as separate silo's.
Simply becasue it still ties dependency management to repo organisation.

The
integration with various source code control systems is easy here,
since it's just 'implement that VCS systems vendor branch pattern'.

Check out the source here for an example:

http://github.com/adamhjk/chef/blob/master/chef/lib/chef/knife/cookbook_site_vendor.rb

You can imagine how that could get re-factored to support multiple VCS
systems without much pain.

OK, this is getting clearer to me

I think we are getting side tracked by the fact that cookbook library
repo organization is being used to proxy for a dependency management
solution.
However, library repos are being justified/defended as easy to
maintain in terms of VCS effort and not dependency management
effort!
In my mind the two activities should be separate but have become conflated
I don't think VCS used has anything to do with dependency management.

In reality the real reason for prefering library repos is not that
library repos are easier to maintain, rather cookbook dependencies
are easier to enforce.
Changing to cookbook repos will raise the same issues in CVS, hg, etc.

• If you are not going to enforce cookbook-library silos, how do you
  manage cookbook dependencies?

The cost of cookbook library repos is giving up distributed cookbook
development and maintenance/admin burdens, having cookbook library
silos develop indepently, etc all outlined in earlier emails
The benefit of cookbook library repos is trivial dependency magement:
Solve dependencies by getting all cookbooks from this cookbook-library
and no others.

The proposal to elevate cookbooks to repos is to try and create a low
pain path for cookbook contributions to flow around the community,
rather than just up-and-down different cookbook-library silos.
To do this we'd need a dependency management approach that works for
Chef's users and use cases.

With respect to the developer workflow:

Here's my specific problem.

My previous chef deployment used my own chef server, which was cool because it allowed me to have multiple cookbook repos that the server pulled from. I maintained an internal repo that had my specific cookbooks for my infrastructure and I had a fork of the opscode repo that I could declare dependencies on from my 'internal repo'. My fork of the opscode repo also allowed me to update those cookbooks when I found problems.

Now I'm trying to use the platform (I'm still just barely scratched the surface of it, so I may be mistaken with how I'm using it), but it looks like I have effectively one cookbook repo which is the 'platform' where I have to push cookbooks into. So if I want to use some opscode cookbooks, I have to copy them and push them up individually. Because the granularity of the repo is not at the cookbook level, I don't see how I could have just a copy of an individual opscode cookbook and still track upstream changes easily.

The message I posted earlier is my answer to this - check it out and
see if it works. The idea is that you would still have your own git
repository full of all your own cookbooks (you want it this way,
because it lets you rebuild everything from scratch with only a copy
of what you care about,) that you're publishing to the platform or a
chef server.

It lets you track the upstream easily, and provides a more flexible
way for you to have custom modifications that travel forward in your
repository. I think it's pretty close.

I 'think' I like the one repo per cookbook strategy because those are the logical modules I think of when developing cookbooks/recipes. Then I would be able to fork and submodule them into my repo. However there is a nagging feeling that makes me feel that's a bit too granular and may get cumbersome to manage.

It would be significantly harder to manage for the maintainers of a
collection of complex, interdependent cookbooks.

Maybe.
Initially, there will be some work to do in resolving dependencies
when merging some cookbook's contents from libraries that are
currently treated as 'off-limits'.
But that work will be there only if you choose to change your
cookbooks, or accept changes into them.
Additional cookbook repo admin can be pushed out to cookbook
maintainers, and they will only make the effort to bring in changes if
they think they add value.
Updating and pushing a library of cookbook submodules involves 3 git commands

   $ # git commit all cookbooks changes in superproject
   $ git add .
   $ git commit -m "Updated submodules."
   $ # git submodule update to push change to the individual

repositories that predate the superproject.
$ git push # only if you have write access

Worth the effort if there are a wider set of cookbook maintainers to
pick and choose from - of course they would have to have described
their cookbook's dependencies correctly - usual open-source issues.

So it seems that the debate over cookbook repo approach really has
little (nothing?) to do with the effort involved in managing repos and
everything to do with the effort required by the dependency magaement
solution adopted.
Which really should be discussed in a separte thread:
'Freeing cookbook dependency management from source repo organization'

Thoughts?

Regards

Adam

--
Opscode, Inc.
Adam Jacob, CTO
T: (206) 508-7449 E: adam@opscode.com

--
πόλλ' οἶδ ἀλώπηξ, ἀλλ' ἐχῖνος ἓν μέγα
[The fox knows many things, but the hedgehog knows one big thing.]
Archilochus, Greek poet (c. 680 BC – c. 645 BC)
http://wiki.hedgehogshiatus.com

[Adam_Jacob]

On Wed, Apr 14, 2010 at 6:19 PM, Hedge Hog hedgehogshiatus@gmail.com wrote:

Is the workflow I outlined earlier not efficient enough still?

Depends.
How long do I have to wait for someone's cookbook change in the repo
to showup in:
/Users/adam/src/sandbox/opscode/chef-repo/cookbooks/apache2.tar.gz

True,

$ ./knife cookbook site vendor apache2 0.9.1

is as good as

$ (cd ; git pull)

I just ask:
Is pay the price of giving up distributed cookbook development worth
the effort saved in dependency management?
I think that depends on the dependency mangement alternatives.

My inital raction is no it is not worth it, and is worth looking into.

Right - I think we need to extend it so you can also do:

$ ./knife cookbook site vendor apache2 HEAD

And we'll handle the case where you are willing to track the upstream
directly without a release.

Adam

--
Opscode, Inc.
Adam Jacob, CTO
T: (206) 508-7449 E: adam@opscode.com

[Adam_Jacob]

On Wed, Apr 14, 2010 at 6:28 PM, dreamcat four dreamcat4@gmail.com wrote:

If I may, I would like to offer some discussion about dependency
resolution and cookbook sources.

Over on cookbooks.opscode.com, we make people register an account
username (or organisational account name, such as '37signals'). Isnt
the account username what we are already registering as the upstream
source / cookbooks namespace?

Then over on github.com, that same username string translates to our
github account name. Eg http://github.com/37signals/cookbooks. Other
SCMs like launchpad and bitbucket should be very easily compatible
with some similar naming convention too. If the chef tools can know to
check at cookbooks.opscode.com for the authority of the registered
account names.

Then it seems to follow that (in some version of chef) a local
cookbooks folder might look like this:

/srv/chef/cookbooks/opscode
/srv/chef/cookbooks/37signals
/srv/chef/cookbooks/dreamcat4
/srv/chef/site-cookbooks/opscode
/srv/chef/site-cookbooks/37signals
/srv/chef/site-cookbooks/dreamcat4

To explicitly reference the 37signals' apache cookbook, I might write:

include_recipe '37signals/apache:recipe_name'

And the same way for roles etc.

When no prefix is specified like for now, the implicit namespace could
default to "opscode". Or we might reference the locally bound
namespace (at some kind of locally bound level).

Im not trying to state that Opscode here can or should really do it
like that. But it seemed kindda worth mentioning here in the
discussion.

Yep - as of right now, we have a flat namespace for cookbooks, in the
same way you have a flat namespace for libraries in rubygems or CPAN.

I have a feeling that something exactly like you are describing is
going to wind up being the de-facto standard - we'll start putting a
prefix in front of the recipes we share to disambiguate.

Adam

--
Opscode, Inc.
Adam Jacob, CTO
T: (206) 508-7449 E: adam@opscode.com

[Adam_Jacob]

On Wed, Apr 14, 2010 at 7:08 PM, Hedge Hog hedgehogshiatus@gmail.com wrote:

This approach still preserves different cookbook-libraries as separate silo's.
Simply becasue it still ties dependency management to repo organisation.

Yes - the places where a given cookbook lives as a unit of
collaboration is a separate silo.

There is also a dependency management problem, but we have the
ground-work already to solve that.

More on it in a sec.

OK, this is getting clearer to me

I think we are getting side tracked by the fact that cookbook library
repo organization is being used to proxy for a dependency management
solution.
However, library repos are being justified/defended as easy to
maintain in terms of VCS effort and not dependency management
effort!
In my mind the two activities should be separate but have become conflated
I don't think VCS used has anything to do with dependency management.

I agree.

In reality the real reason for prefering library repos is not that
library repos are easier to maintain, rather cookbook dependencies
are easier to enforce.
Changing to cookbook repos will raise the same issues in CVS, hg, etc.

• If you are not going to enforce cookbook-library silos, how do you
  manage cookbook dependencies?

The cost of cookbook library repos is giving up distributed cookbook
development and maintenance/admin burdens, having cookbook library
silos develop indepently, etc all outlined in earlier emails
The benefit of cookbook library repos is trivial dependency magement:
Solve dependencies by getting all cookbooks from this cookbook-library
and no others.

The proposal to elevate cookbooks to repos is to try and create a low
pain path for cookbook contributions to flow around the community,
rather than just up-and-down different cookbook-library silos.
To do this we'd need a dependency management approach that works for
Chef's users and use cases.

I think there is a third way, which is that we already have a
publishing platform for cookbooks, and the cookbooks themselves have
dependencies. I've extended the vendor stuff I wrote before, so that
it now downloads and vendors the first cookbook, and then reads the
metadata and repeats the process recursively for all the dependencies:

% ./knife cookbook site vendor wordpress 0.5.0 -d

Now grabs wordpress and all it's dependencies, giving you a tree like this:

% ls -la cookbooks
total 8
drwxr-xr-x 8 adam adam 272 Apr 15 14:32 ./
drwxr-xr-x 11 adam adam 374 Apr 15 14:32 ../
-rw-r--r-- 1 adam adam 151 Apr 15 14:32 README
drwxr-xr-x 10 adam adam 340 Mar 10 16:43 apache2/
drwxr-xr-x 11 adam adam 374 Apr 15 14:32 mysql/
drwxr-xr-x 7 adam adam 238 Feb 26 14:35 openssl/
drwxr-xr-x 8 adam adam 272 Apr 15 14:32 php/
drwxr-xr-x 8 adam adam 272 Apr 15 14:32 wordpress/

% git branch
chef-vendor-apache2
chef-vendor-mysql
chef-vendor-openssl
chef-vendor-php
chef-vendor-wordpress

• master

% git tag
chef-vendor-apache2-0.10.1
chef-vendor-mysql-0.15.0
chef-vendor-openssl-0.1.0
chef-vendor-php-0.7.0
chef-vendor-wordpress-0.5.0

If we start publishing cookbooks and specifying dependencies in
metadata, and extend the cookbooks site to allow the upstream
developers to put the source control end-point for development in,
we'll have versioned cookbooks, with dependency management, along with
the mechanisms required to live on the bleeding edge. And it'll be
discoverable.

Sound good?

So it seems that the debate over cookbook repo approach really has
little (nothing?) to do with the effort involved in managing repos and
everything to do with the effort required by the dependency magaement
solution adopted.
Which really should be discussed in a separte thread:
'Freeing cookbook dependency management from source repo organization'

Thoughts?

I think we agree. There are (at least) 3 things here:

1. How do you discover cookbooks you want to use
2. How do you track them over time, and potentially make site-specific
   changes, and track those over time.
3. How do you track and resolve the dependencies one cookbook has on another

I think I've got the 'knife cookbook site vendor' stuff far enough
that it does all 3 of these based on the cookbook site information
today, and we can focus on adding in the functionality we need to
cover the remaining use cases.

You?

Adam

--
Opscode, Inc.
Adam Jacob, CTO
T: (206) 508-7449 E: adam@opscode.com