Cookbook Releases, summit edition

#1

Ohai!

Last week we had the first Opscode Chef Community Summit in Seattle. A hot topic from that related to cookbooks is whether the project should stay as a single large repository with the 118 cookbooks, or whether they should be split up into separate per-cookbook repositories. The overwhelming majority of people that we talked to were in favor of separate repositories because it makes it easier to contribute, and manage for end-users. This has also been requested over the last couple years since the creation of the cookbooks project in the first place. It will require additional work, workflow documentation and supporting code for us to manage, so please bear with us as we get the repositories ready and released.

With that said, here’s the highlights of cookbook releases from the past week or so:

mysql - v1.2.2

nginx - v0.99.2

haproxy - v1.0.4

nagios - v1.0.4

–
Opscode, Inc
Joshua Timberman, Technical Program Manager
IRC, Skype, Twitter, Github: jtimberman

#2

Hi,
For those unaware there is http://github.com/cookbooks … which
currently has one cookbook per repo.
I’m quite happy to make change to how this account is run. However,
[why is there always a qualification :)]

  • this account’s repos are always open to tracking non-opscode cookbooks
  • who is upstream is a function of quality, so the opscode’s repo
    does not have to be upstream, but it generally is for the same
    reason: quality.
  • we don’t ‘drop’ a repo, it can be dormant if its upstream is dormant.
  • I repos upstream can change, but is always tracked by the master branch.
  • contributions are pulled into the qa branch and this is the
    branch each cookbook team has responsibility for, and ideally is the
    branch that people should track, and issue pull requests against.
  • Handling issues, pull requests, and general cookbook housekeeping
    is the responsibility of each cookbook’s ‘team’
  • Up to now team membership has been quite ‘open’, i.e. if you seem
    to know what you are doing with the cookbook (i.e get someone to
    second a pull request of yours), and you know how to make a clean pull
    request, then youre added to the team.
  • We haven’t been swamped by volunteers, so maybe something isn’t yet
    quite right in the above, although I do ack that the auto updates of
    the repos to their upstream has been less frequent than I’d like.
  • Of course team membership can just as easily be revoked.

Having said that [yes another qualification], apart from the first
point, I’m quite open to suggestions about how to proceed.

HTH

On Tue, Dec 6, 2011 at 2:39 AM, Joshua Timberman joshua@opscode.com wrote:

Ohai!

Last week we had the first Opscode Chef Community Summit in Seattle. A hot topic from that related to cookbooks is whether the project should stay as a single large repository with the 118 cookbooks, or whether they should be split up into separate per-cookbook repositories. The overwhelming majority of people that we talked to were in favor of separate repositories because it makes it easier to contribute, and manage for end-users. This has also been requested over the last couple years since the creation of the cookbooks project in the first place. It will require additional work, workflow documentation and supporting code for us to manage, so please bear with us as we get the repositories ready and released.

With that said, here’s the highlights of cookbook releases from the past week or so:

mysql - v1.2.2

nginx - v0.99.2

haproxy - v1.0.4

nagios - v1.0.4

–
Opscode, Inc
Joshua Timberman, Technical Program Manager
IRC, Skype, Twitter, Github: jtimberman

–
πόλλ’ οἶδ ἀλώπηξ, ἀλλ’ ἐχῖνος ἓν μέγα
[The fox knows many things, but the hedgehog knows one big thing.]
Archilochus, Greek poet (c. 680 BC – c. 645 BC)
http://hedgehogshiatus.com

#3

Having said that [yes another qualification], apart from the first
point, I’m quite open to suggestions about how to proceed.

I don’t know anything about chef yet so this may be a bit naive but
would it be possible to deliver chef cookbooks as gems? That way you
can have a bundler gemfile which lists all your cookbooks.

If this is a silly suggestion please excuse my ignorance.

#4

Tim,

If you’re looking for a dependency resolver/fetcher for cookbooks, check
out https://github.com/applicationsonline/librarian.

You can declare the cookbooks you need in your Cheffile, including any
version constraints you require and any non-Opscode (e.g. GitHub) sources,
and Librarian will resolve and fetch those cookbooks as well as any other
cookbooks you need.

Let me know if it matches what you’re looking for.

Cheers,
Jay

On Mon, Dec 5, 2011 at 3:51 PM, Tim Uckun timuckun@gmail.com wrote:

Having said that [yes another qualification], apart from the first
point, I’m quite open to suggestions about how to proceed.

I don’t know anything about chef yet so this may be a bit naive but
would it be possible to deliver chef cookbooks as gems? That way you
can have a bundler gemfile which lists all your cookbooks.

If this is a silly suggestion please excuse my ignorance.

#5

On Tue, Dec 6, 2011 at 7:51 AM, Tim Uckun timuckun@gmail.com wrote:

Having said that [yes another qualification], apart from the first
point, I’m quite open to suggestions about how to proceed.

I don’t know anything about chef yet so this may be a bit naive but
would it be possible to deliver chef cookbooks as gems? That way you
can have a bundler gemfile which lists all your cookbooks.

If this is a silly suggestion please excuse my ignorance.

No, its not silly, and indeed I have tried just that. Bundler, 1.0
and it seems 1.1 too, is effectively broken (from memory see issue
#67) when it comes to using git repos in they way you’d need to with
Chef cookbooks.
I did try patching Bundler and submitting a pull request which was
either too much or too bad, likely both :wink:
Anyway, I’d counsel against anyone trying to make Bundler work - it is
complex and I found its test suite a nightmare and next to no use in
describing behavior. Hopefully things have/will improve, but be
forewarned.
I think opscode have tackled dep resolution in a nice way on the
server side, but that doesn’t address chef-solo use cases. AFAICT
until Bundler can handle git repo’s well, gems are not really a nice
option, but you could maybe shoe-horn them into it.

I think there is a good case that you should not delegate something as
important a production software installation to a dependencu
auto-resolution tool. Notwithstanding all my effort in trying to do
exactly this :slight_smile:
I think pulling a commit hash into your local cookbook and deploying
that is not a terrible way to go, esp given that you’d test the
daylights out of an installation before deploying any software change.
The Bundler et.al sweet spot is: a) deliver a conveneint starting
point and b) rapid development iterations, and even Bundler seems to
ack this with the package/deployment options that freeze everything
when you deploy.
That said I’d prefer if Bundler did ‘just-work’, and I could simply
use Bundler’s deploy/package options… may be in time :slight_smile:

HTH

–
πόλλ’ οἶδ ἀλώπηξ, ἀλλ’ ἐχῖνος ἓν μέγα
[The fox knows many things, but the hedgehog knows one big thing.]
Archilochus, Greek poet (c. 680 BC – c. 645 BC)
http://hedgehogshiatus.com

#6

On Tue, Dec 6, 2011 at 8:04 AM, Jay Feldblum y_feldblum@yahoo.com wrote:

Tim,

If you’re looking for a dependency resolver/fetcher for cookbooks, check
out https://github.com/applicationsonline/librarian.

Looks very interesting, I’m in different phase now, so may take some
time to fully check it out.

Thanks

You can declare the cookbooks you need in your Cheffile, including any
version constraints you require and any non-Opscode (e.g. GitHub) sources,
and Librarian will resolve and fetch those cookbooks as well as any other
cookbooks you need.

Let me know if it matches what you’re looking for.

Cheers,
Jay

On Mon, Dec 5, 2011 at 3:51 PM, Tim Uckun timuckun@gmail.com wrote:

Having said that [yes another qualification], apart from the first
point, I’m quite open to suggestions about how to proceed.

I don’t know anything about chef yet so this may be a bit naive but
would it be possible to deliver chef cookbooks as gems? That way you
can have a bundler gemfile which lists all your cookbooks.

If this is a silly suggestion please excuse my ignorance.

–
πόλλ’ οἶδ ἀλώπηξ, ἀλλ’ ἐχῖνος ἓν μέγα
[The fox knows many things, but the hedgehog knows one big thing.]
Archilochus, Greek poet (c. 680 BC – c. 645 BC)
http://hedgehogshiatus.com

#7

On 5 December 2011 19:40, Hedge Hog hedgehogshiatus@gmail.com wrote:

For those unaware there is http://github.com/cookbooks … which
currently has one cookbook per repo.

I would prefer to see an Opscode implemented approach which ties into
opscode.com under the Community Cookbooks site, and has some of the
’social’ aspects which were discussed in depth during the Summit. Not
to say a 3rd party approach won’t be individually successful and
viable to continue.

-Alex

#8

On Wed, Dec 7, 2011 at 6:35 AM, Alex Howells lists@howells.me wrote:

On 5 December 2011 19:40, Hedge Hog hedgehogshiatus@gmail.com wrote:

For those unaware there is http://github.com/cookbooks … which
currently has one cookbook per repo.

I would prefer to see an Opscode implemented approach which ties into
opscode.com under the Community Cookbooks site, and has some of the
’social’ aspects which were discussed in depth during the Summit. Not
to say a 3rd party approach won’t be individually successful and
viable to continue.

Definitely, this isn’t an either or proposition. I see this account
doing things that complements whatever Opscode does or comes up with,
as well as whatever any other consulting shop creates:

  • Continue to track cookbooks Opscode, or any other upstream that is
    followed, drops. See the recent email outlining Opscode-discontinued
    cookbooks, one of which had a team member step-up earlier.
  • Track cookbooks that Opscode et. al. don’t and won’t

Recall from the earlier email discussion Opscode had to adopt a
monlithic repo (cookbooks) as a mechanism for ensuring compatability
between cookbooks. I thought Bundler would allow individual repos,
and while that won’t work for the foreseeable future, it seems
librarian might work out. So essentially Librarian allows you to even
be independent of github.com/cookbooks :slight_smile:

From what you mention and from Github’s site motto (social-coding) it
really sounds like Opscode is closer to specializing and competing
with Github that with what this account does.
As it stands, this early and with only moderate levels of effort,
there are 190-odd repos tracked in github.com/cookbooks, I’d envision
this growing over time into the 1000’s, judging from the discontinued
cookbooks it seems clear that Opscode envisions the Opscode repo count
being in the 100’s.

Again just what happens at github.com/cookbooks really depends on the
number and activity of team members.

Hope that clarifies

-Alex

–
πόλλ’ οἶδ ἀλώπηξ, ἀλλ’ ἐχῖνος ἓν μέγα
[The fox knows many things, but the hedgehog knows one big thing.]
Archilochus, Greek poet (c. 680 BC – c. 645 BC)
http://hedgehogshiatus.com

#9

Ohai,

On Tue, Dec 6, 2011 at 7:39 PM, Hedge Hog hedgehogshiatus@gmail.com wrote:

Recall from the earlier email discussion Opscode had to adopt a
monlithic repo (cookbooks) as a mechanism for ensuring compatability
between cookbooks.

Let me clarify.

The monolithic repo was used for a couple reasons.

  1. The Community site hadn’t been built yet, though it was planned and
    in progress.

  2. Chef Server didn’t have an API for uploading and downloading cookbooks.

  3. Just about everyone used the “cookbook shadowing” feature of
    cookbook_path for managing multiple cookbook directories.

  4. Cookbook versions didn’t matter and were rather arbitrary. This is
    a subtle but important thing.

  5. It was highly convenient to have a single repository to push and
    pull both for me as the maintainer, and adding new Opscode team
    members to collaboration of the repository.

Fast forward to now, we have the Community site where people can share
versioned releases of their cookbooks. Since cookbooks are "like"
packages, versioning is important. The Community site also has an API
to download specific versions of cookbooks, and will have some major
improvements coming very soon (serious, I saw them earlier today,
you’re going to love it). The Chef Server API itself is quite good -
thousands of people use it every day on Opscode Hosted Chef, and the
same API works on open source Chef for an unknown number of people out
there. The shadowing feature of cookbook_path is deprecated, because
we feel pretty strongly in the released cookbook model.

Finally, it makes sense, following with the idea of released versioned
cookbooks to have separate repositories, from a maintenance
standpoint. While it does take some initial effort, it won’t be bad
long term. There’s over 100 cookbooks in our repository now, but most
of those don’t see regular changes and releases. Also, since it’s git,
we’ll add typical git workflow tasks such as tagging to the release
process.

–
Opscode, Inc
Joshua Timberman, Technical Program Manager
IRC, Skype, Twitter, Github: jtimberman