Cookbook Management for Complex Infrastructure

Chef Infra (archive)
1

Background:
We currently have several hundred nodes managed by Chef 0.10(.4) on a
private Chef server. We are using librarian-chef to manage off-the-shelf
cookbooks (~30 or so), and have a single repository for the remaining
custom cookbooks (>60) [*]

I am looking to modernize this setup. It is likely[**] to be a clean-slate
rebuild with Chef 12, pulling in cookbooks (more likely portions of
recipes) as needed. This is also a chance to align our processes with the
state-of-the-art thinking on how to manage these resources.

Berkshelf is the current, prescribed tool for managing cookbooks. Berkshelf
(appears to) require each cookbook in its own repository. This is great for
modularity, it forces a separation of concerns, but while I plan to reduce
the number of custom, private cookbooks (removing cruft, replacing some
with off-the-shelf cookbooks configured with attributes), I still expect to
have enough cookbooks that this would consume all of our allotment of
private repositories.

In light of that, I have tried to set up a repository with cookbooks as
peers inside, but the more hoops I jump through to make that work, the more
sure I become that I’m missing something obvious in the way to manage
cookbooks. (Especially right now where I cannot test a custom cookbook with
chefspec if it depends on another custom/private cookbook).

So, how do larger installations manage their private cookbooks? What have I
missed?

-e

[*] I don’t know if that constitutes “large” in the community, but I’m
comfortable saying there’s a fair bit of complexity to it.

[**] For reasons that are way out of scope of this message.

–
Erik Ogan
erik@change.org
415.BE.MESSY

2

While we solved this with separate repositories/projects per cookbook, I don’t see why you can’t store them in a single, monolithic, repo, together.

I would suggest that your path to solve this is to use your Chef Server, combined with Berkshelf and a Berkshelf API Server (which can run alongside the Chef Server), to handle the dependency management.

Here’s an example Berksfile that, in addition to using the community Berkshelf endpoint (i.e., Supermarket), also uses another one:

source 'https://supermarket.chef.io’
source ‘https://berks.compant.com’

metadata

In this way you could have a private cookbook that, using Berks, you upload to your Chef Server. From there, you can then, in another cookbook, have that one as a dependency and, again using Berks, resolve that dependency via your private Berkshelf API.

Thankfully, there is a cookbook to set up the Berkshelf API server: https://supermarket.chef.io/cookbooks/berkshelf-api-server (there are two others, but this is the one we’ve been using successfully, and is also the “official” one that’s part of the Berkshelf API project).

–
Jeff Byrnes
@thejeffbyrnes
Lead DevOps Engineer
EverTrue
704.516.4628

On July 20, 2015 at 4:48:03 PM, Erik Ogan (erik@change.org) wrote:

Background:
We currently have several hundred nodes managed by Chef 0.10(.4) on a private Chef server. We are using librarian-chef to manage off-the-shelf cookbooks (~30 or so), and have a single repository for the remaining custom cookbooks (>60) [*]

I am looking to modernize this setup. It is likely[**] to be a clean-slate rebuild with Chef 12, pulling in cookbooks (more likely portions of recipes) as needed. This is also a chance to align our processes with the state-of-the-art thinking on how to manage these resources.

Berkshelf is the current, prescribed tool for managing cookbooks. Berkshelf (appears to) require each cookbook in its own repository. This is great for modularity, it forces a separation of concerns, but while I plan to reduce the number of custom, private cookbooks (removing cruft, replacing some with off-the-shelf cookbooks configured with attributes), I still expect to have enough cookbooks that this would consume all of our allotment of private repositories.

In light of that, I have tried to set up a repository with cookbooks as peers inside, but the more hoops I jump through to make that work, the more sure I become that I’m missing something obvious in the way to manage cookbooks. (Especially right now where I cannot test a custom cookbook with chefspec if it depends on another custom/private cookbook).

So, how do larger installations manage their private cookbooks? What have I missed?

-e

[*] I don’t know if that constitutes “large” in the community, but I’m comfortable saying there’s a fair bit of complexity to it.

[**] For reasons that are way out of scope of this message.

–
Erik Ogan
erik@change.org
415.BE.MESSY

3

Ugh, for whatever reason, did not see all of the replies to this already. Sorry for the doubling up!

–
Jeff Byrnes
@thejeffbyrnes
Lead DevOps Engineer
EverTrue
704.516.4628

On July 21, 2015 at 4:46:19 PM, Jeff Byrnes (jeff@evertrue.com) wrote:

While we solved this with separate repositories/projects per cookbook, I don’t see why you can’t store them in a single, monolithic, repo, together.

I would suggest that your path to solve this is to use your Chef Server, combined with Berkshelf and a Berkshelf API Server (which can run alongside the Chef Server), to handle the dependency management.

Here’s an example Berksfile that, in addition to using the community Berkshelf endpoint (i.e., Supermarket), also uses another one:

source 'https://supermarket.chef.io’
source ‘https://berks.compant.com’

metadata

In this way you could have a private cookbook that, using Berks, you upload to your Chef Server. From there, you can then, in another cookbook, have that one as a dependency and, again using Berks, resolve that dependency via your private Berkshelf API.

Thankfully, there is a cookbook to set up the Berkshelf API server: https://supermarket.chef.io/cookbooks/berkshelf-api-server (there are two others, but this is the one we’ve been using successfully, and is also the “official” one that’s part of the Berkshelf API project).

–
Jeff Byrnes
@thejeffbyrnes
Lead DevOps Engineer
EverTrue
704.516.4628

On July 20, 2015 at 4:48:03 PM, Erik Ogan (erik@change.org) wrote:

Background:
We currently have several hundred nodes managed by Chef 0.10(.4) on a private Chef server. We are using librarian-chef to manage off-the-shelf cookbooks (~30 or so), and have a single repository for the remaining custom cookbooks (>60) [*]

I am looking to modernize this setup. It is likely[**] to be a clean-slate rebuild with Chef 12, pulling in cookbooks (more likely portions of recipes) as needed. This is also a chance to align our processes with the state-of-the-art thinking on how to manage these resources.

Berkshelf is the current, prescribed tool for managing cookbooks. Berkshelf (appears to) require each cookbook in its own repository. This is great for modularity, it forces a separation of concerns, but while I plan to reduce the number of custom, private cookbooks (removing cruft, replacing some with off-the-shelf cookbooks configured with attributes), I still expect to have enough cookbooks that this would consume all of our allotment of private repositories.

In light of that, I have tried to set up a repository with cookbooks as peers inside, but the more hoops I jump through to make that work, the more sure I become that I’m missing something obvious in the way to manage cookbooks. (Especially right now where I cannot test a custom cookbook with chefspec if it depends on another custom/private cookbook).

So, how do larger installations manage their private cookbooks? What have I missed?

-e

[*] I don’t know if that constitutes “large” in the community, but I’m comfortable saying there’s a fair bit of complexity to it.

[**] For reasons that are way out of scope of this message.

–
Erik Ogan
erik@change.org
415.BE.MESSY

4

On 07/20/2015 01:47 PM, Erik Ogan wrote:

Berkshelf is the current, prescribed tool for managing cookbooks.
Berkshelf (appears to) require each cookbook in its own repository.

As others have posted this is not correct.

Berkshelf was created to make the one-git-repo-per-cookbook model work
really well, and there was a lot of drum beating that went along with
the creating of the tool, but you can also use Berkshelf in at least two
other modes of operation:

1 git repo, Berksfiles-per-cookbook: This setup has cookbooks that each
have individual Berksfiles in them which look like
one-git-repo-per-cookbook-style cookbooks, but all of the
role/wrapper/custom cookbooks are checked into one monolithic git repo.

1 git repo, 1 Berksfile: The setup just has a single monolithic
Berksfile in the cookbooks directory of the monolithic git-repo, used to
pull down all the community cookbooks and lock them all via Berksfile.lock.

5

Yes Berksfile == git repo per cookbook is NOT correct. The tool supports the "rel:" keyword, so I have this lines in my Berksfile

cookbook 'my_coobook1', git: 'git@gitserver.domain.com/monolithic-repo.git', rel: 'cookbooks/my_cookbook1'
cookbook 'my_coobook2', git: 'git@gitserver.domain.com/monolithic-repo.git', rel: 'cookbooks/my_cookbook2'

Chris

-----Original Message-----
From: Lamont Granquist [mailto:lamont@chef.io]
Sent: Monday, July 27, 2015 2:10 PM
To: chef@lists.opscode.com
Subject: [chef] Re: Cookbook Management for Complex Infrastructure

On 07/20/2015 01:47 PM, Erik Ogan wrote:

Berkshelf is the current, prescribed tool for managing cookbooks.
Berkshelf (appears to) require each cookbook in its own repository.

As others have posted this is not correct.

Berkshelf was created to make the one-git-repo-per-cookbook model work really well, and there was a lot of drum beating that went along with the creating of the tool, but you can also use Berkshelf in at least two other modes of operation:

1 git repo, Berksfiles-per-cookbook: This setup has cookbooks that each have individual Berksfiles in them which look like one-git-repo-per-cookbook-style cookbooks, but all of the role/wrapper/custom cookbooks are checked into one monolithic git repo.

1 git repo, 1 Berksfile: The setup just has a single monolithic Berksfile in the cookbooks directory of the monolithic git-repo, used to pull down all the community cookbooks and lock them all via Berksfile.lock.