RE: Cookbook Management for Complex Infrastructure

kkeane · July 20, 2015, 9:13pm

I think there may be some confusion about the relationship between berkshelf and repositories.

First of all, Berkshelf’s recommendation is about git repositories, not about chef repositories. It is also a recommendation, not a requirement.

Kevin Keane

The NetTech

http://www.4nettech.com

Our values: Privacy, Liberty, Justice

See https://www.4nettech.com/corp/the-nettech-values.html

-----Original message-----
From: Erik Ogan erik@change.org
Sent: Monday 20th July 2015 13:48
To: chef@lists.opscode.com
Subject: [chef] Cookbook Management for Complex Infrastructure

Background:
We currently have several hundred nodes managed by Chef 0.10(.4) on a private Chef server. We are using librarian-chef to manage off-the-shelf cookbooks (~30 or so), and have a single repository for the remaining custom cookbooks (>60) [*]

I am looking to modernize this setup. It is likely[**] to be a clean-slate rebuild with Chef 12, pulling in cookbooks (more likely portions of recipes) as needed. This is also a chance to align our processes with the state-of-the-art thinking on how to manage these resources.

Berkshelf is the current, prescribed tool for managing cookbooks. Berkshelf (appears to) require each cookbook in its own repository. This is great for modularity, it forces a separation of concerns, but while I plan to reduce the number of custom, private cookbooks (removing cruft, replacing some with off-the-shelf cookbooks configured with attributes), I still expect to have enough cookbooks that this would consume all of our allotment of private repositories.

In light of that, I have tried to set up a repository with cookbooks as peers inside, but the more hoops I jump through to make that work, the more sure I become that I’m missing something obvious in the way to manage cookbooks. (Especially right now where I cannot test a custom cookbook with chefspec if it depends on another custom/private cookbook).

So, how do larger installations manage their private cookbooks? What have I missed?

-e

[*] I don’t know if that constitutes “large” in the community, but I’m comfortable saying there’s a fair bit of complexity to it.

[**] For reasons that are way out of scope of this message.

–
Erik Ogan
erik@change.org mailto:erik@change.org
415.BE.MESSY

Ranjib · July 20, 2015, 9:27pm

berkshelf does not require one repo per cookbook. We use berks, and a
single repo for all our custom cookbook. Berksfile is standard ruby file,
you can do Dir['./site-cookbook/*'], and add your local entries.

On Mon, Jul 20, 2015 at 2:13 PM, Kevin Keane Subscription <
subscription@kkeane.com> wrote:

I think there may be some confusion about the relationship between
berkshelf and repositories.

First of all, Berkshelf's recommendation is about git repositories, not
about chef repositories. It is also a recommendation, not a requirement.

Kevin Keane

The NetTech

http://www.4nettech.com

Our values: Privacy, Liberty, Justice

See https://www.4nettech.com/corp/the-nettech-values.html

-----Original message-----
From: Erik Ogan erik@change.org
Sent: Monday 20th July 2015 13:48
To: chef@lists.opscode.com
Subject: [chef] Cookbook Management for Complex Infrastructure

Background:
We currently have several hundred nodes managed by Chef 0.10(.4) on a
private Chef server. We are using librarian-chef to manage off-the-shelf
cookbooks (~30 or so), and have a single repository for the remaining
custom cookbooks (>60) [*]

I am looking to modernize this setup. It is likely[**] to be a clean-slate
rebuild with Chef 12, pulling in cookbooks (more likely portions of
recipes) as needed. This is also a chance to align our processes with the
state-of-the-art thinking on how to manage these resources.

Berkshelf is the current, prescribed tool for managing cookbooks.
Berkshelf (appears to) require each cookbook in its own repository. This is
great for modularity, it forces a separation of concerns, but while I plan
to reduce the number of custom, private cookbooks (removing cruft,
replacing some with off-the-shelf cookbooks configured with attributes), I
still expect to have enough cookbooks that this would consume all of our
allotment of private repositories.

In light of that, I have tried to set up a repository with cookbooks as
peers inside, but the more hoops I jump through to make that work, the more
sure I become that I’m missing something obvious in the way to manage
cookbooks. (Especially right now where I cannot test a custom cookbook with
chefspec if it depends on another custom/private cookbook).

So, how do larger installations manage their private cookbooks? What have
I missed?

-e

[*] I don’t know if that constitutes “large” in the community, but I’m
comfortable saying there’s a fair bit of complexity to it.

[**] For reasons that are way out of scope of this message.

--
Erik Ogan
erik@change.org
415.BE.MESSY

Yoshi_Spendiff · July 20, 2015, 9:47pm

You can sort of have your cake and eat it too buy treating each cookbook as
it's own independent object, but still putting each in the one repository
in their own directory. In your Berksfiles you can specify the rel: keyword
to use a specific directory in the git repo for each cookbook if you need
to.

Depends on if you want to go for the 'monolithic' or isolated cookbook
method. Being able to test each cookbook in isolation is certainly an
advantage.

I think Policyfiles have some sort of dependency mapper/puller as well.

On Mon, Jul 20, 2015 at 2:27 PM, Ranjib Dey dey.ranjib@gmail.com wrote:

berkshelf does not require one repo per cookbook. We use berks, and a
single repo for all our custom cookbook. Berksfile is standard ruby file,
you can do Dir['./site-cookbook/*'], and add your local entries.

On Mon, Jul 20, 2015 at 2:13 PM, Kevin Keane Subscription <
subscription@kkeane.com> wrote:

I think there may be some confusion about the relationship between
berkshelf and repositories.

First of all, Berkshelf's recommendation is about git repositories, not
about chef repositories. It is also a recommendation, not a requirement.

Kevin Keane

The NetTech

http://www.4nettech.com

Our values: Privacy, Liberty, Justice

See https://www.4nettech.com/corp/the-nettech-values.html

-----Original message-----
From: Erik Ogan erik@change.org
Sent: Monday 20th July 2015 13:48
To: chef@lists.opscode.com
Subject: [chef] Cookbook Management for Complex Infrastructure

Background:
We currently have several hundred nodes managed by Chef 0.10(.4) on a
private Chef server. We are using librarian-chef to manage off-the-shelf
cookbooks (~30 or so), and have a single repository for the remaining
custom cookbooks (>60) [*]

I am looking to modernize this setup. It is likely[**] to be a
clean-slate rebuild with Chef 12, pulling in cookbooks (more likely
portions of recipes) as needed. This is also a chance to align our
processes with the state-of-the-art thinking on how to manage these
resources.

Berkshelf is the current, prescribed tool for managing cookbooks.
Berkshelf (appears to) require each cookbook in its own repository. This is
great for modularity, it forces a separation of concerns, but while I plan
to reduce the number of custom, private cookbooks (removing cruft,
replacing some with off-the-shelf cookbooks configured with attributes), I
still expect to have enough cookbooks that this would consume all of our
allotment of private repositories.

In light of that, I have tried to set up a repository with cookbooks as
peers inside, but the more hoops I jump through to make that work, the more
sure I become that I’m missing something obvious in the way to manage
cookbooks. (Especially right now where I cannot test a custom cookbook with
chefspec if it depends on another custom/private cookbook).

So, how do larger installations manage their private cookbooks? What have
I missed?

-e

[*] I don’t know if that constitutes “large” in the community, but I’m
comfortable saying there’s a fair bit of complexity to it.

[**] For reasons that are way out of scope of this message.

--
Erik Ogan
erik@change.org
415.BE.MESSY

--
Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com

wjimenez5271 · July 20, 2015, 11:43pm

There are benefits to both models, but I haven't seen any issues with tools
used in Chef development with either approach, so that shouldn't be a
constraint to the decision. Many notable organizations have multiple
cookbooks per repo (aka monolithic repo), Facebook comes to mind in this
regard.

On Mon, Jul 20, 2015 at 2:47 PM, Yoshi Spendiff <
yoshi.spendiff@indochino.com> wrote:

You can sort of have your cake and eat it too buy treating each cookbook
as it's own independent object, but still putting each in the one
repository in their own directory. In your Berksfiles you can specify the
rel: keyword to use a specific directory in the git repo for each cookbook
if you need to.

Depends on if you want to go for the 'monolithic' or isolated cookbook
method. Being able to test each cookbook in isolation is certainly an
advantage.

I think Policyfiles have some sort of dependency mapper/puller as well.

On Mon, Jul 20, 2015 at 2:27 PM, Ranjib Dey dey.ranjib@gmail.com wrote:

berkshelf does not require one repo per cookbook. We use berks, and a
single repo for all our custom cookbook. Berksfile is standard ruby file,
you can do Dir['./site-cookbook/*'], and add your local entries.

On Mon, Jul 20, 2015 at 2:13 PM, Kevin Keane Subscription <
subscription@kkeane.com> wrote:

I think there may be some confusion about the relationship between
berkshelf and repositories.

First of all, Berkshelf's recommendation is about git repositories,
not about chef repositories. It is also a recommendation, not a requirement.

Kevin Keane

The NetTech

http://www.4nettech.com

Our values: Privacy, Liberty, Justice

See https://www.4nettech.com/corp/the-nettech-values.html

-----Original message-----
From: Erik Ogan erik@change.org
Sent: Monday 20th July 2015 13:48
To: chef@lists.opscode.com
Subject: [chef] Cookbook Management for Complex Infrastructure

Background:
We currently have several hundred nodes managed by Chef 0.10(.4) on a
private Chef server. We are using librarian-chef to manage off-the-shelf
cookbooks (~30 or so), and have a single repository for the remaining
custom cookbooks (>60) [*]

I am looking to modernize this setup. It is likely[**] to be a
clean-slate rebuild with Chef 12, pulling in cookbooks (more likely
portions of recipes) as needed. This is also a chance to align our
processes with the state-of-the-art thinking on how to manage these
resources.

Berkshelf is the current, prescribed tool for managing cookbooks.
Berkshelf (appears to) require each cookbook in its own repository. This is
great for modularity, it forces a separation of concerns, but while I plan
to reduce the number of custom, private cookbooks (removing cruft,
replacing some with off-the-shelf cookbooks configured with attributes), I
still expect to have enough cookbooks that this would consume all of our
allotment of private repositories.

In light of that, I have tried to set up a repository with cookbooks as
peers inside, but the more hoops I jump through to make that work, the more
sure I become that I’m missing something obvious in the way to manage
cookbooks. (Especially right now where I cannot test a custom cookbook with
chefspec if it depends on another custom/private cookbook).

So, how do larger installations manage their private cookbooks? What
have I missed?

-e

[*] I don’t know if that constitutes “large” in the community, but I’m
comfortable saying there’s a fair bit of complexity to it.

[**] For reasons that are way out of scope of this message.

--
Erik Ogan
erik@change.org
415.BE.MESSY

--
Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com

AJ_Christensen2 · July 21, 2015, 12:19am

Here's a Berksfile [0] based on one we use (at Pantheon) which is now the
result of a migration to cookbook-per-repository on a dedicated cookbooks
organization. You can make use of two loops when transitioning. We called
our monolithic infrastructure cookbooks directory site-cookbooks, like a
throwback to the old times!

Hope this helps?

cheers,

--aj

[0] Private git / site-cookbooks monolithic Berksfile · GitHub

On Tue, Jul 21, 2015 at 11:44 AM William Jimenez wjimenez5271@gmail.com
wrote:

There are benefits to both models, but I haven't seen any issues with
tools used in Chef development with either approach, so that shouldn't be a
constraint to the decision. Many notable organizations have multiple
cookbooks per repo (aka monolithic repo), Facebook comes to mind in this
regard.

On Mon, Jul 20, 2015 at 2:47 PM, Yoshi Spendiff <
yoshi.spendiff@indochino.com> wrote:

You can sort of have your cake and eat it too buy treating each cookbook
as it's own independent object, but still putting each in the one
repository in their own directory. In your Berksfiles you can specify the
rel: keyword to use a specific directory in the git repo for each cookbook
if you need to.

Depends on if you want to go for the 'monolithic' or isolated cookbook
method. Being able to test each cookbook in isolation is certainly an
advantage.

I think Policyfiles have some sort of dependency mapper/puller as well.

On Mon, Jul 20, 2015 at 2:27 PM, Ranjib Dey dey.ranjib@gmail.com wrote:

berkshelf does not require one repo per cookbook. We use berks, and a
single repo for all our custom cookbook. Berksfile is standard ruby file,
you can do Dir['./site-cookbook/*'], and add your local entries.

On Mon, Jul 20, 2015 at 2:13 PM, Kevin Keane Subscription <
subscription@kkeane.com> wrote:

I think there may be some confusion about the relationship between
berkshelf and repositories.

First of all, Berkshelf's recommendation is about git repositories,
not about chef repositories. It is also a recommendation, not a requirement.

Kevin Keane

The NetTech

http://www.4nettech.com

Our values: Privacy, Liberty, Justice

See https://www.4nettech.com/corp/the-nettech-values.html

-----Original message-----
From: Erik Ogan erik@change.org
Sent: Monday 20th July 2015 13:48
To: chef@lists.opscode.com
Subject: [chef] Cookbook Management for Complex Infrastructure

Background:
We currently have several hundred nodes managed by Chef 0.10(.4) on a
private Chef server. We are using librarian-chef to manage off-the-shelf
cookbooks (~30 or so), and have a single repository for the remaining
custom cookbooks (>60) [*]

I am looking to modernize this setup. It is likely[**] to be a
clean-slate rebuild with Chef 12, pulling in cookbooks (more likely
portions of recipes) as needed. This is also a chance to align our
processes with the state-of-the-art thinking on how to manage these
resources.

Berkshelf is the current, prescribed tool for managing cookbooks.
Berkshelf (appears to) require each cookbook in its own repository. This is
great for modularity, it forces a separation of concerns, but while I plan
to reduce the number of custom, private cookbooks (removing cruft,
replacing some with off-the-shelf cookbooks configured with attributes), I
still expect to have enough cookbooks that this would consume all of our
allotment of private repositories.

In light of that, I have tried to set up a repository with cookbooks as
peers inside, but the more hoops I jump through to make that work, the more
sure I become that I’m missing something obvious in the way to manage
cookbooks. (Especially right now where I cannot test a custom cookbook with
chefspec if it depends on another custom/private cookbook).

So, how do larger installations manage their private cookbooks? What
have I missed?

-e

[*] I don’t know if that constitutes “large” in the community, but I’m
comfortable saying there’s a fair bit of complexity to it.

[**] For reasons that are way out of scope of this message.

--
Erik Ogan
erik@change.org
415.BE.MESSY

--

Yoshi Spendiff

Ops Engineer
Indochino

Mobile: +1 778 952 2025

Email: yoshi.spendiff@indochino.com

Chris_F · July 21, 2015, 8:05pm

I started with one git repo per cookbook, but then switched to a monolithic repo where I have all my cookbooks in one git repo to support my “one-changeset-one-commit” philosophy. In my case a lot, not all, of my changes spanned multiple cookbooks, especially role cookbooks, so when I do a git commit I want to see “one” changeset.

Chris

From: William Jimenez [mailto:wjimenez5271@gmail.com]
Sent: Monday, July 20, 2015 7:43 PM
To: chef@lists.opscode.com
Subject: [chef] Re: Re: Re: RE: Cookbook Management for Complex Infrastructure

There are benefits to both models, but I haven’t seen any issues with tools used in Chef development with either approach, so that shouldn’t be a constraint to the decision. Many notable organizations have multiple cookbooks per repo (aka monolithic repo), Facebook comes to mind in this regard.

On Mon, Jul 20, 2015 at 2:47 PM, Yoshi Spendiff <yoshi.spendiff@indochino.com mailto:yoshi.spendiff@indochino.com> wrote:
You can sort of have your cake and eat it too buy treating each cookbook as it’s own independent object, but still putting each in the one repository in their own directory. In your Berksfiles you can specify the rel: keyword to use a specific directory in the git repo for each cookbook if you need to.
Depends on if you want to go for the ‘monolithic’ or isolated cookbook method. Being able to test each cookbook in isolation is certainly an advantage.
I think Policyfiles have some sort of dependency mapper/puller as well.

On Mon, Jul 20, 2015 at 2:27 PM, Ranjib Dey <dey.ranjib@gmail.com mailto:dey.ranjib@gmail.com> wrote:
berkshelf does not require one repo per cookbook. We use berks, and a single repo for all our custom cookbook. Berksfile is standard ruby file, you can do Dir[’./site-cookbook/*’], and add your local entries.

On Mon, Jul 20, 2015 at 2:13 PM, Kevin Keane Subscription <subscription@kkeane.com mailto:subscription@kkeane.com> wrote: