RE: Single Repo vs Repo per cookbook

Chef Infra (archive)
1

Personally, the almost-forced repo-per-cookbook approach is one thing that bothers me about berkshelf.

Some cookbooks are indeed independent entities, and should have their own repos.

But role cookbooks, and in my case a master cookbook that controls it all, tend to inherently be integrated with each other and should be in the same git repo. A change to one of these cookbooks will often lead to changes in others, and those should be tracked together.

So which cookbook should have their own repos? Generally, cookbooks with a well-defined, stable interface should have their own repos. Can you think of the cookbook’s interface as an API? If not, then it should better be placed in a common repo.

The problem with berkshelf is that you can specify a git repository as source per cookbook, but you can’t really specify a git repository that contains multiple cookbooks.

Kevin Keane

The NetTech

http://www.4nettech.com

Our values: Privacy, Liberty, Justice

See https://www.4nettech.com/corp/the-nettech-values.html

-----Original message-----
From: Yoshi Spendiff yoshi.spendiff@indochino.com
Sent: Monday 13th April 2015 11:49
To: chef@lists.opscode.com
Subject: [chef] Single Repo vs Repo per cookbook

Hi,

I know this horse has been flogged to death but I’ve a slightly different permutation on the question that I’m having trouble working out.
I’m fully on board with having cookbooks be isolated objects with their own dependencies, using Berkshelf and testing cookbooks in isolation, but I’m in a situation here where we’re putting in a new Chef implmenetation and I need to convince people here to use one repo per cookbook with examples of why, otherwise it will be done in one repo. It’s easy to show arguments for why cookbooks should be essentially independent and developed/tested as such but I’m not sure that necessarily means they have to be in their own repository each.

The other way around this that I can see is to simply use directories under the cookbooks repo to do the same thing (not a chef repo, more just like a cookbook specific repo). So you would have a structure like this:
-cookbooks_repo
–company-apache2 (wrapper cookbook)
—Berksfile
–company-php (wrapper cookbook)
—Berksfile
–company-cookbook (role containing cookbook)
—Berksfiles
—recipes\webserver.rb
—metadata.rb (pointing to company-apache2 and company-php)

The Berksfiles for the downstream cookbooks could reference the others by git ref using rel: option for a directory, which is how the separation is achieved. Each cookbook would also have it’s own Vagrantfile for testing and metadata dependencies listed.

I guess the problem I’m having is I don’t really see any major disadvantage of this myself, so can anyone help inform me why I should use one cookbook per repo instead of managing them in directories like this (but still treating them as essentially separate objects otherwise). All I could think of was it means people might work on two cookbooks in a single branch/commit, but I’m not sure that’s necessarily a bad thing.

I should also add that these are all internal cookbooks that are company specific, they would never be shared in the community.

–
Yoshi Spendiff
DevOps Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com mailto:yoshi.spendiff@indochino.com

2

Kevin,
Thats a good point about Berkshelf. I've found that Berkshelf isn't
incompatible with a monolithic repo model, it just requires a little bit of
tooling. I've been using Facebook's grocery delivery tool to manage syncing
chef servers with a single git repo, and with some recent pull requests it
supports resolving dependencies with Berkshelf:
GitHub - facebook/grocery-delivery: The Grocery Delivery utility for managing cookbook uploads to distributed Chef backends..

You mentioned specifying a cookbook contained within a repo as a source.
That is something a better solution is needed for. We can resolve those
cookbooks in some form as they are indexed by berks-api from a chef server,
however that is limited to the last committed version so if you had
multiple cookbook changes not yet committed you wanted to test with that
might be tricky.

-William

On Mon, Apr 13, 2015 at 1:17 PM, Kevin Keane Subscription <
subscription@kkeane.com> wrote:

Personally, the almost-forced repo-per-cookbook approach is one thing
that bothers me about berkshelf.

Some cookbooks are indeed independent entities, and should have their
own repos.

But role cookbooks, and in my case a master cookbook that controls it all,
tend to inherently be integrated with each other and should be in the same
git repo. A change to one of these cookbooks will often lead to changes in
others, and those should be tracked together.

So which cookbook should have their own repos? Generally, cookbooks with
a well-defined, stable interface should have their own repos. Can you think
of the cookbook's interface as an API? If not, then it should better be
placed in a common repo.

The problem with berkshelf is that you can specify a git repository as
source per cookbook, but you can't really specify a git repository that
contains multiple cookbooks.

Kevin Keane

The NetTech

http://www.4nettech.com

Our values: Privacy, Liberty, Justice

See https://www.4nettech.com/corp/the-nettech-values.html

-----Original message-----
From: Yoshi Spendiff yoshi.spendiff@indochino.com
Sent: Monday 13th April 2015 11:49
To: chef@lists.opscode.com
Subject: [chef] Single Repo vs Repo per cookbook

Hi,

I know this horse has been flogged to death but I've a slightly different
permutation on the question that I'm having trouble working out.

I'm fully on board with having cookbooks be isolated objects with their
own dependencies, using Berkshelf and testing cookbooks in isolation, but
I'm in a situation here where we're putting in a new Chef implmenetation
and I need to convince people here to use one repo per cookbook with
examples of why, otherwise it will be done in one repo. It's easy to show
arguments for why cookbooks should be essentially independent and
developed/tested as such but I'm not sure that necessarily means they have
to be in their own repository each.

The other way around this that I can see is to simply use directories
under the cookbooks repo to do the same thing (not a chef repo, more just
like a cookbook specific repo). So you would have a structure like this:
-cookbooks_repo
--company-apache2 (wrapper cookbook)
---Berksfile
--company-php (wrapper cookbook)
---Berksfile
--company-cookbook (role containing cookbook)
---Berksfiles
---recipes\webserver.rb
---metadata.rb (pointing to company-apache2 and company-php)

The Berksfiles for the downstream cookbooks could reference the others by
git ref using rel: option for a directory, which is how the separation is
achieved. Each cookbook would also have it's own Vagrantfile for testing
and metadata dependencies listed.

I guess the problem I'm having is I don't really see any major
disadvantage of this myself, so can anyone help inform me why I should use
one cookbook per repo instead of managing them in directories like this
(but still treating them as essentially separate objects otherwise). All I
could think of was it means people might work on two cookbooks in a single
branch/commit, but I'm not sure that's necessarily a bad thing.

I should also add that these are all internal cookbooks that are company
specific, they would never be shared in the community.

--
Yoshi Spendiff
DevOps Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com

3

"Not even sure whether Berkshelf would handle that gracefully if you have
2 cookbook dependencies using rel: that are in the same repo but checked
out at different commits / tags..."
It definitely works if you just use HEAD/master but I've not tested this
for using the ref/tag/branch options to a Berksfile entry as well. I would
assume it would work but if not it's definitely the the end of the
discussion.

Does anyone know if Berkshelf has any link between versions of cookbooks
(by metadata.rb or a version specified in the Berksfile entry) and tag
names for the repository? As far as I can work out if you want a specific
version of a cookbook using a git source then you use a tag instead of a
version (or both), because only 1 version is available in the tag. What I'm
getting at is that it seems the tag names don't really matter so the
- idea should work in theory. However if you specify a
particular version in your Berksfile plus a git source and it translates
that to a tag name for the source then you have to split it by repo.

On Mon, Apr 13, 2015 at 1:30 PM, William Jimenez wjimenez5271@gmail.com
wrote:

Kevin,
Thats a good point about Berkshelf. I've found that Berkshelf isn't
incompatible with a monolithic repo model, it just requires a little bit of
tooling. I've been using Facebook's grocery delivery tool to manage syncing
chef servers with a single git repo, and with some recent pull requests it
supports resolving dependencies with Berkshelf:
GitHub - facebook/grocery-delivery: The Grocery Delivery utility for managing cookbook uploads to distributed Chef backends..

You mentioned specifying a cookbook contained within a repo as a source.
That is something a better solution is needed for. We can resolve those
cookbooks in some form as they are indexed by berks-api from a chef server,
however that is limited to the last committed version so if you had
multiple cookbook changes not yet committed you wanted to test with that
might be tricky.

-William

On Mon, Apr 13, 2015 at 1:17 PM, Kevin Keane Subscription <
subscription@kkeane.com> wrote:

Personally, the almost-forced repo-per-cookbook approach is one thing
that bothers me about berkshelf.

Some cookbooks are indeed independent entities, and should have their
own repos.

But role cookbooks, and in my case a master cookbook that controls it
all, tend to inherently be integrated with each other and should be in the
same git repo. A change to one of these cookbooks will often lead to
changes in others, and those should be tracked together.

So which cookbook should have their own repos? Generally, cookbooks
with a well-defined, stable interface should have their own repos. Can you
think of the cookbook's interface as an API? If not, then it should better
be placed in a common repo.

The problem with berkshelf is that you can specify a git repository as
source per cookbook, but you can't really specify a git repository that
contains multiple cookbooks.

Kevin Keane

The NetTech

http://www.4nettech.com

Our values: Privacy, Liberty, Justice

See https://www.4nettech.com/corp/the-nettech-values.html

-----Original message-----
From: Yoshi Spendiff yoshi.spendiff@indochino.com
Sent: Monday 13th April 2015 11:49
To: chef@lists.opscode.com
Subject: [chef] Single Repo vs Repo per cookbook

Hi,

I know this horse has been flogged to death but I've a slightly different
permutation on the question that I'm having trouble working out.

I'm fully on board with having cookbooks be isolated objects with their
own dependencies, using Berkshelf and testing cookbooks in isolation, but
I'm in a situation here where we're putting in a new Chef implmenetation
and I need to convince people here to use one repo per cookbook with
examples of why, otherwise it will be done in one repo. It's easy to show
arguments for why cookbooks should be essentially independent and
developed/tested as such but I'm not sure that necessarily means they have
to be in their own repository each.

The other way around this that I can see is to simply use directories
under the cookbooks repo to do the same thing (not a chef repo, more just
like a cookbook specific repo). So you would have a structure like this:
-cookbooks_repo
--company-apache2 (wrapper cookbook)
---Berksfile
--company-php (wrapper cookbook)
---Berksfile
--company-cookbook (role containing cookbook)
---Berksfiles
---recipes\webserver.rb
---metadata.rb (pointing to company-apache2 and company-php)

The Berksfiles for the downstream cookbooks could reference the others by
git ref using rel: option for a directory, which is how the separation is
achieved. Each cookbook would also have it's own Vagrantfile for testing
and metadata dependencies listed.

I guess the problem I'm having is I don't really see any major
disadvantage of this myself, so can anyone help inform me why I should use
one cookbook per repo instead of managing them in directories like this
(but still treating them as essentially separate objects otherwise). All I
could think of was it means people might work on two cookbooks in a single
branch/commit, but I'm not sure that's necessarily a bad thing.

I should also add that these are all internal cookbooks that are company
specific, they would never be shared in the community.

--
Yoshi Spendiff
DevOps Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com

--
Yoshi Spendiff
DevOps Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com

4

On 4/13/15 1:17 PM, Kevin Keane Subscription wrote:

RE: [chef] Single Repo vs Repo per cookbook

The problem with berkshelf is that you can specify a git repository as
source per cookbook, but you can't really specify a git repository
that contains multiple cookbooks.

Isn't that only a problem if you have multiple repos with multiple
cookbooks? If you've got one repo with all your role and wrapper
cookbooks, and then peel off individual library cookbooks which are
one-cookbook-per-repo then it should work okay and you can just use
path: ../some_cookbook in the Berksfile for the internal dependencies
of the monolithic repo, and avoid having the library cookbooks having
deps back into your monolithic repo (which would smell like you're
starting to create circular dependencies).

Berkshelf should work fine for this model as well.

5

Isn't that only a problem if you have multiple repos with multiple
cookbooks? If you've got one repo with all your role and wrapper
cookbooks, and then peel off individual library cookbooks which are
one-cookbook-per-repo then it should work okay and you can just use path: ../some_cookbook in the Berksfile for the internal dependencies of the
monolithic repo, and avoid having the library cookbooks having deps back
into your monolithic repo (which would smell like you're starting to create
circular dependencies).

  • Berkshelf should work fine for this model as well.*
    I think this is a bit of a moot point isn't it? You CAN actually specify a
    git repository that has multiple cookbooks, using the rel: option. You just
    can't do this in a single Berksfile declaration, which is never going to be
    supported.

I was more asking if you SHOULD rather than if you COULD. It's definitely
possible.

On Tue, Apr 14, 2015 at 11:11 AM, Lamont Granquist lamont@chef.io wrote:

On 4/13/15 1:17 PM, Kevin Keane Subscription wrote:

The problem with berkshelf is that you can specify a git repository as
source per cookbook, but you can't really specify a git repository that
contains multiple cookbooks.

Isn't that only a problem if you have multiple repos with multiple
cookbooks? If you've got one repo with all your role and wrapper
cookbooks, and then peel off individual library cookbooks which are
one-cookbook-per-repo then it should work okay and you can just use path: ../some_cookbook in the Berksfile for the internal dependencies of the
monolithic repo, and avoid having the library cookbooks having deps back
into your monolithic repo (which would smell like you're starting to create
circular dependencies).

Berkshelf should work fine for this model as well.

--
Yoshi Spendiff
DevOps Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com

6

I've never needed to point into a multiple-cookbook repo so not familiar
with using the rel: option. I'd still argue that you shouldn't need to
do that, but yeah that renders that question moot.

So, really it sounds like Berkshelf fully supports this method of
arranging cookbooks. Period. Full Stop. You do not need to rearrange
your cookbooks into single git repos to use Berkshelf.

The problem is that Berkshelf authors and users were noisy about the
single git repo model being superior, and that's been conflated with the
tool not being usable if you didn't drink all the kool aid.

On 4/14/15 11:15 AM, Yoshi Spendiff wrote:

/Isn't that only a problem if you have multiple repos with multiple
cookbooks? If you've got one repo with all your role and wrapper
cookbooks, and then peel off individual library cookbooks which are
one-cookbook-per-repo then it should work okay and you can just use
path: ../some_cookbook in the Berksfile for the internal
dependencies of the monolithic repo, and avoid having the library
cookbooks having deps back into your monolithic repo (which would
smell like you're starting to create circular dependencies).
/ /
Berkshelf should work fine for this model as well.

/
I think this is a bit of a moot point isn't it? You CAN actually
specify a git repository that has multiple cookbooks, using the rel:
option. You just can't do this in a single Berksfile declaration,
which is never going to be supported.

I was more asking if you SHOULD rather than if you COULD. It's
definitely possible.

On Tue, Apr 14, 2015 at 11:11 AM, Lamont Granquist <lamont@chef.io
mailto:lamont@chef.io> wrote:

On 4/13/15 1:17 PM, Kevin Keane Subscription wrote:

The problem with berkshelf is that you can specify a git
repository as source per cookbook, but you can't really specify a
git repository that contains multiple cookbooks.

Isn't that only a problem if you have multiple repos with multiple
cookbooks?  If you've got one repo with all your role and wrapper
cookbooks, and then peel off individual library cookbooks which
are one-cookbook-per-repo then it should work okay and you can
just use `path: ../some_cookbook` in the Berksfile for the
internal dependencies of the monolithic repo, and avoid having the
library cookbooks having deps back into your monolithic repo
(which would smell like you're starting to create circular
dependencies).

Berkshelf should work fine for this model as well.

--
Yoshi Spendiff
DevOps Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com mailto:yoshi.spendiff@indochino.com

7

Yeah that's the thought train I was heading down too, which is why I was
wondering if anyone had any experience to the contrary. There's obvious
benefits to having cookbooks as independent objects but that doesn't mean
you need to follow the whole preachy philosophy.

On Tue, Apr 14, 2015 at 11:27 AM, Lamont Granquist lamont@chef.io wrote:

I've never needed to point into a multiple-cookbook repo so not familiar
with using the rel: option. I'd still argue that you shouldn't need to do
that, but yeah that renders that question moot.

So, really it sounds like Berkshelf fully supports this method of
arranging cookbooks. Period. Full Stop. You do not need to rearrange
your cookbooks into single git repos to use Berkshelf.

The problem is that Berkshelf authors and users were noisy about the
single git repo model being superior, and that's been conflated with the
tool not being usable if you didn't drink all the kool aid.

On 4/14/15 11:15 AM, Yoshi Spendiff wrote:

*Isn't that only a problem if you have multiple repos with multiple
cookbooks? If you've got one repo with all your role and wrapper
cookbooks, and then peel off individual library cookbooks which are
one-cookbook-per-repo then it should work okay and you can just use path: ../some_cookbook in the Berksfile for the internal dependencies of the
monolithic repo, and avoid having the library cookbooks having deps back
into your monolithic repo (which would smell like you're starting to create
circular dependencies). *

  • Berkshelf should work fine for this model as well. *
    I think this is a bit of a moot point isn't it? You CAN actually specify a
    git repository that has multiple cookbooks, using the rel: option. You just
    can't do this in a single Berksfile declaration, which is never going to be
    supported.

I was more asking if you SHOULD rather than if you COULD. It's definitely
possible.

On Tue, Apr 14, 2015 at 11:11 AM, Lamont Granquist lamont@chef.io wrote:

On 4/13/15 1:17 PM, Kevin Keane Subscription wrote:

The problem with berkshelf is that you can specify a git repository as
source per cookbook, but you can't really specify a git repository that
contains multiple cookbooks.

Isn't that only a problem if you have multiple repos with multiple
cookbooks? If you've got one repo with all your role and wrapper
cookbooks, and then peel off individual library cookbooks which are
one-cookbook-per-repo then it should work okay and you can just use path: ../some_cookbook in the Berksfile for the internal dependencies of the
monolithic repo, and avoid having the library cookbooks having deps back
into your monolithic repo (which would smell like you're starting to create
circular dependencies).

Berkshelf should work fine for this model as well.

--
Yoshi Spendiff
DevOps Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com

--
Yoshi Spendiff
DevOps Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com

8

As to the "how does berks work with a monolithic repo with multiple
cookbooks", here's a couple of lines from a Berksfile we use that points to
a github repo that has a bunch of cookbooks in it, and are referenced by
paths.

# External cookbooks to <THIS> repo
cookbook 'dd-base', git: 'git@github.com:DataDog/ourmonolithicrepo.git',

branch: 'master', rel: 'site-cookbooks/dd-base'
cookbook 'dd-consul', git: 'git@github.com:DataDog/ourmonolithicrepo.git',
branch: 'prod', rel: 'site-cookbooks/dd-consul'

 # cookbooks in <THIS> repo, relative to the Berksfile
 cookbook 'workspace', path: './site-cookbooks/workspace'

Does this clutter up the local ~/.berkshelf/ cache directory with multiple
copies of the same git repo? Maybe. But who cares? It works. :slight_smile:

It's pretty flexible - so as long as you figure out "when does my team need
to create a new cookbook repo" vs "when does my team need to create a new
cookbook in the existing repo" - that's a workflow problem, and berks will
play nice either way.

-M

On Tue, Apr 14, 2015 at 7:45 PM, Yoshi Spendiff <
yoshi.spendiff@indochino.com> wrote:

Yeah that's the thought train I was heading down too, which is why I was
wondering if anyone had any experience to the contrary. There's obvious
benefits to having cookbooks as independent objects but that doesn't mean
you need to follow the whole preachy philosophy.

On Tue, Apr 14, 2015 at 11:27 AM, Lamont Granquist lamont@chef.io wrote:

I've never needed to point into a multiple-cookbook repo so not familiar
with using the rel: option. I'd still argue that you shouldn't need to do
that, but yeah that renders that question moot.

So, really it sounds like Berkshelf fully supports this method of
arranging cookbooks. Period. Full Stop. You do not need to rearrange
your cookbooks into single git repos to use Berkshelf.

The problem is that Berkshelf authors and users were noisy about the
single git repo model being superior, and that's been conflated with the
tool not being usable if you didn't drink all the kool aid.

On 4/14/15 11:15 AM, Yoshi Spendiff wrote:

*Isn't that only a problem if you have multiple repos with multiple
cookbooks? If you've got one repo with all your role and wrapper
cookbooks, and then peel off individual library cookbooks which are
one-cookbook-per-repo then it should work okay and you can just use path: ../some_cookbook in the Berksfile for the internal dependencies of the
monolithic repo, and avoid having the library cookbooks having deps back
into your monolithic repo (which would smell like you're starting to create
circular dependencies). *

  • Berkshelf should work fine for this model as well. *
    I think this is a bit of a moot point isn't it? You CAN actually specify
    a git repository that has multiple cookbooks, using the rel: option. You
    just can't do this in a single Berksfile declaration, which is never going
    to be supported.

I was more asking if you SHOULD rather than if you COULD. It's
definitely possible.

On Tue, Apr 14, 2015 at 11:11 AM, Lamont Granquist lamont@chef.io
wrote:

On 4/13/15 1:17 PM, Kevin Keane Subscription wrote:

The problem with berkshelf is that you can specify a git repository as
source per cookbook, but you can't really specify a git repository that
contains multiple cookbooks.

Isn't that only a problem if you have multiple repos with multiple
cookbooks? If you've got one repo with all your role and wrapper
cookbooks, and then peel off individual library cookbooks which are
one-cookbook-per-repo then it should work okay and you can just use path: ../some_cookbook in the Berksfile for the internal dependencies of the
monolithic repo, and avoid having the library cookbooks having deps back
into your monolithic repo (which would smell like you're starting to create
circular dependencies).

Berkshelf should work fine for this model as well.

--
Yoshi Spendiff
DevOps Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com

--
Yoshi Spendiff
DevOps Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com

9

Thanks Mike, I didn't know about the path option actually.

Though In my case I would keep a Berksfile in each cookbook still, which is
another way of doing it.

On Tue, Apr 14, 2015 at 3:09 PM, Mike miketheman@gmail.com wrote:

As to the "how does berks work with a monolithic repo with multiple
cookbooks", here's a couple of lines from a Berksfile we use that points to
a github repo that has a bunch of cookbooks in it, and are referenced by
paths.

# External cookbooks to <THIS> repo
cookbook 'dd-base', git: 'git@github.com:DataDog/ourmonolithicrepo.git',

branch: 'master', rel: 'site-cookbooks/dd-base'
cookbook 'dd-consul', git: 'git@github.com:DataDog/ourmonolithicrepo.git',
branch: 'prod', rel: 'site-cookbooks/dd-consul'

 # cookbooks in <THIS> repo, relative to the Berksfile
 cookbook 'workspace', path: './site-cookbooks/workspace'

Does this clutter up the local ~/.berkshelf/ cache directory with multiple
copies of the same git repo? Maybe. But who cares? It works. :slight_smile:

It's pretty flexible - so as long as you figure out "when does my team
need to create a new cookbook repo" vs "when does my team need to create a
new cookbook in the existing repo" - that's a workflow problem, and berks
will play nice either way.

-M

On Tue, Apr 14, 2015 at 7:45 PM, Yoshi Spendiff <
yoshi.spendiff@indochino.com> wrote:

Yeah that's the thought train I was heading down too, which is why I was
wondering if anyone had any experience to the contrary. There's obvious
benefits to having cookbooks as independent objects but that doesn't mean
you need to follow the whole preachy philosophy.

On Tue, Apr 14, 2015 at 11:27 AM, Lamont Granquist lamont@chef.io
wrote:

I've never needed to point into a multiple-cookbook repo so not familiar
with using the rel: option. I'd still argue that you shouldn't need to do
that, but yeah that renders that question moot.

So, really it sounds like Berkshelf fully supports this method of
arranging cookbooks. Period. Full Stop. You do not need to rearrange
your cookbooks into single git repos to use Berkshelf.

The problem is that Berkshelf authors and users were noisy about the
single git repo model being superior, and that's been conflated with the
tool not being usable if you didn't drink all the kool aid.

On 4/14/15 11:15 AM, Yoshi Spendiff wrote:

*Isn't that only a problem if you have multiple repos with multiple
cookbooks? If you've got one repo with all your role and wrapper
cookbooks, and then peel off individual library cookbooks which are
one-cookbook-per-repo then it should work okay and you can just use path: ../some_cookbook in the Berksfile for the internal dependencies of the
monolithic repo, and avoid having the library cookbooks having deps back
into your monolithic repo (which would smell like you're starting to create
circular dependencies). *

  • Berkshelf should work fine for this model as well. *
    I think this is a bit of a moot point isn't it? You CAN actually specify
    a git repository that has multiple cookbooks, using the rel: option. You
    just can't do this in a single Berksfile declaration, which is never going
    to be supported.

I was more asking if you SHOULD rather than if you COULD. It's
definitely possible.

On Tue, Apr 14, 2015 at 11:11 AM, Lamont Granquist lamont@chef.io
wrote:

On 4/13/15 1:17 PM, Kevin Keane Subscription wrote:

The problem with berkshelf is that you can specify a git repository
as source per cookbook, but you can't really specify a git repository that
contains multiple cookbooks.

Isn't that only a problem if you have multiple repos with multiple
cookbooks? If you've got one repo with all your role and wrapper
cookbooks, and then peel off individual library cookbooks which are
one-cookbook-per-repo then it should work okay and you can just use path: ../some_cookbook in the Berksfile for the internal dependencies of the
monolithic repo, and avoid having the library cookbooks having deps back
into your monolithic repo (which would smell like you're starting to create
circular dependencies).

Berkshelf should work fine for this model as well.

--
Yoshi Spendiff
DevOps Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com

--
Yoshi Spendiff
DevOps Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com

--
Yoshi Spendiff
DevOps Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com

10

at PagerDuty, we use a mix of both approaches. Our main infra repo hosts
all cookbooks inside site-cookbooks, we consume a lot of community
cookbooks. Berkshelf assemble all of these for us. Local cookbooks are
assembled via looping over Dir['site-cookbooks/*'] and the declaring path.

Over the time, we have faced several issues including versioning, circular
dependencies, performance and vendoring. But none of them was show stopper.
Above all, though opinionated berkshelf authors never left known bugs open,
or limit the tools usage (\m/). But this also means you should do some
tooling around berks vendoring. i.e. never point any cookbooks upload
machinary against site-cookbooks. All should point to berks vendor, and
keep the vendoring process ephemeral. i.e nuke and re-create the folder
before every cookbook upload process (we do it via CI or chatbots).

My recommendation will be to start with the single repo approach and fan
out as and when need, if they are being consumed by other groups/tools
beyond the initial monolithic repo. That way you can incrementally fine
tune your own versioning policy/convention etc, as and when you encounter
problems. If you break the repo upfront, it will add lot more cognitive
load as u'll switch back and forth between repos to get anything useful.
Which is the case unless you are maintaining community cookbooks. Most of
the internal cookbooks generally makes sense together, not in isolation.
when we write them we think of addressing a specific use case [like ldap
server to back service foo], we dont really start with the requirement of
'lets write a cookbook for ldap which can be re-used across a wide range of
platform'. The re-usable bit involves lot more discovery, and needs lot
more maintenance (you have to think of chef version specific features etc)

hope this help
ranjib

On Tue, Apr 14, 2015 at 3:32 PM, Yoshi Spendiff <
yoshi.spendiff@indochino.com> wrote:

Thanks Mike, I didn't know about the path option actually.

Though In my case I would keep a Berksfile in each cookbook still, which
is another way of doing it.

On Tue, Apr 14, 2015 at 3:09 PM, Mike miketheman@gmail.com wrote:

As to the "how does berks work with a monolithic repo with multiple
cookbooks", here's a couple of lines from a Berksfile we use that points to
a github repo that has a bunch of cookbooks in it, and are referenced by
paths.

# External cookbooks to <THIS> repo
cookbook 'dd-base', git: 'git@github.com:DataDog/ourmonolithicrepo.git',

branch: 'master', rel: 'site-cookbooks/dd-base'
cookbook 'dd-consul', git: 'git@github.com:DataDog/ourmonolithicrepo.git',
branch: 'prod', rel: 'site-cookbooks/dd-consul'

 # cookbooks in <THIS> repo, relative to the Berksfile
 cookbook 'workspace', path: './site-cookbooks/workspace'

Does this clutter up the local ~/.berkshelf/ cache directory with
multiple copies of the same git repo? Maybe. But who cares? It works. :slight_smile:

It's pretty flexible - so as long as you figure out "when does my team
need to create a new cookbook repo" vs "when does my team need to create a
new cookbook in the existing repo" - that's a workflow problem, and berks
will play nice either way.

-M

On Tue, Apr 14, 2015 at 7:45 PM, Yoshi Spendiff <
yoshi.spendiff@indochino.com> wrote:

Yeah that's the thought train I was heading down too, which is why I was
wondering if anyone had any experience to the contrary. There's obvious
benefits to having cookbooks as independent objects but that doesn't mean
you need to follow the whole preachy philosophy.

On Tue, Apr 14, 2015 at 11:27 AM, Lamont Granquist lamont@chef.io
wrote:

I've never needed to point into a multiple-cookbook repo so not
familiar with using the rel: option. I'd still argue that you shouldn't
need to do that, but yeah that renders that question moot.

So, really it sounds like Berkshelf fully supports this method of
arranging cookbooks. Period. Full Stop. You do not need to rearrange
your cookbooks into single git repos to use Berkshelf.

The problem is that Berkshelf authors and users were noisy about the
single git repo model being superior, and that's been conflated with the
tool not being usable if you didn't drink all the kool aid.

On 4/14/15 11:15 AM, Yoshi Spendiff wrote:

*Isn't that only a problem if you have multiple repos with multiple
cookbooks? If you've got one repo with all your role and wrapper
cookbooks, and then peel off individual library cookbooks which are
one-cookbook-per-repo then it should work okay and you can just use path: ../some_cookbook in the Berksfile for the internal dependencies of the
monolithic repo, and avoid having the library cookbooks having deps back
into your monolithic repo (which would smell like you're starting to create
circular dependencies). *

  • Berkshelf should work fine for this model as well. *
    I think this is a bit of a moot point isn't it? You CAN actually
    specify a git repository that has multiple cookbooks, using the rel:
    option. You just can't do this in a single Berksfile declaration, which is
    never going to be supported.

I was more asking if you SHOULD rather than if you COULD. It's
definitely possible.

On Tue, Apr 14, 2015 at 11:11 AM, Lamont Granquist lamont@chef.io
wrote:

On 4/13/15 1:17 PM, Kevin Keane Subscription wrote:

The problem with berkshelf is that you can specify a git repository
as source per cookbook, but you can't really specify a git repository that
contains multiple cookbooks.

Isn't that only a problem if you have multiple repos with multiple
cookbooks? If you've got one repo with all your role and wrapper
cookbooks, and then peel off individual library cookbooks which are
one-cookbook-per-repo then it should work okay and you can just use path: ../some_cookbook in the Berksfile for the internal dependencies of the
monolithic repo, and avoid having the library cookbooks having deps back
into your monolithic repo (which would smell like you're starting to create
circular dependencies).

Berkshelf should work fine for this model as well.

--
Yoshi Spendiff
DevOps Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com

--
Yoshi Spendiff
DevOps Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com

--
Yoshi Spendiff
DevOps Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com

11

I asked a similar question in the Berkshelf issues section of github which
had a couple of interesting answers:

On Tue, Apr 14, 2015 at 4:16 PM, Ranjib Dey dey.ranjib@gmail.com wrote:

at PagerDuty, we use a mix of both approaches. Our main infra repo hosts
all cookbooks inside site-cookbooks, we consume a lot of community
cookbooks. Berkshelf assemble all of these for us. Local cookbooks are
assembled via looping over Dir['site-cookbooks/*'] and the declaring path.

Over the time, we have faced several issues including versioning, circular
dependencies, performance and vendoring. But none of them was show stopper.
Above all, though opinionated berkshelf authors never left known bugs open,
or limit the tools usage (\m/). But this also means you should do some
tooling around berks vendoring. i.e. never point any cookbooks upload
machinary against site-cookbooks. All should point to berks vendor, and
keep the vendoring process ephemeral. i.e nuke and re-create the folder
before every cookbook upload process (we do it via CI or chatbots).

My recommendation will be to start with the single repo approach and fan
out as and when need, if they are being consumed by other groups/tools
beyond the initial monolithic repo. That way you can incrementally fine
tune your own versioning policy/convention etc, as and when you encounter
problems. If you break the repo upfront, it will add lot more cognitive
load as u'll switch back and forth between repos to get anything useful.
Which is the case unless you are maintaining community cookbooks. Most of
the internal cookbooks generally makes sense together, not in isolation.
when we write them we think of addressing a specific use case [like ldap
server to back service foo], we dont really start with the requirement of
'lets write a cookbook for ldap which can be re-used across a wide range of
platform'. The re-usable bit involves lot more discovery, and needs lot
more maintenance (you have to think of chef version specific features etc)

hope this help
ranjib

On Tue, Apr 14, 2015 at 3:32 PM, Yoshi Spendiff <
yoshi.spendiff@indochino.com> wrote:

Thanks Mike, I didn't know about the path option actually.

Though In my case I would keep a Berksfile in each cookbook still, which
is another way of doing it.

On Tue, Apr 14, 2015 at 3:09 PM, Mike miketheman@gmail.com wrote:

As to the "how does berks work with a monolithic repo with multiple
cookbooks", here's a couple of lines from a Berksfile we use that points to
a github repo that has a bunch of cookbooks in it, and are referenced by
paths.

# External cookbooks to <THIS> repo
cookbook 'dd-base', git: 'git@github.com:DataDog/ourmonolithicrepo.git',

branch: 'master', rel: 'site-cookbooks/dd-base'
cookbook 'dd-consul', git: 'git@github.com:DataDog/ourmonolithicrepo.git',
branch: 'prod', rel: 'site-cookbooks/dd-consul'

 # cookbooks in <THIS> repo, relative to the Berksfile
 cookbook 'workspace', path: './site-cookbooks/workspace'

Does this clutter up the local ~/.berkshelf/ cache directory with
multiple copies of the same git repo? Maybe. But who cares? It works. :slight_smile:

It's pretty flexible - so as long as you figure out "when does my team
need to create a new cookbook repo" vs "when does my team need to create a
new cookbook in the existing repo" - that's a workflow problem, and berks
will play nice either way.

-M

On Tue, Apr 14, 2015 at 7:45 PM, Yoshi Spendiff <
yoshi.spendiff@indochino.com> wrote:

Yeah that's the thought train I was heading down too, which is why I
was wondering if anyone had any experience to the contrary. There's obvious
benefits to having cookbooks as independent objects but that doesn't mean
you need to follow the whole preachy philosophy.

On Tue, Apr 14, 2015 at 11:27 AM, Lamont Granquist lamont@chef.io
wrote:

I've never needed to point into a multiple-cookbook repo so not
familiar with using the rel: option. I'd still argue that you shouldn't
need to do that, but yeah that renders that question moot.

So, really it sounds like Berkshelf fully supports this method of
arranging cookbooks. Period. Full Stop. You do not need to rearrange
your cookbooks into single git repos to use Berkshelf.

The problem is that Berkshelf authors and users were noisy about the
single git repo model being superior, and that's been conflated with the
tool not being usable if you didn't drink all the kool aid.

On 4/14/15 11:15 AM, Yoshi Spendiff wrote:

*Isn't that only a problem if you have multiple repos with multiple
cookbooks? If you've got one repo with all your role and wrapper
cookbooks, and then peel off individual library cookbooks which are
one-cookbook-per-repo then it should work okay and you can just use path: ../some_cookbook in the Berksfile for the internal dependencies of the
monolithic repo, and avoid having the library cookbooks having deps back
into your monolithic repo (which would smell like you're starting to create
circular dependencies). *

  • Berkshelf should work fine for this model as well. *
    I think this is a bit of a moot point isn't it? You CAN actually
    specify a git repository that has multiple cookbooks, using the rel:
    option. You just can't do this in a single Berksfile declaration, which is
    never going to be supported.

I was more asking if you SHOULD rather than if you COULD. It's
definitely possible.

On Tue, Apr 14, 2015 at 11:11 AM, Lamont Granquist lamont@chef.io
wrote:

On 4/13/15 1:17 PM, Kevin Keane Subscription wrote:

The problem with berkshelf is that you can specify a git repository
as source per cookbook, but you can't really specify a git repository that
contains multiple cookbooks.

Isn't that only a problem if you have multiple repos with multiple
cookbooks? If you've got one repo with all your role and wrapper
cookbooks, and then peel off individual library cookbooks which are
one-cookbook-per-repo then it should work okay and you can just use path: ../some_cookbook in the Berksfile for the internal dependencies of the
monolithic repo, and avoid having the library cookbooks having deps back
into your monolithic repo (which would smell like you're starting to create
circular dependencies).

Berkshelf should work fine for this model as well.

--
Yoshi Spendiff
DevOps Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com

--
Yoshi Spendiff
DevOps Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com

--
Yoshi Spendiff
DevOps Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com

--
Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com