Where do YOU put your environment and role files?

Chef Infra (archive)
1

When I started working with Chef, I adopted what the guy who was doing this
at my company was doing, which was to work with a traditional "Chef repo"
checked into GitHub as one monolithic thing, containing all the cookbooks,
environments, roles, data bags, etc. But he was doing a lot of things very
wrong. I’m also the only person working with Chef here, so I can make
changes without really bothering anyone much.

I understand the modern recommendation is to keep your cookbooks in
individual git repos, and that’s nice. I do most of my development using
Vagrant and a Chef provisioner, and as you probably know, you can set the
path to your environments or roles in your Vagrantfile.

It bears mentioning that I typically only have a single cookbook in my
nodes’ run lists, and that single cookbook uses depends in metadata.rb and
include_recipe in recipes liberally. I may or may not use environments, and
rarely use roles, but sometimes they have a place in what I’m doing.

I could just gut the old Chef repo, removing all the cookbooks and leaving
only the environments, roles and data bags, then point my Vagrantfiles to
the same paths for those types of resources every time, but most of the
time, the environments or roles make no sense except when considered in the
context of a given cookbook.

It would make sense (to me at least) to keep them with the cookbook they’re
meant to be used with, and to put them in the same git repo as the
cookbook, but then when I use Berkshelf to create a cookbook, it creates
the standard directory structure and it feels weird to me to add a
./environments and a ./roles under my cookbook’s directory structure,
alongside the ./attributes, ./recipes and so forth.

So does it make more sense to keep the environments and roles in the same
git repo as a cookbook, or continue to keep all environments and roles in a
central repo, like so:

chef-repo
|_ cookbooks
|_ cookbook1
|_ individual git repo
|_ Vagrantfile pointed at …/…/ for environments and roles
|_ cookbook2
|_ individual git repo
|_ Vagrantfile pointed at …/…/ for environments and roles
|_ etc.
|_ environments
|_ all environments ever written
|_ roles
|_ all roles ever written
|_ .gitignore to ignore contents of ./cookbooks

What about data bags? Keep them directly under the repo, or with individual
cookbooks? Or both, depending on what makes the most sense?

I know this isn’t really a black or white issue, so really I’m just asking
for some opinions.

2

We have a single git repo that contains all of our data bags, environments
and roles. It is managed just like our cookbooks.

Kent

On Mon, Mar 9, 2015 at 1:21 PM, Fabien Delpierre <fabien.delpierre@gmail.com

wrote:

When I started working with Chef, I adopted what the guy who was doing
this at my company was doing, which was to work with a traditional "Chef
repo" checked into GitHub as one monolithic thing, containing all the
cookbooks, environments, roles, data bags, etc. But he was doing a lot of
things very wrong. I'm also the only person working with Chef here, so I
can make changes without really bothering anyone much.

I understand the modern recommendation is to keep your cookbooks in
individual git repos, and that's nice. I do most of my development using
Vagrant and a Chef provisioner, and as you probably know, you can set the
path to your environments or roles in your Vagrantfile.

It bears mentioning that I typically only have a single cookbook in my
nodes' run lists, and that single cookbook uses depends in metadata.rb
and include_recipe in recipes liberally. I may or may not use
environments, and rarely use roles, but sometimes they have a place in what
I'm doing.

I could just gut the old Chef repo, removing all the cookbooks and leaving
only the environments, roles and data bags, then point my Vagrantfiles to
the same paths for those types of resources every time, but most of the
time, the environments or roles make no sense except when considered in the
context of a given cookbook.

It would make sense (to me at least) to keep them with the cookbook
they're meant to be used with, and to put them in the same git repo as the
cookbook, but then when I use Berkshelf to create a cookbook, it creates
the standard directory structure and it feels weird to me to add a
./environments and a ./roles under my cookbook's directory structure,
alongside the ./attributes, ./recipes and so forth.

So does it make more sense to keep the environments and roles in the same
git repo as a cookbook, or continue to keep all environments and roles in a
central repo, like so:

chef-repo
|_ cookbooks
|_ cookbook1
|_ individual git repo
|_ Vagrantfile pointed at ../../ for environments and roles
|_ cookbook2
|_ individual git repo
|_ Vagrantfile pointed at ../../ for environments and roles
|_ etc.
|_ environments
|_ all environments ever written
|_ roles
|_ all roles ever written
|_ .gitignore to ignore contents of ./cookbooks

What about data bags? Keep them directly under the repo, or with
individual cookbooks? Or both, depending on what makes the most sense?

I know this isn't really a black or white issue, so really I'm just asking
for some opinions.

3

Personally, each individual cookbook manage their own roles in
cookbook1/roles dir.
I update roles with spiceweasel at upload time, so it keeps them
synchronized with the (latest uploaded) repo version.

Environments are pretty much 'empty box' and are stored in a specific
top-level "org" repo (still uploaded with spiceweasel)

On Mon, Mar 9, 2015 at 7:28 PM, Kent Perrier kent.perrier@gmail.com wrote:

We have a single git repo that contains all of our data bags, environments
and roles. It is managed just like our cookbooks.

Kent

On Mon, Mar 9, 2015 at 1:21 PM, Fabien Delpierre <
fabien.delpierre@gmail.com> wrote:

When I started working with Chef, I adopted what the guy who was doing
this at my company was doing, which was to work with a traditional "Chef
repo" checked into GitHub as one monolithic thing, containing all the
cookbooks, environments, roles, data bags, etc. But he was doing a lot of
things very wrong. I'm also the only person working with Chef here, so I
can make changes without really bothering anyone much.

I understand the modern recommendation is to keep your cookbooks in
individual git repos, and that's nice. I do most of my development using
Vagrant and a Chef provisioner, and as you probably know, you can set the
path to your environments or roles in your Vagrantfile.

It bears mentioning that I typically only have a single cookbook in my
nodes' run lists, and that single cookbook uses depends in metadata.rb
and include_recipe in recipes liberally. I may or may not use
environments, and rarely use roles, but sometimes they have a place in what
I'm doing.

I could just gut the old Chef repo, removing all the cookbooks and
leaving only the environments, roles and data bags, then point my
Vagrantfiles to the same paths for those types of resources every time, but
most of the time, the environments or roles make no sense except when
considered in the context of a given cookbook.

It would make sense (to me at least) to keep them with the cookbook
they're meant to be used with, and to put them in the same git repo as the
cookbook, but then when I use Berkshelf to create a cookbook, it creates
the standard directory structure and it feels weird to me to add a
./environments and a ./roles under my cookbook's directory structure,
alongside the ./attributes, ./recipes and so forth.

So does it make more sense to keep the environments and roles in the same
git repo as a cookbook, or continue to keep all environments and roles in a
central repo, like so:

chef-repo
|_ cookbooks
|_ cookbook1
|_ individual git repo
|_ Vagrantfile pointed at ../../ for environments and roles
|_ cookbook2
|_ individual git repo
|_ Vagrantfile pointed at ../../ for environments and roles
|_ etc.
|_ environments
|_ all environments ever written
|_ roles
|_ all roles ever written
|_ .gitignore to ignore contents of ./cookbooks

What about data bags? Keep them directly under the repo, or with
individual cookbooks? Or both, depending on what makes the most sense?

I know this isn't really a black or white issue, so really I'm just
asking for some opinions.

--
Barthélemy Vessemont - bvessemont@gmail.com
Ingénieur en informatique diplômé de l'UTC (Compiègne)

4

On 2015-03-09 14:21, Fabien Delpierre wrote:

[...]
I could just gut the old Chef repo, removing all the cookbooks and leaving
only the environments, roles and data bags, then point my Vagrantfiles to
the same paths for those types of resources every time, but most of the
time, the environments or roles make no sense except when considered in the
context of a given cookbook.

It would make sense (to me at least) to keep them with the cookbook they're
meant to be used with, and to put them in the same git repo as the
cookbook, but then when I use Berkshelf to create a cookbook, it creates
the standard directory structure and it feels weird to me to add a
./environments and a ./roles under my cookbook's directory structure,
alongside the ./attributes, ./recipes and so forth.

So does it make more sense to keep the environments and roles in the same
git repo as a cookbook, or continue to keep all environments and roles in a
central repo, like so:

chef-repo
|_ cookbooks
|_ cookbook1
|_ individual git repo
|_ Vagrantfile pointed at ../../ for environments and roles
|_ cookbook2
|_ individual git repo
|_ Vagrantfile pointed at ../../ for environments and roles
|_ etc.
|_ environments
|_ all environments ever written
|_ roles
|_ all roles ever written
|_ .gitignore to ignore contents of ./cookbooks

What about data bags? Keep them directly under the repo, or with individual
cookbooks? Or both, depending on what makes the most sense?

In our case, we are using a provisioning cookbook that uses the
chef-provisioning gem and we put the environments and the data bags inside this
cookbook recipes. Something like this:

myapp_provisioning::environments
myapp_provisioning::data_bags
myapp_provisioning::roles
...

You could put them in the same cookbook, in a recipe called something like
myapp::provisioning. But that could be confusing, because those recipes are
used differently.

The downside is you can not include them with vagrant so easily :frowning:

https://docs.chef.io/provisioning.html#chef-environment

--
Xabier de Zuazo

5

Over here, we are deploying micro services to environments. Our layout is
that chef_repo is a sibling to all of our application cookbooks ( and
cookbooks dir in chef_repo is empty ) This allows us to have both shared
and application specific attributes in the environment file and deploy apps
configured correctly for each of our services.

On Mon, Mar 9, 2015 at 12:18 PM, Xabier de Zuazo xabier@onddo.com wrote:

On 2015-03-09 14:21, Fabien Delpierre wrote:

[...]
I could just gut the old Chef repo, removing all the cookbooks and
leaving
only the environments, roles and data bags, then point my Vagrantfiles to
the same paths for those types of resources every time, but most of the
time, the environments or roles make no sense except when considered in
the
context of a given cookbook.

It would make sense (to me at least) to keep them with the cookbook
they're
meant to be used with, and to put them in the same git repo as the
cookbook, but then when I use Berkshelf to create a cookbook, it creates
the standard directory structure and it feels weird to me to add a
./environments and a ./roles under my cookbook's directory structure,
alongside the ./attributes, ./recipes and so forth.

So does it make more sense to keep the environments and roles in the same
git repo as a cookbook, or continue to keep all environments and roles
in a
central repo, like so:

chef-repo
|_ cookbooks
|_ cookbook1
|_ individual git repo
|_ Vagrantfile pointed at ../../ for environments and roles
|_ cookbook2
|_ individual git repo
|_ Vagrantfile pointed at ../../ for environments and roles
|_ etc.
|_ environments
|_ all environments ever written
|_ roles
|_ all roles ever written
|_ .gitignore to ignore contents of ./cookbooks

What about data bags? Keep them directly under the repo, or with
individual
cookbooks? Or both, depending on what makes the most sense?

In our case, we are using a provisioning cookbook that uses the
chef-provisioning gem and we put the environments and the data bags inside
this
cookbook recipes. Something like this:

myapp_provisioning::environments
myapp_provisioning::data_bags
myapp_provisioning::roles
...

You could put them in the same cookbook, in a recipe called something like
myapp::provisioning. But that could be confusing, because those recipes
are
used differently.

The downside is you can not include them with vagrant so easily :frowning:

https://docs.chef.io/provisioning.html#chef-environment

--
Xabier de Zuazo

6

Personally I make a clear cut between "cookbook repositories" and
"infrastructure repositories": the former ones contain the reusable
and versioned cookbooks, while the latter ones represent my
infrastructure with its environments (dev -> test -> staging -> prod
etc). Typically I have one "infrastructure repo" per project /
application, and the structure resembles that of a chef repo quite
much, except that it does not contain any cookbooks. Any cookbooks
needed come in via dependency management (berks).

Here is an example using Vagrant as the "infrastructure DSL" (have not
looked into chef-provisioning yet):

Cheers, Torben

P.S.: might be worth noting that I don't use berks apply to lock the
cookbook dependency graph via environments. Instead I lock the graph
in the "top-level" cookbook's metadata. This lets you use environments
in a more natural way, but it also has its drawbacks and there are
several discussions and opinions on how to do it, including the new
Policyfile feature as yet another alternative (which I have not tried
yet).

On Mon, Mar 9, 2015 at 8:31 PM, Michael Lindsay mlindsay@metrodigi.com wrote:

Over here, we are deploying micro services to environments. Our layout is
that chef_repo is a sibling to all of our application cookbooks ( and
cookbooks dir in chef_repo is empty ) This allows us to have both shared
and application specific attributes in the environment file and deploy apps
configured correctly for each of our services.

On Mon, Mar 9, 2015 at 12:18 PM, Xabier de Zuazo xabier@onddo.com wrote:

On 2015-03-09 14:21, Fabien Delpierre wrote:

[...]
I could just gut the old Chef repo, removing all the cookbooks and
leaving
only the environments, roles and data bags, then point my Vagrantfiles
to
the same paths for those types of resources every time, but most of the
time, the environments or roles make no sense except when considered in
the
context of a given cookbook.

It would make sense (to me at least) to keep them with the cookbook
they're
meant to be used with, and to put them in the same git repo as the
cookbook, but then when I use Berkshelf to create a cookbook, it creates
the standard directory structure and it feels weird to me to add a
./environments and a ./roles under my cookbook's directory structure,
alongside the ./attributes, ./recipes and so forth.

So does it make more sense to keep the environments and roles in the
same
git repo as a cookbook, or continue to keep all environments and roles
in a
central repo, like so:

chef-repo
|_ cookbooks
|_ cookbook1
|_ individual git repo
|_ Vagrantfile pointed at ../../ for environments and roles
|_ cookbook2
|_ individual git repo
|_ Vagrantfile pointed at ../../ for environments and roles
|_ etc.
|_ environments
|_ all environments ever written
|_ roles
|_ all roles ever written
|_ .gitignore to ignore contents of ./cookbooks

What about data bags? Keep them directly under the repo, or with
individual
cookbooks? Or both, depending on what makes the most sense?

In our case, we are using a provisioning cookbook that uses the
chef-provisioning gem and we put the environments and the data bags inside
this
cookbook recipes. Something like this:

myapp_provisioning::environments
myapp_provisioning::data_bags
myapp_provisioning::roles
...

You could put them in the same cookbook, in a recipe called something like
myapp::provisioning. But that could be confusing, because those recipes
are
used differently.

The downside is you can not include them with vagrant so easily :frowning:

https://docs.chef.io/provisioning.html#chef-environment

--
Xabier de Zuazo

7

Keep in mind that the monolithic cookbook repository is not wrong, and
we’ve committed to supporting it:

You should switch only if you can see where the workflow improves for
you. If not, you’ll buy yourself a lot more management cost (more git
repos, more per-repo files to synchronize across multiple repos) and if
there’s no other benefit, then you’ll spend the costs of switching plus
the additional per-repo maintenance, for no gain. If you want to do CI
and test cookbooks in isolation and you want to break up the monolithic
repo to avoid having engineers stepping on each other toes when doing
edits to the single repo or stuff like that, then it makes sense. If
its just you, it may not make sense.

8

Our experience working with many clients is strongly pro-monolithic, to a
point:

I draw a distinction between general wrapper cookbooks and specific logic
cookbooks. I've seen myriad names for both these ideas, here's the
difference:

General cookbooks: These are typically wrappers that provide a standard,
within your org, means of doing something. If you regularly deployed LAMP
stacks, this might be a cookbook that pulls in Apache, MySQL, and PHP,
setting them up your standard fashion. Similarly, if you had a suite of
security mandated things that apply everywhere, you could have a single
cookbook that delivers the same setup everywhere. These are essentially
"dumb" cookbooks, that do the same thing everywhere.

Each of these make sense in their own repos, because they do not have
interdependencies with other cookbooks you manage. They should be
independently testable, and essentially provide known, org-specific service
installations/configurations. They would generally not deploy any
application code, but rather prepare a system for the application
deployment.

Specific cookbooks: These are cookbooks that tie things together, and
deliver application "aware" configuration logic. Here you might find "role
cookbooks", or other approaches to deliver fairly granular configuration to
your deployments. Another example might be a monitoring cookbook that has
distinct recipes that correspond to different applications or services.

These go in a monolithic repo, because each one has little-to-no meaning
outside of the collective. This repo also holds data bags, roles, and
environments. It is the only repo that uploads directly to the Chef Server.
This keeps your ultimate concerns (the application and service deployment
and configuration) all together, while allowing you to offload as many
general items to cookbooks that are then treated as upstream dependencies.

Michael

--
Michael F. Weinberg | Director of Operations
http://heavywaterops.com | @heavywaterops

On Mon, Mar 9, 2015 at 1:12 PM, Lamont Granquist lamont@chef.io wrote:

Keep in mind that the monolithic cookbook repository is not wrong, and
we've committed to supporting it:

https://github.com/chef/chef-rfc/blob/master/rfc019-chef-workflows.md

You should switch only if you can see where the workflow improves for
you. If not, you'll buy yourself a lot more management cost (more git
repos, more per-repo files to synchronize across multiple repos) and if
there's no other benefit, then you'll spend the costs of switching plus the
additional per-repo maintenance, for no gain. If you want to do CI and test
cookbooks in isolation and you want to break up the monolithic repo to
avoid having engineers stepping on each other toes when doing edits to the
single repo or stuff like that, then it makes sense. If its just you, it
may not make sense.

9

Thanks everyone for sharing your thoughts! I have things to think about.

On Mon, Mar 9, 2015 at 4:29 PM, Michael Weinberg michael@hw-ops.com wrote:

Our experience working with many clients is strongly pro-monolithic, to a
point:

I draw a distinction between general wrapper cookbooks and specific logic
cookbooks. I've seen myriad names for both these ideas, here's the
difference:

General cookbooks: These are typically wrappers that provide a standard,
within your org, means of doing something. If you regularly deployed LAMP
stacks, this might be a cookbook that pulls in Apache, MySQL, and PHP,
setting them up your standard fashion. Similarly, if you had a suite of
security mandated things that apply everywhere, you could have a single
cookbook that delivers the same setup everywhere. These are essentially
"dumb" cookbooks, that do the same thing everywhere.

Each of these make sense in their own repos, because they do not have
interdependencies with other cookbooks you manage. They should be
independently testable, and essentially provide known, org-specific service
installations/configurations. They would generally not deploy any
application code, but rather prepare a system for the application
deployment.

Specific cookbooks: These are cookbooks that tie things together, and
deliver application "aware" configuration logic. Here you might find "role
cookbooks", or other approaches to deliver fairly granular configuration to
your deployments. Another example might be a monitoring cookbook that has
distinct recipes that correspond to different applications or services.

These go in a monolithic repo, because each one has little-to-no meaning
outside of the collective. This repo also holds data bags, roles, and
environments. It is the only repo that uploads directly to the Chef Server.
This keeps your ultimate concerns (the application and service deployment
and configuration) all together, while allowing you to offload as many
general items to cookbooks that are then treated as upstream dependencies.

Michael

--
Michael F. Weinberg | Director of Operations
http://heavywaterops.com | @heavywaterops

On Mon, Mar 9, 2015 at 1:12 PM, Lamont Granquist lamont@chef.io wrote:

Keep in mind that the monolithic cookbook repository is not wrong, and
we've committed to supporting it:

https://github.com/chef/chef-rfc/blob/master/rfc019-chef-workflows.md

You should switch only if you can see where the workflow improves for
you. If not, you'll buy yourself a lot more management cost (more git
repos, more per-repo files to synchronize across multiple repos) and if
there's no other benefit, then you'll spend the costs of switching plus the
additional per-repo maintenance, for no gain. If you want to do CI and test
cookbooks in isolation and you want to break up the monolithic repo to
avoid having engineers stepping on each other toes when doing edits to the
single repo or stuff like that, then it makes sense. If its just you, it
may not make sense.