Environments

Chef Infra (archive)
1

We put every cookbook in its own git repo. Is there any way to put
environment data, related to just that cookbook actually IN the cookbook,
rather than a common environments directory?

I ask this because it seems strange to me that cookbooks are supposed to be
a unit of software, and yet environment data, related to the cookbook has
to be put into a common location. Seems strange. You can get around this
with role attributes by creating wrapper cookbooks, but … for
environments?

Doug

2

you can have a condition based on the chef_environment in your cookbooks
attributes or recipes.

In attribute file:

If chef_environment == 'blarg'
default[:mything][:blah] = 'whee'
end

On Fri, Mar 14, 2014 at 4:11 PM, Douglas Garstang
doug.garstang@gmail.comwrote:

We put every cookbook in its own git repo. Is there any way to put
environment data, related to just that cookbook actually IN the cookbook,
rather than a common environments directory?

I ask this because it seems strange to me that cookbooks are supposed to
be a unit of software, and yet environment data, related to the cookbook
has to be put into a common location. Seems strange. You can get around
this with role attributes by creating wrapper cookbooks, but .... for
environments?

Doug

3

Jesse,

Thanks, but not what I was looking for. :slight_smile:

Doug

On Fri, Mar 14, 2014 at 4:17 PM, Jesse Nelson spheromak@gmail.com wrote:

you can have a condition based on the chef_environment in your cookbooks
attributes or recipes.

In attribute file:

If chef_environment == 'blarg'
default[:mything][:blah] = 'whee'
end

On Fri, Mar 14, 2014 at 4:11 PM, Douglas Garstang <doug.garstang@gmail.com

wrote:

We put every cookbook in its own git repo. Is there any way to put
environment data, related to just that cookbook actually IN the cookbook,
rather than a common environments directory?

I ask this because it seems strange to me that cookbooks are supposed to
be a unit of software, and yet environment data, related to the cookbook
has to be put into a common location. Seems strange. You can get around
this with role attributes by creating wrapper cookbooks, but .... for
environments?

Doug

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

4

Can you explain what you consider to be an environment and what data you’re looking to store in an environment? We have a single environment for production (vs staging environments / development) that’s shared across multiple data centers (physical and EC2 regions) with zero attributes in it. Discovery of services (you brought up db server hostnames and s3 buckets in an earlier email) could be done through chef searches and data bags.

On Friday, March 14, 2014 at 4:30 PM, Douglas Garstang wrote:

Jesse,

Thanks, but not what I was looking for. :slight_smile:

Doug

On Fri, Mar 14, 2014 at 4:17 PM, Jesse Nelson <spheromak@gmail.com (mailto:spheromak@gmail.com)> wrote:

you can have a condition based on the chef_environment in your cookbooks attributes or recipes.

In attribute file:

If chef_environment == 'blarg'
default[:mything][:blah] = 'whee'
end

On Fri, Mar 14, 2014 at 4:11 PM, Douglas Garstang <doug.garstang@gmail.com (mailto:doug.garstang@gmail.com)> wrote:

We put every cookbook in its own git repo. Is there any way to put environment data, related to just that cookbook actually IN the cookbook, rather than a common environments directory?

I ask this because it seems strange to me that cookbooks are supposed to be a unit of software, and yet environment data, related to the cookbook has to be put into a common location. Seems strange. You can get around this with role attributes by creating wrapper cookbooks, but .... for environments?

Doug

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com (mailto:doug.garstang@gmail.com)
Cell: +1-805-340-5627

5

I'd be curious on this as well - what data are you storing in them? Just
cookbook pins? Or environment attributes?

We generate environments dynamically on our CI system and upload them to
the server - so for us they only exist on the chef server and are
immutable. But we only store cookbook versions in there.

-Pete

On Fri, Mar 14, 2014 at 7:35 PM, Daniel Condomitti daniel@condomitti.comwrote:

Can you explain what you consider to be an environment and what data
you're looking to store in an environment? We have a single environment for
production (vs staging environments / development) that's shared across
multiple data centers (physical and EC2 regions) with zero attributes in
it. Discovery of services (you brought up db server hostnames and s3
buckets in an earlier email) could be done through chef searches and data
bags.

On Friday, March 14, 2014 at 4:30 PM, Douglas Garstang wrote:

Jesse,

Thanks, but not what I was looking for. :slight_smile:

Doug

On Fri, Mar 14, 2014 at 4:17 PM, Jesse Nelson spheromak@gmail.com wrote:

you can have a condition based on the chef_environment in your cookbooks
attributes or recipes.

In attribute file:

If chef_environment == 'blarg'
default[:mything][:blah] = 'whee'
end

On Fri, Mar 14, 2014 at 4:11 PM, Douglas Garstang <doug.garstang@gmail.com

wrote:

We put every cookbook in its own git repo. Is there any way to put
environment data, related to just that cookbook actually IN the cookbook,
rather than a common environments directory?

I ask this because it seems strange to me that cookbooks are supposed to
be a unit of software, and yet environment data, related to the cookbook
has to be put into a common location. Seems strange. You can get around
this with role attributes by creating wrapper cookbooks, but .... for
environments?

Doug

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

6

The data is mostly related to the location in which the cookbook runs. I
wouldn't even consider it environmental, as any location could be in any
number of environments (prod, stage etc). The us-west-1 location (i.e.
Amazon zone) could have systems in stage, or prod, or something else.
That's one reason for not wanting to use environments, as we'd then have
prod_us_west_1, stage_us_west_1 and so on.

Currently, most of the data is for locating servers, so yes, that could
theoretically be put into chef search. How do you unit test the cookbook
with vagrant tho? You can run chef zero (supports search, right?), but then
you'd have to bring up an entire cluster just to test the functionality of
a single cookbook.

What's the benefit of putting this data into a data bag? Wouldn't that
suffer exactly the same issues as a role, or environment?

There is other data too that isn't just used for the location of services.
One example is whether or not any given system will use the apt-cache proxy
server in a given location. For various reasons, that isn't going to be
used everywhere, so there's an attribute that says whether or not it will
be used. That can't be looked up with chef search. It really belongs in a
role... or somewhere....

As for DNS, that's great, but then your cookbook is making assumptions
about it's environment, rather than explicitly stating the attributes
somewhere and that seems more fragile to me. As soon as you hit a situation
that falls outside the norms (and that will happen... we are a startup), your
entire model has to be duct taped. You could come up with a mechanism to do
a chef search, and if the attribute doesn't exist there, then fall back to
an attribute from a role, but then your cookbook becomes even more complex,
and every single cookbook has to have a lot of nested conditions to perform
this logic.

Doug.

On Fri, Mar 14, 2014 at 4:38 PM, Pete Cheslock petecheslock@gmail.comwrote:

I'd be curious on this as well - what data are you storing in them? Just
cookbook pins? Or environment attributes?

We generate environments dynamically on our CI system and upload them to
the server - so for us they only exist on the chef server and are
immutable. But we only store cookbook versions in there.

-Pete

On Fri, Mar 14, 2014 at 7:35 PM, Daniel Condomitti daniel@condomitti.comwrote:

Can you explain what you consider to be an environment and what data
you're looking to store in an environment? We have a single environment for
production (vs staging environments / development) that's shared across
multiple data centers (physical and EC2 regions) with zero attributes in
it. Discovery of services (you brought up db server hostnames and s3
buckets in an earlier email) could be done through chef searches and data
bags.

On Friday, March 14, 2014 at 4:30 PM, Douglas Garstang wrote:

Jesse,

Thanks, but not what I was looking for. :slight_smile:

Doug

On Fri, Mar 14, 2014 at 4:17 PM, Jesse Nelson spheromak@gmail.comwrote:

you can have a condition based on the chef_environment in your cookbooks
attributes or recipes.

In attribute file:

If chef_environment == 'blarg'
default[:mything][:blah] = 'whee'
end

On Fri, Mar 14, 2014 at 4:11 PM, Douglas Garstang <
doug.garstang@gmail.com> wrote:

We put every cookbook in its own git repo. Is there any way to put
environment data, related to just that cookbook actually IN the cookbook,
rather than a common environments directory?

I ask this because it seems strange to me that cookbooks are supposed to
be a unit of software, and yet environment data, related to the cookbook
has to be put into a common location. Seems strange. You can get around
this with role attributes by creating wrapper cookbooks, but .... for
environments?

Doug

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

7

I just tried to dumb down my apt wrapper cookbook, use some defaults, make
it a bit simpler, and see how that pans out.

Each location could probably conform to a standard, whereby our internal
apt repo server could be addressed as http://repo.$location.foo.com/repo,
but I can't say with 100% confidence that this would always be the case.
One exception straight away is testing with vagrant. If the repo server
attribute is explicitly set somewhere I can pull it and look it up and
there will never be a corner case. We are back to again to where should I
put this data? If I put it in a 'location' role I have this giant ugly json
hash that's hard to manage.

As for chef search, not all of our infrastructure is close to being in chef
search, least not out apt repo servers. For this to work with chef search,
they'd have to be under chef control.

Ugh.

Doug.

On Sun, Mar 16, 2014 at 11:43 AM, Douglas Garstang
doug.garstang@gmail.comwrote:

The data is mostly related to the location in which the cookbook runs. I
wouldn't even consider it environmental, as any location could be in any
number of environments (prod, stage etc). The us-west-1 location (i.e.
Amazon zone) could have systems in stage, or prod, or something else.
That's one reason for not wanting to use environments, as we'd then have
prod_us_west_1, stage_us_west_1 and so on.

Currently, most of the data is for locating servers, so yes, that could
theoretically be put into chef search. How do you unit test the cookbook
with vagrant tho? You can run chef zero (supports search, right?), but then
you'd have to bring up an entire cluster just to test the functionality of
a single cookbook.

What's the benefit of putting this data into a data bag? Wouldn't that
suffer exactly the same issues as a role, or environment?

There is other data too that isn't just used for the location of services.
One example is whether or not any given system will use the apt-cache proxy
server in a given location. For various reasons, that isn't going to be
used everywhere, so there's an attribute that says whether or not it will
be used. That can't be looked up with chef search. It really belongs in a
role... or somewhere....

As for DNS, that's great, but then your cookbook is making assumptions
about it's environment, rather than explicitly stating the attributes
somewhere and that seems more fragile to me. As soon as you hit a situation
that falls outside the norms (and that will happen... we are a startup), your
entire model has to be duct taped. You could come up with a mechanism to do
a chef search, and if the attribute doesn't exist there, then fall back to
an attribute from a role, but then your cookbook becomes even more complex,
and every single cookbook has to have a lot of nested conditions to perform
this logic.

Doug.

On Fri, Mar 14, 2014 at 4:38 PM, Pete Cheslock petecheslock@gmail.comwrote:

I'd be curious on this as well - what data are you storing in them? Just
cookbook pins? Or environment attributes?

We generate environments dynamically on our CI system and upload them to
the server - so for us they only exist on the chef server and are
immutable. But we only store cookbook versions in there.

-Pete

On Fri, Mar 14, 2014 at 7:35 PM, Daniel Condomitti <daniel@condomitti.com

wrote:

Can you explain what you consider to be an environment and what data
you're looking to store in an environment? We have a single environment for
production (vs staging environments / development) that's shared across
multiple data centers (physical and EC2 regions) with zero attributes in
it. Discovery of services (you brought up db server hostnames and s3
buckets in an earlier email) could be done through chef searches and data
bags.

On Friday, March 14, 2014 at 4:30 PM, Douglas Garstang wrote:

Jesse,

Thanks, but not what I was looking for. :slight_smile:

Doug

On Fri, Mar 14, 2014 at 4:17 PM, Jesse Nelson spheromak@gmail.comwrote:

you can have a condition based on the chef_environment in your cookbooks
attributes or recipes.

In attribute file:

If chef_environment == 'blarg'
default[:mything][:blah] = 'whee'
end

On Fri, Mar 14, 2014 at 4:11 PM, Douglas Garstang <
doug.garstang@gmail.com> wrote:

We put every cookbook in its own git repo. Is there any way to put
environment data, related to just that cookbook actually IN the cookbook,
rather than a common environments directory?

I ask this because it seems strange to me that cookbooks are supposed to
be a unit of software, and yet environment data, related to the cookbook
has to be put into a common location. Seems strange. You can get around
this with role attributes by creating wrapper cookbooks, but .... for
environments?

Doug

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627