Running chef in multiple environments

Chef Infra (archive)
1

Hi,

We’re using chef to build our new hosting environment, which comprises of 3 environments (dev, staging and production), and manage our own chef server. So far we’ve been using chef environments to change variables etc between the 3 environments. However, we’ve come across an issue whereby we need to test a new role in our dev environement.

With cookbooks we can restrict the version that is used between environments, but there doesn’t seem to be anything similar with roles (or data bags, which is another issue).

I’m just wondering, what do people do in this situation? Should we be looking to run a separate chef server in each environment instead of using chef environments?

Thanks
Tom

HBVB trading as Bauer Corporate Services (BCS) is a division of the Bauer Media
Group the largest consumer publisher in the UK, and second largest commercial
radio broadcaster. BCS provides financial services and manages and develops IT
systems on which our UK publishing, broadcast, digital and partner businesses depend.

The information in this email is intended only for the addressee(s) named above.
Access to this email by anyone else is unauthorised. If you are not the intended
recipient of this message any disclosure, copying, distribution or any action
taken in reliance on it is prohibited and may be unlawful. HBVB do not warrant that
any attachments are free from viruses or other defects and accept no liability for
any losses resulting from infected email transmissions.

Please note that any views expressed in this email may be those of the originator
and do not necessarily reflect those of this organisation.

HBVB is registered in England; Registered address is
1 Lincoln Court, Lincoln Road, Peterborough, PE1 2RF.

Registration number 8453545

2

Tom,

We have spent much time with Opscode coming onsite to help train us. We also have similar environments to you. After talking with Opscode instructors it was decided to create a chef server for each environment. For us it works well.

It may work well for you too.

Justin Franks
Lead Operations Engineer
SaaS, Cloud, Data Centers & Infrastructure
Lithium Technologies, Inc
225 Bush St., 15th Floor
San Francisco, CA 94104
tel: +1 415 757 3100 x3219

From: Deprez, Tom tom.deprez@bauerservices.co.uk
Sent: Friday, July 25, 2014 1:56 AM
To: chef@lists.opscode.com
Subject: [chef] Running chef in multiple environments

Hi,

We’re using chef to build our new hosting environment, which comprises of 3 environments (dev, staging and production), and manage our own chef server. So far we’ve been using chef environments to change variables etc between the 3 environments. However, we’ve come across an issue whereby we need to test a new role in our dev environement.

With cookbooks we can restrict the version that is used between environments, but there doesn’t seem to be anything similar with roles (or data bags, which is another issue).

I’m just wondering, what do people do in this situation? Should we be looking to run a separate chef server in each environment instead of using chef environments?

Thanks
Tom

HBVB trading as Bauer Corporate Services (BCS) is a division of the Bauer Media
Group the largest consumer publisher in the UK, and second largest commercial
radio broadcaster. BCS provides financial services and manages and develops IT
systems on which our UK publishing, broadcast, digital and partner businesses depend.

The information in this email is intended only for the addressee(s) named above.
Access to this email by anyone else is unauthorised. If you are not the intended
recipient of this message any disclosure, copying, distribution or any action
taken in reliance on it is prohibited and may be unlawful. HBVB do not warrant that
any attachments are free from viruses or other defects and accept no liability for
any losses resulting from infected email transmissions.

Please note that any views expressed in this email may be those of the originator
and do not necessarily reflect those of this organisation.

HBVB is registered in England; Registered address is
1 Lincoln Court, Lincoln Road, Peterborough, PE1 2RF.

Registration number 8453545

3

Role data is global to all environments... its one of the "dont use
roles" crowd's rallying points.

Set data in environments and consume in recipes if you need to test changes.

-s

On Fri, Jul 25, 2014 at 9:01 AM, Justin Franks
justin.franks@lithium.com wrote:

Tom,

We have spent much time with Opscode coming onsite to help train us. We also
have similar environments to you. After talking with Opscode instructors it
was decided to create a chef server for each environment. For us it works
well.

It may work well for you too.

Justin Franks
Lead Operations Engineer
SaaS, Cloud, Data Centers & Infrastructure
Lithium Technologies, Inc
225 Bush St., 15th Floor
San Francisco, CA 94104
tel: +1 415 757 3100 x3219

From: Deprez, Tom tom.deprez@bauerservices.co.uk
Sent: Friday, July 25, 2014 1:56 AM
To: chef@lists.opscode.com
Subject: [chef] Running chef in multiple environments

Hi,

We’re using chef to build our new hosting environment, which comprises of 3
environments (dev, staging and production), and manage our own chef server.
So far we’ve been using chef environments to change variables etc between
the 3 environments. However, we’ve come across an issue whereby we need to
test a new role in our dev environement.

With cookbooks we can restrict the version that is used between
environments, but there doesn’t seem to be anything similar with roles (or
data bags, which is another issue).

I’m just wondering, what do people do in this situation? Should we be
looking to run a separate chef server in each environment instead of using
chef environments?

Thanks

Tom

HBVB trading as Bauer Corporate Services (BCS) is a division of the Bauer
Media
Group the largest consumer publisher in the UK, and second largest
commercial
radio broadcaster. BCS provides financial services and manages and develops
IT
systems on which our UK publishing, broadcast, digital and partner
businesses depend.

The information in this email is intended only for the addressee(s) named
above.
Access to this email by anyone else is unauthorised. If you are not the
intended
recipient of this message any disclosure, copying, distribution or any
action
taken in reliance on it is prohibited and may be unlawful. HBVB do not
warrant that
any attachments are free from viruses or other defects and accept no
liability for
any losses resulting from infected email transmissions.

Please note that any views expressed in this email may be those of the
originator
and do not necessarily reflect those of this organisation.

HBVB is registered in England; Registered address is
1 Lincoln Court, Lincoln Road, Peterborough, PE1 2RF.

Registration number 8453545

4

Check this post on using role cookbooks:
http://realityforge.org/code/2012/11/19/role-cookbooks-and-wrapper-cookbooks.html

From: Deprez, Tom [mailto:tom.deprez@bauerservices.co.uk]
Sent: Friday, July 25, 2014 4:56 AM
To: chef@lists.opscode.com
Subject: [chef] Running chef in multiple environments

Hi,

We’re using chef to build our new hosting environment, which comprises of 3 environments (dev, staging and production), and manage our own chef server. So far we’ve been using chef environments to change variables etc between the 3 environments. However, we’ve come across an issue whereby we need to test a new role in our dev environement.

With cookbooks we can restrict the version that is used between environments, but there doesn’t seem to be anything similar with roles (or data bags, which is another issue).

I’m just wondering, what do people do in this situation? Should we be looking to run a separate chef server in each environment instead of using chef environments?

Thanks
Tom

HBVB trading as Bauer Corporate Services (BCS) is a division of the Bauer Media
Group the largest consumer publisher in the UK, and second largest commercial
radio broadcaster. BCS provides financial services and manages and develops IT
systems on which our UK publishing, broadcast, digital and partner businesses depend.

The information in this email is intended only for the addressee(s) named above.
Access to this email by anyone else is unauthorised. If you are not the intended
recipient of this message any disclosure, copying, distribution or any action
taken in reliance on it is prohibited and may be unlawful. HBVB do not warrant that
any attachments are free from viruses or other defects and accept no liability for
any losses resulting from infected email transmissions.

Please note that any views expressed in this email may be those of the originator
and do not necessarily reflect those of this organisation.

HBVB is registered in England; Registered address is
1 Lincoln Court, Lincoln Road, Peterborough, PE1 2RF.

Registration number 8453545

5

That's a pretty old post, but I like the spirit.
One thing I'd change is node.default instead of node.override

Try not to think of "kinds of cookbooks"... they're all Just Cookbooks.

-s

On Fri, Jul 25, 2014 at 9:05 AM, Fitzwater, Brian K (CTR)
brian.k.fitzwater@uscis.dhs.gov wrote:

Check this post on using role cookbooks:

Role Cookbooks and Wrapper Cookbooks

From: Deprez, Tom [mailto:tom.deprez@bauerservices.co.uk]

Sent: Friday, July 25, 2014 4:56 AM
To: chef@lists.opscode.com
Subject: [chef] Running chef in multiple environments

Hi,

We’re using chef to build our new hosting environment, which comprises of 3
environments (dev, staging and production), and manage our own chef server.
So far we’ve been using chef environments to change variables etc between
the 3 environments. However, we’ve come across an issue whereby we need to
test a new role in our dev environement.

With cookbooks we can restrict the version that is used between
environments, but there doesn’t seem to be anything similar with roles (or
data bags, which is another issue).

I’m just wondering, what do people do in this situation? Should we be
looking to run a separate chef server in each environment instead of using
chef environments?

Thanks

Tom

HBVB trading as Bauer Corporate Services (BCS) is a division of the Bauer
Media
Group the largest consumer publisher in the UK, and second largest
commercial
radio broadcaster. BCS provides financial services and manages and develops
IT
systems on which our UK publishing, broadcast, digital and partner
businesses depend.

The information in this email is intended only for the addressee(s) named
above.
Access to this email by anyone else is unauthorised. If you are not the
intended
recipient of this message any disclosure, copying, distribution or any
action
taken in reliance on it is prohibited and may be unlawful. HBVB do not
warrant that
any attachments are free from viruses or other defects and accept no
liability for
any losses resulting from infected email transmissions.

Please note that any views expressed in this email may be those of the
originator
and do not necessarily reflect those of this organisation.

HBVB is registered in England; Registered address is
1 Lincoln Court, Lincoln Road, Peterborough, PE1 2RF.

Registration number 8453545

6

Use multiple Chef Servers or adopt "The Berkshelf Way"

--AJ

On Sat, Jul 26, 2014 at 2:04 AM, Sean OMeara someara@opscode.com wrote:

That's a pretty old post, but I like the spirit.
One thing I'd change is node.default instead of node.override

Try not to think of "kinds of cookbooks"... they're all Just Cookbooks.

-s

On Fri, Jul 25, 2014 at 9:05 AM, Fitzwater, Brian K (CTR)
brian.k.fitzwater@uscis.dhs.gov wrote:

Check this post on using role cookbooks:

Role Cookbooks and Wrapper Cookbooks

From: Deprez, Tom [mailto:tom.deprez@bauerservices.co.uk]

Sent: Friday, July 25, 2014 4:56 AM
To: chef@lists.opscode.com
Subject: [chef] Running chef in multiple environments

Hi,

We’re using chef to build our new hosting environment, which comprises of 3
environments (dev, staging and production), and manage our own chef server.
So far we’ve been using chef environments to change variables etc between
the 3 environments. However, we’ve come across an issue whereby we need to
test a new role in our dev environement.

With cookbooks we can restrict the version that is used between
environments, but there doesn’t seem to be anything similar with roles (or
data bags, which is another issue).

I’m just wondering, what do people do in this situation? Should we be
looking to run a separate chef server in each environment instead of using
chef environments?

Thanks

Tom

HBVB trading as Bauer Corporate Services (BCS) is a division of the Bauer
Media
Group the largest consumer publisher in the UK, and second largest
commercial
radio broadcaster. BCS provides financial services and manages and develops
IT
systems on which our UK publishing, broadcast, digital and partner
businesses depend.

The information in this email is intended only for the addressee(s) named
above.
Access to this email by anyone else is unauthorised. If you are not the
intended
recipient of this message any disclosure, copying, distribution or any
action
taken in reliance on it is prohibited and may be unlawful. HBVB do not
warrant that
any attachments are free from viruses or other defects and accept no
liability for
any losses resulting from infected email transmissions.

Please note that any views expressed in this email may be those of the
originator
and do not necessarily reflect those of this organisation.

HBVB is registered in England; Registered address is
1 Lincoln Court, Lincoln Road, Peterborough, PE1 2RF.

Registration number 8453545

7

There are other ways of setting this up - I've seen a successful example of
automated promotion using a few complicated "version pinning" mechanisms,
both via environment and 'wrapper' cookbook. The key thing I noted in each
of the situations where this was successful was a well defined promotion
path for each cookbook as it makes its way through the environments AND
some kind of automation (be it in CI or just scripts or whatever) that
enforce this.

I've seen first-hand an attempt to 'promote' cookbooks between multiple
chef servers and that got ugly really fast (not to say it's not possible).

Personally I've found the concept of a 'wrapper' cookbook that you can
version independent of the version of the cookbook they pull in to be
successful, but your mileage will vary depending on your node
setup/cookbook setup/workflow, etc.

On Fri, Jul 25, 2014 at 4:09 PM, AJ Christensen aj@junglist.io wrote:

Use multiple Chef Servers or adopt "The Berkshelf Way"

--AJ

On Sat, Jul 26, 2014 at 2:04 AM, Sean OMeara someara@opscode.com wrote:

That's a pretty old post, but I like the spirit.
One thing I'd change is node.default instead of node.override

Try not to think of "kinds of cookbooks"... they're all Just Cookbooks.

-s

On Fri, Jul 25, 2014 at 9:05 AM, Fitzwater, Brian K (CTR)
brian.k.fitzwater@uscis.dhs.gov wrote:

Check this post on using role cookbooks:

http://realityforge.org/code/2012/11/19/role-cookbooks-and-wrapper-cookbooks.html

From: Deprez, Tom [mailto:tom.deprez@bauerservices.co.uk]

Sent: Friday, July 25, 2014 4:56 AM
To: chef@lists.opscode.com
Subject: [chef] Running chef in multiple environments

Hi,

We’re using chef to build our new hosting environment, which comprises
of 3
environments (dev, staging and production), and manage our own chef
server.
So far we’ve been using chef environments to change variables etc
between
the 3 environments. However, we’ve come across an issue whereby we need
to
test a new role in our dev environement.

With cookbooks we can restrict the version that is used between
environments, but there doesn’t seem to be anything similar with roles
(or
data bags, which is another issue).

I’m just wondering, what do people do in this situation? Should we be
looking to run a separate chef server in each environment instead of
using
chef environments?

Thanks

Tom

HBVB trading as Bauer Corporate Services (BCS) is a division of the
Bauer
Media
Group the largest consumer publisher in the UK, and second largest
commercial
radio broadcaster. BCS provides financial services and manages and
develops
IT
systems on which our UK publishing, broadcast, digital and partner
businesses depend.

The information in this email is intended only for the addressee(s)
named
above.
Access to this email by anyone else is unauthorised. If you are not the
intended
recipient of this message any disclosure, copying, distribution or any
action
taken in reliance on it is prohibited and may be unlawful. HBVB do not
warrant that
any attachments are free from viruses or other defects and accept no
liability for
any losses resulting from infected email transmissions.

Please note that any views expressed in this email may be those of the
originator
and do not necessarily reflect those of this organisation.

HBVB is registered in England; Registered address is
1 Lincoln Court, Lincoln Road, Peterborough, PE1 2RF.

Registration number 8453545

--

Scott Hain — Software Development Engineer

360.360.7433 – shain@getchef.com – *my: *Linkedin http://www.linkedin.com/
Twitter http://www.twitter.com/elejia

CHEF

GETCHEF.COM http://www.getchef.com/

TM

getchef.com http://www.getchef.com/ Blog
http://www.opscode.com/blog/ Facebook
https://www.facebook.com/getchefdotcom Twitter
https://twitter.com/chef Youtube https://www.youtube.com/getchef

8

​Hi Tom,

same problem here. I think the "Berkshelf Way" and the patterns that Jamie
describe are perfectly valid:
http://blog.vialstudios.com/the-environment-cookbook-pattern/

However I don't like the "environment cookbook" pattern because it uses
environments for pinning cookbook versions, i.e. you can't use them for
'dev' / 'prod' / 'staging' anymore. Instead of environments I pin the
cookbook dependency versions in a "top-level" cookbook's metadata.rb. This
gets me the whole dependency graph pinned and also allows me to use
environments for 'dev' / 'prod' / 'staging'. Within the environments
I only pin
the "top-level" cookbooks:
http://lists.opscode.com/sympa/arc/chef/2014-01/msg00419.html

More discussion here and here:

Cheers,
Torben

On Sat, Jul 26, 2014 at 1:22 AM, Scott Hain shain@getchef.com wrote:

There are other ways of setting this up - I've seen a successful example
of automated promotion using a few complicated "version pinning"
mechanisms, both via environment and 'wrapper' cookbook. The key thing I
noted in each of the situations where this was successful was a well
defined promotion path for each cookbook as it makes its way through the
environments AND some kind of automation (be it in CI or just scripts or
whatever) that enforce this.

I've seen first-hand an attempt to 'promote' cookbooks between multiple
chef servers and that got ugly really fast (not to say it's not possible).

Personally I've found the concept of a 'wrapper' cookbook that you can
version independent of the version of the cookbook they pull in to be
successful, but your mileage will vary depending on your node
setup/cookbook setup/workflow, etc.

On Fri, Jul 25, 2014 at 4:09 PM, AJ Christensen aj@junglist.io wrote:

Use multiple Chef Servers or adopt "The Berkshelf Way"

--AJ

On Sat, Jul 26, 2014 at 2:04 AM, Sean OMeara someara@opscode.com wrote:

That's a pretty old post, but I like the spirit.
One thing I'd change is node.default instead of node.override

Try not to think of "kinds of cookbooks"... they're all Just Cookbooks.

-s

On Fri, Jul 25, 2014 at 9:05 AM, Fitzwater, Brian K (CTR)
brian.k.fitzwater@uscis.dhs.gov wrote:

Check this post on using role cookbooks:

http://realityforge.org/code/2012/11/19/role-cookbooks-and-wrapper-cookbooks.html

From: Deprez, Tom [mailto:tom.deprez@bauerservices.co.uk]

Sent: Friday, July 25, 2014 4:56 AM
To: chef@lists.opscode.com
Subject: [chef] Running chef in multiple environments

Hi,

We’re using chef to build our new hosting environment, which comprises
of 3
environments (dev, staging and production), and manage our own chef
server.
So far we’ve been using chef environments to change variables etc
between
the 3 environments. However, we’ve come across an issue whereby we
need to
test a new role in our dev environement.

With cookbooks we can restrict the version that is used between
environments, but there doesn’t seem to be anything similar with roles
(or
data bags, which is another issue).

I’m just wondering, what do people do in this situation? Should we be
looking to run a separate chef server in each environment instead of
using
chef environments?

Thanks

Tom

HBVB trading as Bauer Corporate Services (BCS) is a division of the
Bauer
Media
Group the largest consumer publisher in the UK, and second largest
commercial
radio broadcaster. BCS provides financial services and manages and
develops
IT
systems on which our UK publishing, broadcast, digital and partner
businesses depend.

The information in this email is intended only for the addressee(s)
named
above.
Access to this email by anyone else is unauthorised. If you are not the
intended
recipient of this message any disclosure, copying, distribution or any
action
taken in reliance on it is prohibited and may be unlawful. HBVB do not
warrant that
any attachments are free from viruses or other defects and accept no
liability for
any losses resulting from infected email transmissions.

Please note that any views expressed in this email may be those of the
originator
and do not necessarily reflect those of this organisation.

HBVB is registered in England; Registered address is
1 Lincoln Court, Lincoln Road, Peterborough, PE1 2RF.

Registration number 8453545

--

Scott Hain — Software Development Engineer

360.360.7433 – shain@getchef.com – *my: *Linkedin
http://www.linkedin.com/ Twitter http://www.twitter.com/elejia

CHEF

GETCHEF.COM http://www.getchef.com/

TM

getchef.com http://www.getchef.com/ Blog
http://www.opscode.com/blog/ Facebook
https://www.facebook.com/getchefdotcom Twitter
https://twitter.com/chef Youtube https://www.youtube.com/getchef

9

It seems there's two common answers to the "role problem":

  • Set up multiple Chef servers
  • Don't use roles

Both are kind of a pain in the ass for various reasons...

Having multiple Chef servers means I can't depend on Chef to seamlessly
provide configuration across environments, where needed (such as relying on
Chef's search function). I also have to ensure all of my Chef servers have
correct versions of Chef artifacts.

Not using roles... Well, I just don't like the idea. Roles are just such a
great way to group things together as well as separate Chef's
functionalities into distinct steps. Roles are also very simple to look at

  • just one file, no versions to increment, low hassle.

So, my point is something my colleague brought up recently - why run
chef-client as a daemon on production runway at all? Why not run it on
demand, such as when changes need to go out to production? From my
experience anyway, our production runways never get any updates when they
are actually serving production traffic, so it's safe to turn chef-client
off to prevent any accidental promotion of cookbooks, environments, or
roles to production.

That way you can keep using just one Chef server and keep using roles.

On Fri, Jul 25, 2014 at 6:01 AM, Justin Franks justin.franks@lithium.com
wrote:

Tom,

We have spent much time with Opscode coming onsite to help train us. We
also have similar environments to you. After talking with Opscode
instructors it was decided to create a chef server for each environment.
For us it works well.

It may work well for you too.

Justin Franks
Lead Operations Engineer
SaaS, Cloud, Data Centers & Infrastructure
Lithium Technologies, Inc
225 Bush St., 15th Floor
San Francisco, CA 94104
tel: +1 415 757 3100 x3219

From: Deprez, Tom tom.deprez@bauerservices.co.uk
Sent: Friday, July 25, 2014 1:56 AM
To: chef@lists.opscode.com
Subject: [chef] Running chef in multiple environments

Hi,

We’re using chef to build our new hosting environment, which comprises of
3 environments (dev, staging and production), and manage our own chef
server. So far we’ve been using chef environments to change variables etc
between the 3 environments. However, we’ve come across an issue whereby we
need to test a new role in our dev environement.

With cookbooks we can restrict the version that is used between
environments, but there doesn’t seem to be anything similar with roles (or
data bags, which is another issue).

I’m just wondering, what do people do in this situation? Should we be
looking to run a separate chef server in each environment instead of using
chef environments?

Thanks

Tom

HBVB trading as Bauer Corporate Services (BCS) is a division of the Bauer
Media
Group the largest consumer publisher in the UK, and second largest
commercial
radio broadcaster. BCS provides financial services and manages and
develops IT
systems on which our UK publishing, broadcast, digital and partner
businesses depend.

The information in this email is intended only for the addressee(s) named
above.
Access to this email by anyone else is unauthorised. If you are not the
intended
recipient of this message any disclosure, copying, distribution or any
action
taken in reliance on it is prohibited and may be unlawful. HBVB do not
warrant that
any attachments are free from viruses or other defects and accept no
liability for
any losses resulting from infected email transmissions.

Please note that any views expressed in this email may be those of the
originator
and do not necessarily reflect those of this organisation.

HBVB is registered in England; Registered address is
1 Lincoln Court, Lincoln Road, Peterborough, PE1 2RF.

Registration number 8453545

--
Best regards, Dmitriy V.

10

On Wed, 30 Jul 2014 00:30:39 -0700
DV vindimy@gmail.com wrote:

So, my point is something my colleague brought up recently - why run
chef-client as a daemon on production runway at all? Why not run it on
demand, such as when changes need to go out to production? From my
experience anyway, our production runways never get any updates when
they are actually serving production traffic, so it's safe to turn
chef-client off to prevent any accidental promotion of cookbooks,
environments, or roles to production.

That way you can keep using just one Chef server and keep using roles.

We haven't got to the point where this has become a problem yet but I
had been thinking the same thing. Not sure whether we'll take that
approach yet.

11

I’m thinking we might have to go down this route and not schedule chef-client to run automatically, at least until we can re-work the roles into cookbooks.

I’d prefer to run separate chef servers to minimise the risk of a cookbook being promoted straight to production (it only takes someone to use –force and it’ll deploy to all environments, overwriting any cookbook restraints), but that makes it harder to run something like nagios which monitors servers from all environments.

Thanks for all the ideas though! Certainly something we need to have a good think over before committing ourselves to a particular route.

It’s a shame there’s no full case studies written up on the getchef.com site, detailing how some larger companies are implementing chef. I know there’s some videos on youtube (ancestry.com etc), but they can cover everything.

Thanks
Tom

From: DV [mailto:vindimy@gmail.com]
Sent: 30 July 2014 08:31
To: chef@lists.opscode.com
Subject: [chef] Re: RE: Running chef in multiple environments

It seems there’s two common answers to the “role problem”:

  • Set up multiple Chef servers
  • Don’t use roles

Both are kind of a pain in the ass for various reasons…

Having multiple Chef servers means I can’t depend on Chef to seamlessly provide configuration across environments, where needed (such as relying on Chef’s search function). I also have to ensure all of my Chef servers have correct versions of Chef artifacts.

Not using roles… Well, I just don’t like the idea. Roles are just such a great way to group things together as well as separate Chef’s functionalities into distinct steps. Roles are also very simple to look at - just one file, no versions to increment, low hassle.

So, my point is something my colleague brought up recently - why run chef-client as a daemon on production runway at all? Why not run it on demand, such as when changes need to go out to production? From my experience anyway, our production runways never get any updates when they are actually serving production traffic, so it’s safe to turn chef-client off to prevent any accidental promotion of cookbooks, environments, or roles to production.

That way you can keep using just one Chef server and keep using roles.

On Fri, Jul 25, 2014 at 6:01 AM, Justin Franks <justin.franks@lithium.commailto:justin.franks@lithium.com> wrote:

Tom,

We have spent much time with Opscode coming onsite to help train us. We also have similar environments to you. After talking with Opscode instructors it was decided to create a chef server for each environment. For us it works well.

It may work well for you too.

Justin Franks
Lead Operations Engineer
SaaS, Cloud, Data Centers & Infrastructure
Lithium Technologies, Inc
225 Bush St., 15th Floor
San Francisco, CA 94104
tel: +1 415 757 3100 x3219tel:%2B1%20415%20757%203100%20x3219

From: Deprez, Tom <tom.deprez@bauerservices.co.ukmailto:tom.deprez@bauerservices.co.uk>
Sent: Friday, July 25, 2014 1:56 AM
To: chef@lists.opscode.commailto:chef@lists.opscode.com
Subject: [chef] Running chef in multiple environments

Hi,

We’re using chef to build our new hosting environment, which comprises of 3 environments (dev, staging and production), and manage our own chef server. So far we’ve been using chef environments to change variables etc between the 3 environments. However, we’ve come across an issue whereby we need to test a new role in our dev environement.

With cookbooks we can restrict the version that is used between environments, but there doesn’t seem to be anything similar with roles (or data bags, which is another issue).

I’m just wondering, what do people do in this situation? Should we be looking to run a separate chef server in each environment instead of using chef environments?

Thanks
Tom

HBVB trading as Bauer Corporate Services (BCS) is a division of the Bauer Media
Group the largest consumer publisher in the UK, and second largest commercial
radio broadcaster. BCS provides financial services and manages and develops IT
systems on which our UK publishing, broadcast, digital and partner businesses depend.

The information in this email is intended only for the addressee(s) named above.
Access to this email by anyone else is unauthorised. If you are not the intended
recipient of this message any disclosure, copying, distribution or any action
taken in reliance on it is prohibited and may be unlawful. HBVB do not warrant that
any attachments are free from viruses or other defects and accept no liability for
any losses resulting from infected email transmissions.

Please note that any views expressed in this email may be those of the originator
and do not necessarily reflect those of this organisation.

HBVB is registered in England; Registered address is
1 Lincoln Court, Lincoln Road, Peterborough, PE1 2RF.

Registration number 8453545

–
Best regards, Dmitriy V.

12

On 7/30/14, 12:30 AM, DV wrote:

So, my point is something my colleague brought up recently - why run
chef-client as a daemon on production runway at all? Why not run it on
demand, such as when changes need to go out to production? From my
experience anyway, our production runways never get any updates when
they are actually serving production traffic, so it's safe to turn
chef-client off to prevent any accidental promotion of cookbooks,
environments, or roles to production.

Historically, change in IT did not happen very often and Chef inherits a
lot of the old CFEngine model where you want to run it periodically and
gain the 'self-healing'/'computer immunity' features so that servers are
always being fixed back into compliance. In places that I've worked in
the past, change management was pretty abusive, so if CFEngine wasn't
running at least nightly then it'd never run for months (or sometimes
years -- I can think of a few servers at a company that I worked at for
3 years which were never deployed to). If you had the model where you
only kicked it off when a 'Change' happened then it became terrifying to
think about what might occur when you ran it because it hadn't been run
in so long.

So, that's the background. That may not make sense any more in a CI/CD
agile kind of world.

13

That's a great point - there'd better be a process to ensure chef-client
runs frequently enough so that all servers are in line with Chef and don't
fall too far behind.

As far as my suggestion of not running chef-client daemon on production
runways - it has at least one weakness: if there's any kind of auto-scaling
going on, of course chef-client will use whatever role happens to be on
Chef server when bootstrapping new servers. Similarly for if a server dies
and needs to be re-built or if there's a host that relies on Chef for
latest configuration (such as munin, or perhaps a client app that needs to
know hostname(s) of database servers to connect to).

My company went with pre-prod + prod instances of Chef server from the very
beginning, and so far it's saved us a lot of trouble since we only allow
the most experienced Chef users to promote changes to prod Chef.

On Wed, Jul 30, 2014 at 9:59 AM, Lamont Granquist lamont@opscode.com
wrote:

On 7/30/14, 12:30 AM, DV wrote:

So, my point is something my colleague brought up recently - why run
chef-client as a daemon on production runway at all? Why not run it on
demand, such as when changes need to go out to production? From my
experience anyway, our production runways never get any updates when they
are actually serving production traffic, so it's safe to turn chef-client
off to prevent any accidental promotion of cookbooks, environments, or
roles to production.

Historically, change in IT did not happen very often and Chef inherits a
lot of the old CFEngine model where you want to run it periodically and
gain the 'self-healing'/'computer immunity' features so that servers are
always being fixed back into compliance. In places that I've worked in the
past, change management was pretty abusive, so if CFEngine wasn't running
at least nightly then it'd never run for months (or sometimes years -- I
can think of a few servers at a company that I worked at for 3 years which
were never deployed to). If you had the model where you only kicked it
off when a 'Change' happened then it became terrifying to think about what
might occur when you ran it because it hadn't been run in so long.

So, that's the background. That may not make sense any more in a CI/CD
agile kind of world.

--
Best regards, Dmitriy V.

14

I'd like to chime in here...

As Lamont said... Chef's lineage definitely derives from CFEngine. The
whole idea is that recipes represent state policy that your machines
are supposed to be in. Running on a control loop ensure that they're
really that way.

Running chef-client on a control loop interval ensures that anything
that perturbs the system (broken code, malicious code, h4x0rs, junior
sysadmins, cowboy developers) will be repaired.

Think about if this way: if your application has a security
vulnerability that allows an attacker to chmod 777 /etc/shadow... do
you want to wait until your next production change window to repair
it? Or do you want to fix it as quickly as possible?

-s

On Wed, Jul 30, 2014 at 6:34 PM, DV vindimy@gmail.com wrote:

That's a great point - there'd better be a process to ensure chef-client
runs frequently enough so that all servers are in line with Chef and don't
fall too far behind.

As far as my suggestion of not running chef-client daemon on production
runways - it has at least one weakness: if there's any kind of auto-scaling
going on, of course chef-client will use whatever role happens to be on Chef
server when bootstrapping new servers. Similarly for if a server dies and
needs to be re-built or if there's a host that relies on Chef for latest
configuration (such as munin, or perhaps a client app that needs to know
hostname(s) of database servers to connect to).

My company went with pre-prod + prod instances of Chef server from the very
beginning, and so far it's saved us a lot of trouble since we only allow the
most experienced Chef users to promote changes to prod Chef.

On Wed, Jul 30, 2014 at 9:59 AM, Lamont Granquist lamont@opscode.com
wrote:

On 7/30/14, 12:30 AM, DV wrote:

So, my point is something my colleague brought up recently - why run
chef-client as a daemon on production runway at all? Why not run it on
demand, such as when changes need to go out to production? From my
experience anyway, our production runways never get any updates when they
are actually serving production traffic, so it's safe to turn chef-client
off to prevent any accidental promotion of cookbooks, environments, or roles
to production.

Historically, change in IT did not happen very often and Chef inherits a
lot of the old CFEngine model where you want to run it periodically and gain
the 'self-healing'/'computer immunity' features so that servers are always
being fixed back into compliance. In places that I've worked in the past,
change management was pretty abusive, so if CFEngine wasn't running at least
nightly then it'd never run for months (or sometimes years -- I can think of
a few servers at a company that I worked at for 3 years which were never
deployed to). If you had the model where you only kicked it off when a
'Change' happened then it became terrifying to think about what might occur
when you ran it because it hadn't been run in so long.

So, that's the background. That may not make sense any more in a CI/CD
agile kind of world.

--
Best regards, Dmitriy V.

15

On Thu Jul 31 16:48:19 2014, Sean OMeara wrote:

Think about if this way: if your application has a security
vulnerability that allows an attacker to chmod 777 /etc/shadow... do
you want to wait until your next production change window to repair
it? Or do you want to fix it as quickly as possible?

Yeah, I'd forgotten about that. In a SOX/PCI-DSS world you really want
to be able to show that you're running it frequently and enforcing
security with it.

One thing I used to do and typically suggest is to run it in a window
once a day with random splay. I used to run CFEngine/Chef from
8pm->8am PST. That would mean that during business hours I could push
changes and they wouldn't immediately take effect until you poked the
box. Then you could make it part of your rollout plan to test on some
canary-style boxes in production and if you discovered a serious
problem you could revert the change without having trashed all of
production. Once you were confident that the canarys were all still
alive then you could manually push it out with knife ssh.

Means you've got a 24h window before you revert issues like the 777
/etc/shadow file. Also means that if you accidentally chmod 400
/etc/resolv.conf you've got 12 hours to catch it as it rolls out though.

Worked great for me (at large scale even), but does not seem to be a
common pattern anyone else uses.

16

On Thu, 31 Jul 2014 19:48:19 -0400
Sean OMeara someara@opscode.com wrote:

I'd like to chime in here...

As Lamont said... Chef's lineage definitely derives from CFEngine. The
whole idea is that recipes represent state policy that your machines
are supposed to be in. Running on a control loop ensure that they're
really that way.

Running chef-client on a control loop interval ensures that anything
that perturbs the system (broken code, malicious code, h4x0rs, junior
sysadmins, cowboy developers) will be repaired.

Think about if this way: if your application has a security
vulnerability that allows an attacker to chmod 777 /etc/shadow... do
you want to wait until your next production change window to repair
it? Or do you want to fix it as quickly as possible?

I don't really agree with this. If someone has broken in, you should
take time to work out exactly what has been changed and how they
managed to do it. You should also take that box offline immediately
instead of relying on Chef to patch it up in the short term.

While Chef's enforcement of permissions is a nice side effect, I don't
think it should be used primarily as a security feature. My cookbooks
use /etc/shadow but given that this file was already present with the
correct permissions before installing Chef, I don't think Chef should
attempt to enforce this. You could write a cookbook to enforce the
permissions of every vaguely important file on the system but this
would be tedious and probably not very effective. Watching for
unexpected file system changes is really a job for AIDE and the like.

Regards,
James

17

This is at the heart of the "immutable infrastructure" vs "managing
configuration drift" models of infrastructure management.

It really depends on the application.

Okay lets throw out security as a concern, because that invokes
irrationality and sacred "best practices".

Lets focus on stability...
control loops re-start services that have crashed.
control loops allow for dynamic reconfiguration of static
configuration files (LB, metrics, monitoring).
control loops enforce policy on user-interactive machines like
workstations or laptops

Not saying its for you, feel free to crank your control loop down to
zero. Trigger it if thats better.

I'm only pointing out that the in lizard brain of convergent
configuration management tools lies self healing via the
re-application of policy over time.

Now back to your regularly scheduled thread.

-s

On Fri, Aug 1, 2014 at 4:51 AM, James Le Cuirot chewi@aura-online.co.uk wrote:

On Thu, 31 Jul 2014 19:48:19 -0400
Sean OMeara someara@opscode.com wrote:

I'd like to chime in here...

As Lamont said... Chef's lineage definitely derives from CFEngine. The
whole idea is that recipes represent state policy that your machines
are supposed to be in. Running on a control loop ensure that they're
really that way.

Running chef-client on a control loop interval ensures that anything
that perturbs the system (broken code, malicious code, h4x0rs, junior
sysadmins, cowboy developers) will be repaired.

Think about if this way: if your application has a security
vulnerability that allows an attacker to chmod 777 /etc/shadow... do
you want to wait until your next production change window to repair
it? Or do you want to fix it as quickly as possible?

I don't really agree with this. If someone has broken in, you should
take time to work out exactly what has been changed and how they
managed to do it. You should also take that box offline immediately
instead of relying on Chef to patch it up in the short term.

While Chef's enforcement of permissions is a nice side effect, I don't
think it should be used primarily as a security feature. My cookbooks
use /etc/shadow but given that this file was already present with the
correct permissions before installing Chef, I don't think Chef should
attempt to enforce this. You could write a cookbook to enforce the
permissions of every vaguely important file on the system but this
would be tedious and probably not very effective. Watching for
unexpected file system changes is really a job for AIDE and the like.

Regards,
James

18

On 8/1/14, 1:51 AM, James Le Cuirot wrote:

I don't really agree with this. If someone has broken in, you should
take time to work out exactly what has been changed and how they
managed to do it. You should also take that box offline immediately
instead of relying on Chef to patch it up in the short term. While
Chef's enforcement of permissions is a nice side effect, I don't think
it should be used primarily as a security feature. My cookbooks use
/etc/shadow but given that this file was already present with the
correct permissions before installing Chef, I don't think Chef should
attempt to enforce this. You could write a cookbook to enforce the
permissions of every vaguely important file on the system but this
would be tedious and probably not very effective. Watching for
unexpected file system changes is really a job for AIDE and the like.

I totally agree that Chef/CFEngine/Puppet/etc make bad Tripwire-like
systems.

I've still absolutely used them as prevent controls in SOX and PCI-DSS
environments and to show compliance with written systems standards. We
can argue about the absolute security utility of that, but practically
it makes the PCI-DSS auditor go away and I like that... =)