Questions About Solo / Local Only Chef


#1

Chefs,

I come primarily from a Puppet background and am curious about using Chef.
Currently we have a nodeless and masterless Puppet implementation that
pulls all modules, manifests, etc. from Git and stores configuration data
in a backend PostgreSQL DB (PuppetDB).

With this being said, I am trying to figure out what similar functionality
Chef offers. Some specific questions about this are listed below.
Basically, I am trying to eliminate the Chef Server, due to scalability /
maintainability reasons, similar to removing a Puppet Master in puppet.

  1. Based off my reading, Chef seems to use either CouchDB or PostgreSQL as
    the backend configuration store on the Chef Server. Can this configuration
    store be accessed by Chef Clients when they are performing a solo / local
    chef run?

  2. Is there a Chef equivalent of Puppet’s storeconfigs?

  3. How can Chef Clients performing solo / local runs share configuration
    information with one another?

  4. Can Chef define node / role information in LDAP, so that a solo /
    locally ran Chef Client can dynamically “realize” what roles / cookbooks to
    apply to itself?

  5. What are some suggested configurations of Chef that don’t involve the
    Chef Server and allow for scaling large deployments?

  6. Can you share some literature, documentations, etc. for the different
    ways to deploy Chef w/o using a Chef Server?

Mike


#2

Nico Kadel-Garcia
Email: nkadel@gmail.commailto:kadel@gmail.com
Sent from iPhone

On Apr 7, 2015, at 0:34, “Michael Pawlak” <mike@colovore.commailto:mike@colovore.com> wrote:

Chefs,

I come primarily from a Puppet background and am curious about using Chef. Currently we have a nodeless and masterless Puppet implementation that pulls all modules, manifests, etc. from Git and stores configuration data in a backend PostgreSQL DB (PuppetDB).

With this being said, I am trying to figure out what similar functionality Chef offers. Some specific questions about this are listed below. Basically, I am trying to eliminate the Chef Server, due to scalability / maintainability reasons, similar to removing a Puppet Master in puppet.

  1. Based off my reading, Chef seems to use either CouchDB or PostgreSQL as the backend configuration store on the Chef Server. Can this configuration store be accessed by Chef Clients when they are performing a solo / local chef run

Can it be accessed? Yes, by having the computer issue queries against the server by hook or by crook. But that us what chef client us for, to access the chef server’s knowledge of this information. As soon as you start querying this central database, you lose most if the point if chef solo.

  1. Is there a Chef equivalent of Puppet’s storeconfigs?

  2. How can Chef Clients performing solo / local runs share configuration information with one another?

However you decide to publish it. Git repos, for example, can allow automated publication if data bags in submodules managed by individual hosts. The possibilities are limited only by your creativity.

It is a direct violation if the standard server/client model, so you’ll be pretty much on your own doing it.

  1. Can Chef define node / role information in LDAP, so that a solo / locally ran Chef Client can dynamically “realize” what roles / cookbooks to apply to itself?

If you can write tools to generate json files from LDAP information, you can write an LDAP->node filegeneration tool.

  1. What are some suggested configurations of Chef that don’t involve the Chef Server and allow for scaling large deployment?

See above. The amount if work doing this from scratch outweighs that of running a chef server pretty quickly.

  1. Can you share some literature, documentations, etc. for the different ways to deploy Chef w/o using a Chef Server?

Mike

I’m afraid there are as many ways as there are chef developers. I personally use chefdk to get chef-solo and Berkshelf to access a git repo with nodes, roles, etc. but my setup is small, and I can generate nagios configs from reading node config files.


#3

On Monday, April 6, 2015 at 9:34 PM, Michael Pawlak wrote:

Chefs,

I come primarily from a Puppet background and am curious about using Chef. Currently we have a nodeless and masterless Puppet implementation that pulls all modules, manifests, etc. from Git and stores configuration data in a backend PostgreSQL DB (PuppetDB).

With this being said, I am trying to figure out what similar functionality Chef offers. Some specific questions about this are listed below. Basically, I am trying to eliminate the Chef Server, due to scalability / maintainability reasons, similar to removing a Puppet Master in puppet.
What are your scalability and maintainability reasons? I know of a lot of users using Chef Server at pretty large scale—it definitely scales higher on a single server than puppet does. At really enormous scale, you would need to federate, but even then you can use Chef Server as your local distribution mechanism. For example: https://github.com/facebook/grocery-delivery

I’m not sure what you mean about “maintainability”, though, can you clarify?

  1. Based off my reading, Chef seems to use either CouchDB or PostgreSQL as the backend configuration store on the Chef Server. Can this configuration store be accessed by Chef Clients when they are performing a solo / local chef run?
    The old Chef Server used CouchDB, but that version is totally EOL. All supported versions of the server are on Postgres. As mentioned in the other reply thread, trying to do a half-serverless model is probably going to cause you a lot of pain; if you need something like that, you’re probably better off owning the codebase yourself.
  1. Is there a Chef equivalent of Puppet’s storeconfigs?

  2. How can Chef Clients performing solo / local runs share configuration information with one another?

  3. Can Chef define node / role information in LDAP, so that a solo / locally ran Chef Client can dynamically “realize” what roles / cookbooks to apply to itself?

  4. What are some suggested configurations of Chef that don’t involve the Chef Server and allow for scaling large deployments?

  5. Can you share some literature, documentations, etc. for the different ways to deploy Chef w/o using a Chef Server?

Mike


Daniel DeLeo


#4

On 4/6/15 9:34 PM, Michael Pawlak wrote:

Chefs,

I come primarily from a Puppet background and am curious about using
Chef. Currently we have a nodeless and masterless Puppet
implementation that pulls all modules, manifests, etc. from Git and
stores configuration data in a backend PostgreSQL DB (PuppetDB).

With this being said, I am trying to figure out what similar
functionality Chef offers. Some specific questions about this are
listed below. Basically, I am trying to eliminate the Chef Server, due
to scalability / maintainability reasons, similar to removing a Puppet
Master in puppet.

Have you actually tested that a Chef Server fails to scale in your
environment? Since the Chef Server does not assemble a manifest like a
Puppet Master does, it is very thin. Mostly it just serves
configuration files. If you are not heavily using search (and moving
to chef local mode will eliminate your ability to use search entirely)
then a single Chef Server should scale to tens of thousands of nodes, or
somewhere around 10x more than a Puppet Master running on the same
hardware does. Most of the computational offload that you’re getting
from the approach you are taking with Puppet is already baked into the
Chef Client.

Where you’re used to the Puppet Master assembling a manifest and
shipping it to the Client, the analogous thing in Chef is the Resource
Collection. That is a dynamically created object that is created in the
Chef Client, rather than the server. Since the Client creates it, you
can more easily dynamically change the contents of the Resource
Collection based on the existing state of the node and the information
that the node can gather about its surroundings, and the processing
overhead is assumed entirely by the Client, not the Server.

TL;DR is don’t blindly treat Chef like Puppet right out of the gate, you
may have learned the wrong lessons.


#5

If you are not heavily using search (and moving
to chef local mode will eliminate your ability to use search entirely)

I thought local mode ( ie - chef zero ) kept the ability to do ‘searches’. At least, it works with the samba cookbook.

chef-solo, on the other hand, has no search ability.


From: Lamont Granquist lamont@chef.io
Sent: Tuesday, April 07, 2015 10:01 AM
To: chef@lists.opscode.com
Subject: [chef] Re: Questions About Solo / Local Only Chef

On 4/6/15 9:34 PM, Michael Pawlak wrote:

Chefs,

I come primarily from a Puppet background and am curious about using
Chef. Currently we have a nodeless and masterless Puppet
implementation that pulls all modules, manifests, etc. from Git and
stores configuration data in a backend PostgreSQL DB (PuppetDB).

With this being said, I am trying to figure out what similar
functionality Chef offers. Some specific questions about this are
listed below. Basically, I am trying to eliminate the Chef Server, due
to scalability / maintainability reasons, similar to removing a Puppet
Master in puppet.

Have you actually tested that a Chef Server fails to scale in your
environment? Since the Chef Server does not assemble a manifest like a
Puppet Master does, it is very thin. Mostly it just serves
configuration files. If you are not heavily using search (and moving
to chef local mode will eliminate your ability to use search entirely)
then a single Chef Server should scale to tens of thousands of nodes, or
somewhere around 10x more than a Puppet Master running on the same
hardware does. Most of the computational offload that you’re getting
from the approach you are taking with Puppet is already baked into the
Chef Client.

Where you’re used to the Puppet Master assembling a manifest and
shipping it to the Client, the analogous thing in Chef is the Resource
Collection. That is a dynamically created object that is created in the
Chef Client, rather than the server. Since the Client creates it, you
can more easily dynamically change the contents of the Resource
Collection based on the existing state of the node and the information
that the node can gather about its surroundings, and the processing
overhead is assumed entirely by the Client, not the Server.

TL;DR is don’t blindly treat Chef like Puppet right out of the gate, you
may have learned the wrong lessons.


#6

Sure, the search API works, but it doesn’t know about the rest of your
infrastructure the way a chef server would, you’d need to prepopulate
critical nodes or something.

On 4/7/15 10:10 AM, Hajducko, Steven wrote:

If you are not heavily using search (and moving
to chef local mode will eliminate your ability to use search entirely)
I thought local mode ( ie - chef zero ) kept the ability to do ‘searches’. At least, it works with the samba cookbook.

chef-solo, on the other hand, has no search ability.


From: Lamont Granquist lamont@chef.io
Sent: Tuesday, April 07, 2015 10:01 AM
To: chef@lists.opscode.com
Subject: [chef] Re: Questions About Solo / Local Only Chef

On 4/6/15 9:34 PM, Michael Pawlak wrote:

Chefs,

I come primarily from a Puppet background and am curious about using
Chef. Currently we have a nodeless and masterless Puppet
implementation that pulls all modules, manifests, etc. from Git and
stores configuration data in a backend PostgreSQL DB (PuppetDB).

With this being said, I am trying to figure out what similar
functionality Chef offers. Some specific questions about this are
listed below. Basically, I am trying to eliminate the Chef Server, due
to scalability / maintainability reasons, similar to removing a Puppet
Master in puppet.

Have you actually tested that a Chef Server fails to scale in your
environment? Since the Chef Server does not assemble a manifest like a
Puppet Master does, it is very thin. Mostly it just serves
configuration files. If you are not heavily using search (and moving
to chef local mode will eliminate your ability to use search entirely)
then a single Chef Server should scale to tens of thousands of nodes, or
somewhere around 10x more than a Puppet Master running on the same
hardware does. Most of the computational offload that you’re getting
from the approach you are taking with Puppet is already baked into the
Chef Client.

Where you’re used to the Puppet Master assembling a manifest and
shipping it to the Client, the analogous thing in Chef is the Resource
Collection. That is a dynamically created object that is created in the
Chef Client, rather than the server. Since the Client creates it, you
can more easily dynamically change the contents of the Resource
Collection based on the existing state of the node and the information
that the node can gather about its surroundings, and the processing
overhead is assumed entirely by the Client, not the Server.

TL;DR is don’t blindly treat Chef like Puppet right out of the gate, you
may have learned the wrong lessons.


#7

IMHO, if you really don’t want chef server, then chef likely is the wrong
choice for you.
Having a standard access to search data is kinda the whole point.

With enough duct tape and bailing wire, you can probably accomplish the
list of features
you have, but you’re essentially reinventing chef server in a one off, hard
to maintain version.

  • Booker C. Bense

On Tue, Apr 7, 2015 at 10:14 AM, Lamont Granquist lamont@chef.io wrote:

Sure, the search API works, but it doesn’t know about the rest of your
infrastructure the way a chef server would, you’d need to prepopulate
critical nodes or something.

On 4/7/15 10:10 AM, Hajducko, Steven wrote:

If you are not heavily using search (and moving

to chef local mode will eliminate your ability to use search entirely)

I thought local mode ( ie - chef zero ) kept the ability to do
’searches’. At least, it works with the samba cookbook.

chef-solo, on the other hand, has no search ability.


From: Lamont Granquist lamont@chef.io
Sent: Tuesday, April 07, 2015 10:01 AM
To: chef@lists.opscode.com
Subject: [chef] Re: Questions About Solo / Local Only Chef

On 4/6/15 9:34 PM, Michael Pawlak wrote:

Chefs,

I come primarily from a Puppet background and am curious about using
Chef. Currently we have a nodeless and masterless Puppet
implementation that pulls all modules, manifests, etc. from Git and
stores configuration data in a backend PostgreSQL DB (PuppetDB).

With this being said, I am trying to figure out what similar
functionality Chef offers. Some specific questions about this are
listed below. Basically, I am trying to eliminate the Chef Server, due
to scalability / maintainability reasons, similar to removing a Puppet
Master in puppet.

Have you actually tested that a Chef Server fails to scale in your
environment? Since the Chef Server does not assemble a manifest like a
Puppet Master does, it is very thin. Mostly it just serves
configuration files. If you are not heavily using search (and moving
to chef local mode will eliminate your ability to use search entirely)
then a single Chef Server should scale to tens of thousands of nodes, or
somewhere around 10x more than a Puppet Master running on the same
hardware does. Most of the computational offload that you’re getting
from the approach you are taking with Puppet is already baked into the
Chef Client.

Where you’re used to the Puppet Master assembling a manifest and
shipping it to the Client, the analogous thing in Chef is the Resource
Collection. That is a dynamically created object that is created in the
Chef Client, rather than the server. Since the Client creates it, you
can more easily dynamically change the contents of the Resource
Collection based on the existing state of the node and the information
that the node can gather about its surroundings, and the processing
overhead is assumed entirely by the Client, not the Server.

TL;DR is don’t blindly treat Chef like Puppet right out of the gate, you
may have learned the wrong lessons.


#8

If a tool works for you in a certain way, to say that you shouldn’t use the tool because it has X functionality that you are not using seems like a very exclusionary stance.

In any case, my point about chef-local and search was simply to point out that chef-local still provides search in a sense. This is important because certain community cookbooks you may want to use do have search functions and will not work at all with solo ( ie - the samba cookbook ), but you can get them to function with chef-local, which will still accomplish the goal of running in a masterless mode ( or even, say, just testing with Test Kitchen on a local instance ).

Sent from my iPhone

On Apr 8, 2015, at 8:16 AM, Booker Bense <bbense@gmail.commailto:bbense@gmail.com> wrote:

IMHO, if you really don’t want chef server, then chef likely is the wrong choice for you.
Having a standard access to search data is kinda the whole point.

With enough duct tape and bailing wire, you can probably accomplish the list of features
you have, but you’re essentially reinventing chef server in a one off, hard to maintain version.

  • Booker C. Bense

On Tue, Apr 7, 2015 at 10:14 AM, Lamont Granquist <lamont@chef.iomailto:lamont@chef.io> wrote:
Sure, the search API works, but it doesn’t know about the rest of your infrastructure the way a chef server would, you’d need to prepopulate critical nodes or something.

On 4/7/15 10:10 AM, Hajducko, Steven wrote:
If you are not heavily using search (and moving
to chef local mode will eliminate your ability to use search entirely)
I thought local mode ( ie - chef zero ) kept the ability to do ‘searches’. At least, it works with the samba cookbook.

chef-solo, on the other hand, has no search ability.


From: Lamont Granquist <lamont@chef.iomailto:lamont@chef.io>
Sent: Tuesday, April 07, 2015 10:01 AM
To: chef@lists.opscode.commailto:chef@lists.opscode.com
Subject: [chef] Re: Questions About Solo / Local Only Chef

On 4/6/15 9:34 PM, Michael Pawlak wrote:
Chefs,

I come primarily from a Puppet background and am curious about using
Chef. Currently we have a nodeless and masterless Puppet
implementation that pulls all modules, manifests, etc. from Git and
stores configuration data in a backend PostgreSQL DB (PuppetDB).

With this being said, I am trying to figure out what similar
functionality Chef offers. Some specific questions about this are
listed below. Basically, I am trying to eliminate the Chef Server, due
to scalability / maintainability reasons, similar to removing a Puppet
Master in puppet.

Have you actually tested that a Chef Server fails to scale in your
environment? Since the Chef Server does not assemble a manifest like a
Puppet Master does, it is very thin. Mostly it just serves
configuration files. If you are not heavily using search (and moving
to chef local mode will eliminate your ability to use search entirely)
then a single Chef Server should scale to tens of thousands of nodes, or
somewhere around 10x more than a Puppet Master running on the same
hardware does. Most of the computational offload that you’re getting
from the approach you are taking with Puppet is already baked into the
Chef Client.

Where you’re used to the Puppet Master assembling a manifest and
shipping it to the Client, the analogous thing in Chef is the Resource
Collection. That is a dynamically created object that is created in the
Chef Client, rather than the server. Since the Client creates it, you
can more easily dynamically change the contents of the Resource
Collection based on the existing state of the node and the information
that the node can gather about its surroundings, and the processing
overhead is assumed entirely by the Client, not the Server.

TL;DR is don’t blindly treat Chef like Puppet right out of the gate, you
may have learned the wrong lessons.


#9

+1, that Steven :slight_smile:

Michael,
Chef server provides two basic service for chef client, artifact server
(for cookbooks., data bags, roles) and metadata (node system info,). you
can by pass each of these. independently. Following are some combination
you can try:

  1. Use event handlers to push data wherever you want (zk, etcd. mysql
    eftc). Zk, etcd bindings are under FOSS, but maintained by community.
  2. Use s3, or github or any other chef server (a standalone chef server
    thanat has no node but cookbooks, databag etc). But git like approach will
    leads to all artifacts being propagated to the nodes.
    Now, on chef-solo and local mode. I personally use it, and it works. As you
    have said, the search is available but no data. Two ways you can address
    that,
  3. Consume the metadata service you have defined, expose it as a library
    helper method. extend that module from recipes you want the discovery
    mechanism.
  4. Prepare stubs. i.e. create the data you want to inject, and then push
    it inside chef-zero using chef api. this is super easy, in most cases you
    will be able to use the knife plugins programmatically. This will also
    allow you to checkin the metadata itself inside the repo and use it from
    the stub. Service stubs are common design pattern, we are just extending it
    to infrastructure.

A big benefit of local-mode is that you dont have to write specific
strategies based on chef-solo or chef-client modes, which impacts how much
we can share the cookbooks.

This means you can reuse whatever metadata service you had in master less
puppet, and plug it in :slight_smile:

Also, chef-solo will never go away. Local-mode is no-where as widely tested
as chef-solo. ChefFS (one of the localmode components) needs lot more
testing, dan just did the socketless localmode works as well, when its
stable and widely tested, it will replace chef-solo internal, which will
use chef-local under the hood, but UX of solo will largely remain same. in
short, the migration should impact lot less for existing consumers. Just be
aware, and try to use upstream versions.

Couple of things to note though. By doing all of these we are pretty much
building a custom chef server but with different components and data flows.
Depending upon what components you choose, u’ll have different experience
(maintenance, features, scalability etc).

Let us know if you need examples of any of these,

On Wed, Apr 8, 2015 at 8:37 AM, Hajducko, Steven <Steven_Hajducko@intuit.com

wrote:

If a tool works for you in a certain way, to say that you shouldn’t use
the tool because it has X functionality that you are not using seems like a
very exclusionary stance.

In any case, my point about chef-local and search was simply to point
out that chef-local still provides search in a sense. This is important
because certain community cookbooks you may want to use do have search
functions and will not work at all with solo ( ie - the samba cookbook ),
but you can get them to function with chef-local, which will still
accomplish the goal of running in a masterless mode ( or even, say, just
testing with Test Kitchen on a local instance ).

Sent from my iPhone

On Apr 8, 2015, at 8:16 AM, Booker Bense bbense@gmail.com wrote:

IMHO, if you really don’t want chef server, then chef likely is the
wrong choice for you.
Having a standard access to search data is kinda the whole point.

With enough duct tape and bailing wire, you can probably accomplish the
list of features
you have, but you’re essentially reinventing chef server in a one off,
hard to maintain version.

  • Booker C. Bense

On Tue, Apr 7, 2015 at 10:14 AM, Lamont Granquist lamont@chef.io wrote:

Sure, the search API works, but it doesn’t know about the rest of your
infrastructure the way a chef server would, you’d need to prepopulate
critical nodes or something.

On 4/7/15 10:10 AM, Hajducko, Steven wrote:

If you are not heavily using search (and moving

to chef local mode will eliminate your ability to use search entirely)

I thought local mode ( ie - chef zero ) kept the ability to do
’searches’. At least, it works with the samba cookbook.

chef-solo, on the other hand, has no search ability.


From: Lamont Granquist lamont@chef.io
Sent: Tuesday, April 07, 2015 10:01 AM
To: chef@lists.opscode.com
Subject: [chef] Re: Questions About Solo / Local Only Chef

On 4/6/15 9:34 PM, Michael Pawlak wrote:

Chefs,

I come primarily from a Puppet background and am curious about using
Chef. Currently we have a nodeless and masterless Puppet
implementation that pulls all modules, manifests, etc. from Git and
stores configuration data in a backend PostgreSQL DB (PuppetDB).

With this being said, I am trying to figure out what similar
functionality Chef offers. Some specific questions about this are
listed below. Basically, I am trying to eliminate the Chef Server, due
to scalability / maintainability reasons, similar to removing a Puppet
Master in puppet.

Have you actually tested that a Chef Server fails to scale in your
environment? Since the Chef Server does not assemble a manifest like a
Puppet Master does, it is very thin. Mostly it just serves
configuration files. If you are not heavily using search (and moving
to chef local mode will eliminate your ability to use search entirely)
then a single Chef Server should scale to tens of thousands of nodes, or
somewhere around 10x more than a Puppet Master running on the same
hardware does. Most of the computational offload that you’re getting
from the approach you are taking with Puppet is already baked into the
Chef Client.

Where you’re used to the Puppet Master assembling a manifest and
shipping it to the Client, the analogous thing in Chef is the Resource
Collection. That is a dynamically created object that is created in the
Chef Client, rather than the server. Since the Client creates it, you
can more easily dynamically change the contents of the Resource
Collection based on the existing state of the node and the information
that the node can gather about its surroundings, and the processing
overhead is assumed entirely by the Client, not the Server.

TL;DR is don’t blindly treat Chef like Puppet right out of the gate, you
may have learned the wrong lessons.


#10

On Wed, Apr 8, 2015 at 8:37 AM, Hajducko, Steven <Steven_Hajducko@intuit.com

wrote:

If a tool works for you in a certain way, to say that you shouldn’t use
the tool because it has X functionality that you are not using seems like a
very exclusionary stance.

My argument is based on the fact that the OP does want exactly what chef
server provides, but does not want to run chef server. If using chef-solo
w/o search works for you then fine, but re-inventing chef search seems like
a sub-optimal choice. To me you lose many of the advantages that chef
provides as a tool at the expense of some serious technical debt. My
intention was not to exclude anybody, but to simply state “here be
dragons”.

One place I think Chef can go really badly wrong is that if you write
enough code you can do anything, but
writing your own code to duplicate existing functionality always has a
cost. Just because you can do anything in a tool,
does not mean you should do everything in that tool.

It’s important to weigh that cost before you invest in a configuration
management system, because CM’s are very sticky.
Once you pick one and build up a code base around it, moving to another
system is a significant cost.

Maybe Chef will be a reasonable choice even with the original constraints,
but I really think the OP should look around. There are more choices than
Chef or Puppet and those might be a better match for what he wants.

  • Booker C. Bense

#11

The cookbooks and configurable node or roles or data bags can work quite well for smaller environment management, without a whiff of the “search” functionality or the necessary investment in a high-availability chef server. There are also profound version management issues if you have multiple people testing new cookbooks on individual servers, using chef-solo and local git repos with Berkshelf helps prevent potentially destruct cookbook conflicts.

This was especially a problem when the new and not-reverse-compatible ‘yum’ and ‘mysql’ cookbooks were published. It was very easy to wind up with the new mysql cookbook and destroy a working system that had never had its mysql cookbook locked to a specific version, or to break other cookbooks with the “yum::epel” recipe being discarded but still a dependency in other cookbooks.

Nico Kadel-Garcia
Lead DevOps Engineer
nkadel@skyhookwireless.commailto:nkadel@skyhookwireless.com

From: Booker Bense [mailto:bbense@gmail.com]
Sent: Wednesday, April 08, 2015 11:15 AM
To: chef@lists.opscode.com
Subject: [chef] Re: Re: Re: Re: Questions About Solo / Local Only Chef

IMHO, if you really don’t want chef server, then chef likely is the wrong choice for you.
Having a standard access to search data is kinda the whole point.

With enough duct tape and bailing wire, you can probably accomplish the list of features
you have, but you’re essentially reinventing chef server in a one off, hard to maintain version.

  • Booker C. Bense

On Tue, Apr 7, 2015 at 10:14 AM, Lamont Granquist <lamont@chef.iomailto:lamont@chef.io> wrote:
Sure, the search API works, but it doesn’t know about the rest of your infrastructure the way a chef server would, you’d need to prepopulate critical nodes or something.

On 4/7/15 10:10 AM, Hajducko, Steven wrote:
If you are not heavily using search (and moving
to chef local mode will eliminate your ability to use search entirely)
I thought local mode ( ie - chef zero ) kept the ability to do ‘searches’. At least, it works with the samba cookbook.

chef-solo, on the other hand, has no search ability.


From: Lamont Granquist <lamont@chef.iomailto:lamont@chef.io>
Sent: Tuesday, April 07, 2015 10:01 AM
To: chef@lists.opscode.commailto:chef@lists.opscode.com
Subject: [chef] Re: Questions About Solo / Local Only Chef

On 4/6/15 9:34 PM, Michael Pawlak wrote:
Chefs,

I come primarily from a Puppet background and am curious about using
Chef. Currently we have a nodeless and masterless Puppet
implementation that pulls all modules, manifests, etc. from Git and
stores configuration data in a backend PostgreSQL DB (PuppetDB).

With this being said, I am trying to figure out what similar
functionality Chef offers. Some specific questions about this are
listed below. Basically, I am trying to eliminate the Chef Server, due
to scalability / maintainability reasons, similar to removing a Puppet
Master in puppet.

Have you actually tested that a Chef Server fails to scale in your
environment? Since the Chef Server does not assemble a manifest like a
Puppet Master does, it is very thin. Mostly it just serves
configuration files. If you are not heavily using search (and moving
to chef local mode will eliminate your ability to use search entirely)
then a single Chef Server should scale to tens of thousands of nodes, or
somewhere around 10x more than a Puppet Master running on the same
hardware does. Most of the computational offload that you’re getting
from the approach you are taking with Puppet is already baked into the
Chef Client.

Where you’re used to the Puppet Master assembling a manifest and
shipping it to the Client, the analogous thing in Chef is the Resource
Collection. That is a dynamically created object that is created in the
Chef Client, rather than the server. Since the Client creates it, you
can more easily dynamically change the contents of the Resource
Collection based on the existing state of the node and the information
that the node can gather about its surroundings, and the processing
overhead is assumed entirely by the Client, not the Server.

TL;DR is don’t blindly treat Chef like Puppet right out of the gate, you
may have learned the wrong lessons.


#12

The first thing to consider here is that Chef does its “thinking” on
the edges, unlike Puppet, where a central server calculates a manifest
and ships to the edges. That design difference alone will probably get
you scaled to where you are now.

A Chef Server is first and foremost an artifact repository for your
Chef cookbooks. It provides authN / authZ as well. Without these,
you’ll have to roll your own solutions. It is also required for the
advanced things like Analytics and Delivery.

Having a cookbook artifact repository lets you make available multiple
versions of your cookbooks, letting you pin down versions with
environments or policyfiles to handle situations like Nico described.

The search functionality is useful, but can you can live without if
you want to manage your network topology manually or with something
like ZK/consul/etcd.

Accessing the backing datastore directly is not recommended. Use the API.

-s

On Thu, Apr 9, 2015 at 6:45 PM, Nico Kadel-Garcia
nkadel@skyhookwireless.com wrote:

The cookbooks and configurable node or roles or data bags can work quite
well for smaller environment management, without a whiff of the “search”
functionality or the necessary investment in a high-availability chef
server. There are also profound version management issues if you have
multiple people testing new cookbooks on individual servers, using chef-solo
and local git repos with Berkshelf helps prevent potentially destruct
cookbook conflicts.

This was especially a problem when the new and not-reverse-compatible ‘yum’
and ‘mysql’ cookbooks were published. It was very easy to wind up with the
new mysql cookbook and destroy a working system that had never had its mysql
cookbook locked to a specific version, or to break other cookbooks with the
“yum::epel” recipe being discarded but still a dependency in other
cookbooks.

Nico Kadel-Garcia

Lead DevOps Engineer

nkadel@skyhookwireless.com

From: Booker Bense [mailto:bbense@gmail.com]
Sent: Wednesday, April 08, 2015 11:15 AM
To: chef@lists.opscode.com
Subject: [chef] Re: Re: Re: Re: Questions About Solo / Local Only Chef

IMHO, if you really don’t want chef server, then chef likely is the wrong
choice for you.

Having a standard access to search data is kinda the whole point.

With enough duct tape and bailing wire, you can probably accomplish the list
of features

you have, but you’re essentially reinventing chef server in a one off, hard
to maintain version.

  • Booker C. Bense

On Tue, Apr 7, 2015 at 10:14 AM, Lamont Granquist lamont@chef.io wrote:

Sure, the search API works, but it doesn’t know about the rest of your
infrastructure the way a chef server would, you’d need to prepopulate
critical nodes or something.

On 4/7/15 10:10 AM, Hajducko, Steven wrote:

If you are not heavily using search (and moving
to chef local mode will eliminate your ability to use search entirely)

I thought local mode ( ie - chef zero ) kept the ability to do ‘searches’.
At least, it works with the samba cookbook.

chef-solo, on the other hand, has no search ability.


From: Lamont Granquist lamont@chef.io
Sent: Tuesday, April 07, 2015 10:01 AM
To: chef@lists.opscode.com
Subject: [chef] Re: Questions About Solo / Local Only Chef

On 4/6/15 9:34 PM, Michael Pawlak wrote:

Chefs,

I come primarily from a Puppet background and am curious about using
Chef. Currently we have a nodeless and masterless Puppet
implementation that pulls all modules, manifests, etc. from Git and
stores configuration data in a backend PostgreSQL DB (PuppetDB).

With this being said, I am trying to figure out what similar
functionality Chef offers. Some specific questions about this are
listed below. Basically, I am trying to eliminate the Chef Server, due
to scalability / maintainability reasons, similar to removing a Puppet
Master in puppet.

Have you actually tested that a Chef Server fails to scale in your
environment? Since the Chef Server does not assemble a manifest like a
Puppet Master does, it is very thin. Mostly it just serves
configuration files. If you are not heavily using search (and moving
to chef local mode will eliminate your ability to use search entirely)
then a single Chef Server should scale to tens of thousands of nodes, or
somewhere around 10x more than a Puppet Master running on the same
hardware does. Most of the computational offload that you’re getting
from the approach you are taking with Puppet is already baked into the
Chef Client.

Where you’re used to the Puppet Master assembling a manifest and
shipping it to the Client, the analogous thing in Chef is the Resource
Collection. That is a dynamically created object that is created in the
Chef Client, rather than the server. Since the Client creates it, you
can more easily dynamically change the contents of the Resource
Collection based on the existing state of the node and the information
that the node can gather about its surroundings, and the processing
overhead is assumed entirely by the Client, not the Server.

TL;DR is don’t blindly treat Chef like Puppet right out of the gate, you
may have learned the wrong lessons.