Chef-provisioning / knife-zero

Chef Infra (archive)
1

Hi everyone!

I’m currently maintaining a couple of small (<25 nodes) knife-solo[1] and knife-zero[2] setups.

Typically the setups are too small to run a traditional chef-server setup, also the owners don’t want to rely on third party services (data protection, latency) - so hosted chef is not a solution.

Recently I’ve re-evaluated the current state of chef-provisioning and it looks way better than a couple of months ago. However I’m not sure how this is meant to replace the “knife-*” solutions out there: While setting up a couple of machines with attributes/run-list is very nice using the Chef-Provisioning Chef-DSL, my customers usually don’t want to (re-)converge all nodes but run chef-client explicitly on one (or more) nodes.

Example provisioning recipe: https://gist.github.com/rmoriz/f293ca1402591c5469ae

I can run “bundle exec chef-client -z provisioning/test.rb” and the instances will be provisioned, bootstrapped and converged. awesome!
But how do I later use chef-zero to converge e.g. only the nginx node? When I re-run the recipe, all nodes will be converged.

As far as I understand, I could add some attributes or even (ab)use local environment variables to execute only specific resources of the chef-provisinioning cookbook, but I’m sure this is not the right way to do it, right?

A “one node per recipe”-approach seems to be very non-DRY to me and breaks all the nice “10.times do |i| … end” examples to create a bunch of identical nodes.

I’m missing something like knife-zero in the mix, unfortunately it currently use a different port for knife-zero forwarding which currently makes it incompatible.

Any hints? Thanks :slight_smile:

best regards
Roland

[1] https://github.com/matschaffer/knife-solo
[2] https://github.com/higanworks/knife-zero

2

As I understand it (and one of the creators/maintainers of
chef-provisioning please step in and correct me if I'm wrong)
chef-provisioning's intended use is for the initial bootstrapping of the
machine, and it would 'replace' commands like knife ec2 server create. The
initial converge when creating the machine would then use something like
the chef-client cookbook to set up chef client against the chef server and
you would use daemonized/cronned/manual chef client runs on the server
itself against the chef server. After the initial bootstrap is complete,
chef-provisioning itself would never run a converge unless the machine
needed to be recreated.

Obviously this is a little different than your use case, and it sounds
similar to something I have done at
https://mharrison.org/post/chef-provisioning-ssh/. However, in the examples
there, I have just set 'converge true' in the machine definition to run a
chef converge every time on all machines, and ignored the problem of what
to do if you only want to converge one machine.

I suspect your idea of using environment variables, or providing json
attributes on the command line with something like "chef-client -z -j
<(echo '{"converge_node": "mynode"}')", is the way to go for now. Compare
that value with the current node name and add 'converge true' to the
machine definition if it matches. The other nodes won't converge (but chef
provisioning will make sure they exist).

On Fri, May 29, 2015 at 6:03 AM, Roland Moriz rmoriz@gmail.com wrote:

Hi everyone!

I'm currently maintaining a couple of small (<25 nodes) knife-solo[1] and
knife-zero[2] setups.

Typically the setups are too small to run a traditional chef-server setup,
also the owners don't want to rely on third party services (data
protection, latency) - so hosted chef is not a solution.

Recently I've re-evaluated the current state of chef-provisioning and it
looks way better than a couple of months ago. However I'm not sure how this
is meant to replace the "knife-*" solutions out there: While setting up a
couple of machines with attributes/run-list is very nice using the
Chef-Provisioning Chef-DSL, my customers usually don't want to
(re-)converge all nodes but run chef-client explicitly on one (or more)
nodes.

Example provisioning recipe:
Chef-Provisioning example · GitHub

I can run "bundle exec chef-client -z provisioning/test.rb" and the
instances will be provisioned, bootstrapped and converged. awesome!
But how do I later use chef-zero to converge e.g. only the nginx node?
When I re-run the recipe, all nodes will be converged.

As far as I understand, I could add some attributes or even (ab)use local
environment variables to execute only specific resources of the
chef-provisinioning cookbook, but I'm sure this is not the right way to do
it, right?

A "one node per recipe"-approach seems to be very non-DRY to me and breaks
all the nice "10.times do |i| … end" examples to create a bunch of
identical nodes.

I'm missing something like knife-zero in the mix, unfortunately it
currently use a different port for knife-zero forwarding which currently
makes it incompatible.

Any hints? Thanks :slight_smile:

best regards
Roland

[1] GitHub - matschaffer/knife-solo: DEPRECATED: Please consider using https://knife-zero.github.io/, ansible, or visit https://www.chef.io/ for other ideas
[2] GitHub - higanworks/knife-zero: Run chef-client/cinc-client at remote node with chef-zero(local-mode) via HTTP over SSH port fowarding.

3

On Friday, May 29, 2015 at 10:45 AM, Mark Harrison wrote:

As I understand it (and one of the creators/maintainers of chef-provisioning please step in and correct me if I'm wrong) chef-provisioning's intended use is for the initial bootstrapping of the machine, and it would 'replace' commands like knife ec2 server create. The initial converge when creating the machine would then use something like the chef-client cookbook to set up chef client against the chef server and you would use daemonized/cronned/manual chef client runs on the server itself against the chef server. After the initial bootstrap is complete, chef-provisioning itself would never run a converge unless the machine needed to be recreated.

Chef Provisioning knows when you’ve updated the run_list or node attributes, and will reconverge a machine if it detects that one of these things has changed. Otherwise, it assumes the node is up to date. I don’t think it can tell if cookbook have been updated outside of Chef Provisioning (e.g., you run Chef Provisioning, update some cookbooks, then re-run Chef Provisioning).

Obviously this is a little different than your use case, and it sounds similar to something I have done at https://mharrison.org/post/chef-provisioning-ssh/ (https://mharrison.org/post/chef-provisioning-ssh/). However, in the examples there, I have just set 'converge true' in the machine definition to run a chef converge every time on all machines, and ignored the problem of what to do if you only want to converge one machine.

I suspect your idea of using environment variables, or providing json attributes on the command line with something like "chef-client -z -j <(echo '{"converge_node": "mynode"}')", is the way to go for now. Compare that value with the current node name and add 'converge true' to the machine definition if it matches. The other nodes won't converge (but chef provisioning will make sure they exist).

On Fri, May 29, 2015 at 6:03 AM, Roland Moriz <rmoriz@gmail.com (mailto:rmoriz@gmail.com)> wrote:

Hi everyone!

I'm currently maintaining a couple of small (<25 nodes) knife-solo[1] and knife-zero[2] setups.

Typically the setups are too small to run a traditional chef-server setup, also the owners don't want to rely on third party services (data protection, latency) - so hosted chef is not a solution.

Recently I've re-evaluated the current state of chef-provisioning and it looks way better than a couple of months ago. However I'm not sure how this is meant to replace the "knife-*" solutions out there: While setting up a couple of machines with attributes/run-list is very nice using the Chef-Provisioning Chef-DSL, my customers usually don't want to (re-)converge all nodes but run chef-client explicitly on one (or more) nodes.

Example provisioning recipe: Chef-Provisioning example · GitHub

I can run "bundle exec chef-client -z provisioning/test.rb" and the instances will be provisioned, bootstrapped and converged. awesome!
But how do I later use chef-zero to converge e.g. only the nginx node? When I re-run the recipe, all nodes will be converged.

As far as I understand, I could add some attributes or even (ab)use local environment variables to execute only specific resources of the chef-provisinioning cookbook, but I'm sure this is not the right way to do it, right?

A "one node per recipe"-approach seems to be very non-DRY to me and breaks all the nice "10.times do |i| … end" examples to create a bunch of identical nodes.
You could hypothetically have each component in your cluster have its own recipe, then include_recipe each of them into a build_the_whole_cluster.rb recipe. Though that would prevent you from building the entire cluster simultaneously, it would create things in a component-wise fashion. Also you wouldn’t be able to converge just one of the N machines of a given type.

I'm missing something like knife-zero in the mix, unfortunately it currently use a different port for knife-zero forwarding which currently makes it incompatible.

Any hints? Thanks :slight_smile:

best regards
Roland

[1] GitHub - matschaffer/knife-solo: DEPRECATED: Please consider using https://knife-zero.github.io/, ansible, or visit https://www.chef.io/ for other ideas
[2] GitHub - higanworks/knife-zero: Run chef-client/cinc-client at remote node with chef-zero(local-mode) via HTTP over SSH port fowarding.

--
Daniel DeLeo

4

Hi,

Am 29.05.2015 um 19:56 schrieb Daniel DeLeo dan@kallistec.com:
On Friday, May 29, 2015 at 10:45 AM, Mark Harrison wrote:

As I understand it (and one of the creators/maintainers of chef-provisioning please step in and correct me if I'm wrong) chef-provisioning's intended use is for the initial bootstrapping of the machine, and it would 'replace' commands like knife ec2 server create. The initial converge when creating the machine would then use something like the chef-client cookbook to set up chef client against the chef server and you would use daemonized/cronned/manual chef client runs on the server itself against the chef server. After the initial bootstrap is complete, chef-provisioning itself would never run a converge unless the machine needed to be recreated.

Chef Provisioning knows when you’ve updated the run_list or node attributes, and will reconverge a machine if it detects that one of these things has changed. Otherwise, it assumes the node is up to date. I don’t think it can tell if cookbook have been updated outside of Chef Provisioning (e.g., you run Chef Provisioning, update some cookbooks, then re-run Chef Provisioning).

Thanks for all replies, I've found a working solution:

  1. Initial bootstrapping with chef-provisioning and the "with_chef_local_server " option which starts a local chef-zero instance:

bundle exec chef-client -z provisioning/example.rb

Example: Chef-Provisioning example · GitHub

  1. single node converging with knife-zero. The key is to configure knife-zero to use the same port number when SSH forwarding the local knife-zero instance to the nodes as chef-provisioning does. See Chef-Provisioning support · Issue #31 · higanworks/knife-zero · GitHub for the knife.rb config option.

    knife zero chef_client "name:<node>"

This combination is IMHO the best "server-less" approach possible with chef because both use chef-zero.

I'll now retiring all my knife-solo/chef-solo setups as soon as possible and migrate to this combination.

best regards
Roland

5

Hi Roland,

if you are managing a fixed set of nodes you might also want to take a look
at vagrant [0]. It plays nice with chef_zero (and all the other chef
provisioners) and allows you to define your nodes in a "Vagrantfile". You
give them a name and then provision them via vagrant provision <name> for
example.

It's just another option, a bit less chef specific though. Depends on what
you need and want...

HTH, Torben

[0] https://www.vagrantup.com/

On Fri, May 29, 2015 at 8:10 PM, Roland Moriz rmoriz@gmail.com wrote:

Hi,

Am 29.05.2015 um 19:56 schrieb Daniel DeLeo dan@kallistec.com:
On Friday, May 29, 2015 at 10:45 AM, Mark Harrison wrote:

As I understand it (and one of the creators/maintainers of
chef-provisioning please step in and correct me if I'm wrong)
chef-provisioning's intended use is for the initial bootstrapping of the
machine, and it would 'replace' commands like knife ec2 server create. The
initial converge when creating the machine would then use something like
the chef-client cookbook to set up chef client against the chef server and
you would use daemonized/cronned/manual chef client runs on the server
itself against the chef server. After the initial bootstrap is complete,
chef-provisioning itself would never run a converge unless the machine
needed to be recreated.

Chef Provisioning knows when you’ve updated the run_list or node
attributes, and will reconverge a machine if it detects that one of these
things has changed. Otherwise, it assumes the node is up to date. I don’t
think it can tell if cookbook have been updated outside of Chef
Provisioning (e.g., you run Chef Provisioning, update some cookbooks, then
re-run Chef Provisioning).

Thanks for all replies, I've found a working solution:

  1. Initial bootstrapping with chef-provisioning and the
    "with_chef_local_server " option which starts a local chef-zero instance:

bundle exec chef-client -z provisioning/example.rb

Example: Chef-Provisioning example · GitHub

  1. single node converging with knife-zero. The key is to configure
    knife-zero to use the same port number when SSH forwarding the local
    knife-zero instance to the nodes as chef-provisioning does. See
    Chef-Provisioning support · Issue #31 · higanworks/knife-zero · GitHub for the knife.rb
    config option.

    knife zero chef_client "name:<node>"

This combination is IMHO the best "server-less" approach possible with
chef because both use chef-zero.

I'll now retiring all my knife-solo/chef-solo setups as soon as possible
and migrate to this combination.

best regards
Roland

6

Hi,

Am 29.05.2015 um 21:55 schrieb Torben Knerr mail@tknerr.de:

Hi Roland,

if you are managing a fixed set of nodes you might also want to take a look at vagrant [0]. It plays nice with chef_zero (and all the other chef provisioners) and allows you to define your nodes in a "Vagrantfile". You give them a name and then provision them via vagrant provision <name> for example.

I’m aware of this but i really like the "all-chef-approach" of chef-provisioning, e.g. converge the chef-provisioning recipe locally on your node.

I also think that Vagrant’s external provider plugins will probably suffer under the impression of the Terraform project, which kind of replaces Vagrant for that. Terraform is nice but it’s another „external“ tool (and it doesn’t support chef-zero atm).

regards
Roland

It's just another option, a bit less chef specific though. Depends on what you need and want...

HTH, Torben

[0] https://www.vagrantup.com/ https://www.vagrantup.com/

On Fri, May 29, 2015 at 8:10 PM, Roland Moriz <rmoriz@gmail.com mailto:rmoriz@gmail.com> wrote:
Hi,

Am 29.05.2015 um 19:56 schrieb Daniel DeLeo <dan@kallistec.com mailto:dan@kallistec.com>:
On Friday, May 29, 2015 at 10:45 AM, Mark Harrison wrote:

As I understand it (and one of the creators/maintainers of chef-provisioning please step in and correct me if I'm wrong) chef-provisioning's intended use is for the initial bootstrapping of the machine, and it would 'replace' commands like knife ec2 server create. The initial converge when creating the machine would then use something like the chef-client cookbook to set up chef client against the chef server and you would use daemonized/cronned/manual chef client runs on the server itself against the chef server. After the initial bootstrap is complete, chef-provisioning itself would never run a converge unless the machine needed to be recreated.

Chef Provisioning knows when you’ve updated the run_list or node attributes, and will reconverge a machine if it detects that one of these things has changed. Otherwise, it assumes the node is up to date. I don’t think it can tell if cookbook have been updated outside of Chef Provisioning (e.g., you run Chef Provisioning, update some cookbooks, then re-run Chef Provisioning).

Thanks for all replies, I've found a working solution:

  1. Initial bootstrapping with chef-provisioning and the "with_chef_local_server " option which starts a local chef-zero instance:

bundle exec chef-client -z provisioning/example.rb

Example: Chef-Provisioning example · GitHub https://gist.github.com/rmoriz/f293ca1402591c5469ae

  1. single node converging with knife-zero. The key is to configure knife-zero to use the same port number when SSH forwarding the local knife-zero instance to the nodes as chef-provisioning does. See Chef-Provisioning support · Issue #31 · higanworks/knife-zero · GitHub https://github.com/higanworks/knife-zero/issues/31 for the knife.rb config option.

    knife zero chef_client "name:<node>"

This combination is IMHO the best "server-less" approach possible with chef because both use chef-zero.

I'll now retiring all my knife-solo/chef-solo setups as soon as possible and migrate to this combination.

best regards
Roland