Testing idempotence of chef cookbooks

Chef Infra (archive)
1

Hi,

is there a general way or best practice for testing idempotence of chef
cookbooks?

best regards,
Johannes

ARZ Allgemeines Rechenzentrum GmbH
Architekturen und Plattformen

A-6020 Innsbruck, Tschamlerstrasse 2
Tel.: +43 / (0)50 4009-1521
Fax: +43 / (0)50 4007-1521
e-Mail: Johannes.Kleinlercher@arz.at
http://www.arz.at
DVR: 0419427

2

Hi,

Am 20.05.2015 um 10:21 schrieb johannes.kleinlercher@arz.at:

Hi,

is there a general way or best practice for testing idempotence of chef cookbooks?

An approach might be to use test-kitchen + duplicate suite names + a custom chef-handler that writes the number of updated resources to disk + serverspec to verify it.

e.g. .kitchen.yaml

-8<---
suites:

  • name: default
    run_list:

    • recipe[chef-handler]
    • recipe[build-essential]
      attributes:

  • name: default
    run_list:

    • recipe[chef-handler]
    • recipe[build-essential]
      attributes:
      -8<---

this will converge the node twice on „kitchen converge“.

An example for a simple chef handler: https://github.com/jtimberman/chef-handler-updated-resources/blob/master/lib/chef/handler/updated_resources.rb https://github.com/jtimberman/chef-handler-updated-resources/blob/master/lib/chef/handler/updated_resources.rb - use chef_handler versions https://supermarket.chef.io/cookbooks/chef_handler to deploy the handler to your testing node.

Instead of logging to Chef::Log you could just write the number of updated resources to a log file e.g. /tmp/updated_resources and use Serverspec - Resource Types http://serverspec.org/resource_types.html#file to verify.

hth

best regards
Roland

3

Isn’t your way the same as running the converge command twice and looking
at the last line of the chef-client output? If so, this way sounds a
little bit easier for me.

    Chef Client finished, 45/61 resources updated in 16.28070151

seconds

If so, what is really the goal of testing idempotence? Should I really
have a “0 resources updated”?
By the way, what do the two numbers mean actually?

regards,
Johannes

Von: Roland Moriz rmoriz@gmail.com
An: chef@lists.opscode.com
Datum: 20.05.2015 13:10
Betreff: [chef] Re: Testing idempotence of chef cookbooks

Hi,

Am 20.05.2015 um 10:21 schrieb johannes.kleinlercher@arz.at:

Hi,

is there a general way or best practice for testing idempotence of chef
cookbooks?

An approach might be to use test-kitchen + duplicate suite names + a
custom chef-handler that writes the number of updated resources to disk +
serverspec to verify it.

e.g. .kitchen.yaml

-8<—
suites:

  • name: default
    run_list:

    • recipe[chef-handler]
    • recipe[build-essential]
      attributes:

  • name: default
    run_list:

    • recipe[chef-handler]
    • recipe[build-essential]
      attributes:
      -8<—

this will converge the node twice on „kitchen converge“.

An example for a simple chef handler:

Instead of logging to Chef::Log you could just write the number of updated
resources to a log file e.g. /tmp/updated_resources and use
http://serverspec.org/resource_types.html#file to verify.

hth

best regards
Roland

4

yeah, its should be 0 resources updated if everything is idempotent and
already converged. those two numbers relfect number of updated resources /
number of total resources. Total resources is sum of all resources defined
by all recipes, lwrps as part of the expanded run list, while updated
resources are a subset of them which was modified (like absent file being
created, absent packaged being installed etc) during the chef run.

On Thu, May 21, 2015 at 8:01 AM, johannes.kleinlercher@arz.at wrote:

Isn't your way the same as running the converge command twice and looking
at the last line of the chef-client output? If so, this way sounds a little
bit easier for me.

    Chef Client finished, 45/61 resources updated in 16.28070151

seconds

If so, what is really the goal of testing idempotence? Should I really
have a "0 resources updated"?
By the way, what do the two numbers mean actually?

regards,
Johannes

Von: Roland Moriz rmoriz@gmail.com
An: chef@lists.opscode.com
Datum: 20.05.2015 13:10
Betreff: [chef] Re: Testing idempotence of chef cookbooks

Hi,

Am 20.05.2015 um 10:21 schrieb johannes.kleinlercher@arz.at
johannes.kleinlercher@arz.at:

Hi,

is there a general way or best practice for testing idempotence of chef
cookbooks?

An approach might be to use test-kitchen + duplicate suite names + a
custom chef-handler that writes the number of updated resources to disk +
serverspec to verify it.

e.g. .kitchen.yaml

-8<---
suites:

  • name: default
    run_list:

    • recipe[chef-handler]
    • recipe[build-essential]
      attributes:

  • name: default
    run_list:

    • recipe[chef-handler]
    • recipe[build-essential]
      attributes:
      -8<---

this will converge the node twice on „kitchen converge“.

An example for a simple chef handler:
https://github.com/jtimberman/chef-handler-updated-resources/blob/master/lib/chef/handler/updated_resources.rb
https://github.com/jtimberman/chef-handler-updated-resources/blob/master/lib/chef/handler/updated_resources.rb

Instead of logging to Chef::Log you could just write the number of updated
resources to a log file e.g. /tmp/updated_resources and use
Serverspec - Resource Types
http://serverspec.org/resource_types.html#file to verify.

hth

best regards
Roland

5

Am 21.05.2015 um 17:30 schrieb Ranjib Dey dey.ranjib@gmail.com:

yeah, its should be 0 resources updated if everything is idempotent and already converged. those two numbers relfect number of updated resources / number of total resources. Total resources is sum of all resources defined by all recipes, lwrps as part of the expanded run list, while updated resources are a subset of them which was modified (like absent file being created, absent packaged being installed etc) during the chef run.

Some resources are very hard to make „chef-idempotent“, e.g. ruby blocks. Even if you make sure your ruby code is idempotent, the execution of the block already counts as „updated resource“. Even popular community cookbooks do have that "problem“.

You probably want to add each converge result to a temp file and calculate/check the difference or some kind of percentage instead.

regards
Roland

On Thu, May 21, 2015 at 8:01 AM, <johannes.kleinlercher@arz.at mailto:johannes.kleinlercher@arz.at> wrote:
Isn't your way the same as running the converge command twice and looking at the last line of the chef-client output? If so, this way sounds a little bit easier for me.

    Chef Client finished, 45/61 resources updated in 16.28070151 seconds

If so, what is really the goal of testing idempotence? Should I really have a "0 resources updated"?
By the way, what do the two numbers mean actually?

regards,
Johannes

Von: Roland Moriz <rmoriz@gmail.com mailto:rmoriz@gmail.com>
An: chef@lists.opscode.com mailto:chef@lists.opscode.com
Datum: 20.05.2015 13:10
Betreff: [chef] Re: Testing idempotence of chef cookbooks

Hi,

Am 20.05.2015 um 10:21 schrieb johannes.kleinlercher@arz.at mailto:johannes.kleinlercher@arz.at:

Hi,

is there a general way or best practice for testing idempotence of chef cookbooks?

An approach might be to use test-kitchen + duplicate suite names + a custom chef-handler that writes the number of updated resources to disk + serverspec to verify it.

e.g. .kitchen.yaml

-8<---
suites:

  • name: default
    run_list:

    • recipe[chef-handler]
    • recipe[build-essential]
      attributes:

  • name: default
    run_list:

    • recipe[chef-handler]
    • recipe[build-essential]
      attributes:
      -8<---

this will converge the node twice on „kitchen converge“.

An example for a simple chef handler: https://github.com/jtimberman/chef-handler-updated-resources/blob/master/lib/chef/handler/updated_resources.rb https://github.com/jtimberman/chef-handler-updated-resources/blob/master/lib/chef/handler/updated_resources.rb - use chef_handler versions https://supermarket.chef.io/cookbooks/chef_handler to deploy the handler to your testing node.

Instead of logging to Chef::Log you could just write the number of updated resources to a log file e.g. /tmp/updated_resources and use Serverspec - Resource Types http://serverspec.org/resource_types.html#file to verify.

hth

best regards
Roland

6

Ok, I see. Well maybe I understood something wrong, but from my
understanding idempotence is the property when you apply a cookbook
several times and the result is always the same. So I always thought it is
enough to run converge multiple times and see if the serverspec tests are
always the same. That is also something you can easily automate for a CI
environment.

The property that only resources are updated that are really needed is
more some kind of efficiency of cookbooks, isn’t it?

And as you described it is not always possible to have “0 resources
updated” so I wonder if such idempotence tests can be standardized or
automated at all.

Von: Roland Moriz rmoriz@gmail.com
An: chef@lists.opscode.com
Datum: 22.05.2015 04:34
Betreff: [chef] Re: Antwort: Re: Testing idempotence of chef
cookbooks

Am 21.05.2015 um 17:30 schrieb Ranjib Dey dey.ranjib@gmail.com:

yeah, its should be 0 resources updated if everything is idempotent and
already converged. those two numbers relfect number of updated resources /
number of total resources. Total resources is sum of all resources defined
by all recipes, lwrps as part of the expanded run list, while updated
resources are a subset of them which was modified (like absent file being
created, absent packaged being installed etc) during the chef run.

Some resources are very hard to make „chef-idempotent“, e.g. ruby blocks.
Even if you make sure your ruby code is idempotent, the execution of the
block already counts as „updated resource“. Even popular community
cookbooks do have that "problem“.

You probably want to add each converge result to a temp file and
calculate/check the difference or some kind of percentage instead.

regards
Roland

On Thu, May 21, 2015 at 8:01 AM, johannes.kleinlercher@arz.at wrote:
Isn’t your way the same as running the converge command twice and looking
at the last line of the chef-client output? If so, this way sounds a
little bit easier for me.

    Chef Client finished, 45/61 resources updated in 16.28070151

seconds

If so, what is really the goal of testing idempotence? Should I really
have a “0 resources updated”?
By the way, what do the two numbers mean actually?

regards,
Johannes

Von: Roland Moriz rmoriz@gmail.com
An: chef@lists.opscode.com
Datum: 20.05.2015 13:10
Betreff: [chef] Re: Testing idempotence of chef cookbooks

Hi,

Am 20.05.2015 um 10:21 schrieb johannes.kleinlercher@arz.at:

Hi,

is there a general way or best practice for testing idempotence of chef
cookbooks?

An approach might be to use test-kitchen + duplicate suite names + a
custom chef-handler that writes the number of updated resources to disk +
serverspec to verify it.

e.g. .kitchen.yaml

-8<—
suites:

  • name: default
    run_list:

    • recipe[chef-handler]
    • recipe[build-essential]
      attributes:

  • name: default
    run_list:

    • recipe[chef-handler]
    • recipe[build-essential]
      attributes:
      -8<—

this will converge the node twice on „kitchen converge“.

An example for a simple chef handler:

Instead of logging to Chef::Log you could just write the number of updated
resources to a log file e.g. /tmp/updated_resources and use
http://serverspec.org/resource_types.html#file to verify.

hth

best regards
Roland

7

yes, you can. But as roland is pointing out, you can have non-idempotent
resources, i.e. resource that are always updating (like execute, bash,
ruby_block), this does necessarily not mean something is wrong, neither it
means they cant be made idempotent (with proper resource guards).
even if there are some resources which are always being updated, you can
white-list them , since they are known, and then check for any additional
resources (as you dont want new ones).
note. we are equating idempotency as not being updated in every run,
but its lot more complicated than that. Sean Omeara wrote some stuff on
idempotency vs convergence.

for all practical purpose, you can just drop statsd or something similar
metrics about updated resources, and alert on increased rates (i.e. first
derivative).

On Thu, May 21, 2015 at 11:45 PM, johannes.kleinlercher@arz.at wrote:

Ok, I see. Well maybe I understood something wrong, but from my
understanding idempotence is the property when you apply a cookbook several
times and the result is always the same. So I always thought it is enough
to run converge multiple times and see if the serverspec tests are always
the same. That is also something you can easily automate for a CI
environment.

The property that only resources are updated that are really needed is
more some kind of efficiency of cookbooks, isn't it?

And as you described it is not always possible to have "0 resources
updated" so I wonder if such idempotence tests can be standardized or
automated at all.

Von: Roland Moriz rmoriz@gmail.com
An: chef@lists.opscode.com
Datum: 22.05.2015 04:34
Betreff: [chef] Re: Antwort: Re: Testing idempotence of chef
cookbooks

Am 21.05.2015 um 17:30 schrieb Ranjib Dey <dey.ranjib@gmail.com
dey.ranjib@gmail.com>:

yeah, its should be 0 resources updated if everything is idempotent and
already converged. those two numbers relfect number of updated resources /
number of total resources. Total resources is sum of all resources defined
by all recipes, lwrps as part of the expanded run list, while updated
resources are a subset of them which was modified (like absent file being
created, absent packaged being installed etc) during the chef run.

Some resources are very hard to make „chef-idempotent“, e.g. ruby blocks.
Even if you make sure your ruby code is idempotent, the execution of the
block already counts as „updated resource“. Even popular community
cookbooks do have that "problem“.

You probably want to add each converge result to a temp file and
calculate/check the difference or some kind of percentage instead.

regards
Roland

On Thu, May 21, 2015 at 8:01 AM, <johannes.kleinlercher@arz.at
johannes.kleinlercher@arz.at> wrote:
Isn't your way the same as running the converge command twice and looking
at the last line of the chef-client output? If so, this way sounds a little
bit easier for me.

   Chef Client finished, 45/61 resources updated in 16.28070151 seconds

If so, what is really the goal of testing idempotence? Should I really
have a "0 resources updated"?
By the way, what do the two numbers mean actually?

regards,
Johannes

Von: Roland Moriz <rmoriz@gmail.com rmoriz@gmail.com>
An: chef@lists.opscode.com chef@lists.opscode.com
Datum: 20.05.2015 13:10
Betreff: [chef] Re: Testing idempotence of chef cookbooks

Hi,

Am 20.05.2015 um 10:21 schrieb johannes.kleinlercher@arz.at
johannes.kleinlercher@arz.at:

Hi,

is there a general way or best practice for testing idempotence of chef
cookbooks?

An approach might be to use test-kitchen + duplicate suite names + a
custom chef-handler that writes the number of updated resources to disk +
serverspec to verify it.

e.g. .kitchen.yaml

-8<---
suites:

  • name: default
    run_list:

    • recipe[chef-handler]
    • recipe[build-essential]
      attributes:

  • name: default
    run_list:

    • recipe[chef-handler]
    • recipe[build-essential]
      attributes:
      -8<---

this will converge the node twice on „kitchen converge“.

An example for a simple chef handler:
https://github.com/jtimberman/chef-handler-updated-resources/blob/master/lib/chef/handler/updated_resources.rb
https://github.com/jtimberman/chef-handler-updated-resources/blob/master/lib/chef/handler/updated_resources.rb

Instead of logging to Chef::Log you could just write the number of updated
resources to a log file e.g. /tmp/updated_resources and use
Serverspec - Resource Types
http://serverspec.org/resource_types.html#file to verify.

hth

best regards
Roland

8

On Fri, May 22, 2015 at 2:45 AM, johannes.kleinlercher@arz.at wrote:

Ok, I see. Well maybe I understood something wrong, but from my
understanding idempotence is the property when you apply a cookbook several
times and the result is always the same. So I always thought it is enough to

It is, and what people frequently refer to as "idempotence" is
actually "convergence", but I've stopped well-actuallying people on
this :slight_smile:

  • Julian

--
[ Julian C. Dunn jdunn@aquezada.com * Sorry, I'm ]
[ WWW: Julian Dunn's Blog - Commentary on media, technology, and everything in between. * only Web 1.0 ]
[ gopher://sdf.org/1/users/keymaker/ * compliant! ]
[ PGP: 91B3 7A9D 683C 7C16 715F 442C 6065 D533 FDC2 05B9 ]