Automated testing of cookbooks


#1

I noticed that Bryan Berry suggested “cookbook testing” as a topic for the
community summit.

 "Cookbook testing and how to automate it, esp. for opscode/cookbooks"

 http://wiki.opscode.com/pages/diffpagesbyversion.action?pageId=20414482&selectedPageVersions=3&selectedPageVersions=2

I’m also interested in that topic. As someone responsible for a
relatively small number of nodes, testability (and the prerequisite
repeatability) have been my biggest benefits from Chef.

So far, what we’ve done for testing is to use rspec for implementing
tests. Here’s an example:

 it "should respond on port 80" do
   lambda {
     TCPSocket.open(@server, 'http')
   }.should_not raise_error
 end

Before running the tests, I have to manually bootstrap a node using knife.
If my instance is the only one in its environment, the spec can find it
using knife’s search feature. The bootstrap takes a few minutes, and the
20 or so tests take about half a minute to run.

While I’m iteratively developing a recipe, my work cycle is to edit
source, upload a cookbook, and rerun chef-client (usually by rerunning
knife boostrap, because the execution environment is different from
invoking chef-client directly). This feels a bit slower than the cycle
I’m used to when coding in Ruby because of the upload and bootstrap steps.

I like rspec over other testing tools because of how it generates handy
reports, such as this one, which displays an English list of covered test
cases:

 $ rspec spec/ -f doc

 Foo role
   should respond on port 80
   should run memcached
   should accept memcached connections
   should have mysql account
   should allow passwordless sudo to user foo as user bar
   should allow passwordless sudo to root as a member of sysadmin
   should allow key login as user bar
   should mount homedirs on ext4, not NFS
   should rotate production.log
   should have baz as default vhost
   ...

That sample report also gives a feel for sort of things we check. So far,
nearly all of our checks are non-intrusive enough to run on a production
system. (The exception is testing of local email delivery
configurations.)

Areas I’d love to see improvement:

 * Shortening the edit-upload-bootstrap-test cycle
 * Automating the bootstrap in the context of testing
 * Adding rspec primitives for Chef-related testing, which might
   include support for multiple platforms

As an example of rspec primitives, instead of:

 it "should respond on port 80" do
   lambda {
     TCPSocket.open(@server, 'http')
   }.should_not raise_error
 end

I’d like to write:

 it { should respond_on_port(80) }

Rspec supports the the syntactic sugar; it’s just a matter of adding some
"matcher" plugins.

How do other chef users verify that recipes work as expected?

I’m not sure how applicable my approach is to opscode/cookbooks because
it relies on having a specific server configuration and can only test a
cookbook in the context of that single server. If we automated the
boostrap step so it could be embedded into the rspec setup blocks, it
would be possible to test a cookbook in several sample contexts, but the
time required to setup each server instance might be prohibitive.

Marcel


#2

I have used both Rspec + ssh as well as fujin’s minitest report handler,
now pretty happy with the report handler based approach, i guess rspec
instead of minitest will be a good stuff try out.

On Wed, Nov 23, 2011 at 9:30 AM, Marcel M. Cary marcel@oak.homeunix.orgwrote:

I noticed that Bryan Berry suggested “cookbook testing” as a topic for the
community summit.

“Cookbook testing and how to automate it, esp. for opscode/cookbooks”

http://wiki.opscode.com/pages/diffpagesbyversion.action?
pageId=20414482&**selectedPageVersions=3&**selectedPageVersions=2http://wiki.opscode.com/pages/diffpagesbyversion.action?pageId=20414482&selectedPageVersions=3&selectedPageVersions=2

I’m also interested in that topic. As someone responsible for a
relatively small number of nodes, testability (and the prerequisite
repeatability) have been my biggest benefits from Chef.

So far, what we’ve done for testing is to use rspec for implementing
tests. Here’s an example:

it “should respond on port 80” do
lambda {
TCPSocket.open(@server, ‘http’)
}.should_not raise_error
end

Before running the tests, I have to manually bootstrap a node using knife.
If my instance is the only one in its environment, the spec can find it
using knife’s search feature. The bootstrap takes a few minutes, and the
20 or so tests take about half a minute to run.

While I’m iteratively developing a recipe, my work cycle is to edit
source, upload a cookbook, and rerun chef-client (usually by rerunning
knife boostrap, because the execution environment is different from
invoking chef-client directly). This feels a bit slower than the cycle I’m
used to when coding in Ruby because of the upload and bootstrap steps.

I like rspec over other testing tools because of how it generates handy
reports, such as this one, which displays an English list of covered test
cases:

$ rspec spec/ -f doc

Foo role
should respond on port 80
should run memcached
should accept memcached connections
should have mysql account
should allow passwordless sudo to user foo as user bar
should allow passwordless sudo to root as a member of sysadmin
should allow key login as user bar
should mount homedirs on ext4, not NFS
should rotate production.log
should have baz as default vhost

That sample report also gives a feel for sort of things we check. So far,
nearly all of our checks are non-intrusive enough to run on a production
system. (The exception is testing of local email delivery configurations.)

Areas I’d love to see improvement:

  • Shortening the edit-upload-bootstrap-test cycle
  • Automating the bootstrap in the context of testing
  • Adding rspec primitives for Chef-related testing, which might
    include support for multiple platforms

As an example of rspec primitives, instead of:

it “should respond on port 80” do
lambda {
TCPSocket.open(@server, ‘http’)
}.should_not raise_error
end

I’d like to write:

it { should respond_on_port(80) }

Rspec supports the the syntactic sugar; it’s just a matter of adding some
"matcher" plugins.

How do other chef users verify that recipes work as expected?

I’m not sure how applicable my approach is to opscode/cookbooks because it
relies on having a specific server configuration and can only test a
cookbook in the context of that single server. If we automated the
boostrap step so it could be embedded into the rspec setup blocks, it would
be possible to test a cookbook in several sample contexts, but the time
required to setup each server instance might be prohibitive.

Marcel


#3

As was mentioned, I have been using my MiniTest cookbook [0] (w/
Report handler) to provide a minimal DSL around the Test Unit style
tests that MiniTest supports. It would be pretty easy to support
MiniTests’ Spec style, as well as potentially abstracting it to run
RSpec tests…

Some examples are here [1]

–AJ

[0] https://github.com/fujin/minitest-cookbook
[1] https://github.com/fujin/minitest-cookbook/blob/develop/cookbooks/minitest/recipes/examples.rb#L57

On 23 November 2011 17:07, Ranjib Dey ranjibd@thoughtworks.com wrote:

I have used both Rspec + ssh as well as fujin’s minitest report handler, now
pretty happy with the report handler based approach, i guess rspec instead
of minitest will be a good stuff try out.

On Wed, Nov 23, 2011 at 9:30 AM, Marcel M. Cary marcel@oak.homeunix.org
wrote:

I noticed that Bryan Berry suggested “cookbook testing” as a topic for the
community summit.

“Cookbook testing and how to automate it, esp. for opscode/cookbooks”

http://wiki.opscode.com/pages/diffpagesbyversion.action?pageId=20414482&selectedPageVersions=3&selectedPageVersions=2

I’m also interested in that topic. As someone responsible for a
relatively small number of nodes, testability (and the prerequisite
repeatability) have been my biggest benefits from Chef.

So far, what we’ve done for testing is to use rspec for implementing
tests. Here’s an example:

it “should respond on port 80” do
lambda {
TCPSocket.open(@server, ‘http’)
}.should_not raise_error
end

Before running the tests, I have to manually bootstrap a node using knife.
If my instance is the only one in its environment, the spec can find it
using knife’s search feature. The bootstrap takes a few minutes, and the 20
or so tests take about half a minute to run.

While I’m iteratively developing a recipe, my work cycle is to edit
source, upload a cookbook, and rerun chef-client (usually by rerunning knife
boostrap, because the execution environment is different from invoking
chef-client directly). This feels a bit slower than the cycle I’m used to
when coding in Ruby because of the upload and bootstrap steps.

I like rspec over other testing tools because of how it generates handy
reports, such as this one, which displays an English list of covered test
cases:

$ rspec spec/ -f doc

Foo role
should respond on port 80
should run memcached
should accept memcached connections
should have mysql account
should allow passwordless sudo to user foo as user bar
should allow passwordless sudo to root as a member of sysadmin
should allow key login as user bar
should mount homedirs on ext4, not NFS
should rotate production.log
should have baz as default vhost

That sample report also gives a feel for sort of things we check. So far,
nearly all of our checks are non-intrusive enough to run on a production
system. (The exception is testing of local email delivery configurations.)

Areas I’d love to see improvement:

  • Shortening the edit-upload-bootstrap-test cycle
  • Automating the bootstrap in the context of testing
  • Adding rspec primitives for Chef-related testing, which might
    include support for multiple platforms

As an example of rspec primitives, instead of:

it “should respond on port 80” do
lambda {
TCPSocket.open(@server, ‘http’)
}.should_not raise_error
end

I’d like to write:

it { should respond_on_port(80) }

Rspec supports the the syntactic sugar; it’s just a matter of adding some
"matcher" plugins.

How do other chef users verify that recipes work as expected?

I’m not sure how applicable my approach is to opscode/cookbooks because it
relies on having a specific server configuration and can only test a
cookbook in the context of that single server. If we automated the boostrap
step so it could be embedded into the rspec setup blocks, it would be
possible to test a cookbook in several sample contexts, but the time
required to setup each server instance might be prohibitive.

Marcel


#4

Integration tests that exercise the service you are building
definitely give you the most bang for buck.

We found the feedback cycle slow as well so I wrote chefspec which
builds on RSpec to support unit testing cookbooks:

This basically fakes a convergence and allows you to make assertions
about the created resources. At first glance Chef’s declarative nature
makes this less useful, but once you start introducing conditional
execution I’ve found this to be a time saver.

If you’re looking to do CI (which you should be) converging twice goes
some way to verifying that your recipes are idempotent.

knife cookbook test is a useful first gate for CI:
http://wiki.opscode.com/display/chef/Knife#Knife-test

Cheers,

Andrew.

On Wed, Nov 23, 2011 at 4:21 AM, AJ Christensen aj@junglist.gen.nz wrote:

As was mentioned, I have been using my MiniTest cookbook [0] (w/
Report handler) to provide a minimal DSL around the Test Unit style
tests that MiniTest supports. It would be pretty easy to support
MiniTests’ Spec style, as well as potentially abstracting it to run
RSpec tests…

Some examples are here [1]

–AJ

[0] https://github.com/fujin/minitest-cookbook
[1] https://github.com/fujin/minitest-cookbook/blob/develop/cookbooks/minitest/recipes/examples.rb#L57

On 23 November 2011 17:07, Ranjib Dey ranjibd@thoughtworks.com wrote:

I have used both Rspec + ssh as well as fujin’s minitest report handler, now
pretty happy with the report handler based approach, i guess rspec instead
of minitest will be a good stuff try out.

On Wed, Nov 23, 2011 at 9:30 AM, Marcel M. Cary marcel@oak.homeunix.org
wrote:

I noticed that Bryan Berry suggested “cookbook testing” as a topic for the
community summit.

“Cookbook testing and how to automate it, esp. for opscode/cookbooks”

http://wiki.opscode.com/pages/diffpagesbyversion.action?pageId=20414482&selectedPageVersions=3&selectedPageVersions=2

I’m also interested in that topic. As someone responsible for a
relatively small number of nodes, testability (and the prerequisite
repeatability) have been my biggest benefits from Chef.

So far, what we’ve done for testing is to use rspec for implementing
tests. Here’s an example:

it “should respond on port 80” do
lambda {
TCPSocket.open(@server, ‘http’)
}.should_not raise_error
end

Before running the tests, I have to manually bootstrap a node using knife.
If my instance is the only one in its environment, the spec can find it
using knife’s search feature. The bootstrap takes a few minutes, and the 20
or so tests take about half a minute to run.

While I’m iteratively developing a recipe, my work cycle is to edit
source, upload a cookbook, and rerun chef-client (usually by rerunning knife
boostrap, because the execution environment is different from invoking
chef-client directly). This feels a bit slower than the cycle I’m used to
when coding in Ruby because of the upload and bootstrap steps.

I like rspec over other testing tools because of how it generates handy
reports, such as this one, which displays an English list of covered test
cases:

$ rspec spec/ -f doc

Foo role
should respond on port 80
should run memcached
should accept memcached connections
should have mysql account
should allow passwordless sudo to user foo as user bar
should allow passwordless sudo to root as a member of sysadmin
should allow key login as user bar
should mount homedirs on ext4, not NFS
should rotate production.log
should have baz as default vhost

That sample report also gives a feel for sort of things we check. So far,
nearly all of our checks are non-intrusive enough to run on a production
system. (The exception is testing of local email delivery configurations.)

Areas I’d love to see improvement:

  • Shortening the edit-upload-bootstrap-test cycle
  • Automating the bootstrap in the context of testing
  • Adding rspec primitives for Chef-related testing, which might
    include support for multiple platforms

As an example of rspec primitives, instead of:

it “should respond on port 80” do
lambda {
TCPSocket.open(@server, ‘http’)
}.should_not raise_error
end

I’d like to write:

it { should respond_on_port(80) }

Rspec supports the the syntactic sugar; it’s just a matter of adding some
"matcher" plugins.

How do other chef users verify that recipes work as expected?

I’m not sure how applicable my approach is to opscode/cookbooks because it
relies on having a specific server configuration and can only test a
cookbook in the context of that single server. If we automated the boostrap
step so it could be embedded into the rspec setup blocks, it would be
possible to test a cookbook in several sample contexts, but the time
required to setup each server instance might be prohibitive.

Marcel


#5

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I noticed that Bryan Berry suggested “cookbook testing” as a topic for the
community summit.

“Cookbook testing and how to automate it, esp. for opscode/cookbooks”

The variety of ways suggested so far to skin this cat seems to makes it seem like a ripe topic for the summit. As a new Chef adopter I’d like to see a convergence upon best practice before trying four different approaches, then building my own.

P.


Peter Burkholder | Sr. System Administrator (consultant)
AARP | Digital Strategy & Operations | 601 E Street NW | Washington, DC 20049
pburkholder@aarp.org | aim: peterbtech | w: 202-434-3530 | c: 202-344-7129
For optimal efficiency, I check email at 2-hour intervals during the workday
(except when on-call). Please use IM or phone to contact me for urgent matters

-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org

iEYEARECAAYFAk7NHkgACgkQLvy74DZcgeSIegCgywRX0tNuj5u7qZhzPfpQLnDl
H2oAoNXEOO/HPC3NODQVSruGKR52muLn
=vNa2
-----END PGP SIGNATURE-----


#6

+1. And I’d like to have options for other languages. It doesn’t
need to be in the chef DSL, but there should be a best practice to a
callout from a recipe to test and to run a test in any language, and
to be able to close the loop and confirm that the test completed and
know its status. We’d like chef to be part of a deployment system and
this is important to controlling the lifecycle of the deployment.

-Peter

On Wed, Nov 23, 2011 at 11:24 AM, Burkholder, Peter
PBurkholder@aarp.org wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I noticed that Bryan Berry suggested “cookbook testing” as a topic for the
community summit.

“Cookbook testing and how to automate it, esp. for opscode/cookbooks”

The variety of ways suggested so far to skin this cat seems to makes it seem like a ripe topic for the summit. As a new Chef adopter I’d like to see a convergence upon best practice before trying four different approaches, then building my own.

P.


Peter Burkholder | Sr. System Administrator (consultant)
AARP | Digital Strategy & Operations | 601 E Street NW | Washington, DC 20049
pburkholder@aarp.org | aim: peterbtech | w: 202-434-3530 | c: 202-344-7129
For optimal efficiency, I check email at 2-hour intervals during the workday
(except when on-call). Please use IM or phone to contact me for urgent matters

-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org

iEYEARECAAYFAk7NHkgACgkQLvy74DZcgeSIegCgywRX0tNuj5u7qZhzPfpQLnDl
H2oAoNXEOO/HPC3NODQVSruGKR52muLn
=vNa2
-----END PGP SIGNATURE-----