Integration testing Chef -- seriously, what is the best approach?

[Jay_Pipes]

Ohai Chefs!

Alrighty, so over in OpenStack + Chef world, we've recently made a lot
of progress in consolidating and cleaning up the myriad cookbook
repositories, adding ChefSpec unit tests for the cookbooks, and getting
all the cookbooks linted with FoodCritic and gated using Gerrit and
Jenkins [1]. All good stuff.

The next big piece of the puzzle is integration testing. Now, I'm
neither a Chef nor a Ruby expert, so I decided to use the Google machine
to determine what the best practices were for doing multi-machine
integration testing of a large set of interrelated cookbooks.

So far, I've been struggling. I found chef-workflow, and that seemed to
be a great solution. It's advantages are:

• pretty well doc'd
• able to use multiple provisioners
• uses minitest to define test cases
• just makes sense for integration testing -- define a workflow (set of
  tasks) that simulate your deployment, spin up a set of boxes or machines
  based on Chef roles, and validate stuff

But it has some downsides that I'm unable to get around:

• does not support Chef 11 -- and the OpenStack + Chef community has
  decided to standardize on Chef 11 and not spend time supporting 10.x
• many of the tools we use -- ChefSpec, Strainer, and others -- depend
  on much newer versions of libs (minitest, json, etc) than chef-workflow
  depends on, which makes it impossible to use in our environment [2]
• lots of "magic" that seems to get glossed over -- for instance, I
  cannot understand if Chef Solo is a requirement and if not, why it gets
  installed on all the boxes I provision -- this is probably just my fault
  in not understanding Chef/Ruby 100%, but when magic seems to happen, my
  coder ears tend to perk up -- and not in a good way

I've also read Josh Timberman's article on Test Kitchen [3] but I
believe we must have different ideas of what integration testing is. My
definition of integration testing is tests that a) don't mock anything
out and b) stress the points of interaction (integration) between
various dependent things (in this case, interrelated cookbooks). I don't
really see how Test Kitchen makes this easy, but of course, I may be
misunderstanding something. Josh, please feel free to enlighten me!

Anyway, the point of this email is to get feedback on what exactly the
state of integration testing of Chef cookbooks is today, where it's
heading, and what people feel should be the direction the OpenStack +
Chef community should go in.

Best,
-jay

[1] For more info, see here:
http://www.joinfu.com/2013/05/working-with-the-openstack-code-review-and-ci-system-chef-edition/
[2] Issue logged in chef-workflow for this... if you have suggestions or
workarounds, please do comment:
https://github.com/chef-workflow/chef-workflow/issues/11
[3]
http://jtimberman.housepub.org/blog/2012/12/13/cookbook-integration-testing-with-real-examples/

[Jesse_Nelson]

I've been down this road with our Infrastructure, and I will share my thoughts. What I started with was workflow, but as the chef11 thing came up I couldn't use that. I've been a fan and continue to be of test-kitchen 1.0 for individual cookbook testing, but the lack of being able to describe infrastructures and run chef-server to test hurts for integration. I believe this function will come to kitchen tho as many people have talked about it, and I am sure Fletcher is on board with the idea just not 100% ready to go a direction yet.

What we use now:
Thor tasks that manage knife-server and spice weasel + knife-vagrant to build out arbitrary infrastructures. I ripped this out of our repo and put it on github during chef hack-day. Example @ GitHub - spheromak/chef-infra: Thor tasks and misc stuff for building infras with chef and spice.. It's less than pretty, but it has been working for us to get our dev work up and running. Just add server-spec[1] tests and a test task to get some testing in there (or whatever you want/need). I am happy to talk more about it if there is interest, but I will just give highlights here.

This approach using a mishmash of tools allows us to swap the infrastructures descriptions out from vagrant to openstack or something else supported by knife. That was an initial requirement, but our testing needs have been driving me down a different path due to vbox being a pos, and the tests taking too long to spin up.

What I am trying to make work now:

At chef-conf I spent some time talking with Chris Roberts, and he introduced me to the thing he was working on in this realm: vagabond[2]. Which is a lxc management framework that integrates with kitchen and has its own hooks for running tests. It handles building a chef-server or chef-zero container, and making your repo assets available to it. Basically it is a better more concise version of what I had done with thor + spice + knife. Over the last week or so I have been working on getting things working with that instead. The vagabond system is much faster and easier to add tests for. Using kitchen for cooks and vagabond for integration testing is working out really nicely for me right now. There are still a lot of edges to be worked out but Chris has been hacking on vagabond pretty heavily.

Caveat emptor: Everything I've been doing in vagabond land has been against development or my own custom hacks. This thing isn't quite ready for general consumption, but It is already useful. I also believe its the right way to move on the integration testing front (at least for my needs).

1
2

Jesse

On May 31, 2013, at 8:11 AM, Jay Pipes jaypipes@gmail.com wrote:

Ohai Chefs!

Alrighty, so over in OpenStack + Chef world, we've recently made a lot
of progress in consolidating and cleaning up the myriad cookbook
repositories, adding ChefSpec unit tests for the cookbooks, and getting
all the cookbooks linted with FoodCritic and gated using Gerrit and
Jenkins [1]. All good stuff.

The next big piece of the puzzle is integration testing. Now, I'm
neither a Chef nor a Ruby expert, so I decided to use the Google machine
to determine what the best practices were for doing multi-machine
integration testing of a large set of interrelated cookbooks.

So far, I've been struggling. I found chef-workflow, and that seemed to
be a great solution. It's advantages are:

• pretty well doc'd
• able to use multiple provisioners
• uses minitest to define test cases
• just makes sense for integration testing -- define a workflow (set of
  tasks) that simulate your deployment, spin up a set of boxes or machines
  based on Chef roles, and validate stuff

But it has some downsides that I'm unable to get around:

• does not support Chef 11 -- and the OpenStack + Chef community has
  decided to standardize on Chef 11 and not spend time supporting 10.x
• many of the tools we use -- ChefSpec, Strainer, and others -- depend
  on much newer versions of libs (minitest, json, etc) than chef-workflow
  depends on, which makes it impossible to use in our environment [2]
• lots of "magic" that seems to get glossed over -- for instance, I
  cannot understand if Chef Solo is a requirement and if not, why it gets
  installed on all the boxes I provision -- this is probably just my fault
  in not understanding Chef/Ruby 100%, but when magic seems to happen, my
  coder ears tend to perk up -- and not in a good way

I've also read Josh Timberman's article on Test Kitchen [3] but I
believe we must have different ideas of what integration testing is. My
definition of integration testing is tests that a) don't mock anything
out and b) stress the points of interaction (integration) between
various dependent things (in this case, interrelated cookbooks). I don't
really see how Test Kitchen makes this easy, but of course, I may be
misunderstanding something. Josh, please feel free to enlighten me!

Anyway, the point of this email is to get feedback on what exactly the
state of integration testing of Chef cookbooks is today, where it's
heading, and what people feel should be the direction the OpenStack +
Chef community should go in.

Best,
-jay

[1] For more info, see here:
Page not found – joinfu
[2] Issue logged in chef-workflow for this... if you have suggestions or
workarounds, please do comment:
https://github.com/chef-workflow/chef-workflow/issues/11
[3]
http://jtimberman.housepub.org/blog/2012/12/13/cookbook-integration-testing-with-real-examples/

[realityforge]

Hi,

We use a heavily customised multi-vm Vagrantfile that is driven by
simple testing infrastructure.

Essentially our tests start by spinning up a chef-server, uploading
the full set of cookbooks/roles/data bags and then instantiating a set
of nodes in a particular order. We then do a few simple tests, just
getting stuff to converge successfully is often a good first step.

We went with the Vagrantfile approach as many of the current test
infrastructure did not exist when we started. Also we use windows that
is not well support in other infra.

On Sat, Jun 1, 2013 at 1:11 AM, Jay Pipes jaypipes@gmail.com wrote:

Ohai Chefs!

Alrighty, so over in OpenStack + Chef world, we've recently made a lot
of progress in consolidating and cleaning up the myriad cookbook
repositories, adding ChefSpec unit tests for the cookbooks, and getting
all the cookbooks linted with FoodCritic and gated using Gerrit and
Jenkins [1]. All good stuff.

The next big piece of the puzzle is integration testing. Now, I'm
neither a Chef nor a Ruby expert, so I decided to use the Google machine
to determine what the best practices were for doing multi-machine
integration testing of a large set of interrelated cookbooks.

So far, I've been struggling. I found chef-workflow, and that seemed to
be a great solution. It's advantages are:

• pretty well doc'd
• able to use multiple provisioners
• uses minitest to define test cases
• just makes sense for integration testing -- define a workflow (set of
  tasks) that simulate your deployment, spin up a set of boxes or machines
  based on Chef roles, and validate stuff

But it has some downsides that I'm unable to get around:

• does not support Chef 11 -- and the OpenStack + Chef community has
  decided to standardize on Chef 11 and not spend time supporting 10.x
• many of the tools we use -- ChefSpec, Strainer, and others -- depend
  on much newer versions of libs (minitest, json, etc) than chef-workflow
  depends on, which makes it impossible to use in our environment [2]
• lots of "magic" that seems to get glossed over -- for instance, I
  cannot understand if Chef Solo is a requirement and if not, why it gets
  installed on all the boxes I provision -- this is probably just my fault
  in not understanding Chef/Ruby 100%, but when magic seems to happen, my
  coder ears tend to perk up -- and not in a good way

I've also read Josh Timberman's article on Test Kitchen [3] but I
believe we must have different ideas of what integration testing is. My
definition of integration testing is tests that a) don't mock anything
out and b) stress the points of interaction (integration) between
various dependent things (in this case, interrelated cookbooks). I don't
really see how Test Kitchen makes this easy, but of course, I may be
misunderstanding something. Josh, please feel free to enlighten me!

Anyway, the point of this email is to get feedback on what exactly the
state of integration testing of Chef cookbooks is today, where it's
heading, and what people feel should be the direction the OpenStack +
Chef community should go in.

Best,
-jay

[1] For more info, see here:
Page not found – joinfu
[2] Issue logged in chef-workflow for this... if you have suggestions or
workarounds, please do comment:
https://github.com/chef-workflow/chef-workflow/issues/11
[3]
http://jtimberman.housepub.org/blog/2012/12/13/cookbook-integration-testing-with-real-examples/

--
Cheers,

Peter Donald

[Sascha_Bates]

I'm curious as to how well Vagabond works on Red Hat/EL? Some of the LXC
support is pretty sketchy in general from what I hear but I have a great
interest in trying it in my EL 6.3 world. Has anyone tried LXC/Vagabond
on Enterprise Linux?

Jesse Nelson wrote:

I've been down this road with our Infrastructure, and I will share
my thoughts. What I started with was workflow, but as the chef11 thing
came up I couldn't use that. I've been a fan and continue to be of
test-kitchen 1.0 for individual cookbook testing, but the lack of
being able to describe infrastructures and run chef-server to test
hurts for integration. I believe this function will come to kitchen
tho as many people have talked about it, and I am sure Fletcher is on
board with the idea just not 100% ready to go a direction yet.

What we use now:
Thor tasks that manage knife-server and spice weasel +
knife-vagrant to build out arbitrary infrastructures. I ripped this
out of our repo and put it on github during chef hack-day. Example @
GitHub - spheromak/chef-infra: Thor tasks and misc stuff for building infras with chef and spice.
http://github.com/spheromak/chef-infra. It's less than pretty, but
it has been working for us to get our dev work up and running. Just
add server-spec[1] tests and a test task to get some testing in there
(or whatever you want/need). I am happy to talk more about it if
there is interest, but I will just give highlights here.
This approach using a mishmash of tools allows us to swap the
infrastructures descriptions out from vagrant to openstack or
something else supported by knife. That was an initial requirement,
but our testing needs have been driving me down a different path due
to vbox being a pos, and the tests taking too long to spin up.

What I am trying to make work now:

At chef-conf I spent some time talking with Chris Roberts, and he
introduced me to the thing he was working on in this realm:
vagabond[2]. Which is a lxc management framework that integrates with
kitchen and has its own hooks for running tests. It handles building a
chef-server or chef-zero container, and making your repo assets
available to it. Basically it is a better more concise version of what
I had done with thor + spice + knife. Over the last week or so I have
been working on getting things working with that instead. The
vagabond system is much faster and easier to add tests for. Using
kitchen for cooks and vagabond for integration testing is working out
really nicely for me right now. There are still a lot of edges to be
worked out but Chris has been hacking on vagabond pretty heavily.

Caveat emptor: Everything I've been doing in vagabond land has been
against development or my own custom hacks. This thing isn't quite
ready for general consumption, but It is already useful. I also
believe its the right way to move on the integration testing front (at
least for my needs).

1
2

Jesse

On May 31, 2013, at 8:11 AM, Jay Pipes <jaypipes@gmail.com
mailto:jaypipes@gmail.com> wrote:

Ohai Chefs!

Alrighty, so over in OpenStack + Chef world, we've recently made a lot
of progress in consolidating and cleaning up the myriad cookbook
repositories, adding ChefSpec unit tests for the cookbooks, and getting
all the cookbooks linted with FoodCritic and gated using Gerrit and
Jenkins [1]. All good stuff.

The next big piece of the puzzle is integration testing. Now, I'm
neither a Chef nor a Ruby expert, so I decided to use the Google machine
to determine what the best practices were for doing multi-machine
integration testing of a large set of interrelated cookbooks.

So far, I've been struggling. I found chef-workflow, and that seemed to
be a great solution. It's advantages are:

• pretty well doc'd
• able to use multiple provisioners
• uses minitest to define test cases
• just makes sense for integration testing -- define a workflow (set of
  tasks) that simulate your deployment, spin up a set of boxes or machines
  based on Chef roles, and validate stuff

But it has some downsides that I'm unable to get around:

• does not support Chef 11 -- and the OpenStack + Chef community has
  decided to standardize on Chef 11 and not spend time supporting 10.x
• many of the tools we use -- ChefSpec, Strainer, and others -- depend
  on much newer versions of libs (minitest, json, etc) than chef-workflow
  depends on, which makes it impossible to use in our environment [2]
• lots of "magic" that seems to get glossed over -- for instance, I
  cannot understand if Chef Solo is a requirement and if not, why it gets
  installed on all the boxes I provision -- this is probably just my fault
  in not understanding Chef/Ruby 100%, but when magic seems to happen, my
  coder ears tend to perk up -- and not in a good way

I've also read Josh Timberman's article on Test Kitchen [3] but I
believe we must have different ideas of what integration testing is. My
definition of integration testing is tests that a) don't mock anything
out and b) stress the points of interaction (integration) between
various dependent things (in this case, interrelated cookbooks). I don't
really see how Test Kitchen makes this easy, but of course, I may be
misunderstanding something. Josh, please feel free to enlighten me!

Anyway, the point of this email is to get feedback on what exactly the
state of integration testing of Chef cookbooks is today, where it's
heading, and what people feel should be the direction the OpenStack +
Chef community should go in.

Best,
-jay

[1] For more info, see here:
Page not found – joinfu
[2] Issue logged in chef-workflow for this... if you have suggestions or
workarounds, please do comment:
https://github.com/chef-workflow/chef-workflow/issues/11
[3]
http://jtimberman.housepub.org/blog/2012/12/13/cookbook-integration-testing-with-real-examples/

[Jesse_Nelson]

The trick is to run yer LXC host on Ubuntu or arch or something that supports LXC well. They are all capable of running containers of rhel/cent

I know from the docker lists that it is a pita to get LXC up on rhel.

On May 31, 2013, at 11:44 PM, Sascha Bates sascha.bates@gmail.com wrote:

I'm curious as to how well Vagabond works on Red Hat/EL? Some of the LXC support is pretty sketchy in general from what I hear but I have a great interest in trying it in my EL 6.3 world. Has anyone tried LXC/Vagabond on Enterprise Linux?

Jesse Nelson wrote:

I've been down this road with our Infrastructure, and I will share my thoughts. What I started with was workflow, but as the chef11 thing came up I couldn't use that. I've been a fan and continue to be of test-kitchen 1.0 for individual cookbook testing, but the lack of being able to describe infrastructures and run chef-server to test hurts for integration. I believe this function will come to kitchen tho as many people have talked about it, and I am sure Fletcher is on board with the idea just not 100% ready to go a direction yet.

What we use now:
Thor tasks that manage knife-server and spice weasel + knife-vagrant to build out arbitrary infrastructures. I ripped this out of our repo and put it on github during chef hack-day. Example @ GitHub - spheromak/chef-infra: Thor tasks and misc stuff for building infras with chef and spice.. It's less than pretty, but it has been working for us to get our dev work up and running. Just add server-spec[1] tests and a test task to get some testing in there (or whatever you want/need). I am happy to talk more about it if there is interest, but I will just give highlights here.

This approach using a mishmash of tools allows us to swap the infrastructures descriptions out from vagrant to openstack or something else supported by knife. That was an initial requirement, but our testing needs have been driving me down a different path due to vbox being a pos, and the tests taking too long to spin up.

What I am trying to make work now:

At chef-conf I spent some time talking with Chris Roberts, and he introduced me to the thing he was working on in this realm: vagabond[2]. Which is a lxc management framework that integrates with kitchen and has its own hooks for running tests. It handles building a chef-server or chef-zero container, and making your repo assets available to it. Basically it is a better more concise version of what I had done with thor + spice + knife. Over the last week or so I have been working on getting things working with that instead. The vagabond system is much faster and easier to add tests for. Using kitchen for cooks and vagabond for integration testing is working out really nicely for me right now. There are still a lot of edges to be worked out but Chris has been hacking on vagabond pretty heavily.

Caveat emptor: Everything I've been doing in vagabond land has been against development or my own custom hacks. This thing isn't quite ready for general consumption, but It is already useful. I also believe its the right way to move on the integration testing front (at least for my needs).

1
2

Jesse

On May 31, 2013, at 8:11 AM, Jay Pipes jaypipes@gmail.com wrote:

Ohai Chefs!

Alrighty, so over in OpenStack + Chef world, we've recently made a lot
of progress in consolidating and cleaning up the myriad cookbook
repositories, adding ChefSpec unit tests for the cookbooks, and getting
all the cookbooks linted with FoodCritic and gated using Gerrit and
Jenkins [1]. All good stuff.

The next big piece of the puzzle is integration testing. Now, I'm
neither a Chef nor a Ruby expert, so I decided to use the Google machine
to determine what the best practices were for doing multi-machine
integration testing of a large set of interrelated cookbooks.

So far, I've been struggling. I found chef-workflow, and that seemed to
be a great solution. It's advantages are:

• pretty well doc'd
• able to use multiple provisioners
• uses minitest to define test cases
• just makes sense for integration testing -- define a workflow (set of
  tasks) that simulate your deployment, spin up a set of boxes or machines
  based on Chef roles, and validate stuff

But it has some downsides that I'm unable to get around:

• does not support Chef 11 -- and the OpenStack + Chef community has
  decided to standardize on Chef 11 and not spend time supporting 10.x
• many of the tools we use -- ChefSpec, Strainer, and others -- depend
  on much newer versions of libs (minitest, json, etc) than chef-workflow
  depends on, which makes it impossible to use in our environment [2]
• lots of "magic" that seems to get glossed over -- for instance, I
  cannot understand if Chef Solo is a requirement and if not, why it gets
  installed on all the boxes I provision -- this is probably just my fault
  in not understanding Chef/Ruby 100%, but when magic seems to happen, my
  coder ears tend to perk up -- and not in a good way

I've also read Josh Timberman's article on Test Kitchen [3] but I
believe we must have different ideas of what integration testing is. My
definition of integration testing is tests that a) don't mock anything
out and b) stress the points of interaction (integration) between
various dependent things (in this case, interrelated cookbooks). I don't
really see how Test Kitchen makes this easy, but of course, I may be
misunderstanding something. Josh, please feel free to enlighten me!

Anyway, the point of this email is to get feedback on what exactly the
state of integration testing of Chef cookbooks is today, where it's
heading, and what people feel should be the direction the OpenStack +
Chef community should go in.

Best,
-jay

[1] For more info, see here:
Page not found – joinfu
[2] Issue logged in chef-workflow for this... if you have suggestions or
workarounds, please do comment:
https://github.com/chef-workflow/chef-workflow/issues/11
[3]
http://jtimberman.housepub.org/blog/2012/12/13/cookbook-integration-testing-with-real-examples/