I’m still in the learning phase for testing with Chef, but I consider it an
essential part, thats why I spent some time on it lately.
See comments inline.
Am 07.10.2012 10:38 schrieb “steve .” firstname.lastname@example.org:
Durf. Never occurred to me to crib off of cucumber-nagios. That’s a
pretty neat idea, and it sure solves the problem of “How do you
explain test-writing?” as Cucumber tests are about as easy to read as
they come. I’m going to give that a try next week.
I like the chefspec/fauxhai concept but I’m lucky enough to be testing
in an environment where spinning up and immediately terminating a
bunch of images is relatively painless.
I like chefspec/fauxhai too. From my experience they are especially (if not
only) useful for testing how combinations of different platforms and
attributes affect the resources that you create.
I quickly felt the urgent need to spin up some images and test real node
convergence as well
Vagrant works well for me here, but the roundtrip times are naturally
longer. I wanted to play with LXC containers instead of using VirtualBox
(ideally Re-using my Vagrantfile), but haven’t come to testing this yet.
Anyone here on the list using LXC containers for testing Chef runs already?
So I will probably just laze
out on that step and write unit/basic integration tests with the
minitest handler, then write up some Cucumber features to test each VM
after convergence and before teardown. One place where I do see value
in that would be for our LWRPs, a few of which are getting pretty
complicated and could probably do with some specs running that could
The only problem I have (and this is due to my particular
implementation) is that I’m testing role/environment/platform
combinations. Can’t really test a multi-tiered app this way. It’s
one of the things test-kitchen does that my little skunkworks project
Yeah, that’s where test-kitchen really rocks.
I guess the answer is to bind some features to a special
full-stack role that we would only use for testing/development, though
that wouldn’t work for apps that require multiple hosts (Mongo springs
For testing a complex setup like this I would go for a multivm Vagrantfile
and test the real scenario with multiple VMs - otherwise you are testing
something that dosen’t reflect the real situation.
Still it would be useful to have something like a special role for
development in addition.
Your documentation and the test-writing docs on the Chefspec and
Fauxhai sites are pretty comprehensive, it seems (or have I just been
reading too much poorly-documented code lately? ). I’m not sure
how many people are interested in converging Vagrant VMs against a
Chef server endpoint (i.e. using the chef-client provisioner instead
of chef-solo), but jtimberman wrote a couple of pretty good blog posts
on the subject (starting with
, again, HI GOOGLE! ) and I could perhaps write something up as well
if the community desires it and existing resources are found wanting.
I think thats useful and will probably do that for my current project. I’ll
probably be using knife-server for painlessly setting up dedicated chef
Servers for my test runs on the fly.
Thanks for the pointer to the blog post, found some valuable tips there!
Anyway, I think this kind of testing is a neat idea and I’m going to
see how far I can get with it. I try to follow this list but I’m
awash in work email most of the time. Comments/off-list conversations
on this subject with interested parties are also welcome.
On Thu, Oct 4, 2012 at 10:22 PM, Torben Knerr email@example.com wrote:
Love the idea of properly tested cookbooks and the fact that you are
your buddies into that direction
I’ve taken some notes and collected some links about test-driven
It covers cookbook testing on several levels:
- syntax & long checking -> foodcritic
- unit-level spec testing -> chefspec and fauxhai
- “smoke tests” from the inside -> minitest-handler
- acceptance-level testing from the outside -> cucumber-nagios
Maybe you can make some use of it or you have some ideas how to improve
Am 05.10.2012 03:17 schrieb “steve .” firstname.lastname@example.org:
Continuous cookbook testing framework (analogous to test-kitchen, but
built on Jenkins, Foodcritic, RVM and Vagrant) … check.
Chef-minitest handler running on VMs and reporting test results back
to CI host … check.
Now I just have to get all my cookbook authors to write tests!
It seems like a lot of us are in this situation, so I thought I’d
share my findings and the direction in which I seem to be headed. We
have to get a bunch of people who are maybe not so used to TDD/BDD or
writing integration tests to suddenly start writing integration tests
in minitest (spec tests, in my exact case).
There’s a growing amount of source code out there, but the best
documentation I’ve found is still in Bryan’s example cookbook (found
here, if this somehow pops up on Google):
So you can use node data, but only within the scope of a particular
test definition. I understand why that is, but folks who know “enough
Ruby for Chef” will probably get tripped up by that.
Still, I need a way to tell my budding cooks what needs testing and
what doesn’t. Can I use existing code coverage tools to do this
outside of converging a node and running the minitest handler? As
far as I can tell the answer is, no, I can’t.
(I base this preliminary conclusion on an indexed conversation in
#chef from July:
… would love to be proven wrong, BTW)
My current idea for drawing attention to which cookbooks are
adequately tested in our Chef repository involves writing some
Foodcritic rules to check for the presence of minitests in each
cookbook and then critique them. I believe I can get started with
this using three rules:
“Does your cookbook have a files/default/tests/minitest directory?”
“Do you have a test/spec.rb file describing each recipe in the
“Does your test/spec for recipe X cover all of the resources specified
in recipe X?”.
Since Foodcritic testing is already in the early stage of my cookbook
testing pipeline, I can easily turn those warnings into pretty graphs.
I’m worried that writing those rules will be a bit brain-melty,
Mission accomplished: I am shaming all my repo contributors for not
writing tests for their cookbook contributions. I still need to make
writing those tests easier for them to pick up or I’m just being mean.
My idea here is to write and exhaustively document a helper gem that
provides convenience functions beyond the resource checks that are
already included, then distribute that along with the recipe that
installs the chef-minitest-handler gem and sets it up as an event
handler on the system. If the helper gem is well-documented and the
tests in the repository leverage the helper properly, I think that
will be enough signposts to get us started. Then It’ll be a lot
easier to accept all these pull requests…
Is anyone else proceeding along these lines? Is there any interest in
adding that functionality to mainline Foodcritic? Is there some other
way of ensuring test coverage on cookbooks? Any fellow travelers know
of any pitfalls or open source projects that might help?
(besides test-kitchen, that is. )