Single cbook testing doesn't interest me as much as the complex interactions of various cookbooks. If you have many people writing cookbooks realistically testing every permutation and output doesn't seem logistically feasible. Pulling a subset of the runlist from prod chef and running it over that would be good. perhaps cookbook meta data could help with knowing what a cookbook should be able to run with or can't run with.
I agree with andrew that we are already declaring a lot of the state, but simply asserting against that state seems a little redundant. Especially when we are dynamically configuring something. I suppose the assumption is that if chef says a package installed then it is installed. At some point we have to trust that the tool is doing what it says, is erroring if its not/can't, and is doing so in a graceful non destructive manner (thats the job of the framework).
Some simple code that could take chef recipes and magic up some cucumber or similar testing framework would be infinitely useful for most people i suspect. If it could generate basic test for cookbooks that plug into cucumber-nagios or something similar. you have some sweet sauce there.
Generic service for testing functional output of a run was also something we were talking about. being able to have confidence in the output being correct for an application at the very minimum being syntactically correct is at least a step. I.e. i want to take output form the apache cookbook which has dynamic inputs and know that on the other end what the human has put in is going to be parsed by httpd without syntax errors. I believe there is value there.
my apologies if this comes off a little scatter brained im just into my first cup of coffee this morning.
On Jul 9, 2010, at 12:20 PM, Jeremy Deininger wrote:
Hey guys,
Thought I'd chime in with my experience testing system configuration code @ RightScale so far. What we've been building are integration style cucumber tests to run a cookbook through it's paces on all platforms and OSs that we support.
First we use our API to spin up 'fresh' server clusters in EC2, one for every platform/OS (variation) that the cookbook will be supporting. The same could be done using other cloud APIs (anyone else doing this with VMware or etc?) Starting from scratch is important because of chef's idempotent nature.
Then a cucumber test is run against every variation in parallel. The cucumber test runs a series of recipes on the cluster then uses what we call 'spot checks' to ensure the cluster is configured and functional. The spot checks are updated when we find a bug, to cover the bug. An example spot check would be, sshing to every server and checking the mysql.err file for bad strings.
These high level integration tests are long running but have been very useful flushing out bugs.
-J