thanks for all the responses!
Some things are getting clearer now for me.
So first: multi-node testing and testing “from the outside” (I’d call
that acceptance / end-to-end testing) are two different pairs of
shoes. They are orthogonal and we can consider them unrelated to each
I love test-kitchen the way it is, and multi-node testing would be a
super useful addition for exactly the kinds of tests Lamont described
As for testing “from the outside”, I’m probably just looking for tool
/ plugin that would integrate to test-kitchen, but that would NOT be
installed inside the VM via busser. Instead it should be testing the
VM from the outside. As far as I understood, this would exclude
serverspec, because serverspec is SSHing into the VM to run its tests,
Btw: has anyone looked into Leibiz actually?
It appears to do some multi-node testing (reusing TK as it looks like)
and it does it’s testing from the outside via cucumber. I’ve seen
cucumber being used a lot for that kind of "from the outside"
acceptance testing, but personally rspec would be just a bit more
appealing to me
On Wed, Apr 22, 2015 at 11:25 PM, Lamont Granquist email@example.com wrote:
On 4/22/15 2:02 PM, Zac Stevens wrote:
On Wed, Apr 22, 2015 at 8:06 PM, Lamont Granquist firstname.lastname@example.org wrote:
On 4/22/15 11:28 AM, Torben Knerr wrote:
A more concrete example: say I have a cookbook for setting up
JenkinsCI with Slaves on multiple nodes, with nginx in front, etc…
I’d like to have a test suite that:
- sets up the multi-node scenario
- then verifies that the front page is served…
- …and that you can create a basic job that is executed on a slave
- …and so on (some basic acceptance level smoke tests)
That’s what I’m up to… did it get clearer?
My intuition is that the way that you’re thinking about that violates
promise theory a bit. All of your tests should be “from the inside”. The
VM that your webapp is being deployed on should really only test the world
from its perspective. It should test that its listening on its http port
and that basic API calls can succeed and that it can make calls to a
database. Trying to think about it begin responsible for testing its own
VIP violates promise theory since now its responsible for distant
infrastructure and it doesn’t have the information that it may need (it
could be failing to register correctly in the VIP but another server in the
cluster could be serving content so the VIP is up). Its better to add tests
to the loadbalancer “from the inside” so that the load balancer checks
itself and validates that all its exepected clients are listed as avaiable
and health checks are passing and the loadbalancer should be checking its
Then what we need is a multi-node test-kitchen which used something like
chef-provisioning to spin up all of these VMs and fire off the serverspec
runs on all of them, and then report back. For deployed infrastructure then
audit mode could be running serverspec tests (and ideally hook busser up to
audit mode so you could audit with bats tests and whatever) and you’d be
getting reports back on every chef-client converge or audit mode run.
I really don’t understand the conclusion that there should not be any
tests “from the outside”. (I also don’t see anything in Torben’s example
prescribing where the outside-in tests should be run from)
All the tests described above are valuable, but testing that individual
components work as expected is not a substitute for verifying that they work
as expected when assembled. Or put another way, unit tests are great but
integration tests are also valuable - I’m of the opinion that this is true
whether the units you’re dealing with are classes or servers.
I’d love to see multi-node testing in test-kitchen. I’d love to spin up a
set of VMs that are configured to work together. I’d love to run serverspec
suites on each of them, to assert their behaviour at that level. I’d love
to spin up another VM to run additional outside-in tests making assertions
about the behaviour of the whole. This would be slow, and I wouldn’t be
running these integration tests unless everything looked good, but they
would be meaningful.
Here’s the kicker: I thought that this was obvious, something that everyone
wanted to do, and it was simply a matter of time before Test Kitchen evolved
to deliver it. Now you’ve given me pause for thought.
Is this a philosophical question and I didn’t realise I was on a side?
Is there no debate and I’m simply wrong?
Did I completely misunderstand your point, and now look a bit silly for
This may be philosophical naval gazing.
The way that I’d test a cluster end-to-end would be to consider the load
balancer part of the cluster and it should issue API calls to its own VIPs
and should also assert that the health checks of all of its servers are
healthy. But the way that I express that is via the load balancer testing
itself, but that API is the externally available interface to the cluster
which is what you are going to test against for ‘integration’ or
So I suspect we get the same test coverage, but I think its important the
way that I state who is responsible for testing what, and I don’t see the
need for ‘external’ testing and more pieces bolted onto the infrastructure
to do ‘new’ kinds of tests. I think the tools are already mostly there if
you squint at it, and I agree that chef-provisioning and test-kitchen and
serverspec/bats/etc seem to be the way forwards to deliver the whole thing
and it will evolve to provide it. All we need is to make test-kitchen more
cluster-aware and then the process of simply being able to spin up the
cluster, login and converge chef on everything and then run the tests on
each individual node should be sufficient. Even though there’s no overall
end-to-end testing framework being additionally added into test-kitchen, I
don’t think you’ll miss it since by each node testing itself they also test
their dependencies (it isn’t like unit testing because you’ll really be
communicating with all your neighbors and not stubs or mocks of your
neighbors – if you fully test yourself, you should have also fully tested
all of your adjacent dependencies and the whole should really be the sum of