How to run integration tests outside of test kitchen?

Chef Infra (archive)
1

This seems like a simple thing, but I’m struggling to find the answer. We are using test kitchen with vagrant and virtualbox for local cookbook development. This works swimmingly. This as resulted in the build up of a suite of integration tests using ChefSpec that we would like to execute outside the context of test kitchen. We have a private cloud based on VMWare vsphere with an automated process using knife vsphere to provision, bootstrap and converge the nodes. Once the node is up, how can I execute the integration tests on that node?

Why do we want to do this? A few reasons:

  • We want to make this part of our CI pipeline. We considered using Chef Provisioning Vsphere instead and it’s built in test driven driver, but we would rather practice (test) like we play (production) and today that is based on knife vsphere
  • There is interest in moving production provisioning to TerraForm and that also doesn’t appear to have any support test kitchen and again we want to practice like we play.
  • Separate from the CI scenarios, there is a desire to have our suite of integration tests executed in a continuous fashion on our nodes. We are at the beginning of a long journey in adopting Chef and this practice will ease some of the transitioning pain.

Thank you in advance

2

chef-provisioning-vsphere includes a driver for Test Kitchen that will allow you to spin up a node in vCenter, converge, run your inspec tests or otherwise, and then destroy the node.

In addition, there are lots of other drivers for TK so you can test on multiple platforms should you do a hybrid cloud deployment at some point in the future.

3

When I was at CenturyLink, we used the Chef Provisioning Vsphere driver not only in development but also in our CI/CD pipeline.I can definitely understand wanting to test the same way that you run in production. I’ve run into several scenarios where kitchen tests did not quite cover several production scenarios. Things like the production living in a separate data center with a different network and also the fact that kitchen in centered around a single instance and production realities are made up of several instances being able to talk to one another and choreograph various tasks successfully.

I like to look at testing concerns as a pyramid. At the bottom you want to test a small piece of the system very quickly and very often. These tests are isolated from the rest of the system. But here errors can be found quickly and fixed cheaply. I think in the middle is where kitchen tests have a place and you might consider using kitchen even above the dev layer in CI. Somewhere at the top, you might have a more holistic test where you do an actual deployment using your normal deployment processes and utilize a test suite that might look different from the suites run inside test-kitchen and thats very specific to the role of your infrastructure and/or its applications. That might be an expensive test that you run much more infrequently than every source control commit but gives you confidence that your system works.

4

Matt - I've followed you work for a long time and thank you very much for all your contributions to the community. And thank you for the reply! Your thoughts hit home - we are quite like minded on this subject. What I think I'm hearing is... tests that live in /cookbooks/test/integration while living in a generically named directory, are for all intents and purposes kitchen tests and are not intended to be executed in any other context. And find another way to test in a more app-centric manner for the higher-level, less frequently rans tests. Fair statements?

5

Definitely fair. Test-Kitchen just doesn’t fit the bill for true end to end tests of complex (or even not so complex) distributed systems.

6

You could try Inspec and Compliance for this. There are some very good blog entries on how to use this here: https://github.com/chef/inspec#documentation

In short, you can place tests somewhere in your filesystem, in a git repo or on the compliance server and then refer to them in your kitchen.yml (taken from one of the blog entries):

suites:
  - name: default
    run_list:
      - recipe[inspec-workshop-cookbook::default]
    verifier:
      inspec_tests:
        - /Path/to/local/folder
        - https://github.com/<username>/<profilename>
        - supermarket://<owner>/<profile-name>
        - compliance://base/ssh

And those tests or compliance profiles can then also be run either in production or on some staging servers using the compliance server and/or the audit cookbook https://github.com/chef-cookbooks/audit

7

We use mini-test handler for this. We leave ChefSpec to do unit style testing only, and all integration/functional test is done via the minitest handler. This allows us to run the same tests that we run in kitchen on our live nodes (and in some cases, some tests which we can’t run in kitchen due to the distributed nature of things).