Automated test-kitchen runs?

Chef Infra (archive)
1

Ohai Chefs,

We have rubocop and foodcritic configured to run on Travis CI to
automatically check our pull requests, which is awesome. I verify each
build with test-kitchen before I cut a new cookbook release, but this is
getting very time consuming to run manually each time. For the java
cookbook, testing different java versions on various platforms results in a
ton of individual runs. I’d love to set up a system to automatically run
test-kitchen and then give me the red/green light.

Has anyone tried doing this and could recommend some tools/workflows?

Thanks,

Chris

2

There was a thread on this last August, but nothing concrete:
http://lists.opscode.com/sympa/arc/chef/2013-08/msg00069.html

On Thu, Mar 27, 2014 at 4:18 PM, Christopher Armstrong <
chris@chrisarmstrong.me> wrote:

Ohai Chefs,

We have rubocop and foodcritic configured to run on Travis CI to
automatically check our pull requests, which is awesome. I verify each
build with test-kitchen before I cut a new cookbook release, but this is
getting very time consuming to run manually each time. For the java
cookbook, testing different java versions on various platforms results in a
ton of individual runs. I'd love to set up a system to automatically run
test-kitchen and then give me the red/green light.

Has anyone tried doing this and could recommend some tools/workflows?

Thanks,

Chris

3

Run test-kitchen from travis? Super easy with kitchen-rackspace and somewhat easy with kitchen-ec2.

--Noah

On Mar 27, 2014, at 4:18 PM, Christopher Armstrong chris@chrisarmstrong.me wrote:

Ohai Chefs,

We have rubocop and foodcritic configured to run on Travis CI to automatically check our pull requests, which is awesome. I verify each build with test-kitchen before I cut a new cookbook release, but this is getting very time consuming to run manually each time. For the java cookbook, testing different java versions on various platforms results in a ton of individual runs. I'd love to set up a system to automatically run test-kitchen and then give me the red/green light.

Has anyone tried doing this and could recommend some tools/workflows?

Thanks,

Chris

4

I just began using the (newish?) paradigm I saw in the Opscode nginx cookbook to test my cookbooks for a side project via Travis and Digital Ocean. The cookbooks at https://github.com/todonts-cookbooks are linted with rubocop and foodcritic, unit tested with chefspec, and and integration tested on DO via the kitchen-digitalocean provider and serverspec (I don't think I've committed any of the serverspec tests yet). One nice thing you can do with Travis is use the build matrix to run your test-kitchen suites in parallel during the build. The most complete example I have up right now is the nginx cookbook for my project (https://github.com/todonts-cookbooks/todonts-nginx), but you can see the build-matrix stuff in action on the todonts-app cookbook (https://github.com/todonts-cookbooks/todonts-app). Props to the folks at Opscode for writing all the original harnessing.

Regards,
Tony

--
Tony Burns
Operations Engineer / Software Developer
Quad Learning, Inc.
1150 17th St Ste 310, Washington, DC 20036
c. 608.799.2000 p. 202.525.1078 f. 202.652.1075

On Mar 27, 2014, at 19:19, Noah Kantrowitz noah@coderanger.net wrote:

Run test-kitchen from travis? Super easy with kitchen-rackspace and somewhat easy with kitchen-ec2.

--Noah

On Mar 27, 2014, at 4:18 PM, Christopher Armstrong chris@chrisarmstrong.me wrote:

Ohai Chefs,

We have rubocop and foodcritic configured to run on Travis CI to automatically check our pull requests, which is awesome. I verify each build with test-kitchen before I cut a new cookbook release, but this is getting very time consuming to run manually each time. For the java cookbook, testing different java versions on various platforms results in a ton of individual runs. I'd love to set up a system to automatically run test-kitchen and then give me the red/green light.

Has anyone tried doing this and could recommend some tools/workflows?

Thanks,

Chris

5

As an example for how I do this on Rackspace, check out https://github.com/poise/poise/blob/master/.travis.yml.

The encrypted blob is my RACKSPACE_USERNAME and RACKSPACE_API_KEY variables, not much to it beyond that.

--Noah

On Mar 27, 2014, at 4:44 PM, Tony Burns tabolario@gmail.com wrote:

I just began using the (newish?) paradigm I saw in the Opscode nginx cookbook to test my cookbooks for a side project via Travis and Digital Ocean. The cookbooks at https://github.com/todonts-cookbooks are linted with rubocop and foodcritic, unit tested with chefspec, and and integration tested on DO via the kitchen-digitalocean provider and serverspec (I don't think I've committed any of the serverspec tests yet). One nice thing you can do with Travis is use the build matrix to run your test-kitchen suites in parallel during the build. The most complete example I have up right now is the nginx cookbook for my project (https://github.com/todonts-cookbooks/todonts-nginx), but you can see the build-matrix stuff in action on the todonts-app cookbook (https://github.com/todonts-cookbooks/todonts-app). Props to the folks at Opscode for writing all the original harnessing.

Regards,
Tony

--
Tony Burns
Operations Engineer / Software Developer
Quad Learning, Inc.
1150 17th St Ste 310, Washington, DC 20036
c. 608.799.2000 p. 202.525.1078 f. 202.652.1075

On Mar 27, 2014, at 19:19, Noah Kantrowitz noah@coderanger.net wrote:

Run test-kitchen from travis? Super easy with kitchen-rackspace and somewhat easy with kitchen-ec2.

--Noah

On Mar 27, 2014, at 4:18 PM, Christopher Armstrong chris@chrisarmstrong.me wrote:

Ohai Chefs,

We have rubocop and foodcritic configured to run on Travis CI to automatically check our pull requests, which is awesome. I verify each build with test-kitchen before I cut a new cookbook release, but this is getting very time consuming to run manually each time. For the java cookbook, testing different java versions on various platforms results in a ton of individual runs. I'd love to set up a system to automatically run test-kitchen and then give me the red/green light.

Has anyone tried doing this and could recommend some tools/workflows?

Thanks,

Chris

6

I've been running a few of our cookbooks through Jenkins with Kitchen
and Docker, since first of January. I had to use either Docker, or LXC,
because my Jenkins build host was already a VMWare VM and getting nested
virtualization working with something like Vagrant/Virtualbox wasn't all
that straightforward.

I broke down the build jobs up into 3 stages, quick tests, integration
tests, and deploy.

Stage 1: Unit and linting tests, with Chefspec, Rubocop, and Foodcritic,
which all takes about 1 minute or less. First stage is triggered by our
Gitlab server, or upstream dependencies.

Stage 2: Use kitchen.ci and kitchen-docker to spin up baseboxes
generated with rinse https://index.docker.io/u/atomicpenguin/centos/
and then run BATS integration tests against those machines. Some of the
cookbooks are EL5 and EL6 which take about 6-8 minutes to test both.
Some just need to be tested with EL6 which take about half the time, 3-4
minutes.

Stage 3: do a berks upload, and berks apply production. If there are
data bags in the cookbook, then I test with chef-zero in Stage 2, and do
a rake databag:upload_all on Stage 3.

If there are downstream dependencies like a role cookbook which depends
on an upstream baseline role cookbook. Then the upstream cookbook
triggers a 3-stage build on downstream cookbooks.

If you want to validate branching behaviour such as in your org-java
cookbook, then a good way to test a lot of corner cases is probably
Chefspec. Kitchen works great doing just a few branching platforms and
automatically integration testing those cases. As you add more
platforms to Kitchen, two things tend to happen. Your integration tests
become more complex, and your build time grows. If your integration
tests for all your combined platforms for a given cookbook don't fit in
a ten to fifteen minute window, then unit tests with Chefspec is
probably going to provide more value and quicker feedback.

Eric G. Wolfe
Senior Linux Administrator,
IT Infrastructure Systems

Marshall University Computing Services
Drinko Library 428-K
One John Marshall Dr.
Huntington, WV 25755
Phone: 304.942.3970
Email: eric.wolfe@marshall.edu

The worst sin towards our fellow creatures is not to hate them,
but to be indifferent to them; that's the essence of inhumanity.
-- G.B. Shaw

On 03/27/2014 07:18 PM, Christopher Armstrong wrote:

Ohai Chefs,

We have rubocop and foodcritic configured to run on Travis CI to
automatically check our pull requests, which is awesome. I verify each
build with test-kitchen before I cut a new cookbook release, but this
is getting very time consuming to run manually each time. For the java
cookbook, testing different java versions on various platforms results
in a ton of individual runs. I'd love to set up a system to
automatically run test-kitchen and then give me the red/green light.

Has anyone tried doing this and could recommend some tools/workflows?

Thanks,

Chris

7

Thanks guys!! Appreciate all the feedback. Going to take a stab at this
tomorrow.

On Thu, Mar 27, 2014 at 5:21 PM, Eric G. Wolfe eric.wolfe@marshall.eduwrote:

I've been running a few of our cookbooks through Jenkins with Kitchen
and Docker, since first of January. I had to use either Docker, or LXC,
because my Jenkins build host was already a VMWare VM and getting nested
virtualization working with something like Vagrant/Virtualbox wasn't all
that straightforward.

I broke down the build jobs up into 3 stages, quick tests, integration
tests, and deploy.

Stage 1: Unit and linting tests, with Chefspec, Rubocop, and Foodcritic,
which all takes about 1 minute or less. First stage is triggered by our
Gitlab server, or upstream dependencies.

Stage 2: Use kitchen.ci and kitchen-docker to spin up baseboxes generated
with rinse https://index.docker.io/u/atomicpenguin/centos/ and then run
BATS integration tests against those machines. Some of the cookbooks are
EL5 and EL6 which take about 6-8 minutes to test both. Some just need to
be tested with EL6 which take about half the time, 3-4 minutes.

Stage 3: do a berks upload, and berks apply production. If there are data
bags in the cookbook, then I test with chef-zero in Stage 2, and do a rake
databag:upload_all on Stage 3.

If there are downstream dependencies like a role cookbook which depends on
an upstream baseline role cookbook. Then the upstream cookbook triggers a
3-stage build on downstream cookbooks.

If you want to validate branching behaviour such as in your org-java
cookbook, then a good way to test a lot of corner cases is probably
Chefspec. Kitchen works great doing just a few branching platforms and
automatically integration testing those cases. As you add more platforms
to Kitchen, two things tend to happen. Your integration tests become more
complex, and your build time grows. If your integration tests for all your
combined platforms for a given cookbook don't fit in a ten to fifteen
minute window, then unit tests with Chefspec is probably going to provide
more value and quicker feedback.

Eric G. Wolfe
Senior Linux Administrator,
IT Infrastructure Systems

Marshall University Computing Services
Drinko Library 428-K
One John Marshall Dr.
Huntington, WV 25755
Phone: 304.942.3970
Email: eric.wolfe@marshall.edu

The worst sin towards our fellow creatures is not to hate them,
but to be indifferent to them; that's the essence of inhumanity.
-- G.B. Shaw

On 03/27/2014 07:18 PM, Christopher Armstrong wrote:

Ohai Chefs,

We have rubocop and foodcritic configured to run on Travis CI to
automatically check our pull requests, which is awesome. I verify each
build with test-kitchen before I cut a new cookbook release, but this is
getting very time consuming to run manually each time. For the java
cookbook, testing different java versions on various platforms results in a
ton of individual runs. I'd love to set up a system to automatically run
test-kitchen and then give me the red/green light.

Has anyone tried doing this and could recommend some tools/workflows?

Thanks,

Chris

8

Yep, I started replicating what Sean O'meara is doing, and I have some
notes here for having Travis drive T-K to do testing on DigitalOcean:

Total rough draft, but it does work. Blog post will follow at some point.

  • Julian

On Thu, Mar 27, 2014 at 7:19 PM, Noah Kantrowitz noah@coderanger.net wrote:

Run test-kitchen from travis? Super easy with kitchen-rackspace and somewhat easy with kitchen-ec2.

--Noah

On Mar 27, 2014, at 4:18 PM, Christopher Armstrong chris@chrisarmstrong.me wrote:

Ohai Chefs,

We have rubocop and foodcritic configured to run on Travis CI to automatically check our pull requests, which is awesome. I verify each build with test-kitchen before I cut a new cookbook release, but this is getting very time consuming to run manually each time. For the java cookbook, testing different java versions on various platforms results in a ton of individual runs. I'd love to set up a system to automatically run test-kitchen and then give me the red/green light.

Has anyone tried doing this and could recommend some tools/workflows?

Thanks,

Chris

--
[ Julian C. Dunn jdunn@aquezada.com * Sorry, I'm ]
[ WWW: http://www.aquezada.com/staff/julian * only Web 1.0 ]
[ gopher://sdf.org/1/users/keymaker/ * compliant! ]
[ PGP: 91B3 7A9D 683C 7C16 715F 442C 6065 D533 FDC2 05B9 ]

9

For long secrets you can also put them in a private S3 (or similar bucket) and just put the access key in your env vars, and then download the file at run time. (ex https://github.com/balanced-cookbooks/balanced-berkshelf-api/blob/master/.travis.yml#L10)

--Noah

On Apr 2, 2014, at 9:20 PM, "Julian C. Dunn" jdunn@aquezada.com wrote:

Yep, I started replicating what Sean O'meara is doing, and I have some
notes here for having Travis drive T-K to do testing on DigitalOcean:

https://github.com/juliandunn/sauceconnect/blob/master/TRAVIS.md

Total rough draft, but it does work. Blog post will follow at some point.

  • Julian

On Thu, Mar 27, 2014 at 7:19 PM, Noah Kantrowitz noah@coderanger.net wrote:

Run test-kitchen from travis? Super easy with kitchen-rackspace and somewhat easy with kitchen-ec2.

--Noah

On Mar 27, 2014, at 4:18 PM, Christopher Armstrong chris@chrisarmstrong.me wrote:

Ohai Chefs,

We have rubocop and foodcritic configured to run on Travis CI to automatically check our pull requests, which is awesome. I verify each build with test-kitchen before I cut a new cookbook release, but this is getting very time consuming to run manually each time. For the java cookbook, testing different java versions on various platforms results in a ton of individual runs. I'd love to set up a system to automatically run test-kitchen and then give me the red/green light.

Has anyone tried doing this and could recommend some tools/workflows?

Thanks,

Chris

--
[ Julian C. Dunn jdunn@aquezada.com * Sorry, I'm ]
[ WWW: http://www.aquezada.com/staff/julian * only Web 1.0 ]
[ gopher://sdf.org/1/users/keymaker/ * compliant! ]
[ PGP: 91B3 7A9D 683C 7C16 715F 442C 6065 D533 FDC2 05B9 ]