Test Kitchen and Outside-in Acceptance Testing?

Chef Infra (archive)
1

Hey guys,

I’m not up-to-date with the latest test-kitchen development - can I run
Cucumber features for testing from the outside with test-kitchen?

If not, what else are you using for this kind of testing?

Cheers,
Torben

2

Hi,

On 18 June 2013 20:34, Torben Knerr ukio@gmx.de wrote:

Hey guys,

I'm not up-to-date with the latest test-kitchen development - can I run
Cucumber features for testing from the outside with test-kitchen?

Not "with" Test Kitchen. However I've been working on a prototype which
uses Test Kitchen's API as a provisioning layer. See http://leibniz.cc

I'm writing it up this afternoon, as it happens.

S.

--
Stephen Nelson-Smith,
Founder, Principal Consultant,
Atalanta Systems Ltd,
Web: http://agilesysadmin.net
Twitter: @lordcope
Skype: atalanta.systems
Direct: +44 (0) 1329 550203
Mobile: +44 (0) 7917 101919

Atalanta Systems: The Agile Infrastructure Enablers
http://atalanta-systems.com

3

bats and minitest-handler

On Tue, Jun 18, 2013 at 3:34 PM, Torben Knerr ukio@gmx.de wrote:

Hey guys,

I'm not up-to-date with the latest test-kitchen development - can I run
Cucumber features for testing from the outside with test-kitchen?

If not, what else are you using for this kind of testing?

Cheers,
Torben

4

you should be able to integrate busser with cucumber, but i think invoking
test-kitchen directly from step definition will be painful, given its a
full blown harness & provisioning system. Im using mostly minitest or
serverspec with lxc

On Tue, Jun 18, 2013 at 12:34 PM, Torben Knerr ukio@gmx.de wrote:

Hey guys,

I'm not up-to-date with the latest test-kitchen development - can I run
Cucumber features for testing from the outside with test-kitchen?

If not, what else are you using for this kind of testing?

Cheers,
Torben

5

Ranjib, when u use busser and serverspec you are definitely doing inside
testing as you are inside the node

can't wait to hear about your adventures w/ serverspec and lxc!

On Tue, Jun 18, 2013 at 9:40 PM, Ranjib Dey dey.ranjib@gmail.com wrote:

you should be able to integrate busser with cucumber, but i think invoking
test-kitchen directly from step definition will be painful, given its a
full blown harness & provisioning system. Im using mostly minitest or
serverspec with lxc

On Tue, Jun 18, 2013 at 12:34 PM, Torben Knerr ukio@gmx.de wrote:

Hey guys,

I'm not up-to-date with the latest test-kitchen development - can I run
Cucumber features for testing from the outside with test-kitchen?

If not, what else are you using for this kind of testing?

Cheers,
Torben

6

See my slides from yesterday:

This is exactly what Leibniz does.

On 18 June 2013 20:40, Ranjib Dey dey.ranjib@gmail.com wrote:

you should be able to integrate busser with cucumber, but i think invoking
test-kitchen directly from step definition will be painful, given its a
full blown harness & provisioning system. Im using mostly minitest or
serverspec with lxc

On Tue, Jun 18, 2013 at 12:34 PM, Torben Knerr ukio@gmx.de wrote:

Hey guys,

I'm not up-to-date with the latest test-kitchen development - can I run
Cucumber features for testing from the outside with test-kitchen?

If not, what else are you using for this kind of testing?

Cheers,
Torben

--
Stephen Nelson-Smith,
Founder, Principal Consultant,
Atalanta Systems Ltd,
Web: http://agilesysadmin.net
Twitter: @lordcope
Skype: atalanta.systems
Direct: +44 (0) 1329 550203
Mobile: +44 (0) 7917 101919

Atalanta Systems: The Agile Infrastructure Enablers
http://atalanta-systems.com

7

Hi,

On 18 June 2013 20:40, Mike miketheman@gmail.com wrote:

bats and minitest-handler
Test Kitchen with bats - A Lightning Guide · GitHub

test-kitchen-example/test/integration/default/bats/example.bats.erb at master · ChrisLundquist/test-kitchen-example · GitHub

Depending on the OP's definition, that's not 'outside in'.

I think what Tobias is asking is whether he can run tests AGAINST systems
built WITH Test Kitchen.

If so, that is exactly what I am doing.

If instead he just means: can I use Test Kitchen to run tests with Cucumber

  • the answer is: yes - write a busser plugin.

But I would argue that's the wrong use of the tool, and serverspec or bats
is much better for this use case.

S.

8

Thanks for the quick replies!

I just noticed that I'm actually looking for something suitable to test my
infrastructure (e.g. represented by a couple of vagrant VMs interacting
with each other) rather than individual cookbooks.

@Stephen: looking forward to see more of it. My first thought on Leibniz
was a bit different though given that we are working in a kitchen, aren't
we?

@Mike: but this runs on the node, doesn't it?

@Ranjib: do you have an example where you are using serverspec? It sounds
like you spin up a VM, then run your serverspec suite against it from the
outside, and tear it down again. Is that so?

On Tue, Jun 18, 2013 at 9:40 PM, Ranjib Dey dey.ranjib@gmail.com wrote:

you should be able to integrate busser with cucumber, but i think invoking
test-kitchen directly from step definition will be painful, given its a
full blown harness & provisioning system. Im using mostly minitest or
serverspec with lxc

On Tue, Jun 18, 2013 at 12:34 PM, Torben Knerr ukio@gmx.de wrote:

Hey guys,

I'm not up-to-date with the latest test-kitchen development - can I run
Cucumber features for testing from the outside with test-kitchen?

If not, what else are you using for this kind of testing?

Cheers,
Torben

9

Hi,

On 18 June 2013 20:40, Ranjib Dey dey.ranjib@gmail.com wrote:

you should be able to integrate busser with cucumber, but i think invoking
test-kitchen directly from step definition will be painful, given its a
full blown harness & provisioning system. Im using mostly minitest or
serverspec with lxc

How painful is this:

Given(/^I have provisioned the following infrastructure:$/) do
|specification|
@infrastructure = Leibniz.build(specification)
end

Given(/^I have run Chef$/) do
@infrastructure.destroy
@infrastructure.converge
end

S.

10

Hi,

On 18 June 2013 20:52, Torben Knerr ukio@gmx.de wrote:

Thanks for the quick replies!

I just noticed that I'm actually looking for something suitable to test my
infrastructure (e.g. represented by a couple of vagrant VMs interacting
with each other) rather than individual cookbooks.

Right - so explicitly this is not what Test Kitchen is designed for. But
we can use the fact that it builds infrastructure and converges nodes.

@Stephen: looking forward to see more of it. My first thought on Leibniz
was a bit different though given that we are working in a kitchen, aren't
we?
File:Butterkeks.jpg - Wikimedia Commons

Hehe. I named it after the dude who, alongside Newton, invevented
integration.

@Mike: but this runs on the node, doesn't it?

Yes - absolutely. This is what Test Kitchen is designed to do.

S.

11

Hi Stephen,

yes, I'm looking for something to test my infrastructure from the outside,
ideally reusing test-kitchen for bringing up and converging the VMs.

Leibniz looks exactly what I was looking for!

Btw: any news on the availability of the 2nd edition of your "Test Driven
Infrastructure" book mentioned on the last slide? I couldn't find it on
amazon yet :slight_smile:

Cheers,
Torben

On Tue, Jun 18, 2013 at 9:58 PM, Stephen Nelson-Smith <
stephen@atalanta-systems.com> wrote:

Hi,

On 18 June 2013 20:52, Torben Knerr ukio@gmx.de wrote:

Thanks for the quick replies!

I just noticed that I'm actually looking for something suitable to test
my infrastructure (e.g. represented by a couple of vagrant VMs interacting
with each other) rather than individual cookbooks.

Right - so explicitly this is not what Test Kitchen is designed for. But
we can use the fact that it builds infrastructure and converges nodes.

@Stephen: looking forward to see more of it. My first thought on
Leibniz was a bit different though given that we are working in a kitchen,
aren't we?
File:Butterkeks.jpg - Wikimedia Commons

Hehe. I named it after the dude who, alongside Newton, invevented
integration.

@Mike: but this runs on the node, doesn't it?

Yes - absolutely. This is what Test Kitchen is designed to do.

S.

12

Also using lxc/ec2/vagrant + bats + serverspec

I've had some success with vagabond for building full infra and running integrations from that. Vagabond still has some rough edges, but it can drive kitchen as well.

On Jun 18, 2013, at 12:40 PM, Ranjib Dey dey.ranjib@gmail.com wrote:

you should be able to integrate busser with cucumber, but i think invoking test-kitchen directly from step definition will be painful, given its a full blown harness & provisioning system. Im using mostly minitest or serverspec with lxc

On Tue, Jun 18, 2013 at 12:34 PM, Torben Knerr ukio@gmx.de wrote:
Hey guys,

I'm not up-to-date with the latest test-kitchen development - can I run Cucumber features for testing from the outside with test-kitchen?

If not, what else are you using for this kind of testing?

Cheers,
Torben

13

Hi Torben (sorry about Tobias!)

On 18 June 2013 21:51, Torben Knerr ukio@gmx.de wrote:

Btw: any news on the availability of the 2nd edition of your "Test Driven
Infrastructure" book mentioned on the last slide? I couldn't find it on
amazon yet :slight_smile:

Finished draft goes to editor today, but we've extended technical review by
1 week as of today, as most tech reviewers have been busy with Velocity
prep. I thought the plan was for it to be on sale from today/tomorrow.
BTW buy it from O'Reilly, not Amazon :slight_smile:

S.

14

Cool, vagabond looks nice too!

Also looked at serverspec. Funny, it's run from the outside but tests from
the inside :wink:

On Tue, Jun 18, 2013 at 10:46 PM, Jesse Nelson spheromak@gmail.com wrote:

Also using lxc/ec2/vagrant + bats + serverspec

I've had some success with vagabond for building full infra and running
integrations from that. Vagabond still has some rough edges, but it can
drive kitchen as well.

On Jun 18, 2013, at 12:40 PM, Ranjib Dey dey.ranjib@gmail.com wrote:

you should be able to integrate busser with cucumber, but i think invoking
test-kitchen directly from step definition will be painful, given its a
full blown harness & provisioning system. Im using mostly minitest or
serverspec with lxc

On Tue, Jun 18, 2013 at 12:34 PM, Torben Knerr ukio@gmx.de wrote:

Hey guys,

I'm not up-to-date with the latest test-kitchen development - can I run
Cucumber features for testing from the outside with test-kitchen?

If not, what else are you using for this kind of testing?

Cheers,
Torben

15

On Tue, Jun 18, 2013 at 10:55 PM, Stephen Nelson-Smith <
stephen@atalanta-systems.com> wrote:

Hi Torben (sorry about Tobias!)

On 18 June 2013 21:51, Torben Knerr ukio@gmx.de wrote:

Btw: any news on the availability of the 2nd edition of your "Test Driven
Infrastructure" book mentioned on the last slide? I couldn't find it on
amazon yet :slight_smile:

Finished draft goes to editor today, but we've extended technical review
by 1 week as of today, as most tech reviewers have been busy with Velocity
prep. I thought the plan was for it to be on sale from today/tomorrow.
BTW buy it from O'Reilly, not Amazon :slight_smile:

On my wishlist :slight_smile:

S.

16

On Tuesday, June 18, 2013 at 18:11, Torben Knerr wrote:

On Tue, Jun 18, 2013 at 10:55 PM, Stephen Nelson-Smith <stephen@atalanta-systems.com (mailto:stephen@atalanta-systems.com)> wrote:

Hi Torben (sorry about Tobias!)

On 18 June 2013 21:51, Torben Knerr <ukio@gmx.de (mailto:ukio@gmx.de)> wrote:

Btw: any news on the availability of the 2nd edition of your "Test Driven Infrastructure" book mentioned on the last slide? I couldn't find it on amazon yet :slight_smile:

Finished draft goes to editor today, but we've extended technical review by 1 week as of today, as most tech reviewers have been busy with Velocity prep. I thought the plan was for it to be on sale from today/tomorrow. BTW buy it from O'Reilly, not Amazon :slight_smile:

On my wishlist :slight_smile:
And on mine! :slight_smile:

Stephen, mind if I ask what’s the test-kitchen “the-stretch” branch for? Should I actually use it, as suggested in your slides?

  • cassiano
17

Hi,

That's just the development branch for the final 1.0 release. I probably
had it there from stuff I was experimenting with on chef zero. Just use
the current pre, or wait for the 1.0 release.

S.

On 19 June 2013 08:06, Cassiano Leal cassianoleal@gmail.com wrote:

On Tuesday, June 18, 2013 at 18:11, Torben Knerr wrote:

On Tue, Jun 18, 2013 at 10:55 PM, Stephen Nelson-Smith <
stephen@atalanta-systems.com> wrote:

Hi Torben (sorry about Tobias!)

On 18 June 2013 21:51, Torben Knerr ukio@gmx.de wrote:

Btw: any news on the availability of the 2nd edition of your "Test Driven
Infrastructure" book mentioned on the last slide? I couldn't find it on
amazon yet :slight_smile:

Finished draft goes to editor today, but we've extended technical review
by 1 week as of today, as most tech reviewers have been busy with Velocity
prep. I thought the plan was for it to be on sale from today/tomorrow.
BTW buy it from O'Reilly, not Amazon :slight_smile:

On my wishlist :slight_smile:

And on mine! :slight_smile:

Stephen, mind if I ask what’s the test-kitchen “the-stretch” branch for?
Should I actually use it, as suggested in your slides?

  • cassiano

--
Stephen Nelson-Smith,
Founder, Principal Consultant,
Atalanta Systems Ltd,
Web: http://agilesysadmin.net
Twitter: @lordcope
Skype: atalanta.systems
Direct: +44 (0) 1329 550203
Mobile: +44 (0) 7917 101919

Atalanta Systems: The Agile Infrastructure Enablers
http://atalanta-systems.com