# Testing Cookbooks vs Testing Infrastructure

**URL:** <https://discourse.chef.io/t/testing-cookbooks-vs-testing-infrastructure/6507>\
**Category:** Chef Infra (archive)\
**Created:** [April 22, 2015, 5:40am UTC](https://discourse.chef.io/t/testing-cookbooks-vs-testing-infrastructure/6507 "2015-04-22T05:40:10Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![tknerr](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/tknerr/32/21_2.png) [@tknerr](https://discourse.chef.io/u/tknerr)\
**Post date:** [April 22, 2015, 5:40am UTC](https://discourse.chef.io/t/testing-cookbooks-vs-testing-infrastructure/6507/1 "2015-04-22T05:40:10Z")

</div>

Hey everybody,

@jtimberman's recent blog post about test-driven infrastructure with  
Chef made me thinking about this again.

All the tools we currently have are focused on cookbook testing, but  
we have nothing established for acceptance level infrastructure  
testing yet, do we?

I mean specifically:

- testing from the outside vs from the inside
- testing multi-vm setups

See also my comment on the original post:

> **[Overview of Test Driven Infrastructure with Chef - Chef Blog](https://blog.chef.io/overview-of-test-driven-infrastructure-with-chef/)**
>
> This post is all about test driven infrastructure with Chef: an overview in testing Chef cookbooks for the current landscape. This post is focused on tools included in ChefDK. Some other tools and projects are mentioned for completeness or historical...

What do we have in this space?

Cheers,  
Torben

---

<div class="post-metadata">

**Author:** ![Tensibai](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/tensibai/32/29_2.png) [@Tensibai](https://discourse.chef.io/u/Tensibai)\
**Post date:** [April 22, 2015, 12:41pm UTC](https://discourse.chef.io/t/testing-cookbooks-vs-testing-infrastructure/6507/2 "2015-04-22T12:41:17Z")

</div>

I had a discussion with a colleague this morning about this and pointed  
to Terraform[1].

As we're already using vagrant with a custom provider (for vsphere)  
we're planning to move toward this path. for now it's just an idea with  
no feedback but the described way Terraform works sounds interesting to  
write "plans" to ensure idem-potency for cookbooks (spin up a machine,  
converge it once, re-converge it, change a parameter somewhere, converge  
again, ensure expectation are there).

That's what we're aiming to do, launched from jenkins on cookbook commit  
after usual linting/chefspec test.

I'll try to write something about it if we end up with something  
functional at end.

Le 2015-04-22 07:40, Torben Knerr a écrit :

> Hey everybody,
> 
> @jtimberman's recent blog post about test-driven infrastructure with  
> Chef made me thinking about this again.
> 
> All the tools we currently have are focused on cookbook testing, but  
> we have nothing established for acceptance level infrastructure  
> testing yet, do we?
> 
> I mean specifically:
> 
> - testing from the outside vs from the inside
> - testing multi-vm setups
> 
> See also my comment on the original post:  
> [https://www.chef.io/blog/2015/04/21/overview-of-test-driven-infrastructure-with-chef/](https://www.chef.io/blog/2015/04/21/overview-of-test-driven-infrastructure-with-chef/) [1]
> 
> What do we have in this space?
> 
> Cheers,  
> Torben

## Links:

[1]  
[https://www.chef.io/blog/2015/04/21/overview-of-test-driven-infrastructure-with-chef/](https://www.chef.io/blog/2015/04/21/overview-of-test-driven-infrastructure-with-chef/)

---

<div class="post-metadata">

**Author:** ![Tensibai](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/tensibai/32/29_2.png) [@Tensibai](https://discourse.chef.io/u/Tensibai)\
**Post date:** [April 22, 2015, 12:45pm UTC](https://discourse.chef.io/t/testing-cookbooks-vs-testing-infrastructure/6507/3 "2015-04-22T12:45:53Z")

</div>

I forgot the link to terraform: [What is Terraform | Terraform | HashiCorp Developer](http://www.terraform.io/intro/index.html)

Le 2015-04-22 14:41, Tensibai a écrit :

> I had a discussion with a colleague this morning about this and pointed to Terraform[1].
> 
> As we're already using vagrant with a custom provider (for vsphere) we're planning to move toward this path. for now it's just an idea with no feedback but the described way Terraform works sounds interesting to write "plans" to ensure idem-potency for cookbooks (spin up a machine, converge it once, re-converge it, change a parameter somewhere, converge again, ensure expectation are there).
> 
> That's what we're aiming to do, launched from jenkins on cookbook commit after usual linting/chefspec test.
> 
> I'll try to write something about it if we end up with something functional at end.
> 
> Le 2015-04-22 07:40, Torben Knerr a écrit :
> 
> > Hey everybody,
> > 
> > @jtimberman's recent blog post about test-driven infrastructure with  
> > Chef made me thinking about this again.
> > 
> > All the tools we currently have are focused on cookbook testing, but  
> > we have nothing established for acceptance level infrastructure  
> > testing yet, do we?
> > 
> > I mean specifically:
> > 
> > - testing from the outside vs from the inside
> > - testing multi-vm setups
> > 
> > See also my comment on the original post:  
> > [https://www.chef.io/blog/2015/04/21/overview-of-test-driven-infrastructure-with-chef/](https://www.chef.io/blog/2015/04/21/overview-of-test-driven-infrastructure-with-chef/) [1]
> > 
> > What do we have in this space?
> > 
> > Cheers,  
> > Torben

## Links:

[1]  
[https://www.chef.io/blog/2015/04/21/overview-of-test-driven-infrastructure-with-chef/](https://www.chef.io/blog/2015/04/21/overview-of-test-driven-infrastructure-with-chef/)

---

<div class="post-metadata">

**Author:** ![tknerr](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/tknerr/32/21_2.png) [@tknerr](https://discourse.chef.io/u/tknerr)\
**Post date:** [April 22, 2015, 1:37pm UTC](https://discourse.chef.io/t/testing-cookbooks-vs-testing-infrastructure/6507/4 "2015-04-22T13:37:51Z")

</div>

Are you planning to (acceptance level) test it from the outside as well?

Or just for convergence to succeed?  
Am 22.04.2015 14:46 schrieb "Tensibai" [tensibai@iabis.net](mailto:tensibai@iabis.net):

> I forgot the link to terraform: [What is Terraform | Terraform | HashiCorp Developer](http://www.terraform.io/intro/index.html)
> 
> Le 2015-04-22 14:41, Tensibai a écrit :
> 
> I had a discussion with a colleague this morning about this and pointed to  
> Terraform[1].
> 
> As we're already using vagrant with a custom provider (for vsphere) we're  
> planning to move toward this path. for now it's just an idea with no  
> feedback but the described way Terraform works sounds interesting to write  
> "plans" to ensure idem-potency for cookbooks (spin up a machine, converge  
> it once, re-converge it, change a parameter somewhere, converge again,  
> ensure expectation are there).
> 
> That's what we're aiming to do, launched from jenkins on cookbook commit  
> after usual linting/chefspec test.
> 
> I'll try to write something about it if we end up with something  
> functional at end.
> 
> Le 2015-04-22 07:40, Torben Knerr a écrit :
> 
> Hey everybody,
> 
> @jtimberman's recent blog post about test-driven infrastructure with  
> Chef made me thinking about this again.
> 
> All the tools we currently have are focused on cookbook testing, but  
> we have nothing established for acceptance level infrastructure  
> testing yet, do we?
> 
> I mean specifically:
> 
> - testing from the outside vs from the inside
> - testing multi-vm setups
> 
> See also my comment on the original post:
> 
> [https://www.chef.io/blog/2015/04/21/overview-of-test-driven-infrastructure-with-chef/](https://www.chef.io/blog/2015/04/21/overview-of-test-driven-infrastructure-with-chef/)
> 
> What do we have in this space?
> 
> Cheers,  
> Torben

---

<div class="post-metadata">

**Author:** ![Tensibai](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/tensibai/32/29_2.png) [@Tensibai](https://discourse.chef.io/u/Tensibai)\
**Post date:** [April 22, 2015, 1:54pm UTC](https://discourse.chef.io/t/testing-cookbooks-vs-testing-infrastructure/6507/5 "2015-04-22T13:54:27Z")

</div>

We wish to have integration tests launched from jenkins too, that's not  
really outside, but the idea is to test the functional state in IC  
before promotion to QA with human testers behind load balancer and  
reverse proxy.

I may misunderstand what you think of under acceptance term, sorry if  
it's the case 🙂

Le 2015-04-22 15:37, Torben Knerr a écrit :

> Are you planning to (acceptance level) test it from the outside as well?
> 
> Or just for convergence to succeed?  
> Am 22.04.2015 14:46 schrieb "Tensibai" [tensibai@iabis.net](mailto:tensibai@iabis.net):
> 
> I forgot the link to terraform: [What is Terraform | Terraform | HashiCorp Developer](http://www.terraform.io/intro/index.html) [1]
> 
> Le 2015-04-22 14:41, Tensibai a écrit :
> 
> I had a discussion with a colleague this morning about this and pointed to Terraform[1].
> 
> As we're already using vagrant with a custom provider (for vsphere) we're planning to move toward this path. for now it's just an idea with no feedback but the described way Terraform works sounds interesting to write "plans" to ensure idem-potency for cookbooks (spin up a machine, converge it once, re-converge it, change a parameter somewhere, converge again, ensure expectation are there).
> 
> That's what we're aiming to do, launched from jenkins on cookbook commit after usual linting/chefspec test.
> 
> I'll try to write something about it if we end up with something functional at end.
> 
> Le 2015-04-22 07:40, Torben Knerr a écrit :  
> Hey everybody,
> 
> @jtimberman's recent blog post about test-driven infrastructure with  
> Chef made me thinking about this again.
> 
> All the tools we currently have are focused on cookbook testing, but  
> we have nothing established for acceptance level infrastructure  
> testing yet, do we?
> 
> I mean specifically:
> 
> - testing from the outside vs from the inside
> - testing multi-vm setups
> 
> See also my comment on the original post:  
> [https://www.chef.io/blog/2015/04/21/overview-of-test-driven-infrastructure-with-chef/](https://www.chef.io/blog/2015/04/21/overview-of-test-driven-infrastructure-with-chef/) [2]
> 
> What do we have in this space?
> 
> Cheers,  
> Torben

## Links:

[1] [What is Terraform | Terraform | HashiCorp Developer](http://www.terraform.io/intro/index.html)  
[2]  
[https://www.chef.io/blog/2015/04/21/overview-of-test-driven-infrastructure-with-chef/](https://www.chef.io/blog/2015/04/21/overview-of-test-driven-infrastructure-with-chef/)

---

<div class="post-metadata">

**Author:** ![tknerr](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/tknerr/32/21_2.png) [@tknerr](https://discourse.chef.io/u/tknerr)\
**Post date:** [April 22, 2015, 6:28pm UTC](https://discourse.chef.io/t/testing-cookbooks-vs-testing-infrastructure/6507/6 "2015-04-22T18:28:25Z")

</div>

Sorry, I was probably not specific enough 🙂

Serverspec is asserting state by ssh'ing into the VM, then checking  
for processes, files being present, services being started, etc... (I  
call that "from the inside")

However, if all that works, it doesn't necessarily mean that it works  
for the end user (lets say the webapp is served on port 80 and  
accessible "from outside" of the VM. That's probably a grey zone where  
infrastructure testing ends and you the acceptance testing for a  
specific application starts.

A more concrete example: say I have a cookbook for setting up  
JenkinsCI with Slaves on multiple nodes, with nginx in front, etc..  
I'd like to have a test suite that:

- sets up the multi-node scenario
- then verifies that the front page is served...
- ...and that you can create a basic job that is executed on a slave
- ...and so on (some basic acceptance level smoke tests)

That's what I'm up to... did it get clearer?

Cheers,  
Torben

On Wed, Apr 22, 2015 at 3:54 PM, Tensibai [tensibai@iabis.net](mailto:tensibai@iabis.net) wrote:

> We wish to have integration tests launched from jenkins too, that's not  
> really outside, but the idea is to test the functional state in IC before  
> promotion to QA with human testers behind load balancer and reverse proxy.
> 
> I may misunderstand what you think of under acceptance term, sorry if it's  
> the case 🙂
> 
> Le 2015-04-22 15:37, Torben Knerr a écrit :
> 
> Are you planning to (acceptance level) test it from the outside as well?
> 
> Or just for convergence to succeed?
> 
> Am 22.04.2015 14:46 schrieb "Tensibai" [tensibai@iabis.net](mailto:tensibai@iabis.net):
> 
> > I forgot the link to terraform: [What is Terraform | Terraform | HashiCorp Developer](http://www.terraform.io/intro/index.html)
> > 
> > Le 2015-04-22 14:41, Tensibai a écrit :
> > 
> > I had a discussion with a colleague this morning about this and pointed to  
> > Terraform[1].
> > 
> > As we're already using vagrant with a custom provider (for vsphere) we're  
> > planning to move toward this path. for now it's just an idea with no  
> > feedback but the described way Terraform works sounds interesting to write  
> > "plans" to ensure idem-potency for cookbooks (spin up a machine, converge it  
> > once, re-converge it, change a parameter somewhere, converge again, ensure  
> > expectation are there).
> > 
> > That's what we're aiming to do, launched from jenkins on cookbook commit  
> > after usual linting/chefspec test.
> > 
> > I'll try to write something about it if we end up with something  
> > functional at end.
> > 
> > Le 2015-04-22 07:40, Torben Knerr a écrit :
> > 
> > Hey everybody,
> > 
> > @jtimberman's recent blog post about test-driven infrastructure with  
> > Chef made me thinking about this again.
> > 
> > All the tools we currently have are focused on cookbook testing, but  
> > we have nothing established for acceptance level infrastructure  
> > testing yet, do we?
> > 
> > I mean specifically:
> > 
> > - testing from the outside vs from the inside
> > - testing multi-vm setups
> > 
> > See also my comment on the original post:
> > 
> > [https://www.chef.io/blog/2015/04/21/overview-of-test-driven-infrastructure-with-chef/](https://www.chef.io/blog/2015/04/21/overview-of-test-driven-infrastructure-with-chef/)
> > 
> > What do we have in this space?
> > 
> > Cheers,  
> > Torben

---

<div class="post-metadata">

**Author:** ![Morgan\_Blackthorne](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/morgan_blackthorne/32/100_2.png) [@Morgan\_Blackthorne](https://discourse.chef.io/u/Morgan_Blackthorne)\
**Post date:** [April 22, 2015, 6:34pm UTC](https://discourse.chef.io/t/testing-cookbooks-vs-testing-infrastructure/6507/7 "2015-04-22T18:34:24Z")

</div>

That's probably best suited by more generalized testing suites like  
Selenium, etc, for web services. However, I thought that serverspec also  
supported basic curl someurl | grep 'somestring' as well? Maybe I'm  
thinking of one of the other testing frameworks.

--  
~_~ StormeRider ~_~

"Every world needs its heroes [...] They inspire us to be better than we  
are. And they protect from the darkness that's just around the corner."

(from Smallville Season 6x1: "Zod")

On why I hate the phrase "that's so lame"... [http://bit.ly/Ps3uSS](http://bit.ly/Ps3uSS)

On Wed, Apr 22, 2015 at 11:28 AM, Torben Knerr [mail@tknerr.de](mailto:mail@tknerr.de) wrote:

> Sorry, I was probably not specific enough 🙂
> 
> Serverspec is asserting state by ssh'ing into the VM, then checking  
> for processes, files being present, services being started, etc... (I  
> call that "from the inside")
> 
> However, if all that works, it doesn't necessarily mean that it works  
> for the end user (lets say the webapp is served on port 80 and  
> accessible "from outside" of the VM. That's probably a grey zone where  
> infrastructure testing ends and you the acceptance testing for a  
> specific application starts.
> 
> A more concrete example: say I have a cookbook for setting up  
> JenkinsCI with Slaves on multiple nodes, with nginx in front, etc..  
> I'd like to have a test suite that:
> 
> - sets up the multi-node scenario
> - then verifies that the front page is served...
> - ...and that you can create a basic job that is executed on a slave
> - ...and so on (some basic acceptance level smoke tests)
> 
> That's what I'm up to... did it get clearer?
> 
> Cheers,  
> Torben
> 
> On Wed, Apr 22, 2015 at 3:54 PM, Tensibai [tensibai@iabis.net](mailto:tensibai@iabis.net) wrote:
> 
> > We wish to have integration tests launched from jenkins too, that's not  
> > really outside, but the idea is to test the functional state in IC before  
> > promotion to QA with human testers behind load balancer and reverse  
> > proxy.
> > 
> > I may misunderstand what you think of under acceptance term, sorry if  
> > it's  
> > the case 🙂
> > 
> > Le 2015-04-22 15:37, Torben Knerr a écrit :
> > 
> > Are you planning to (acceptance level) test it from the outside as well?
> > 
> > Or just for convergence to succeed?
> > 
> > Am 22.04.2015 14:46 schrieb "Tensibai" [tensibai@iabis.net](mailto:tensibai@iabis.net):
> > 
> > > I forgot the link to terraform:  
> > > [What is Terraform | Terraform | HashiCorp Developer](http://www.terraform.io/intro/index.html)
> > > 
> > > Le 2015-04-22 14:41, Tensibai a écrit :
> > > 
> > > I had a discussion with a colleague this morning about this and pointed  
> > > to  
> > > Terraform[1].
> > > 
> > > As we're already using vagrant with a custom provider (for vsphere)  
> > > we're  
> > > planning to move toward this path. for now it's just an idea with no  
> > > feedback but the described way Terraform works sounds interesting to  
> > > write  
> > > "plans" to ensure idem-potency for cookbooks (spin up a machine,  
> > > converge it  
> > > once, re-converge it, change a parameter somewhere, converge again,  
> > > ensure  
> > > expectation are there).
> > > 
> > > That's what we're aiming to do, launched from jenkins on cookbook commit  
> > > after usual linting/chefspec test.
> > > 
> > > I'll try to write something about it if we end up with something  
> > > functional at end.
> > > 
> > > Le 2015-04-22 07:40, Torben Knerr a écrit :
> > > 
> > > Hey everybody,
> > > 
> > > @jtimberman's recent blog post about test-driven infrastructure with  
> > > Chef made me thinking about this again.
> > > 
> > > All the tools we currently have are focused on cookbook testing, but  
> > > we have nothing established for acceptance level infrastructure  
> > > testing yet, do we?
> > > 
> > > I mean specifically:
> > > 
> > > - testing from the outside vs from the inside
> > > - testing multi-vm setups
> > > 
> > > See also my comment on the original post:
> 
> [https://www.chef.io/blog/2015/04/21/overview-of-test-driven-infrastructure-with-chef/](https://www.chef.io/blog/2015/04/21/overview-of-test-driven-infrastructure-with-chef/)
> 
> > > What do we have in this space?
> > > 
> > > Cheers,  
> > > Torben

---

<div class="post-metadata">

**Author:** ![Mike](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/mike/32/18_2.png) [@Mike](https://discourse.chef.io/u/Mike)\
**Post date:** [April 22, 2015, 6:35pm UTC](https://discourse.chef.io/t/testing-cookbooks-vs-testing-infrastructure/6507/8 "2015-04-22T18:35:45Z")

</div>

Torben,

I think it's easy to get really deep in how one wants to perform acceptance  
testing.

In our deployment cycle, we deploy the changes to a node, then run a series  
of phantomjs tests that load a web page, log in, clock a few things, assert  
behavior.

This is totally customized for our app, but there's no reason anyone  
couldn't write their own tests.

I don't think it's ever going to be the responsibility of  
Test-Kitchen/Serverspec - rather these methods could use bussers to launch  
your acceptance testing, or not - that's up to you.

-M

On Wed, Apr 22, 2015 at 2:28 PM, Torben Knerr [mail@tknerr.de](mailto:mail@tknerr.de) wrote:

> Sorry, I was probably not specific enough 🙂
> 
> Serverspec is asserting state by ssh'ing into the VM, then checking  
> for processes, files being present, services being started, etc... (I  
> call that "from the inside")
> 
> However, if all that works, it doesn't necessarily mean that it works  
> for the end user (lets say the webapp is served on port 80 and  
> accessible "from outside" of the VM. That's probably a grey zone where  
> infrastructure testing ends and you the acceptance testing for a  
> specific application starts.
> 
> A more concrete example: say I have a cookbook for setting up  
> JenkinsCI with Slaves on multiple nodes, with nginx in front, etc..  
> I'd like to have a test suite that:
> 
> - sets up the multi-node scenario
> - then verifies that the front page is served...
> - ...and that you can create a basic job that is executed on a slave
> - ...and so on (some basic acceptance level smoke tests)
> 
> That's what I'm up to... did it get clearer?
> 
> Cheers,  
> Torben
> 
> On Wed, Apr 22, 2015 at 3:54 PM, Tensibai [tensibai@iabis.net](mailto:tensibai@iabis.net) wrote:
> 
> > We wish to have integration tests launched from jenkins too, that's not  
> > really outside, but the idea is to test the functional state in IC before  
> > promotion to QA with human testers behind load balancer and reverse  
> > proxy.
> > 
> > I may misunderstand what you think of under acceptance term, sorry if  
> > it's  
> > the case 🙂
> > 
> > Le 2015-04-22 15:37, Torben Knerr a écrit :
> > 
> > Are you planning to (acceptance level) test it from the outside as well?
> > 
> > Or just for convergence to succeed?
> > 
> > Am 22.04.2015 14:46 schrieb "Tensibai" [tensibai@iabis.net](mailto:tensibai@iabis.net):
> > 
> > > I forgot the link to terraform:  
> > > [What is Terraform | Terraform | HashiCorp Developer](http://www.terraform.io/intro/index.html)
> > > 
> > > Le 2015-04-22 14:41, Tensibai a écrit :
> > > 
> > > I had a discussion with a colleague this morning about this and pointed  
> > > to  
> > > Terraform[1].
> > > 
> > > As we're already using vagrant with a custom provider (for vsphere)  
> > > we're  
> > > planning to move toward this path. for now it's just an idea with no  
> > > feedback but the described way Terraform works sounds interesting to  
> > > write  
> > > "plans" to ensure idem-potency for cookbooks (spin up a machine,  
> > > converge it  
> > > once, re-converge it, change a parameter somewhere, converge again,  
> > > ensure  
> > > expectation are there).
> > > 
> > > That's what we're aiming to do, launched from jenkins on cookbook commit  
> > > after usual linting/chefspec test.
> > > 
> > > I'll try to write something about it if we end up with something  
> > > functional at end.
> > > 
> > > Le 2015-04-22 07:40, Torben Knerr a écrit :
> > > 
> > > Hey everybody,
> > > 
> > > @jtimberman's recent blog post about test-driven infrastructure with  
> > > Chef made me thinking about this again.
> > > 
> > > All the tools we currently have are focused on cookbook testing, but  
> > > we have nothing established for acceptance level infrastructure  
> > > testing yet, do we?
> > > 
> > > I mean specifically:
> > > 
> > > - testing from the outside vs from the inside
> > > - testing multi-vm setups
> > > 
> > > See also my comment on the original post:
> 
> [https://www.chef.io/blog/2015/04/21/overview-of-test-driven-infrastructure-with-chef/](https://www.chef.io/blog/2015/04/21/overview-of-test-driven-infrastructure-with-chef/)
> 
> > > What do we have in this space?
> > > 
> > > Cheers,  
> > > Torben

---

<div class="post-metadata">

**Author:** ![Lamont\_Granquist](https://avatars.discourse-cdn.com/v4/letter/l/d26b3c/32.png) [@Lamont\_Granquist](https://discourse.chef.io/u/Lamont_Granquist)\
**Post date:** [April 22, 2015, 7:06pm UTC](https://discourse.chef.io/t/testing-cookbooks-vs-testing-infrastructure/6507/9 "2015-04-22T19:06:17Z")

</div>

On 4/22/15 11:28 AM, Torben Knerr wrote:

> Sorry, I was probably not specific enough 🙂
> 
> Serverspec is asserting state by ssh'ing into the VM, then checking  
> for processes, files being present, services being started, etc... (I  
> call that "from the inside")
> 
> However, if all that works, it doesn't necessarily mean that it works  
> for the end user (lets say the webapp is served on port 80 and  
> accessible "from outside" of the VM. That's probably a grey zone where  
> infrastructure testing ends and you the acceptance testing for a  
> specific application starts.
> 
> A more concrete example: say I have a cookbook for setting up  
> JenkinsCI with Slaves on multiple nodes, with nginx in front, etc..  
> I'd like to have a test suite that:
> 
> - sets up the multi-node scenario
> - then verifies that the front page is served...
> - ...and that you can create a basic job that is executed on a slave
> - ...and so on (some basic acceptance level smoke tests)
> 
> That's what I'm up to... did it get clearer?

My intuition is that the way that you're thinking about that violates  
promise theory a bit. All of your tests should be "from the inside".  
The VM that your webapp is being deployed on should really only test the  
world from its perspective. It should test that its listening on its  
http port and that basic API calls can succeed and that it can make  
calls to a database. Trying to think about it begin responsible for  
testing its own VIP violates promise theory since now its responsible  
for distant infrastructure and it doesn't have the information that it  
may need (it could be failing to register correctly in the VIP but  
another server in the cluster could be serving content so the VIP is  
up). Its better to add tests to the loadbalancer "from the inside" so  
that the load balancer checks itself and validates that all its  
exepected clients are listed as avaiable and health checks are passing  
and the loadbalancer should be checking its own VIP.

Then what we need is a multi-node test-kitchen which used something like  
chef-provisioning to spin up all of these VMs and fire off the  
serverspec runs on all of them, and then report back. For deployed  
infrastructure then audit mode could be running serverspec tests (and  
ideally hook busser up to audit mode so you could audit with bats tests  
and whatever) and you'd be getting reports back on every chef-client  
converge or audit mode run.

---

<div class="post-metadata">

**Author:** ![Tensibai](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/tensibai/32/29_2.png) [@Tensibai](https://discourse.chef.io/u/Tensibai)\
**Post date:** [April 22, 2015, 8:05pm UTC](https://discourse.chef.io/t/testing-cookbooks-vs-testing-infrastructure/6507/10 "2015-04-22T20:05:39Z")

</div>

Oops, forgot to say we're doing this integration tests with gatling to test the service from Jenkins but targeting the app server directly and not through all the bricks a user will

Le 22 avr. 2015 20:28, Torben Knerr [mail@tknerr.de](mailto:mail@tknerr.de) a écrit :

> Sorry, I was probably not specific enough 🙂
> 
> Serverspec is asserting state by ssh'ing into the VM, then checking  
> for processes, files being present, services being started, etc... (I  
> call that "from the inside")
> 
> However, if all that works, it doesn't necessarily mean that it works  
> for the end user (lets say the webapp is served on port 80 and  
> accessible "from outside" of the VM. That's probably a grey zone where  
> infrastructure testing ends and you the acceptance testing for a  
> specific application starts.
> 
> A more concrete example: say I have a cookbook for setting up  
> JenkinsCI with Slaves on multiple nodes, with nginx in front, etc..  
> I'd like to have a test suite that:
> 
> - sets up the multi-node scenario
> - then verifies that the front page is served...
> - ...and that you can create a basic job that is executed on a slave
> - ...and so on (some basic acceptance level smoke tests)
> 
> That's what I'm up to... did it get clearer?
> 
> Cheers,  
> Torben
> 
> On Wed, Apr 22, 2015 at 3:54 PM, Tensibai [tensibai@iabis.net](mailto:tensibai@iabis.net) wrote:
> 
> > We wish to have integration tests launched from jenkins too, that's not  
> > really outside, but the idea is to test the functional state in IC before  
> > promotion to QA with human testers behind load balancer and reverse proxy.
> > 
> > I may misunderstand what you think of under acceptance term, sorry if it's  
> > the case 🙂
> > 
> > Le 2015-04-22 15:37, Torben Knerr a écrit :
> > 
> > Are you planning to (acceptance level) test it from the outside as well?
> > 
> > Or just for convergence to succeed?
> > 
> > Am 22.04.2015 14:46 schrieb "Tensibai" [tensibai@iabis.net](mailto:tensibai@iabis.net):
> > 
> > > I forgot the link to terraform: [What is Terraform | Terraform | HashiCorp Developer](http://www.terraform.io/intro/index.html)
> > > 
> > > Le 2015-04-22 14:41, Tensibai a écrit :
> > > 
> > > I had a discussion with a colleague this morning about this and pointed to  
> > > Terraform[1].
> > > 
> > > As we're already using vagrant with a custom provider (for vsphere) we're  
> > > planning to move toward this path. for now it's just an idea with no  
> > > feedback but the described way Terraform works sounds interesting to write  
> > > "plans" to ensure idem-potency for cookbooks (spin up a machine, converge it  
> > > once, re-converge it, change a parameter somewhere, converge again, ensure  
> > > expectation are there).
> > > 
> > > That's what we're aiming to do, launched from jenkins on cookbook commit  
> > > after usual linting/chefspec test.
> > > 
> > > I'll try to write something about it if we end up with something  
> > > functional at end.
> > > 
> > > Le 2015-04-22 07:40, Torben Knerr a écrit :
> > > 
> > > Hey everybody,
> > > 
> > > @jtimberman's recent blog post about test-driven infrastructure with  
> > > Chef made me thinking about this again.
> > > 
> > > All the tools we currently have are focused on cookbook testing, but  
> > > we have nothing established for acceptance level infrastructure  
> > > testing yet, do we?
> > > 
> > > I mean specifically:
> > > 
> > > - testing from the outside vs from the inside
> > > - testing multi-vm setups
> > > 
> > > See also my comment on the original post:
> > > 
> > > [https://www.chef.io/blog/2015/04/21/overview-of-test-driven-infrastructure-with-chef/](https://www.chef.io/blog/2015/04/21/overview-of-test-driven-infrastructure-with-chef/)
> > > 
> > > What do we have in this space?
> > > 
> > > Cheers,  
> > > Torben

---

<div class="post-metadata">

**Author:** ![zts](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/zts/32/111_2.png) [@zts](https://discourse.chef.io/u/zts)\
**Post date:** [April 22, 2015, 9:02pm UTC](https://discourse.chef.io/t/testing-cookbooks-vs-testing-infrastructure/6507/11 "2015-04-22T21:02:00Z")

</div>

On Wed, Apr 22, 2015 at 8:06 PM, Lamont Granquist [lamont@chef.io](mailto:lamont@chef.io) wrote:

> On 4/22/15 11:28 AM, Torben Knerr wrote:
> 
> > A more concrete example: say I have a cookbook for setting up  
> > JenkinsCI with Slaves on multiple nodes, with nginx in front, etc..  
> > I'd like to have a test suite that:
> > 
> > - sets up the multi-node scenario
> > - then verifies that the front page is served...
> > - ...and that you can create a basic job that is executed on a slave
> > - ...and so on (some basic acceptance level smoke tests)
> > 
> > That's what I'm up to... did it get clearer?
> > 
> > My intuition is that the way that you're thinking about that violates  
> > promise theory a bit. All of your tests should be "from the inside". The  
> > VM that your webapp is being deployed on should really only test the world  
> > from its perspective. It should test that its listening on its http port  
> > and that basic API calls can succeed and that it can make calls to a  
> > database. Trying to think about it begin responsible for testing its own  
> > VIP violates promise theory since now its responsible for distant  
> > infrastructure and it doesn't have the information that it may need (it  
> > could be failing to register correctly in the VIP but another server in the  
> > cluster could be serving content so the VIP is up). Its better to add  
> > tests to the loadbalancer "from the inside" so that the load balancer  
> > checks itself and validates that all its exepected clients are listed as  
> > avaiable and health checks are passing and the loadbalancer should be  
> > checking its own VIP.
> 
> Then what we need is a multi-node test-kitchen which used something like  
> chef-provisioning to spin up all of these VMs and fire off the serverspec  
> runs on all of them, and then report back. For deployed infrastructure  
> then audit mode could be running serverspec tests (and ideally hook busser  
> up to audit mode so you could audit with bats tests and whatever) and you'd  
> be getting reports back on every chef-client converge or audit mode run.

I really don't understand the conclusion that there _should not_ be any  
tests "from the outside". (I also don't see anything in Torben's example  
prescribing where the outside-in tests should be run from)

All the tests described above are valuable, but testing that individual  
components work as expected is not a substitute for verifying that they  
work as expected when assembled. Or put another way, unit tests are great  
but integration tests are also valuable - I'm of the opinion that this is  
true whether the units you're dealing with are classes or servers.

I'd love to see multi-node testing in test-kitchen. I'd love to spin up a  
set of VMs that are configured to work together. I'd love to run  
serverspec suites on each of them, to assert their behaviour at that  
level. I'd love to spin up another VM to run additional outside-in tests  
making assertions about the behaviour of the whole. This would be _slow_,  
and I wouldn't be running these integration tests unless everything looked  
good, but they would be _meaningful_.

Here's the kicker: I thought that this was obvious, something that everyone  
wanted to do, and it was simply a matter of time before Test Kitchen  
evolved to deliver it. Now you've given me pause for thought.

Is this a philosophical question and I didn't realise I was on a side?  
Is there no debate and I'm simply wrong?  
Did I completely misunderstand your point, and now look a bit silly for  
writing this?

Zac

---

<div class="post-metadata">

**Author:** ![Nate\_DAmico](https://avatars.discourse-cdn.com/v4/letter/n/848f3c/32.png) [@Nate\_DAmico](https://discourse.chef.io/u/Nate_DAmico)\
**Post date:** [April 22, 2015, 9:02pm UTC](https://discourse.chef.io/t/testing-cookbooks-vs-testing-infrastructure/6507/12 "2015-04-22T21:02:55Z")

</div>

We have been doing range of this type of testing for while..., serverspec  
rocks, everybody testing infrastructure/service should be using it. It can  
be used locally or remote via ssh.., we primarily use locally. Our style  
is in test mode run serverspec, and possibly other test scripts, on each  
node with agent. We run an orchestrator to do mix of testing consisting of  
a couple modes:

-post deploy+config testing/assertion  
-post stage assertion (ie: spin up nodes 1,2,3.., configure stages 1,2..,  
then assert things are correct, if not, error out)

To @lamont's point.., if you are automating nodes, configuration, network,  
etc.., as part of a service rollout you might want/need to run external  
tests to assert the things are configured correctly for connectivity and  
certain SLA's that need to be met. We typically will coordinate a bastion  
to do some of this work, or some equivalent to test for instance  
connectivity between an app node and db node for example.

On Wed, Apr 22, 2015 at 1:05 PM, Tensibai Zhaoying [tensibai@iabis.net](mailto:tensibai@iabis.net)  
wrote:

> Oops, forgot to say we're doing this integration tests with gatling to  
> test the service from Jenkins but targeting the app server directly and not  
> through all the bricks a user will
> 
> Le 22 avr. 2015 20:28, Torben Knerr [mail@tknerr.de](mailto:mail@tknerr.de) a écrit :
> 
> > Sorry, I was probably not specific enough 🙂
> > 
> > Serverspec is asserting state by ssh'ing into the VM, then checking  
> > for processes, files being present, services being started, etc... (I  
> > call that "from the inside")
> > 
> > However, if all that works, it doesn't necessarily mean that it works  
> > for the end user (lets say the webapp is served on port 80 and  
> > accessible "from outside" of the VM. That's probably a grey zone where  
> > infrastructure testing ends and you the acceptance testing for a  
> > specific application starts.
> > 
> > A more concrete example: say I have a cookbook for setting up  
> > JenkinsCI with Slaves on multiple nodes, with nginx in front, etc..  
> > I'd like to have a test suite that:
> > 
> > - sets up the multi-node scenario
> > - then verifies that the front page is served...
> > - ...and that you can create a basic job that is executed on a slave
> > - ...and so on (some basic acceptance level smoke tests)
> > 
> > That's what I'm up to... did it get clearer?
> > 
> > Cheers,  
> > Torben
> > 
> > On Wed, Apr 22, 2015 at 3:54 PM, Tensibai [tensibai@iabis.net](mailto:tensibai@iabis.net) wrote:
> > 
> > > We wish to have integration tests launched from jenkins too, that's not  
> > > really outside, but the idea is to test the functional state in IC  
> > > before  
> > > promotion to QA with human testers behind load balancer and reverse  
> > > proxy.
> > > 
> > > I may misunderstand what you think of under acceptance term, sorry if  
> > > it's  
> > > the case 🙂
> > > 
> > > Le 2015-04-22 15:37, Torben Knerr a écrit :
> > > 
> > > Are you planning to (acceptance level) test it from the outside as  
> > > well?
> > > 
> > > Or just for convergence to succeed?
> > > 
> > > Am 22.04.2015 14:46 schrieb "Tensibai" [tensibai@iabis.net](mailto:tensibai@iabis.net):
> > > 
> > > > I forgot the link to terraform:  
> > > > [What is Terraform | Terraform | HashiCorp Developer](http://www.terraform.io/intro/index.html)
> > > > 
> > > > Le 2015-04-22 14:41, Tensibai a écrit :
> > > > 
> > > > I had a discussion with a colleague this morning about this and  
> > > > pointed to  
> > > > Terraform[1].
> > > > 
> > > > As we're already using vagrant with a custom provider (for vsphere)  
> > > > we're  
> > > > planning to move toward this path. for now it's just an idea with no  
> > > > feedback but the described way Terraform works sounds interesting to  
> > > > write  
> > > > "plans" to ensure idem-potency for cookbooks (spin up a machine,  
> > > > converge it  
> > > > once, re-converge it, change a parameter somewhere, converge again,  
> > > > ensure  
> > > > expectation are there).
> > > > 
> > > > That's what we're aiming to do, launched from jenkins on cookbook  
> > > > commit  
> > > > after usual linting/chefspec test.
> > > > 
> > > > I'll try to write something about it if we end up with something  
> > > > functional at end.
> > > > 
> > > > Le 2015-04-22 07:40, Torben Knerr a écrit :
> > > > 
> > > > Hey everybody,
> > > > 
> > > > @jtimberman's recent blog post about test-driven infrastructure with  
> > > > Chef made me thinking about this again.
> > > > 
> > > > All the tools we currently have are focused on cookbook testing, but  
> > > > we have nothing established for acceptance level infrastructure  
> > > > testing yet, do we?
> > > > 
> > > > I mean specifically:
> > > > 
> > > > - testing from the outside vs from the inside
> > > > - testing multi-vm setups
> > > > 
> > > > See also my comment on the original post:
> 
> [https://www.chef.io/blog/2015/04/21/overview-of-test-driven-infrastructure-with-chef/](https://www.chef.io/blog/2015/04/21/overview-of-test-driven-infrastructure-with-chef/)
> 
> > > > What do we have in this space?
> > > > 
> > > > Cheers,  
> > > > Torben

---

<div class="post-metadata">

**Author:** ![Lamont\_Granquist](https://avatars.discourse-cdn.com/v4/letter/l/d26b3c/32.png) [@Lamont\_Granquist](https://discourse.chef.io/u/Lamont_Granquist)\
**Post date:** [April 22, 2015, 9:25pm UTC](https://discourse.chef.io/t/testing-cookbooks-vs-testing-infrastructure/6507/13 "2015-04-22T21:25:44Z")

</div>

On 4/22/15 2:02 PM, Zac Stevens wrote:

> On Wed, Apr 22, 2015 at 8:06 PM, Lamont Granquist \<[lamont@chef.io](mailto:lamont@chef.io)  
> [mailto:lamont@chef.io](mailto:lamont@chef.io)\> wrote:
> 
> ```
> On 4/22/15 11:28 AM, Torben Knerr wrote:
> 
> A more concrete example: say I have a cookbook for setting up
> JenkinsCI with Slaves on multiple nodes, with nginx in front,
> etc..
> I'd like to have a test suite that:
> - sets up the multi-node scenario
> - then verifies that the front page is served...
> - ...and that you can create a basic job that is executed on a
> slave
> - ...and so on (some basic acceptance level smoke tests)
> 
> That's what I'm up to... did it get clearer?
> 
> My intuition is that the way that you're thinking about that
> violates promise theory a bit. All of your tests should be "from
> the inside". The VM that your webapp is being deployed on should
> really only test the world from its perspective. It should test
> that its listening on its http port and that basic API calls can
> succeed and that it can make calls to a database. Trying to think
> about it begin responsible for testing its own VIP violates
> promise theory since now its responsible for distant
> infrastructure and it doesn't have the information that it may
> need (it could be failing to register correctly in the VIP but
> another server in the cluster could be serving content so the VIP
> is up). Its better to add tests to the loadbalancer "from the
> inside" so that the load balancer checks itself and validates that
> all its exepected clients are listed as avaiable and health checks
> are passing and the loadbalancer should be checking its own VIP.
> 
> Then what we need is a multi-node test-kitchen which used
> something like chef-provisioning to spin up all of these VMs and
> fire off the serverspec runs on all of them, and then report
> back. For deployed infrastructure then audit mode could be
> running serverspec tests (and ideally hook busser up to audit mode
> so you could audit with bats tests and whatever) and you'd be
> getting reports back on every chef-client converge or audit mode run.
> 
> ```
> 
> I really don't understand the conclusion that there _should not_ be  
> any tests "from the outside". (I also don't see anything in Torben's  
> example prescribing where the outside-in tests should be run from)
> 
> All the tests described above are valuable, but testing that  
> individual components work as expected is not a substitute for  
> verifying that they work as expected when assembled. Or put another  
> way, unit tests are great but integration tests are also valuable -  
> I'm of the opinion that this is true whether the units you're dealing  
> with are classes or servers.
> 
> I'd love to see multi-node testing in test-kitchen. I'd love to spin  
> up a set of VMs that are configured to work together. I'd love to run  
> serverspec suites on each of them, to assert their behaviour at that  
> level. I'd love to spin up another VM to run additional outside-in  
> tests making assertions about the behaviour of the whole. This would  
> be _slow_, and I wouldn't be running these integration tests unless  
> everything looked good, but they would be _meaningful_.
> 
> Here's the kicker: I thought that this was obvious, something that  
> everyone wanted to do, and it was simply a matter of time before Test  
> Kitchen evolved to deliver it. Now you've given me pause for thought.
> 
> Is this a philosophical question and I didn't realise I was on a side?  
> Is there no debate and I'm simply wrong?  
> Did I completely misunderstand your point, and now look a bit silly  
> for writing this?
> 
> Zac

This may be philosophical naval gazing.

The way that I'd test a cluster end-to-end would be to consider the load  
balancer part of the cluster and it should issue API calls to its own  
VIPs and should also assert that the health checks of all of its servers  
are healthy. But the way that I express that is via the load balancer  
testing itself, but that API is the externally available interface to  
the cluster which is what you are going to test against for  
'integration' or 'end-to-end' tests.

So I suspect we get the same test coverage, but I think its important  
the way that I state who is responsible for testing what, and I don't  
see the need for 'external' testing and more pieces bolted onto the  
infrastructure to do 'new' kinds of tests. I think the tools are  
already mostly there if you squint at it, and I agree that  
chef-provisioning and test-kitchen and serverspec/bats/etc seem to be  
the way forwards to deliver the whole thing and it will evolve to  
provide it. All we need is to make test-kitchen more cluster-aware and  
then the process of simply being able to spin up the cluster, login and  
converge chef on everything and then run the tests on each individual  
node should be sufficient. Even though there's no overall end-to-end  
testing framework being additionally added into test-kitchen, I don't  
think you'll miss it since by each node testing itself they also test  
their dependencies (it isn't like unit testing because you'll really be  
communicating with all your neighbors and not stubs or mocks of your  
neighbors -- if you fully test yourself, you should have also fully  
tested all of your adjacent dependencies and the whole should really be  
the sum of the parts).

---

<div class="post-metadata">

**Author:** ![Lamont\_Granquist](https://avatars.discourse-cdn.com/v4/letter/l/d26b3c/32.png) [@Lamont\_Granquist](https://discourse.chef.io/u/Lamont_Granquist)\
**Post date:** [April 22, 2015, 9:25pm UTC](https://discourse.chef.io/t/testing-cookbooks-vs-testing-infrastructure/6507/14 "2015-04-22T21:25:52Z")

</div>

On 4/22/15 2:02 PM, Zac Stevens wrote:

> On Wed, Apr 22, 2015 at 8:06 PM, Lamont Granquist \<[lamont@chef.io](mailto:lamont@chef.io)  
> [mailto:lamont@chef.io](mailto:lamont@chef.io)\> wrote:
> 
> ```
> On 4/22/15 11:28 AM, Torben Knerr wrote:
> 
> A more concrete example: say I have a cookbook for setting up
> JenkinsCI with Slaves on multiple nodes, with nginx in front,
> etc..
> I'd like to have a test suite that:
> - sets up the multi-node scenario
> - then verifies that the front page is served...
> - ...and that you can create a basic job that is executed on a
> slave
> - ...and so on (some basic acceptance level smoke tests)
> 
> That's what I'm up to... did it get clearer?
> 
> My intuition is that the way that you're thinking about that
> violates promise theory a bit. All of your tests should be "from
> the inside". The VM that your webapp is being deployed on should
> really only test the world from its perspective. It should test
> that its listening on its http port and that basic API calls can
> succeed and that it can make calls to a database. Trying to think
> about it begin responsible for testing its own VIP violates
> promise theory since now its responsible for distant
> infrastructure and it doesn't have the information that it may
> need (it could be failing to register correctly in the VIP but
> another server in the cluster could be serving content so the VIP
> is up). Its better to add tests to the loadbalancer "from the
> inside" so that the load balancer checks itself and validates that
> all its exepected clients are listed as avaiable and health checks
> are passing and the loadbalancer should be checking its own VIP.
> 
> Then what we need is a multi-node test-kitchen which used
> something like chef-provisioning to spin up all of these VMs and
> fire off the serverspec runs on all of them, and then report
> back. For deployed infrastructure then audit mode could be
> running serverspec tests (and ideally hook busser up to audit mode
> so you could audit with bats tests and whatever) and you'd be
> getting reports back on every chef-client converge or audit mode run.
> 
> ```
> 
> I really don't understand the conclusion that there _should not_ be  
> any tests "from the outside". (I also don't see anything in Torben's  
> example prescribing where the outside-in tests should be run from)
> 
> All the tests described above are valuable, but testing that  
> individual components work as expected is not a substitute for  
> verifying that they work as expected when assembled. Or put another  
> way, unit tests are great but integration tests are also valuable -  
> I'm of the opinion that this is true whether the units you're dealing  
> with are classes or servers.
> 
> I'd love to see multi-node testing in test-kitchen. I'd love to spin  
> up a set of VMs that are configured to work together. I'd love to run  
> serverspec suites on each of them, to assert their behaviour at that  
> level. I'd love to spin up another VM to run additional outside-in  
> tests making assertions about the behaviour of the whole. This would  
> be _slow_, and I wouldn't be running these integration tests unless  
> everything looked good, but they would be _meaningful_.
> 
> Here's the kicker: I thought that this was obvious, something that  
> everyone wanted to do, and it was simply a matter of time before Test  
> Kitchen evolved to deliver it. Now you've given me pause for thought.
> 
> Is this a philosophical question and I didn't realise I was on a side?  
> Is there no debate and I'm simply wrong?  
> Did I completely misunderstand your point, and now look a bit silly  
> for writing this?
> 
> Zac

This may be philosophical naval gazing.

The way that I'd test a cluster end-to-end would be to consider the load  
balancer part of the cluster and it should issue API calls to its own  
VIPs and should also assert that the health checks of all of its servers  
are healthy. But the way that I express that is via the load balancer  
testing itself, but that API is the externally available interface to  
the cluster which is what you are going to test against for  
'integration' or 'end-to-end' tests.

So I suspect we get the same test coverage, but I think its important  
the way that I state who is responsible for testing what, and I don't  
see the need for 'external' testing and more pieces bolted onto the  
infrastructure to do 'new' kinds of tests. I think the tools are  
already mostly there if you squint at it, and I agree that  
chef-provisioning and test-kitchen and serverspec/bats/etc seem to be  
the way forwards to deliver the whole thing and it will evolve to  
provide it. All we need is to make test-kitchen more cluster-aware and  
then the process of simply being able to spin up the cluster, login and  
converge chef on everything and then run the tests on each individual  
node should be sufficient. Even though there's no overall end-to-end  
testing framework being additionally added into test-kitchen, I don't  
think you'll miss it since by each node testing itself they also test  
their dependencies (it isn't like unit testing because you'll really be  
communicating with all your neighbors and not stubs or mocks of your  
neighbors -- if you fully test yourself, you should have also fully  
tested all of your adjacent dependencies and the whole should really be  
the sum of the parts).

---

<div class="post-metadata">

**Author:** ![tknerr](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/tknerr/32/21_2.png) [@tknerr](https://discourse.chef.io/u/tknerr)\
**Post date:** [April 23, 2015, 5:37am UTC](https://discourse.chef.io/t/testing-cookbooks-vs-testing-infrastructure/6507/15 "2015-04-23T05:37:42Z")

</div>

Hey,

thanks for all the responses!

Some things are getting clearer now for me.

So first: multi-node testing and testing "from the outside" (I'd call  
that acceptance / end-to-end testing) are two different pairs of  
shoes. They are orthogonal and we can consider them unrelated to each  
other.

I love test-kitchen the way it is, and multi-node testing would be a  
super useful addition for exactly the kinds of tests Lamont described  
above.

As for testing "from the outside", I'm probably just looking for tool  
/ plugin that would integrate to test-kitchen, but that would NOT be  
installed inside the VM via busser. Instead it should be testing the  
VM from the outside. As far as I understood, this would exclude  
serverspec, because serverspec is SSHing into the VM to run its tests,  
right?

Btw: has anyone looked into Leibiz actually?

> **[GitHub - Atalanta/leibniz: Leibniz is an integration testing framework for Chef](https://github.com/Atalanta/leibniz)**
>
> Leibniz is an integration testing framework for Chef - GitHub - Atalanta/leibniz: Leibniz is an integration testing framework for Chef

It appears to do some multi-node testing (reusing TK as it looks like)  
and it does it's testing from the outside via cucumber. I've seen  
cucumber being used a lot for that kind of "from the outside"  
acceptance testing, but personally rspec would be just a bit more  
appealing to me 😉

Cheers,  
Torben

On Wed, Apr 22, 2015 at 11:25 PM, Lamont Granquist [lamont@chef.io](mailto:lamont@chef.io) wrote:

> On 4/22/15 2:02 PM, Zac Stevens wrote:
> 
> On Wed, Apr 22, 2015 at 8:06 PM, Lamont Granquist [lamont@chef.io](mailto:lamont@chef.io) wrote:
> 
> > On 4/22/15 11:28 AM, Torben Knerr wrote:
> > 
> > > A more concrete example: say I have a cookbook for setting up  
> > > JenkinsCI with Slaves on multiple nodes, with nginx in front, etc..  
> > > I'd like to have a test suite that:
> > > 
> > > - sets up the multi-node scenario
> > > - then verifies that the front page is served...
> > > - ...and that you can create a basic job that is executed on a slave
> > > - ...and so on (some basic acceptance level smoke tests)
> > > 
> > > That's what I'm up to... did it get clearer?
> > 
> > My intuition is that the way that you're thinking about that violates  
> > promise theory a bit. All of your tests should be "from the inside". The  
> > VM that your webapp is being deployed on should really only test the world  
> > from its perspective. It should test that its listening on its http port  
> > and that basic API calls can succeed and that it can make calls to a  
> > database. Trying to think about it begin responsible for testing its own  
> > VIP violates promise theory since now its responsible for distant  
> > infrastructure and it doesn't have the information that it may need (it  
> > could be failing to register correctly in the VIP but another server in the  
> > cluster could be serving content so the VIP is up). Its better to add tests  
> > to the loadbalancer "from the inside" so that the load balancer checks  
> > itself and validates that all its exepected clients are listed as avaiable  
> > and health checks are passing and the loadbalancer should be checking its  
> > own VIP.
> > 
> > Then what we need is a multi-node test-kitchen which used something like  
> > chef-provisioning to spin up all of these VMs and fire off the serverspec  
> > runs on all of them, and then report back. For deployed infrastructure then  
> > audit mode could be running serverspec tests (and ideally hook busser up to  
> > audit mode so you could audit with bats tests and whatever) and you'd be  
> > getting reports back on every chef-client converge or audit mode run.
> 
> I really don't understand the conclusion that there _should not_ be any  
> tests "from the outside". (I also don't see anything in Torben's example  
> prescribing where the outside-in tests should be run from)
> 
> All the tests described above are valuable, but testing that individual  
> components work as expected is not a substitute for verifying that they work  
> as expected when assembled. Or put another way, unit tests are great but  
> integration tests are also valuable - I'm of the opinion that this is true  
> whether the units you're dealing with are classes or servers.
> 
> I'd love to see multi-node testing in test-kitchen. I'd love to spin up a  
> set of VMs that are configured to work together. I'd love to run serverspec  
> suites on each of them, to assert their behaviour at that level. I'd love  
> to spin up another VM to run additional outside-in tests making assertions  
> about the behaviour of the whole. This would be _slow_, and I wouldn't be  
> running these integration tests unless everything looked good, but they  
> would be _meaningful_.
> 
> Here's the kicker: I thought that this was obvious, something that everyone  
> wanted to do, and it was simply a matter of time before Test Kitchen evolved  
> to deliver it. Now you've given me pause for thought.
> 
> Is this a philosophical question and I didn't realise I was on a side?  
> Is there no debate and I'm simply wrong?  
> Did I completely misunderstand your point, and now look a bit silly for  
> writing this?
> 
> Zac
> 
> This may be philosophical naval gazing.
> 
> The way that I'd test a cluster end-to-end would be to consider the load  
> balancer part of the cluster and it should issue API calls to its own VIPs  
> and should also assert that the health checks of all of its servers are  
> healthy. But the way that I express that is via the load balancer testing  
> itself, but that API is the externally available interface to the cluster  
> which is what you are going to test against for 'integration' or  
> 'end-to-end' tests.
> 
> So I suspect we get the same test coverage, but I think its important the  
> way that I state who is responsible for testing what, and I don't see the  
> need for 'external' testing and more pieces bolted onto the infrastructure  
> to do 'new' kinds of tests. I think the tools are already mostly there if  
> you squint at it, and I agree that chef-provisioning and test-kitchen and  
> serverspec/bats/etc seem to be the way forwards to deliver the whole thing  
> and it will evolve to provide it. All we need is to make test-kitchen more  
> cluster-aware and then the process of simply being able to spin up the  
> cluster, login and converge chef on everything and then run the tests on  
> each individual node should be sufficient. Even though there's no overall  
> end-to-end testing framework being additionally added into test-kitchen, I  
> don't think you'll miss it since by each node testing itself they also test  
> their dependencies (it isn't like unit testing because you'll really be  
> communicating with all your neighbors and not stubs or mocks of your  
> neighbors -- if you fully test yourself, you should have also fully tested  
> all of your adjacent dependencies and the whole should really be the sum of  
> the parts).

---

<div class="post-metadata">

**Author:** ![zts](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/zts/32/111_2.png) [@zts](https://discourse.chef.io/u/zts)\
**Post date:** [April 23, 2015, 8:20am UTC](https://discourse.chef.io/t/testing-cookbooks-vs-testing-infrastructure/6507/16 "2015-04-23T08:20:22Z")

</div>

On Thu, Apr 23, 2015 at 6:37 AM, Torben Knerr [mail@tknerr.de](mailto:mail@tknerr.de) wrote:

> As for testing "from the outside", I'm probably just looking for tool  
> / plugin that would integrate to test-kitchen, but that would NOT be  
> installed inside the VM via busser. Instead it should be testing the  
> VM from the outside. As far as I understood, this would exclude  
> serverspec, because serverspec is SSHing into the VM to run its tests,  
> right?

No, wrong - serverspec _can_ SSH into a remote host, and it can run tests  
locally (with `set :backend :exec`).  
As you observe, busser installs serverspec on the VM, copies the test suite  
to the VM, and runs the suite locally.

In the same space, you might also want to look at [http://infrataster.net](http://infrataster.net)

Zac

---

<div class="post-metadata">

**Author:** ![Cassiano\_Leal](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/cassiano_leal/32/86_2.png) [@Cassiano\_Leal](https://discourse.chef.io/u/Cassiano_Leal)\
**Post date:** [April 23, 2015, 8:27am UTC](https://discourse.chef.io/t/testing-cookbooks-vs-testing-infrastructure/6507/17 "2015-04-23T08:27:30Z")

</div>

That’s true, but unless I’ve missed something, that won’t do any outside-in testing either.

Serverspec always tests local resources, either by SSHing in first, or by assuming it’s already running on the node it will test. In either case it’s testing from within the node under test.

Thanks for the link to Infrataster, I hadn’t heard about it. 🙂

–  
Cassiano Leal  
On 23 April 2015 at 09:20:59, Zac Stevens ([zts@cryptocracy.com](mailto:zts@cryptocracy.com)) wrote:

On Thu, Apr 23, 2015 at 6:37 AM, Torben Knerr [mail@tknerr.de](mailto:mail@tknerr.de) wrote:  
As for testing “from the outside”, I’m probably just looking for tool  
/ plugin that would integrate to test-kitchen, but that would NOT be  
installed inside the VM via busser. Instead it should be testing the  
VM from the outside. As far as I understood, this would exclude  
serverspec, because serverspec is SSHing into the VM to run its tests,  
right?

No, wrong - serverspec _can_ SSH into a remote host, and it can run tests locally (with `set :backend :exec`).  
As you observe, busser installs serverspec on the VM, copies the test suite to the VM, and runs the suite locally.

In the same space, you might also want to look at [http://infrataster.net](http://infrataster.net)

Zac

---

<div class="post-metadata">

**Author:** ![tknerr](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/tknerr/32/21_2.png) [@tknerr](https://discourse.chef.io/u/tknerr)\
**Post date:** [April 23, 2015, 8:35am UTC](https://discourse.chef.io/t/testing-cookbooks-vs-testing-infrastructure/6507/18 "2015-04-23T08:35:14Z")

</div>

Oh, infratester is nice indeed!

kitchen-infratester anyone? 😉  
Am 23.04.2015 10:27 schrieb "Cassiano Leal" [cassianoleal@gmail.com](mailto:cassianoleal@gmail.com):

> That’s true, but unless I’ve missed something, that won’t do any  
> outside-in testing either.
> 
> Serverspec always tests local resources, either by SSHing in first, or by  
> assuming it’s already running on the node it will test. In either case it’s  
> testing from within the node under test.
> 
> Thanks for the link to Infrataster, I hadn’t heard about it. 🙂
> 
> --  
> Cassiano Leal
> 
> On 23 April 2015 at 09:20:59, Zac Stevens ([zts@cryptocracy.com](mailto:zts@cryptocracy.com)) wrote:
> 
> On Thu, Apr 23, 2015 at 6:37 AM, Torben Knerr [mail@tknerr.de](mailto:mail@tknerr.de) wrote:
> 
> > As for testing "from the outside", I'm probably just looking for tool  
> > / plugin that would integrate to test-kitchen, but that would NOT be  
> > installed inside the VM via busser. Instead it should be testing the  
> > VM from the outside. As far as I understood, this would exclude  
> > serverspec, because serverspec is SSHing into the VM to run its tests,  
> > right?
> 
> No, wrong - serverspec _can_ SSH into a remote host, and it can run tests  
> locally (with `set :backend :exec`).  
> As you observe, busser installs serverspec on the VM, copies the test  
> suite to the VM, and runs the suite locally.
> 
> In the same space, you might also want to look at [http://infrataster.net](http://infrataster.net)
> 
> Zac

---

<div class="post-metadata">

**Author:** ![zts](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/zts/32/111_2.png) [@zts](https://discourse.chef.io/u/zts)\
**Post date:** [April 23, 2015, 9:09am UTC](https://discourse.chef.io/t/testing-cookbooks-vs-testing-infrastructure/6507/19 "2015-04-23T09:09:21Z")

</div>

On Thursday, April 23, 2015, Cassiano Leal [cassianoleal@gmail.com](mailto:cassianoleal@gmail.com) wrote:

> That’s true, but unless I’ve missed something, that won’t do any  
> outside-in testing either.
> 
> Serverspec always tests local resources, either by SSHing in first, or by  
> assuming it’s already running on the node it will test. In either case it’s  
> testing from within the node under test.
> 
> Thanks for the link to Infrataster, I hadn’t heard about it. 🙂

My bad - I could have sworn it had an http resource, but apparently I've  
just been abusing the command resource to run functional tests 🙂

Zac