Testing Cookbooks vs Testing Infrastructure

Hey everybody,

@jtimberman's recent blog post about test-driven infrastructure with
Chef made me thinking about this again.

All the tools we currently have are focused on cookbook testing, but
we have nothing established for acceptance level infrastructure
testing yet, do we?

I mean specifically:

  • testing from the outside vs from the inside
  • testing multi-vm setups

See also my comment on the original post:

What do we have in this space?

Cheers,
Torben

I had a discussion with a colleague this morning about this and pointed
to Terraform[1].

As we're already using vagrant with a custom provider (for vsphere)
we're planning to move toward this path. for now it's just an idea with
no feedback but the described way Terraform works sounds interesting to
write "plans" to ensure idem-potency for cookbooks (spin up a machine,
converge it once, re-converge it, change a parameter somewhere, converge
again, ensure expectation are there).

That's what we're aiming to do, launched from jenkins on cookbook commit
after usual linting/chefspec test.

I'll try to write something about it if we end up with something
functional at end.

Le 2015-04-22 07:40, Torben Knerr a écrit :

Hey everybody,

@jtimberman's recent blog post about test-driven infrastructure with
Chef made me thinking about this again.

All the tools we currently have are focused on cookbook testing, but
we have nothing established for acceptance level infrastructure
testing yet, do we?

I mean specifically:

  • testing from the outside vs from the inside
  • testing multi-vm setups

See also my comment on the original post:
https://www.chef.io/blog/2015/04/21/overview-of-test-driven-infrastructure-with-chef/ [1]

What do we have in this space?

Cheers,
Torben

Links:

[1]
https://www.chef.io/blog/2015/04/21/overview-of-test-driven-infrastructure-with-chef/

I forgot the link to terraform: What is Terraform | Terraform | HashiCorp Developer

Le 2015-04-22 14:41, Tensibai a écrit :

I had a discussion with a colleague this morning about this and pointed to Terraform[1].

As we're already using vagrant with a custom provider (for vsphere) we're planning to move toward this path. for now it's just an idea with no feedback but the described way Terraform works sounds interesting to write "plans" to ensure idem-potency for cookbooks (spin up a machine, converge it once, re-converge it, change a parameter somewhere, converge again, ensure expectation are there).

That's what we're aiming to do, launched from jenkins on cookbook commit after usual linting/chefspec test.

I'll try to write something about it if we end up with something functional at end.

Le 2015-04-22 07:40, Torben Knerr a écrit :

Hey everybody,

@jtimberman's recent blog post about test-driven infrastructure with
Chef made me thinking about this again.

All the tools we currently have are focused on cookbook testing, but
we have nothing established for acceptance level infrastructure
testing yet, do we?

I mean specifically:

  • testing from the outside vs from the inside
  • testing multi-vm setups

See also my comment on the original post:
https://www.chef.io/blog/2015/04/21/overview-of-test-driven-infrastructure-with-chef/ [1]

What do we have in this space?

Cheers,
Torben

Links:

[1]
https://www.chef.io/blog/2015/04/21/overview-of-test-driven-infrastructure-with-chef/

Are you planning to (acceptance level) test it from the outside as well?

Or just for convergence to succeed?
Am 22.04.2015 14:46 schrieb "Tensibai" tensibai@iabis.net:

I forgot the link to terraform: What is Terraform | Terraform | HashiCorp Developer

Le 2015-04-22 14:41, Tensibai a écrit :

I had a discussion with a colleague this morning about this and pointed to
Terraform[1].

As we're already using vagrant with a custom provider (for vsphere) we're
planning to move toward this path. for now it's just an idea with no
feedback but the described way Terraform works sounds interesting to write
"plans" to ensure idem-potency for cookbooks (spin up a machine, converge
it once, re-converge it, change a parameter somewhere, converge again,
ensure expectation are there).

That's what we're aiming to do, launched from jenkins on cookbook commit
after usual linting/chefspec test.

I'll try to write something about it if we end up with something
functional at end.

Le 2015-04-22 07:40, Torben Knerr a écrit :

Hey everybody,

@jtimberman's recent blog post about test-driven infrastructure with
Chef made me thinking about this again.

All the tools we currently have are focused on cookbook testing, but
we have nothing established for acceptance level infrastructure
testing yet, do we?

I mean specifically:

  • testing from the outside vs from the inside
  • testing multi-vm setups

See also my comment on the original post:

https://www.chef.io/blog/2015/04/21/overview-of-test-driven-infrastructure-with-chef/

What do we have in this space?

Cheers,
Torben

We wish to have integration tests launched from jenkins too, that's not
really outside, but the idea is to test the functional state in IC
before promotion to QA with human testers behind load balancer and
reverse proxy.

I may misunderstand what you think of under acceptance term, sorry if
it's the case :slight_smile:

Le 2015-04-22 15:37, Torben Knerr a écrit :

Are you planning to (acceptance level) test it from the outside as well?

Or just for convergence to succeed?
Am 22.04.2015 14:46 schrieb "Tensibai" tensibai@iabis.net:

I forgot the link to terraform: What is Terraform | Terraform | HashiCorp Developer [1]

Le 2015-04-22 14:41, Tensibai a écrit :

I had a discussion with a colleague this morning about this and pointed to Terraform[1].

As we're already using vagrant with a custom provider (for vsphere) we're planning to move toward this path. for now it's just an idea with no feedback but the described way Terraform works sounds interesting to write "plans" to ensure idem-potency for cookbooks (spin up a machine, converge it once, re-converge it, change a parameter somewhere, converge again, ensure expectation are there).

That's what we're aiming to do, launched from jenkins on cookbook commit after usual linting/chefspec test.

I'll try to write something about it if we end up with something functional at end.

Le 2015-04-22 07:40, Torben Knerr a écrit :
Hey everybody,

@jtimberman's recent blog post about test-driven infrastructure with
Chef made me thinking about this again.

All the tools we currently have are focused on cookbook testing, but
we have nothing established for acceptance level infrastructure
testing yet, do we?

I mean specifically:

  • testing from the outside vs from the inside
  • testing multi-vm setups

See also my comment on the original post:
https://www.chef.io/blog/2015/04/21/overview-of-test-driven-infrastructure-with-chef/ [2]

What do we have in this space?

Cheers,
Torben

Links:

[1] What is Terraform | Terraform | HashiCorp Developer
[2]
https://www.chef.io/blog/2015/04/21/overview-of-test-driven-infrastructure-with-chef/

Sorry, I was probably not specific enough :slight_smile:

Serverspec is asserting state by ssh'ing into the VM, then checking
for processes, files being present, services being started, etc... (I
call that "from the inside")

However, if all that works, it doesn't necessarily mean that it works
for the end user (lets say the webapp is served on port 80 and
accessible "from outside" of the VM. That's probably a grey zone where
infrastructure testing ends and you the acceptance testing for a
specific application starts.

A more concrete example: say I have a cookbook for setting up
JenkinsCI with Slaves on multiple nodes, with nginx in front, etc..
I'd like to have a test suite that:

  • sets up the multi-node scenario
  • then verifies that the front page is served...
  • ...and that you can create a basic job that is executed on a slave
  • ...and so on (some basic acceptance level smoke tests)

That's what I'm up to... did it get clearer?

Cheers,
Torben

On Wed, Apr 22, 2015 at 3:54 PM, Tensibai tensibai@iabis.net wrote:

We wish to have integration tests launched from jenkins too, that's not
really outside, but the idea is to test the functional state in IC before
promotion to QA with human testers behind load balancer and reverse proxy.

I may misunderstand what you think of under acceptance term, sorry if it's
the case :slight_smile:

Le 2015-04-22 15:37, Torben Knerr a écrit :

Are you planning to (acceptance level) test it from the outside as well?

Or just for convergence to succeed?

Am 22.04.2015 14:46 schrieb "Tensibai" tensibai@iabis.net:

I forgot the link to terraform: What is Terraform | Terraform | HashiCorp Developer

Le 2015-04-22 14:41, Tensibai a écrit :

I had a discussion with a colleague this morning about this and pointed to
Terraform[1].

As we're already using vagrant with a custom provider (for vsphere) we're
planning to move toward this path. for now it's just an idea with no
feedback but the described way Terraform works sounds interesting to write
"plans" to ensure idem-potency for cookbooks (spin up a machine, converge it
once, re-converge it, change a parameter somewhere, converge again, ensure
expectation are there).

That's what we're aiming to do, launched from jenkins on cookbook commit
after usual linting/chefspec test.

I'll try to write something about it if we end up with something
functional at end.

Le 2015-04-22 07:40, Torben Knerr a écrit :

Hey everybody,

@jtimberman's recent blog post about test-driven infrastructure with
Chef made me thinking about this again.

All the tools we currently have are focused on cookbook testing, but
we have nothing established for acceptance level infrastructure
testing yet, do we?

I mean specifically:

  • testing from the outside vs from the inside
  • testing multi-vm setups

See also my comment on the original post:

https://www.chef.io/blog/2015/04/21/overview-of-test-driven-infrastructure-with-chef/

What do we have in this space?

Cheers,
Torben

That's probably best suited by more generalized testing suites like
Selenium, etc, for web services. However, I thought that serverspec also
supported basic curl someurl | grep 'somestring' as well? Maybe I'm
thinking of one of the other testing frameworks.

--
~~ StormeRider ~~

"Every world needs its heroes [...] They inspire us to be better than we
are. And they protect from the darkness that's just around the corner."

(from Smallville Season 6x1: "Zod")

On why I hate the phrase "that's so lame"... http://bit.ly/Ps3uSS

On Wed, Apr 22, 2015 at 11:28 AM, Torben Knerr mail@tknerr.de wrote:

Sorry, I was probably not specific enough :slight_smile:

Serverspec is asserting state by ssh'ing into the VM, then checking
for processes, files being present, services being started, etc... (I
call that "from the inside")

However, if all that works, it doesn't necessarily mean that it works
for the end user (lets say the webapp is served on port 80 and
accessible "from outside" of the VM. That's probably a grey zone where
infrastructure testing ends and you the acceptance testing for a
specific application starts.

A more concrete example: say I have a cookbook for setting up
JenkinsCI with Slaves on multiple nodes, with nginx in front, etc..
I'd like to have a test suite that:

  • sets up the multi-node scenario
  • then verifies that the front page is served...
  • ...and that you can create a basic job that is executed on a slave
  • ...and so on (some basic acceptance level smoke tests)

That's what I'm up to... did it get clearer?

Cheers,
Torben

On Wed, Apr 22, 2015 at 3:54 PM, Tensibai tensibai@iabis.net wrote:

We wish to have integration tests launched from jenkins too, that's not
really outside, but the idea is to test the functional state in IC before
promotion to QA with human testers behind load balancer and reverse
proxy.

I may misunderstand what you think of under acceptance term, sorry if
it's
the case :slight_smile:

Le 2015-04-22 15:37, Torben Knerr a écrit :

Are you planning to (acceptance level) test it from the outside as well?

Or just for convergence to succeed?

Am 22.04.2015 14:46 schrieb "Tensibai" tensibai@iabis.net:

I forgot the link to terraform:
What is Terraform | Terraform | HashiCorp Developer

Le 2015-04-22 14:41, Tensibai a écrit :

I had a discussion with a colleague this morning about this and pointed
to
Terraform[1].

As we're already using vagrant with a custom provider (for vsphere)
we're
planning to move toward this path. for now it's just an idea with no
feedback but the described way Terraform works sounds interesting to
write
"plans" to ensure idem-potency for cookbooks (spin up a machine,
converge it
once, re-converge it, change a parameter somewhere, converge again,
ensure
expectation are there).

That's what we're aiming to do, launched from jenkins on cookbook commit
after usual linting/chefspec test.

I'll try to write something about it if we end up with something
functional at end.

Le 2015-04-22 07:40, Torben Knerr a écrit :

Hey everybody,

@jtimberman's recent blog post about test-driven infrastructure with
Chef made me thinking about this again.

All the tools we currently have are focused on cookbook testing, but
we have nothing established for acceptance level infrastructure
testing yet, do we?

I mean specifically:

  • testing from the outside vs from the inside
  • testing multi-vm setups

See also my comment on the original post:

https://www.chef.io/blog/2015/04/21/overview-of-test-driven-infrastructure-with-chef/

What do we have in this space?

Cheers,
Torben

Torben,

I think it's easy to get really deep in how one wants to perform acceptance
testing.

In our deployment cycle, we deploy the changes to a node, then run a series
of phantomjs tests that load a web page, log in, clock a few things, assert
behavior.

This is totally customized for our app, but there's no reason anyone
couldn't write their own tests.

I don't think it's ever going to be the responsibility of
Test-Kitchen/Serverspec - rather these methods could use bussers to launch
your acceptance testing, or not - that's up to you.

-M

On Wed, Apr 22, 2015 at 2:28 PM, Torben Knerr mail@tknerr.de wrote:

Sorry, I was probably not specific enough :slight_smile:

Serverspec is asserting state by ssh'ing into the VM, then checking
for processes, files being present, services being started, etc... (I
call that "from the inside")

However, if all that works, it doesn't necessarily mean that it works
for the end user (lets say the webapp is served on port 80 and
accessible "from outside" of the VM. That's probably a grey zone where
infrastructure testing ends and you the acceptance testing for a
specific application starts.

A more concrete example: say I have a cookbook for setting up
JenkinsCI with Slaves on multiple nodes, with nginx in front, etc..
I'd like to have a test suite that:

  • sets up the multi-node scenario
  • then verifies that the front page is served...
  • ...and that you can create a basic job that is executed on a slave
  • ...and so on (some basic acceptance level smoke tests)

That's what I'm up to... did it get clearer?

Cheers,
Torben

On Wed, Apr 22, 2015 at 3:54 PM, Tensibai tensibai@iabis.net wrote:

We wish to have integration tests launched from jenkins too, that's not
really outside, but the idea is to test the functional state in IC before
promotion to QA with human testers behind load balancer and reverse
proxy.

I may misunderstand what you think of under acceptance term, sorry if
it's
the case :slight_smile:

Le 2015-04-22 15:37, Torben Knerr a écrit :

Are you planning to (acceptance level) test it from the outside as well?

Or just for convergence to succeed?

Am 22.04.2015 14:46 schrieb "Tensibai" tensibai@iabis.net:

I forgot the link to terraform:
What is Terraform | Terraform | HashiCorp Developer

Le 2015-04-22 14:41, Tensibai a écrit :

I had a discussion with a colleague this morning about this and pointed
to
Terraform[1].

As we're already using vagrant with a custom provider (for vsphere)
we're
planning to move toward this path. for now it's just an idea with no
feedback but the described way Terraform works sounds interesting to
write
"plans" to ensure idem-potency for cookbooks (spin up a machine,
converge it
once, re-converge it, change a parameter somewhere, converge again,
ensure
expectation are there).

That's what we're aiming to do, launched from jenkins on cookbook commit
after usual linting/chefspec test.

I'll try to write something about it if we end up with something
functional at end.

Le 2015-04-22 07:40, Torben Knerr a écrit :

Hey everybody,

@jtimberman's recent blog post about test-driven infrastructure with
Chef made me thinking about this again.

All the tools we currently have are focused on cookbook testing, but
we have nothing established for acceptance level infrastructure
testing yet, do we?

I mean specifically:

  • testing from the outside vs from the inside
  • testing multi-vm setups

See also my comment on the original post:

https://www.chef.io/blog/2015/04/21/overview-of-test-driven-infrastructure-with-chef/

What do we have in this space?

Cheers,
Torben

On 4/22/15 11:28 AM, Torben Knerr wrote:

Sorry, I was probably not specific enough :slight_smile:

Serverspec is asserting state by ssh'ing into the VM, then checking
for processes, files being present, services being started, etc... (I
call that "from the inside")

However, if all that works, it doesn't necessarily mean that it works
for the end user (lets say the webapp is served on port 80 and
accessible "from outside" of the VM. That's probably a grey zone where
infrastructure testing ends and you the acceptance testing for a
specific application starts.

A more concrete example: say I have a cookbook for setting up
JenkinsCI with Slaves on multiple nodes, with nginx in front, etc..
I'd like to have a test suite that:

  • sets up the multi-node scenario
  • then verifies that the front page is served...
  • ...and that you can create a basic job that is executed on a slave
  • ...and so on (some basic acceptance level smoke tests)

That's what I'm up to... did it get clearer?

My intuition is that the way that you're thinking about that violates
promise theory a bit. All of your tests should be "from the inside".
The VM that your webapp is being deployed on should really only test the
world from its perspective. It should test that its listening on its
http port and that basic API calls can succeed and that it can make
calls to a database. Trying to think about it begin responsible for
testing its own VIP violates promise theory since now its responsible
for distant infrastructure and it doesn't have the information that it
may need (it could be failing to register correctly in the VIP but
another server in the cluster could be serving content so the VIP is
up). Its better to add tests to the loadbalancer "from the inside" so
that the load balancer checks itself and validates that all its
exepected clients are listed as avaiable and health checks are passing
and the loadbalancer should be checking its own VIP.

Then what we need is a multi-node test-kitchen which used something like
chef-provisioning to spin up all of these VMs and fire off the
serverspec runs on all of them, and then report back. For deployed
infrastructure then audit mode could be running serverspec tests (and
ideally hook busser up to audit mode so you could audit with bats tests
and whatever) and you'd be getting reports back on every chef-client
converge or audit mode run.

Oops, forgot to say we're doing this integration tests with gatling to test the service from Jenkins but targeting the app server directly and not through all the bricks a user will

Le 22 avr. 2015 20:28, Torben Knerr mail@tknerr.de a écrit :

Sorry, I was probably not specific enough :slight_smile:

Serverspec is asserting state by ssh'ing into the VM, then checking
for processes, files being present, services being started, etc... (I
call that "from the inside")

However, if all that works, it doesn't necessarily mean that it works
for the end user (lets say the webapp is served on port 80 and
accessible "from outside" of the VM. That's probably a grey zone where
infrastructure testing ends and you the acceptance testing for a
specific application starts.

A more concrete example: say I have a cookbook for setting up
JenkinsCI with Slaves on multiple nodes, with nginx in front, etc..
I'd like to have a test suite that:

  • sets up the multi-node scenario
  • then verifies that the front page is served...
  • ...and that you can create a basic job that is executed on a slave
  • ...and so on (some basic acceptance level smoke tests)

That's what I'm up to... did it get clearer?

Cheers,
Torben

On Wed, Apr 22, 2015 at 3:54 PM, Tensibai tensibai@iabis.net wrote:

We wish to have integration tests launched from jenkins too, that's not
really outside, but the idea is to test the functional state in IC before
promotion to QA with human testers behind load balancer and reverse proxy.

I may misunderstand what you think of under acceptance term, sorry if it's
the case :slight_smile:

Le 2015-04-22 15:37, Torben Knerr a écrit :

Are you planning to (acceptance level) test it from the outside as well?

Or just for convergence to succeed?

Am 22.04.2015 14:46 schrieb "Tensibai" tensibai@iabis.net:

I forgot the link to terraform: What is Terraform | Terraform | HashiCorp Developer

Le 2015-04-22 14:41, Tensibai a écrit :

I had a discussion with a colleague this morning about this and pointed to
Terraform[1].

As we're already using vagrant with a custom provider (for vsphere) we're
planning to move toward this path. for now it's just an idea with no
feedback but the described way Terraform works sounds interesting to write
"plans" to ensure idem-potency for cookbooks (spin up a machine, converge it
once, re-converge it, change a parameter somewhere, converge again, ensure
expectation are there).

That's what we're aiming to do, launched from jenkins on cookbook commit
after usual linting/chefspec test.

I'll try to write something about it if we end up with something
functional at end.

Le 2015-04-22 07:40, Torben Knerr a écrit :

Hey everybody,

@jtimberman's recent blog post about test-driven infrastructure with
Chef made me thinking about this again.

All the tools we currently have are focused on cookbook testing, but
we have nothing established for acceptance level infrastructure
testing yet, do we?

I mean specifically:

  • testing from the outside vs from the inside
  • testing multi-vm setups

See also my comment on the original post:

https://www.chef.io/blog/2015/04/21/overview-of-test-driven-infrastructure-with-chef/

What do we have in this space?

Cheers,
Torben

On Wed, Apr 22, 2015 at 8:06 PM, Lamont Granquist lamont@chef.io wrote:

On 4/22/15 11:28 AM, Torben Knerr wrote:

A more concrete example: say I have a cookbook for setting up
JenkinsCI with Slaves on multiple nodes, with nginx in front, etc..
I'd like to have a test suite that:

  • sets up the multi-node scenario
  • then verifies that the front page is served...
  • ...and that you can create a basic job that is executed on a slave
  • ...and so on (some basic acceptance level smoke tests)

That's what I'm up to... did it get clearer?

My intuition is that the way that you're thinking about that violates
promise theory a bit. All of your tests should be "from the inside". The
VM that your webapp is being deployed on should really only test the world
from its perspective. It should test that its listening on its http port
and that basic API calls can succeed and that it can make calls to a
database. Trying to think about it begin responsible for testing its own
VIP violates promise theory since now its responsible for distant
infrastructure and it doesn't have the information that it may need (it
could be failing to register correctly in the VIP but another server in the
cluster could be serving content so the VIP is up). Its better to add
tests to the loadbalancer "from the inside" so that the load balancer
checks itself and validates that all its exepected clients are listed as
avaiable and health checks are passing and the loadbalancer should be
checking its own VIP.

Then what we need is a multi-node test-kitchen which used something like
chef-provisioning to spin up all of these VMs and fire off the serverspec
runs on all of them, and then report back. For deployed infrastructure
then audit mode could be running serverspec tests (and ideally hook busser
up to audit mode so you could audit with bats tests and whatever) and you'd
be getting reports back on every chef-client converge or audit mode run.

I really don't understand the conclusion that there should not be any
tests "from the outside". (I also don't see anything in Torben's example
prescribing where the outside-in tests should be run from)

All the tests described above are valuable, but testing that individual
components work as expected is not a substitute for verifying that they
work as expected when assembled. Or put another way, unit tests are great
but integration tests are also valuable - I'm of the opinion that this is
true whether the units you're dealing with are classes or servers.

I'd love to see multi-node testing in test-kitchen. I'd love to spin up a
set of VMs that are configured to work together. I'd love to run
serverspec suites on each of them, to assert their behaviour at that
level. I'd love to spin up another VM to run additional outside-in tests
making assertions about the behaviour of the whole. This would be slow,
and I wouldn't be running these integration tests unless everything looked
good, but they would be meaningful.

Here's the kicker: I thought that this was obvious, something that everyone
wanted to do, and it was simply a matter of time before Test Kitchen
evolved to deliver it. Now you've given me pause for thought.

Is this a philosophical question and I didn't realise I was on a side?
Is there no debate and I'm simply wrong?
Did I completely misunderstand your point, and now look a bit silly for
writing this?

Zac

We have been doing range of this type of testing for while..., serverspec
rocks, everybody testing infrastructure/service should be using it. It can
be used locally or remote via ssh.., we primarily use locally. Our style
is in test mode run serverspec, and possibly other test scripts, on each
node with agent. We run an orchestrator to do mix of testing consisting of
a couple modes:

-post deploy+config testing/assertion
-post stage assertion (ie: spin up nodes 1,2,3.., configure stages 1,2..,
then assert things are correct, if not, error out)

To @lamont's point.., if you are automating nodes, configuration, network,
etc.., as part of a service rollout you might want/need to run external
tests to assert the things are configured correctly for connectivity and
certain SLA's that need to be met. We typically will coordinate a bastion
to do some of this work, or some equivalent to test for instance
connectivity between an app node and db node for example.

On Wed, Apr 22, 2015 at 1:05 PM, Tensibai Zhaoying tensibai@iabis.net
wrote:

Oops, forgot to say we're doing this integration tests with gatling to
test the service from Jenkins but targeting the app server directly and not
through all the bricks a user will

Le 22 avr. 2015 20:28, Torben Knerr mail@tknerr.de a écrit :

Sorry, I was probably not specific enough :slight_smile:

Serverspec is asserting state by ssh'ing into the VM, then checking
for processes, files being present, services being started, etc... (I
call that "from the inside")

However, if all that works, it doesn't necessarily mean that it works
for the end user (lets say the webapp is served on port 80 and
accessible "from outside" of the VM. That's probably a grey zone where
infrastructure testing ends and you the acceptance testing for a
specific application starts.

A more concrete example: say I have a cookbook for setting up
JenkinsCI with Slaves on multiple nodes, with nginx in front, etc..
I'd like to have a test suite that:

  • sets up the multi-node scenario
  • then verifies that the front page is served...
  • ...and that you can create a basic job that is executed on a slave
  • ...and so on (some basic acceptance level smoke tests)

That's what I'm up to... did it get clearer?

Cheers,
Torben

On Wed, Apr 22, 2015 at 3:54 PM, Tensibai tensibai@iabis.net wrote:

We wish to have integration tests launched from jenkins too, that's not
really outside, but the idea is to test the functional state in IC
before
promotion to QA with human testers behind load balancer and reverse
proxy.

I may misunderstand what you think of under acceptance term, sorry if
it's
the case :slight_smile:

Le 2015-04-22 15:37, Torben Knerr a écrit :

Are you planning to (acceptance level) test it from the outside as
well?

Or just for convergence to succeed?

Am 22.04.2015 14:46 schrieb "Tensibai" tensibai@iabis.net:

I forgot the link to terraform:
What is Terraform | Terraform | HashiCorp Developer

Le 2015-04-22 14:41, Tensibai a écrit :

I had a discussion with a colleague this morning about this and
pointed to
Terraform[1].

As we're already using vagrant with a custom provider (for vsphere)
we're
planning to move toward this path. for now it's just an idea with no
feedback but the described way Terraform works sounds interesting to
write
"plans" to ensure idem-potency for cookbooks (spin up a machine,
converge it
once, re-converge it, change a parameter somewhere, converge again,
ensure
expectation are there).

That's what we're aiming to do, launched from jenkins on cookbook
commit
after usual linting/chefspec test.

I'll try to write something about it if we end up with something
functional at end.

Le 2015-04-22 07:40, Torben Knerr a écrit :

Hey everybody,

@jtimberman's recent blog post about test-driven infrastructure with
Chef made me thinking about this again.

All the tools we currently have are focused on cookbook testing, but
we have nothing established for acceptance level infrastructure
testing yet, do we?

I mean specifically:

  • testing from the outside vs from the inside
  • testing multi-vm setups

See also my comment on the original post:

https://www.chef.io/blog/2015/04/21/overview-of-test-driven-infrastructure-with-chef/

What do we have in this space?

Cheers,
Torben

On 4/22/15 2:02 PM, Zac Stevens wrote:

On Wed, Apr 22, 2015 at 8:06 PM, Lamont Granquist <lamont@chef.io
mailto:lamont@chef.io> wrote:

On 4/22/15 11:28 AM, Torben Knerr wrote:

    A more concrete example: say I have a cookbook for setting up
    JenkinsCI with Slaves on multiple nodes, with nginx in front,
    etc..
    I'd like to have a test suite that:
    - sets up the multi-node scenario
    - then verifies that the front page is served...
    - ...and that you can create a basic job that is executed on a
    slave
    - ...and so on (some basic acceptance level smoke tests)

    That's what I'm up to... did it get clearer?

My intuition is that the way that you're thinking about that
violates promise theory a bit.  All of your tests should be "from
the inside".  The VM that your webapp is being deployed on should
really only test the world from its perspective.  It should test
that its listening on its http port and that basic API calls can
succeed and that it can make calls to a database.  Trying to think
about it begin responsible for testing its own VIP violates
promise theory since now its responsible for distant
infrastructure and it doesn't have the information that it may
need (it could be failing to register correctly in the VIP but
another server in the cluster could be serving content so the VIP
is up).  Its better to add tests to the loadbalancer "from the
inside" so that the load balancer checks itself and validates that
all its exepected clients are listed as avaiable and health checks
are passing and the loadbalancer should be checking its own VIP.

Then what we need is a multi-node test-kitchen which used
something like chef-provisioning to spin up all of these VMs and
fire off the serverspec runs on all of them, and then report
back.  For deployed infrastructure then audit mode could be
running serverspec tests (and ideally hook busser up to audit mode
so you could audit with bats tests and whatever) and you'd be
getting reports back on every chef-client converge or audit mode run.

I really don't understand the conclusion that there should not be
any tests "from the outside". (I also don't see anything in Torben's
example prescribing where the outside-in tests should be run from)

All the tests described above are valuable, but testing that
individual components work as expected is not a substitute for
verifying that they work as expected when assembled. Or put another
way, unit tests are great but integration tests are also valuable -
I'm of the opinion that this is true whether the units you're dealing
with are classes or servers.

I'd love to see multi-node testing in test-kitchen. I'd love to spin
up a set of VMs that are configured to work together. I'd love to run
serverspec suites on each of them, to assert their behaviour at that
level. I'd love to spin up another VM to run additional outside-in
tests making assertions about the behaviour of the whole. This would
be slow, and I wouldn't be running these integration tests unless
everything looked good, but they would be meaningful.

Here's the kicker: I thought that this was obvious, something that
everyone wanted to do, and it was simply a matter of time before Test
Kitchen evolved to deliver it. Now you've given me pause for thought.

Is this a philosophical question and I didn't realise I was on a side?
Is there no debate and I'm simply wrong?
Did I completely misunderstand your point, and now look a bit silly
for writing this?

Zac

This may be philosophical naval gazing.

The way that I'd test a cluster end-to-end would be to consider the load
balancer part of the cluster and it should issue API calls to its own
VIPs and should also assert that the health checks of all of its servers
are healthy. But the way that I express that is via the load balancer
testing itself, but that API is the externally available interface to
the cluster which is what you are going to test against for
'integration' or 'end-to-end' tests.

So I suspect we get the same test coverage, but I think its important
the way that I state who is responsible for testing what, and I don't
see the need for 'external' testing and more pieces bolted onto the
infrastructure to do 'new' kinds of tests. I think the tools are
already mostly there if you squint at it, and I agree that
chef-provisioning and test-kitchen and serverspec/bats/etc seem to be
the way forwards to deliver the whole thing and it will evolve to
provide it. All we need is to make test-kitchen more cluster-aware and
then the process of simply being able to spin up the cluster, login and
converge chef on everything and then run the tests on each individual
node should be sufficient. Even though there's no overall end-to-end
testing framework being additionally added into test-kitchen, I don't
think you'll miss it since by each node testing itself they also test
their dependencies (it isn't like unit testing because you'll really be
communicating with all your neighbors and not stubs or mocks of your
neighbors -- if you fully test yourself, you should have also fully
tested all of your adjacent dependencies and the whole should really be
the sum of the parts).

On 4/22/15 2:02 PM, Zac Stevens wrote:

On Wed, Apr 22, 2015 at 8:06 PM, Lamont Granquist <lamont@chef.io
mailto:lamont@chef.io> wrote:

On 4/22/15 11:28 AM, Torben Knerr wrote:

    A more concrete example: say I have a cookbook for setting up
    JenkinsCI with Slaves on multiple nodes, with nginx in front,
    etc..
    I'd like to have a test suite that:
    - sets up the multi-node scenario
    - then verifies that the front page is served...
    - ...and that you can create a basic job that is executed on a
    slave
    - ...and so on (some basic acceptance level smoke tests)

    That's what I'm up to... did it get clearer?

My intuition is that the way that you're thinking about that
violates promise theory a bit.  All of your tests should be "from
the inside".  The VM that your webapp is being deployed on should
really only test the world from its perspective.  It should test
that its listening on its http port and that basic API calls can
succeed and that it can make calls to a database.  Trying to think
about it begin responsible for testing its own VIP violates
promise theory since now its responsible for distant
infrastructure and it doesn't have the information that it may
need (it could be failing to register correctly in the VIP but
another server in the cluster could be serving content so the VIP
is up).  Its better to add tests to the loadbalancer "from the
inside" so that the load balancer checks itself and validates that
all its exepected clients are listed as avaiable and health checks
are passing and the loadbalancer should be checking its own VIP.

Then what we need is a multi-node test-kitchen which used
something like chef-provisioning to spin up all of these VMs and
fire off the serverspec runs on all of them, and then report
back.  For deployed infrastructure then audit mode could be
running serverspec tests (and ideally hook busser up to audit mode
so you could audit with bats tests and whatever) and you'd be
getting reports back on every chef-client converge or audit mode run.

I really don't understand the conclusion that there should not be
any tests "from the outside". (I also don't see anything in Torben's
example prescribing where the outside-in tests should be run from)

All the tests described above are valuable, but testing that
individual components work as expected is not a substitute for
verifying that they work as expected when assembled. Or put another
way, unit tests are great but integration tests are also valuable -
I'm of the opinion that this is true whether the units you're dealing
with are classes or servers.

I'd love to see multi-node testing in test-kitchen. I'd love to spin
up a set of VMs that are configured to work together. I'd love to run
serverspec suites on each of them, to assert their behaviour at that
level. I'd love to spin up another VM to run additional outside-in
tests making assertions about the behaviour of the whole. This would
be slow, and I wouldn't be running these integration tests unless
everything looked good, but they would be meaningful.

Here's the kicker: I thought that this was obvious, something that
everyone wanted to do, and it was simply a matter of time before Test
Kitchen evolved to deliver it. Now you've given me pause for thought.

Is this a philosophical question and I didn't realise I was on a side?
Is there no debate and I'm simply wrong?
Did I completely misunderstand your point, and now look a bit silly
for writing this?

Zac

This may be philosophical naval gazing.

The way that I'd test a cluster end-to-end would be to consider the load
balancer part of the cluster and it should issue API calls to its own
VIPs and should also assert that the health checks of all of its servers
are healthy. But the way that I express that is via the load balancer
testing itself, but that API is the externally available interface to
the cluster which is what you are going to test against for
'integration' or 'end-to-end' tests.

So I suspect we get the same test coverage, but I think its important
the way that I state who is responsible for testing what, and I don't
see the need for 'external' testing and more pieces bolted onto the
infrastructure to do 'new' kinds of tests. I think the tools are
already mostly there if you squint at it, and I agree that
chef-provisioning and test-kitchen and serverspec/bats/etc seem to be
the way forwards to deliver the whole thing and it will evolve to
provide it. All we need is to make test-kitchen more cluster-aware and
then the process of simply being able to spin up the cluster, login and
converge chef on everything and then run the tests on each individual
node should be sufficient. Even though there's no overall end-to-end
testing framework being additionally added into test-kitchen, I don't
think you'll miss it since by each node testing itself they also test
their dependencies (it isn't like unit testing because you'll really be
communicating with all your neighbors and not stubs or mocks of your
neighbors -- if you fully test yourself, you should have also fully
tested all of your adjacent dependencies and the whole should really be
the sum of the parts).

Hey,

thanks for all the responses!

Some things are getting clearer now for me.

So first: multi-node testing and testing "from the outside" (I'd call
that acceptance / end-to-end testing) are two different pairs of
shoes. They are orthogonal and we can consider them unrelated to each
other.

I love test-kitchen the way it is, and multi-node testing would be a
super useful addition for exactly the kinds of tests Lamont described
above.

As for testing "from the outside", I'm probably just looking for tool
/ plugin that would integrate to test-kitchen, but that would NOT be
installed inside the VM via busser. Instead it should be testing the
VM from the outside. As far as I understood, this would exclude
serverspec, because serverspec is SSHing into the VM to run its tests,
right?

Btw: has anyone looked into Leibiz actually?

It appears to do some multi-node testing (reusing TK as it looks like)
and it does it's testing from the outside via cucumber. I've seen
cucumber being used a lot for that kind of "from the outside"
acceptance testing, but personally rspec would be just a bit more
appealing to me :wink:

Cheers,
Torben

On Wed, Apr 22, 2015 at 11:25 PM, Lamont Granquist lamont@chef.io wrote:

On 4/22/15 2:02 PM, Zac Stevens wrote:

On Wed, Apr 22, 2015 at 8:06 PM, Lamont Granquist lamont@chef.io wrote:

On 4/22/15 11:28 AM, Torben Knerr wrote:

A more concrete example: say I have a cookbook for setting up
JenkinsCI with Slaves on multiple nodes, with nginx in front, etc..
I'd like to have a test suite that:

  • sets up the multi-node scenario
  • then verifies that the front page is served...
  • ...and that you can create a basic job that is executed on a slave
  • ...and so on (some basic acceptance level smoke tests)

That's what I'm up to... did it get clearer?

My intuition is that the way that you're thinking about that violates
promise theory a bit. All of your tests should be "from the inside". The
VM that your webapp is being deployed on should really only test the world
from its perspective. It should test that its listening on its http port
and that basic API calls can succeed and that it can make calls to a
database. Trying to think about it begin responsible for testing its own
VIP violates promise theory since now its responsible for distant
infrastructure and it doesn't have the information that it may need (it
could be failing to register correctly in the VIP but another server in the
cluster could be serving content so the VIP is up). Its better to add tests
to the loadbalancer "from the inside" so that the load balancer checks
itself and validates that all its exepected clients are listed as avaiable
and health checks are passing and the loadbalancer should be checking its
own VIP.

Then what we need is a multi-node test-kitchen which used something like
chef-provisioning to spin up all of these VMs and fire off the serverspec
runs on all of them, and then report back. For deployed infrastructure then
audit mode could be running serverspec tests (and ideally hook busser up to
audit mode so you could audit with bats tests and whatever) and you'd be
getting reports back on every chef-client converge or audit mode run.

I really don't understand the conclusion that there should not be any
tests "from the outside". (I also don't see anything in Torben's example
prescribing where the outside-in tests should be run from)

All the tests described above are valuable, but testing that individual
components work as expected is not a substitute for verifying that they work
as expected when assembled. Or put another way, unit tests are great but
integration tests are also valuable - I'm of the opinion that this is true
whether the units you're dealing with are classes or servers.

I'd love to see multi-node testing in test-kitchen. I'd love to spin up a
set of VMs that are configured to work together. I'd love to run serverspec
suites on each of them, to assert their behaviour at that level. I'd love
to spin up another VM to run additional outside-in tests making assertions
about the behaviour of the whole. This would be slow, and I wouldn't be
running these integration tests unless everything looked good, but they
would be meaningful.

Here's the kicker: I thought that this was obvious, something that everyone
wanted to do, and it was simply a matter of time before Test Kitchen evolved
to deliver it. Now you've given me pause for thought.

Is this a philosophical question and I didn't realise I was on a side?
Is there no debate and I'm simply wrong?
Did I completely misunderstand your point, and now look a bit silly for
writing this?

Zac

This may be philosophical naval gazing.

The way that I'd test a cluster end-to-end would be to consider the load
balancer part of the cluster and it should issue API calls to its own VIPs
and should also assert that the health checks of all of its servers are
healthy. But the way that I express that is via the load balancer testing
itself, but that API is the externally available interface to the cluster
which is what you are going to test against for 'integration' or
'end-to-end' tests.

So I suspect we get the same test coverage, but I think its important the
way that I state who is responsible for testing what, and I don't see the
need for 'external' testing and more pieces bolted onto the infrastructure
to do 'new' kinds of tests. I think the tools are already mostly there if
you squint at it, and I agree that chef-provisioning and test-kitchen and
serverspec/bats/etc seem to be the way forwards to deliver the whole thing
and it will evolve to provide it. All we need is to make test-kitchen more
cluster-aware and then the process of simply being able to spin up the
cluster, login and converge chef on everything and then run the tests on
each individual node should be sufficient. Even though there's no overall
end-to-end testing framework being additionally added into test-kitchen, I
don't think you'll miss it since by each node testing itself they also test
their dependencies (it isn't like unit testing because you'll really be
communicating with all your neighbors and not stubs or mocks of your
neighbors -- if you fully test yourself, you should have also fully tested
all of your adjacent dependencies and the whole should really be the sum of
the parts).

On Thu, Apr 23, 2015 at 6:37 AM, Torben Knerr mail@tknerr.de wrote:

As for testing "from the outside", I'm probably just looking for tool
/ plugin that would integrate to test-kitchen, but that would NOT be
installed inside the VM via busser. Instead it should be testing the
VM from the outside. As far as I understood, this would exclude
serverspec, because serverspec is SSHing into the VM to run its tests,
right?

No, wrong - serverspec can SSH into a remote host, and it can run tests
locally (with set :backend :exec).
As you observe, busser installs serverspec on the VM, copies the test suite
to the VM, and runs the suite locally.

In the same space, you might also want to look at http://infrataster.net

Zac

That’s true, but unless I’ve missed something, that won’t do any outside-in testing either.

Serverspec always tests local resources, either by SSHing in first, or by assuming it’s already running on the node it will test. In either case it’s testing from within the node under test.

Thanks for the link to Infrataster, I hadn’t heard about it. :slight_smile:


Cassiano Leal
On 23 April 2015 at 09:20:59, Zac Stevens (zts@cryptocracy.com) wrote:

On Thu, Apr 23, 2015 at 6:37 AM, Torben Knerr mail@tknerr.de wrote:
As for testing “from the outside”, I’m probably just looking for tool
/ plugin that would integrate to test-kitchen, but that would NOT be
installed inside the VM via busser. Instead it should be testing the
VM from the outside. As far as I understood, this would exclude
serverspec, because serverspec is SSHing into the VM to run its tests,
right?

No, wrong - serverspec can SSH into a remote host, and it can run tests locally (with set :backend :exec).
As you observe, busser installs serverspec on the VM, copies the test suite to the VM, and runs the suite locally.

In the same space, you might also want to look at http://infrataster.net

Zac

Oh, infratester is nice indeed!

kitchen-infratester anyone? :wink:
Am 23.04.2015 10:27 schrieb "Cassiano Leal" cassianoleal@gmail.com:

That’s true, but unless I’ve missed something, that won’t do any
outside-in testing either.

Serverspec always tests local resources, either by SSHing in first, or by
assuming it’s already running on the node it will test. In either case it’s
testing from within the node under test.

Thanks for the link to Infrataster, I hadn’t heard about it. :slight_smile:

--
Cassiano Leal

On 23 April 2015 at 09:20:59, Zac Stevens (zts@cryptocracy.com) wrote:

On Thu, Apr 23, 2015 at 6:37 AM, Torben Knerr mail@tknerr.de wrote:

As for testing "from the outside", I'm probably just looking for tool
/ plugin that would integrate to test-kitchen, but that would NOT be
installed inside the VM via busser. Instead it should be testing the
VM from the outside. As far as I understood, this would exclude
serverspec, because serverspec is SSHing into the VM to run its tests,
right?

No, wrong - serverspec can SSH into a remote host, and it can run tests
locally (with set :backend :exec).
As you observe, busser installs serverspec on the VM, copies the test
suite to the VM, and runs the suite locally.

In the same space, you might also want to look at http://infrataster.net

Zac

On Thursday, April 23, 2015, Cassiano Leal cassianoleal@gmail.com wrote:

That’s true, but unless I’ve missed something, that won’t do any
outside-in testing either.

Serverspec always tests local resources, either by SSHing in first, or by
assuming it’s already running on the node it will test. In either case it’s
testing from within the node under test.

Thanks for the link to Infrataster, I hadn’t heard about it. :slight_smile:

My bad - I could have sworn it had an http resource, but apparently I've
just been abusing the command resource to run functional tests :slight_smile:

Zac