Ohai Chefs!
I’m currently struggling with writing cucumber features for my
recipes, and I’m not sure whether Cuken[1] or cucumber-nagios[2] would
be the “better” library.
Can you recommend either of them, or do you have examples of cookbooks
being tested with them?
Cheers,
Torben
[1] https://github.com/hedgehog/cuken
[2] https://github.com/auxesis/cucumber-nagios
On Wed, Aug 29, 2012 at 12:41 AM, Torben Knerr ukio@gmx.de wrote:
Ohai Chefs!
I'm currently struggling with writing cucumber features for my
recipes, and I'm not sure whether Cuken[1] or cucumber-nagios[2] would
be the "better" library.
Cuken grew out of cucumber-nagios, and was inspired by Aruba.
Cucken aims to cover more than nagios use cases.
While I have contributed to both, I wouldn't be willing to opine on
which is better.
We currently use cuken.
Can you recommend either of them, or do you have examples of cookbooks
being tested with them?
Cucken is not aimed at the test/spec or TDD/BDD of individual cookbooks.
To my mind testing/describing specific cookbooks starts to resemble
testing chef[-server/-client/-solo].
Rather, Cuken is intended to help with describing what you have/want
after Chef has done its thing.
This can mean features being run on the machine (local), but
generally, this means features being run against a machine (remote).
This approach also means you will generally spec/test the net result
of more than one cookbook/recipe.
To my mind this makes sense because there is no (practical/real)
concept in chef of having the result of exactly one cookbook - all
chef runs tend to be the net result of having run several cookbooks.
Of course you are not limited to using cuken in this way.
In fact the zenoss example (see the relishapp representation)
illustrates a description of a local development environment, as well
as some VMs.
The proof-of-concept was to see if we could get to the point of
editing one source (the feature files) to describe the dev (or prod)
environment as well as the actual servers.
This seems to work for us. We now:
All done automatically, with the code described under our features folder.
Note that we use Librarian-Chef and our cookbook recipes are not
currently 'described' using cuken, although we could do this just as
easily as we did with Vangrant's files.
The reason for this is that recipe changes can be rapid and, as
mentioned, the results are rarely atomic.
We may yet expand cuken's scope to our production recipes - which are
more stable.
However, I'm currently of the view that cuken's sweetspot is (yes, I
should update the project description):
This means we don't spec cookbooks per se.
Rather:
a) the environment in which cookbooks run and
b) the results of running those cookboooks.
Hope that helps clarify?
Best wishes
Cheers,
Torben[1] GitHub - hedgehog/cuken: Reusable Cucumber steps and API for post-convergence system integration descriptions
[2] GitHub - auxesis/cucumber-nagios: Systems testing plugin for Nagios with Cucumber + Webrat + Mechanize + Net::SSH
--
πόλλ' οἶδ ἀλώπηξ, ἀλλ' ἐχῖνος ἓν μέγα
[The fox knows many things, but the hedgehog knows one big thing.]
Archilochus, Greek poet (c. 680 BC – c. 645 BC)
http://hedgehogshiatus.com
Hi,
thanks, that clarifies a lot!
I don't quite get what you mean with environment though (maybe I'm
confused with Chef environments :-)) but I guess you mean the Chef
Server and which Recipes it knows, right?
For me there is value in both, testing individual cookbooks as well as
their composition (roles, node's run_list). For individual cookbooks,
I'd test on all levels like foodcritic, chefspec,
minitest-chef-handler and feature tests. For any composition of
cookbooks I'd be more interested in the net outcome, i.e. cucumber
features tests against a machine (remote). Don't know if spec tests
would make sense at that level.
Btw: I was wondering why cucumber features should be run on a
machine (locally), but it seems to be quite often used in that way
(e.g. test-kitchen does that as well). I always thought of
cucumber/gherkin as a tool that is especially well suited for
acceptance-level testing, which IMHO should always be outside-in.
Would be interested to hear the community's thoughts about that...
Anyways, Cuken vs. cucumber-nagios is now much clearer to me. It's
interesting to see that current master of cucumber-nagios is now
re-using the Cuken steps, so with the next gem release you will
probably get Cuken as well when you use cucumber-nagios 
Thanks a lot,
Torben
On Thu, Aug 30, 2012 at 1:56 AM, Hedge Hog hedgehogshiatus@gmail.com wrote:
On Wed, Aug 29, 2012 at 12:41 AM, Torben Knerr ukio@gmx.de wrote:
Ohai Chefs!
I'm currently struggling with writing cucumber features for my
recipes, and I'm not sure whether Cuken[1] or cucumber-nagios[2] would
be the "better" library.Cuken grew out of cucumber-nagios, and was inspired by Aruba.
Cucken aims to cover more than nagios use cases.
While I have contributed to both, I wouldn't be willing to opine on
which is better.
We currently use cuken.Can you recommend either of them, or do you have examples of cookbooks
being tested with them?Cucken is not aimed at the test/spec or TDD/BDD of individual cookbooks.
To my mind testing/describing specific cookbooks starts to resemble
testing chef[-server/-client/-solo].Rather, Cuken is intended to help with describing what you have/want
after Chef has done its thing.
This can mean features being run on the machine (local), but
generally, this means features being run against a machine (remote).
This approach also means you will generally spec/test the net result
of more than one cookbook/recipe.
To my mind this makes sense because there is no (practical/real)
concept in chef of having the result of exactly one cookbook - all
chef runs tend to be the net result of having run several cookbooks.Of course you are not limited to using cuken in this way.
In fact the zenoss example (see the relishapp representation)
illustrates a description of a local development environment, as well
as some VMs.The proof-of-concept was to see if we could get to the point of
editing one source (the feature files) to describe the dev (or prod)
environment as well as the actual servers.This seems to work for us. We now:
- build a couple of AMIs from base images (client/server kernel specifics)
- rebundle these AMIs as our base images
- build custom OS specific images from the rebaundled sys-base images
(package/app specifics)- rebundle these AMIs as our server/client images
- test the Chef configuration results.
All done automatically, with the code described under our features folder.
Note that we use Librarian-Chef and our cookbook recipes are not
currently 'described' using cuken, although we could do this just as
easily as we did with Vangrant's files.
The reason for this is that recipe changes can be rapid and, as
mentioned, the results are rarely atomic.We may yet expand cuken's scope to our production recipes - which are
more stable.
However, I'm currently of the view that cuken's sweetspot is (yes, I
should update the project description):
- pre system-convergence (describing the environment of a chef run: dev/prod)
- post system-convergence (describing the results of a chef run: web
page availability, security tests)This means we don't spec cookbooks per se.
Rather:
a) the environment in which cookbooks run and
b) the results of running those cookboooks.Hope that helps clarify?
Best wishesCheers,
Torben[1] GitHub - hedgehog/cuken: Reusable Cucumber steps and API for post-convergence system integration descriptions
[2] GitHub - auxesis/cucumber-nagios: Systems testing plugin for Nagios with Cucumber + Webrat + Mechanize + Net::SSH--
πόλλ' οἶδ ἀλώπηξ, ἀλλ' ἐχῖνος ἓν μέγα
[The fox knows many things, but the hedgehog knows one big thing.]
Archilochus, Greek poet (c. 680 BC – c. 645 BC)
http://hedgehogshiatus.com
On Fri, Aug 31, 2012 at 4:49 PM, Torben Knerr ukio@gmx.de wrote:
Hi,
thanks, that clarifies a lot!
I don't quite get what you mean with environment though (maybe I'm
confused with Chef environments :-)) but I guess you mean the Chef
Server and which Recipes it knows, right?
Environment is an overloaded term, 'context' is a better synonym.
We try to avoid single point of failure, so we don't use Chef Server
and rely on chef-solo.
I emphasize that using chef-solo is not possible in all use cases, and
can require you to structure your app in a particular way.
For me there is value in both, testing individual cookbooks as well as
their composition (roles, node's run_list). For individual cookbooks,
I'd test on all levels like foodcritic, chefspec,
minitest-chef-handler and feature tests. For any composition of
cookbooks I'd be more interested in the net outcome, i.e. cucumber
features tests against a machine (remote). Don't know if spec tests
would make sense at that level.Btw: I was wondering why cucumber features should be run on a
machine (locally), but it seems to be quite often used in that way
Cuken is not limited to chef/cloud/networked use cases.
It can be the case that you want to verify a single machine has been
setup correctly without network (remote) access.
So far I've run cucumber features locally, after a chef run, when I am
paranoid, e.g. have we really pinned down all versions sufficiently,
or could a system package update still break something.
HTH
(e.g. test-kitchen does that as well). I always thought of
cucumber/gherkin as a tool that is especially well suited for
acceptance-level testing, which IMHO should always be outside-in.
Would be interested to hear the community's thoughts about that...Anyways, Cuken vs. cucumber-nagios is now much clearer to me. It's
interesting to see that current master of cucumber-nagios is now
re-using the Cuken steps, so with the next gem release you will
probably get Cuken as well when you use cucumber-nagiosThanks a lot,
TorbenOn Thu, Aug 30, 2012 at 1:56 AM, Hedge Hog hedgehogshiatus@gmail.com wrote:
On Wed, Aug 29, 2012 at 12:41 AM, Torben Knerr ukio@gmx.de wrote:
Ohai Chefs!
I'm currently struggling with writing cucumber features for my
recipes, and I'm not sure whether Cuken[1] or cucumber-nagios[2] would
be the "better" library.Cuken grew out of cucumber-nagios, and was inspired by Aruba.
Cucken aims to cover more than nagios use cases.
While I have contributed to both, I wouldn't be willing to opine on
which is better.
We currently use cuken.Can you recommend either of them, or do you have examples of cookbooks
being tested with them?Cucken is not aimed at the test/spec or TDD/BDD of individual cookbooks.
To my mind testing/describing specific cookbooks starts to resemble
testing chef[-server/-client/-solo].Rather, Cuken is intended to help with describing what you have/want
after Chef has done its thing.
This can mean features being run on the machine (local), but
generally, this means features being run against a machine (remote).
This approach also means you will generally spec/test the net result
of more than one cookbook/recipe.
To my mind this makes sense because there is no (practical/real)
concept in chef of having the result of exactly one cookbook - all
chef runs tend to be the net result of having run several cookbooks.Of course you are not limited to using cuken in this way.
In fact the zenoss example (see the relishapp representation)
illustrates a description of a local development environment, as well
as some VMs.The proof-of-concept was to see if we could get to the point of
editing one source (the feature files) to describe the dev (or prod)
environment as well as the actual servers.This seems to work for us. We now:
- build a couple of AMIs from base images (client/server kernel specifics)
- rebundle these AMIs as our base images
- build custom OS specific images from the rebaundled sys-base images
(package/app specifics)- rebundle these AMIs as our server/client images
- test the Chef configuration results.
All done automatically, with the code described under our features folder.
Note that we use Librarian-Chef and our cookbook recipes are not
currently 'described' using cuken, although we could do this just as
easily as we did with Vangrant's files.
The reason for this is that recipe changes can be rapid and, as
mentioned, the results are rarely atomic.We may yet expand cuken's scope to our production recipes - which are
more stable.
However, I'm currently of the view that cuken's sweetspot is (yes, I
should update the project description):
- pre system-convergence (describing the environment of a chef run: dev/prod)
- post system-convergence (describing the results of a chef run: web
page availability, security tests)This means we don't spec cookbooks per se.
Rather:
a) the environment in which cookbooks run and
b) the results of running those cookboooks.Hope that helps clarify?
Best wishesCheers,
Torben[1] GitHub - hedgehog/cuken: Reusable Cucumber steps and API for post-convergence system integration descriptions
[2] GitHub - auxesis/cucumber-nagios: Systems testing plugin for Nagios with Cucumber + Webrat + Mechanize + Net::SSH--
πόλλ' οἶδ ἀλώπηξ, ἀλλ' ἐχῖνος ἓν μέγα
[The fox knows many things, but the hedgehog knows one big thing.]
Archilochus, Greek poet (c. 680 BC – c. 645 BC)
http://hedgehogshiatus.com
--
πόλλ' οἶδ ἀλώπηξ, ἀλλ' ἐχῖνος ἓν μέγα
[The fox knows many things, but the hedgehog knows one big thing.]
Archilochus, Greek poet (c. 680 BC – c. 645 BC)
http://hedgehogshiatus.com
Am 03.09.2012 03:29 schrieb "Hedge Hog" hedgehogshiatus@gmail.com:
On Fri, Aug 31, 2012 at 4:49 PM, Torben Knerr ukio@gmx.de wrote:
Hi,
thanks, that clarifies a lot!
I don't quite get what you mean with environment though (maybe I'm
confused with Chef environments :-)) but I guess you mean the Chef
Server and which Recipes it knows, right?Environment is an overloaded term, 'context' is a better synonym.
We try to avoid single point of failure, so we don't use Chef Server
and rely on chef-solo.
I emphasize that using chef-solo is not possible in all use cases, and
can require you to structure your app in a particular way.
Ok, understood, thanks.
For me there is value in both, testing individual cookbooks as well as
their composition (roles, node's run_list). For individual cookbooks,
I'd test on all levels like foodcritic, chefspec,
minitest-chef-handler and feature tests. For any composition of
cookbooks I'd be more interested in the net outcome, i.e. cucumber
features tests against a machine (remote). Don't know if spec tests
would make sense at that level.Btw: I was wondering why cucumber features should be run on a
machine (locally), but it seems to be quite often used in that wayCuken is not limited to chef/cloud/networked use cases.
It can be the case that you want to verify a single machine has been
setup correctly without network (remote) access.
True, didn't consider this case so far - then it makes sense running the
features locally...
So far I've run cucumber features locally, after a chef run, when I am
paranoid, e.g. have we really pinned down all versions sufficiently,
or could a system package update still break something.HTH
Yes, it helped, thanks
(e.g. test-kitchen does that as well). I always thought of
cucumber/gherkin as a tool that is especially well suited for
acceptance-level testing, which IMHO should always be outside-in.
Would be interested to hear the community's thoughts about that...Anyways, Cuken vs. cucumber-nagios is now much clearer to me. It's
interesting to see that current master of cucumber-nagios is now
re-using the Cuken steps, so with the next gem release you will
probably get Cuken as well when you use cucumber-nagiosThanks a lot,
TorbenOn Thu, Aug 30, 2012 at 1:56 AM, Hedge Hog hedgehogshiatus@gmail.com
wrote:On Wed, Aug 29, 2012 at 12:41 AM, Torben Knerr ukio@gmx.de wrote:
Ohai Chefs!
I'm currently struggling with writing cucumber features for my
recipes, and I'm not sure whether Cuken[1] or cucumber-nagios[2] would
be the "better" library.Cuken grew out of cucumber-nagios, and was inspired by Aruba.
Cucken aims to cover more than nagios use cases.
While I have contributed to both, I wouldn't be willing to opine on
which is better.
We currently use cuken.Can you recommend either of them, or do you have examples of cookbooks
being tested with them?Cucken is not aimed at the test/spec or TDD/BDD of individual
cookbooks.
To my mind testing/describing specific cookbooks starts to resemble
testing chef[-server/-client/-solo].Rather, Cuken is intended to help with describing what you have/want
after Chef has done its thing.
This can mean features being run on the machine (local), but
generally, this means features being run against a machine (remote).
This approach also means you will generally spec/test the net result
of more than one cookbook/recipe.
To my mind this makes sense because there is no (practical/real)
concept in chef of having the result of exactly one cookbook - all
chef runs tend to be the net result of having run several cookbooks.Of course you are not limited to using cuken in this way.
In fact the zenoss example (see the relishapp representation)
illustrates a description of a local development environment, as well
as some VMs.The proof-of-concept was to see if we could get to the point of
editing one source (the feature files) to describe the dev (or prod)
environment as well as the actual servers.This seems to work for us. We now:
- build a couple of AMIs from base images (client/server kernel
specifics)- rebundle these AMIs as our base images
- build custom OS specific images from the rebaundled sys-base images
(package/app specifics)- rebundle these AMIs as our server/client images
- test the Chef configuration results.
All done automatically, with the code described under our features
folder.Note that we use Librarian-Chef and our cookbook recipes are not
currently 'described' using cuken, although we could do this just as
easily as we did with Vangrant's files.
The reason for this is that recipe changes can be rapid and, as
mentioned, the results are rarely atomic.We may yet expand cuken's scope to our production recipes - which are
more stable.
However, I'm currently of the view that cuken's sweetspot is (yes, I
should update the project description):
- pre system-convergence (describing the environment of a chef run:
dev/prod)- post system-convergence (describing the results of a chef run: web
page availability, security tests)This means we don't spec cookbooks per se.
Rather:
a) the environment in which cookbooks run and
b) the results of running those cookboooks.Hope that helps clarify?
Best wishesCheers,
Torben[1] GitHub - hedgehog/cuken: Reusable Cucumber steps and API for post-convergence system integration descriptions
[2] GitHub - auxesis/cucumber-nagios: Systems testing plugin for Nagios with Cucumber + Webrat + Mechanize + Net::SSH--
πόλλ' οἶδ ἀλώπηξ, ἀλλ' ἐχῖνος ἓν μέγα
[The fox knows many things, but the hedgehog knows one big thing.]
Archilochus, Greek poet (c. 680 BC – c. 645 BC)
http://hedgehogshiatus.com--
πόλλ' οἶδ ἀλώπηξ, ἀλλ' ἐχῖνος ἓν μέγα
[The fox knows many things, but the hedgehog knows one big thing.]
Archilochus, Greek poet (c. 680 BC – c. 645 BC)
http://hedgehogshiatus.com