Ohai Chefs!
Today we released test-kitchen, a framework for running integration tests in an isolated environment. It uses Vagrant and Chef under the hood to set up a single system for converging your cookbook, and optionally running rspec/minitest/cucumber tests.
It is released as a RubyGem, and issues are tracked in our Open Source ticket system.
–
Opscode, Inc
Joshua Timberman, Technical Program Manager
IRC, Skype, Twitter, Github: jtimberman
Interesting... Will try to convert my old vagrant/chef solutions to test my
code.
2012/8/17 Joshua Timberman joshua@opscode.com
Ohai Chefs!
Today we released test-kitchen, a framework for running integration tests
in an isolated environment. It uses Vagrant and Chef under the hood to set
up a single system for converging your cookbook, and optionally running
rspec/minitest/cucumber tests.It is released as a RubyGem, and issues are tracked in our Open Source
ticket system.--
Opscode, Inc
Joshua Timberman, Technical Program Manager
IRC, Skype, Twitter, Github: jtimberman
BTW, is it helps to test selfmade cookbooks too?
2012/8/17 Joshua Timberman joshua@opscode.com
Ohai Chefs!
Today we released test-kitchen, a framework for running integration tests
in an isolated environment. It uses Vagrant and Chef under the hood to set
up a single system for converging your cookbook, and optionally running
rspec/minitest/cucumber tests.It is released as a RubyGem, and issues are tracked in our Open Source
ticket system.--
Opscode, Inc
Joshua Timberman, Technical Program Manager
IRC, Skype, Twitter, Github: jtimberman
Ohai,
On Aug 17, 2012, at 4:52 PM, Akzhan Abdulin akzhan.abdulin@gmail.com wrote:
BTW, is it helps to test selfmade cookbooks too?
Totally! We consider our own cookbooks to be self-made, after all :).
Interesting... Will try to convert my old vagrant/chef solutions to test my code.
If you have existing baseboxes that you want to use instead of the ones provided by bento, or used by default in test-kitchen, put this in your test/kitchen/Kitchenfile:
platform :platform_to_test do
version "its version" do
box "name-of-existing-box"
box_url "where it can be downloaded but not used if it exists"
end
end
For example, I had an existing vagrant box:
% vagrant box list
precise64
So I have just this in my Kitchenfile:
platform :ubuntu do
version "12.04" do
box "precise64"
box_url "https://s3.amazonaws.com/mybucket/precise64.box
end
end
(the url is made up, I built it locally)
--
Opscode, Inc
Joshua Timberman, Technical Program Manager
IRC, Skype, Twitter, Github: jtimberman
Absolutely Nice!
So far I have just read the README.md and it looks very very nice. I
have not actually tried it out yet, but from reading the README some
questions came up:
Having noticed the librarian Cheffile in the
apache2/test/kitchen/cookbooks directory, I wonder whether the
cookbook dependencies therein are automatically resolved by
test-kitchen, are they? Would a Berksfile be supported as well?
If these dependencies are indeed automatically resolved then that
would be worth mentioning in the README.md
Do you have an example with test-kitchen tests running on Travis
CI? Btw: can you actually spin up a vagrant VM on Travis?
Concerning fast running tests:
It would be great to have fast running unit-like tests (which don't
require to spin up a vagrant VM first) covered by test-kitchen as
well. Chefspec comes to my mind, but I haven't found it referenced in
the README.
Oh wait, isn't that already supported by the cookbook / script
parameter which defaults to 'rspec spec'? Or would I do that within
the cookbook / preflight command to make sure this runs before the
vagrant VM is spun up?
The README "Kitchenfile DSL" section talks about a features dir
underneath the /test/kitchen dir, but in the examples I see
only /test/features?
Also, is the "integration_test" block in the Kitchenfile really
necessary to run the features? From the "Cucumber" section above,
which says "Test Kitchen will run any features in the test/features
subdirectory tagged with the configuration name in order to check that
the service is working as expected.", it sounds like you wouldn't need
to do anything extra to make the cucumber features run. In general,
the "integration_test" section really confused me...
I don't get what the cookbook and integration_test runtimes parameter
is good for. Why does RVM need to be installed on the base box? Are
you running the features locally on the target box? I would have
expected that the features are run from the outside (e.g. on my
workstation) against the provisioned target vm (the "black box"), so
why would I care that RVM is installed inside that black box?!?
I guess most of my questions are based on misunderstandings. Please
forgive the fact that I haven't actually run test-kitchen yet... 
Anyways, thanks for all the Chef awesomeness!
Cheers,
Torben
On Fri, Aug 17, 2012 at 6:00 PM, Joshua Timberman joshua@opscode.com wrote:
Ohai Chefs!
Today we released test-kitchen, a framework for running integration tests in an isolated environment. It uses Vagrant and Chef under the hood to set up a single system for converging your cookbook, and optionally running rspec/minitest/cucumber tests.
It is released as a RubyGem, and issues are tracked in our Open Source ticket system.
--
Opscode, Inc
Joshua Timberman, Technical Program Manager
IRC, Skype, Twitter, Github: jtimberman
On Aug 18, 2012, at 8:45 AM, Torben Knerr ukio@gmx.de wrote:
Absolutely Nice!
Thanks! We're all pretty excited about this. Andrew did a great job with it.
- Concerning cookbooks with dependencies to other cookbooks:
Having noticed the librarian Cheffile in the
apache2/test/kitchen/cookbooks directory, I wonder whether the
cookbook dependencies therein are automatically resolved by
test-kitchen, are they? Would a Berksfile be supported as well?
Yes - test-kitchen wraps up librarian-chef in the base runner:
https://github.com/opscode/test-kitchen/blob/master/lib/test-kitchen/runner/base.rb#L95-106
The reason Berkshelf isn't supported is simply because it didn't exist (publicly available) when test-kitchen was created and developed.
If these dependencies are indeed automatically resolved then that
would be worth mentioning in the README.md
Would you please open a ticket in the KITCHEN project for that?
- Do you have an example with test-kitchen tests running on Travis
CI? Btw: can you actually spin up a vagrant VM on Travis?
I'm not sure if Travis would support that. I haven't done much with Travis besides wire it up with foodcritic and knife cookbook test. And by that I mean I've merged pull requests that do that :).
We have Jenkins internally for other projects, like building our omnibus packages, and the services that run Hosted/Private Chef. We also have the cookbooks tested under test-kitchen, though it is very simplistic right now.
I'd like to see Travis.CI support in each of our cookbooks, whether they're just doing the foodcritic/cookbook test, or running test-kitchen, that'd be great.
- Concerning fast running tests:
It would be great to have fast running unit-like tests (which don't
require to spin up a vagrant VM first) covered by test-kitchen as
well. Chefspec comes to my mind, but I haven't found it referenced in
the README.Oh wait, isn't that already supported by the cookbook / script
parameter which defaults to 'rspec spec'? Or would I do that within
the cookbook / preflight command to make sure this runs before the
vagrant VM is spun up?
Yes, that should be done by the "script" parameter. You should be able to just run that in your cookbook directory without test-kitchen if you don't want the full provisioning cycle.
- Concerning integration_test:
The README "Kitchenfile DSL" section talks about a features dir
underneath the /test/kitchen dir, but in the examples I see
only /test/features?Also, is the "integration_test" block in the Kitchenfile really
necessary to run the features? From the "Cucumber" section above,
which says "Test Kitchen will run any features in the test/features
subdirectory tagged with the configuration name in order to check that
the service is working as expected.", it sounds like you wouldn't need
to do anything extra to make the cucumber features run. In general,
the "integration_test" section really confused me...
The cookbook features will automatically get run by test-kitchen if the directory exists.
The integration_test block is for non-cookbook projects (like mixlib-shellout) that have unit or integration testing. While mixlib-shellout is kind of a contrived example, a better one would be a Rails application that has its own test suite that should be run in a fully deployed scenario with its database, etc.
- Runtimes?
I don't get what the cookbook and integration_test runtimes parameter
is good for. Why does RVM need to be installed on the base box? Are
you running the features locally on the target box? I would have
expected that the features are run from the outside (e.g. on my
workstation) against the provisioned target vm (the "black box"), so
why would I care that RVM is installed inside that black box?!?
Rvm was used as a relatively simple way to get specific versions of Ruby installed for performing the tests. Think of a non-cookbook project, like a Ruby library that you want to test on different versions of Ruby (like 1.8.7 vs 1.9.3, or even JRuby). My personal preference for cookbook testing is that we make the RVM part optional, and use the Ruby that is included in Chef on the base boxes already.
I created a ticket to track this:
http://tickets.opscode.com/browse/KITCHEN-4
--
Opscode, Inc
Joshua Timberman, Technical Program Manager
IRC, Skype, Twitter, Github: jtimberman
Hi Joshua,
thanks for the quick reply!
If these dependencies are indeed automatically resolved then that
would be worth mentioning in the README.mdWould you please open a ticket in the KITCHEN project for that?
Cool, having automatic dependency resolution is great!
I have created
http://tickets.opscode.com/browse/KITCHEN-8 for the missing README documentation
and
http://tickets.opscode.com/browse/KITCHEN-9 for Berksfile support
I'd like to see Travis.CI support in each of our cookbooks, whether they're just doing the foodcritic/cookbook test, or running test-kitchen, that'd be great.
Fully support that
Concerning the provisioning of a Vagrant VM on Travis, I will give it
a try next week. If someone has already done that or a working example
please let me know!
Oh wait, isn't that already supported by the cookbook / script
parameter which defaults to 'rspec spec'? Or would I do that within
the cookbook / preflight command to make sure this runs before the
vagrant VM is spun up?Yes, that should be done by the "script" parameter. You should be able to just run that in your cookbook directory without test-kitchen if you don't want the full provisioning cycle.
Ok, great!
Also, is the "integration_test" block in the Kitchenfile really
necessary to run the features?The cookbook features will automatically get run by test-kitchen if the directory exists.
Perfect, good to know,
The integration_test block is for non-cookbook projects (like mixlib-shellout) that have unit or integration testing. While mixlib-shellout is kind of a contrived example, a better one would be a Rails application that has its own test suite that should be run in a fully deployed scenario with its database, etc.
Now that I read it again it's clear
Rvm was used as a relatively simple way to get specific versions of Ruby installed for performing the tests. Think of a non-cookbook project, like a Ruby library that you want to test on different versions of Ruby (like 1.8.7 vs 1.9.3, or even JRuby). My personal preference for cookbook testing is that we make the RVM part optional, and use the Ruby that is included in Chef on the base boxes already.
I created a ticket to track this:
Yes, for non-cookbook projects using "integration_test" that makes sense.
For cookbook projects using "cookbook" the rvm option still does not
(generally) make sense to me. Okay, if I know I have a basebox with
rvm and some Rubies installed, then it would be fair to state that you
want to test against several different rubies. Yes, but this makes
some heavy assumptions about rvm and rubies being installed on the
basebox, so I would rather see "runtimes" as an option within
"platform", but not within "cookbooks", e.g.
"""
platform :centos do
version "5.8" do
box "opscode-centos-5.8"
box_url "https://opscode-vm.s3.amazonaws.com/vagrant/boxes/opscode-centos-5.8.box"
runtimes ['1.9.2', '1.9.3']
end
end
"""
The default of would mean that the system ruby that is installed
along with Chef of basebox is used. If "runtimes" is specified it
would explicitly mean rvm rubies.
For sure, you could then also exclude specific runtimes:
"""
cookbook "apache2" do
configuration "default"
exclude :platform => 'centos', :runtimes => ['1.9.2']
run_list_extras ['apache2_test::setup']
end
"""
Does that make sense?
Finally, this part of the current "cookbook" / "runtimes"
documentation confuses me most: "The default for cookbook projects is
, which effectively disables spec/feature tests"
Why should spec/feature tests be disabled in this case?
Cheers,
Torben
--
Opscode, Inc
Joshua Timberman, Technical Program Manager
IRC, Skype, Twitter, Github: jtimberman
Ohai,
Thanks for opening the tickets, it's important to get those things tracked so folks can submit feedback, or contribute.
On Aug 23, 2012, at 12:46 AM, Torben Knerr ukio@gmx.de wrote:
Yes, for non-cookbook projects using "integration_test" that makes sense.
For cookbook projects using "cookbook" the rvm option still does not
(generally) make sense to me. Okay, if I know I have a basebox with
rvm and some Rubies installed, then it would be fair to state that you
want to test against several different rubies. Yes, but this makes
some heavy assumptions about rvm and rubies being installed on the
basebox, so I would rather see "runtimes" as an option within
"platform", but not within "cookbooks", e.g."""
platform :centos do
version "5.8" do
box "opscode-centos-5.8"
box_url "https://opscode-vm.s3.amazonaws.com/vagrant/boxes/opscode-centos-5.8.box"
runtimes ['1.9.2', '1.9.3']
end
end
"""The default of would mean that the system ruby that is installed
along with Chef of basebox is used. If "runtimes" is specified it
would explicitly mean rvm rubies.For sure, you could then also exclude specific runtimes:
"""
cookbook "apache2" do
configuration "default"
exclude :platform => 'centos', :runtimes => ['1.9.2']
run_list_extras ['apache2_test::setup']
end
"""Does that make sense?
Finally, this part of the current "cookbook" / "runtimes"
documentation confuses me most: "The default for cookbook projects is
, which effectively disables spec/feature tests"Why should spec/feature tests be disabled in this case?
You also posted this to the appropriate tickets, yes? I just want to make sure I interpreted the comments correctly.
Thanks!
Hi Joshua,
yep, it's all added as comments to the tickets now, except...
Am 24.08.2012 04:50 schrieb "Joshua Timberman" joshua@opscode.com:
Ohai,
Thanks for opening the tickets, it's important to get those things tracked
so folks can submit feedback, or contribute.On Aug 23, 2012, at 12:46 AM, Torben Knerr ukio@gmx.de wrote:
Yes, for non-cookbook projects using "integration_test" that makes sense.
For cookbook projects using "cookbook" the rvm option still does not
(generally) make sense to me. Okay, if I know I have a basebox with
rvm and some Rubies installed, then it would be fair to state that you
want to test against several different rubies. Yes, but this makes
some heavy assumptions about rvm and rubies being installed on the
basebox, so I would rather see "runtimes" as an option within
"platform", but not within "cookbooks", e.g."""
platform :centos do
version "5.8" do
box "opscode-centos-5.8"
box_url "
https://opscode-vm.s3.amazonaws.com/vagrant/boxes/opscode-centos-5.8.box"
runtimes ['1.9.2', '1.9.3']
end
end
"""The default of would mean that the system ruby that is installed
along with Chef of basebox is used. If "runtimes" is specified it
would explicitly mean rvm rubies.For sure, you could then also exclude specific runtimes:
"""
cookbook "apache2" do
configuration "default"
exclude :platform => 'centos', :runtimes => ['1.9.2']
run_list_extras ['apache2_test::setup']
end
"""Does that make sense?
Finally, this part of the current "cookbook" / "runtimes"
documentation confuses me most: "The default for cookbook projects is
, which effectively disables spec/feature tests"Why should spec/feature tests be disabled in this case?
You also posted this to the appropriate tickets, yes? I just want to make
sure I interpreted the comments correctly.
...that I've changed my mind a bit here: assuming that runtimes makes
sense only within integration_test but not within cookbook (as argued
in the commentshttp://tickets.opscode.com/browse/KITCHEN-4?focusedCommentId=26525&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-26525),
I would not add runtimes to platform and keep it really specific to the
non-cookbook integration tests. Thus I did not add the code snippets above
to the ticket.
Cheers,
Torben
Thanks!
What is the easiest approach for a newbie to create tests using test kitchen and inspec to be deployed into a jenkins pipeline using the bitbucket hook?
Should all cookbooks reside within the test folder or in the recipe folder and referenced in the test/default ruby file via include..?