# Meet test-kitchen

**URL:** <https://discourse.chef.io/t/meet-test-kitchen/3025>\
**Category:** Chef Infra (archive)\
**Created:** [August 17, 2012, 4:00pm UTC](https://discourse.chef.io/t/meet-test-kitchen/3025 "2012-08-17T16:00:48Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Joshua\_Timberman](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/joshua_timberman/32/424_2.png) [@Joshua\_Timberman](https://discourse.chef.io/u/Joshua_Timberman)\
**Post date:** [August 17, 2012, 4:00pm UTC](https://discourse.chef.io/t/meet-test-kitchen/3025/1 "2012-08-17T16:00:48Z")

</div>

Ohai Chefs!

Today we released test-kitchen, a framework for running integration tests in an isolated environment. It uses Vagrant and Chef under the hood to set up a single system for converging your cookbook, and optionally running rspec/minitest/cucumber tests.

It is released as a RubyGem, and issues are tracked in our Open Source ticket system.

- [http://bit.ly/test-kitchen](http://bit.ly/test-kitchen)

–  
Opscode, Inc  
Joshua Timberman, Technical Program Manager  
IRC, Skype, Twitter, Github: jtimberman

---

<div class="post-metadata">

**Author:** ![Akzhan\_Abdulin](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/akzhan_abdulin/32/465_2.png) [@Akzhan\_Abdulin](https://discourse.chef.io/u/Akzhan_Abdulin)\
**Post date:** [August 17, 2012, 10:51pm UTC](https://discourse.chef.io/t/meet-test-kitchen/3025/2 "2012-08-17T22:51:18Z")

</div>

Interesting... Will try to convert my old vagrant/chef solutions to test my  
code.

2012/8/17 Joshua Timberman [joshua@opscode.com](mailto:joshua@opscode.com)

> Ohai Chefs!
> 
> Today we released test-kitchen, a framework for running integration tests  
> in an isolated environment. It uses Vagrant and Chef under the hood to set  
> up a single system for converging your cookbook, and optionally running  
> rspec/minitest/cucumber tests.
> 
> It is released as a RubyGem, and issues are tracked in our Open Source  
> ticket system.
> 
> - [http://bit.ly/test-kitchen](http://bit.ly/test-kitchen)
> 
> --  
> Opscode, Inc  
> Joshua Timberman, Technical Program Manager  
> IRC, Skype, Twitter, Github: jtimberman

---

<div class="post-metadata">

**Author:** ![Akzhan\_Abdulin](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/akzhan_abdulin/32/465_2.png) [@Akzhan\_Abdulin](https://discourse.chef.io/u/Akzhan_Abdulin)\
**Post date:** [August 17, 2012, 10:52pm UTC](https://discourse.chef.io/t/meet-test-kitchen/3025/3 "2012-08-17T22:52:16Z")

</div>

BTW, is it helps to test selfmade cookbooks too?

2012/8/17 Joshua Timberman [joshua@opscode.com](mailto:joshua@opscode.com)

> Ohai Chefs!
> 
> Today we released test-kitchen, a framework for running integration tests  
> in an isolated environment. It uses Vagrant and Chef under the hood to set  
> up a single system for converging your cookbook, and optionally running  
> rspec/minitest/cucumber tests.
> 
> It is released as a RubyGem, and issues are tracked in our Open Source  
> ticket system.
> 
> - [http://bit.ly/test-kitchen](http://bit.ly/test-kitchen)
> 
> --  
> Opscode, Inc  
> Joshua Timberman, Technical Program Manager  
> IRC, Skype, Twitter, Github: jtimberman

---

<div class="post-metadata">

**Author:** ![Joshua\_Timberman](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/joshua_timberman/32/424_2.png) [@Joshua\_Timberman](https://discourse.chef.io/u/Joshua_Timberman)\
**Post date:** [August 17, 2012, 11:03pm UTC](https://discourse.chef.io/t/meet-test-kitchen/3025/4 "2012-08-17T23:03:43Z")

</div>

Ohai,

On Aug 17, 2012, at 4:52 PM, Akzhan Abdulin [akzhan.abdulin@gmail.com](mailto:akzhan.abdulin@gmail.com) wrote:

> BTW, is it helps to test selfmade cookbooks too?

Totally! We consider our own cookbooks to be self-made, after all :).

> Interesting... Will try to convert my old vagrant/chef solutions to test my code.

If you have existing baseboxes that you want to use instead of the ones provided by bento, or used by default in test-kitchen, put this in your test/kitchen/Kitchenfile:

platform :platform\_to\_test do  
version "its version" do  
box "name-of-existing-box"  
box\_url "where it can be downloaded but not used if it exists"  
end  
end

For example, I had an existing vagrant box:

% vagrant box list  
precise64

So I have just this in my Kitchenfile:

platform :ubuntu do  
version "12.04" do  
box "precise64"  
box\_url "[https://s3.amazonaws.com/mybucket/precise64.box](https://s3.amazonaws.com/mybucket/precise64.box)  
end  
end

(the url is made up, I built it locally)

--  
Opscode, Inc  
Joshua Timberman, Technical Program Manager  
IRC, Skype, Twitter, Github: jtimberman

---

<div class="post-metadata">

**Author:** ![Torben\_Knerr](https://avatars.discourse-cdn.com/v4/letter/t/ba8739/32.png) [@Torben\_Knerr](https://discourse.chef.io/u/Torben_Knerr)\
**Post date:** [August 18, 2012, 2:45pm UTC](https://discourse.chef.io/t/meet-test-kitchen/3025/5 "2012-08-18T14:45:15Z")

</div>

Absolutely Nice!

So far I have just read the README.md and it looks very very nice. I  
have not actually tried it out yet, but from reading the README some  
questions came up:

1. Concerning cookbooks with dependencies to other cookbooks:

Having noticed the librarian Cheffile in the  
apache2/test/kitchen/cookbooks directory, I wonder whether the  
cookbook dependencies therein are automatically resolved by  
test-kitchen, are they? Would a Berksfile be supported as well?

If these dependencies are indeed automatically resolved then that  
would be worth mentioning in the README.md

1. Do you have an example with test-kitchen tests running on Travis  
CI? Btw: can you actually spin up a vagrant VM on Travis?

2. Concerning fast running tests:

It would be great to have fast running unit-like tests (which don't  
require to spin up a vagrant VM first) covered by test-kitchen as  
well. Chefspec comes to my mind, but I haven't found it referenced in  
the README.

Oh wait, isn't that already supported by the cookbook / script  
parameter which defaults to 'rspec spec'? Or would I do that within  
the cookbook / preflight command to make sure this runs _before_ the  
vagrant VM is spun up?

1. Concerning integration\_test:

The README "Kitchenfile DSL" section talks about a features dir  
underneath the /test/kitchen dir, but in the examples I see  
only /test/features?

Also, is the "integration\_test" block in the Kitchenfile really  
necessary to run the features? From the "Cucumber" section above,  
which says "Test Kitchen will run any features in the test/features  
subdirectory tagged with the configuration name in order to check that  
the service is working as expected.", it sounds like you wouldn't need  
to do anything extra to make the cucumber features run. In general,  
the "integration\_test" section really confused me...

1. Runtimes?

I don't get what the cookbook and integration\_test runtimes parameter  
is good for. Why does RVM need to be installed on the base box? Are  
you running the features locally on the target box? I would have  
expected that the features are run from the outside (e.g. on my  
workstation) against the provisioned target vm (the "black box"), so  
why would I care that RVM is installed inside that black box?!?

I guess most of my questions are based on misunderstandings. Please  
forgive the fact that I haven't actually run test-kitchen yet... 🙂

Anyways, thanks for all the Chef awesomeness!

Cheers,  
Torben

On Fri, Aug 17, 2012 at 6:00 PM, Joshua Timberman [joshua@opscode.com](mailto:joshua@opscode.com) wrote:

> Ohai Chefs!
> 
> Today we released test-kitchen, a framework for running integration tests in an isolated environment. It uses Vagrant and Chef under the hood to set up a single system for converging your cookbook, and optionally running rspec/minitest/cucumber tests.
> 
> It is released as a RubyGem, and issues are tracked in our Open Source ticket system.
> 
> - [Announcing Test Kitchen - Chef Blog | Chef](http://bit.ly/test-kitchen)
> 
> --  
> Opscode, Inc  
> Joshua Timberman, Technical Program Manager  
> IRC, Skype, Twitter, Github: jtimberman

---

<div class="post-metadata">

**Author:** ![Joshua\_Timberman](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/joshua_timberman/32/424_2.png) [@Joshua\_Timberman](https://discourse.chef.io/u/Joshua_Timberman)\
**Post date:** [August 18, 2012, 4:03pm UTC](https://discourse.chef.io/t/meet-test-kitchen/3025/6 "2012-08-18T16:03:37Z")

</div>

On Aug 18, 2012, at 8:45 AM, Torben Knerr [ukio@gmx.de](mailto:ukio@gmx.de) wrote:

> Absolutely Nice!

Thanks! We're all pretty excited about this. Andrew did a great job with it.

> 1. Concerning cookbooks with dependencies to other cookbooks:
> 
> Having noticed the librarian Cheffile in the  
> apache2/test/kitchen/cookbooks directory, I wonder whether the  
> cookbook dependencies therein are automatically resolved by  
> test-kitchen, are they? Would a Berksfile be supported as well?

Yes - test-kitchen wraps up librarian-chef in the base runner:

[https://github.com/opscode/test-kitchen/blob/master/lib/test-kitchen/runner/base.rb#L95-106](https://github.com/opscode/test-kitchen/blob/master/lib/test-kitchen/runner/base.rb#L95-106)

The reason Berkshelf isn't supported is simply because it didn't exist (publicly available) when test-kitchen was created and developed.

> If these dependencies are indeed automatically resolved then that  
> would be worth mentioning in the README.md

Would you please open a ticket in the KITCHEN project for that?

> 1. Do you have an example with test-kitchen tests running on Travis  
> CI? Btw: can you actually spin up a vagrant VM on Travis?

I'm not sure if Travis would support that. I haven't done much with Travis besides wire it up with foodcritic and knife cookbook test. And by that I mean I've merged pull requests that do that :).

We have Jenkins internally for other projects, like building our omnibus packages, and the services that run Hosted/Private Chef. We also have the cookbooks tested under test-kitchen, though it is very simplistic right now.

I'd like to see Travis.CI support in each of our cookbooks, whether they're just doing the foodcritic/cookbook test, or running test-kitchen, that'd be great.

> 1. Concerning fast running tests:
> 
> It would be great to have fast running unit-like tests (which don't  
> require to spin up a vagrant VM first) covered by test-kitchen as  
> well. Chefspec comes to my mind, but I haven't found it referenced in  
> the README.
> 
> Oh wait, isn't that already supported by the cookbook / script  
> parameter which defaults to 'rspec spec'? Or would I do that within  
> the cookbook / preflight command to make sure this runs _before_ the  
> vagrant VM is spun up?

Yes, that should be done by the "script" parameter. You should be able to just run that in your cookbook directory without test-kitchen if you don't want the full provisioning cycle.

> 1. Concerning integration\_test:
> 
> The README "Kitchenfile DSL" section talks about a features dir  
> underneath the /test/kitchen dir, but in the examples I see  
> only /test/features?
> 
> Also, is the "integration\_test" block in the Kitchenfile really  
> necessary to run the features? From the "Cucumber" section above,  
> which says "Test Kitchen will run any features in the test/features  
> subdirectory tagged with the configuration name in order to check that  
> the service is working as expected.", it sounds like you wouldn't need  
> to do anything extra to make the cucumber features run. In general,  
> the "integration\_test" section really confused me...

The cookbook features will automatically get run by test-kitchen if the directory exists.

The integration\_test block is for non-cookbook projects (like mixlib-shellout) that have unit or integration testing. While mixlib-shellout is kind of a contrived example, a better one would be a Rails application that has its own test suite that should be run in a fully deployed scenario with its database, etc.

> 1. Runtimes?
> 
> I don't get what the cookbook and integration\_test runtimes parameter  
> is good for. Why does RVM need to be installed on the base box? Are  
> you running the features locally on the target box? I would have  
> expected that the features are run from the outside (e.g. on my  
> workstation) against the provisioned target vm (the "black box"), so  
> why would I care that RVM is installed inside that black box?!?

Rvm was used as a relatively simple way to get specific versions of Ruby installed for performing the tests. Think of a non-cookbook project, like a Ruby library that you want to test on different versions of Ruby (like 1.8.7 vs 1.9.3, or even JRuby). My personal preference for cookbook testing is that we make the RVM part optional, and use the Ruby that is included in Chef on the base boxes already.

I created a ticket to track this:

[http://tickets.opscode.com/browse/KITCHEN-4](http://tickets.opscode.com/browse/KITCHEN-4)

--  
Opscode, Inc  
Joshua Timberman, Technical Program Manager  
IRC, Skype, Twitter, Github: jtimberman

---

<div class="post-metadata">

**Author:** ![Torben\_Knerr](https://avatars.discourse-cdn.com/v4/letter/t/ba8739/32.png) [@Torben\_Knerr](https://discourse.chef.io/u/Torben_Knerr)\
**Post date:** [August 23, 2012, 6:46am UTC](https://discourse.chef.io/t/meet-test-kitchen/3025/7 "2012-08-23T06:46:24Z")

</div>

Hi Joshua,

thanks for the quick reply!

> > If these dependencies are indeed automatically resolved then that  
> > would be worth mentioning in the README.md
> 
> Would you please open a ticket in the KITCHEN project for that?

Cool, having automatic dependency resolution is great!

I have created  
[http://tickets.opscode.com/browse/KITCHEN-8](http://tickets.opscode.com/browse/KITCHEN-8) for the missing README documentation  
and  
[http://tickets.opscode.com/browse/KITCHEN-9](http://tickets.opscode.com/browse/KITCHEN-9) for Berksfile support

> I'd like to see Travis.CI support in each of our cookbooks, whether they're just doing the foodcritic/cookbook test, or running test-kitchen, that'd be great.

Fully support that 🙂

Concerning the provisioning of a Vagrant VM on Travis, I will give it  
a try next week. If someone has already done that or a working example  
please let me know!

> > Oh wait, isn't that already supported by the cookbook / script  
> > parameter which defaults to 'rspec spec'? Or would I do that within  
> > the cookbook / preflight command to make sure this runs _before_ the  
> > vagrant VM is spun up?
> 
> Yes, that should be done by the "script" parameter. You should be able to just run that in your cookbook directory without test-kitchen if you don't want the full provisioning cycle.

Ok, great!

> > Also, is the "integration\_test" block in the Kitchenfile really  
> > necessary to run the features?
> 
> The cookbook features will automatically get run by test-kitchen if the directory exists.

Perfect, good to know,

> The integration\_test block is for non-cookbook projects (like mixlib-shellout) that have unit or integration testing. While mixlib-shellout is kind of a contrived example, a better one would be a Rails application that has its own test suite that should be run in a fully deployed scenario with its database, etc.

Now that I read it again it's clear 🙂

> Rvm was used as a relatively simple way to get specific versions of Ruby installed for performing the tests. Think of a non-cookbook project, like a Ruby library that you want to test on different versions of Ruby (like 1.8.7 vs 1.9.3, or even JRuby). My personal preference for cookbook testing is that we make the RVM part optional, and use the Ruby that is included in Chef on the base boxes already.
> 
> I created a ticket to track this:
> 
> [http://tickets.opscode.com/browse/KITCHEN-4](http://tickets.opscode.com/browse/KITCHEN-4)

Yes, for non-cookbook projects using "integration\_test" that makes sense.

For cookbook projects using "cookbook" the rvm option still does not  
(generally) make sense to me. Okay, if I know I have a basebox with  
rvm and some Rubies installed, then it would be fair to state that you  
want to test against several different rubies. Yes, but this makes  
some heavy assumptions about rvm and rubies being installed on the  
basebox, so I would rather see "runtimes" as an option within  
"platform", but not within "cookbooks", e.g.

"""  
platform :centos do  
version "5.8" do  
box "opscode-centos-5.8"  
box\_url "[https://opscode-vm.s3.amazonaws.com/vagrant/boxes/opscode-centos-5.8.box](https://opscode-vm.s3.amazonaws.com/vagrant/boxes/opscode-centos-5.8.box)"  
runtimes ['1.9.2', '1.9.3']  
end  
end  
"""

The default of would mean that the system ruby that is installed  
along with Chef of basebox is used. If "runtimes" is specified it  
would explicitly mean rvm rubies.

For sure, you could then also exclude specific runtimes:

"""  
cookbook "apache2" do  
configuration "default"  
exclude :platform =\> 'centos', :runtimes =\> ['1.9.2']  
run\_list\_extras ['apache2\_test::setup']  
end  
"""

Does that make sense?

Finally, this part of the current "cookbook" / "runtimes"  
documentation confuses me most: "The default for cookbook projects is  
, which effectively disables spec/feature tests"

Why should spec/feature tests be disabled in this case?

Cheers,  
Torben

> --  
> Opscode, Inc  
> Joshua Timberman, Technical Program Manager  
> IRC, Skype, Twitter, Github: jtimberman

---

<div class="post-metadata">

**Author:** ![Joshua\_Timberman](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/joshua_timberman/32/424_2.png) [@Joshua\_Timberman](https://discourse.chef.io/u/Joshua_Timberman)\
**Post date:** [August 24, 2012, 2:50am UTC](https://discourse.chef.io/t/meet-test-kitchen/3025/8 "2012-08-24T02:50:48Z")

</div>

Ohai,

Thanks for opening the tickets, it's important to get those things tracked so folks can submit feedback, or contribute.

On Aug 23, 2012, at 12:46 AM, Torben Knerr [ukio@gmx.de](mailto:ukio@gmx.de) wrote:

> Yes, for non-cookbook projects using "integration\_test" that makes sense.
> 
> For cookbook projects using "cookbook" the rvm option still does not  
> (generally) make sense to me. Okay, if I know I have a basebox with  
> rvm and some Rubies installed, then it would be fair to state that you  
> want to test against several different rubies. Yes, but this makes  
> some heavy assumptions about rvm and rubies being installed on the  
> basebox, so I would rather see "runtimes" as an option within  
> "platform", but not within "cookbooks", e.g.
> 
> """  
> platform :centos do  
> version "5.8" do  
> box "opscode-centos-5.8"  
> box\_url "[https://opscode-vm.s3.amazonaws.com/vagrant/boxes/opscode-centos-5.8.box](https://opscode-vm.s3.amazonaws.com/vagrant/boxes/opscode-centos-5.8.box)"  
> runtimes ['1.9.2', '1.9.3']  
> end  
> end  
> """
> 
> The default of would mean that the system ruby that is installed  
> along with Chef of basebox is used. If "runtimes" is specified it  
> would explicitly mean rvm rubies.
> 
> For sure, you could then also exclude specific runtimes:
> 
> """  
> cookbook "apache2" do  
> configuration "default"  
> exclude :platform =\> 'centos', :runtimes =\> ['1.9.2']  
> run\_list\_extras ['apache2\_test::setup']  
> end  
> """
> 
> Does that make sense?
> 
> Finally, this part of the current "cookbook" / "runtimes"  
> documentation confuses me most: "The default for cookbook projects is  
> , which effectively disables spec/feature tests"
> 
> Why should spec/feature tests be disabled in this case?

You also posted this to the appropriate tickets, yes? I just want to make sure I interpreted the comments correctly.

Thanks!

---

<div class="post-metadata">

**Author:** ![Torben\_Knerr](https://avatars.discourse-cdn.com/v4/letter/t/ba8739/32.png) [@Torben\_Knerr](https://discourse.chef.io/u/Torben_Knerr)\
**Post date:** [August 24, 2012, 5:11am UTC](https://discourse.chef.io/t/meet-test-kitchen/3025/9 "2012-08-24T05:11:10Z")

</div>

Hi Joshua,

yep, it's all added as comments to the tickets now, except...

Am 24.08.2012 04:50 schrieb "Joshua Timberman" [joshua@opscode.com](mailto:joshua@opscode.com):

> Ohai,
> 
> Thanks for opening the tickets, it's important to get those things tracked  
> so folks can submit feedback, or contribute.
> 
> On Aug 23, 2012, at 12:46 AM, Torben Knerr [ukio@gmx.de](mailto:ukio@gmx.de) wrote:
> 
> > Yes, for non-cookbook projects using "integration\_test" that makes sense.
> > 
> > For cookbook projects using "cookbook" the rvm option still does not  
> > (generally) make sense to me. Okay, if I know I have a basebox with  
> > rvm and some Rubies installed, then it would be fair to state that you  
> > want to test against several different rubies. Yes, but this makes  
> > some heavy assumptions about rvm and rubies being installed on the  
> > basebox, so I would rather see "runtimes" as an option within  
> > "platform", but not within "cookbooks", e.g.
> > 
> > """  
> > platform :centos do  
> > version "5.8" do  
> > box "opscode-centos-5.8"  
> > box\_url "  
> > [https://opscode-vm.s3.amazonaws.com/vagrant/boxes/opscode-centos-5.8.box](https://opscode-vm.s3.amazonaws.com/vagrant/boxes/opscode-centos-5.8.box)"  
> > runtimes ['1.9.2', '1.9.3']  
> > end  
> > end  
> > """
> > 
> > The default of would mean that the system ruby that is installed  
> > along with Chef of basebox is used. If "runtimes" is specified it  
> > would explicitly mean rvm rubies.
> > 
> > For sure, you could then also exclude specific runtimes:
> > 
> > """  
> > cookbook "apache2" do  
> > configuration "default"  
> > exclude :platform =\> 'centos', :runtimes =\> ['1.9.2']  
> > run\_list\_extras ['apache2\_test::setup']  
> > end  
> > """
> > 
> > Does that make sense?
> > 
> > Finally, this part of the current "cookbook" / "runtimes"  
> > documentation confuses me most: "The default for cookbook projects is  
> > , which effectively disables spec/feature tests"
> > 
> > Why should spec/feature tests be disabled in this case?
> 
> You also posted this to the appropriate tickets, yes? I just want to make  
> sure I interpreted the comments correctly.

...that I've changed my mind a bit here: assuming that `runtimes` makes  
sense only within `integration_test` but not within `cookbook` (as argued  
in the comments[http://tickets.opscode.com/browse/KITCHEN-4?focusedCommentId=26525&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-26525](http://tickets.opscode.com/browse/KITCHEN-4?focusedCommentId=26525&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-26525)),  
I would not add `runtimes` to `platform` and keep it really specific to the  
non-cookbook integration tests. Thus I did not add the code snippets above  
to the ticket.

Cheers,  
Torben

> Thanks!

---

<div class="post-metadata">

**Author:** ![gholtzhausen](https://avatars.discourse-cdn.com/v4/letter/g/aeb1de/32.png) [@gholtzhausen](https://discourse.chef.io/u/gholtzhausen)\
**Post date:** [September 9, 2019, 6:04pm UTC](https://discourse.chef.io/t/meet-test-kitchen/3025/10 "2019-09-09T18:04:56Z")

</div>

> [@Joshua\_Timberman](#):
>
> [Announcing Test Kitchen - Chef Blog | Chef](http://bit.ly/test-kitchen)

What is the easiest approach for a newbie to create tests using test kitchen and inspec to be deployed into a jenkins pipeline using the bitbucket hook?  
Should all cookbooks reside within the test folder or in the recipe folder and referenced in the test/default ruby file via include..?