Status of test-kitchen?


#1

I really like test-kitchen and I think it has tremendous potential however
I find that in its current state it is not a productive tool for me. There
are many outstanding bugs, as one can see here:
http://tickets.opscode.com/secure/IssueNavigator.jspa?reset=true&jqlQuery=project+%3D+KITCHEN+AND+fixVersion+%3D+EMPTY
but
the one that I find particularly challenging is that I cannot get
test-kitchen to test only one virtualbox image rather than all 4 (centos
5,6, ubuntu 10,11). Further, I can’t figure out how to run librarian-chef update if I make any changes to my Cheffile.

I have gotten the distinct impression that test-kitchen is the preferred
testing tool for the opscode-cookbooks/ set of cookbooks. However, at this
stage I find myself just spending to much time fighting w/ test-kitchen. I
would like to know if anyone plans to put in the time and effort to bring
test-kitchen to a more stable and usable form. I would love to pitch in,
but right now my priority is to improve the state of the
database/postgresql and java-related cookbooks.

In the meantime I will refocus my workflow, which I call "The Riot Way"
based on Jamie Winsor’s cookbook authoring guide
http://vialstudios.com/guide-authoring-cookbooks.html

BryanWB


#2

Last time I checked, test-kitchen required some custom Vagrant boxes
to work. It wasn’t even possible to use the boxes built with Bento,
not to speak of http://files.vagrantup.com/lucid32.box and friends.

I’d like to know the status as well.

-Mathias

On Wed, Sep 26, 2012 at 4:13 PM, Bryan Berry bryan.berry@gmail.com wrote:

I really like test-kitchen and I think it has tremendous potential however I
find that in its current state it is not a productive tool for me. There
are many outstanding bugs, as one can see here:
http://tickets.opscode.com/secure/IssueNavigator.jspa?reset=true&jqlQuery=project+%3D+KITCHEN+AND+fixVersion+%3D+EMPTY
but the one that I find particularly challenging is that I cannot get
test-kitchen to test only one virtualbox image rather than all 4 (centos
5,6, ubuntu 10,11). Further, I can’t figure out how to run librarian-chef update if I make any changes to my Cheffile.

I have gotten the distinct impression that test-kitchen is the preferred
testing tool for the opscode-cookbooks/ set of cookbooks. However, at this
stage I find myself just spending to much time fighting w/ test-kitchen. I
would like to know if anyone plans to put in the time and effort to bring
test-kitchen to a more stable and usable form. I would love to pitch in, but
right now my priority is to improve the state of the database/postgresql and
java-related cookbooks.

In the meantime I will refocus my workflow, which I call "The Riot Way"
based on Jamie Winsor’s cookbook authoring guide
http://vialstudios.com/guide-authoring-cookbooks.html

BryanWB


#3

Hi Bryan,

On Sep 26, 2012, at 8:13 AM, Bryan Berry bryan.berry@gmail.com wrote:

I really like test-kitchen and I think it has tremendous potential however I find that in its current state it is not a productive tool for me. There are many outstanding bugs, as one can see here: http://tickets.opscode.com/secure/IssueNavigator.jspa?reset=true&jqlQuery=project+%3D+KITCHEN+AND+fixVersion+%3D+EMPTY but the one that I find particularly challenging is that I cannot get test-kitchen to test only one virtualbox image rather than all 4 (centos 5,6, ubuntu 10,11). Further, I can’t figure out how to run librarian-chef update if I make any changes to my Cheffile.

Our current code review meetings cover tickets that are resolved. That is, we review pull requests/patches. Most of the tickets in Test Kitchen aren’t resolved, so there isn’t much to review (yet).

That said, I see four tickets in resolved state. They’re all things I’d like to get into the next release (0.5.6), but I don’t have an ETA.

  • KITCHEN-4
  • KITCHEN-22
  • KITCHEN-35
  • KITCHEN-37

Much of our general code review effort recently has been focused on getting 10.14 regressions fixed (mainly in the deploy resource)[0]. This week we had a couple people out of the office, too.

Bringing up the topic on the mailing list(s) is good though, and this is exactly the kind of topic that the chef-dev list and #chef-hacking IRC channel are intended to facilitate.

I have gotten the distinct impression that test-kitchen is the preferred testing tool for the opscode-cookbooks/ set of cookbooks. However, at this stage I find myself just spending to much time fighting w/ test-kitchen. I would like to know if anyone plans to put in the time and effort to bring test-kitchen to a more stable and usable form. I would love to pitch in, but right now my priority is to improve the state of the database/postgresql and java-related cookbooks.

Your contributions to the cookbooks are awesome, thank you. The 1.6.0 release of the Java cookbook passed a test-kitchen run, too :-).

As you’ve noticed, one of the things we’re doing is add Test Kitchen support to our cookbooks, and that is revealing issues that need to be fixed. For example, we can’t really do much useful testing of heavily search based cookbooks (like nagios or munin) in test kitchen because it only supports Chef Solo[1]. Our internal CI system uses OpenStack, so a runner for that in Test Kitchen would be awesome[2].

In the meantime I will refocus my workflow, which I call “The Riot Way” based on Jamie Winsor’s cookbook authoring guide http://vialstudios.com/guide-authoring-cookbooks.html

I’ve only skimmed the guide, but it looks great. Jamie and Riot are awesome.

[0]: 10.14.4 was just tagged for release while I wrote this post.
[1]: https://tickets.opscode.com/browse/KITCHEN-31
[2]: http://tickets.opscode.com/browse/KITCHEN-5

Opscode, Inc
Joshua Timberman, Technical Community Manager
IRC, Skype, Twitter, Github: jtimberman


#4

I know I talked about this before but this seems like a great time to bring
it up again. What about chef-solo-search0? It gives chef solo the ability
to use many search features. Obviously it can’t completely replace
searching against a chef server. At least I don’t think it can. But can it
help with some of the issues you are running into? If so is there any
technical or philosophical reason preventing it from being rolled into the
chef client code?


#5

+1, I would love to know the same

On Fri, Sep 28, 2012 at 2:15 AM, Jeremiah Snapp
jeremiah.snapp@gmail.com wrote:

I know I talked about this before but this seems like a great time to bring
it up again. What about chef-solo-search0? It gives chef solo the ability
to use many search features. Obviously it can’t completely replace searching
against a chef server. At least I don’t think it can. But can it help with
some of the issues you are running into? If so is there any technical or
philosophical reason preventing it from being rolled into the chef client
code?


#6

On Fri, Sep 28, 2012 at 1:36 AM, Joshua Timberman joshua@opscode.com wrote:

Hi Bryan,

On Sep 26, 2012, at 8:13 AM, Bryan Berry bryan.berry@gmail.com wrote:
Bringing up the topic on the mailing list(s) is good though, and this is exactly the kind of topic that the chef-dev list and #chef-hacking IRC channel are intended to facilitate.

I have gotten the distinct impression that test-kitchen is the preferred testing tool for the opscode-cookbooks/ set of cookbooks. However, at this stage I find myself just spending to much time fighting w/ test-kitchen. I would like to know if anyone plans to put in the time and effort to bring test-kitchen to a more stable and usable form. I would love to pitch in, but right now my priority is to improve the state of the database/postgresql and java-related cookbooks.

Your contributions to the cookbooks are awesome, thank you. The 1.6.0 release of the Java cookbook passed a test-kitchen run, too :-).

Tks, I used test-kitchen for my changes to the java cookbok but I have
to say I found it quite frustrating and time-consuming. I hope the
process gets the smoother in the coming weeks.

Good to hear that you guys are still focused on improving test-kitchen


#7

+1. would love to see solo and client parity


Edmund Haselwanter
edmund@haselwanter.com

On 28.09.2012, at 11:00, Bryan Berry bryan.berry@gmail.com wrote:

+1, I would love to know the same

On Fri, Sep 28, 2012 at 2:15 AM, Jeremiah Snapp
jeremiah.snapp@gmail.com wrote:

I know I talked about this before but this seems like a great time to bring
it up again. What about chef-solo-search0? It gives chef solo the ability
to use many search features. Obviously it can’t completely replace searching
against a chef server. At least I don’t think it can. But can it help with
some of the issues you are running into? If so is there any technical or
philosophical reason preventing it from being rolled into the chef client
code?