Couple of test-kitchen questions


#1

Coming back to test-kitchen after a little time off, and glad to see lots
of improvements in very short period of time. I’ve got two items which are
slowing me down that I’m hoping might have easy solutions:

  1. Shutting down a Configuration after test: If I have multiple
    configurations lets say 4 ( a client and a server configuration on 2
    platforms). It seems like the VMs are spun up serially, and one is
    completed its left on while the next is started. Is it possible to have the
    original shutdown (without destroy) at the end of the run, so I’m not left
    with 4 VMs spinning by the end of the test? The --teardown option seems to
    imply the VMs will be destroyed?

  2. Almost in contrast of my first question, I’m wondering how to speed up
    tests during quick iteration cycles. Before test-kitchen, using standalone
    Vagrant files, I could just run vagrant provision blah over and over as I
    fine tune things for a set configuration. I understand that I can use the
    –configuration and the --platform options to kitchen test to limit the
    scope of the test, but ideally I’d like to skip all the pre-checks as well,
    especially the “Validating ruby files” which seems obnoxiously slow on my
    machine. I still want all checks to run when I think I have a finished
    product, just not while I’m iterating through and fine tuning a recipe. Any
    way to pull that off?


#2

Ohai!

On Nov 2, 2012, at 16:20, “David Petzel” davidpetzel@gmail.com wrote:

  1. Shutting down a Configuration after test: If I have multiple configurations lets say 4 ( a client and a server configuration on 2 platforms). It seems like the VMs are spun up serially, and one is completed its left on while the next is started. Is it possible to have the original shutdown (without destroy) at the end of the run, so I’m not left with 4 VMs spinning by the end of the test? The --teardown option seems to imply the VMs will be destroyed?

What about simply leveraging your shell?

bundle exec kitchen test && bundle exec kitchen destroy

Then the environment would only be torn down if the run was successful.

  1. Almost in contrast of my first question, I’m wondering how to speed up tests during quick iteration cycles. Before test-kitchen, using standalone Vagrant files, I could just run vagrant provision blah over and over as I fine tune things for a set configuration. I understand that I can use the --configuration and the --platform options to kitchen test to limit the scope of the test, but ideally I’d like to skip all the pre-checks as well, especially the “Validating ruby files” which seems obnoxiously slow on my machine. I still want all checks to run when I think I have a finished product, just not while I’m iterating through and fine tuning a recipe. Any way to pull that off?

Validating ruby files is the knife cookbook test command that checks syntax of the rb and erb. That should be pretty fast, though as it stores the checksums with a time stamp and only checks files that have changed.

There seem to be other opportunities for optimization, though.

  1. The cookbook assembly with librarian happens very run. I am not sure if this is due to librarian or test kitchens use of it.

  2. Berkshelf may be faster at resolution of dependencies based on conversations I hea at the chef summit. A ticket is open for this in the kitchen project.

  3. The test-kitchen cookbook itself does some work that may not be required, such as using the rvm cookbook and installing fit and other things beforehand.

If you can think of others, please open tickets.


#3

Thanks for the info Joshua

What about simply leveraging your shell?

bundle exec kitchen test && bundle exec kitchen destroy

Then the environment would only be torn down if the run was successful.

The tearing down isn’t so much my problem as having all the VMs up and
running at once. Unless I’m misunderstanding, I believe the command line
above would still in result in all VMs running at the same time by the
end of the successful run? Basically what I’m looking for is on a
successful run of a configuration on a platform, shutdown (not destroy)
that platform, before moving onto the next

Validating ruby files is the knife cookbook test command that checks
syntax of the rb and erb. That should be pretty fast, though as it stores
the checksums with a time stamp and only checks files that have changed.

I probably should clarify that I’m on a windows workstation, so Rubyin
general is just obnoxiously slow to start at times, so I think the syntax
check times are more a result of how long it takes ruby to initialize on my
workstation than the checks themselves. Whatever the cause of the slowness
(and the quantify of it), it would be nice to be able to just "skip it"
when desired.

There seem to be other opportunities for optimization, though.

  1. The test-kitchen cookbook itself does some work that may not be
    required, such as using the rvm cookbook and installing fit and other
    things beforehand.

I don’t mind the work that is done, and I don’t want to skip it all the
time (which can be done fairly easily via the Kitchenfile, but I’d like the
ability to skip on demand. I.E., while iterating, say “don’t do all the
cool stuff, just provision.”. Its obviously not a huge deal, and the time
that test-kitchen saves dwarfs this by orders of magnitude, so I don’ want
it to sound like a complaint per say, but rather just exploring tiny time
savers


#4

As a side note is anyone actively working on abstracting the
cook-management out on kitchen right now?(KITCHEN-9). I Started
looking into it, but don’t want to step on toes.

On Sat, Nov 3, 2012 at 7:42 PM, David Petzel davidpetzel@gmail.com wrote:

Thanks for the info Joshua

What about simply leveraging your shell?

bundle exec kitchen test && bundle exec kitchen destroy

Then the environment would only be torn down if the run was successful.

The tearing down isn’t so much my problem as having all the VMs up and
running at once. Unless I’m misunderstanding, I believe the command line
above would still in result in all VMs running at the same time by the end
of the successful run? Basically what I’m looking for is on a successful run
of a configuration on a platform, shutdown (not destroy) that platform,
before moving onto the next

Validating ruby files is the knife cookbook test command that checks
syntax of the rb and erb. That should be pretty fast, though as it stores
the checksums with a time stamp and only checks files that have changed.

I probably should clarify that I’m on a windows workstation, so Rubyin
general is just obnoxiously slow to start at times, so I think the syntax
check times are more a result of how long it takes ruby to initialize on my
workstation than the checks themselves. Whatever the cause of the slowness
(and the quantify of it), it would be nice to be able to just “skip it” when
desired.

There seem to be other opportunities for optimization, though.

  1. The test-kitchen cookbook itself does some work that may not be
    required, such as using the rvm cookbook and installing fit and other things
    beforehand.

I don’t mind the work that is done, and I don’t want to skip it all the time
(which can be done fairly easily via the Kitchenfile, but I’d like the
ability to skip on demand. I.E., while iterating, say “don’t do all the cool
stuff, just provision.”. Its obviously not a huge deal, and the time that
test-kitchen saves dwarfs this by orders of magnitude, so I don’ want it to
sound like a complaint per say, but rather just exploring tiny time savers


#5

On Nov 3, 2012, at 8:42 PM, David Petzel davidpetzel@gmail.com wrote:

The tearing down isn’t so much my problem as having all the VMs up and running at once. Unless I’m misunderstanding, I believe the command line above would still in result in all VMs running at the same time by the end of the successful run? Basically what I’m looking for is on a successful run of a configuration on a platform, shutdown (not destroy) that platform, before moving onto the next

Ah, yes. All the vagrant boxes will be running until the kitchen destroy command is issued.

I probably should clarify that I’m on a windows workstation, so Rubyin general is just obnoxiously slow to start at times, so I think the syntax check times are more a result of how long it takes ruby to initialize on my workstation than the checks themselves. Whatever the cause of the slowness (and the quantify of it), it would be nice to be able to just “skip it” when desired.

Everything working okay on Windows? There were conversations at the summit that indicated that test kitchen doesn’t actually yet run on Windows. If you had to do anything special to get that going, or if it just worked following the README, I’d like to hear it.

I don’t mind the work that is done, and I don’t want to skip it all the time (which can be done fairly easily via the Kitchenfile, but I’d like the ability to skip on demand. I.E., while iterating, say “don’t do all the cool stuff, just provision.”. Its obviously not a huge deal, and the time that test-kitchen saves dwarfs this by orders of magnitude, so I don’ want it to sound like a complaint per say, but rather just exploring tiny time savers

Time savers add up, of course. If you can determine where additional optimization can occur, please do open a KITCHEN ticket.

Cheers!


Opscode, Inc
Joshua Timberman, Technical Community Manager
IRC, Skype, Twitter, Github: jtimberman


#6

On Sun, Nov 4, 2012 at 8:18 PM, Joshua Timberman joshua@opscode.com wrote:

Everything working okay on Windows? There were conversations at the summit
that indicated that test kitchen doesn’t actually yet run on Windows. If
you had to do anything special to get that going, or if it just worked
following the README, I’d like to hear it.

Nothing special really. The only thing outside of the readme I ran into,
which I don’t believe is specific to Windows is
http://tickets.opscode.com/browse/KITCHEN-11

Aside from that its working fine on my Windows 7 Laptop.