Kitchen-Docker Much Slower than Kitchen-Vagrant

Hi,

I am trying out kitchen-docker as a replacement for kitchen-vagrant, hoping
to speed up my integration tests. Unfortunately, the tests take ten times
as long to run with kitchen-docker compared to kitchen vagrant. I am on
test-kitchen 1.3.1 and kitchen-docker 1.7.0. Below is my .kitchen.yml. I
used Boot2Docker to install Docker and Vagrant on my machine. Thank you
for your help.

Jimmy

https://mail.google.com/mail/u/0/#inbox?compose=14c84e4a519b2984


driver:
name: docker

provisioner:
name: chef_zero
data_bags_path: test/fixtures/data_bags
environments_path: test/fixtures/environments

platforms:

  • name: ubuntu-14.04
    driver_config:
    image: ubuntu:14.04
    container_name: ubuntu_test_1404
    platform: ubuntu
    attributes:

suites:

  • name: stag
    provisioner:
    client_rb:
    environment: masterwrap-stag
    run_list:
    • recipe[masterwrap::default]
      attributes:

Hi,

Il giorno sab 4 apr 2015 alle ore 16:47 Jimmy Huang
jimmy.huang@duragility.com ha scritto:

I am trying out kitchen-docker as a replacement for kitchen-vagrant, hoping to speed up my integration tests. Unfortunately, the tests take ten times as long to run with kitchen-docker compared to kitchen vagrant. I am on test-kitchen 1.3.1 and kitchen-docker 1.7.0. Below is my .kitchen.yml. I used Boot2Docker to install Docker and Vagrant on my machine. Thank you for your help.

This is not a surprise, if you use boot2docker on windows/mac you are
just adding an additional layer in the virtualization stack and I/O
performances can't be better than Virtualbox stand-alone.

There’s some caveats with kitchen-docker. When the cache is cold, things are slow. Once it’s warmed up, things tend to be really fast. In my experience, the first run with kitchen-docker is about the same speed or a bit slower than an equivalent Vagrant run.

Also, you can’t do your first kitchen run in parallel with a cold boot2docker. The disk usage in the vm tends to increase faster than the guest-disk-auto-resize in virtualbox can manage, and my runs fail as a result. Serialize the first time through.

All that said, subsequent runs are much, much faster and parallelize no problem.

My setup:

Retina MBP Mid–2012 (Core i7 16GB RAM)
OSX 10.10.2
boot2docker 1.5.0 (from brew cask)
docker 1.5.0 (from brew cask)
chefdk 0.4.0
VMWare Fusion 7.1.1
Vagrant 1.7.2
vagrant-vmware-fusion (3.2.0)
Have my vagrant box already cached
Here’s a repo with a very simple recipe, just one template and an attribute feature flag. The repo contains a .kitchen.yml that does 3 suites: https://github.com/charlesjohnson/fundamentals-with-tests/blob/master/chef-repo/cookbooks/motd/.kitchen.yml

Here’s my .kitchen.local.yml that I layer in on top of it:

driver_plugin: docker
driver_config:
provision_command: curl -L https://www.opscode.com/chef/install.sh | bash
require_chef_omnibus: false
use_sudo: false
Turning off the require_chef_omnibus flag and manually curl-pipe-bash’ing puts the installation of chef-client into its own layer in the docker cache. Slow the first time, zippy as hell afterward.

Here’s some quick informal benchmarks for those runs, just doing a kitchen destroy; kitchen converge (no busser):

Kitchen-Docker (cold booted VM, no layers cached at all, no parallel):
-----> Kitchen is finished. (2m38.15s)

Kitchen-Docker (warm cache, no parallel): -----> Kitchen is finished. (0m22.73s)

Kitchen-Docker (warm cache, parallel): -----> Kitchen is finished. (0m7.02s)

So by using kitchen-docker I get 8-second kitchen runs, including machine creation from scratch each time. Not bad at all.

Thanks,
–Charles

Charles Johnson — Product Engineer
(510) 545-9485 – charles@chef.io – my: Linkedin Twitter

CHEF

TM
chef.io Blog Facebook Twitter Youtube

On April 4, 2015 at 7:47:45 AM, Jimmy Huang (jimmy.huang@duragility.com) wrote:

Hi,

I am trying out kitchen-docker as a replacement for kitchen-vagrant, hoping to speed up my integration tests. Unfortunately, the tests take ten times as long to run with kitchen-docker compared to kitchen vagrant. I am on test-kitchen 1.3.1 and kitchen-docker 1.7.0. Below is my .kitchen.yml. I used Boot2Docker to install Docker and Vagrant on my machine. Thank you for your help.

Jimmy

https://mail.google.com/mail/u/0/#inbox?compose=14c84e4a519b2984


driver:
name: docker

provisioner:
name: chef_zero
data_bags_path: test/fixtures/data_bags
environments_path: test/fixtures/environments

platforms:

  • name: ubuntu-14.04
    driver_config:
    image: ubuntu:14.04
    container_name: ubuntu_test_1404
    platform: ubuntu
    attributes:

suites:

  • name: stag
    provisioner:
    client_rb:
    environment: masterwrap-stag
    run_list:
  • recipe[masterwrap::default]
    attributes:

Hey Charles,

thanks for posting this!

I now get similar numbers on a Windows host with docker 1.6 / boot2docker.
This is making test-kitchen on windows fun again! :slight_smile:

As for the provision command, I'm now using this:

driver_config:
provision_command:
- curl -L https://www.opscode.com/chef/install.sh | bash -s -- -p -v 12.3.0
- env GEM_HOME=/tmp/verifier/gems GEM_PATH=/tmp/verifier/gems
GEM_CACHE=/tmp/verifier/gems/cache /opt/chef/embedded/bin/gem install
thor busser busser-serverspec serverspec bundler && chown -R
kitchen:kitchen /tmp/verifier

  1. installs a specific chef version, 12.3.0 rc pre-release in this case
  2. pre-caches the busser installed gems in for test-kitchen 1.4.0.rc1
    (thanks @coderanger!)

Gives me around -----> Kitchen is finished. (0m21.66s) for a full converge

  • test + destroy run (i.e. kitchen test -c) in the above motd example

Cheers,
Torben

On Mon, Apr 6, 2015 at 10:57 PM, Charles Johnson charles@chef.io wrote:

There’s some caveats with kitchen-docker. When the cache is cold, things
are slow. Once it’s warmed up, things tend to be really fast. In my
experience, the first run with kitchen-docker is about the same speed or a
bit slower than an equivalent Vagrant run.

Also, you can’t do your first kitchen run in parallel with a cold
boot2docker. The disk usage in the vm tends to increase faster than the
guest-disk-auto-resize in virtualbox can manage, and my runs fail as a
result. Serialize the first time through.

All that said, subsequent runs are much, much faster and parallelize no
problem.

My setup:

  • Retina MBP Mid–2012 (Core i7 16GB RAM)
  • OSX 10.10.2
  • boot2docker 1.5.0 (from brew cask)
  • docker 1.5.0 (from brew cask)
  • chefdk 0.4.0
  • VMWare Fusion 7.1.1
  • Vagrant 1.7.2
  • vagrant-vmware-fusion (3.2.0)
  • Have my vagrant box already cached

Here’s a repo with a very simple recipe, just one template and an
attribute feature flag. The repo contains a .kitchen.yml that does 3
suites:
https://github.com/charlesjohnson/fundamentals-with-tests/blob/master/chef-repo/cookbooks/motd/.kitchen.yml

Here’s my .kitchen.local.yml that I layer in on top of it:

driver_plugin: docker
driver_config:
provision_command: curl -L https://www.opscode.com/chef/install.sh | bash
require_chef_omnibus: false
use_sudo: false

Turning off the require_chef_omnibus flag and manually curl-pipe-bash’ing
puts the installation of chef-client into its own layer in the docker
cache. Slow the first time, zippy as hell afterward.

Here’s some quick informal benchmarks for those runs, just doing a kitchen
destroy; kitchen converge (no busser):

Kitchen-Docker (cold booted VM, no layers cached at all, no parallel):
-----> Kitchen is finished. (2m38.15s)

Kitchen-Docker (warm cache, no parallel): -----> Kitchen is finished.
(0m22.73s)

Kitchen-Docker (warm cache, parallel): -----> Kitchen is finished.
(0m7.02s)

So by using kitchen-docker I get 8-second kitchen runs, including machine
creation from scratch each time. Not bad at all.

Thanks,
--Charles

Charles Johnson — Product Engineer

(510) 545-9485 – http://mailto:charles@getchef.com/charles@chef.io
charles@opscode.com – *my: *Linkedin
http://www.linkedin.com/pub/charles-johnson/1/b38/6a5 Twitter
https://twitter.com/chipadeedoodah

CHEF

CHEF.IO http://www.getchef.com/

TM

chef.io http://www.getchef.com/ http://www.getchef.com/blog/Blog
http://www.opscode.com/blog/ Facebook
https://www.facebook.com/getchefdotcom Twitter
https://twitter.com/getchefdotcom Youtube
https://www.youtube.com/getchef

On April 4, 2015 at 7:47:45 AM, Jimmy Huang (jimmy.huang@duragility.com)
wrote:

Hi,

I am trying out kitchen-docker as a replacement for kitchen-vagrant,
hoping to speed up my integration tests. Unfortunately, the tests take ten
times as long to run with kitchen-docker compared to kitchen vagrant. I am
on test-kitchen 1.3.1 and kitchen-docker 1.7.0. Below is my .kitchen.yml.
I used Boot2Docker to install Docker and Vagrant on my machine. Thank you
for your help.

Jimmy

https://mail.google.com/mail/u/0/#inbox?compose=14c84e4a519b2984


driver:
name: docker

provisioner:
name: chef_zero
data_bags_path: test/fixtures/data_bags
environments_path: test/fixtures/environments

platforms:

  • name: ubuntu-14.04
    driver_config:
    image: ubuntu:14.04
    container_name: ubuntu_test_1404
    platform: ubuntu
    attributes:

suites:

  • name: stag
    provisioner:
    client_rb:
    environment: masterwrap-stag
    run_list:
  • recipe[masterwrap::default]
    attributes: