Vagrant 1.7.3 chef-zero provisioner mostly unusable


#1

Previous versions of Vagrant (1.7.2 and prior) used a fudged Chef Zero
provisioner that was actually Chef Solo under the covers:

This meant that all your data bags and the code that depended on them had
to be fooled, stubbed, or coerced if you wanted to test it in Vagrant.
This was generally seen as a bummer and widely regarded as a time sink that
produced little business value. “That’s what you get for using open source
tools, Greg.”

We’d been waiting for Vagrant 1.7.3 over here for some time because it
promised real Chef Zero support: This should have made our hacks and dumb
shell scripts obsolete, allowing me to give my dev’s a Chef role and
nothing else for configuring their development VMs

And so it was like a child starting its first day of school and immediately
after dreaming of christmas morning that we watched the above PR get merged
and thought to ourselves “probably won’t be more than 5 or 6 months before
that gets bundled into an official release.”

Today I can say that this does not fit in our workflow. The Chef Zero
provisioner is taking anywhere from 20 to 40 minutes on different laptops
just to load our cookbooks into memory from local disk. Then there’s the
speed at which it seems to be configuring the resources in a run list. I
don’t have any real figures to refer to, just anecdotal evidence, so I’ll
resort to hyperbole. Vagrant 1.7.2 and Chef Solo seemed zippy enough. I
could spin up a development vagrant machine in about 5-10 minutes depending
on role and network. 1.7.3 and Chef Zero are reminiscent of a snail race
through knee-deep peanut butter.

$ time vagrant provision


wait for it…

pain…
continue waiting


real 79m59.131s
user 0m4.463s
sys 0m1.405s

Sometimes we use Vagrant to test Chef changes locally before uploading to a
CI server that will build a branch of chef code on EC2 and run some tests
before we mark it as merge-able. I’m telling Chef dev’s in this
organization that Vagrant is supposed to save them time, that local
development is a convenience… and now this. I literally can’t even.

Who’s got a workaround for all these network round trips and molasses
uploads? Do I need more ergs? Should I just rub some Docker on it? Aside
from declaring ‘no joy’ and rolling back to Vagrant 1.7.2 what are my
options here? We’ve tried different folder syncing methods (nfs, rsync)
and found them all wanting. I’m afraid this is a feature, not a bug:

Greg Damiani | Senior Kitten Delivery Engineer | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7E

Greg Damiani | Senior System Administrator | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7EB


#2

Out of interest do you get similar times if you spool up a local chef-zero
instance and knife upload and/or spool up an instance using the chef_zero
provider, just to make sure it’s not Vagrant doing something strange?

That seems like a long, long time for spooling to me. I used kitchen-ec2 to
spool instances on AWS and, granted my cookbook stack is probably smaller
than yours, it takes maybe 15-20 minutes to spool an instance (so
transferring over the internet).

On Fri, Jul 17, 2015 at 6:48 AM, Greg Damiani greg.damiani@buzzfeed.com
wrote:

Previous versions of Vagrant (1.7.2 and prior) used a fudged Chef Zero
provisioner that was actually Chef Solo under the covers:
https://github.com/mitchellh/vagrant/issues/5072

This meant that all your data bags and the code that depended on them had
to be fooled, stubbed, or coerced if you wanted to test it in Vagrant.
This was generally seen as a bummer and widely regarded as a time sink that
produced little business value. “That’s what you get for using open source
tools, Greg.”

We’d been waiting for Vagrant 1.7.3 over here for some time because it
promised real Chef Zero support: This should have made our hacks and dumb
shell scripts obsolete, allowing me to give my dev’s a Chef role and
nothing else for configuring their development VMs
https://github.com/mitchellh/vagrant/pull/5339

And so it was like a child starting its first day of school and
immediately after dreaming of christmas morning that we watched the above
PR get merged and thought to ourselves “probably won’t be more than 5 or 6
months before that gets bundled into an official release.”

Today I can say that this does not fit in our workflow. The Chef Zero
provisioner is taking anywhere from 20 to 40 minutes on different laptops
just to load our cookbooks into memory from local disk. Then there’s the
speed at which it seems to be configuring the resources in a run list. I
don’t have any real figures to refer to, just anecdotal evidence, so I’ll
resort to hyperbole. Vagrant 1.7.2 and Chef Solo seemed zippy enough. I
could spin up a development vagrant machine in about 5-10 minutes depending
on role and network. 1.7.3 and Chef Zero are reminiscent of a snail race
through knee-deep peanut butter.

$ time vagrant provision


wait for it…

pain…
continue waiting


real 79m59.131s
user 0m4.463s
sys 0m1.405s

Sometimes we use Vagrant to test Chef changes locally before uploading to
a CI server that will build a branch of chef code on EC2 and run some tests
before we mark it as merge-able. I’m telling Chef dev’s in this
organization that Vagrant is supposed to save them time, that local
development is a convenience… and now this. I literally can’t even.

Who’s got a workaround for all these network round trips and molasses
uploads? Do I need more ergs? Should I just rub some Docker on it? Aside
from declaring ‘no joy’ and rolling back to Vagrant 1.7.2 what are my
options here? We’ve tried different folder syncing methods (nfs, rsync)
and found them all wanting. I’m afraid this is a feature, not a bug:

https://github.com/mitchellh/vagrant/issues/5972

Greg Damiani | Senior Kitten Delivery Engineer | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7E

Greg Damiani | Senior System Administrator | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7EB


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com


#3

and/or spool up an instance using the chef_zero provider in test kitchen*

On Fri, Jul 17, 2015 at 8:41 AM, Yoshi Spendiff <
yoshi.spendiff@indochino.com> wrote:

Out of interest do you get similar times if you spool up a local chef-zero
instance and knife upload and/or spool up an instance using the chef_zero
provider, just to make sure it’s not Vagrant doing something strange?

That seems like a long, long time for spooling to me. I used kitchen-ec2
to spool instances on AWS and, granted my cookbook stack is probably
smaller than yours, it takes maybe 15-20 minutes to spool an instance (so
transferring over the internet).

On Fri, Jul 17, 2015 at 6:48 AM, Greg Damiani greg.damiani@buzzfeed.com
wrote:

Previous versions of Vagrant (1.7.2 and prior) used a fudged Chef Zero
provisioner that was actually Chef Solo under the covers:
https://github.com/mitchellh/vagrant/issues/5072

This meant that all your data bags and the code that depended on them had
to be fooled, stubbed, or coerced if you wanted to test it in Vagrant.
This was generally seen as a bummer and widely regarded as a time sink that
produced little business value. “That’s what you get for using open source
tools, Greg.”

We’d been waiting for Vagrant 1.7.3 over here for some time because it
promised real Chef Zero support: This should have made our hacks and dumb
shell scripts obsolete, allowing me to give my dev’s a Chef role and
nothing else for configuring their development VMs
https://github.com/mitchellh/vagrant/pull/5339

And so it was like a child starting its first day of school and
immediately after dreaming of christmas morning that we watched the above
PR get merged and thought to ourselves “probably won’t be more than 5 or 6
months before that gets bundled into an official release.”

Today I can say that this does not fit in our workflow. The Chef Zero
provisioner is taking anywhere from 20 to 40 minutes on different laptops
just to load our cookbooks into memory from local disk. Then there’s the
speed at which it seems to be configuring the resources in a run list. I
don’t have any real figures to refer to, just anecdotal evidence, so I’ll
resort to hyperbole. Vagrant 1.7.2 and Chef Solo seemed zippy enough. I
could spin up a development vagrant machine in about 5-10 minutes depending
on role and network. 1.7.3 and Chef Zero are reminiscent of a snail race
through knee-deep peanut butter.

$ time vagrant provision


wait for it…

pain…
continue waiting


real 79m59.131s
user 0m4.463s
sys 0m1.405s

Sometimes we use Vagrant to test Chef changes locally before uploading to
a CI server that will build a branch of chef code on EC2 and run some tests
before we mark it as merge-able. I’m telling Chef dev’s in this
organization that Vagrant is supposed to save them time, that local
development is a convenience… and now this. I literally can’t even.

Who’s got a workaround for all these network round trips and molasses
uploads? Do I need more ergs? Should I just rub some Docker on it? Aside
from declaring ‘no joy’ and rolling back to Vagrant 1.7.2 what are my
options here? We’ve tried different folder syncing methods (nfs, rsync)
and found them all wanting. I’m afraid this is a feature, not a bug:

https://github.com/mitchellh/vagrant/issues/5972

Greg Damiani | Senior Kitten Delivery Engineer | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7E

Greg Damiani | Senior System Administrator | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7EB


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com


#4

Yep, test kitchen / kitchen-vagrant converge times haven’t changed much (or
at all) with Vagrant 1.7.3 installed.

On Fri, Jul 17, 2015 at 4:44 PM, Yoshi Spendiff <
yoshi.spendiff@indochino.com> wrote:

and/or spool up an instance using the chef_zero provider in test
kitchen
*

On Fri, Jul 17, 2015 at 8:41 AM, Yoshi Spendiff <
yoshi.spendiff@indochino.com> wrote:

Out of interest do you get similar times if you spool up a local
chef-zero instance and knife upload and/or spool up an instance using the
chef_zero provider, just to make sure it’s not Vagrant doing something
strange?

That seems like a long, long time for spooling to me. I used kitchen-ec2
to spool instances on AWS and, granted my cookbook stack is probably
smaller than yours, it takes maybe 15-20 minutes to spool an instance (so
transferring over the internet).

On Fri, Jul 17, 2015 at 6:48 AM, Greg Damiani greg.damiani@buzzfeed.com
wrote:

Previous versions of Vagrant (1.7.2 and prior) used a fudged Chef Zero
provisioner that was actually Chef Solo under the covers:
https://github.com/mitchellh/vagrant/issues/5072

This meant that all your data bags and the code that depended on them
had to be fooled, stubbed, or coerced if you wanted to test it in Vagrant.
This was generally seen as a bummer and widely regarded as a time sink that
produced little business value. “That’s what you get for using open source
tools, Greg.”

We’d been waiting for Vagrant 1.7.3 over here for some time because it
promised real Chef Zero support: This should have made our hacks and dumb
shell scripts obsolete, allowing me to give my dev’s a Chef role and
nothing else for configuring their development VMs
https://github.com/mitchellh/vagrant/pull/5339

And so it was like a child starting its first day of school and
immediately after dreaming of christmas morning that we watched the above
PR get merged and thought to ourselves “probably won’t be more than 5 or 6
months before that gets bundled into an official release.”

Today I can say that this does not fit in our workflow. The Chef Zero
provisioner is taking anywhere from 20 to 40 minutes on different laptops
just to load our cookbooks into memory from local disk. Then there’s the
speed at which it seems to be configuring the resources in a run list. I
don’t have any real figures to refer to, just anecdotal evidence, so I’ll
resort to hyperbole. Vagrant 1.7.2 and Chef Solo seemed zippy enough. I
could spin up a development vagrant machine in about 5-10 minutes depending
on role and network. 1.7.3 and Chef Zero are reminiscent of a snail race
through knee-deep peanut butter.

$ time vagrant provision


wait for it…

pain…
continue waiting


real 79m59.131s
user 0m4.463s
sys 0m1.405s

Sometimes we use Vagrant to test Chef changes locally before uploading
to a CI server that will build a branch of chef code on EC2 and run some
tests before we mark it as merge-able. I’m telling Chef dev’s in this
organization that Vagrant is supposed to save them time, that local
development is a convenience… and now this. I literally can’t even.

Who’s got a workaround for all these network round trips and molasses
uploads? Do I need more ergs? Should I just rub some Docker on it? Aside
from declaring ‘no joy’ and rolling back to Vagrant 1.7.2 what are my
options here? We’ve tried different folder syncing methods (nfs, rsync)
and found them all wanting. I’m afraid this is a feature, not a bug:

https://github.com/mitchellh/vagrant/issues/5972

Greg Damiani | Senior Kitten Delivery Engineer | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7E

Greg Damiani | Senior System Administrator | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7EB


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com

Greg Damiani | Senior System Administrator | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7EB


#5

What version of ChefDK?
-s

On Fri, Jul 17, 2015 at 12:39 PM, Greg Damiani
greg.damiani@buzzfeed.com wrote:

Yep, test kitchen / kitchen-vagrant converge times haven’t changed much (or
at all) with Vagrant 1.7.3 installed.

On Fri, Jul 17, 2015 at 4:44 PM, Yoshi Spendiff
yoshi.spendiff@indochino.com wrote:

and/or spool up an instance using the chef_zero provider in test kitchen*

On Fri, Jul 17, 2015 at 8:41 AM, Yoshi Spendiff
yoshi.spendiff@indochino.com wrote:

Out of interest do you get similar times if you spool up a local
chef-zero instance and knife upload and/or spool up an instance using the
chef_zero provider, just to make sure it’s not Vagrant doing something
strange?

That seems like a long, long time for spooling to me. I used kitchen-ec2
to spool instances on AWS and, granted my cookbook stack is probably smaller
than yours, it takes maybe 15-20 minutes to spool an instance (so
transferring over the internet).

On Fri, Jul 17, 2015 at 6:48 AM, Greg Damiani greg.damiani@buzzfeed.com
wrote:

Previous versions of Vagrant (1.7.2 and prior) used a fudged Chef Zero
provisioner that was actually Chef Solo under the covers:
https://github.com/mitchellh/vagrant/issues/5072

This meant that all your data bags and the code that depended on them
had to be fooled, stubbed, or coerced if you wanted to test it in Vagrant.
This was generally seen as a bummer and widely regarded as a time sink that
produced little business value. “That’s what you get for using open source
tools, Greg.”

We’d been waiting for Vagrant 1.7.3 over here for some time because it
promised real Chef Zero support: This should have made our hacks and dumb
shell scripts obsolete, allowing me to give my dev’s a Chef role and nothing
else for configuring their development VMs
https://github.com/mitchellh/vagrant/pull/5339

And so it was like a child starting its first day of school and
immediately after dreaming of christmas morning that we watched the above PR
get merged and thought to ourselves “probably won’t be more than 5 or 6
months before that gets bundled into an official release.”

Today I can say that this does not fit in our workflow. The Chef Zero
provisioner is taking anywhere from 20 to 40 minutes on different laptops
just to load our cookbooks into memory from local disk. Then there’s the
speed at which it seems to be configuring the resources in a run list. I
don’t have any real figures to refer to, just anecdotal evidence, so I’ll
resort to hyperbole. Vagrant 1.7.2 and Chef Solo seemed zippy enough. I
could spin up a development vagrant machine in about 5-10 minutes depending
on role and network. 1.7.3 and Chef Zero are reminiscent of a snail race
through knee-deep peanut butter.

$ time vagrant provision


wait for it…

pain…
continue waiting


real 79m59.131s
user 0m4.463s
sys 0m1.405s

Sometimes we use Vagrant to test Chef changes locally before uploading
to a CI server that will build a branch of chef code on EC2 and run some
tests before we mark it as merge-able. I’m telling Chef dev’s in this
organization that Vagrant is supposed to save them time, that local
development is a convenience… and now this. I literally can’t even.

Who’s got a workaround for all these network round trips and molasses
uploads? Do I need more ergs? Should I just rub some Docker on it? Aside
from declaring ‘no joy’ and rolling back to Vagrant 1.7.2 what are my
options here? We’ve tried different folder syncing methods (nfs, rsync) and
found them all wanting. I’m afraid this is a feature, not a bug:

https://github.com/mitchellh/vagrant/issues/5972

Greg Damiani | Senior Kitten Delivery Engineer | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7E

Greg Damiani | Senior System Administrator | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7EB


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com

Greg Damiani | Senior System Administrator | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7EB


#6

Mac x86_64 v0.6.2

On Fri, Jul 17, 2015 at 6:12 PM, Sean OMeara someara@chef.io wrote:

What version of ChefDK?
-s

On Fri, Jul 17, 2015 at 12:39 PM, Greg Damiani
greg.damiani@buzzfeed.com wrote:

Yep, test kitchen / kitchen-vagrant converge times haven’t changed much
(or
at all) with Vagrant 1.7.3 installed.

On Fri, Jul 17, 2015 at 4:44 PM, Yoshi Spendiff
yoshi.spendiff@indochino.com wrote:

and/or spool up an instance using the chef_zero provider in test
kitchen*

On Fri, Jul 17, 2015 at 8:41 AM, Yoshi Spendiff
yoshi.spendiff@indochino.com wrote:

Out of interest do you get similar times if you spool up a local
chef-zero instance and knife upload and/or spool up an instance using
the

chef_zero provider, just to make sure it’s not Vagrant doing something
strange?

That seems like a long, long time for spooling to me. I used
kitchen-ec2

to spool instances on AWS and, granted my cookbook stack is probably
smaller

than yours, it takes maybe 15-20 minutes to spool an instance (so
transferring over the internet).

On Fri, Jul 17, 2015 at 6:48 AM, Greg Damiani <
greg.damiani@buzzfeed.com>

wrote:

Previous versions of Vagrant (1.7.2 and prior) used a fudged Chef Zero
provisioner that was actually Chef Solo under the covers:
https://github.com/mitchellh/vagrant/issues/5072

This meant that all your data bags and the code that depended on them
had to be fooled, stubbed, or coerced if you wanted to test it in
Vagrant.

This was generally seen as a bummer and widely regarded as a time
sink that

produced little business value. "That’s what you get for using open
source

tools, Greg."

We’d been waiting for Vagrant 1.7.3 over here for some time because it
promised real Chef Zero support: This should have made our hacks and
dumb

shell scripts obsolete, allowing me to give my dev’s a Chef role and
nothing

else for configuring their development VMs
https://github.com/mitchellh/vagrant/pull/5339

And so it was like a child starting its first day of school and
immediately after dreaming of christmas morning that we watched the
above PR

get merged and thought to ourselves "probably won’t be more than 5 or
6

months before that gets bundled into an official release."

Today I can say that this does not fit in our workflow. The Chef Zero
provisioner is taking anywhere from 20 to 40 minutes on different
laptops

just to load our cookbooks into memory from local disk. Then there’s
the

speed at which it seems to be configuring the resources in a run
list. I

don’t have any real figures to refer to, just anecdotal evidence, so
I’ll

resort to hyperbole. Vagrant 1.7.2 and Chef Solo seemed zippy
enough. I

could spin up a development vagrant machine in about 5-10 minutes
depending

on role and network. 1.7.3 and Chef Zero are reminiscent of a snail
race

through knee-deep peanut butter.

$ time vagrant provision


wait for it…

pain…
continue waiting


real 79m59.131s
user 0m4.463s
sys 0m1.405s

Sometimes we use Vagrant to test Chef changes locally before uploading
to a CI server that will build a branch of chef code on EC2 and run
some

tests before we mark it as merge-able. I’m telling Chef dev’s in this
organization that Vagrant is supposed to save them time, that local
development is a convenience… and now this. I literally can’t even.

Who’s got a workaround for all these network round trips and molasses
uploads? Do I need more ergs? Should I just rub some Docker on it?
Aside

from declaring ‘no joy’ and rolling back to Vagrant 1.7.2 what are my
options here? We’ve tried different folder syncing methods (nfs,
rsync) and

found them all wanting. I’m afraid this is a feature, not a bug:

https://github.com/mitchellh/vagrant/issues/5972

Greg Damiani | Senior Kitten Delivery Engineer | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7E

Greg Damiani | Senior System Administrator | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7EB


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com

Greg Damiani | Senior System Administrator | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7EB

Greg Damiani | Senior System Administrator | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7EB


#7

Sorry, I mean is it significantly slower to run *vagrant up *vs kitchen
converge
for the same developer on the same machine running the same logic
path. I.e if you make a .kitchen.yml in the same directory as your
Vagrantfile and configure kitchen to create an instance in exactly the same
manner are the times similar or is vagrant slower? In the effort to isolate
the problem, being file transfer, Vagrant or chef-zero. For me Vagrant is
actually faster…

There’s a couple of interesting things happening in the pipeline to speed
up test kitchen runs using chef-zero, maybe similar could help you out if
you’re willing to sit out another monkeypatch into merge situation:

https://github.com/test-kitchen/test-kitchen/pull/466
https://github.com/test-kitchen/test-kitchen/pull/620

If you’re using AWS for spooling instances I also found moving to t2.small
from t2.micro to be a speed step up.

On Fri, Jul 17, 2015 at 9:39 AM, Greg Damiani greg.damiani@buzzfeed.com
wrote:

Yep, test kitchen / kitchen-vagrant converge times haven’t changed much
(or at all) with Vagrant 1.7.3 installed.

On Fri, Jul 17, 2015 at 4:44 PM, Yoshi Spendiff <
yoshi.spendiff@indochino.com> wrote:

and/or spool up an instance using the chef_zero provider in test
kitchen
*

On Fri, Jul 17, 2015 at 8:41 AM, Yoshi Spendiff <
yoshi.spendiff@indochino.com> wrote:

Out of interest do you get similar times if you spool up a local
chef-zero instance and knife upload and/or spool up an instance using the
chef_zero provider, just to make sure it’s not Vagrant doing something
strange?

That seems like a long, long time for spooling to me. I used kitchen-ec2
to spool instances on AWS and, granted my cookbook stack is probably
smaller than yours, it takes maybe 15-20 minutes to spool an instance (so
transferring over the internet).

On Fri, Jul 17, 2015 at 6:48 AM, Greg Damiani <greg.damiani@buzzfeed.com

wrote:

Previous versions of Vagrant (1.7.2 and prior) used a fudged Chef Zero
provisioner that was actually Chef Solo under the covers:
https://github.com/mitchellh/vagrant/issues/5072

This meant that all your data bags and the code that depended on them
had to be fooled, stubbed, or coerced if you wanted to test it in Vagrant.
This was generally seen as a bummer and widely regarded as a time sink that
produced little business value. “That’s what you get for using open source
tools, Greg.”

We’d been waiting for Vagrant 1.7.3 over here for some time because it
promised real Chef Zero support: This should have made our hacks and dumb
shell scripts obsolete, allowing me to give my dev’s a Chef role and
nothing else for configuring their development VMs
https://github.com/mitchellh/vagrant/pull/5339

And so it was like a child starting its first day of school and
immediately after dreaming of christmas morning that we watched the above
PR get merged and thought to ourselves “probably won’t be more than 5 or 6
months before that gets bundled into an official release.”

Today I can say that this does not fit in our workflow. The Chef Zero
provisioner is taking anywhere from 20 to 40 minutes on different laptops
just to load our cookbooks into memory from local disk. Then there’s the
speed at which it seems to be configuring the resources in a run list. I
don’t have any real figures to refer to, just anecdotal evidence, so I’ll
resort to hyperbole. Vagrant 1.7.2 and Chef Solo seemed zippy enough. I
could spin up a development vagrant machine in about 5-10 minutes depending
on role and network. 1.7.3 and Chef Zero are reminiscent of a snail race
through knee-deep peanut butter.

$ time vagrant provision


wait for it…

pain…
continue waiting


real 79m59.131s
user 0m4.463s
sys 0m1.405s

Sometimes we use Vagrant to test Chef changes locally before uploading
to a CI server that will build a branch of chef code on EC2 and run some
tests before we mark it as merge-able. I’m telling Chef dev’s in this
organization that Vagrant is supposed to save them time, that local
development is a convenience… and now this. I literally can’t even.

Who’s got a workaround for all these network round trips and molasses
uploads? Do I need more ergs? Should I just rub some Docker on it? Aside
from declaring ‘no joy’ and rolling back to Vagrant 1.7.2 what are my
options here? We’ve tried different folder syncing methods (nfs, rsync)
and found them all wanting. I’m afraid this is a feature, not a bug:

https://github.com/mitchellh/vagrant/issues/5972

Greg Damiani | Senior Kitten Delivery Engineer | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7E

Greg Damiani | Senior System Administrator | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7EB


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com

Greg Damiani | Senior System Administrator | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7EB


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com


#8

I’m seeing the opposite. Same machine, same run_list. “vagrant up” takes
about 80 minutes, “kitchen converge” takes 7-8 minutes.

On Fri, Jul 17, 2015 at 7:06 PM, Yoshi Spendiff <
yoshi.spendiff@indochino.com> wrote:

Sorry, I mean is it significantly slower to run *vagrant up *vs kitchen
converge
for the same developer on the same machine running the same
logic path. I.e if you make a .kitchen.yml in the same directory as your
Vagrantfile and configure kitchen to create an instance in exactly the same
manner are the times similar or is vagrant slower? In the effort to isolate
the problem, being file transfer, Vagrant or chef-zero. For me Vagrant is
actually faster…

There’s a couple of interesting things happening in the pipeline to speed
up test kitchen runs using chef-zero, maybe similar could help you out if
you’re willing to sit out another monkeypatch into merge situation:

https://github.com/test-kitchen/test-kitchen/pull/466
https://github.com/test-kitchen/test-kitchen/pull/620

If you’re using AWS for spooling instances I also found moving to t2.small
from t2.micro to be a speed step up.

On Fri, Jul 17, 2015 at 9:39 AM, Greg Damiani greg.damiani@buzzfeed.com
wrote:

Yep, test kitchen / kitchen-vagrant converge times haven’t changed much
(or at all) with Vagrant 1.7.3 installed.

On Fri, Jul 17, 2015 at 4:44 PM, Yoshi Spendiff <
yoshi.spendiff@indochino.com> wrote:

and/or spool up an instance using the chef_zero provider in test
kitchen
*

On Fri, Jul 17, 2015 at 8:41 AM, Yoshi Spendiff <
yoshi.spendiff@indochino.com> wrote:

Out of interest do you get similar times if you spool up a local
chef-zero instance and knife upload and/or spool up an instance using the
chef_zero provider, just to make sure it’s not Vagrant doing something
strange?

That seems like a long, long time for spooling to me. I used
kitchen-ec2 to spool instances on AWS and, granted my cookbook stack is
probably smaller than yours, it takes maybe 15-20 minutes to spool an
instance (so transferring over the internet).

On Fri, Jul 17, 2015 at 6:48 AM, Greg Damiani <
greg.damiani@buzzfeed.com> wrote:

Previous versions of Vagrant (1.7.2 and prior) used a fudged Chef Zero
provisioner that was actually Chef Solo under the covers:
https://github.com/mitchellh/vagrant/issues/5072

This meant that all your data bags and the code that depended on them
had to be fooled, stubbed, or coerced if you wanted to test it in Vagrant.
This was generally seen as a bummer and widely regarded as a time sink that
produced little business value. “That’s what you get for using open source
tools, Greg.”

We’d been waiting for Vagrant 1.7.3 over here for some time because it
promised real Chef Zero support: This should have made our hacks and dumb
shell scripts obsolete, allowing me to give my dev’s a Chef role and
nothing else for configuring their development VMs
https://github.com/mitchellh/vagrant/pull/5339

And so it was like a child starting its first day of school and
immediately after dreaming of christmas morning that we watched the above
PR get merged and thought to ourselves “probably won’t be more than 5 or 6
months before that gets bundled into an official release.”

Today I can say that this does not fit in our workflow. The Chef Zero
provisioner is taking anywhere from 20 to 40 minutes on different laptops
just to load our cookbooks into memory from local disk. Then there’s the
speed at which it seems to be configuring the resources in a run list. I
don’t have any real figures to refer to, just anecdotal evidence, so I’ll
resort to hyperbole. Vagrant 1.7.2 and Chef Solo seemed zippy enough. I
could spin up a development vagrant machine in about 5-10 minutes depending
on role and network. 1.7.3 and Chef Zero are reminiscent of a snail race
through knee-deep peanut butter.

$ time vagrant provision


wait for it…

pain…
continue waiting


real 79m59.131s
user 0m4.463s
sys 0m1.405s

Sometimes we use Vagrant to test Chef changes locally before uploading
to a CI server that will build a branch of chef code on EC2 and run some
tests before we mark it as merge-able. I’m telling Chef dev’s in this
organization that Vagrant is supposed to save them time, that local
development is a convenience… and now this. I literally can’t even.

Who’s got a workaround for all these network round trips and molasses
uploads? Do I need more ergs? Should I just rub some Docker on it? Aside
from declaring ‘no joy’ and rolling back to Vagrant 1.7.2 what are my
options here? We’ve tried different folder syncing methods (nfs, rsync)
and found them all wanting. I’m afraid this is a feature, not a bug:

https://github.com/mitchellh/vagrant/issues/5972

Greg Damiani | Senior Kitten Delivery Engineer | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7E

Greg Damiani | Senior System Administrator | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7EB


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com

Greg Damiani | Senior System Administrator | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7EB


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com

Greg Damiani | Senior System Administrator | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7EB


#9

Are you using chef_zero or chef_solo as your kitchen provisioner?

On Mon, Jul 20, 2015 at 7:01 AM, Greg Damiani greg.damiani@buzzfeed.com
wrote:

I’m seeing the opposite. Same machine, same run_list. “vagrant up” takes
about 80 minutes, “kitchen converge” takes 7-8 minutes.

On Fri, Jul 17, 2015 at 7:06 PM, Yoshi Spendiff <
yoshi.spendiff@indochino.com> wrote:

Sorry, I mean is it significantly slower to run *vagrant up *vs kitchen
converge
for the same developer on the same machine running the same
logic path. I.e if you make a .kitchen.yml in the same directory as your
Vagrantfile and configure kitchen to create an instance in exactly the same
manner are the times similar or is vagrant slower? In the effort to isolate
the problem, being file transfer, Vagrant or chef-zero. For me Vagrant is
actually faster…

There’s a couple of interesting things happening in the pipeline to speed
up test kitchen runs using chef-zero, maybe similar could help you out if
you’re willing to sit out another monkeypatch into merge situation:

https://github.com/test-kitchen/test-kitchen/pull/466
https://github.com/test-kitchen/test-kitchen/pull/620

If you’re using AWS for spooling instances I also found moving to
t2.small from t2.micro to be a speed step up.

On Fri, Jul 17, 2015 at 9:39 AM, Greg Damiani greg.damiani@buzzfeed.com
wrote:

Yep, test kitchen / kitchen-vagrant converge times haven’t changed much
(or at all) with Vagrant 1.7.3 installed.

On Fri, Jul 17, 2015 at 4:44 PM, Yoshi Spendiff <
yoshi.spendiff@indochino.com> wrote:

and/or spool up an instance using the chef_zero provider in test
kitchen
*

On Fri, Jul 17, 2015 at 8:41 AM, Yoshi Spendiff <
yoshi.spendiff@indochino.com> wrote:

Out of interest do you get similar times if you spool up a local
chef-zero instance and knife upload and/or spool up an instance using the
chef_zero provider, just to make sure it’s not Vagrant doing something
strange?

That seems like a long, long time for spooling to me. I used
kitchen-ec2 to spool instances on AWS and, granted my cookbook stack is
probably smaller than yours, it takes maybe 15-20 minutes to spool an
instance (so transferring over the internet).

On Fri, Jul 17, 2015 at 6:48 AM, Greg Damiani <
greg.damiani@buzzfeed.com> wrote:

Previous versions of Vagrant (1.7.2 and prior) used a fudged Chef
Zero provisioner that was actually Chef Solo under the covers:
https://github.com/mitchellh/vagrant/issues/5072

This meant that all your data bags and the code that depended on them
had to be fooled, stubbed, or coerced if you wanted to test it in Vagrant.
This was generally seen as a bummer and widely regarded as a time sink that
produced little business value. “That’s what you get for using open source
tools, Greg.”

We’d been waiting for Vagrant 1.7.3 over here for some time because
it promised real Chef Zero support: This should have made our hacks and
dumb shell scripts obsolete, allowing me to give my dev’s a Chef role and
nothing else for configuring their development VMs
https://github.com/mitchellh/vagrant/pull/5339

And so it was like a child starting its first day of school and
immediately after dreaming of christmas morning that we watched the above
PR get merged and thought to ourselves “probably won’t be more than 5 or 6
months before that gets bundled into an official release.”

Today I can say that this does not fit in our workflow. The Chef
Zero provisioner is taking anywhere from 20 to 40 minutes on different
laptops just to load our cookbooks into memory from local disk. Then
there’s the speed at which it seems to be configuring the resources in a
run list. I don’t have any real figures to refer to, just anecdotal
evidence, so I’ll resort to hyperbole. Vagrant 1.7.2 and Chef Solo seemed
zippy enough. I could spin up a development vagrant machine in about 5-10
minutes depending on role and network. 1.7.3 and Chef Zero are reminiscent
of a snail race through knee-deep peanut butter.

$ time vagrant provision


wait for it…

pain…
continue waiting


real 79m59.131s
user 0m4.463s
sys 0m1.405s

Sometimes we use Vagrant to test Chef changes locally before
uploading to a CI server that will build a branch of chef code on EC2 and
run some tests before we mark it as merge-able. I’m telling Chef dev’s in
this organization that Vagrant is supposed to save them time, that local
development is a convenience… and now this. I literally can’t even.

Who’s got a workaround for all these network round trips and molasses
uploads? Do I need more ergs? Should I just rub some Docker on it? Aside
from declaring ‘no joy’ and rolling back to Vagrant 1.7.2 what are my
options here? We’ve tried different folder syncing methods (nfs, rsync)
and found them all wanting. I’m afraid this is a feature, not a bug:

https://github.com/mitchellh/vagrant/issues/5972

Greg Damiani | Senior Kitten Delivery Engineer | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7E

Greg Damiani | Senior System Administrator | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7EB


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com

Greg Damiani | Senior System Administrator | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7EB


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com

Greg Damiani | Senior System Administrator | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7EB


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com


#10

One thing I’ve found, for both Vagrant and any kitchen driver, is making sure my chefignore excludes the .kitchen directory. When Berkshelf scans the cookbook directory, it likes to scan everything, and depending on how much is in your .kitchen folder, that could take a bit. Plus, some drivers have open handles to files (especially on Windows) which can lead to errors.

Steve

Steven Murawski
Community Software Development Engineer @ Chef
Microsoft MVP - PowerShell
http://stevenmurawski.com [http://stevenmurawski.com/]
On 7/20/2015 10:01:03 AM, Yoshi Spendiff yoshi.spendiff@indochino.com wrote:
Are you using chef_zero or chef_solo as your kitchen provisioner?

On Mon, Jul 20, 2015 at 7:01 AM, Greg Damiani <greg.damiani@buzzfeed.com [mailto:greg.damiani@buzzfeed.com]> wrote:

I’m seeing the opposite. Same machine, same run_list. “vagrant up” takes about 80 minutes, “kitchen converge” takes 7-8 minutes.

On Fri, Jul 17, 2015 at 7:06 PM, Yoshi Spendiff <yoshi.spendiff@indochino.com [mailto:yoshi.spendiff@indochino.com]> wrote:

Sorry, I mean is it significantly slower to run vagrant up vs kitchen converge for the same developer on the same machine running the same logic path. I.e if you make a .kitchen.yml in the same directory as your Vagrantfile and configure kitchen to create an instance in exactly the same manner are the times similar or is vagrant slower? In the effort to isolate the problem, being file transfer, Vagrant or chef-zero. For me Vagrant is actually faster…

There’s a couple of interesting things happening in the pipeline to speed up test kitchen runs using chef-zero, maybe similar could help you out if you’re willing to sit out another monkeypatch into merge situation:

https://github.com/test-kitchen/test-kitchen/pull/466 [https://github.com/test-kitchen/test-kitchen/pull/466]
https://github.com/test-kitchen/test-kitchen/pull/620 [https://github.com/test-kitchen/test-kitchen/pull/620]

If you’re using AWS for spooling instances I also found moving to t2.small from t2.micro to be a speed step up.

On Fri, Jul 17, 2015 at 9:39 AM, Greg Damiani <greg.damiani@buzzfeed.com [mailto:greg.damiani@buzzfeed.com]> wrote:

Yep, test kitchen / kitchen-vagrant converge times haven’t changed much (or at all) with Vagrant 1.7.3 installed.

On Fri, Jul 17, 2015 at 4:44 PM, Yoshi Spendiff <yoshi.spendiff@indochino.com [mailto:yoshi.spendiff@indochino.com]> wrote:

and/or spool up an instance using the chef_zero provider in test kitchen*

On Fri, Jul 17, 2015 at 8:41 AM, Yoshi Spendiff <yoshi.spendiff@indochino.com [mailto:yoshi.spendiff@indochino.com]> wrote:

Out of interest do you get similar times if you spool up a local chef-zero instance and knife upload and/or spool up an instance using the chef_zero provider, just to make sure it’s not Vagrant doing something strange?

That seems like a long, long time for spooling to me. I used kitchen-ec2 to spool instances on AWS and, granted my cookbook stack is probably smaller than yours, it takes maybe 15-20 minutes to spool an instance (so transferring over the internet).

On Fri, Jul 17, 2015 at 6:48 AM, Greg Damiani <greg.damiani@buzzfeed.com [mailto:greg.damiani@buzzfeed.com]> wrote:

Previous versions of Vagrant (1.7.2 and prior) used a fudged Chef Zero provisioner that was actually Chef Solo under the covers: https://github.com/mitchellh/vagrant/issues/5072 [https://github.com/mitchellh/vagrant/issues/5072]

This meant that all your data bags and the code that depended on them had to be fooled, stubbed, or coerced if you wanted to test it in Vagrant. This was generally seen as a bummer and widely regarded as a time sink that produced little business value. “That’s what you get for using open source tools, Greg.”

We’d been waiting for Vagrant 1.7.3 over here for some time because it promised real Chef Zero support: This should have made our hacks and dumb shell scripts obsolete, allowing me to give my dev’s a Chef role and nothing else for configuring their development VMs https://github.com/mitchellh/vagrant/pull/5339 [https://github.com/mitchellh/vagrant/pull/5339]

And so it was like a child starting its first day of school and immediately after dreaming of christmas morning that we watched the above PR get merged and thought to ourselves “probably won’t be more than 5 or 6 months before that gets bundled into an official release.”

Today I can say that this does not fit in our workflow. The Chef Zero provisioner is taking anywhere from 20 to 40 minutes on different laptops just to load our cookbooks into memory from local disk. Then there’s the speed at which it seems to be configuring the resources in a run list. I don’t have any real figures to refer to, just anecdotal evidence, so I’ll resort to hyperbole. Vagrant 1.7.2 and Chef Solo seemed zippy enough. I could spin up a development vagrant machine in about 5-10 minutes depending on role and network. 1.7.3 and Chef Zero are reminiscent of a snail race through knee-deep peanut butter.

$ time vagrant provision



wait for it…

pain…
continue waiting


real 79m59.131s
user 0m4.463s
sys 0m1.405s

Sometimes we use Vagrant to test Chef changes locally before uploading to a CI server that will build a branch of chef code on EC2 and run some tests before we mark it as merge-able. I’m telling Chef dev’s in this organization that Vagrant is supposed to save them time, that local development is a convenience… and now this. I literally can’t even.

Who’s got a workaround for all these network round trips and molasses uploads? Do I need more ergs? Should I just rub some Docker on it? Aside from declaring ‘no joy’ and rolling back to Vagrant 1.7.2 what are my options here? We’ve tried different folder syncing methods (nfs, rsync) and found them all wanting. I’m afraid this is a feature, not a bug:

https://github.com/mitchellh/vagrant/issues/5972 [https://github.com/mitchellh/vagrant/issues/5972]

Greg Damiani | Senior Kitten Delivery Engineer | @damianigreg
BuzzFeed: The Social News and Entertainment Company
40 Argyll Street, 2nd Floor, London, W1F 7E

Greg Damiani | Senior System Administrator | @damianigreg
BuzzFeed: The Social News and Entertainment Company
40 Argyll Street, 2nd Floor, London, W1F 7EB

Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025 [tel:%2B1%20778%20952%202025]
Email: yoshi.spendiff@indochino.com [mailto:yoshi.spendiff@indochino.com]

Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025 [tel:%2B1%20778%20952%202025]
Email: yoshi.spendiff@indochino.com [mailto:yoshi.spendiff@indochino.com]

Greg Damiani | Senior System Administrator | @damianigreg
BuzzFeed: The Social News and Entertainment Company
40 Argyll Street, 2nd Floor, London, W1F 7EB

Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025 [tel:%2B1%20778%20952%202025]
Email: yoshi.spendiff@indochino.com [mailto:yoshi.spendiff@indochino.com]

Greg Damiani | Senior System Administrator | @damianigreg
BuzzFeed: The Social News and Entertainment Company
40 Argyll Street, 2nd Floor, London, W1F 7EB

Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com [mailto:yoshi.spendiff@indochino.com]


#11

I’ve been experiencing a similar problem on one of my machines (Windows
8.1) in a slightly different setup - using chef-zero standalone to
bootstrap VMs that I’ve created using vagrant.

If I run chef-zero on either a host private network (e.g. using -H10.0.1.1)
or on localhost (default), berkps upload perhaps loads a cookbook every
10-20 seconds. If I instead use the public IP of the host, it takes 1 sec
or less per cookbook.

I’m pretty sure this is a problem with my network setup (e.g. routing
internal traffic wrong), just haven’t had the motivation/ergs to figure out
what, given I have the workaround of using my public IP address. I only
mention this in case it helps suggest what might be going wrong for Greg’s
case. Of course, I wouldn’t be sad if it helped me figure out my issue too!

Regards,
Christine

On Mon, Jul 20, 2015 at 11:07 AM, Steven Murawski <steven.murawski@gmail.com

wrote:

One thing I’ve found, for both Vagrant and any kitchen driver, is making
sure my chefignore excludes the .kitchen directory. When Berkshelf scans
the cookbook directory, it likes to scan everything, and depending on how
much is in your .kitchen folder, that could take a bit. Plus, some drivers
have open handles to files (especially on Windows) which can lead to errors.

Steve

Steven Murawski
Community Software Development Engineer @ Chef
Microsoft MVP - PowerShell
http://stevenmurawski.com

On 7/20/2015 10:01:03 AM, Yoshi Spendiff yoshi.spendiff@indochino.com
wrote:
Are you using chef_zero or chef_solo as your kitchen provisioner?

On Mon, Jul 20, 2015 at 7:01 AM, Greg Damiani greg.damiani@buzzfeed.com
wrote:

I’m seeing the opposite. Same machine, same run_list. "vagrant up"
takes about 80 minutes, “kitchen converge” takes 7-8 minutes.

On Fri, Jul 17, 2015 at 7:06 PM, Yoshi Spendiff <
yoshi.spendiff@indochino.com> wrote:

Sorry, I mean is it significantly slower to run *vagrant up *vs kitchen
converge
for the same developer on the same machine running the same
logic path. I.e if you make a .kitchen.yml in the same directory as your
Vagrantfile and configure kitchen to create an instance in exactly the same
manner are the times similar or is vagrant slower? In the effort to isolate
the problem, being file transfer, Vagrant or chef-zero. For me Vagrant is
actually faster…

There’s a couple of interesting things happening in the pipeline to
speed up test kitchen runs using chef-zero, maybe similar could help you
out if you’re willing to sit out another monkeypatch into merge situation:

https://github.com/test-kitchen/test-kitchen/pull/466
https://github.com/test-kitchen/test-kitchen/pull/620

If you’re using AWS for spooling instances I also found moving to
t2.small from t2.micro to be a speed step up.

On Fri, Jul 17, 2015 at 9:39 AM, Greg Damiani <greg.damiani@buzzfeed.com

wrote:

Yep, test kitchen / kitchen-vagrant converge times haven’t changed much
(or at all) with Vagrant 1.7.3 installed.

On Fri, Jul 17, 2015 at 4:44 PM, Yoshi Spendiff <
yoshi.spendiff@indochino.com> wrote:

and/or spool up an instance using the chef_zero provider in test
kitchen
*

On Fri, Jul 17, 2015 at 8:41 AM, Yoshi Spendiff <
yoshi.spendiff@indochino.com> wrote:

Out of interest do you get similar times if you spool up a local
chef-zero instance and knife upload and/or spool up an instance using the
chef_zero provider, just to make sure it’s not Vagrant doing something
strange?

That seems like a long, long time for spooling to me. I used
kitchen-ec2 to spool instances on AWS and, granted my cookbook stack is
probably smaller than yours, it takes maybe 15-20 minutes to spool an
instance (so transferring over the internet).

On Fri, Jul 17, 2015 at 6:48 AM, Greg Damiani <
greg.damiani@buzzfeed.com> wrote:

Previous versions of Vagrant (1.7.2 and prior) used a fudged Chef
Zero provisioner that was actually Chef Solo under the covers:
https://github.com/mitchellh/vagrant/issues/5072

This meant that all your data bags and the code that depended on
them had to be fooled, stubbed, or coerced if you wanted to test it in
Vagrant. This was generally seen as a bummer and widely regarded as a time
sink that produced little business value. “That’s what you get for using
open source tools, Greg.”

We’d been waiting for Vagrant 1.7.3 over here for some time because
it promised real Chef Zero support: This should have made our hacks and
dumb shell scripts obsolete, allowing me to give my dev’s a Chef role and
nothing else for configuring their development VMs
https://github.com/mitchellh/vagrant/pull/5339

And so it was like a child starting its first day of school and
immediately after dreaming of christmas morning that we watched the above
PR get merged and thought to ourselves “probably won’t be more than 5 or 6
months before that gets bundled into an official release.”

Today I can say that this does not fit in our workflow. The Chef
Zero provisioner is taking anywhere from 20 to 40 minutes on different
laptops just to load our cookbooks into memory from local disk. Then
there’s the speed at which it seems to be configuring the resources in a
run list. I don’t have any real figures to refer to, just anecdotal
evidence, so I’ll resort to hyperbole. Vagrant 1.7.2 and Chef Solo seemed
zippy enough. I could spin up a development vagrant machine in about 5-10
minutes depending on role and network. 1.7.3 and Chef Zero are reminiscent
of a snail race through knee-deep peanut butter.

$ time vagrant provision


wait for it…

pain…
continue waiting


real 79m59.131s
user 0m4.463s
sys 0m1.405s

Sometimes we use Vagrant to test Chef changes locally before
uploading to a CI server that will build a branch of chef code on EC2 and
run some tests before we mark it as merge-able. I’m telling Chef dev’s in
this organization that Vagrant is supposed to save them time, that local
development is a convenience… and now this. I literally can’t even.

Who’s got a workaround for all these network round trips and
molasses uploads? Do I need more ergs? Should I just rub some Docker on
it? Aside from declaring ‘no joy’ and rolling back to Vagrant 1.7.2 what
are my options here? We’ve tried different folder syncing methods (nfs,
rsync) and found them all wanting. I’m afraid this is a feature, not a bug:

https://github.com/mitchellh/vagrant/issues/5972

Greg Damiani | Senior Kitten Delivery Engineer | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7E

Greg Damiani | Senior System Administrator | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7EB


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com

Greg Damiani | Senior System Administrator | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7EB


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com

Greg Damiani | Senior System Administrator | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7EB


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com


ThirdWave Insights, LLC I (512) 971-8727 <%28512%29%20656-7724> I
www.ThirdWaveInsights.com I P.O. Box 500134 I Austin, TX 78750


#12

I was experiencing extremely slow chef runs as well with vagrant 1.7.4 and
the chef_zero provisioner.

It turned out to be vagrant-cachier in my case, which caches
"/var/chef/cache" [0] where chef_zero also stores it’s cookbooks
internally. Having vagrant-cachier caching these file by file was super
slow, even with only a minimum set of cookbooks…

Actually I find it quite useful to cache “/var/chef/cache”.

Is there an option in chef_zero config to store the cookbooks anywhere
outside the file_cache_path?

Cheers, Torben

[0] http://fgrehm.viewdocs.io/vagrant-cachier/buckets/chef/

On Wed, Jul 22, 2015 at 3:29 PM, Christine Draper <
christine_draper@thirdwaveinsights.com> wrote:

I’ve been experiencing a similar problem on one of my machines (Windows
8.1) in a slightly different setup - using chef-zero standalone to
bootstrap VMs that I’ve created using vagrant.

If I run chef-zero on either a host private network (e.g. using
-H10.0.1.1) or on localhost (default), berkps upload perhaps loads a
cookbook every 10-20 seconds. If I instead use the public IP of the host,
it takes 1 sec or less per cookbook.

I’m pretty sure this is a problem with my network setup (e.g. routing
internal traffic wrong), just haven’t had the motivation/ergs to figure out
what, given I have the workaround of using my public IP address. I only
mention this in case it helps suggest what might be going wrong for Greg’s
case. Of course, I wouldn’t be sad if it helped me figure out my issue too!

Regards,
Christine

On Mon, Jul 20, 2015 at 11:07 AM, Steven Murawski <
steven.murawski@gmail.com> wrote:

One thing I’ve found, for both Vagrant and any kitchen driver, is making
sure my chefignore excludes the .kitchen directory. When Berkshelf scans
the cookbook directory, it likes to scan everything, and depending on how
much is in your .kitchen folder, that could take a bit. Plus, some drivers
have open handles to files (especially on Windows) which can lead to errors.

Steve

Steven Murawski
Community Software Development Engineer @ Chef
Microsoft MVP - PowerShell
http://stevenmurawski.com

On 7/20/2015 10:01:03 AM, Yoshi Spendiff yoshi.spendiff@indochino.com
wrote:
Are you using chef_zero or chef_solo as your kitchen provisioner?

On Mon, Jul 20, 2015 at 7:01 AM, Greg Damiani greg.damiani@buzzfeed.com
wrote:

I’m seeing the opposite. Same machine, same run_list. "vagrant up"
takes about 80 minutes, “kitchen converge” takes 7-8 minutes.

On Fri, Jul 17, 2015 at 7:06 PM, Yoshi Spendiff <
yoshi.spendiff@indochino.com> wrote:

Sorry, I mean is it significantly slower to run *vagrant up *vs kitchen
converge
for the same developer on the same machine running the same
logic path. I.e if you make a .kitchen.yml in the same directory as your
Vagrantfile and configure kitchen to create an instance in exactly the same
manner are the times similar or is vagrant slower? In the effort to isolate
the problem, being file transfer, Vagrant or chef-zero. For me Vagrant is
actually faster…

There’s a couple of interesting things happening in the pipeline to
speed up test kitchen runs using chef-zero, maybe similar could help you
out if you’re willing to sit out another monkeypatch into merge situation:

https://github.com/test-kitchen/test-kitchen/pull/466
https://github.com/test-kitchen/test-kitchen/pull/620

If you’re using AWS for spooling instances I also found moving to
t2.small from t2.micro to be a speed step up.

On Fri, Jul 17, 2015 at 9:39 AM, Greg Damiani <
greg.damiani@buzzfeed.com> wrote:

Yep, test kitchen / kitchen-vagrant converge times haven’t changed
much (or at all) with Vagrant 1.7.3 installed.

On Fri, Jul 17, 2015 at 4:44 PM, Yoshi Spendiff <
yoshi.spendiff@indochino.com> wrote:

and/or spool up an instance using the chef_zero provider in test
kitchen
*

On Fri, Jul 17, 2015 at 8:41 AM, Yoshi Spendiff <
yoshi.spendiff@indochino.com> wrote:

Out of interest do you get similar times if you spool up a local
chef-zero instance and knife upload and/or spool up an instance using the
chef_zero provider, just to make sure it’s not Vagrant doing something
strange?

That seems like a long, long time for spooling to me. I used
kitchen-ec2 to spool instances on AWS and, granted my cookbook stack is
probably smaller than yours, it takes maybe 15-20 minutes to spool an
instance (so transferring over the internet).

On Fri, Jul 17, 2015 at 6:48 AM, Greg Damiani <
greg.damiani@buzzfeed.com> wrote:

Previous versions of Vagrant (1.7.2 and prior) used a fudged Chef
Zero provisioner that was actually Chef Solo under the covers:
https://github.com/mitchellh/vagrant/issues/5072

This meant that all your data bags and the code that depended on
them had to be fooled, stubbed, or coerced if you wanted to test it in
Vagrant. This was generally seen as a bummer and widely regarded as a time
sink that produced little business value. “That’s what you get for using
open source tools, Greg.”

We’d been waiting for Vagrant 1.7.3 over here for some time because
it promised real Chef Zero support: This should have made our hacks and
dumb shell scripts obsolete, allowing me to give my dev’s a Chef role and
nothing else for configuring their development VMs
https://github.com/mitchellh/vagrant/pull/5339

And so it was like a child starting its first day of school and
immediately after dreaming of christmas morning that we watched the above
PR get merged and thought to ourselves “probably won’t be more than 5 or 6
months before that gets bundled into an official release.”

Today I can say that this does not fit in our workflow. The Chef
Zero provisioner is taking anywhere from 20 to 40 minutes on different
laptops just to load our cookbooks into memory from local disk. Then
there’s the speed at which it seems to be configuring the resources in a
run list. I don’t have any real figures to refer to, just anecdotal
evidence, so I’ll resort to hyperbole. Vagrant 1.7.2 and Chef Solo seemed
zippy enough. I could spin up a development vagrant machine in about 5-10
minutes depending on role and network. 1.7.3 and Chef Zero are reminiscent
of a snail race through knee-deep peanut butter.

$ time vagrant provision


wait for it…

pain…
continue waiting


real 79m59.131s
user 0m4.463s
sys 0m1.405s

Sometimes we use Vagrant to test Chef changes locally before
uploading to a CI server that will build a branch of chef code on EC2 and
run some tests before we mark it as merge-able. I’m telling Chef dev’s in
this organization that Vagrant is supposed to save them time, that local
development is a convenience… and now this. I literally can’t even.

Who’s got a workaround for all these network round trips and
molasses uploads? Do I need more ergs? Should I just rub some Docker on
it? Aside from declaring ‘no joy’ and rolling back to Vagrant 1.7.2 what
are my options here? We’ve tried different folder syncing methods (nfs,
rsync) and found them all wanting. I’m afraid this is a feature, not a bug:

https://github.com/mitchellh/vagrant/issues/5972

Greg Damiani | Senior Kitten Delivery Engineer | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7E

Greg Damiani | Senior System Administrator | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7EB


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com

Greg Damiani | Senior System Administrator | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7EB


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com

Greg Damiani | Senior System Administrator | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7EB


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com


ThirdWave Insights, LLC I (512) 971-8727 <%28512%29%20656-7724> I
www.ThirdWaveInsights.com I P.O. Box 500134 I Austin, TX 78750


#13

Ooops I was wrong: it was vagrant-cachier which made the situation actually
a bit better. With cachier disabled it happens now every run, not only the
first one.

Thought that these messages came from cachier, but it was chef-zero indeed?


==> sample-app: resolving cookbooks for run list: [“sample-app”]
==> sample-app: [2015-08-09T22:36:17+00:00] INFO: Loading cookbooks
[sample-app@0.2.1, apache2@3.0.1, apt@2.7.0, iptables@1.0.0, logrotate@1.9.1
]
==> sample-app: Synchronizing cookbooks
==> sample-app: [2015-08-09T22:36:21+00:00] INFO: Storing updated
cookbooks/sample-app/.rubocop.yml in the cache.
==> sample-app: [2015-08-09T22:36:21+00:00] INFO: Storing updated
cookbooks/sample-app/Berksfile in the cache.
==> sample-app: [2015-08-09T22:36:21+00:00] INFO: Storing updated
cookbooks/sample-app/Berksfile.lock in the cache.
==> sample-app: [2015-08-09T22:36:21+00:00] INFO: Storing updated
cookbooks/sample-app/CHANGELOG.md in the cache.
==> sample-app: [2015-08-09T22:36:21+00:00] INFO: Storing updated
cookbooks/sample-app/.kitchen.lxc.yml in the cache.
==> sample-app: [2015-08-09T22:36:21+00:00] INFO: Storing updated
cookbooks/sample-app/attributes/default.rb in the cache.
==> sample-app: [2015-08-09T22:36:22+00:00] INFO: Storing updated
cookbooks/sample-app/templates/default/sample.html.erb in the cache.
==> sample-app: [2015-08-09T22:36:22+00:00] INFO: Storing updated
cookbooks/sample-app/recipes/default.rb in the cache.

With all the dependent cookbooks the “Synchronizing cookbooks” takes quite
an amount of extra time (~4 minutes for apt, apache2, iptables, logrotate)

Would be great if there were a way to disable “Synchronizing cookbooks” for
the vagrant case, because the cookbooks are mounted via synced_folder
anyway, so there is no need for caching it internally

Any ideas?

Cheers, Torben

On Mon, Aug 10, 2015 at 12:12 AM, Torben Knerr mail@tknerr.de wrote:

I was experiencing extremely slow chef runs as well with vagrant 1.7.4 and
the chef_zero provisioner.

It turned out to be vagrant-cachier in my case, which caches
"/var/chef/cache" [0] where chef_zero also stores it’s cookbooks
internally. Having vagrant-cachier caching these file by file was super
slow, even with only a minimum set of cookbooks…

Actually I find it quite useful to cache “/var/chef/cache”.

Is there an option in chef_zero config to store the cookbooks anywhere
outside the file_cache_path?

Cheers, Torben

[0] http://fgrehm.viewdocs.io/vagrant-cachier/buckets/chef/

On Wed, Jul 22, 2015 at 3:29 PM, Christine Draper <
christine_draper@thirdwaveinsights.com> wrote:

I’ve been experiencing a similar problem on one of my machines (Windows
8.1) in a slightly different setup - using chef-zero standalone to
bootstrap VMs that I’ve created using vagrant.

If I run chef-zero on either a host private network (e.g. using
-H10.0.1.1) or on localhost (default), berkps upload perhaps loads a
cookbook every 10-20 seconds. If I instead use the public IP of the host,
it takes 1 sec or less per cookbook.

I’m pretty sure this is a problem with my network setup (e.g. routing
internal traffic wrong), just haven’t had the motivation/ergs to figure out
what, given I have the workaround of using my public IP address. I only
mention this in case it helps suggest what might be going wrong for Greg’s
case. Of course, I wouldn’t be sad if it helped me figure out my issue too!

Regards,
Christine

On Mon, Jul 20, 2015 at 11:07 AM, Steven Murawski <
steven.murawski@gmail.com> wrote:

One thing I’ve found, for both Vagrant and any kitchen driver, is making
sure my chefignore excludes the .kitchen directory. When Berkshelf scans
the cookbook directory, it likes to scan everything, and depending on how
much is in your .kitchen folder, that could take a bit. Plus, some drivers
have open handles to files (especially on Windows) which can lead to errors.

Steve

Steven Murawski
Community Software Development Engineer @ Chef
Microsoft MVP - PowerShell
http://stevenmurawski.com

On 7/20/2015 10:01:03 AM, Yoshi Spendiff yoshi.spendiff@indochino.com
wrote:
Are you using chef_zero or chef_solo as your kitchen provisioner?

On Mon, Jul 20, 2015 at 7:01 AM, Greg Damiani <greg.damiani@buzzfeed.com

wrote:

I’m seeing the opposite. Same machine, same run_list. "vagrant up"
takes about 80 minutes, “kitchen converge” takes 7-8 minutes.

On Fri, Jul 17, 2015 at 7:06 PM, Yoshi Spendiff <
yoshi.spendiff@indochino.com> wrote:

Sorry, I mean is it significantly slower to run *vagrant up *vs kitchen
converge
for the same developer on the same machine running the same
logic path. I.e if you make a .kitchen.yml in the same directory as your
Vagrantfile and configure kitchen to create an instance in exactly the same
manner are the times similar or is vagrant slower? In the effort to isolate
the problem, being file transfer, Vagrant or chef-zero. For me Vagrant is
actually faster…

There’s a couple of interesting things happening in the pipeline to
speed up test kitchen runs using chef-zero, maybe similar could help you
out if you’re willing to sit out another monkeypatch into merge situation:

https://github.com/test-kitchen/test-kitchen/pull/466
https://github.com/test-kitchen/test-kitchen/pull/620

If you’re using AWS for spooling instances I also found moving to
t2.small from t2.micro to be a speed step up.

On Fri, Jul 17, 2015 at 9:39 AM, Greg Damiani <
greg.damiani@buzzfeed.com> wrote:

Yep, test kitchen / kitchen-vagrant converge times haven’t changed
much (or at all) with Vagrant 1.7.3 installed.

On Fri, Jul 17, 2015 at 4:44 PM, Yoshi Spendiff <
yoshi.spendiff@indochino.com> wrote:

and/or spool up an instance using the chef_zero provider in test
kitchen
*

On Fri, Jul 17, 2015 at 8:41 AM, Yoshi Spendiff <
yoshi.spendiff@indochino.com> wrote:

Out of interest do you get similar times if you spool up a local
chef-zero instance and knife upload and/or spool up an instance using the
chef_zero provider, just to make sure it’s not Vagrant doing something
strange?

That seems like a long, long time for spooling to me. I used
kitchen-ec2 to spool instances on AWS and, granted my cookbook stack is
probably smaller than yours, it takes maybe 15-20 minutes to spool an
instance (so transferring over the internet).

On Fri, Jul 17, 2015 at 6:48 AM, Greg Damiani <
greg.damiani@buzzfeed.com> wrote:

Previous versions of Vagrant (1.7.2 and prior) used a fudged Chef
Zero provisioner that was actually Chef Solo under the covers:
https://github.com/mitchellh/vagrant/issues/5072

This meant that all your data bags and the code that depended on
them had to be fooled, stubbed, or coerced if you wanted to test it in
Vagrant. This was generally seen as a bummer and widely regarded as a time
sink that produced little business value. “That’s what you get for using
open source tools, Greg.”

We’d been waiting for Vagrant 1.7.3 over here for some time
because it promised real Chef Zero support: This should have made our
hacks and dumb shell scripts obsolete, allowing me to give my dev’s a Chef
role and nothing else for configuring their development VMs
https://github.com/mitchellh/vagrant/pull/5339

And so it was like a child starting its first day of school and
immediately after dreaming of christmas morning that we watched the above
PR get merged and thought to ourselves “probably won’t be more than 5 or 6
months before that gets bundled into an official release.”

Today I can say that this does not fit in our workflow. The Chef
Zero provisioner is taking anywhere from 20 to 40 minutes on different
laptops just to load our cookbooks into memory from local disk. Then
there’s the speed at which it seems to be configuring the resources in a
run list. I don’t have any real figures to refer to, just anecdotal
evidence, so I’ll resort to hyperbole. Vagrant 1.7.2 and Chef Solo seemed
zippy enough. I could spin up a development vagrant machine in about 5-10
minutes depending on role and network. 1.7.3 and Chef Zero are reminiscent
of a snail race through knee-deep peanut butter.

$ time vagrant provision


wait for it…

pain…
continue waiting


real 79m59.131s
user 0m4.463s
sys 0m1.405s

Sometimes we use Vagrant to test Chef changes locally before
uploading to a CI server that will build a branch of chef code on EC2 and
run some tests before we mark it as merge-able. I’m telling Chef dev’s in
this organization that Vagrant is supposed to save them time, that local
development is a convenience… and now this. I literally can’t even.

Who’s got a workaround for all these network round trips and
molasses uploads? Do I need more ergs? Should I just rub some Docker on
it? Aside from declaring ‘no joy’ and rolling back to Vagrant 1.7.2 what
are my options here? We’ve tried different folder syncing methods (nfs,
rsync) and found them all wanting. I’m afraid this is a feature, not a bug:

https://github.com/mitchellh/vagrant/issues/5972

Greg Damiani | Senior Kitten Delivery Engineer | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7E

Greg Damiani | Senior System Administrator | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7EB


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com

Greg Damiani | Senior System Administrator | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7EB


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com

Greg Damiani | Senior System Administrator | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7EB


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com


ThirdWave Insights, LLC I (512) 971-8727 <%28512%29%20656-7724> I
www.ThirdWaveInsights.com I P.O. Box 500134 I Austin, TX 78750


#14

I don’t use Vagrant, but I have noticed that "synchronizing cookbooks"
takes a while lately…

What version of chef-client did “synchronizing cookbooks” start taking
forever?

-s

On Sun, Aug 9, 2015 at 6:43 PM, Torben Knerr mail@tknerr.de wrote:

Ooops I was wrong: it was vagrant-cachier which made the situation
actually a bit better. With cachier disabled it happens now every run, not
only the first one.

Thought that these messages came from cachier, but it was chef-zero indeed?


==> sample-app: resolving cookbooks for run list: [“sample-app”]
==> sample-app: [2015-08-09T22:36:17+00:00] INFO: Loading cookbooks
[sample-app@0.2.1, apache2@3.0.1, apt@2.7.0, iptables@1.0.0,
logrotate@1.9.1]
==> sample-app: Synchronizing cookbooks
==> sample-app: [2015-08-09T22:36:21+00:00] INFO: Storing updated
cookbooks/sample-app/.rubocop.yml in the cache.
==> sample-app: [2015-08-09T22:36:21+00:00] INFO: Storing updated
cookbooks/sample-app/Berksfile in the cache.
==> sample-app: [2015-08-09T22:36:21+00:00] INFO: Storing updated
cookbooks/sample-app/Berksfile.lock in the cache.
==> sample-app: [2015-08-09T22:36:21+00:00] INFO: Storing updated
cookbooks/sample-app/CHANGELOG.md in the cache.
==> sample-app: [2015-08-09T22:36:21+00:00] INFO: Storing updated
cookbooks/sample-app/.kitchen.lxc.yml in the cache.
==> sample-app: [2015-08-09T22:36:21+00:00] INFO: Storing updated
cookbooks/sample-app/attributes/default.rb in the cache.
==> sample-app: [2015-08-09T22:36:22+00:00] INFO: Storing updated
cookbooks/sample-app/templates/default/sample.html.erb in the cache.
==> sample-app: [2015-08-09T22:36:22+00:00] INFO: Storing updated
cookbooks/sample-app/recipes/default.rb in the cache.

With all the dependent cookbooks the “Synchronizing cookbooks” takes quite
an amount of extra time (~4 minutes for apt, apache2, iptables, logrotate)

Would be great if there were a way to disable "Synchronizing cookbooks"
for the vagrant case, because the cookbooks are mounted via synced_folder
anyway, so there is no need for caching it internally

Any ideas?

Cheers, Torben

On Mon, Aug 10, 2015 at 12:12 AM, Torben Knerr mail@tknerr.de wrote:

I was experiencing extremely slow chef runs as well with vagrant 1.7.4
and the chef_zero provisioner.

It turned out to be vagrant-cachier in my case, which caches
"/var/chef/cache" [0] where chef_zero also stores it’s cookbooks
internally. Having vagrant-cachier caching these file by file was super
slow, even with only a minimum set of cookbooks…

Actually I find it quite useful to cache “/var/chef/cache”.

Is there an option in chef_zero config to store the cookbooks anywhere
outside the file_cache_path?

Cheers, Torben

[0] http://fgrehm.viewdocs.io/vagrant-cachier/buckets/chef/

On Wed, Jul 22, 2015 at 3:29 PM, Christine Draper <
christine_draper@thirdwaveinsights.com> wrote:

I’ve been experiencing a similar problem on one of my machines (Windows
8.1) in a slightly different setup - using chef-zero standalone to
bootstrap VMs that I’ve created using vagrant.

If I run chef-zero on either a host private network (e.g. using
-H10.0.1.1) or on localhost (default), berkps upload perhaps loads a
cookbook every 10-20 seconds. If I instead use the public IP of the host,
it takes 1 sec or less per cookbook.

I’m pretty sure this is a problem with my network setup (e.g. routing
internal traffic wrong), just haven’t had the motivation/ergs to figure out
what, given I have the workaround of using my public IP address. I only
mention this in case it helps suggest what might be going wrong for Greg’s
case. Of course, I wouldn’t be sad if it helped me figure out my issue too!

Regards,
Christine

On Mon, Jul 20, 2015 at 11:07 AM, Steven Murawski <
steven.murawski@gmail.com> wrote:

One thing I’ve found, for both Vagrant and any kitchen driver, is
making sure my chefignore excludes the .kitchen directory. When Berkshelf
scans the cookbook directory, it likes to scan everything, and depending on
how much is in your .kitchen folder, that could take a bit. Plus, some
drivers have open handles to files (especially on Windows) which can lead
to errors.

Steve

Steven Murawski
Community Software Development Engineer @ Chef
Microsoft MVP - PowerShell
http://stevenmurawski.com

On 7/20/2015 10:01:03 AM, Yoshi Spendiff yoshi.spendiff@indochino.com
wrote:
Are you using chef_zero or chef_solo as your kitchen provisioner?

On Mon, Jul 20, 2015 at 7:01 AM, Greg Damiani <
greg.damiani@buzzfeed.com> wrote:

I’m seeing the opposite. Same machine, same run_list. "vagrant up"
takes about 80 minutes, “kitchen converge” takes 7-8 minutes.

On Fri, Jul 17, 2015 at 7:06 PM, Yoshi Spendiff <
yoshi.spendiff@indochino.com> wrote:

Sorry, I mean is it significantly slower to run *vagrant up *vs kitchen
converge
for the same developer on the same machine running the
same logic path. I.e if you make a .kitchen.yml in the same directory as
your Vagrantfile and configure kitchen to create an instance in exactly the
same manner are the times similar or is vagrant slower? In the effort to
isolate the problem, being file transfer, Vagrant or chef-zero. For me
Vagrant is actually faster…

There’s a couple of interesting things happening in the pipeline to
speed up test kitchen runs using chef-zero, maybe similar could help you
out if you’re willing to sit out another monkeypatch into merge situation:

https://github.com/test-kitchen/test-kitchen/pull/466
https://github.com/test-kitchen/test-kitchen/pull/620

If you’re using AWS for spooling instances I also found moving to
t2.small from t2.micro to be a speed step up.

On Fri, Jul 17, 2015 at 9:39 AM, Greg Damiani <
greg.damiani@buzzfeed.com> wrote:

Yep, test kitchen / kitchen-vagrant converge times haven’t changed
much (or at all) with Vagrant 1.7.3 installed.

On Fri, Jul 17, 2015 at 4:44 PM, Yoshi Spendiff <
yoshi.spendiff@indochino.com> wrote:

and/or spool up an instance using the chef_zero provider in test
kitchen
*

On Fri, Jul 17, 2015 at 8:41 AM, Yoshi Spendiff <
yoshi.spendiff@indochino.com> wrote:

Out of interest do you get similar times if you spool up a local
chef-zero instance and knife upload and/or spool up an instance using the
chef_zero provider, just to make sure it’s not Vagrant doing something
strange?

That seems like a long, long time for spooling to me. I used
kitchen-ec2 to spool instances on AWS and, granted my cookbook stack is
probably smaller than yours, it takes maybe 15-20 minutes to spool an
instance (so transferring over the internet).

On Fri, Jul 17, 2015 at 6:48 AM, Greg Damiani <
greg.damiani@buzzfeed.com> wrote:

Previous versions of Vagrant (1.7.2 and prior) used a fudged Chef
Zero provisioner that was actually Chef Solo under the covers:
https://github.com/mitchellh/vagrant/issues/5072

This meant that all your data bags and the code that depended on
them had to be fooled, stubbed, or coerced if you wanted to test it in
Vagrant. This was generally seen as a bummer and widely regarded as a time
sink that produced little business value. “That’s what you get for using
open source tools, Greg.”

We’d been waiting for Vagrant 1.7.3 over here for some time
because it promised real Chef Zero support: This should have made our
hacks and dumb shell scripts obsolete, allowing me to give my dev’s a Chef
role and nothing else for configuring their development VMs
https://github.com/mitchellh/vagrant/pull/5339

And so it was like a child starting its first day of school and
immediately after dreaming of christmas morning that we watched the above
PR get merged and thought to ourselves “probably won’t be more than 5 or 6
months before that gets bundled into an official release.”

Today I can say that this does not fit in our workflow. The Chef
Zero provisioner is taking anywhere from 20 to 40 minutes on different
laptops just to load our cookbooks into memory from local disk. Then
there’s the speed at which it seems to be configuring the resources in a
run list. I don’t have any real figures to refer to, just anecdotal
evidence, so I’ll resort to hyperbole. Vagrant 1.7.2 and Chef Solo seemed
zippy enough. I could spin up a development vagrant machine in about 5-10
minutes depending on role and network. 1.7.3 and Chef Zero are reminiscent
of a snail race through knee-deep peanut butter.

$ time vagrant provision


wait for it…

pain…
continue waiting


real 79m59.131s
user 0m4.463s
sys 0m1.405s

Sometimes we use Vagrant to test Chef changes locally before
uploading to a CI server that will build a branch of chef code on EC2 and
run some tests before we mark it as merge-able. I’m telling Chef dev’s in
this organization that Vagrant is supposed to save them time, that local
development is a convenience… and now this. I literally can’t even.

Who’s got a workaround for all these network round trips and
molasses uploads? Do I need more ergs? Should I just rub some Docker on
it? Aside from declaring ‘no joy’ and rolling back to Vagrant 1.7.2 what
are my options here? We’ve tried different folder syncing methods (nfs,
rsync) and found them all wanting. I’m afraid this is a feature, not a bug:

https://github.com/mitchellh/vagrant/issues/5972

Greg Damiani | Senior Kitten Delivery Engineer | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7E

Greg Damiani | Senior System Administrator | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7EB


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com

Greg Damiani | Senior System Administrator | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7EB


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com

Greg Damiani | Senior System Administrator | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7EB


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com


ThirdWave Insights, LLC I (512) 971-8727 <%28512%29%20656-7724> I
www.ThirdWaveInsights.com I P.O. Box 500134 I Austin, TX 78750


#15

12.3.0 that was.

So it’s chef-client filling it’s cache rather than the in-memory chef-zero?

Cheers, Torben
Am 10.08.2015 05:19 schrieb “Sean OMeara” someara@chef.io:

I don’t use Vagrant, but I have noticed that "synchronizing cookbooks"
takes a while lately…

What version of chef-client did “synchronizing cookbooks” start taking
forever?

-s

On Sun, Aug 9, 2015 at 6:43 PM, Torben Knerr mail@tknerr.de wrote:

Ooops I was wrong: it was vagrant-cachier which made the situation
actually a bit better. With cachier disabled it happens now every run, not
only the first one.

Thought that these messages came from cachier, but it was chef-zero
indeed?


==> sample-app: resolving cookbooks for run list: [“sample-app”]
==> sample-app: [2015-08-09T22:36:17+00:00] INFO: Loading cookbooks
[sample-app@0.2.1, apache2@3.0.1, apt@2.7.0, iptables@1.0.0,
logrotate@1.9.1]
==> sample-app: Synchronizing cookbooks
==> sample-app: [2015-08-09T22:36:21+00:00] INFO: Storing updated
cookbooks/sample-app/.rubocop.yml in the cache.
==> sample-app: [2015-08-09T22:36:21+00:00] INFO: Storing updated
cookbooks/sample-app/Berksfile in the cache.
==> sample-app: [2015-08-09T22:36:21+00:00] INFO: Storing updated
cookbooks/sample-app/Berksfile.lock in the cache.
==> sample-app: [2015-08-09T22:36:21+00:00] INFO: Storing updated
cookbooks/sample-app/CHANGELOG.md in the cache.
==> sample-app: [2015-08-09T22:36:21+00:00] INFO: Storing updated
cookbooks/sample-app/.kitchen.lxc.yml in the cache.
==> sample-app: [2015-08-09T22:36:21+00:00] INFO: Storing updated
cookbooks/sample-app/attributes/default.rb in the cache.
==> sample-app: [2015-08-09T22:36:22+00:00] INFO: Storing updated
cookbooks/sample-app/templates/default/sample.html.erb in the cache.
==> sample-app: [2015-08-09T22:36:22+00:00] INFO: Storing updated
cookbooks/sample-app/recipes/default.rb in the cache.

With all the dependent cookbooks the “Synchronizing cookbooks” takes
quite an amount of extra time (~4 minutes for apt, apache2, iptables,
logrotate)

Would be great if there were a way to disable "Synchronizing cookbooks"
for the vagrant case, because the cookbooks are mounted via synced_folder
anyway, so there is no need for caching it internally

Any ideas?

Cheers, Torben

On Mon, Aug 10, 2015 at 12:12 AM, Torben Knerr mail@tknerr.de wrote:

I was experiencing extremely slow chef runs as well with vagrant 1.7.4
and the chef_zero provisioner.

It turned out to be vagrant-cachier in my case, which caches
"/var/chef/cache" [0] where chef_zero also stores it’s cookbooks
internally. Having vagrant-cachier caching these file by file was super
slow, even with only a minimum set of cookbooks…

Actually I find it quite useful to cache “/var/chef/cache”.

Is there an option in chef_zero config to store the cookbooks anywhere
outside the file_cache_path?

Cheers, Torben

[0] http://fgrehm.viewdocs.io/vagrant-cachier/buckets/chef/

On Wed, Jul 22, 2015 at 3:29 PM, Christine Draper <
christine_draper@thirdwaveinsights.com> wrote:

I’ve been experiencing a similar problem on one of my machines (Windows
8.1) in a slightly different setup - using chef-zero standalone to
bootstrap VMs that I’ve created using vagrant.

If I run chef-zero on either a host private network (e.g. using
-H10.0.1.1) or on localhost (default), berkps upload perhaps loads a
cookbook every 10-20 seconds. If I instead use the public IP of the host,
it takes 1 sec or less per cookbook.

I’m pretty sure this is a problem with my network setup (e.g. routing
internal traffic wrong), just haven’t had the motivation/ergs to figure out
what, given I have the workaround of using my public IP address. I only
mention this in case it helps suggest what might be going wrong for Greg’s
case. Of course, I wouldn’t be sad if it helped me figure out my issue too!

Regards,
Christine

On Mon, Jul 20, 2015 at 11:07 AM, Steven Murawski <
steven.murawski@gmail.com> wrote:

One thing I’ve found, for both Vagrant and any kitchen driver, is
making sure my chefignore excludes the .kitchen directory. When Berkshelf
scans the cookbook directory, it likes to scan everything, and depending on
how much is in your .kitchen folder, that could take a bit. Plus, some
drivers have open handles to files (especially on Windows) which can lead
to errors.

Steve

Steven Murawski
Community Software Development Engineer @ Chef
Microsoft MVP - PowerShell
http://stevenmurawski.com

On 7/20/2015 10:01:03 AM, Yoshi Spendiff yoshi.spendiff@indochino.com
wrote:
Are you using chef_zero or chef_solo as your kitchen provisioner?

On Mon, Jul 20, 2015 at 7:01 AM, Greg Damiani <
greg.damiani@buzzfeed.com> wrote:

I’m seeing the opposite. Same machine, same run_list. "vagrant up"
takes about 80 minutes, “kitchen converge” takes 7-8 minutes.

On Fri, Jul 17, 2015 at 7:06 PM, Yoshi Spendiff <
yoshi.spendiff@indochino.com> wrote:

Sorry, I mean is it significantly slower to run *vagrant up *vs kitchen
converge
for the same developer on the same machine running the
same logic path. I.e if you make a .kitchen.yml in the same directory as
your Vagrantfile and configure kitchen to create an instance in exactly the
same manner are the times similar or is vagrant slower? In the effort to
isolate the problem, being file transfer, Vagrant or chef-zero. For me
Vagrant is actually faster…

There’s a couple of interesting things happening in the pipeline to
speed up test kitchen runs using chef-zero, maybe similar could help you
out if you’re willing to sit out another monkeypatch into merge situation:

https://github.com/test-kitchen/test-kitchen/pull/466
https://github.com/test-kitchen/test-kitchen/pull/620

If you’re using AWS for spooling instances I also found moving to
t2.small from t2.micro to be a speed step up.

On Fri, Jul 17, 2015 at 9:39 AM, Greg Damiani <
greg.damiani@buzzfeed.com> wrote:

Yep, test kitchen / kitchen-vagrant converge times haven’t changed
much (or at all) with Vagrant 1.7.3 installed.

On Fri, Jul 17, 2015 at 4:44 PM, Yoshi Spendiff <
yoshi.spendiff@indochino.com> wrote:

and/or spool up an instance using the chef_zero provider in test
kitchen
*

On Fri, Jul 17, 2015 at 8:41 AM, Yoshi Spendiff <
yoshi.spendiff@indochino.com> wrote:

Out of interest do you get similar times if you spool up a local
chef-zero instance and knife upload and/or spool up an instance using the
chef_zero provider, just to make sure it’s not Vagrant doing something
strange?

That seems like a long, long time for spooling to me. I used
kitchen-ec2 to spool instances on AWS and, granted my cookbook stack is
probably smaller than yours, it takes maybe 15-20 minutes to spool an
instance (so transferring over the internet).

On Fri, Jul 17, 2015 at 6:48 AM, Greg Damiani <
greg.damiani@buzzfeed.com> wrote:

Previous versions of Vagrant (1.7.2 and prior) used a fudged
Chef Zero provisioner that was actually Chef Solo under the covers:
https://github.com/mitchellh/vagrant/issues/5072

This meant that all your data bags and the code that depended on
them had to be fooled, stubbed, or coerced if you wanted to test it in
Vagrant. This was generally seen as a bummer and widely regarded as a time
sink that produced little business value. “That’s what you get for using
open source tools, Greg.”

We’d been waiting for Vagrant 1.7.3 over here for some time
because it promised real Chef Zero support: This should have made our
hacks and dumb shell scripts obsolete, allowing me to give my dev’s a Chef
role and nothing else for configuring their development VMs
https://github.com/mitchellh/vagrant/pull/5339

And so it was like a child starting its first day of school and
immediately after dreaming of christmas morning that we watched the above
PR get merged and thought to ourselves “probably won’t be more than 5 or 6
months before that gets bundled into an official release.”

Today I can say that this does not fit in our workflow. The
Chef Zero provisioner is taking anywhere from 20 to 40 minutes on different
laptops just to load our cookbooks into memory from local disk. Then
there’s the speed at which it seems to be configuring the resources in a
run list. I don’t have any real figures to refer to, just anecdotal
evidence, so I’ll resort to hyperbole. Vagrant 1.7.2 and Chef Solo seemed
zippy enough. I could spin up a development vagrant machine in about 5-10
minutes depending on role and network. 1.7.3 and Chef Zero are reminiscent
of a snail race through knee-deep peanut butter.

$ time vagrant provision


wait for it…

pain…
continue waiting


real 79m59.131s
user 0m4.463s
sys 0m1.405s

Sometimes we use Vagrant to test Chef changes locally before
uploading to a CI server that will build a branch of chef code on EC2 and
run some tests before we mark it as merge-able. I’m telling Chef dev’s in
this organization that Vagrant is supposed to save them time, that local
development is a convenience… and now this. I literally can’t even.

Who’s got a workaround for all these network round trips and
molasses uploads? Do I need more ergs? Should I just rub some Docker on
it? Aside from declaring ‘no joy’ and rolling back to Vagrant 1.7.2 what
are my options here? We’ve tried different folder syncing methods (nfs,
rsync) and found them all wanting. I’m afraid this is a feature, not a bug:

https://github.com/mitchellh/vagrant/issues/5972

Greg Damiani | Senior Kitten Delivery Engineer | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7E

Greg Damiani | Senior System Administrator | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7EB


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com

Greg Damiani | Senior System Administrator | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7EB


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com

Greg Damiani | Senior System Administrator | @damianigreg

BuzzFeed: The Social News and Entertainment Company

40 Argyll Street, 2nd Floor, London, W1F 7EB


Yoshi Spendiff
Ops Engineer
Indochino
Mobile: +1 778 952 2025
Email: yoshi.spendiff@indochino.com


ThirdWave Insights, LLC I (512) 971-8727 <%28512%29%20656-7724> I
www.ThirdWaveInsights.com I P.O. Box 500134 I Austin, TX 78750