Chef at Airbnb

Hi Chefs!

We just put up a blog post which details our tooling and workflow around
Chef Solo at Airbnb. I thought I'd let the list know, it's here:

Along with the post, we open-sourced several tools, including stemcell (
https://github.com/airbnb/stemcell) which we use to launch and bootstrap
instances and optica (https://github.com/airbnb/optica) which we use to
keep track of our instances and their last converge state.

I'm happy to answer questions about our approach here if anyone is
interested.

--Igor

Great post, thanks for sharing.

On Tue, Oct 15, 2013 at 6:55 PM, Igor Serebryany <igor.serebryany@airbnb.com

wrote:

Hi Chefs!

We just put up a blog post which details our tooling and workflow around
Chef Solo at Airbnb. I thought I'd let the list know, it's here:
Making Breakfast: Chef at Airbnb | by AirbnbEng | The Airbnb Tech Blog | Medium

Along with the post, we open-sourced several tools, including stemcell (
GitHub - airbnb/stemcell: Airbnb's EC2 instance creation and bootstrapping tool) which we use to launch and bootstrap
instances and optica (GitHub - airbnb/optica: A tool for keeping track of nodes in your infrastructure) which we use to
keep track of our instances and their last converge state.

I'm happy to answer questions about our approach here if anyone is
interested.

--Igor

Hi Igor,

When you say that the chef server approach was not scaling, did you
consider using "the berkshelf way" approach and having the cookbook build
and uploaded via CI server? If so, what issue did you run into that the
berkshelf way did not solve?

Best,

-- Sander

On Tue, Oct 15, 2013 at 3:55 PM, Igor Serebryany <igor.serebryany@airbnb.com

wrote:

Hi Chefs!

We just put up a blog post which details our tooling and workflow around
Chef Solo at Airbnb. I thought I'd let the list know, it's here:
Making Breakfast: Chef at Airbnb | by AirbnbEng | The Airbnb Tech Blog | Medium

Along with the post, we open-sourced several tools, including stemcell (
GitHub - airbnb/stemcell: Airbnb's EC2 instance creation and bootstrapping tool) which we use to launch and bootstrap
instances and optica (GitHub - airbnb/optica: A tool for keeping track of nodes in your infrastructure) which we use to
keep track of our instances and their last converge state.

I'm happy to answer questions about our approach here if anyone is
interested.

--Igor

--
Sander van Zoest
sander@vanzoest.com
San Diego, CA, USA
http://Sander.vanZoest.com/

Sander, at the time, we googled around and learned about additional tooling
which might have been able to help. but both figuring out the tooling AND
getting it going on CI AND teaching it to the rest of our team was
yak-shaving. our task was to get our chaotic legacy infrastructure on chef,
as quickly as possible, and what we ended up doing was extremely
light-weight.

even today, i'm not convinced that the berkshelf way has any advantages
over our approach. i think that what we're doing is still much easier to
operate, and much easier to teach. for developing and testing cookbooks,
our approach involves no additional tooling beyond just git. and as far as
ops work,stemcell https://github.com/airbnb/stemcell and
opticahttps://github.com/airbnb/optica neatly
replace knife and chef-server, and arguably do a better job. (stemcell,
because of how it keeps instance metadata in the role files, and optica
because of how flexible and easy to use it is).

actually, i think the biggest drawback to what we're doing is in community
integration; i have a few cookbooks i'd like to share, but (a) i'd have to
break them out into separate repos to do so and (b) i'd have to start doing
all that manual versioning i've been lucky to get to ignore. also, our
approach invites people to just edit upstream cookbooks because they're in
our repo. once or twice so far, we've had to back-port our changes to newer
versions of, say, the runit
cookbookhttps://github.com/opscode-cookbooks/runit (this
was complicated when that cookbook was basically completely rewritten as
resource/provider libraries).

--igor

On Wed, Oct 16, 2013 at 11:31 AM, Sander van Zoest sander@vanzoest.comwrote:

Hi Igor,

When you say that the chef server approach was not scaling, did you
consider using "the berkshelf way" approach and having the cookbook build
and uploaded via CI server? If so, what issue did you run into that the
berkshelf way did not solve?

Best,

-- Sander

On Tue, Oct 15, 2013 at 3:55 PM, Igor Serebryany <
igor.serebryany@airbnb.com> wrote:

Hi Chefs!

We just put up a blog post which details our tooling and workflow around
Chef Solo at Airbnb. I thought I'd let the list know, it's here:
Making Breakfast: Chef at Airbnb | by AirbnbEng | The Airbnb Tech Blog | Medium

Along with the post, we open-sourced several tools, including stemcell (
GitHub - airbnb/stemcell: Airbnb's EC2 instance creation and bootstrapping tool) which we use to launch and bootstrap
instances and optica (GitHub - airbnb/optica: A tool for keeping track of nodes in your infrastructure) which we use to
keep track of our instances and their last converge state.

I'm happy to answer questions about our approach here if anyone is
interested.

--Igor

--
Sander van Zoest
sander@vanzoest.com
San Diego, CA, USA
http://Sander.vanZoest.com/

Rewriting a "chef server" could arguably be yak-shaving -- not picking on your decision, just sayin'.

On Wednesday, October 16, 2013 at 1:33 PM, Igor Serebryany wrote:

Sander, at the time, we googled around and learned about additional tooling which might have been able to help. but both figuring out the tooling AND getting it going on CI AND teaching it to the rest of our team was yak-shaving. our task was to get our chaotic legacy infrastructure on chef, as quickly as possible, and what we ended up doing was extremely light-weight.

even today, i'm not convinced that the berkshelf way has any advantages over our approach. i think that what we're doing is still much easier to operate, and much easier to teach. for developing and testing cookbooks, our approach involves no additional tooling beyond just git. and as far as ops work,stemcell (GitHub - airbnb/stemcell: Airbnb's EC2 instance creation and bootstrapping tool) and optica (GitHub - airbnb/optica: A tool for keeping track of nodes in your infrastructure) neatly replace knife and chef-server, and arguably do a better job. (stemcell, because of how it keeps instance metadata in the role files, and optica because of how flexible and easy to use it is).

actually, i think the biggest drawback to what we're doing is in community integration; i have a few cookbooks i'd like to share, but (a) i'd have to break them out into separate repos to do so and (b) i'd have to start doing all that manual versioning i've been lucky to get to ignore. also, our approach invites people to just edit upstream cookbooks because they're in our repo. once or twice so far, we've had to back-port our changes to newer versions of, say, the runit cookbook (GitHub - chef-cookbooks/runit: Development repository for the Chef Runit Cookbook) (this was complicated when that cookbook was basically completely rewritten as resource/provider libraries).

--igor

On Wed, Oct 16, 2013 at 11:31 AM, Sander van Zoest <sander@vanzoest.com (mailto:sander@vanzoest.com)> wrote:

Hi Igor,

When you say that the chef server approach was not scaling, did you consider using "the berkshelf way" approach and having the cookbook build and uploaded via CI server? If so, what issue did you run into that the berkshelf way did not solve?

Best,

-- Sander

On Tue, Oct 15, 2013 at 3:55 PM, Igor Serebryany <igor.serebryany@airbnb.com (mailto:igor.serebryany@airbnb.com)> wrote:

Hi Chefs!

We just put up a blog post which details our tooling and workflow around Chef Solo at Airbnb. I thought I'd let the list know, it's here: Making Breakfast: Chef at Airbnb | by AirbnbEng | The Airbnb Tech Blog | Medium

Along with the post, we open-sourced several tools, including stemcell (GitHub - airbnb/stemcell: Airbnb's EC2 instance creation and bootstrapping tool) which we use to launch and bootstrap instances and optica (GitHub - airbnb/optica: A tool for keeping track of nodes in your infrastructure) which we use to keep track of our instances and their last converge state.

I'm happy to answer questions about our approach here if anyone is interested.

--Igor

--
Sander van Zoest sander@vanzoest.com (mailto:sander@vanzoest.com)
San Diego, CA, USA http://Sander.vanZoest.com/

I prefer calling it bike-shedding. However, having ran chef-servers at
scale, I could argue its a necessity.

On Wed, Oct 16, 2013 at 4:40 PM, John Dewey john@dewey.ws wrote:

Rewriting a "chef server" could arguably be yak-shaving -- not picking on
your decision, just sayin'.

I really like the idea of separating node attributes, data_bags, etc from
the cookbooks "repo". It's a major shortcoming for open source chef IMO.
We ended up hooking up our application recipes directly to our API server
for better isolation.

On Thu, Oct 17, 2013 at 9:40 AM, John Dewey john@dewey.ws wrote:

Rewriting a "chef server" could arguably be yak-shaving -- not picking on
your decision, just sayin'.

On Wednesday, October 16, 2013 at 1:33 PM, Igor Serebryany wrote:

Sander, at the time, we googled around and learned about additional
tooling which might have been able to help. but both figuring out the
tooling AND getting it going on CI AND teaching it to the rest of our team
was yak-shaving. our task was to get our chaotic legacy infrastructure on
chef, as quickly as possible, and what we ended up doing was extremely
light-weight.

even today, i'm not convinced that the berkshelf way has any advantages
over our approach. i think that what we're doing is still much easier to
operate, and much easier to teach. for developing and testing cookbooks,
our approach involves no additional tooling beyond just git. and as far as
ops work,stemcell https://github.com/airbnb/stemcell and opticahttps://github.com/airbnb/optica neatly
replace knife and chef-server, and arguably do a better job. (stemcell,
because of how it keeps instance metadata in the role files, and optica
because of how flexible and easy to use it is).

actually, i think the biggest drawback to what we're doing is in community
integration; i have a few cookbooks i'd like to share, but (a) i'd have to
break them out into separate repos to do so and (b) i'd have to start doing
all that manual versioning i've been lucky to get to ignore. also, our
approach invites people to just edit upstream cookbooks because they're in
our repo. once or twice so far, we've had to back-port our changes to newer
versions of, say, the runit cookbookhttps://github.com/opscode-cookbooks/runit (this
was complicated when that cookbook was basically completely rewritten as
resource/provider libraries).

--igor

On Wed, Oct 16, 2013 at 11:31 AM, Sander van Zoest sander@vanzoest.comwrote:

Hi Igor,

When you say that the chef server approach was not scaling, did you
consider using "the berkshelf way" approach and having the cookbook build
and uploaded via CI server? If so, what issue did you run into that the
berkshelf way did not solve?

Best,

-- Sander

On Tue, Oct 15, 2013 at 3:55 PM, Igor Serebryany <
igor.serebryany@airbnb.com> wrote:

Hi Chefs!

We just put up a blog post which details our tooling and workflow around
Chef Solo at Airbnb. I thought I'd let the list know, it's here:
Making Breakfast: Chef at Airbnb | by AirbnbEng | The Airbnb Tech Blog | Medium

Along with the post, we open-sourced several tools, including stemcell (
GitHub - airbnb/stemcell: Airbnb's EC2 instance creation and bootstrapping tool) which we use to launch and bootstrap
instances and optica (GitHub - airbnb/optica: A tool for keeping track of nodes in your infrastructure) which we use to
keep track of our instances and their last converge state.

I'm happy to answer questions about our approach here if anyone is
interested.

--Igor

--
Sander van Zoest
sander@vanzoest.com
San Diego, CA, USA
http://Sander.vanZoest.com/

optica https://github.com/airbnb/optica doesn't re-implement chef server,
it just keeps track of nodes that have converged and their last converge
status/output.

we don't have anything that stores node metadata, cookbooks, or anything
else for that matter. the code for the converge right now comes straight
from our git repo, and will soon be coming from a tarball published by CI.

On Wed, Oct 16, 2013 at 1:40 PM, John Dewey john@dewey.ws wrote:

Rewriting a "chef server" could arguably be yak-shaving -- not picking on
your decision, just sayin'.

On Wednesday, October 16, 2013 at 1:33 PM, Igor Serebryany wrote:

Sander, at the time, we googled around and learned about additional
tooling which might have been able to help. but both figuring out the
tooling AND getting it going on CI AND teaching it to the rest of our team
was yak-shaving. our task was to get our chaotic legacy infrastructure on
chef, as quickly as possible, and what we ended up doing was extremely
light-weight.

even today, i'm not convinced that the berkshelf way has any advantages
over our approach. i think that what we're doing is still much easier to
operate, and much easier to teach. for developing and testing cookbooks,
our approach involves no additional tooling beyond just git. and as far as
ops work,stemcell https://github.com/airbnb/stemcell and opticahttps://github.com/airbnb/optica neatly
replace knife and chef-server, and arguably do a better job. (stemcell,
because of how it keeps instance metadata in the role files, and optica
because of how flexible and easy to use it is).

actually, i think the biggest drawback to what we're doing is in community
integration; i have a few cookbooks i'd like to share, but (a) i'd have to
break them out into separate repos to do so and (b) i'd have to start doing
all that manual versioning i've been lucky to get to ignore. also, our
approach invites people to just edit upstream cookbooks because they're in
our repo. once or twice so far, we've had to back-port our changes to newer
versions of, say, the runit cookbookhttps://github.com/opscode-cookbooks/runit (this
was complicated when that cookbook was basically completely rewritten as
resource/provider libraries).

--igor

On Wed, Oct 16, 2013 at 11:31 AM, Sander van Zoest sander@vanzoest.comwrote:

Hi Igor,

When you say that the chef server approach was not scaling, did you
consider using "the berkshelf way" approach and having the cookbook build
and uploaded via CI server? If so, what issue did you run into that the
berkshelf way did not solve?

Best,

-- Sander

On Tue, Oct 15, 2013 at 3:55 PM, Igor Serebryany <
igor.serebryany@airbnb.com> wrote:

Hi Chefs!

We just put up a blog post which details our tooling and workflow around
Chef Solo at Airbnb. I thought I'd let the list know, it's here:
Making Breakfast: Chef at Airbnb | by AirbnbEng | The Airbnb Tech Blog | Medium

Along with the post, we open-sourced several tools, including stemcell (
GitHub - airbnb/stemcell: Airbnb's EC2 instance creation and bootstrapping tool) which we use to launch and bootstrap
instances and optica (GitHub - airbnb/optica: A tool for keeping track of nodes in your infrastructure) which we use to
keep track of our instances and their last converge state.

I'm happy to answer questions about our approach here if anyone is
interested.

--Igor

--
Sander van Zoest
sander@vanzoest.com
San Diego, CA, USA
http://Sander.vanZoest.com/

On Wed, Oct 16, 2013 at 5:54 PM, Igor Serebryany <igor.serebryany@airbnb.com

wrote:

and will soon be coming from a tarball published by CI.

That's the model I've been toying with as well. It's a lot less, um,
"exciting" than most of the common chef workflows, but much easier to grok.

+1 for using Chef Solo :slight_smile:

I guess we have the same motivation: avoid the additional complexity of
Chef Server
unless you really need it!

For myself I have found a sweet spot using Chef Solo, Vagrant and
Berkshelf, which,
apart from the Chef Solo bit, is quite different from your approach, but
probably worth
sharing as well.

The key points to make it play nicely together are:

  • use single-repo-per-cookbook rather than a monolithic chef-repo [0]
  • pull in cookbook dependencies via Berkshelf / Berksfile [1]
  • use application cookbooks [2] rather than roles
  • differentiate between cookbook repositories [3] vs. infrastructure
    repositories [4]

A Vagrant-based infrastructure repository yields some really nice
properties:

  • different providers: deploy your infrastructure on virtualbox, lxc, aws,
    rackspace, anywhere...
  • use the same configuration mechanism (Vagrantfile) no matter where you
    deploy
  • use the same CLI (vagrant up) no matter where you deploy
  • use the same plugin ecosystem (vagrant-omnibus, vagrant-berkshelf,
    vagrant-cachier, etc..) no matter where you deploy
  • define the infrastructure-specific environments, data_bags and node
    configuration where they belong to

This approach works really well if you have a static environment where you
can still name
all of your machines (i.e. no elasticity / autoscaling). In this case Chef
Solo + Vagrant
seem to be a perfect fit.

If you have something more dynamic with autoscaling etc. you probably need
Chef Server
and search. There's some overhead involved in running an additional piece
of infrastructure,
but also the workflow gets slightly more complicated when using Chef Server.

Happy to hear what others think about it!

Refs:
[0]

[1] Redirecting…
[2]
devopsanywhere: How to Write Reusable Chef Cookbooks, Gangnam Style
[3] GitHub - tknerr/sample-toplevel-cookbook: A sample "top-level" cookbook hosting a website
[4]
sample-infrastructure-repo/Vagrantfile at master · tknerr/sample-infrastructure-repo · GitHub

On Wed, Oct 16, 2013 at 10:33 PM, Igor Serebryany <
igor.serebryany@airbnb.com> wrote:

Sander, at the time, we googled around and learned about additional
tooling which might have been able to help. but both figuring out the
tooling AND getting it going on CI AND teaching it to the rest of our team
was yak-shaving. our task was to get our chaotic legacy infrastructure on
chef, as quickly as possible, and what we ended up doing was extremely
light-weight.

even today, i'm not convinced that the berkshelf way has any advantages
over our approach. i think that what we're doing is still much easier to
operate, and much easier to teach. for developing and testing cookbooks,
our approach involves no additional tooling beyond just git. and as far as
ops work,stemcell https://github.com/airbnb/stemcell and opticahttps://github.com/airbnb/optica neatly
replace knife and chef-server, and arguably do a better job. (stemcell,
because of how it keeps instance metadata in the role files, and optica
because of how flexible and easy to use it is).

actually, i think the biggest drawback to what we're doing is in community
integration; i have a few cookbooks i'd like to share, but (a) i'd have to
break them out into separate repos to do so and (b) i'd have to start doing
all that manual versioning i've been lucky to get to ignore. also, our
approach invites people to just edit upstream cookbooks because they're in
our repo. once or twice so far, we've had to back-port our changes to newer
versions of, say, the runit cookbookhttps://github.com/opscode-cookbooks/runit (this
was complicated when that cookbook was basically completely rewritten as
resource/provider libraries).

--igor

On Wed, Oct 16, 2013 at 11:31 AM, Sander van Zoest sander@vanzoest.comwrote:

Hi Igor,

When you say that the chef server approach was not scaling, did you
consider using "the berkshelf way" approach and having the cookbook build
and uploaded via CI server? If so, what issue did you run into that the
berkshelf way did not solve?

Best,

-- Sander

On Tue, Oct 15, 2013 at 3:55 PM, Igor Serebryany <
igor.serebryany@airbnb.com> wrote:

Hi Chefs!

We just put up a blog post which details our tooling and workflow around
Chef Solo at Airbnb. I thought I'd let the list know, it's here:
Making Breakfast: Chef at Airbnb | by AirbnbEng | The Airbnb Tech Blog | Medium

Along with the post, we open-sourced several tools, including stemcell (
GitHub - airbnb/stemcell: Airbnb's EC2 instance creation and bootstrapping tool) which we use to launch and
bootstrap instances and optica (GitHub - airbnb/optica: A tool for keeping track of nodes in your infrastructure) which
we use to keep track of our instances and their last converge state.

I'm happy to answer questions about our approach here if anyone is
interested.

--Igor

--
Sander van Zoest
sander@vanzoest.com
San Diego, CA, USA
http://Sander.vanZoest.com/

It's actually not that hard to split out a unified repository into
individual cookbook repos as long as you're not afraid of git filter-branch
... the company technically owns the shell script I wrote to do it so I
can't just post it here. I remember thinking it was going to be a lot
harder than it actually was...

On Wed, Oct 16, 2013 at 5:49 PM, Torben Knerr ukio@gmx.de wrote:

+1 for using Chef Solo :slight_smile:

I guess we have the same motivation: avoid the additional complexity of
Chef Server
unless you really need it!

For myself I have found a sweet spot using Chef Solo, Vagrant and
Berkshelf, which,
apart from the Chef Solo bit, is quite different from your approach, but
probably worth
sharing as well.

The key points to make it play nicely together are:

  • use single-repo-per-cookbook rather than a monolithic chef-repo [0]
  • pull in cookbook dependencies via Berkshelf / Berksfile [1]
  • use application cookbooks [2] rather than roles
  • differentiate between cookbook repositories [3] vs. infrastructure
    repositories [4]

A Vagrant-based infrastructure repository yields some really nice
properties:

  • different providers: deploy your infrastructure on virtualbox, lxc,
    aws, rackspace, anywhere...
  • use the same configuration mechanism (Vagrantfile) no matter where you
    deploy
  • use the same CLI (vagrant up) no matter where you deploy
  • use the same plugin ecosystem (vagrant-omnibus, vagrant-berkshelf,
    vagrant-cachier, etc..) no matter where you deploy
  • define the infrastructure-specific environments, data_bags and node
    configuration where they belong to

This approach works really well if you have a static environment where you
can still name
all of your machines (i.e. no elasticity / autoscaling). In this case Chef
Solo + Vagrant
seem to be a perfect fit.

If you have something more dynamic with autoscaling etc. you probably need
Chef Server
and search. There's some overhead involved in running an additional piece
of infrastructure,
but also the workflow gets slightly more complicated when using Chef
Server.

Happy to hear what others think about it!

Refs:
[0]
Cookbooks Migrated to New GitHub Organization - Chef Blog | Chef
[1] Redirecting…
[2]
devopsanywhere: How to Write Reusable Chef Cookbooks, Gangnam Style
[3] GitHub - tknerr/sample-toplevel-cookbook: A sample "top-level" cookbook hosting a website
[4]
sample-infrastructure-repo/Vagrantfile at master · tknerr/sample-infrastructure-repo · GitHub

On Wed, Oct 16, 2013 at 10:33 PM, Igor Serebryany <
igor.serebryany@airbnb.com> wrote:

Sander, at the time, we googled around and learned about additional
tooling which might have been able to help. but both figuring out the
tooling AND getting it going on CI AND teaching it to the rest of our team
was yak-shaving. our task was to get our chaotic legacy infrastructure on
chef, as quickly as possible, and what we ended up doing was extremely
light-weight.

even today, i'm not convinced that the berkshelf way has any advantages
over our approach. i think that what we're doing is still much easier to
operate, and much easier to teach. for developing and testing cookbooks,
our approach involves no additional tooling beyond just git. and as far as
ops work,stemcell https://github.com/airbnb/stemcell and opticahttps://github.com/airbnb/optica neatly
replace knife and chef-server, and arguably do a better job. (stemcell,
because of how it keeps instance metadata in the role files, and optica
because of how flexible and easy to use it is).

actually, i think the biggest drawback to what we're doing is in
community integration; i have a few cookbooks i'd like to share, but (a)
i'd have to break them out into separate repos to do so and (b) i'd have to
start doing all that manual versioning i've been lucky to get to ignore.
also, our approach invites people to just edit upstream cookbooks because
they're in our repo. once or twice so far, we've had to back-port our
changes to newer versions of, say, the runit cookbookhttps://github.com/opscode-cookbooks/runit (this
was complicated when that cookbook was basically completely rewritten as
resource/provider libraries).

--igor

On Wed, Oct 16, 2013 at 11:31 AM, Sander van Zoest sander@vanzoest.comwrote:

Hi Igor,

When you say that the chef server approach was not scaling, did you
consider using "the berkshelf way" approach and having the cookbook build
and uploaded via CI server? If so, what issue did you run into that the
berkshelf way did not solve?

Best,

-- Sander

On Tue, Oct 15, 2013 at 3:55 PM, Igor Serebryany <
igor.serebryany@airbnb.com> wrote:

Hi Chefs!

We just put up a blog post which details our tooling and workflow
around Chef Solo at Airbnb. I thought I'd let the list know, it's here:
Making Breakfast: Chef at Airbnb | by AirbnbEng | The Airbnb Tech Blog | Medium

Along with the post, we open-sourced several tools, including stemcell (
GitHub - airbnb/stemcell: Airbnb's EC2 instance creation and bootstrapping tool) which we use to launch and
bootstrap instances and optica (GitHub - airbnb/optica: A tool for keeping track of nodes in your infrastructure)
which we use to keep track of our instances and their last converge state.

I'm happy to answer questions about our approach here if anyone is
interested.

--Igor

--
Sander van Zoest
sander@vanzoest.com
San Diego, CA, USA
http://Sander.vanZoest.com/

the hard part is not splitting out a cookbook, the hard part is maintaining
the code in two separate repos. i think you can do this with git patch and
git apply, but it's not a familiar workflow.

On Thu, Oct 17, 2013 at 4:15 PM, steve . leftathome@gmail.com wrote:

It's actually not that hard to split out a unified repository into
individual cookbook repos as long as you're not afraid of git filter-branch
... the company technically owns the shell script I wrote to do it so I
can't just post it here. I remember thinking it was going to be a lot
harder than it actually was...

On Wed, Oct 16, 2013 at 5:49 PM, Torben Knerr ukio@gmx.de wrote:

+1 for using Chef Solo :slight_smile:

I guess we have the same motivation: avoid the additional complexity of
Chef Server
unless you really need it!

For myself I have found a sweet spot using Chef Solo, Vagrant and
Berkshelf, which,
apart from the Chef Solo bit, is quite different from your approach, but
probably worth
sharing as well.

The key points to make it play nicely together are:

  • use single-repo-per-cookbook rather than a monolithic chef-repo [0]
  • pull in cookbook dependencies via Berkshelf / Berksfile [1]
  • use application cookbooks [2] rather than roles
  • differentiate between cookbook repositories [3] vs. infrastructure
    repositories [4]

A Vagrant-based infrastructure repository yields some really nice
properties:

  • different providers: deploy your infrastructure on virtualbox, lxc,
    aws, rackspace, anywhere...
  • use the same configuration mechanism (Vagrantfile) no matter where you
    deploy
  • use the same CLI (vagrant up) no matter where you deploy
  • use the same plugin ecosystem (vagrant-omnibus, vagrant-berkshelf,
    vagrant-cachier, etc..) no matter where you deploy
  • define the infrastructure-specific environments, data_bags and node
    configuration where they belong to

This approach works really well if you have a static environment where
you can still name
all of your machines (i.e. no elasticity / autoscaling). In this case
Chef Solo + Vagrant
seem to be a perfect fit.

If you have something more dynamic with autoscaling etc. you probably
need Chef Server
and search. There's some overhead involved in running an additional piece
of infrastructure,
but also the workflow gets slightly more complicated when using Chef
Server.

Happy to hear what others think about it!

Refs:
[0]
Cookbooks Migrated to New GitHub Organization - Chef Blog | Chef
[1] Redirecting…
[2]
devopsanywhere: How to Write Reusable Chef Cookbooks, Gangnam Style
[3] GitHub - tknerr/sample-toplevel-cookbook: A sample "top-level" cookbook hosting a website
[4]
sample-infrastructure-repo/Vagrantfile at master · tknerr/sample-infrastructure-repo · GitHub

On Wed, Oct 16, 2013 at 10:33 PM, Igor Serebryany <
igor.serebryany@airbnb.com> wrote:

Sander, at the time, we googled around and learned about additional
tooling which might have been able to help. but both figuring out the
tooling AND getting it going on CI AND teaching it to the rest of our team
was yak-shaving. our task was to get our chaotic legacy infrastructure on
chef, as quickly as possible, and what we ended up doing was extremely
light-weight.

even today, i'm not convinced that the berkshelf way has any advantages
over our approach. i think that what we're doing is still much easier to
operate, and much easier to teach. for developing and testing cookbooks,
our approach involves no additional tooling beyond just git. and as far as
ops work,stemcell https://github.com/airbnb/stemcell and opticahttps://github.com/airbnb/optica neatly
replace knife and chef-server, and arguably do a better job. (stemcell,
because of how it keeps instance metadata in the role files, and optica
because of how flexible and easy to use it is).

actually, i think the biggest drawback to what we're doing is in
community integration; i have a few cookbooks i'd like to share, but (a)
i'd have to break them out into separate repos to do so and (b) i'd have to
start doing all that manual versioning i've been lucky to get to ignore.
also, our approach invites people to just edit upstream cookbooks because
they're in our repo. once or twice so far, we've had to back-port our
changes to newer versions of, say, the runit cookbookhttps://github.com/opscode-cookbooks/runit (this
was complicated when that cookbook was basically completely rewritten as
resource/provider libraries).

--igor

On Wed, Oct 16, 2013 at 11:31 AM, Sander van Zoest sander@vanzoest.comwrote:

Hi Igor,

When you say that the chef server approach was not scaling, did you
consider using "the berkshelf way" approach and having the cookbook build
and uploaded via CI server? If so, what issue did you run into that the
berkshelf way did not solve?

Best,

-- Sander

On Tue, Oct 15, 2013 at 3:55 PM, Igor Serebryany <
igor.serebryany@airbnb.com> wrote:

Hi Chefs!

We just put up a blog post which details our tooling and workflow
around Chef Solo at Airbnb. I thought I'd let the list know, it's here:
Making Breakfast: Chef at Airbnb | by AirbnbEng | The Airbnb Tech Blog | Medium

Along with the post, we open-sourced several tools, including stemcell
(GitHub - airbnb/stemcell: Airbnb's EC2 instance creation and bootstrapping tool) which we use to launch and
bootstrap instances and optica (GitHub - airbnb/optica: A tool for keeping track of nodes in your infrastructure)
which we use to keep track of our instances and their last converge state.

I'm happy to answer questions about our approach here if anyone is
interested.

--Igor

--
Sander van Zoest
sander@vanzoest.com
San Diego, CA, USA
http://Sander.vanZoest.com/

So far we've found that the filter-branch results in the same refs in the
generated target repo each time, so we can safely run the cookbook-splitter
as part of our regular CI-driven release process. However, we're about to
switch over to the one-cookbook-per-repo model for future releases, which
means this won't really be an issue past the end of this month.

(And! We'll finally be free of scheduled releases!)

On Thu, Oct 17, 2013 at 4:17 PM, Igor Serebryany <igor.serebryany@airbnb.com

wrote:

the hard part is not splitting out a cookbook, the hard part is
maintaining the code in two separate repos. i think you can do this with
git patch and git apply, but it's not a familiar workflow.

On Thu, Oct 17, 2013 at 4:15 PM, steve . leftathome@gmail.com wrote:

It's actually not that hard to split out a unified repository into
individual cookbook repos as long as you're not afraid of git filter-branch
... the company technically owns the shell script I wrote to do it so I
can't just post it here. I remember thinking it was going to be a lot
harder than it actually was...

On Wed, Oct 16, 2013 at 5:49 PM, Torben Knerr ukio@gmx.de wrote:

+1 for using Chef Solo :slight_smile:

I guess we have the same motivation: avoid the additional complexity of
Chef Server
unless you really need it!

For myself I have found a sweet spot using Chef Solo, Vagrant and
Berkshelf, which,
apart from the Chef Solo bit, is quite different from your approach, but
probably worth
sharing as well.

The key points to make it play nicely together are:

  • use single-repo-per-cookbook rather than a monolithic chef-repo [0]
  • pull in cookbook dependencies via Berkshelf / Berksfile [1]
  • use application cookbooks [2] rather than roles
  • differentiate between cookbook repositories [3] vs. infrastructure
    repositories [4]

A Vagrant-based infrastructure repository yields some really nice
properties:

  • different providers: deploy your infrastructure on virtualbox, lxc,
    aws, rackspace, anywhere...
  • use the same configuration mechanism (Vagrantfile) no matter where
    you deploy
  • use the same CLI (vagrant up) no matter where you deploy
  • use the same plugin ecosystem (vagrant-omnibus, vagrant-berkshelf,
    vagrant-cachier, etc..) no matter where you deploy
  • define the infrastructure-specific environments, data_bags and node
    configuration where they belong to

This approach works really well if you have a static environment where
you can still name
all of your machines (i.e. no elasticity / autoscaling). In this case
Chef Solo + Vagrant
seem to be a perfect fit.

If you have something more dynamic with autoscaling etc. you probably
need Chef Server
and search. There's some overhead involved in running an additional
piece of infrastructure,
but also the workflow gets slightly more complicated when using Chef
Server.

Happy to hear what others think about it!

Refs:
[0]
Cookbooks Migrated to New GitHub Organization - Chef Blog | Chef
[1] Redirecting…
[2]
devopsanywhere: How to Write Reusable Chef Cookbooks, Gangnam Style
[3] GitHub - tknerr/sample-toplevel-cookbook: A sample "top-level" cookbook hosting a website
[4]
sample-infrastructure-repo/Vagrantfile at master · tknerr/sample-infrastructure-repo · GitHub

On Wed, Oct 16, 2013 at 10:33 PM, Igor Serebryany <
igor.serebryany@airbnb.com> wrote:

Sander, at the time, we googled around and learned about additional
tooling which might have been able to help. but both figuring out the
tooling AND getting it going on CI AND teaching it to the rest of our team
was yak-shaving. our task was to get our chaotic legacy infrastructure on
chef, as quickly as possible, and what we ended up doing was extremely
light-weight.

even today, i'm not convinced that the berkshelf way has any advantages
over our approach. i think that what we're doing is still much easier to
operate, and much easier to teach. for developing and testing cookbooks,
our approach involves no additional tooling beyond just git. and as far as
ops work,stemcell https://github.com/airbnb/stemcell and opticahttps://github.com/airbnb/optica neatly
replace knife and chef-server, and arguably do a better job. (stemcell,
because of how it keeps instance metadata in the role files, and optica
because of how flexible and easy to use it is).

actually, i think the biggest drawback to what we're doing is in
community integration; i have a few cookbooks i'd like to share, but (a)
i'd have to break them out into separate repos to do so and (b) i'd have to
start doing all that manual versioning i've been lucky to get to ignore.
also, our approach invites people to just edit upstream cookbooks because
they're in our repo. once or twice so far, we've had to back-port our
changes to newer versions of, say, the runit cookbookhttps://github.com/opscode-cookbooks/runit (this
was complicated when that cookbook was basically completely rewritten as
resource/provider libraries).

--igor

On Wed, Oct 16, 2013 at 11:31 AM, Sander van Zoest <sander@vanzoest.com

wrote:

Hi Igor,

When you say that the chef server approach was not scaling, did you
consider using "the berkshelf way" approach and having the cookbook build
and uploaded via CI server? If so, what issue did you run into that the
berkshelf way did not solve?

Best,

-- Sander

On Tue, Oct 15, 2013 at 3:55 PM, Igor Serebryany <
igor.serebryany@airbnb.com> wrote:

Hi Chefs!

We just put up a blog post which details our tooling and workflow
around Chef Solo at Airbnb. I thought I'd let the list know, it's here:
Making Breakfast: Chef at Airbnb | by AirbnbEng | The Airbnb Tech Blog | Medium

Along with the post, we open-sourced several tools, including
stemcell (GitHub - airbnb/stemcell: Airbnb's EC2 instance creation and bootstrapping tool) which we use to launch
and bootstrap instances and optica (GitHub - airbnb/optica: A tool for keeping track of nodes in your infrastructure)
which we use to keep track of our instances and their last converge state.

I'm happy to answer questions about our approach here if anyone is
interested.

--Igor

--
Sander van Zoest
sander@vanzoest.com
San Diego, CA, USA
http://Sander.vanZoest.com/

Hi,

On Fri, Oct 18, 2013 at 10:17 AM, Igor Serebryany
igor.serebryany@airbnb.com wrote:

the hard part is not splitting out a cookbook, the hard part is maintaining
the code in two separate repos. i think you can do this with git patch and
git apply, but it's not a familiar workflow.

We use braid [1] fairly successfully to do this.

Merge the mycookbook to cookbooks/mycookbook

braid add git://github.com/mybiz/chef-mycookbook.git cookbooks/mycookbook

Make some changes in aggregate cookbook

vi cookbooks/mycookbook/...
git commit -a -m "Mycookbook changes"

Compare the local repo with the last import

braid diff cookbooks/mycookbook

Update the local cookbook copy with the upstream changes

braid update cookbooks/mycookbook

You can also push changes directly from your local repo back to the
upstream but we tend to got through the generate a patch and go
through review/pull request system when merging back.

[1] GitHub - realityforge/braid: Simple tool to help track git vendor branches in a git repository. WARNING: Changes integrated to upstream

--
Cheers,

Peter Donald

i have learned hard way not to treat code as artifact (git submodule-> git
subtree -> artifactory/gemserver/apt|rpmrepo etc). stuff inside git repo is
code, as long as you manage them via git tools, if you use berks or
librarian to access the same, you will be treating them as artifact. you
can use git sha/tags etc as your artifact version, but coupling them inside
the packaging too can increase friction in update process. we have seen
this exact problem in user space tool develeopment, language specific
library development (remember the days before rubygems & pypi :slight_smile: ). I
think we'll follow the same. But its also true we might not be able to
adopt the artifact style practices up front, we have to wait till the
tooling around artifact/package management is matured, as well as the
artifacts itself are stable enough that we can include them without looking
at the code, which clearly not the case with vast majority of community
cookbooks.

On Thu, Oct 17, 2013 at 4:26 PM, steve . leftathome@gmail.com wrote:

So far we've found that the filter-branch results in the same refs in the
generated target repo each time, so we can safely run the cookbook-splitter
as part of our regular CI-driven release process. However, we're about to
switch over to the one-cookbook-per-repo model for future releases, which
means this won't really be an issue past the end of this month.

(And! We'll finally be free of scheduled releases!)

On Thu, Oct 17, 2013 at 4:17 PM, Igor Serebryany <
igor.serebryany@airbnb.com> wrote:

the hard part is not splitting out a cookbook, the hard part is
maintaining the code in two separate repos. i think you can do this with
git patch and git apply, but it's not a familiar workflow.

On Thu, Oct 17, 2013 at 4:15 PM, steve . leftathome@gmail.com wrote:

It's actually not that hard to split out a unified repository into
individual cookbook repos as long as you're not afraid of git filter-branch
... the company technically owns the shell script I wrote to do it so I
can't just post it here. I remember thinking it was going to be a lot
harder than it actually was...

On Wed, Oct 16, 2013 at 5:49 PM, Torben Knerr ukio@gmx.de wrote:

+1 for using Chef Solo :slight_smile:

I guess we have the same motivation: avoid the additional complexity of
Chef Server
unless you really need it!

For myself I have found a sweet spot using Chef Solo, Vagrant and
Berkshelf, which,
apart from the Chef Solo bit, is quite different from your approach,
but probably worth
sharing as well.

The key points to make it play nicely together are:

  • use single-repo-per-cookbook rather than a monolithic chef-repo [0]
  • pull in cookbook dependencies via Berkshelf / Berksfile [1]
  • use application cookbooks [2] rather than roles
  • differentiate between cookbook repositories [3] vs. infrastructure
    repositories [4]

A Vagrant-based infrastructure repository yields some really nice
properties:

  • different providers: deploy your infrastructure on virtualbox, lxc,
    aws, rackspace, anywhere...
  • use the same configuration mechanism (Vagrantfile) no matter where
    you deploy
  • use the same CLI (vagrant up) no matter where you deploy
  • use the same plugin ecosystem (vagrant-omnibus, vagrant-berkshelf,
    vagrant-cachier, etc..) no matter where you deploy
  • define the infrastructure-specific environments, data_bags and node
    configuration where they belong to

This approach works really well if you have a static environment where
you can still name
all of your machines (i.e. no elasticity / autoscaling). In this case
Chef Solo + Vagrant
seem to be a perfect fit.

If you have something more dynamic with autoscaling etc. you probably
need Chef Server
and search. There's some overhead involved in running an additional
piece of infrastructure,
but also the workflow gets slightly more complicated when using Chef
Server.

Happy to hear what others think about it!

Refs:
[0]
Cookbooks Migrated to New GitHub Organization - Chef Blog | Chef
[1] Redirecting…
[2]
devopsanywhere: How to Write Reusable Chef Cookbooks, Gangnam Style
[3] GitHub - tknerr/sample-toplevel-cookbook: A sample "top-level" cookbook hosting a website
[4]
sample-infrastructure-repo/Vagrantfile at master · tknerr/sample-infrastructure-repo · GitHub

On Wed, Oct 16, 2013 at 10:33 PM, Igor Serebryany <
igor.serebryany@airbnb.com> wrote:

Sander, at the time, we googled around and learned about additional
tooling which might have been able to help. but both figuring out the
tooling AND getting it going on CI AND teaching it to the rest of our team
was yak-shaving. our task was to get our chaotic legacy infrastructure on
chef, as quickly as possible, and what we ended up doing was extremely
light-weight.

even today, i'm not convinced that the berkshelf way has any
advantages over our approach. i think that what we're doing is still much
easier to operate, and much easier to teach. for developing and testing
cookbooks, our approach involves no additional tooling beyond just git. and
as far as ops work,stemcell https://github.com/airbnb/stemcell and
optica https://github.com/airbnb/optica neatly replace knife and
chef-server, and arguably do a better job. (stemcell, because of how it
keeps instance metadata in the role files, and optica because of how
flexible and easy to use it is).

actually, i think the biggest drawback to what we're doing is in
community integration; i have a few cookbooks i'd like to share, but (a)
i'd have to break them out into separate repos to do so and (b) i'd have to
start doing all that manual versioning i've been lucky to get to ignore.
also, our approach invites people to just edit upstream cookbooks because
they're in our repo. once or twice so far, we've had to back-port our
changes to newer versions of, say, the runit cookbookhttps://github.com/opscode-cookbooks/runit (this
was complicated when that cookbook was basically completely rewritten as
resource/provider libraries).

--igor

On Wed, Oct 16, 2013 at 11:31 AM, Sander van Zoest <
sander@vanzoest.com> wrote:

Hi Igor,

When you say that the chef server approach was not scaling, did you
consider using "the berkshelf way" approach and having the cookbook build
and uploaded via CI server? If so, what issue did you run into that the
berkshelf way did not solve?

Best,

-- Sander

On Tue, Oct 15, 2013 at 3:55 PM, Igor Serebryany <
igor.serebryany@airbnb.com> wrote:

Hi Chefs!

We just put up a blog post which details our tooling and workflow
around Chef Solo at Airbnb. I thought I'd let the list know, it's here:
Making Breakfast: Chef at Airbnb | by AirbnbEng | The Airbnb Tech Blog | Medium

Along with the post, we open-sourced several tools, including
stemcell (GitHub - airbnb/stemcell: Airbnb's EC2 instance creation and bootstrapping tool) which we use to
launch and bootstrap instances and optica (
GitHub - airbnb/optica: A tool for keeping track of nodes in your infrastructure) which we use to keep track of our
instances and their last converge state.

I'm happy to answer questions about our approach here if anyone is
interested.

--Igor

--
Sander van Zoest
sander@vanzoest.com
San Diego, CA, USA
http://Sander.vanZoest.com/