On Chef patterns - ways to organize a Chef kitchen

Hi list,

I’m wondering Chef kitchen patterns. Right now, I use knife-solo, and I have a general-purpose kitchen. I add nodes as needed (not necessarily for the same app). Since I don’t really deal with a project that requires a more than one (or two) server(s), yet, it’s easy to just do something like:

➜ mykitchenv2 git:(master) ✗ ls
Berksfile Berksfile.lock Gemfile Gemfile.lock Vagrantfile cookbooks data_bags environments nodes roles site-cookbooks

➜ roles git:(master) ✗ ls
lamp.json rack.json

➜ nodes git:(master) ✗ ls
127.0.0.1.json anotherhost.json …

So, it’s basically a general-purpose kitchen. It’s not really geared towards a specific application. My workflow looks like this:

  1. New project, hack, app ready to be deployed
  2. Create new VM @ digital ocean, grab the IP, create a new node json in the kitchen
  3. It’s probably a Rails app. Add the rack role[lamp] to the run_list for this node. The rack role basically setups a new rack box using the rackbox cookbook (https://github.com/teohm/rackbox-cookbook)
  4. knife solo prepare root@newvm_ip
  5. knife solo cook root@newvm_ip
  6. Use capistrano to deploy the app.

Then, I do the same thing if I need a staging server.

It works pretty well. However, most of the apps I develop right now are apps with a quite simple infrastructure. They don’t require more than 2 servers (prod,staging). I did work with more elaborated apps before with up to 10 different servers (but without using Chef at the time, go figure).

The problem with this approach is that I end up having several unrelated nodes, and in the end, I don’t really describe the infrastructure of a single app/system, but of several ones, which can become clunky. Basically, for now, I’m just using Chef to get the sever ready, but not really using it to manage the server afterwards.

That being said, I’d like some insights on Chef patterns related to this, for example, when is it time to create a kitchen solely for an app?

Thanks in advance,


Marcelo

"It depends" :wink:

When I was doing client work I generally kept one kitchen per client just
to make the final delivery as clean as possible. For myself I tend to keep
kitchens logically separate by project.

If you need to share stuff between kitchens I recommend pushing all that
stuff into cookbooks so that you can use berkshelf or librarian-chef to
load in the shared cookbooks from their own dedicated git repositories.
This also helps avoid loading too much info into roles which is another
good practice.

Finally, you should try to get into the practice of using chef to provision
as well as manage the servers since it's where configuration "management"
really starts to shine. Of course there are other options but if you're on
a lamp stack, generating configuration files and/or data from chef will
probably be the easiest to set up.

Hope that helps some. I think this is one of those things that varies a bit
from project to project.

Feel free to open a knife-solo issue if you run into trouble there.

-Mat

On Sun, Feb 2, 2014 at 12:00 AM, Marcelo Serpa mlists@fullofcaffeine.comwrote:

Hi list,

I’m wondering Chef kitchen patterns. Right now, I use knife-solo, and I
have a general-purpose kitchen. I add nodes as needed (not necessarily for
the same app). Since I don’t really deal with a project that requires a
more than one (or two) server(s), yet, it’s easy to just do something like:

➜ mykitchenv2 git:(master) ✗ ls
Berksfile Berksfile.lock Gemfile Gemfile.lock Vagrantfile
cookbooks data_bags environments nodes roles
site-cookbooks

➜ roles git:(master) ✗ ls
lamp.json rack.json

➜ nodes git:(master) ✗ ls
127.0.0.1.json anotherhost.json …

So, it’s basically a general-purpose kitchen. It’s not really geared
towards a specific application. My workflow looks like this:

  1. New project, hack, app ready to be deployed
  2. Create new VM @ digital ocean, grab the IP, create a new node json in
    the kitchen
  3. It’s probably a Rails app. Add the rack role[lamp] to the run_list for
    this node. The rack role basically setups a new rack box using the rackbox
    cookbook (GitHub - teohm/rackbox-cookbook: Setup a Rack-based application server to run Unicorn & Passenger apps.)
  4. knife solo prepare root@newvm_ip
  5. knife solo cook root@newvm_ip
  6. Use capistrano to deploy the app.

Then, I do the same thing if I need a staging server.

It works pretty well. However, most of the apps I develop right now are
apps with a quite simple infrastructure. They don’t require more than 2
servers (prod,staging). I did work with more elaborated apps before with up
to 10 different servers (but without using Chef at the time, go figure).

The problem with this approach is that I end up having several unrelated
nodes, and in the end, I don’t really describe the infrastructure of a
single app/system, but of several ones, which can become clunky. Basically,
for now, I’m just using Chef to get the sever ready, but not really using
it to manage the server afterwards.

That being said, I’d like some insights on Chef patterns related to this,
for example, when is it time to create a kitchen solely for an app?

Thanks in advance,

--
Marcelo

Hi Mat,

Thank you for taking the time to answer. I appreciate it.

This also helps avoid loading too much info into roles which is another good practice.

Interesting… could you elaborate on this one? An example would be nice :slight_smile:

Finally, you should try to get into the practice of using chef to provision as well as manage the servers since it's where configuration "management" really starts to shine.
Excellent point. Now I see that the main reason why I haven’t been using a kitchen per project, is that I’ve been using Chef only to provision servers (hence, we could call that a provision kitchen ;), and not manage them. For me, so far, it was the most painful SCM aspect. Since my apps’ don’t really change a lot when it comes to infrastructure, it has been enough to do those changes by hand. So far, it has been enough to scale vertically (basically expanding the RAM of a single node), which I can do easily on digitalocean (or AWS or any cloud VM provider ).

By the way, how would this work with knife solo? Or do you think that when the infrastructure becomes complex to the point where Chef is useful to manage it, then it’s time to add a Chef server to the equation?

Also, I’ve been playing with PaaS solutions lately. One of them is Tsuru, written in Go, and Dokku, which is pretty easy to setup on digital ocean. Perhaps the best combo for apps that don’t require a very complex infrastructure, would be to use Chef to provision, say, Dokku and also setup the necessary plugins (for db access, for example), and then just use Dokku from then on. I think this could work for simpler apps, but as soon as you need more server side components, then it becomes a burden to deal with PaaS plugins. What do you think about this?

Lastly, do you think it would be feasible to have the kitchen integrated in the app project itself? Let’s say I have an app built on Rails:

$ rails new test

Then, create the kitchen inside the app’s dir"

$ cd test
$ knife solo init .

Ending up with:

➜ ls
Gemfile app config.ru db log roles tmp
README.rdoc bin cookbooks environments nodes site-cookbooks vendor
Rakefile config data_bags lib public test

So, we have the app, and the code that does all the SCM (potentially adding Capistrano to the mix).


Marcelo

On Sunday, February 2, 2014 at 5:13 PM, Mat Schaffer wrote:

"It depends" :wink:

When I was doing client work I generally kept one kitchen per client just to make the final delivery as clean as possible. For myself I tend to keep kitchens logically separate by project.

If you need to share stuff between kitchens I recommend pushing all that stuff into cookbooks so that you can use berkshelf or librarian-chef to load in the shared cookbooks from their own dedicated git repositories. This also helps avoid loading too much info into roles which is another good practice.

Finally, you should try to get into the practice of using chef to provision as well as manage the servers since it's where configuration "management" really starts to shine. Of course there are other options but if you're on a lamp stack, generating configuration files and/or data from chef will probably be the easiest to set up.

Hope that helps some. I think this is one of those things that varies a bit from project to project.

Feel free to open a knife-solo issue if you run into trouble there.

-Mat

Mat Schaffer - Gifu Japan | about.me (Mat Schaffer - Gifu Japan | about.me)

On Sun, Feb 2, 2014 at 12:00 AM, Marcelo Serpa <mlists@fullofcaffeine.com (mailto:mlists@fullofcaffeine.com)> wrote:

Hi list,

I’m wondering Chef kitchen patterns. Right now, I use knife-solo, and I have a general-purpose kitchen. I add nodes as needed (not necessarily for the same app). Since I don’t really deal with a project that requires a more than one (or two) server(s), yet, it’s easy to just do something like:

➜ mykitchenv2 git:(master) ✗ ls
Berksfile Berksfile.lock Gemfile Gemfile.lock Vagrantfile cookbooks data_bags environments nodes roles site-cookbooks

➜ roles git:(master) ✗ ls
lamp.json rack.json

➜ nodes git:(master) ✗ ls
127.0.0.1.json anotherhost.json …

So, it’s basically a general-purpose kitchen. It’s not really geared towards a specific application. My workflow looks like this:

  1. New project, hack, app ready to be deployed
  2. Create new VM @ digital ocean, grab the IP, create a new node json in the kitchen
  3. It’s probably a Rails app. Add the rack role[lamp] to the run_list for this node. The rack role basically setups a new rack box using the rackbox cookbook (GitHub - teohm/rackbox-cookbook: Setup a Rack-based application server to run Unicorn & Passenger apps.)
  4. knife solo prepare root@newvm_ip
  5. knife solo cook root@newvm_ip
  6. Use capistrano to deploy the app.

Then, I do the same thing if I need a staging server.

It works pretty well. However, most of the apps I develop right now are apps with a quite simple infrastructure. They don’t require more than 2 servers (prod,staging). I did work with more elaborated apps before with up to 10 different servers (but without using Chef at the time, go figure).

The problem with this approach is that I end up having several unrelated nodes, and in the end, I don’t really describe the infrastructure of a single app/system, but of several ones, which can become clunky. Basically, for now, I’m just using Chef to get the sever ready, but not really using it to manage the server afterwards.

That being said, I’d like some insights on Chef patterns related to this, for example, when is it time to create a kitchen solely for an app?

Thanks in advance,

--
Marcelo

The whole roles vs cookbooks thing is an ongoing discussion to be sure. But
last I checked roles don't have any sort of versioning support which can
them a bit of a liability when you start sharing them between projects or
apps. It was covered pretty well on episode 36 of the foodfight show so
I'll point you there: Roles, Environments, Attributes, and Data Bags - Food Fight

The win for really using chef is partly in the "scaling out" but it also
helps when you need to replicate your current stack. For example, imagine
that DO machine were somehow lost and you needed to start over again. Any
manual changes or updates would now have to get redone but if you've been
managing with chef it should be easy to recreate everything again.

Personally, if I needed a basic Rails PaaS, I'd go with heroku. The
run-your-own PaaS is interesting, but I'm not sure it's worth the overhead
for personal use. I could see doing that if you needed a place for a dev
team to run internal tooling. Mesos is also interesting there but I haven't
really tried it yet.

As for mixing the kitchen with the project, I don't see any issue with
that. Might be a little messy in terms of directory structure though.
Another option would be to have a "deployment" or "chef" directory and put
everything in there. I've done that in the past too.

-Mat

On Sun, Feb 2, 2014 at 4:06 PM, Marcelo Serpa mlists@fullofcaffeine.comwrote:

Hi Mat,

Thank you for taking the time to answer. I appreciate it.

This also helps avoid loading too much info into roles which is another
good practice.

Interesting… could you elaborate on this one? An example would be nice :slight_smile:

Finally, you should try to get into the practice of using chef to
provision as well as manage the servers since it's where configuration
"management" really starts to shine.

Excellent point. Now I see that the main reason why I haven’t been using a
kitchen per project, is that I’ve been using Chef only to provision servers
(hence, we could call that a provision kitchen ;), and not manage them. For
me, so far, it was the most painful SCM aspect. Since my apps’ don’t really
change a lot when it comes to infrastructure, it has been enough to do
those changes by hand. So far, it has been enough to scale vertically
(basically expanding the RAM of a single node), which I can do easily on
digitalocean (or AWS or any cloud VM provider ).

By the way, how would this work with knife solo? Or do you think that when
the infrastructure becomes complex to the point where Chef is useful to
manage it, then it’s time to add a Chef server to the equation?

Also, I’ve been playing with PaaS solutions lately. One of them is Tsuru,
written in Go, and Dokku, which is pretty easy to setup on digital ocean.
Perhaps the best combo for apps that don’t require a very complex
infrastructure, would be to use Chef to provision, say, Dokku and also
setup the necessary plugins (for db access, for example), and then just use
Dokku from then on. I think this could work for simpler apps, but as soon
as you need more server side components, then it becomes a burden to deal
with PaaS plugins. What do you think about this?

Lastly, do you think it would be feasible to have the kitchen integrated
in the app project itself? Let’s say I have an app built on Rails:

$ rails new test

Then, create the kitchen inside the app’s dir"

$ cd test
$ knife solo init .

Ending up with:

➜ ls
Gemfile app config.ru db log
roles tmp
README.rdoc bin cookbooks environments nodes
site-cookbooks vendor
Rakefile config data_bags lib public
test

So, we have the app, and the code that does all the SCM (potentially
adding Capistrano to the mix).


Marcelo

On Sunday, February 2, 2014 at 5:13 PM, Mat Schaffer wrote:

"It depends" :wink:

When I was doing client work I generally kept one kitchen per client just
to make the final delivery as clean as possible. For myself I tend to keep
kitchens logically separate by project.

If you need to share stuff between kitchens I recommend pushing all that
stuff into cookbooks so that you can use berkshelf or librarian-chef to
load in the shared cookbooks from their own dedicated git repositories.
This also helps avoid loading too much info into roles which is another
good practice.

Finally, you should try to get into the practice of using chef to
provision as well as manage the servers since it's where configuration
"management" really starts to shine. Of course there are other options but
if you're on a lamp stack, generating configuration files and/or data from
chef will probably be the easiest to set up.

Hope that helps some. I think this is one of those things that varies a
bit from project to project.

Feel free to open a knife-solo issue if you run into trouble there.

-Mat

Mat Schaffer - Gifu Japan | about.me

On Sun, Feb 2, 2014 at 12:00 AM, Marcelo Serpa mlists@fullofcaffeine.comwrote:

Hi list,

I’m wondering Chef kitchen patterns. Right now, I use knife-solo, and I
have a general-purpose kitchen. I add nodes as needed (not necessarily for
the same app). Since I don’t really deal with a project that requires a
more than one (or two) server(s), yet, it’s easy to just do something like:

➜ mykitchenv2 git:(master) ✗ ls
Berksfile Berksfile.lock Gemfile Gemfile.lock Vagrantfile
cookbooks data_bags environments nodes roles
site-cookbooks

➜ roles git:(master) ✗ ls
lamp.json rack.json

➜ nodes git:(master) ✗ ls
127.0.0.1.json anotherhost.json …

So, it’s basically a general-purpose kitchen. It’s not really geared
towards a specific application. My workflow looks like this:

  1. New project, hack, app ready to be deployed
  2. Create new VM @ digital ocean, grab the IP, create a new node json in
    the kitchen
  3. It’s probably a Rails app. Add the rack role[lamp] to the run_list for
    this node. The rack role basically setups a new rack box using the rackbox
    cookbook (GitHub - teohm/rackbox-cookbook: Setup a Rack-based application server to run Unicorn & Passenger apps.)
  4. knife solo prepare root@newvm_ip
  5. knife solo cook root@newvm_ip
  6. Use capistrano to deploy the app.

Then, I do the same thing if I need a staging server.

It works pretty well. However, most of the apps I develop right now are
apps with a quite simple infrastructure. They don’t require more than 2
servers (prod,staging). I did work with more elaborated apps before with up
to 10 different servers (but without using Chef at the time, go figure).

The problem with this approach is that I end up having several unrelated
nodes, and in the end, I don’t really describe the infrastructure of a
single app/system, but of several ones, which can become clunky. Basically,
for now, I’m just using Chef to get the sever ready, but not really using
it to manage the server afterwards.

That being said, I’d like some insights on Chef patterns related to this,
for example, when is it time to create a kitchen solely for an app?

Thanks in advance,

--
Marcelo