Hello,
I’ve been mucking around with Chef for a week or so. It’s pretty
impressive what I’m seeing so far. Coming from LAMP + Perl land (think
traditional sysadmin stuff) , the learning curve for Chef is a bit
steep, but I’m learning. 
I’m interested in using Chef with about 100-200 Linux nodes, and we have
about 10-ish types of hosts (web servers, API, backend, etc). We’ve been
setting up our CentOS servers with Kickstart, then apply shell scripts
to automate deployment.
We want to be able to do something similar with Chef - the goal being,
kickstart a host, then install Chef RPMs, then the host should just
automagically get configured according to its IP, hostname, or some kind
of identifying criteria. Any ideas on how to do this?
So far, the workflow that Chef expects is:
- install chef-client on node.
- chef-client auto-registers itself with either validation cert or
password
- Use knife to apply role(s) to the newly created node
It’s the manual application of role(s) to the newly created node that I
would like to see automated. I don’t think there’s a way in Chef to set
a rule saying, “if a host is in 10.1.12.* subnet, apply role[a],
role[b], role[z], etc”. If I can figure out this last step, life would
be just awesome.
Thanks in advance for your help.
-Paul
Adding a bit more...
One idea I have is, if it's possible:
After installing chef-client, the node should run a custom script that
does a POST to /nodes/:
{
"run_list": [
"role[a]",
"role[c]"
]
}
I don't know if a node has the right to modify its run_list or not.
On 5/14/10 11:11 AM, Paul Choi wrote:
Hello,
I've been mucking around with Chef for a week or so. It's pretty
impressive what I'm seeing so far. Coming from LAMP + Perl land (think
traditional sysadmin stuff) , the learning curve for Chef is a bit
steep, but I'm learning. 
I'm interested in using Chef with about 100-200 Linux nodes, and we
have about 10-ish types of hosts (web servers, API, backend, etc).
We've been setting up our CentOS servers with Kickstart, then apply
shell scripts to automate deployment.
We want to be able to do something similar with Chef - the goal being,
kickstart a host, then install Chef RPMs, then the host should just
automagically get configured according to its IP, hostname, or some
kind of identifying criteria. Any ideas on how to do this?
So far, the workflow that Chef expects is:
- install chef-client on node.
- chef-client auto-registers itself with either validation cert or
password
- Use knife to apply role(s) to the newly created node
It's the manual application of role(s) to the newly created node that
I would like to see automated. I don't think there's a way in Chef to
set a rule saying, "if a host is in 10.1.12.* subnet, apply role[a],
role[b], role[z], etc". If I can figure out this last step, life would
be just awesome.
Thanks in advance for your help.
-Paul
On Fri, May 14, 2010 at 11:27 AM, Paul Choi paulchoi@plaxo.com wrote:
Adding a bit more...
One idea I have is, if it's possible:
After installing chef-client, the node should run a custom script that does
a POST to /nodes/:
{
"run_list": [
"role[a]",
"role[c]"
]
}
You can pass a file containing JSON to chef-client with -j. This is
how I've bootstrapped vm-builder (Ubuntu JeOS) nodes using the
firstboot parameter.
Bryan
On May 14, 2010, at 11:11 AM, Paul Choi wrote:
We want to be able to do something similar with Chef - the goal being, kickstart a host, then install Chef RPMs, then the host should just automagically get configured according to its IP, hostname, or some kind of identifying criteria. Any ideas on how to do this?
I'm doing this, so it is possible.
So far, the workflow that Chef expects is:
- install chef-client on node.
- chef-client auto-registers itself with either validation cert or password
- Use knife to apply role(s) to the newly created node
This is what I do, except for the last part.
I have a separate piece of code that is driven by a data file that maps virtual hosts onto virtual servers onto physical machines. The code sets up the name servers, creates the machines through my hosting provider API and sets up the roles for the servers in chef. You can actually create the nodes before the client actually connects for the first time. The common key between the virtual and the physical is the hostname of the machine. I know what the machine name will be e.g 'webserver6' and the boot script names the machine that before the chef client code is installed.
It's the manual application of role(s) to the newly created node that I would like to see automated. I don't think there's a way in Chef to set a rule saying, "if a host is in 10.1.12.* subnet, apply role[a], role[b], role[z], etc". If I can figure out this last step, life would be just awesome.
You might be able to do this by writing some ruby code in a role file..... I'm not sure how the chef client determines which cookbooks to download from the server... it might be static.... others on the list can probably answer this.
However, I'd have thought that basing roles on ip addresses is going to remove some flexibility that you might need later on.
John
--
John Merrells
+1.415.244.5808
Ah, thank you! That's exactly what I need.
-Paul
On 5/14/10 11:34 AM, Bryan McLellan wrote:
On Fri, May 14, 2010 at 11:27 AM, Paul Choipaulchoi@plaxo.com wrote:
Adding a bit more...
One idea I have is, if it's possible:
After installing chef-client, the node should run a custom script that does
a POST to/nodes/:
{
"run_list": [
"role[a]",
"role[c]"
]
}
You can pass a file containing JSON to chef-client with -j. This is
how I've bootstrapped vm-builder (Ubuntu JeOS) nodes using the
firstboot parameter.
Bryan
On 5/14/10 11:37 AM, John Merrells wrote:
On May 14, 2010, at 11:11 AM, Paul Choi wrote:
We want to be able to do something similar with Chef - the goal being, kickstart a host, then install Chef RPMs, then the host should just automagically get configured according to its IP, hostname, or some kind of identifying criteria. Any ideas on how to do this?
I'm doing this, so it is possible.
So far, the workflow that Chef expects is:
- install chef-client on node.
- chef-client auto-registers itself with either validation cert or password
- Use knife to apply role(s) to the newly created node
This is what I do, except for the last part.
I have a separate piece of code that is driven by a data file that maps virtual hosts onto virtual servers onto physical machines. The code sets up the name servers, creates the machines through my hosting provider API and sets up the roles for the servers in chef. You can actually create the nodes before the client actually connects for the first time. The common key between the virtual and the physical is the hostname of the machine. I know what the machine name will be e.g 'webserver6' and the boot script names the machine that before the chef client code is installed.
That's good to know, thanks.
It's the manual application of role(s) to the newly created node that I would like to see automated. I don't think there's a way in Chef to set a rule saying, "if a host is in 10.1.12.* subnet, apply role[a], role[b], role[z], etc". If I can figure out this last step, life would be just awesome.
You might be able to do this by writing some ruby code in a role file..... I'm not sure how the chef client determines which cookbooks to download from the server... it might be static.... others on the list can probably answer this.
However, I'd have thought that basing roles on ip addresses is going to remove some flexibility that you might need later on.
Since our servers are not in the cloud, it's actually good for us to tie
stuff to host IP. The firewall allows access based on IP, and physical
server specs are different based on their role.
As for cookbooks, I might be misunderstanding what you are saying, but
after applying the correct roles to a node, recipes are applied
according to the run list. I think that's what I'm looking for, unless
there's more flexibility with downloading cookbooks that you might want
to tell me about.
John
On May 14, 2010, at 12:00 PM, Paul Choi wrote:
Since our servers are not in the cloud, it's actually good for us to tie stuff to host IP. The firewall allows access based on IP, and physical server specs are different based on their role.
Ack.
As for cookbooks, I might be misunderstanding what you are saying, but after applying the correct roles to a node, recipes are applied according to the run list. I think that's what I'm looking for,
Yes, that's it.
unless there's more flexibility with downloading cookbooks that you might want to tell me about.
I don't think there is.
John
--
John Merrells
+1.415.244.5808
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hello!
On May 14, 2010, at 12:37 PM, John Merrells wrote:
You might be able to do this by writing some ruby code in a role
file..... I'm not sure how the chef client determines which
cookbooks to download from the server... it might be static....
others on the list can probably answer this.
The server sends the client the cookbooks mentioned in the run_list,
and the dependencies those cookbooks have based on their Metadata. If
you want to use the libraries, resources, providers or definitions
between cookbooks, you need to specify dependencies in the metadata.
Likewise if you include_recipe, you have to make sure the included
cookbook is in the metadata.
http://wiki.opscode.com/display/chef/Metadata
Opscode, Inc
Joshua Timberman, Senior Solutions Engineer
C: 720.334.RUBY E: joshua@opscode.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (Darwin)
iEYEARECAAYFAkvtpb4ACgkQO97WSdVpzT1ckgCcCSwO8PPvzG0lRhh2iacuevJj
hlIAn1KeLFYF1iyDUNRTS0v2rkssTVoA
=MG+T
-----END PGP SIGNATURE-----
On Fri, May 14, 2010 at 7:37 PM, John Merrells john@merrells.com wrote:
I have a separate piece of code that is driven by a data file that maps virtual hosts onto virtual servers onto physical machines. The code sets up the name servers, creates the machines through my hosting provider API and sets up the roles for the servers in chef. You can
Any opportunity to see this script? It sounds worth looking at.
actually create the nodes before the client actually connects for the
first time. The common key between the virtual and the physical is the
hostname of the machine. I know what the machine name will be e.g
'webserver6' and the boot script names the machine that before the
chef client code is installed.
It's the manual application of role(s) to the newly created node that I would like to see automated. I don't think there's a way in Chef to set a rule saying, "if a host is in 10.1.12.* subnet, apply role[a], role[b], role[z], etc". If I can figure out this last step, life would be just awesome.
You might be able to do this by writing some ruby code in a role file..... I'm not sure how the chef client determines which cookbooks to download from the server... it might be static.... others on the list can probably answer this.
However, I'd have thought that basing roles on ip addresses is going to remove some flexibility that you might need later on.
John
--
John Merrells
http://johnmerrells.com
+1.415.244.5808
On May 14, 2010, at 12:48 PM, dreamcat four wrote:
I have a separate piece of code that is driven by a data file that maps virtual hosts onto virtual servers onto physical machines. The code sets up the name servers, creates the machines through my hosting provider API and sets up the roles for the servers in chef. You can
Any opportunity to see this script? It sounds worth looking at.
It's not in a sharable state right now, as it's a bit intertwined with other things, kinda specific to my requirements, and also has a work around for a bug in the hosting provider api..... so it's still a work in progress.... I'll keep it in mind though.
Is anyone else using linode?
John
--
John Merrells
+1.415.244.5808
For what it’s worth, I’ve been using PoolParty (http://auser.github.com/poolparty/
) to define multi-machine deployments on EC2 and I’ve found that it
works pretty well. There have been a few quirks but it’s under active
development.
Cheers!
-Cameorn
Cameron Pope
camerooni@gmail.com
Ouch, old thread, sorry.
What we have at peerpong is a very simple boot.sh sitting on the
image. You still have to log in, run boot.sh -> this registers the
node on the chef server. then go on the server UI, apply roles to the
new node.
I guess it would not be difficult to apply roles for the new machine
from boot.sh using knife though. Command line would look like
./boot.sh [database|app||...]
Might spend some time on that tomorrow actually
--Gilles
On Sun, May 16, 2010 at 6:41 PM, Cameron Pope camerooni@gmail.com wrote:
For what it's worth, I've been using PoolParty
(http://auser.github.com/poolparty/) to define multi-machine deployments on
EC2 and I've found that it works pretty well. There have been a few quirks
but it's under active development.
Cheers!
-Cameorn
Cameron Pope
camerooni@gmail.com
Hi Gilles,
Actually, I'm really interested in what others have to say about this
idea, so thanks for your input.
Bryan McLellan responded saying you can do "chef-client -j "
to apply a role from a node.
So, in my kickstart script's %post section, I have:
curl http:///cgi-bin/chef_role.cgi? > /tmp/role.json
chef-client -j /tmp/role.json
And chef_role.cgi is just a little perl script I wrote that takes the
hostname and spits out role/cluster assignment.
I did think about maybe setting up knife on each host. Or using
Chef::REST with an admin user's client PEM cert on the node, but I
thought I'll just do "chef-client -j", since it at least works for me.
-Paul
On 5/20/10 10:34 PM, Gilles Devaux wrote:
Ouch, old thread, sorry.
What we have at peerpong is a very simple boot.sh sitting on the
image. You still have to log in, run boot.sh -> this registers the
node on the chef server. then go on the server UI, apply roles to the
new node.
I guess it would not be difficult to apply roles for the new machine
from boot.sh using knife though. Command line would look like
./boot.sh [database|app||...]
Might spend some time on that tomorrow actually
--Gilles
On Sun, May 16, 2010 at 6:41 PM, Cameron Popecamerooni@gmail.com wrote:
For what it's worth, I've been using PoolParty
(http://auser.github.com/poolparty/) to define multi-machine deployments on
EC2 and I've found that it works pretty well. There have been a few quirks
but it's under active development.
Cheers!
-Cameorn
Cameron Pope
camerooni@gmail.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hello!
On May 20, 2010, at 11:34 PM, Gilles Devaux wrote:
I guess it would not be difficult to apply roles for the new machine
from boot.sh using knife though. Command line would look like
./boot.sh [database|app||...]
You might look at what we have for supporting rackspace and terremark
server creation with knife here:
http://github.com/opscode/chef/blob/master/chef/lib/chef/knife/rackspace_server_create.rb
http://github.com/opscode/chef/blob/master/chef/lib/chef/knife/terremark_server_create.rb
Opscode, Inc
Joshua Timberman, Senior Solutions Engineer
C: 720.334.RUBY E: joshua@opscode.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (Darwin)
iEYEARECAAYFAkv2ImMACgkQO97WSdVpzT3S5QCfZRFA64aWinBmw+V75hH+hdK5
JrYAnj0r3w+4BZvtGEV6IB/6XRJ8O9UR
=zmcY
-----END PGP SIGNATURE-----
Now this is awesome!
I can easily build on that.
Thanks a lot
--Gilles
On Thu, May 20, 2010 at 11:04 PM, Joshua Timberman joshua@opscode.com wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hello!
On May 20, 2010, at 11:34 PM, Gilles Devaux wrote:
I guess it would not be difficult to apply roles for the new machine
from boot.sh using knife though. Command line would look like
./boot.sh [database|app||...]
You might look at what we have for supporting rackspace and terremark server
creation with knife here:
http://github.com/opscode/chef/blob/master/chef/lib/chef/knife/rackspace_server_create.rb
http://github.com/opscode/chef/blob/master/chef/lib/chef/knife/terremark_server_create.rb
Opscode, Inc
Joshua Timberman, Senior Solutions Engineer
C: 720.334.RUBY E: joshua@opscode.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (Darwin)
iEYEARECAAYFAkv2ImMACgkQO97WSdVpzT3S5QCfZRFA64aWinBmw+V75hH+hdK5
JrYAnj0r3w+4BZvtGEV6IB/6XRJ8O9UR
=zmcY
-----END PGP SIGNATURE-----