Hi,
We have a requirement soon to add several hundred nodes to our OpenSource Chef server. My question is: does anyone know a good way to handle this?
I’ve googled quite a lot and cannot find much info about do a ‘mass bootstrapping’.
I will have a list of Environments and the nodes contained within those environments, and I suspect with a little ruby code I could get a script together to parse a file containing this information and bootstrap nodes appropriately, but this seems a little inelegant.
Are there tools out there to help? Anyone devise a good method to do this?
We have a requirement soon to add several hundred nodes to our OpenSource Chef server. My question is: does anyone know a good way to handle this?
I’ve googled quite a lot and cannot find much info about do a ‘mass bootstrapping’.
I will have a list of Environments and the nodes contained within those environments, and I suspect with a little ruby code I could get a script together to parse a file containing this information and bootstrap nodes appropriately, but this seems a little inelegant.
Are there tools out there to help? Anyone devise a good method to do this?
populate a validation.pem, and a client.rb on each node.
Install chef-client on the nodes en mass (should be trivial task for linux
or windows)
This will read the client.rb, and validation.pem and connect to the chef
server.
IF you have recipes/roles you want to apply off the bat, you could generate
the nodes in the chef server. Then, when the node connects, it creates the
matching client, reads its run_list, and BOOM, right after the install is
done, when it tries to do chef-client, it will actually have a run_list.
We have a requirement soon to add several hundred nodes to our
OpenSource Chef server. My question is: does anyone know a good way to
handle this?
I’ve googled quite a lot and cannot find much info about do a ‘mass
bootstrapping’.
I will have a list of Environments and the nodes contained within those
environments, and I suspect with a little ruby code I could get a script
together to parse a file containing this information and bootstrap nodes
appropriately, but this seems a little inelegant.
Are there tools out there to help? Anyone devise a good method to do
this?
populate a validation.pem, and a client.rb on each node.
Install chef-client on the nodes en mass (should be trivial task for linux
or windows)
This will read the client.rb, and validation.pem and connect to the chef
server.
IF you have recipes/roles you want to apply off the bat, you could
generate the nodes in the chef server. Then, when the node connects, it
creates the matching client, reads its run_list, and BOOM, right after the
install is done, when it tries to do chef-client, it will actually have a
run_list.
We have a requirement soon to add several hundred nodes to our
OpenSource Chef server. My question is: does anyone know a good way to
handle this?
I’ve googled quite a lot and cannot find much info about do a ‘mass
bootstrapping’.
I will have a list of Environments and the nodes contained within those
environments, and I suspect with a little ruby code I could get a script
together to parse a file containing this information and bootstrap nodes
appropriately, but this seems a little inelegant.
Are there tools out there to help? Anyone devise a good method to do
this?
i use ansible or knife-ssh (as api, not as cli). bootstrapping has always
been pain, because its ssh based, and also i needed ordered bootstrapping
(ops services -> persistence layer -> backends -> frontends -> load
balancers).
spice weasel is awesome if you can model everything using knife cli.
if you want to parallelize certain steps. retry , callbacks then ansible
can do those (like send a failure metric to statds if certain step fails).
first use case (parallel execution) can be done with with yaml only
interface, callbacks require python.
chef-metal is new kid in this zone . its super alpha but it lets you do
all of these assuming you are running a controller node somewhere (may be
your workstation) and environment is modeled as chef recipe (chef-metal
provides machine as a resource).
note: knife-ssh is also fairly easy to extend, and you can build custom
knife extensions for your bootstrapping needs.
We have a requirement soon to add several hundred nodes to our
OpenSource Chef server. My question is: does anyone know a good way to
handle this?
I’ve googled quite a lot and cannot find much info about do a ‘mass
bootstrapping’.
I will have a list of Environments and the nodes contained within those
environments, and I suspect with a little ruby code I could get a script
together to parse a file containing this information and bootstrap nodes
appropriately, but this seems a little inelegant.
Are there tools out there to help? Anyone devise a good method to do
this?
I think the best way to accomplish this is to trigger the bootstrap from
the node itself, rather than spawn off a ton of knife commands en masse.
chef-client 11.12.x supports the -r parameter, which allows you to specify
the run list.
I achieve this in my projects by creating a template that has an init
script that triggers the chef-client run. A gist of what I use is at GitHub - ameir/chef-initstrap: Bootstrap Chef nodes via an init script if you want to adapt it for your
needs. It's been working well for us thus far.
We have a requirement soon to add several hundred nodes to our
OpenSource Chef server. My question is: does anyone know a good way to
handle this?
I’ve googled quite a lot and cannot find much info about do a ‘mass
bootstrapping’.
I will have a list of Environments and the nodes contained within those
environments, and I suspect with a little ruby code I could get a script
together to parse a file containing this information and bootstrap nodes
appropriately, but this seems a little inelegant.
Are there tools out there to help? Anyone devise a good method to do
this?
who triggers chef-initstrap ? how you spawn a cluster of say 10 nodes? what
if some them requires different runlist, currently its hard coded in your
script.
On Wed, May 21, 2014 at 12:34 PM, ameirh . ameirh@gmail.com wrote:
I think the best way to accomplish this is to trigger the bootstrap from
the node itself, rather than spawn off a ton of knife commands en masse.
chef-client 11.12.x supports the -r parameter, which allows you to specify
the run list.
I achieve this in my projects by creating a template that has an init
script that triggers the chef-client run. A gist of what I use is at GitHub - ameir/chef-initstrap: Bootstrap Chef nodes via an init script if you want to adapt it for your
needs. It's been working well for us thus far.
We have a requirement soon to add several hundred nodes to our
OpenSource Chef server. My question is: does anyone know a good way to
handle this?
I’ve googled quite a lot and cannot find much info about do a ‘mass
bootstrapping’.
I will have a list of Environments and the nodes contained within those
environments, and I suspect with a little ruby code I could get a script
together to parse a file containing this information and bootstrap nodes
appropriately, but this seems a little inelegant.
Are there tools out there to help? Anyone devise a good method to do
this?
who triggers chef-initstrap ? how you spawn a cluster of say 10 nodes?
what if some them requires different runlist, currently its hard coded in
your script.
On Wed, May 21, 2014 at 12:34 PM, ameirh . ameirh@gmail.com wrote:
I think the best way to accomplish this is to trigger the bootstrap from
the node itself, rather than spawn off a ton of knife commands en masse.
chef-client 11.12.x supports the -r parameter, which allows you to specify
the run list.
I achieve this in my projects by creating a template that has an init
script that triggers the chef-client run. A gist of what I use is at https://github.com/ameir/chef-initstrap if you want to adapt it for your
needs. It's been working well for us thus far.
We have a requirement soon to add several hundred nodes to our
OpenSource Chef server. My question is: does anyone know a good way to
handle this?
I’ve googled quite a lot and cannot find much info about do a ‘mass
bootstrapping’.
I will have a list of Environments and the nodes contained within those
environments, and I suspect with a little ruby code I could get a script
together to parse a file containing this information and bootstrap nodes
appropriately, but this seems a little inelegant.
Are there tools out there to help? Anyone devise a good method to do
this?
Ranjib, if you have a template and install the file on the 10 nodes, they
will each be bootstrapped at boot time. You would place the file in
/etc/init.d, chmod +x it, and enable it via chkconfig/update-rc.d in your
template. You can also run it manually with service chef-initstrap start.
If all of your nodes are identical, this would work well, but if you have
different run lists, you'd have to rinse and repeat this process if you
keep the script as-is. You could expand it further to, for example, use
AWS user data/metadata, or contact some other external source for the
config options; they don't necessarily have to be defined in the script.
It's pretty much as flexible as your coding abilities will allow.
who triggers chef-initstrap ? how you spawn a cluster of say 10 nodes?
what if some them requires different runlist, currently its hard coded in
your script.
On Wed, May 21, 2014 at 12:34 PM, ameirh . ameirh@gmail.com wrote:
I think the best way to accomplish this is to trigger the bootstrap from
the node itself, rather than spawn off a ton of knife commands en masse.
chef-client 11.12.x supports the -r parameter, which allows you to specify
the run list.
I achieve this in my projects by creating a template that has an init
script that triggers the chef-client run. A gist of what I use is at https://github.com/ameir/chef-initstrap if you want to adapt it for your
needs. It's been working well for us thus far.
We have a requirement soon to add several hundred nodes to our
OpenSource Chef server. My question is: does anyone know a good way to
handle this?
I’ve googled quite a lot and cannot find much info about do a ‘mass
bootstrapping’.
I will have a list of Environments and the nodes contained within those
environments, and I suspect with a little ruby code I could get a script
together to parse a file containing this information and bootstrap nodes
appropriately, but this seems a little inelegant.
Are there tools out there to help? Anyone devise a good method to do
this?
Kenneth, yes, the run list specified in -r persists (as in a node.save
occurs, and saves to the Chef server), as opposed to -o (which is used to
override the run list). This does require a recent Chef version, though (I
believe 11.12.x).
who triggers chef-initstrap ? how you spawn a cluster of say 10 nodes?
what if some them requires different runlist, currently its hard coded in
your script.
On Wed, May 21, 2014 at 12:34 PM, ameirh . ameirh@gmail.com wrote:
I think the best way to accomplish this is to trigger the bootstrap from
the node itself, rather than spawn off a ton of knife commands en masse.
chef-client 11.12.x supports the -r parameter, which allows you to specify
the run list.
I achieve this in my projects by creating a template that has an init
script that triggers the chef-client run. A gist of what I use is at GitHub - ameir/chef-initstrap: Bootstrap Chef nodes via an init script if you want to adapt it for
your needs. It's been working well for us thus far.
We have a requirement soon to add several hundred nodes to our
OpenSource Chef server. My question is: does anyone know a good way to
handle this?
I’ve googled quite a lot and cannot find much info about do a ‘mass
bootstrapping’.
I will have a list of Environments and the nodes contained within those
environments, and I suspect with a little ruby code I could get a script
together to parse a file containing this information and bootstrap nodes
appropriately, but this seems a little inelegant.
Are there tools out there to help? Anyone devise a good method to do
this?
Ameir, so you are manually doing what knife bootstrap already does? or the
reason is to pass this data via aws user data field? also how you spawn ec2
instances , because knife-ec2 can do what you are mentioning already (and
more).
on older versions you can pass -j /path/to/file.json where json file can
contain runlist key (value is used as runlist item) , knife bootstrap uses
the same trick to create a first-boot.json and invoke chef-client -j
/etc/chef/first-boot.json
I launch these instances via AWS autoscaling, so the goal is that I do not
have any manual involvement in any process of the provisioning or
bootstrapping process. knife-ec2 can provision instances, albeit manually
(unless you have something else triggering knife).
Using -j is perfect, but I didn’t want to mess with JSON in Bash. -r has
been working great for our needs, and I’m not aware of any downsides to it
versus -j. If you can think of any, I’d love to hear them.
We have a requirement soon to add several hundred nodes to our OpenSource
Chef server. My question is: does anyone know a good way to handle this?
I’ve googled quite a lot and cannot find much info about do a ‘mass
bootstrapping’.
I will have a list of Environments and the nodes contained within those
environments, and I suspect with a little ruby code I could get a script
together to parse a file containing this information and bootstrap nodes
appropriately, but this seems a little inelegant.
Are there tools out there to help? Anyone devise a good method to do this?
Thanks very much for the information guys, lots to look at.
I think spiceweasel or a ruby script might be the better option for us.
I’ve missed off my original email some pertinent information.
The nodes I want to strap already exist and they are all Windows servers, I’m using WinRM to bootstrap the test nodes but there require some changes to the WinRM settings, but that is something we can script changes for.
Hi Ranjib,
I launch these instances via AWS autoscaling, so the goal is that I do not have any manual involvement in any process of the provisioning or bootstrapping process. knife-ec2 can provision instances, albeit manually (unless you have something else triggering knife).
Using -j is perfect, but I didn’t want to mess with JSON in Bash. -r has been working great for our needs, and I’m not aware of any downsides to it versus -j. If you can think of any, I’d love to hear them.
For what it’s worth, I’ve done some work like this using cloud-init and handing each node a shell script that has what run-list and other parameters I want for each node. You can take your pseudocode for such a script directly from knife’s bootstrap steps if you like.
From: Ranjib Dey [dey.ranjib@gmail.com]
Sent: Wednesday, May 21, 2014 3:07 PM
To: chef@lists.opscode.com
Subject: [chef] Re: Re: Bootstrapping a lot of nodes
who triggers chef-initstrap ? how you spawn a cluster of say 10 nodes? what if some them requires different runlist, currently its hard coded in your script.
On Wed, May 21, 2014 at 12:34 PM, ameirh . <ameirh@gmail.commailto:ameirh@gmail.com> wrote:
I think the best way to accomplish this is to trigger the bootstrap from the node itself, rather than spawn off a ton of knife commands en masse. chef-client 11.12.x supports the -r parameter, which allows you to specify the run list.
I achieve this in my projects by creating a template that has an init script that triggers the chef-client run. A gist of what I use is at https://github.com/ameir/chef-initstrap if you want to adapt it for your needs. It’s been working well for us thus far.
Thanks,
Ameir
On Wed, May 21, 2014 at 1:00 PM, ChristopherHall@air-watch.commailto:ChristopherHall@air-watch.com <ChristopherHall@air-watch.commailto:ChristopherHall@air-watch.com> wrote:
Hi,
We have a requirement soon to add several hundred nodes to our OpenSource Chef server. My question is: does anyone know a good way to handle this?
I’ve googled quite a lot and cannot find much info about do a ‘mass bootstrapping’.
I will have a list of Environments and the nodes contained within those environments, and I suspect with a little ruby code I could get a script together to parse a file containing this information and bootstrap nodes appropriately, but this seems a little inelegant.
Are there tools out there to help? Anyone devise a good method to do this?
. its super alpha but it lets you do