How to deploy to nodes synchronously?


#1

I’m new to chef and still trying to understand how it works. I searched the
FAQ and couldn’t find the answer to this. Please forgive me if the question
has been answered before.

Let’s say I have 100 nodes running chef-client daemon. If I update the
config for those nodes in chef-server my understanding is that all those
chef-clients will detect the changes for its own node and go ahead to do
whatever the recipes say to do. How can I control the process
synchronously. I don’t want all my 100 nodes updating at the same time
because the update process may require restarting Apache. I know one way to
do it is to run the chef-client in non-daemon mode. e.g. run “knife ssh
chef-client” on nodes one by one. I’m wondering is it possible to
still run the chef-client in daemon mode but control the process on server
side so that the update happens one node at a time? Maybe I can update the
setting one node at a time. But how can I know the chef-client on a
particular node is done so that I can move to update the setting for next
node?

Thanks

Irvin


#2

You can use the -s option when calling chef-client:

$ chef-client -h
Usage: chef-client (options)


-s, --splay SECONDS The splay time for running at intervals, in seconds

On Monday, July 30, 2012 at 4:20 PM, Irvin Fan wrote:

I’m new to chef and still trying to understand how it works. I searched the FAQ and couldn’t find the answer to this. Please forgive me if the question has been answered before.

Let’s say I have 100 nodes running chef-client daemon. If I update the config for those nodes in chef-server my understanding is that all those chef-clients will detect the changes for its own node and go ahead to do whatever the recipes say to do. How can I control the process synchronously. I don’t want all my 100 nodes updating at the same time because the update process may require restarting Apache. I know one way to do it is to run the chef-client in non-daemon mode. e.g. run “knife ssh chef-client” on nodes one by one. I’m wondering is it possible to still run the chef-client in daemon mode but control the process on server side so that the update happens one node at a time? Maybe I can update the setting one node at a time. But how can I know the chef-client on a particular node is done so that I can move to update the setting for next node?

Thanks

Irvin


#3

Hi Daniel,

Thanks for the tip. I think it works even though I was looking for a more
precise control.

Irvin

On Mon, Jul 30, 2012 at 4:23 PM, Daniel Condomitti daniel@condomitti.comwrote:

You can use the -s option when calling chef-client:

$ chef-client -h
Usage: chef-client (options)

-s, --splay SECONDS The splay time for running at
intervals, in seconds

On Monday, July 30, 2012 at 4:20 PM, Irvin Fan wrote:

I’m new to chef and still trying to understand how it works. I searched
the FAQ and couldn’t find the answer to this. Please forgive me if the
question has been answered before.

Let’s say I have 100 nodes running chef-client daemon. If I update the
config for those nodes in chef-server my understanding is that all those
chef-clients will detect the changes for its own node and go ahead to do
whatever the recipes say to do. How can I control the process
synchronously. I don’t want all my 100 nodes updating at the same time
because the update process may require restarting Apache. I know one way to
do it is to run the chef-client in non-daemon mode. e.g. run “knife ssh
chef-client” on nodes one by one. I’m wondering is it possible to
still run the chef-client in daemon mode but control the process on server
side so that the update happens one node at a time? Maybe I can update the
setting one node at a time. But how can I know the chef-client on a
particular node is done so that I can move to update the setting for next
node?

Thanks

Irvin


#4

Another option is to use the knife command with a search that will find and all the nodes, setting concurrency to some number less than your total nodes.

For example:

knife ssh “role:app_server” “sudo chef-client” -C 10

Will call chef-client on all nodes that have the “app_server” role, executing only 50 at a time. This guarantees that no more than 10 nodes will ever be mid-deploy during execution.


Robby Grossman

http://rob.by (http://rob.by/)

On Monday, July 30, 2012 at 8:26 PM, Irvin Fan wrote:

Hi Daniel,

Thanks for the tip. I think it works even though I was looking for a more precise control.

Irvin

On Mon, Jul 30, 2012 at 4:23 PM, Daniel Condomitti <daniel@condomitti.com (mailto:daniel@condomitti.com)> wrote:

You can use the -s option when calling chef-client:

$ chef-client -h
Usage: chef-client (options)


-s, --splay SECONDS The splay time for running at intervals, in seconds

On Monday, July 30, 2012 at 4:20 PM, Irvin Fan wrote:

I’m new to chef and still trying to understand how it works. I searched the FAQ and couldn’t find the answer to this. Please forgive me if the question has been answered before.

Let’s say I have 100 nodes running chef-client daemon. If I update the config for those nodes in chef-server my understanding is that all those chef-clients will detect the changes for its own node and go ahead to do whatever the recipes say to do. How can I control the process synchronously. I don’t want all my 100 nodes updating at the same time because the update process may require restarting Apache. I know one way to do it is to run the chef-client in non-daemon mode. e.g. run “knife ssh chef-client” on nodes one by one. I’m wondering is it possible to still run the chef-client in daemon mode but control the process on server side so that the update happens one node at a time? Maybe I can update the setting one node at a time. But how can I know the chef-client on a particular node is done so that I can move to update the setting for next node?

Thanks

Irvin


#5

On Monday, July 30, 2012 at 4:20 PM, Irvin Fan wrote:

I’m new to chef and still trying to understand how it works. I searched the FAQ and couldn’t find the answer to this. Please forgive me if the question has been answered before.

Let’s say I have 100 nodes running chef-client daemon. If I update the config for those nodes in chef-server my understanding is that all those chef-clients will detect the changes for its own node and go ahead to do whatever the recipes say to do. How can I control the process synchronously. I don’t want all my 100 nodes updating at the same time because the update process may require restarting Apache. I know one way to do it is to run the chef-client in non-daemon mode. e.g. run “knife ssh chef-client” on nodes one by one. I’m wondering is it possible to still run the chef-client in daemon mode but control the process on server side so that the update happens one node at a time? Maybe I can update the setting one node at a time. But how can I know the chef-client on a particular node is done so that I can move to update the setting for next node?

Thanks

Irvin
There are a lot of different ways to handle this, depending on what your requirements are.

One way is the data-driven approach, where you pre-partition your fleet into smaller clusters, which you assign by role or node attribute. Then, you drive any config changes which require app restarts using data in roles or data bags, with the application version (or package that installs the app, whatever) set per-cluster. On a small scale, you might have something like:

Node Attributes:
node1[“my_app”][“cluster_number”] == 0
node2[“my_app”][“cluster_number”] == 1

Data Bag Item:
“cluster_0_app_version”: "1.2.3"
“cluster_1_app_version”: “1.2.2”

Then your recipes would figure out the version of the app to run based on their cluster assignment, which should be a no-op if you’re on the correct version already.

The only option for getting feedback about chef-client success or failure at the moment is to use exception/report handlers[0].

  1. http://wiki.opscode.com/display/chef/Exception+and+Report+Handlers


Dan DeLeo


#6

Correction, the command I mentioned would execute 10 at a time, not 50; the -C flag sets concurrency:

knife ssh “role:app_server” “sudo chef-client” -C 10


Robby Grossman
@freerobby (http://twitter.com/freerobby)
http://rob.by (http://rob.by/)

On Monday, July 30, 2012 at 8:32 PM, Robby Grossman wrote:

Another option is to use the knife command with a search that will find and all the nodes, setting concurrency to some number less than your total nodes.

For example:

knife ssh “role:app_server” “sudo chef-client” -C 10

Will call chef-client on all nodes that have the “app_server” role, executing only 50 at a time. This guarantees that no more than 10 nodes will ever be mid-deploy during execution.


Robby Grossman
http://rob.by (http://rob.by/)

On Monday, July 30, 2012 at 8:26 PM, Irvin Fan wrote:

Hi Daniel,

Thanks for the tip. I think it works even though I was looking for a more precise control.

Irvin

On Mon, Jul 30, 2012 at 4:23 PM, Daniel Condomitti <daniel@condomitti.com (mailto:daniel@condomitti.com)> wrote:

You can use the -s option when calling chef-client:

$ chef-client -h
Usage: chef-client (options)


-s, --splay SECONDS The splay time for running at intervals, in seconds

On Monday, July 30, 2012 at 4:20 PM, Irvin Fan wrote:

I’m new to chef and still trying to understand how it works. I searched the FAQ and couldn’t find the answer to this. Please forgive me if the question has been answered before.

Let’s say I have 100 nodes running chef-client daemon. If I update the config for those nodes in chef-server my understanding is that all those chef-clients will detect the changes for its own node and go ahead to do whatever the recipes say to do. How can I control the process synchronously. I don’t want all my 100 nodes updating at the same time because the update process may require restarting Apache. I know one way to do it is to run the chef-client in non-daemon mode. e.g. run “knife ssh chef-client” on nodes one by one. I’m wondering is it possible to still run the chef-client in daemon mode but control the process on server side so that the update happens one node at a time? Maybe I can update the setting one node at a time. But how can I know the chef-client on a particular node is done so that I can move to update the setting for next node?

Thanks

Irvin


#7

If you prefer to just have chef-clients running
in daemon mode all the time, then this may be
a good way to ensure that your core services do
not get restarted all at once:

On Mon, 30 Jul 2012 16:20:37 -0700
Irvin Fan irvin@corp.oodle.com wrote:

I’m new to chef and still trying to understand how it works. I
searched the FAQ and couldn’t find the answer to this. Please forgive
me if the question has been answered before.

Let’s say I have 100 nodes running chef-client daemon. If I update the
config for those nodes in chef-server my understanding is that all
those chef-clients will detect the changes for its own node and go
ahead to do whatever the recipes say to do. How can I control the
process synchronously. I don’t want all my 100 nodes updating at the
same time because the update process may require restarting Apache. I
know one way to do it is to run the chef-client in non-daemon mode.
e.g. run “knife ssh chef-client” on nodes one by one. I’m
wondering is it possible to still run the chef-client in daemon mode
but control the process on server side so that the update happens one
node at a time? Maybe I can update the setting one node at a time.
But how can I know the chef-client on a particular node is done so
that I can move to update the setting for next node?

Thanks

Irvin