# Simultaneous software upgrade on multiple nodes

**URL:** <https://discourse.chef.io/t/simultaneous-software-upgrade-on-multiple-nodes/2134>\
**Category:** Chef Infra (archive)\
**Created:** [October 22, 2011, 8:10am UTC](https://discourse.chef.io/t/simultaneous-software-upgrade-on-multiple-nodes/2134 "2011-10-22T08:10:40Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Anton\_Koldaev](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/anton_koldaev/32/91_2.png) [@Anton\_Koldaev](https://discourse.chef.io/u/Anton_Koldaev)\
**Post date:** [October 22, 2011, 8:10am UTC](https://discourse.chef.io/t/simultaneous-software-upgrade-on-multiple-nodes/2134/1 "2011-10-22T08:10:40Z")

</div>

Hi  
Let’s imagine we have 50 servers with and a distributed web application and  
we need to upgrade software on all of them. New software requires downtime  
while upgrade.  
If upgrade is performed simultaneously on all servers the application will  
be totally unavailable for this time. How to avoid it?

My suggestion is to use one of the following ways:

1. Manually run chef clients with knife for blocks of 10 servers in series.
2. Create some complex cookbook that search for how much servers are being  
upgraded at this moment and not run software upgrade recipe if there are  
already 10 servers in upgrade queue(chef attribute). At the end of upgrade  
recipe there will be some notification to start chef run on the next 10  
nodes if it was the last upgrade node upgrade(will search for  
node[:upgrade][:in\_process]). In this case there is no manual work - just  
change role version and run chef-client for all of 50 nodes, all upgrade  
logic will be in the recipes.

Which way is better? Maybe there are another great ways to perform partly  
upgrade?

–  
Best regards,  
Koldaev Anton

---

<div class="post-metadata">

**Author:** ![Andrea\_Campi](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/andrea_campi/32/514_2.png) [@Andrea\_Campi](https://discourse.chef.io/u/Andrea_Campi)\
**Post date:** [October 22, 2011, 8:28am UTC](https://discourse.chef.io/t/simultaneous-software-upgrade-on-multiple-nodes/2134/2 "2011-10-22T08:28:03Z")

</div>

I usually go the first way for simplicity--I keep a set of environments (production1..n) and assign nodes in a sensible fashion. Cookbooks are always pinned to a version; then I can just bump version one env at a time.

I have never needed your second approach, but it's an interesting idea, shouldn't be too hard to implement. But I like the total control of the first one.

On Oct 22, 2011, at 10:10 AM, Anton Koldaev [koldaevav@gmail.com](mailto:koldaevav@gmail.com) wrote:

> Hi  
> Let's imagine we have 50 servers with and a distributed web application and we need to upgrade software on all of them. New software requires downtime while upgrade.  
> If upgrade is performed simultaneously on all servers the application will be totally unavailable for this time. How to avoid it?
> 
> My suggestion is to use one of the following ways:
> 
> 1. Manually run chef clients with knife for blocks of 10 servers in series.
> 2. Create some complex cookbook that search for how much servers are being upgraded at this moment and not run software upgrade recipe if there are already 10 servers in upgrade queue(chef attribute). At the end of upgrade recipe there will be some notification to start chef run on the next 10 nodes if it was the last upgrade node upgrade(will search for node[:upgrade][:in\_process]). In this case there is no manual work - just change role version and run chef-client for all of 50 nodes, all upgrade logic will be in the recipes.
> 
> Which way is better? Maybe there are another great ways to perform partly upgrade?
> 
> --  
> Best regards,  
> Koldaev Anton

---

<div class="post-metadata">

**Author:** ![Brad\_Knowles](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/brad_knowles/32/716_2.png) [@Brad\_Knowles](https://discourse.chef.io/u/Brad_Knowles)\
**Post date:** [October 22, 2011, 1:55pm UTC](https://discourse.chef.io/t/simultaneous-software-upgrade-on-multiple-nodes/2134/3 "2011-10-22T13:55:31Z")

</div>

On Oct 22, 2011, at 3:10 AM, Anton Koldaev wrote:

> Let's imagine we have 50 servers with and a distributed web application and we need to upgrade software on all of them. New software requires downtime while upgrade.  
> If upgrade is performed simultaneously on all servers the application will be totally unavailable for this time. How to avoid it?

I don't have personal experience with it, but other Chef experts I've heard have talked about using "rundeck" to handle the orchestration of things like this. I would be very interested to hear what your thoughts would be on this software.

--  
Brad Knowles [bknowles@ihiji.com](mailto:bknowles@ihiji.com)  
SAGE Level IV, Chef Level 0.0.1

---

<div class="post-metadata">

**Author:** ![Rob\_Guttman](https://avatars.discourse-cdn.com/v4/letter/r/a9adbd/32.png) [@Rob\_Guttman](https://discourse.chef.io/u/Rob_Guttman)\
**Post date:** [October 22, 2011, 8:22pm UTC](https://discourse.chef.io/t/simultaneous-software-upgrade-on-multiple-nodes/2134/4 "2011-10-22T20:22:14Z")

</div>

If you use aws ec2 and Trac, I wrote a Trac plugin that does the multi-node serial orchestration for push-button deployments:

[http://trac-hacks.org/wiki/CloudPlugin](http://trac-hacks.org/wiki/CloudPlugin)

It's really just a thin webui wrapper to pychef and boto.

- Rob

On Oct 22, 2011, at 9:55 AM, Brad Knowles wrote:

> On Oct 22, 2011, at 3:10 AM, Anton Koldaev wrote:
> 
> > Let's imagine we have 50 servers with and a distributed web application and we need to upgrade software on all of them. New software requires downtime while upgrade.  
> > If upgrade is performed simultaneously on all servers the application will be totally unavailable for this time. How to avoid it?
> 
> I don't have personal experience with it, but other Chef experts I've heard have talked about using "rundeck" to handle the orchestration of things like this. I would be very interested to hear what your thoughts would be on this software.
> 
> --  
> Brad Knowles [bknowles@ihiji.com](mailto:bknowles@ihiji.com)  
> SAGE Level IV, Chef Level 0.0.1

---

<div class="post-metadata">

**Author:** ![Ranjib\_Dey](https://avatars.discourse-cdn.com/v4/letter/r/8e8cbc/32.png) [@Ranjib\_Dey](https://discourse.chef.io/u/Ranjib_Dey)\
**Post date:** [October 24, 2011, 4:51am UTC](https://discourse.chef.io/t/simultaneous-software-upgrade-on-multiple-nodes/2134/5 "2011-10-24T04:51:08Z")

</div>

I'll prefer knife ssh and pass -C 1 (or 5 or whatever block side i need)

On Sun, Oct 23, 2011 at 1:52 AM, Rob Guttman [robguttman@gmail.com](mailto:robguttman@gmail.com) wrote:

> If you use aws ec2 and Trac, I wrote a Trac plugin that does the multi-node  
> serial orchestration for push-button deployments:
> 
> [http://trac-hacks.org/wiki/CloudPlugin](http://trac-hacks.org/wiki/CloudPlugin)
> 
> It's really just a thin webui wrapper to pychef and boto.
> 
> - Rob
> 
> On Oct 22, 2011, at 9:55 AM, Brad Knowles wrote:
> 
> On Oct 22, 2011, at 3:10 AM, Anton Koldaev wrote:
> 
> Let's imagine we have 50 servers with and a distributed web application and  
> we need to upgrade software on all of them. New software requires downtime  
> while upgrade.
> 
> If upgrade is performed simultaneously on all servers the application will  
> be totally unavailable for this time. How to avoid it?
> 
> I don't have personal experience with it, but other Chef experts I've heard  
> have talked about using "rundeck" to handle the orchestration of things like  
> this. I would be very interested to hear what your thoughts would be on  
> this software.
> 
> --  
> Brad Knowles [bknowles@ihiji.com](mailto:bknowles@ihiji.com)  
> SAGE Level IV, Chef Level 0.0.1

---

<div class="post-metadata">

**Author:** ![ianmmeyer](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/ianmmeyer/32/341_2.png) [@ianmmeyer](https://discourse.chef.io/u/ianmmeyer)\
**Post date:** [November 11, 2011, 12:16am UTC](https://discourse.chef.io/t/simultaneous-software-upgrade-on-multiple-nodes/2134/6 "2011-11-11T00:16:05Z")

</div>

I wrote a knife plugin called "batch" (gem install knife-batch) that is  
basically knife ssh, with the ability to specify how many servers to  
operate on at once with a sleep of however long you want in between these  
batches.

knife batch "role:foo" "command" -B 10 -W 10 (run command on 10 servers at  
once, with a wait of 10 seconds per batch)

It's pretty handy in cases where the -C option to knife ssh won't fit your  
needs.

- Ian

On Mon, Oct 24, 2011 at 12:51 AM, Ranjib Dey [ranjibd@thoughtworks.com](mailto:ranjibd@thoughtworks.com)wrote:

> I'll prefer knife ssh and pass -C 1 (or 5 or whatever block side i need)
> 
> On Sun, Oct 23, 2011 at 1:52 AM, Rob Guttman [robguttman@gmail.com](mailto:robguttman@gmail.com) wrote:
> 
> > If you use aws ec2 and Trac, I wrote a Trac plugin that does the  
> > multi-node serial orchestration for push-button deployments:
> > 
> > [http://trac-hacks.org/wiki/CloudPlugin](http://trac-hacks.org/wiki/CloudPlugin)
> > 
> > It's really just a thin webui wrapper to pychef and boto.
> > 
> > - Rob
> > 
> > On Oct 22, 2011, at 9:55 AM, Brad Knowles wrote:
> > 
> > On Oct 22, 2011, at 3:10 AM, Anton Koldaev wrote:
> > 
> > Let's imagine we have 50 servers with and a distributed web application  
> > and we need to upgrade software on all of them. New software requires  
> > downtime while upgrade.
> > 
> > If upgrade is performed simultaneously on all servers the application  
> > will be totally unavailable for this time. How to avoid it?
> > 
> > I don't have personal experience with it, but other Chef experts I've  
> > heard have talked about using "rundeck" to handle the orchestration of  
> > things like this. I would be very interested to hear what your thoughts  
> > would be on this software.
> > 
> > --  
> > Brad Knowles [bknowles@ihiji.com](mailto:bknowles@ihiji.com)  
> > SAGE Level IV, Chef Level 0.0.1

---

<div class="post-metadata">

**Author:** ![KC\_Braunschweig](https://avatars.discourse-cdn.com/v4/letter/k/ea5d25/32.png) [@KC\_Braunschweig](https://discourse.chef.io/u/KC_Braunschweig)\
**Post date:** [November 11, 2011, 12:30am UTC](https://discourse.chef.io/t/simultaneous-software-upgrade-on-multiple-nodes/2134/7 "2011-11-11T00:30:52Z")

</div>

I haven't tried it myself, but saw this a while back and it looked promising:

> **[GitHub - websterclay/chef-dominodes: A Chef resource for mutual exclusion of...](https://github.com/websterclay/chef-dominodes)**
>
> A Chef resource for mutual exclusion of blocks of recipe code. Useful for cross-cluster rolling restarts. - GitHub - websterclay/chef-dominodes: A Chef resource for mutual exclusion of blocks of ...

Basically does rolling restarts using a databag to implement locking.

KC

On Thu, Nov 10, 2011 at 4:16 PM, Ian Meyer [ianmmeyer@gmail.com](mailto:ianmmeyer@gmail.com) wrote:

> I wrote a knife plugin called "batch" (gem install knife-batch) that is  
> basically knife ssh, with the ability to specify how many servers to operate  
> on at once with a sleep of however long you want in between these batches.  
> knife batch "role:foo" "command" -B 10 -W 10 (run command on 10 servers at  
> once, with a wait of 10 seconds per batch)  
> It's pretty handy in cases where the -C option to knife ssh won't fit your  
> needs.
> 
> - Ian
> 
> On Mon, Oct 24, 2011 at 12:51 AM, Ranjib Dey [ranjibd@thoughtworks.com](mailto:ranjibd@thoughtworks.com)  
> wrote:
> 
> > I'll prefer knife ssh and pass -C 1 (or 5 or whatever block side i need)
> > 
> > On Sun, Oct 23, 2011 at 1:52 AM, Rob Guttman [robguttman@gmail.com](mailto:robguttman@gmail.com) wrote:
> > 
> > > If you use aws ec2 and Trac, I wrote a Trac plugin that does the  
> > > multi-node serial orchestration for push-button deployments:  
> > > [http://trac-hacks.org/wiki/CloudPlugin](http://trac-hacks.org/wiki/CloudPlugin)  
> > > It's really just a thin webui wrapper to pychef and boto.
> > > 
> > > - Rob
> > > 
> > > On Oct 22, 2011, at 9:55 AM, Brad Knowles wrote:
> > > 
> > > On Oct 22, 2011, at 3:10 AM, Anton Koldaev wrote:
> > > 
> > > Let's imagine we have 50 servers with and a distributed web application  
> > > and we need to upgrade software on all of them. New software requires  
> > > downtime while upgrade.
> > > 
> > > If upgrade is performed simultaneously on all servers the application  
> > > will be totally unavailable for this time. How to avoid it?
> > > 
> > > I don't have personal experience with it, but other Chef experts I've  
> > > heard have talked about using "rundeck" to handle the orchestration of  
> > > things like this. I would be very interested to hear what your thoughts  
> > > would be on this software.
> > > 
> > > --  
> > > Brad Knowles [bknowles@ihiji.com](mailto:bknowles@ihiji.com)  
> > > SAGE Level IV, Chef Level 0.0.1