How to coordinate restart across roles


#1

Hi,

I have three three roles, rabbit, worker servers, django servers.

If I want to restart the django servers I need to restart rabbit first,
then the work servers and then django servers.

Off of the top of my head, I am thinkg about zookeep for co coordinating
where each server is listing for a signal to restart.

How would I use chef and hosted chef for this co-ordination? So

  1. A signal is sent to the rabbit roles to restart. It then sends this
    signal to the worker roles to restart then send a signal for the django
    servers t restart.

Since I have never done this before…I have no clue on how to proceed with
chef. Or is using a service like zookeeper best practice.

Thanks


#2

I’d recommend looking at Chef + ZK.

We have a rough implementation here that we’ve used for a few
successful tests, but nothing major like coordinated rolling
restarts.: https://github.com/chrisroberts/flock_of_chefs

This is complex stuff. You should ‘learn how to do this’ first, so to
speak! I’d recommend playing with tools like Doozer (Go), Paxos
(multiple lang impls.), Celluloid and DCell too (in Ruby land)

It’s much easier to coordinate processes while learning instead of
trying to coordinate huge behemoth systems.

It’s also worth considering that Opscode will probably release Pushy
soon for Hosted/Private, although I have no visibility into that /
comments.

Cheers,

AJ

On 3 March 2013 02:33, David Montgomery davidmontgomery@gmail.com wrote:

Hi,

I have three three roles, rabbit, worker servers, django servers.

If I want to restart the django servers I need to restart rabbit first, then
the work servers and then django servers.

Off of the top of my head, I am thinkg about zookeep for co coordinating
where each server is listing for a signal to restart.

How would I use chef and hosted chef for this co-ordination? So

  1. A signal is sent to the rabbit roles to restart. It then sends this
    signal to the worker roles to restart then send a signal for the django
    servers t restart.

Since I have never done this before…I have no clue on how to proceed with
chef. Or is using a service like zookeeper best practice.

Thanks


#3

On Mar 2, 2013, at 2:17 PM, AJ Christensen wrote:

I’d recommend looking at Chef + ZK.

We have a rough implementation here that we’ve used for a few
successful tests, but nothing major like coordinated rolling
restarts.: https://github.com/chrisroberts/flock_of_chefs

This is complex stuff. You should ‘learn how to do this’ first, so to
speak! I’d recommend playing with tools like Doozer (Go), Paxos
(multiple lang impls.), Celluloid and DCell too (in Ruby land)

As a warning, Doozerd is basically abandoned, though it looks like someone merged a few README fixes today after a year of nothing.

It’s much easier to coordinate processes while learning instead of
trying to coordinate huge behemoth systems.

My personal fav for this kind of task if Fabric + PyChef :slight_smile:

–Noah


#4

Dooozer is not abandoned There are a couple companies taking it up and
collaborating on it. Last week they had a hangout and discussion on how to
move forward. ActiveState, Bitly, Soundcloud, Torbit, and the guys working
on skynet are all contributing. The initial push is going to get doozer in
the “ha” org to build on go. Each of the orgs I mentioned above all have
their own versions that are more up to date and which have various
features. We are using skynet version of doozerd internally for evaluation
as a network lock service.

State of doozer discussion:
https://groups.google.com/d/topic/doozer/fVcS0y3KuHQ/discussion

Notes from the meeting:
https://groups.google.com/d/topic/doozer/oUuhBwQ_Ga4/discussion

Skynet doozerd:
https://github.com/skynetservices/doozerd

On Sat, Mar 2, 2013 at 2:29 PM, Noah Kantrowitz noah@coderanger.net wrote:

On Mar 2, 2013, at 2:17 PM, AJ Christensen wrote:

I’d recommend looking at Chef + ZK.

We have a rough implementation here that we’ve used for a few
successful tests, but nothing major like coordinated rolling
restarts.: https://github.com/chrisroberts/flock_of_chefs

This is complex stuff. You should ‘learn how to do this’ first, so to
speak! I’d recommend playing with tools like Doozer (Go), Paxos
(multiple lang impls.), Celluloid and DCell too (in Ruby land)

As a warning, Doozerd is basically abandoned, though it looks like someone
merged a few README fixes today after a year of nothing.

It’s much easier to coordinate processes while learning instead of
trying to coordinate huge behemoth systems.

My personal fav for this kind of task if Fabric + PyChef :slight_smile:

–Noah


#5

I would love to see this as a chef server feature In the future. Chef
chubby server !

This chubby: http://research.google.com/archive/chubby.html :smiley:

On Sat, Mar 2, 2013 at 2:44 PM, Jesse Nelson spheromak@gmail.com wrote:

Dooozer is not abandoned There are a couple companies taking it up and
collaborating on it. Last week they had a hangout and discussion on how to
move forward. ActiveState, Bitly, Soundcloud, Torbit, and the guys working
on skynet are all contributing. The initial push is going to get doozer in
the “ha” org to build on go. Each of the orgs I mentioned above all have
their own versions that are more up to date and which have various
features. We are using skynet version of doozerd internally for evaluation
as a network lock service.

State of doozer discussion:
https://groups.google.com/d/topic/doozer/fVcS0y3KuHQ/discussion

Notes from the meeting:
https://groups.google.com/d/topic/doozer/oUuhBwQ_Ga4/discussion

Skynet doozerd:
https://github.com/skynetservices/doozerd

On Sat, Mar 2, 2013 at 2:29 PM, Noah Kantrowitz noah@coderanger.netwrote:

On Mar 2, 2013, at 2:17 PM, AJ Christensen wrote:

I’d recommend looking at Chef + ZK.

We have a rough implementation here that we’ve used for a few
successful tests, but nothing major like coordinated rolling
restarts.: https://github.com/chrisroberts/flock_of_chefs

This is complex stuff. You should ‘learn how to do this’ first, so to
speak! I’d recommend playing with tools like Doozer (Go), Paxos
(multiple lang impls.), Celluloid and DCell too (in Ruby land)

As a warning, Doozerd is basically abandoned, though it looks like
someone merged a few README fixes today after a year of nothing.

It’s much easier to coordinate processes while learning instead of
trying to coordinate huge behemoth systems.

My personal fav for this kind of task if Fabric + PyChef :slight_smile:

–Noah