"Cluster" Management


#1

We’re running a deployment of 3 rails app servers, and we only need one to run the database setup.

What’s the best solution for allowing Chef to determine which app should run the db:setup?

Is it safe to run a search for ‘role:app’, then only run db:setup if current node is first node? We can even sort the search response to make sure it’s always the same ordered response.

I’d love to hear your suggestions!

Thanks,
Curtis


#2

What you’re describing is a distributed leader election problem. If I was you, I’d manually manage a tag to pick the node that runs db setup and migrations. If the leader leaves the cluster (hardware failure, etc.), then tag a different node in the cluster with knife tag or something. If that ever becomes a burden on your team, then you need to run zookeeper or similar to get a really reliable leader election system that automatically handles systems going down, race conditions, etc.


Daniel DeLeo

On Thursday, April 10, 2014 at 8:47 AM, Stewart, Curtis wrote:

We’re running a deployment of 3 rails app servers, and we only need one to run the database setup.

What’s the best solution for allowing Chef to determine which app should run the db:setup?

Is it safe to run a search for ‘role:app’, then only run db:setup if current node is first node? We can even sort the search response to make sure it’s always the same ordered response.

I’d love to hear your suggestions!

Thanks,
Curtis


#3

how you are doing deployment? db:* tasks should be triggered via the same
tool. if its chef (using the deploy resource) then yeah, search, sort and
use a particular node (as you have mentioned , first node) to trigger the
db:* tasks. Following are the things you have to ensure:
0) remember to sort the search results. :slight_smile:

  1. the first node converges first (tune chef run cron times)
  2. other app instances should not invoke deploy resource if the first app
    node’s convergence /deploy resource failed (set an attribute [e.g.
    last_migration = revision] to detect this).

this is definitely a yak shaving, but it works if your setup is smaller.
etcd, zookeeper is better way to attain this. As dan has already mentioned,
the same scenario is also used for HA, and called as leader election… but
here i’ll just say a singleton task (leaders generally used in master-slave
setups, here all app nodes are similar).
with zk you have to do some more yak shaving on the client side, while with
etcd mod_leader, and mod_lock is part of the core, thus its bit easier to
use.

btw, rails db migrations are idempotent, so, it should not matter even if
you run it from multiple hosts,

p.s. there are zk handler (though i am not sure report handlers are best
here), and etcd bindings for chef. You can use them along with the new even
handler system to do this cleanly
cheers
ranjib

On Thu, Apr 10, 2014 at 8:47 AM, Stewart, Curtis cstewart@momentumsi.comwrote:

We’re running a deployment of 3 rails app servers, and we only need one
to run the database setup.

What’s the best solution for allowing Chef to determine which app should
run the db:setup?

Is it safe to run a search for ‘role:app’, then only run db:setup if
current node is first node? We can even sort the search response to make
sure it’s always the same ordered response.

I’d love to hear your suggestions!

Thanks,
Curtis


#4

Thanks to both of you, Daniel and Ranjib!

Yes, we’re using Chef for the deployment and triggering db:* tasks through Chef.

I think we’ll try to go the route of sorting Chef search results for now. Although we’ll have to look into zookeeper down the road

Thanks again!
Curtis

On Apr 10, 2014, at 11:47 AM, Ranjib Dey <dey.ranjib@gmail.commailto:dey.ranjib@gmail.com> wrote:

how you are doing deployment? db:* tasks should be triggered via the same tool. if its chef (using the deploy resource) then yeah, search, sort and use a particular node (as you have mentioned , first node) to trigger the db:* tasks. Following are the things you have to ensure:
0) remember to sort the search results. :slight_smile:

  1. the first node converges first (tune chef run cron times)
  2. other app instances should not invoke deploy resource if the first app node’s convergence /deploy resource failed (set an attribute [e.g. last_migration = revision] to detect this).

this is definitely a yak shaving, but it works if your setup is smaller.
etcd, zookeeper is better way to attain this. As dan has already mentioned, the same scenario is also used for HA, and called as leader election… but here i’ll just say a singleton task (leaders generally used in master-slave setups, here all app nodes are similar).
with zk you have to do some more yak shaving on the client side, while with etcd mod_leader, and mod_lock is part of the core, thus its bit easier to use.

btw, rails db migrations are idempotent, so, it should not matter even if you run it from multiple hosts,

p.s. there are zk handler (though i am not sure report handlers are best here), and etcd bindings for chef. You can use them along with the new even handler system to do this cleanly
cheers
ranjib

On Thu, Apr 10, 2014 at 8:47 AM, Stewart, Curtis <cstewart@momentumsi.commailto:cstewart@momentumsi.com> wrote:
We’re running a deployment of 3 rails app servers, and we only need one to run the database setup.

What’s the best solution for allowing Chef to determine which app should run the db:setup?

Is it safe to run a search for ‘role:app’, then only run db:setup if current node is first node? We can even sort the search response to make sure it’s always the same ordered response.

I’d love to hear your suggestions!

Thanks,
Curtis