Sequencing node operations - db, multiple apps


#1

Hello again!
This one has probably already been discussed, and apologies if that is
the case, but I have a situation where I have a single DB master/slave
cluster, and between 2 and 4 application nodes pointing at the DB
cluster.

I can run DB updates/migrations from the DB server, or from one of the
application servers, location doesn’t particularly matter.
However… I need sequencing to be correct to avoid the app breaking
itself or the DB.

  1. Before the DB migrations start, all of the app nodes need to have
    stopped (tomcat app)
  2. Before the app servers start back up, all of the db migrations
    need to have completed.

I can add something to the tomcat startup that causes it to wait for a
schema version to show up in the database, which satisfies condition
2… but it isn’t the cleanest method.
I can have the db server run the migrations, refusing to do it until a
search for app servers reveals that they are all running the expected
version…

Or is there some newfangled cookbook that tracks things through
attributes and search that automagically keeps everything in sync?
It would need to only handle the sync, and not be like application
that handles everything, as our app doesn’t currently fit the mold
covered by application…


#2

What you need is an orchestrator that can dispatch chef-client runs or
special ssh commands across nodes in certain order. I’ll suggest using
mcollective or something similar (i use champu for the same
https://github.com/ranjibd/champu, but its very primitive as of now).

On Sun, Aug 5, 2012 at 11:12 PM, Jesse Campbell hikeit@gmail.com wrote:

Hello again!
This one has probably already been discussed, and apologies if that is
the case, but I have a situation where I have a single DB master/slave
cluster, and between 2 and 4 application nodes pointing at the DB
cluster.

I can run DB updates/migrations from the DB server, or from one of the
application servers, location doesn’t particularly matter.
However… I need sequencing to be correct to avoid the app breaking
itself or the DB.

  1. Before the DB migrations start, all of the app nodes need to have
    stopped (tomcat app)
  2. Before the app servers start back up, all of the db migrations
    need to have completed.

I can add something to the tomcat startup that causes it to wait for a
schema version to show up in the database, which satisfies condition
2… but it isn’t the cleanest method.
I can have the db server run the migrations, refusing to do it until a
search for app servers reveals that they are all running the expected
version…

Or is there some newfangled cookbook that tracks things through
attributes and search that automagically keeps everything in sync?
It would need to only handle the sync, and not be like application
that handles everything, as our app doesn’t currently fit the mold
covered by application…