Chef Orchestration

Chef Infra (archive)
1

Just curious to know if anyone is using chef to do multi-node deployment
orchestration.

If so, how are you conducting the orchestration?

What other non-chef tools are you using for the orchestration?

–
Thanks,

Mark

2

knife can do some basic orchestration tasks but it can't provide any
transaction support. If you want to set up 3 servers, you can just
wrap around the knife api using rake or thor tasks. However, if you
actually want to have any real coordination you will need a
distributed lock manager, like Apache ZooKeeper. However, though I
have no experience w/ it. I know that heroku created doozerd, which
may be lighter weight, but I have no idea how well-maintained or
complete it is.

On Mon, Oct 22, 2012 at 7:52 PM, Mark Pimentel markpimentel22@gmail.com wrote:

Just curious to know if anyone is using chef to do multi-node deployment
orchestration.

If so, how are you conducting the orchestration?

What other non-chef tools are you using for the orchestration?

--
Thanks,

Mark

3

Hi,

On Tue, Oct 23, 2012 at 4:52 AM, Mark Pimentel markpimentel22@gmail.comwrote:

Just curious to know if anyone is using chef to do multi-node deployment
orchestration.

If so, how are you conducting the orchestration?

What other non-chef tools are you using for the orchestration?

When we need to do a multi-node release we typically;

  • flip the release flag wherever that may be (typically this means updating
    a data bag item with a new version)
  • each release is broken into several stages, for each stage we
    • perform a search to return a list of nodes to converge
    • converge a batch of nodes from the list of nodes until there are none
      left to converge

if any converge fails we have a semi-adhoc process semi automated process
of roll back and correct which usually just means reverting the data bag
item and re running the release.

We currently drive this from jenkins and had planned to use rundeck but
never quite got around to it.

--
Cheers,

Peter Donald

4

I get the release flag part as I have considered doing that as a trigger
point to trip the deployment.
I was looking for some way to manage an upgrade of an agent on 100+ nodes
that depend on whether the agent is running a job or not. So if running a
job, wait, otherwise shutdown, upgrade, start.

As for zookeeper, while I have taken a look, is there anything lighter
weight? Zookeeper seems like quite the application.

Any suggestions would be greatly appreciated.

Thanks.

On Mon, Oct 22, 2012 at 6:25 PM, Peter Donald peter@realityforge.orgwrote:

Hi,

On Tue, Oct 23, 2012 at 4:52 AM, Mark Pimentel markpimentel22@gmail.comwrote:

Just curious to know if anyone is using chef to do multi-node deployment
orchestration.

If so, how are you conducting the orchestration?

What other non-chef tools are you using for the orchestration?

When we need to do a multi-node release we typically;

  • flip the release flag wherever that may be (typically this means
    updating a data bag item with a new version)
  • each release is broken into several stages, for each stage we
    • perform a search to return a list of nodes to converge
    • converge a batch of nodes from the list of nodes until there are none
      left to converge

if any converge fails we have a semi-adhoc process semi automated process
of roll back and correct which usually just means reverting the data bag
item and re running the release.

We currently drive this from jenkins and had planned to use rundeck but
never quite got around to it.

--
Cheers,

Peter Donald

--
Thanks,

Mark

5

you could do that with knife exec

You don't really need orchestration as the operation completed on each
node is not dependent on the state of other nodes

http://wiki.opscode.com/display/chef/Knife+Exec

On Tue, Oct 23, 2012 at 2:54 PM, Mark Pimentel markpimentel22@gmail.com wrote:

I get the release flag part as I have considered doing that as a trigger
point to trip the deployment.
I was looking for some way to manage an upgrade of an agent on 100+ nodes
that depend on whether the agent is running a job or not. So if running a
job, wait, otherwise shutdown, upgrade, start.

As for zookeeper, while I have taken a look, is there anything lighter
weight? Zookeeper seems like quite the application.

Any suggestions would be greatly appreciated.

Thanks.

On Mon, Oct 22, 2012 at 6:25 PM, Peter Donald peter@realityforge.org
wrote:

Hi,

On Tue, Oct 23, 2012 at 4:52 AM, Mark Pimentel markpimentel22@gmail.com
wrote:

Just curious to know if anyone is using chef to do multi-node deployment
orchestration.

If so, how are you conducting the orchestration?

What other non-chef tools are you using for the orchestration?

When we need to do a multi-node release we typically;

  • flip the release flag wherever that may be (typically this means
    updating a data bag item with a new version)
  • each release is broken into several stages, for each stage we
    • perform a search to return a list of nodes to converge
    • converge a batch of nodes from the list of nodes until there are none
      left to converge

if any converge fails we have a semi-adhoc process semi automated process
of roll back and correct which usually just means reverting the data bag
item and re running the release.

We currently drive this from jenkins and had planned to use rundeck but
never quite got around to it.

--
Cheers,

Peter Donald

--
Thanks,

Mark