Hi
As we move configuration of more critical components to our chef server, we need to implement some sort of high availability solution for our chef server.
Any one has experience in that matter ?
a) Is replication of couchdb, solr index viable ? or
b) it is more simple to just re-create the database from source code of cookbook, roles and databags ? in that case how to make sure that we don´t have to re-register each node
On Mon, Jul 4, 2011 at 7:30 AM, le.huy@ingdirect.es wrote:
As we move configuration of more critical components to our chef server, we
need to implement some sort of high availability solution for our chef
server.Any one has experience in that matter ?
a) Is replication of couchdb, solr index viable ? or
It depends on what level of HA you require. For both CouchDB and Solr,
you can handle this with DRBD and passive failover for what is usually
sub-second takeover on failure - this is also good for things like
cookbook uploads (stick /var/chef on the DRBD drives.)
You can also make each component HA on their own, using the mechanisms
the upstream recommends - Replication in both cases you list above.
b) it is more simple to just re-create the database from source code
of cookbook, roles and databags ? in that case how to make sure that we
don´t have to re-register each node
You likely want to be keeping an active replica (either with app
replication or disk-level replication) for your failover - this helps
in DR, but that's what backups are for.
Best,
Adam
--
Opscode, Inc.
Adam Jacob, Chief Product Officer
T: (206) 619-7151 E: adam@opscode.com
You may also (in addition to the HA setup) want to check out Spiceweasel: GitHub - mattray/spiceweasel: Generates Chef knife commands from a simple JSON or YAML file. which will allow you to reference all of your nodes, cookbooks, roles etc in a yaml file and bulk load from it.
Ant
On Jul 5, 2011, at 1:42 PM, Adam Jacob wrote:
On Mon, Jul 4, 2011 at 7:30 AM, le.huy@ingdirect.es wrote:
As we move configuration of more critical components to our chef server, we
need to implement some sort of high availability solution for our chef
server.Any one has experience in that matter ?
a) Is replication of couchdb, solr index viable ? or
It depends on what level of HA you require. For both CouchDB and Solr,
you can handle this with DRBD and passive failover for what is usually
sub-second takeover on failure - this is also good for things like
cookbook uploads (stick /var/chef on the DRBD drives.)You can also make each component HA on their own, using the mechanisms
the upstream recommends - Replication in both cases you list above.b) it is more simple to just re-create the database from source code
of cookbook, roles and databags ? in that case how to make sure that we
don´t have to re-register each nodeYou likely want to be keeping an active replica (either with app
replication or disk-level replication) for your failover - this helps
in DR, but that's what backups are for.Best,
Adam--
Opscode, Inc.
Adam Jacob, Chief Product Officer
T: (206) 619-7151 E: adam@opscode.com
I'd be interested in people's experience with the capacity of various
Chef server configurations. I'm working on the HA setup as well.
Active/passive failover seems relatively straightforward but the next
question is when that won't be good enough. Anyone have numbers they'd
be willing to post? Haven't started testing yet to see what the first
bottleneck will be. Hoping that it'll scale well enough to handle our
needs (only a couple thousand nodes) so that we won't have to do much
more than split off the data components on separate nodes. Not
terribly interested in mimicking the full multi-tier stack I imagine
the platform runs on if I can avoid it.
KC
On Tue, Jul 5, 2011 at 11:00 AM, Anthony Goddard agoddard@mbl.edu wrote:
You may also (in addition to the HA setup) want to check out Spiceweasel:
GitHub - mattray/spiceweasel: Generates Chef knife commands from a simple JSON or YAML file. which will allow you to reference all
of your nodes, cookbooks, roles etc in a yaml file and bulk load from it.
AntOn Jul 5, 2011, at 1:42 PM, Adam Jacob wrote:
On Mon, Jul 4, 2011 at 7:30 AM, le.huy@ingdirect.es wrote:
As we move configuration of more critical components to our chef server, we
need to implement some sort of high availability solution for our chef
server.
Any one has experience in that matter ?
a) Is replication of couchdb, solr index viable ? or
It depends on what level of HA you require. For both CouchDB and Solr,
you can handle this with DRBD and passive failover for what is usually
sub-second takeover on failure - this is also good for things like
cookbook uploads (stick /var/chef on the DRBD drives.)You can also make each component HA on their own, using the mechanisms
the upstream recommends - Replication in both cases you list above.b) it is more simple to just re-create the database from source code
of cookbook, roles and databags ? in that case how to make sure that we
don´t have to re-register each node
You likely want to be keeping an active replica (either with app
replication or disk-level replication) for your failover - this helps
in DR, but that's what backups are for.Best,
Adam--
Opscode, Inc.
Adam Jacob, Chief Product Officer
T: (206) 619-7151 E: adam@opscode.com
On Tue, Jul 5, 2011 at 11:51 AM, KC Braunschweig
kcbraunschweig@gmail.com wrote:
I'd be interested in people's experience with the capacity of various
Chef server configurations. I'm working on the HA setup as well.
Active/passive failover seems relatively straightforward but the next
question is when that won't be good enough. Anyone have numbers they'd
be willing to post? Haven't started testing yet to see what the first
bottleneck will be. Hoping that it'll scale well enough to handle our
needs (only a couple thousand nodes) so that we won't have to do much
more than split off the data components on separate nodes. Not
terribly interested in mimicking the full multi-tier stack I imagine
the platform runs on if I can avoid it.
For a couple of thousand nodes, assuming you have control over the
hardware, you can scale active/passive vertically.
You need to be aware of, and control:
The frequency of convergence on the chef-clients. (How often do
they check in, at what splay.) This will impact the number of API
workers you run (we recommend you do it on unicorn,) and the
configuration of the upstream proxy (we use nginx.) This is often
bound by memory, although CPU can be a pain point.
Make sure you are compacting the CouchDB database and the views on
a regular basis.
Solr may need tuning, specifically memory/heap/permgen space.
RabbitMQ really only does reliable HA by persisting to disk.
Best,
Adam
--
Opscode, Inc.
Adam Jacob, Chief Product Officer
T: (206) 619-7151 E: adam@opscode.com