Expander redundancy

Reichert_Wil · April 5, 2012, 1:20am

I’m running chef .10.8 and trying to configure 2 expanders to each pull
messages off rabbit & populate 2 independent solr instances.
Documentation mentions separate 1 to n and 1 to (1 of n) behaviors but
it’s not clear if this is still the case with chef .10. Regardless of
what I set amqp_consumer_id, index, or node_count to I get the same
behavior - the first expander subscribes to all 1024 vnodes & the second
refuses to do anything as there’s already a consumer on the queues.
Chef-expanderctl node-status confirms the log messages. Searching
hasn’t pulled up much:

http://wiki.opscode.com/display/chef/Chef+Configuration+Settings doesn’t
have anything regarding the expander

http://wiki.opscode.com/display/chef/Chef+Indexer covers the some
settings but they don’t seem to work or aren’t applicable to this
version of chef.

Is this type of configuration possible?

Wil Reichert | Systems Operations

Wil.Reichert@monster.com | T : 415-820-3430 | M : 703-200-4607 | F :
415-820-0552

MILITARY.COM 799 Market Street, Suite 700, San Francisco, CA 94103

kallistec · April 5, 2012, 3:21pm

On Wednesday, April 4, 2012 at 6:20 PM, Reichert, Wil wrote:

I'm running chef .10.8 and trying to configure 2 expanders to each pull messages off rabbit & populate 2 independent solr instances. Documentation mentions separate 1 to n and 1 to (1 of n) behaviors but it’s not clear if this is still the case with chef .10. Regardless of what I set amqp_consumer_id, index, or node_count to I get the same behavior – the first expander subscribes to all 1024 vnodes & the second refuses to do anything as there’s already a consumer on the queues. Chef-expanderctl node-status confirms the log messages. Searching hasn’t pulled up much:

http://wiki.opscode.com/display/chef/Chef+Configuration+Settings doesn’t have anything regarding the expander

http://wiki.opscode.com/display/chef/Chef+Indexer covers the some settings but they don’t seem to work or aren’t applicable to this version of chef.

Is this type of configuration possible?

I don't think this is possible any longer. As you've seen, expander refuses to operate when there are other consumers on the queues, this is a safety feature to prevent data loss if the cluster somehow gets in a bad state.

I'd look in to Solr replication, it should be much more sane with the version of Solr in Chef 0.10.x

--
Dan DeLeo

Joseph_Holsten · April 5, 2012, 5:25pm

On 2012-04-05, at 15:21, Daniel DeLeo wrote:

On Wednesday, April 4, 2012 at 6:20 PM, Reichert, Wil wrote:

I'm running chef .10.8 and trying to configure 2 expanders to each pull messages off rabbit & populate 2 independent solr instances.

I don't think this is possible any longer. As you've seen, expander refuses to operate when there are other consumers on the queues, this is a safety feature to prevent data loss if the cluster somehow gets in a bad state.

Sorry to derail, but are you saying the amqp queues are flagged exclusive, and the consumers use a no-ack of false? Wouldn't we get the same safety if we switched both of those settings, use durable queues and (possibly) messages flagged persistent?

Reichert_Wil · April 5, 2012, 5:41pm

Solr replication works well, it does a master / slave pull that can be setup for reversal with a restart. My goal was an full active / active configuration. The solr cloud feature currently in trunk does distributed indexing via shards & seems like a nice option once its baked, that’s a bit down the road tho.

How are others handling redundant chef configs? Requests to chef-server can be balanced and couch & rabbit happily cluster themselves, that all seems fairly obvious. It’s the expander / solr bit that’s got me scratching my head on the best approach here.

Wil Reichert | Systems Operations

Wil.Reichert@monster.com | T : 415-820-3430 | M : 703-200-4607 | F : 415-820-0552