Solr fun


#1

hi chefs

i have a question about the solr query that a node does at the
beginning of its run – this one

select?q=content:hostname__%3D__*&wt=json&fq=%2BX_CHEF_database_CHEF_X:chef+%2BX_CHEF_type_CHEF_X:node&rows=1000

it looks like its asking for a list of every node, which at first
seems fine. but its causing me a lot of grief and i’m wondering what
to do about it.

if i query solr in a browser it returns very quickly, and i can even
see that in the solr log when my chef servers run the query. the
problem is that if this list has 400-ish or more entries in it my chef
server nodes start using a ton of cpu and are slow to respond to
clients. it seems like the chef servers are having a hard time
processing the results. i have 1300 clients split across 2 chef
servers (api only. solr, couch, and rabbit are on a different host),
each with 8GB and 4 cores. do i simply need more chef server nodes?


Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.


#2

On Wednesday, March 6, 2013 at 7:35 AM, Chris wrote:

hi chefs

i have a question about the solr query that a node does at the
beginning of its run – this one

select?q=content:hostname__%3D__*&wt=json&fq=%2BX_CHEF_database_CHEF_X:chef+%2BX_CHEF_type_CHEF_X:node&rows=1000

it looks like its asking for a list of every node, which at first
seems fine. but its causing me a lot of grief and i’m wondering what
to do about it.

if i query solr in a browser it returns very quickly, and i can even
see that in the solr log when my chef servers run the query. the
problem is that if this list has 400-ish or more entries in it my chef
server nodes start using a ton of cpu and are slow to respond to
clients. it seems like the chef servers are having a hard time
processing the results. i have 1300 clients split across 2 chef
servers (api only. solr, couch, and rabbit are on a different host),
each with 8GB and 4 cores. do i simply need more chef server nodes?


Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.

What version of the server is this?


Daniel DeLeo


#3

server is a bit old, 10.16.2.

On Wed, Mar 6, 2013 at 8:23 AM, Daniel DeLeo dan@kallistec.com wrote:

On Wednesday, March 6, 2013 at 7:35 AM, Chris wrote:

hi chefs

i have a question about the solr query that a node does at the
beginning of its run – this one

select?q=content:hostname__%3D__*&wt=json&fq=%2BX_CHEF_database_CHEF_X:chef+%2BX_CHEF_type_CHEF_X:node&rows=1000

it looks like its asking for a list of every node, which at first
seems fine. but its causing me a lot of grief and i’m wondering what
to do about it.

if i query solr in a browser it returns very quickly, and i can even
see that in the solr log when my chef servers run the query. the
problem is that if this list has 400-ish or more entries in it my chef
server nodes start using a ton of cpu and are slow to respond to
clients. it seems like the chef servers are having a hard time
processing the results. i have 1300 clients split across 2 chef
servers (api only. solr, couch, and rabbit are on a different host),
each with 8GB and 4 cores. do i simply need more chef server nodes?


Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.

What version of the server is this?


Daniel DeLeo


Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.


#4

On Wednesday, March 6, 2013 at 8:25 AM, Chris wrote:

server is a bit old, 10.16.2.

On Wed, Mar 6, 2013 at 8:23 AM, Daniel DeLeo <dan@kallistec.com (mailto:dan@kallistec.com)> wrote:

On Wednesday, March 6, 2013 at 7:35 AM, Chris wrote:

hi chefs

i have a question about the solr query that a node does at the
beginning of its run – this one

select?q=content:hostname__%3D__*&wt=json&fq=%2BX_CHEF_database_CHEF_X:chef+%2BX_CHEF_type_CHEF_X:node&rows=1000

it looks like its asking for a list of every node, which at first
seems fine. but its causing me a lot of grief and i’m wondering what
to do about it.

if i query solr in a browser it returns very quickly, and i can even
see that in the solr log when my chef servers run the query. the
problem is that if this list has 400-ish or more entries in it my chef
server nodes start using a ton of cpu and are slow to respond to
clients. it seems like the chef servers are having a hard time
processing the results. i have 1300 clients split across 2 chef
servers (api only. solr, couch, and rabbit are on a different host),
each with 8GB and 4 cores. do i simply need more chef server nodes?


Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.

What version of the server is this?


Daniel DeLeo


Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.

Upgrading to Chef 11 server is probably your best bet. We don’t have official migration tools yet, but you can follow this guide:

http://www.ducea.com/2013/03/05/howto-migrate-to-chef-11/

Chef 10.x clients are compatible with server 11.x, though I’d recommend upgrading knife on your workstation to at least 10.18.2 (10.24.0 is the latest 10.x) to get some small forward compat fixes (affects knife client reregister).

If that’s not an option, then you probably need to scale out chef-server workers.


Daniel DeLeo


#5

Would adding 2 additional workers to each api node be enough? Getting
new VMs might be an issue.

On Wed, Mar 6, 2013 at 8:53 AM, Daniel DeLeo dan@kallistec.com wrote:

On Wednesday, March 6, 2013 at 8:25 AM, Chris wrote:

server is a bit old, 10.16.2.

On Wed, Mar 6, 2013 at 8:23 AM, Daniel DeLeo dan@kallistec.com wrote:

On Wednesday, March 6, 2013 at 7:35 AM, Chris wrote:

hi chefs

i have a question about the solr query that a node does at the
beginning of its run – this one

select?q=content:hostname__%3D__*&wt=json&fq=%2BX_CHEF_database_CHEF_X:chef+%2BX_CHEF_type_CHEF_X:node&rows=1000

it looks like its asking for a list of every node, which at first
seems fine. but its causing me a lot of grief and i’m wondering what
to do about it.

if i query solr in a browser it returns very quickly, and i can even
see that in the solr log when my chef servers run the query. the
problem is that if this list has 400-ish or more entries in it my chef
server nodes start using a ton of cpu and are slow to respond to
clients. it seems like the chef servers are having a hard time
processing the results. i have 1300 clients split across 2 chef
servers (api only. solr, couch, and rabbit are on a different host),
each with 8GB and 4 cores. do i simply need more chef server nodes?


Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.

What version of the server is this?


Daniel DeLeo


Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.

Upgrading to Chef 11 server is probably your best bet. We don’t have
official migration tools yet, but you can follow this guide:

http://www.ducea.com/2013/03/05/howto-migrate-to-chef-11/

Chef 10.x clients are compatible with server 11.x, though I’d recommend
upgrading knife on your workstation to at least 10.18.2 (10.24.0 is the
latest 10.x) to get some small forward compat fixes (affects knife client reregister).

If that’s not an option, then you probably need to scale out chef-server
workers.


Daniel DeLeo


Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.


#6

As Dan said, upgrading to Chef 11 will help a lot here. Partial search is
your friend as well.

Best,
Adam

On 3/6/13 7:35 AM, “Chris” grocerylist@gmail.com wrote:

hi chefs

i have a question about the solr query that a node does at the
beginning of its run – this one

select?q=content:hostname__%3D__*&wt=json&fq=%2BX_CHEF_database_CHEF_X:che
f+%2BX_CHEF_type_CHEF_X:node&rows=1000

it looks like its asking for a list of every node, which at first
seems fine. but its causing me a lot of grief and i’m wondering what
to do about it.

if i query solr in a browser it returns very quickly, and i can even
see that in the solr log when my chef servers run the query. the
problem is that if this list has 400-ish or more entries in it my chef
server nodes start using a ton of cpu and are slow to respond to
clients. it seems like the chef servers are having a hard time
processing the results. i have 1300 clients split across 2 chef
servers (api only. solr, couch, and rabbit are on a different host),
each with 8GB and 4 cores. do i simply need more chef server nodes?


Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.