Node attributes aren't searchable for some reason?

I’m working with the following two nodes, which are supposed to be part of
a pacemaker cluster:
[imarlier@imarlier-mac chef-repo (hc_production)] k node show login01.fal --attribute cluster --format json { "cluster": { "heartbeat_coredumps": "true", "uuid": "dff82156-9145-4a89-b92a-ba3bb238442b", "udpport": "759", "heartbeat_compression": "bz2", "pacemaker": { "resource_dir": "/usr/lib/ocf/resource.d" }, "deadtime": 60, "initdead": 90, "warntime": 30, "keepalive": 5 } } [imarlier@imarlier-mac chef-repo (hc_production)] k node show login02.fal
–attribute cluster --format json
{
“cluster”: {
“heartbeat_coredumps”: “true”,
“uuid”: “dff82156-9145-4a89-b92a-ba3bb238442b”,
“udpport”: “759”,
“heartbeat_compression”: “bz2”,
“deadtime”: 60,
“pacemaker”: {
“resource_dir”: “/usr/lib/ocf/resource.d”
},
“initdead”: 90,
“keepalive”: 5,
“warntime”: 30
}
}
[imarlier@imarlier-mac chef-repo (hc_production)]$

However, for reasons that I can’t explain, searching based on cluster UUID
doesn’t return these nodes:

[imarlier@imarlier-mac chef-repo (hc_production)]$ k search node
’cluster_uuid:dff82156-9145-4a89-b92a-ba3bb238442b’
0 items found

[imarlier@imarlier-mac chef-repo (hc_production)]$

Anyone have any idea why this would be? Is there a way to check the index
status for a given node or something like that, or to force re-indexing of
the node? (Note: I did try editing the node, removing the UUID parameter,
saving the node, then editing again and adding the UUID parameter back in.)

Thanks,

Ian


Ian Marlier | Senior Systems Engineer
Brightcove, Inc.
290 Congress Street, 4th Floor, Boston, MA 02110
imarlier@brightcove.com

Hi,

Do the nodes that are not being returned from search have a lot of
attribute data? If so the problem may be that they are failing to be
indexed by solr as you are exceeding the setting maxFieldLength. If
this is the case look for the solr config file on your system and
increase the size of this. (Just be warned the config file actually
appears twice on ubuntu systems for some reason).

HTH

On Wed, May 30, 2012 at 3:00 AM, Ian Marlier imarlier@brightcove.com wrote:

I'm working with the following two nodes, which are supposed to be part of a
pacemaker cluster:
[imarlier@imarlier-mac chef-repo (hc_production)]$ k node show login01.fal
--attribute cluster --format json
{
"cluster": {
"heartbeat_coredumps": "true",
"uuid": "dff82156-9145-4a89-b92a-ba3bb238442b",
"udpport": "759",
"heartbeat_compression": "bz2",
"pacemaker": {
"resource_dir": "/usr/lib/ocf/resource.d"
},
"deadtime": 60,
"initdead": 90,
"warntime": 30,
"keepalive": 5
}
}
[imarlier@imarlier-mac chef-repo (hc_production)]$ k node show login02.fal
--attribute cluster --format json
{
"cluster": {
"heartbeat_coredumps": "true",
"uuid": "dff82156-9145-4a89-b92a-ba3bb238442b",
"udpport": "759",
"heartbeat_compression": "bz2",
"deadtime": 60,
"pacemaker": {
"resource_dir": "/usr/lib/ocf/resource.d"
},
"initdead": 90,
"keepalive": 5,
"warntime": 30
}
}
[imarlier@imarlier-mac chef-repo (hc_production)]$

However, for reasons that I can't explain, searching based on cluster UUID
doesn't return these nodes:

[imarlier@imarlier-mac chef-repo (hc_production)]$ k search node
'cluster_uuid:dff82156-9145-4a89-b92a-ba3bb238442b'
0 items found

[imarlier@imarlier-mac chef-repo (hc_production)]$

Anyone have any idea why this would be? Is there a way to check the index
status for a given node or something like that, or to force re-indexing of
the node? (Note: I did try editing the node, removing the UUID parameter,
saving the node, then editing again and adding the UUID parameter back in.)

Thanks,

Ian

--
Ian Marlier | Senior Systems Engineer
Brightcove, Inc.
290 Congress Street, 4th Floor, Boston, MA 02110
imarlier@brightcove.com

--
Cheers,

Peter Donald

I'll take a look at that setting, Peter. Thanks for the pointer. I'll
report back once I've got something.

On Tue, May 29, 2012 at 9:15 PM, Peter Donald peter@realityforge.orgwrote:

Hi,

Do the nodes that are not being returned from search have a lot of
attribute data? If so the problem may be that they are failing to be
indexed by solr as you are exceeding the setting maxFieldLength. If
this is the case look for the solr config file on your system and
increase the size of this. (Just be warned the config file actually
appears twice on ubuntu systems for some reason).

HTH

On Wed, May 30, 2012 at 3:00 AM, Ian Marlier imarlier@brightcove.com
wrote:

I'm working with the following two nodes, which are supposed to be part
of a
pacemaker cluster:
[imarlier@imarlier-mac chef-repo (hc_production)]$ k node show
login01.fal
--attribute cluster --format json
{
"cluster": {
"heartbeat_coredumps": "true",
"uuid": "dff82156-9145-4a89-b92a-ba3bb238442b",
"udpport": "759",
"heartbeat_compression": "bz2",
"pacemaker": {
"resource_dir": "/usr/lib/ocf/resource.d"
},
"deadtime": 60,
"initdead": 90,
"warntime": 30,
"keepalive": 5
}
}
[imarlier@imarlier-mac chef-repo (hc_production)]$ k node show
login02.fal
--attribute cluster --format json
{
"cluster": {
"heartbeat_coredumps": "true",
"uuid": "dff82156-9145-4a89-b92a-ba3bb238442b",
"udpport": "759",
"heartbeat_compression": "bz2",
"deadtime": 60,
"pacemaker": {
"resource_dir": "/usr/lib/ocf/resource.d"
},
"initdead": 90,
"keepalive": 5,
"warntime": 30
}
}
[imarlier@imarlier-mac chef-repo (hc_production)]$

However, for reasons that I can't explain, searching based on cluster
UUID
doesn't return these nodes:

[imarlier@imarlier-mac chef-repo (hc_production)]$ k search node
'cluster_uuid:dff82156-9145-4a89-b92a-ba3bb238442b'
0 items found

[imarlier@imarlier-mac chef-repo (hc_production)]$

Anyone have any idea why this would be? Is there a way to check the
index
status for a given node or something like that, or to force re-indexing
of
the node? (Note: I did try editing the node, removing the UUID
parameter,
saving the node, then editing again and adding the UUID parameter back
in.)

Thanks,

Ian

--
Ian Marlier | Senior Systems Engineer
Brightcove, Inc.
290 Congress Street, 4th Floor, Boston, MA 02110
imarlier@brightcove.com

--
Cheers,

Peter Donald

--
Ian Marlier | Senior Systems Engineer
Brightcove, Inc.
290 Congress Street, 4th Floor, Boston, MA 02110
imarlier@brightcove.com

Sadly, changing that parameter and restarting solr doesn't seem to have
helped. I changed it in both locations (/etc/solr/conf/solrconfig.xml and
/var/lib/chef/solr/conf/solrconfig.xml) and restarted chef-solr after doing
so. I initially changed it from the default value of 10000 to 20000, and
then tried again raising it to 120000

In order to try to force reindexing I've done a number of things, including
editing the nodes and deleting the cluster/uuid parameter entirely, and
then editing again and adding it back. I've even tried dumping the entire
node out to a JSON file, deleting it entirely from Chef, and then adding it
back in again. Neither of those worked.

Querying solr directly, it's clear that the issue is that this particular
attribute isn't making it in there, but I really can't comprehend why.

On Tue, May 29, 2012 at 10:39 PM, Ian Marlier imarlier@brightcove.comwrote:

I'll take a look at that setting, Peter. Thanks for the pointer. I'll
report back once I've got something.

On Tue, May 29, 2012 at 9:15 PM, Peter Donald peter@realityforge.orgwrote:

Hi,

Do the nodes that are not being returned from search have a lot of
attribute data? If so the problem may be that they are failing to be
indexed by solr as you are exceeding the setting maxFieldLength. If
this is the case look for the solr config file on your system and
increase the size of this. (Just be warned the config file actually
appears twice on ubuntu systems for some reason).

HTH

On Wed, May 30, 2012 at 3:00 AM, Ian Marlier imarlier@brightcove.com
wrote:

I'm working with the following two nodes, which are supposed to be part
of a
pacemaker cluster:
[imarlier@imarlier-mac chef-repo (hc_production)]$ k node show
login01.fal
--attribute cluster --format json
{
"cluster": {
"heartbeat_coredumps": "true",
"uuid": "dff82156-9145-4a89-b92a-ba3bb238442b",
"udpport": "759",
"heartbeat_compression": "bz2",
"pacemaker": {
"resource_dir": "/usr/lib/ocf/resource.d"
},
"deadtime": 60,
"initdead": 90,
"warntime": 30,
"keepalive": 5
}
}
[imarlier@imarlier-mac chef-repo (hc_production)]$ k node show
login02.fal
--attribute cluster --format json
{
"cluster": {
"heartbeat_coredumps": "true",
"uuid": "dff82156-9145-4a89-b92a-ba3bb238442b",
"udpport": "759",
"heartbeat_compression": "bz2",
"deadtime": 60,
"pacemaker": {
"resource_dir": "/usr/lib/ocf/resource.d"
},
"initdead": 90,
"keepalive": 5,
"warntime": 30
}
}
[imarlier@imarlier-mac chef-repo (hc_production)]$

However, for reasons that I can't explain, searching based on cluster
UUID
doesn't return these nodes:

[imarlier@imarlier-mac chef-repo (hc_production)]$ k search node
'cluster_uuid:dff82156-9145-4a89-b92a-ba3bb238442b'
0 items found

[imarlier@imarlier-mac chef-repo (hc_production)]$

Anyone have any idea why this would be? Is there a way to check the
index
status for a given node or something like that, or to force re-indexing
of
the node? (Note: I did try editing the node, removing the UUID
parameter,
saving the node, then editing again and adding the UUID parameter back
in.)

Thanks,

Ian

--
Ian Marlier | Senior Systems Engineer
Brightcove, Inc.
290 Congress Street, 4th Floor, Boston, MA 02110
imarlier@brightcove.com

--
Cheers,

Peter Donald

--
Ian Marlier | Senior Systems Engineer
Brightcove, Inc.
290 Congress Street, 4th Floor, Boston, MA 02110
imarlier@brightcove.com

--
Ian Marlier | Senior Systems Engineer
Brightcove, Inc.
290 Congress Street, 4th Floor, Boston, MA 02110
imarlier@brightcove.com

So, it turned out that this DID fix my problem, but there's a trick:
/var/lib/chef/solr/conf/solrconfig.xml has the maxFieldLength field set
twice, and you need to change the value in both places for it to take.

On Tue, May 29, 2012 at 11:14 PM, Ian Marlier imarlier@brightcove.comwrote:

Sadly, changing that parameter and restarting solr doesn't seem to have
helped. I changed it in both locations (/etc/solr/conf/solrconfig.xml and
/var/lib/chef/solr/conf/solrconfig.xml) and restarted chef-solr after doing
so. I initially changed it from the default value of 10000 to 20000, and
then tried again raising it to 120000

In order to try to force reindexing I've done a number of things,
including editing the nodes and deleting the cluster/uuid parameter
entirely, and then editing again and adding it back. I've even tried
dumping the entire node out to a JSON file, deleting it entirely from Chef,
and then adding it back in again. Neither of those worked.

Querying solr directly, it's clear that the issue is that this particular
attribute isn't making it in there, but I really can't comprehend why.

On Tue, May 29, 2012 at 10:39 PM, Ian Marlier imarlier@brightcove.comwrote:

I'll take a look at that setting, Peter. Thanks for the pointer. I'll
report back once I've got something.

On Tue, May 29, 2012 at 9:15 PM, Peter Donald peter@realityforge.orgwrote:

Hi,

Do the nodes that are not being returned from search have a lot of
attribute data? If so the problem may be that they are failing to be
indexed by solr as you are exceeding the setting maxFieldLength. If
this is the case look for the solr config file on your system and
increase the size of this. (Just be warned the config file actually
appears twice on ubuntu systems for some reason).

HTH

On Wed, May 30, 2012 at 3:00 AM, Ian Marlier imarlier@brightcove.com
wrote:

I'm working with the following two nodes, which are supposed to be
part of a
pacemaker cluster:
[imarlier@imarlier-mac chef-repo (hc_production)]$ k node show
login01.fal
--attribute cluster --format json
{
"cluster": {
"heartbeat_coredumps": "true",
"uuid": "dff82156-9145-4a89-b92a-ba3bb238442b",
"udpport": "759",
"heartbeat_compression": "bz2",
"pacemaker": {
"resource_dir": "/usr/lib/ocf/resource.d"
},
"deadtime": 60,
"initdead": 90,
"warntime": 30,
"keepalive": 5
}
}
[imarlier@imarlier-mac chef-repo (hc_production)]$ k node show
login02.fal
--attribute cluster --format json
{
"cluster": {
"heartbeat_coredumps": "true",
"uuid": "dff82156-9145-4a89-b92a-ba3bb238442b",
"udpport": "759",
"heartbeat_compression": "bz2",
"deadtime": 60,
"pacemaker": {
"resource_dir": "/usr/lib/ocf/resource.d"
},
"initdead": 90,
"keepalive": 5,
"warntime": 30
}
}
[imarlier@imarlier-mac chef-repo (hc_production)]$

However, for reasons that I can't explain, searching based on cluster
UUID
doesn't return these nodes:

[imarlier@imarlier-mac chef-repo (hc_production)]$ k search node
'cluster_uuid:dff82156-9145-4a89-b92a-ba3bb238442b'
0 items found

[imarlier@imarlier-mac chef-repo (hc_production)]$

Anyone have any idea why this would be? Is there a way to check the
index
status for a given node or something like that, or to force
re-indexing of
the node? (Note: I did try editing the node, removing the UUID
parameter,
saving the node, then editing again and adding the UUID parameter back
in.)

Thanks,

Ian

--
Ian Marlier | Senior Systems Engineer
Brightcove, Inc.
290 Congress Street, 4th Floor, Boston, MA 02110
imarlier@brightcove.com

--
Cheers,

Peter Donald

--
Ian Marlier | Senior Systems Engineer
Brightcove, Inc.
290 Congress Street, 4th Floor, Boston, MA 02110
imarlier@brightcove.com

--
Ian Marlier | Senior Systems Engineer
Brightcove, Inc.
290 Congress Street, 4th Floor, Boston, MA 02110
imarlier@brightcove.com

--
Ian Marlier | Senior Systems Engineer
Brightcove, Inc.
290 Congress Street, 4th Floor, Boston, MA 02110
imarlier@brightcove.com

I would love to know your configuration, I've never been able to make this work. Every time I try solr stops responding and chef server eventually crashes. I'd like to be able to blame it on centos 5.5 and ruby 1.8.7, but I dont anything to compare to

Sent from a phone

On May 31, 2012, at 2:10 PM, Ian Marlier imarlier@brightcove.com wrote:

So, it turned out that this DID fix my problem, but there's a trick: /var/lib/chef/solr/conf/solrconfig.xml has the maxFieldLength field set twice, and you need to change the value in both places for it to take.

On Tue, May 29, 2012 at 11:14 PM, Ian Marlier imarlier@brightcove.com wrote:
Sadly, changing that parameter and restarting solr doesn't seem to have helped. I changed it in both locations (/etc/solr/conf/solrconfig.xml and /var/lib/chef/solr/conf/solrconfig.xml) and restarted chef-solr after doing so. I initially changed it from the default value of 10000 to 20000, and then tried again raising it to 120000

In order to try to force reindexing I've done a number of things, including editing the nodes and deleting the cluster/uuid parameter entirely, and then editing again and adding it back. I've even tried dumping the entire node out to a JSON file, deleting it entirely from Chef, and then adding it back in again. Neither of those worked.

Querying solr directly, it's clear that the issue is that this particular attribute isn't making it in there, but I really can't comprehend why.

On Tue, May 29, 2012 at 10:39 PM, Ian Marlier imarlier@brightcove.com wrote:
I'll take a look at that setting, Peter. Thanks for the pointer. I'll report back once I've got something.

On Tue, May 29, 2012 at 9:15 PM, Peter Donald peter@realityforge.org wrote:
Hi,

Do the nodes that are not being returned from search have a lot of
attribute data? If so the problem may be that they are failing to be
indexed by solr as you are exceeding the setting maxFieldLength. If
this is the case look for the solr config file on your system and
increase the size of this. (Just be warned the config file actually
appears twice on ubuntu systems for some reason).

HTH

On Wed, May 30, 2012 at 3:00 AM, Ian Marlier imarlier@brightcove.com wrote:

I'm working with the following two nodes, which are supposed to be part of a
pacemaker cluster:
[imarlier@imarlier-mac chef-repo (hc_production)]$ k node show login01.fal
--attribute cluster --format json
{
"cluster": {
"heartbeat_coredumps": "true",
"uuid": "dff82156-9145-4a89-b92a-ba3bb238442b",
"udpport": "759",
"heartbeat_compression": "bz2",
"pacemaker": {
"resource_dir": "/usr/lib/ocf/resource.d"
},
"deadtime": 60,
"initdead": 90,
"warntime": 30,
"keepalive": 5
}
}
[imarlier@imarlier-mac chef-repo (hc_production)]$ k node show login02.fal
--attribute cluster --format json
{
"cluster": {
"heartbeat_coredumps": "true",
"uuid": "dff82156-9145-4a89-b92a-ba3bb238442b",
"udpport": "759",
"heartbeat_compression": "bz2",
"deadtime": 60,
"pacemaker": {
"resource_dir": "/usr/lib/ocf/resource.d"
},
"initdead": 90,
"keepalive": 5,
"warntime": 30
}
}
[imarlier@imarlier-mac chef-repo (hc_production)]$

However, for reasons that I can't explain, searching based on cluster UUID
doesn't return these nodes:

[imarlier@imarlier-mac chef-repo (hc_production)]$ k search node
'cluster_uuid:dff82156-9145-4a89-b92a-ba3bb238442b'
0 items found

[imarlier@imarlier-mac chef-repo (hc_production)]$

Anyone have any idea why this would be? Is there a way to check the index
status for a given node or something like that, or to force re-indexing of
the node? (Note: I did try editing the node, removing the UUID parameter,
saving the node, then editing again and adding the UUID parameter back in.)

Thanks,

Ian

--
Ian Marlier | Senior Systems Engineer
Brightcove, Inc.
290 Congress Street, 4th Floor, Boston, MA 02110
imarlier@brightcove.com

--
Cheers,

Peter Donald

--
Ian Marlier | Senior Systems Engineer
Brightcove, Inc.
290 Congress Street, 4th Floor, Boston, MA 02110
imarlier@brightcove.com

--
Ian Marlier | Senior Systems Engineer
Brightcove, Inc.
290 Congress Street, 4th Floor, Boston, MA 02110
imarlier@brightcove.com

--
Ian Marlier | Senior Systems Engineer
Brightcove, Inc.
290 Congress Street, 4th Floor, Boston, MA 02110
imarlier@brightcove.com

I've made two changes:

  • maxFieldLength from 10k to 120k (sort of a random number, mostly just
    going for "biggerer"). There are two places that this appears in
    /var/lib/chef/solr/conf/solrconfig.xml, and it needs to be changed in both
    of them.
  • -Xms and -Xmx changed to 2GB. This was done in /etc/chef/solr.rb, by
    changing the variable "solr_heap_size". I didn't want to max out on it,
    because I was afraid of GC behavior, but I suspect that it's not really
    relevant since the resident set size doesn't seem to go above 750MB or
    thereabouts. As the number of machines that we have on this Chef instance
    rises, I suspect that I'll need to spend some time looking into that GC
    behavior and maybe doing some tuning.

Other than that, everything is stock.

OS is Ubuntu 11.04, Ruby version 1.9.2p290. Running in EC2 on an m1.xlarge
instance dedicated to the application and to Solr (Couch is running on a
separate box).

On Fri, Jun 1, 2012 at 10:10 AM, Chris grocerylist@gmail.com wrote:

I would love to know your configuration, I've never been able to make this
work. Every time I try solr stops responding and chef server eventually
crashes. I'd like to be able to blame it on centos 5.5 and ruby 1.8.7, but
I dont anything to compare to

Sent from a phone

On May 31, 2012, at 2:10 PM, Ian Marlier imarlier@brightcove.com wrote:

So, it turned out that this DID fix my problem, but there's a trick:
/var/lib/chef/solr/conf/solrconfig.xml has the maxFieldLength field set
twice, and you need to change the value in both places for it to take.

On Tue, May 29, 2012 at 11:14 PM, Ian Marlier imarlier@brightcove.comwrote:

Sadly, changing that parameter and restarting solr doesn't seem to have
helped. I changed it in both locations (/etc/solr/conf/solrconfig.xml and
/var/lib/chef/solr/conf/solrconfig.xml) and restarted chef-solr after doing
so. I initially changed it from the default value of 10000 to 20000, and
then tried again raising it to 120000

In order to try to force reindexing I've done a number of things,
including editing the nodes and deleting the cluster/uuid parameter
entirely, and then editing again and adding it back. I've even tried
dumping the entire node out to a JSON file, deleting it entirely from Chef,
and then adding it back in again. Neither of those worked.

Querying solr directly, it's clear that the issue is that this particular
attribute isn't making it in there, but I really can't comprehend why.

On Tue, May 29, 2012 at 10:39 PM, Ian Marlier imarlier@brightcove.comwrote:

I'll take a look at that setting, Peter. Thanks for the pointer. I'll
report back once I've got something.

On Tue, May 29, 2012 at 9:15 PM, Peter Donald peter@realityforge.orgwrote:

Hi,

Do the nodes that are not being returned from search have a lot of
attribute data? If so the problem may be that they are failing to be
indexed by solr as you are exceeding the setting maxFieldLength. If
this is the case look for the solr config file on your system and
increase the size of this. (Just be warned the config file actually
appears twice on ubuntu systems for some reason).

HTH

On Wed, May 30, 2012 at 3:00 AM, Ian Marlier imarlier@brightcove.com
wrote:

I'm working with the following two nodes, which are supposed to be
part of a
pacemaker cluster:
[imarlier@imarlier-mac chef-repo (hc_production)]$ k node show
login01.fal
--attribute cluster --format json
{
"cluster": {
"heartbeat_coredumps": "true",
"uuid": "dff82156-9145-4a89-b92a-ba3bb238442b",
"udpport": "759",
"heartbeat_compression": "bz2",
"pacemaker": {
"resource_dir": "/usr/lib/ocf/resource.d"
},
"deadtime": 60,
"initdead": 90,
"warntime": 30,
"keepalive": 5
}
}
[imarlier@imarlier-mac chef-repo (hc_production)]$ k node show
login02.fal
--attribute cluster --format json
{
"cluster": {
"heartbeat_coredumps": "true",
"uuid": "dff82156-9145-4a89-b92a-ba3bb238442b",
"udpport": "759",
"heartbeat_compression": "bz2",
"deadtime": 60,
"pacemaker": {
"resource_dir": "/usr/lib/ocf/resource.d"
},
"initdead": 90,
"keepalive": 5,
"warntime": 30
}
}
[imarlier@imarlier-mac chef-repo (hc_production)]$

However, for reasons that I can't explain, searching based on cluster
UUID
doesn't return these nodes:

[imarlier@imarlier-mac chef-repo (hc_production)]$ k search node
'cluster_uuid:dff82156-9145-4a89-b92a-ba3bb238442b'
0 items found

[imarlier@imarlier-mac chef-repo (hc_production)]$

Anyone have any idea why this would be? Is there a way to check the
index
status for a given node or something like that, or to force
re-indexing of
the node? (Note: I did try editing the node, removing the UUID
parameter,
saving the node, then editing again and adding the UUID parameter
back in.)

Thanks,

Ian

--
Ian Marlier | Senior Systems Engineer
Brightcove, Inc.
290 Congress Street, 4th Floor, Boston, MA 02110
imarlier@brightcove.com

--
Cheers,

Peter Donald

--
Ian Marlier | Senior Systems Engineer
Brightcove, Inc.
290 Congress Street, 4th Floor, Boston, MA 02110
imarlier@brightcove.com

--
Ian Marlier | Senior Systems Engineer
Brightcove, Inc.
290 Congress Street, 4th Floor, Boston, MA 02110
imarlier@brightcove.com

--
Ian Marlier | Senior Systems Engineer
Brightcove, Inc.
290 Congress Street, 4th Floor, Boston, MA 02110
imarlier@brightcove.com

--
Ian Marlier | Senior Systems Engineer
Brightcove, Inc.
290 Congress Street, 4th Floor, Boston, MA 02110
imarlier@brightcove.com

Thanks Ian

I figured out that 50k is the minimum for my setup, I just wish i
could keep the system stable. Maybe moving to Ruby 1.9.x will finally
fix this problem. I've been able to work around this by using Adam
Jacobs whitelist cookbook. But it would be nice not to have to worry
about updating the cookbook if i want to add a new attribute.

On Fri, Jun 1, 2012 at 11:53 AM, Ian Marlier imarlier@brightcove.com wrote:

I've made two changes:

  • maxFieldLength from 10k to 120k (sort of a random number, mostly just
    going for "biggerer"). There are two places that this appears in
    /var/lib/chef/solr/conf/solrconfig.xml, and it needs to be changed in both
    of them.
  • -Xms and -Xmx changed to 2GB. This was done in /etc/chef/solr.rb, by
    changing the variable "solr_heap_size". I didn't want to max out on it,
    because I was afraid of GC behavior, but I suspect that it's not really
    relevant since the resident set size doesn't seem to go above 750MB or
    thereabouts. As the number of machines that we have on this Chef instance
    rises, I suspect that I'll need to spend some time looking into that GC
    behavior and maybe doing some tuning.

Other than that, everything is stock.

OS is Ubuntu 11.04, Ruby version 1.9.2p290. Running in EC2 on an m1.xlarge
instance dedicated to the application and to Solr (Couch is running on a
separate box).

On Fri, Jun 1, 2012 at 10:10 AM, Chris grocerylist@gmail.com wrote:

I would love to know your configuration, I've never been able to make this
work. Every time I try solr stops responding and chef server eventually
crashes. I'd like to be able to blame it on centos 5.5 and ruby 1.8.7, but I
dont anything to compare to

Sent from a phone

On May 31, 2012, at 2:10 PM, Ian Marlier imarlier@brightcove.com wrote:

So, it turned out that this DID fix my problem, but there's a trick:
/var/lib/chef/solr/conf/solrconfig.xml has the maxFieldLength field set
twice, and you need to change the value in both places for it to take.

On Tue, May 29, 2012 at 11:14 PM, Ian Marlier imarlier@brightcove.com
wrote:

Sadly, changing that parameter and restarting solr doesn't seem to have
helped. I changed it in both locations (/etc/solr/conf/solrconfig.xml and
/var/lib/chef/solr/conf/solrconfig.xml) and restarted chef-solr after doing
so. I initially changed it from the default value of 10000 to 20000, and
then tried again raising it to 120000

In order to try to force reindexing I've done a number of things,
including editing the nodes and deleting the cluster/uuid parameter
entirely, and then editing again and adding it back. I've even tried
dumping the entire node out to a JSON file, deleting it entirely from Chef,
and then adding it back in again. Neither of those worked.

Querying solr directly, it's clear that the issue is that this particular
attribute isn't making it in there, but I really can't comprehend why.

On Tue, May 29, 2012 at 10:39 PM, Ian Marlier imarlier@brightcove.com
wrote:

I'll take a look at that setting, Peter. Thanks for the pointer. I'll
report back once I've got something.

On Tue, May 29, 2012 at 9:15 PM, Peter Donald peter@realityforge.org
wrote:

Hi,

Do the nodes that are not being returned from search have a lot of
attribute data? If so the problem may be that they are failing to be
indexed by solr as you are exceeding the setting maxFieldLength. If
this is the case look for the solr config file on your system and
increase the size of this. (Just be warned the config file actually
appears twice on ubuntu systems for some reason).

HTH

On Wed, May 30, 2012 at 3:00 AM, Ian Marlier imarlier@brightcove.com
wrote:

I'm working with the following two nodes, which are supposed to be
part of a
pacemaker cluster:
[imarlier@imarlier-mac chef-repo (hc_production)]$ k node show
login01.fal
--attribute cluster --format json
{
"cluster": {
"heartbeat_coredumps": "true",
"uuid": "dff82156-9145-4a89-b92a-ba3bb238442b",
"udpport": "759",
"heartbeat_compression": "bz2",
"pacemaker": {
"resource_dir": "/usr/lib/ocf/resource.d"
},
"deadtime": 60,
"initdead": 90,
"warntime": 30,
"keepalive": 5
}
}
[imarlier@imarlier-mac chef-repo (hc_production)]$ k node show
login02.fal
--attribute cluster --format json
{
"cluster": {
"heartbeat_coredumps": "true",
"uuid": "dff82156-9145-4a89-b92a-ba3bb238442b",
"udpport": "759",
"heartbeat_compression": "bz2",
"deadtime": 60,
"pacemaker": {
"resource_dir": "/usr/lib/ocf/resource.d"
},
"initdead": 90,
"keepalive": 5,
"warntime": 30
}
}
[imarlier@imarlier-mac chef-repo (hc_production)]$

However, for reasons that I can't explain, searching based on cluster
UUID
doesn't return these nodes:

[imarlier@imarlier-mac chef-repo (hc_production)]$ k search node
'cluster_uuid:dff82156-9145-4a89-b92a-ba3bb238442b'
0 items found

[imarlier@imarlier-mac chef-repo (hc_production)]$

Anyone have any idea why this would be? Is there a way to check the
index
status for a given node or something like that, or to force
re-indexing of
the node? (Note: I did try editing the node, removing the UUID
parameter,
saving the node, then editing again and adding the UUID parameter
back in.)

Thanks,

Ian

--
Ian Marlier | Senior Systems Engineer
Brightcove, Inc.
290 Congress Street, 4th Floor, Boston, MA 02110
imarlier@brightcove.com

--
Cheers,

Peter Donald

--
Ian Marlier | Senior Systems Engineer
Brightcove, Inc.
290 Congress Street, 4th Floor, Boston, MA 02110
imarlier@brightcove.com

--
Ian Marlier | Senior Systems Engineer
Brightcove, Inc.
290 Congress Street, 4th Floor, Boston, MA 02110
imarlier@brightcove.com

--
Ian Marlier | Senior Systems Engineer
Brightcove, Inc.
290 Congress Street, 4th Floor, Boston, MA 02110
imarlier@brightcove.com

--
Ian Marlier | Senior Systems Engineer
Brightcove, Inc.
290 Congress Street, 4th Floor, Boston, MA 02110
imarlier@brightcove.com

--
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.