# Problems with Search?

**URL:** <https://discourse.chef.io/t/problems-with-search/1671>\
**Category:** Chef Infra (archive)\
**Created:** [January 27, 2011, 12:09am UTC](https://discourse.chef.io/t/problems-with-search/1671 "2011-01-27T00:09:18Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ash\_Berlin](https://avatars.discourse-cdn.com/v4/letter/a/0ea827/32.png) [@Ash\_Berlin](https://discourse.chef.io/u/Ash_Berlin)\
**Post date:** [January 27, 2011, 12:09am UTC](https://discourse.chef.io/t/problems-with-search/1671/1 "2011-01-27T00:09:18Z")

</div>

So some of my nagios checks started failing just now because the search that fills out allowed\_hosts in nrpe.cfg was empty. This is the second time this happened to me - the first was immediately after upgrading to chef-server 0.9.12.

The server is The clients in question are still on 0.9.8.

My cookbook code is (probably the standard nagios::client cookbook, possibly a few versions out of date):

search(:node, “roles:monitoring AND app\_environment:#{node[:app\_environment]}”) do |n|  
mon\_host \<\< n[‘ipaddress’]  
end

…

template “/etc/nagios/nrpe.cfg” do  
source "nrpe.cfg.erb"  
owner "nagios"  
group "nagios"  
mode "0644"  
variables :mon\_host =\> mon\_host  
notifies :restart, resources(:service =\> “nagios-nrpe-server”)  
end  
**END**

The initial problem on upgrading the server was the search was previously “role:…” which was working fine but needed to be changed to “roles:…”.

However this time nothing changed and simply logging in to the problem hosts and re-running chef-client fixed it.

What went wrong and how to I track it down so it doesn’t happen again?

-ash

---

<div class="post-metadata">

**Author:** ![Adam\_Jacob](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/adam_jacob/32/292_2.png) [@Adam\_Jacob](https://discourse.chef.io/u/Adam_Jacob)\
**Post date:** [January 27, 2011, 12:12am UTC](https://discourse.chef.io/t/problems-with-search/1671/2 "2011-01-27T00:12:15Z")

</div>

Ash, can you give use some gists of the log output from the Chef  
Server at the time?

Solr logs would help as well.

Adam

On Wed, Jan 26, 2011 at 4:09 PM, Ash Berlin [ash\_opscode@firemirror.com](mailto:ash_opscode@firemirror.com) wrote:

> So some of my nagios checks started failing just now because the search that fills out allowed\_hosts in nrpe.cfg was empty. This is the second time this happened to me - the first was immediately after upgrading to chef-server 0.9.12.
> 
> The server is The clients in question are still on 0.9.8.
> 
> My cookbook code is (probably the standard nagios::client cookbook, possibly a few versions out of date):
> 
> search(:node, "roles:monitoring AND app\_environment:#{node[:app\_environment]}") do |n|  
> mon\_host \<\< n['ipaddress']  
> end
> 
> ...
> 
> template "/etc/nagios/nrpe.cfg" do  
> source "nrpe.cfg.erb"  
> owner "nagios"  
> group "nagios"  
> mode "0644"  
> variables :mon\_host =\> mon\_host  
> notifies :restart, resources(:service =\> "nagios-nrpe-server")  
> end  
> **END**
> 
> The initial problem on upgrading the server was the search was previously "role:..." which was working fine but needed to be changed to "roles:...".
> 
> However this time nothing changed and simply logging in to the problem hosts and re-running chef-client fixed it.
> 
> What went wrong and how to I track it down so it doesn't happen again?
> 
> -ash

--  
Opscode, Inc.  
Adam Jacob, Chief Product Officer  
T: (206) 619-7151 E: [adam@opscode.com](mailto:adam@opscode.com)

---

<div class="post-metadata">

**Author:** ![Ash\_Berlin](https://avatars.discourse-cdn.com/v4/letter/a/0ea827/32.png) [@Ash\_Berlin](https://discourse.chef.io/u/Ash_Berlin)\
**Post date:** [January 27, 2011, 12:30am UTC](https://discourse.chef.io/t/problems-with-search/1671/3 "2011-01-27T00:30:34Z")

</div>

[https://gist.github.com/860921611bbcaca7b338](https://gist.github.com/860921611bbcaca7b338)

Let me know if you need more and I'll get it in the morning - its 00:30 here now.

-ash

On 27 Jan 2011, at 00:12, Adam Jacob wrote:

> Ash, can you give use some gists of the log output from the Chef  
> Server at the time?
> 
> Solr logs would help as well.
> 
> Adam
> 
> On Wed, Jan 26, 2011 at 4:09 PM, Ash Berlin [ash\_opscode@firemirror.com](mailto:ash_opscode@firemirror.com) wrote:
> 
> > So some of my nagios checks started failing just now because the search that fills out allowed\_hosts in nrpe.cfg was empty. This is the second time this happened to me - the first was immediately after upgrading to chef-server 0.9.12.
> > 
> > The server is The clients in question are still on 0.9.8.
> > 
> > My cookbook code is (probably the standard nagios::client cookbook, possibly a few versions out of date):
> > 
> > search(:node, "roles:monitoring AND app\_environment:#{node[:app\_environment]}") do |n|  
> > mon\_host \<\< n['ipaddress']  
> > end
> > 
> > ...
> > 
> > template "/etc/nagios/nrpe.cfg" do  
> > source "nrpe.cfg.erb"  
> > owner "nagios"  
> > group "nagios"  
> > mode "0644"  
> > variables :mon\_host =\> mon\_host  
> > notifies :restart, resources(:service =\> "nagios-nrpe-server")  
> > end  
> > **END**
> > 
> > The initial problem on upgrading the server was the search was previously "role:..." which was working fine but needed to be changed to "roles:...".
> > 
> > However this time nothing changed and simply logging in to the problem hosts and re-running chef-client fixed it.
> > 
> > What went wrong and how to I track it down so it doesn't happen again?
> > 
> > -ash

---

<div class="post-metadata">

**Author:** ![Adam\_Jacob](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/adam_jacob/32/292_2.png) [@Adam\_Jacob](https://discourse.chef.io/u/Adam_Jacob)\
**Post date:** [January 27, 2011, 2:52am UTC](https://discourse.chef.io/t/problems-with-search/1671/4 "2011-01-27T02:52:33Z")

</div>

Odd - can you give us another gist with some of the solr log going  
further back, say a few minutes before where the log starts now?

In your chef server logs, what was happening with node convergence for  
the machine with the actual monitoring role at the time the search  
request came in?

Adam

On Wed, Jan 26, 2011 at 4:30 PM, Ash Berlin [ash\_opscode@firemirror.com](mailto:ash_opscode@firemirror.com) wrote:

> [https://gist.github.com/860921611bbcaca7b338](https://gist.github.com/860921611bbcaca7b338)
> 
> Let me know if you need more and I'll get it in the morning - its 00:30 here now.
> 
> -ash
> 
> On 27 Jan 2011, at 00:12, Adam Jacob wrote:
> 
> > Ash, can you give use some gists of the log output from the Chef  
> > Server at the time?
> > 
> > Solr logs would help as well.
> > 
> > Adam
> > 
> > On Wed, Jan 26, 2011 at 4:09 PM, Ash Berlin [ash\_opscode@firemirror.com](mailto:ash_opscode@firemirror.com) wrote:
> > 
> > > So some of my nagios checks started failing just now because the search that fills out allowed\_hosts in nrpe.cfg was empty. This is the second time this happened to me - the first was immediately after upgrading to chef-server 0.9.12.
> > > 
> > > The server is The clients in question are still on 0.9.8.
> > > 
> > > My cookbook code is (probably the standard nagios::client cookbook, possibly a few versions out of date):
> > > 
> > > search(:node, "roles:monitoring AND app\_environment:#{node[:app\_environment]}") do |n|  
> > > mon\_host \<\< n['ipaddress']  
> > > end
> > > 
> > > ...
> > > 
> > > template "/etc/nagios/nrpe.cfg" do  
> > > source "nrpe.cfg.erb"  
> > > owner "nagios"  
> > > group "nagios"  
> > > mode "0644"  
> > > variables :mon\_host =\> mon\_host  
> > > notifies :restart, resources(:service =\> "nagios-nrpe-server")  
> > > end  
> > > **END**
> > > 
> > > The initial problem on upgrading the server was the search was previously "role:..." which was working fine but needed to be changed to "roles:...".
> > > 
> > > However this time nothing changed and simply logging in to the problem hosts and re-running chef-client fixed it.
> > > 
> > > What went wrong and how to I track it down so it doesn't happen again?
> > > 
> > > -ash

--  
Opscode, Inc.  
Adam Jacob, Chief Product Officer  
T: (206) 619-7151 E: [adam@opscode.com](mailto:adam@opscode.com)

---

<div class="post-metadata">

**Author:** ![Gilles\_Devaux](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/gilles_devaux/32/744_2.png) [@Gilles\_Devaux](https://discourse.chef.io/u/Gilles_Devaux)\
**Post date:** [January 27, 2011, 4:54pm UTC](https://discourse.chef.io/t/problems-with-search/1671/5 "2011-01-27T16:54:45Z")

</div>

I reported the same issue under the title "search sometimes returns no  
result", I pasted my logs in the email.

On Wed, Jan 26, 2011 at 6:52 PM, Adam Jacob [adam@opscode.com](mailto:adam@opscode.com) wrote:

> Odd - can you give us another gist with some of the solr log going  
> further back, say a few minutes before where the log starts now?
> 
> In your chef server logs, what was happening with node convergence for  
> the machine with the actual monitoring role at the time the search  
> request came in?
> 
> Adam
> 
> On Wed, Jan 26, 2011 at 4:30 PM, Ash Berlin [ash\_opscode@firemirror.com](mailto:ash_opscode@firemirror.com) wrote:
> 
> > [https://gist.github.com/860921611bbcaca7b338](https://gist.github.com/860921611bbcaca7b338)
> > 
> > Let me know if you need more and I'll get it in the morning - its 00:30 here now.
> > 
> > -ash
> > 
> > On 27 Jan 2011, at 00:12, Adam Jacob wrote:
> > 
> > > Ash, can you give use some gists of the log output from the Chef  
> > > Server at the time?
> > > 
> > > Solr logs would help as well.
> > > 
> > > Adam
> > > 
> > > On Wed, Jan 26, 2011 at 4:09 PM, Ash Berlin [ash\_opscode@firemirror.com](mailto:ash_opscode@firemirror.com) wrote:
> > > 
> > > > So some of my nagios checks started failing just now because the search that fills out allowed\_hosts in nrpe.cfg was empty. This is the second time this happened to me - the first was immediately after upgrading to chef-server 0.9.12.
> > > > 
> > > > The server is The clients in question are still on 0.9.8.
> > > > 
> > > > My cookbook code is (probably the standard nagios::client cookbook, possibly a few versions out of date):
> > > > 
> > > > search(:node, "roles:monitoring AND app\_environment:#{node[:app\_environment]}") do |n|  
> > > > mon\_host \<\< n['ipaddress']  
> > > > end
> > > > 
> > > > ...
> > > > 
> > > > template "/etc/nagios/nrpe.cfg" do  
> > > > source "nrpe.cfg.erb"  
> > > > owner "nagios"  
> > > > group "nagios"  
> > > > mode "0644"  
> > > > variables :mon\_host =\> mon\_host  
> > > > notifies :restart, resources(:service =\> "nagios-nrpe-server")  
> > > > end  
> > > > **END**
> > > > 
> > > > The initial problem on upgrading the server was the search was previously "role:..." which was working fine but needed to be changed to "roles:...".
> > > > 
> > > > However this time nothing changed and simply logging in to the problem hosts and re-running chef-client fixed it.
> > > > 
> > > > What went wrong and how to I track it down so it doesn't happen again?
> > > > 
> > > > -ash
> 
> --  
> Opscode, Inc.  
> Adam Jacob, Chief Product Officer  
> T: (206) 619-7151 E: [adam@opscode.com](mailto:adam@opscode.com)

---

<div class="post-metadata">

**Author:** ![kallistec](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/kallistec/32/23_2.png) [@kallistec](https://discourse.chef.io/u/kallistec)\
**Post date:** [January 27, 2011, 5:11pm UTC](https://discourse.chef.io/t/problems-with-search/1671/6 "2011-01-27T17:11:51Z")

</div>

On Thu, Jan 27, 2011 at 8:54 AM, Gilles Devaux [gilles.devaux@gmail.com](mailto:gilles.devaux@gmail.com) wrote:

> I reported the same issue under the title "search sometimes returns no  
> result", I pasted my logs in the email.

Is this a search for one specific host? Do the bad searches coincide  
with chef client runs on the searched-for system? If so, I think this  
is a resolved bug on the client, and you can fix it by upgrading the  
client on the searched-for box to 0.9.12.

The issue I'm thinking of is that the node is saved at the beginning  
of the run so that it has the correct run list when the client asks  
the server for the list of cookbooks to sync. However, some of the  
attributes have been wiped and not yet rebuilt when this node save  
occurs. Sound like your issue?

Thanks,  
Dan

---

<div class="post-metadata">

**Author:** ![Ash\_Berlin](https://avatars.discourse-cdn.com/v4/letter/a/0ea827/32.png) [@Ash\_Berlin](https://discourse.chef.io/u/Ash_Berlin)\
**Post date:** [January 27, 2011, 5:18pm UTC](https://discourse.chef.io/t/problems-with-search/1671/7 "2011-01-27T17:18:33Z")

</div>

This could be it yes - basically i'm searching on nrpe client machines for the host running nagois to allow nrpe connections from just this host, and all my clients are still on 0.9.8.

-ash

On 27 Jan 2011, at 17:11, Daniel DeLeo wrote:

> On Thu, Jan 27, 2011 at 8:54 AM, Gilles Devaux [gilles.devaux@gmail.com](mailto:gilles.devaux@gmail.com) wrote:
> 
> > I reported the same issue under the title "search sometimes returns no  
> > result", I pasted my logs in the email.
> 
> Is this a search for one specific host? Do the bad searches coincide  
> with chef client runs on the searched-for system? If so, I think this  
> is a resolved bug on the client, and you can fix it by upgrading the  
> client on the searched-for box to 0.9.12.
> 
> The issue I'm thinking of is that the node is saved at the beginning  
> of the run so that it has the correct run list when the client asks  
> the server for the list of cookbooks to sync. However, some of the  
> attributes have been wiped and not yet rebuilt when this node save  
> occurs. Sound like your issue?
> 
> Thanks,  
> Dan

---

<div class="post-metadata">

**Author:** ![Paul\_Paradise](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/paul_paradise/32/846_2.png) [@Paul\_Paradise](https://discourse.chef.io/u/Paul_Paradise)\
**Post date:** [January 28, 2011, 9:01pm UTC](https://discourse.chef.io/t/problems-with-search/1671/8 "2011-01-28T21:01:27Z")

</div>

I've had similar problems in the past where the nagios would fail to restart  
because a client saved its node[:ipaddress] as nil, and the next chef-client  
run on my nagios server broke it. Anecdotally, it only happens when I reboot  
a node. I suspect there's a race condition at startup where chef-client  
starts running (and collects the ohai attributes) before the node gets a  
DHCP lease, and ends up writing a nil node[:ipaddress] when the run  
completes.

Perhaps you have the same problem, but with the node[:ipaddress]  
disappearing from your nagios node rather than the monitored nodes?

-Paul

On Thu, Jan 27, 2011 at 9:18 AM, Ash Berlin [ash\_opscode@firemirror.com](mailto:ash_opscode@firemirror.com)wrote:

> This could be it yes - basically i'm searching on nrpe client machines for  
> the host running nagois to allow nrpe connections from just this host, and  
> all my clients are still on 0.9.8.
> 
> -ash
> 
> On 27 Jan 2011, at 17:11, Daniel DeLeo wrote:
> 
> > On Thu, Jan 27, 2011 at 8:54 AM, Gilles Devaux [gilles.devaux@gmail.com](mailto:gilles.devaux@gmail.com)  
> > wrote:
> > 
> > > I reported the same issue under the title "search sometimes returns no  
> > > result", I pasted my logs in the email.
> > 
> > Is this a search for one specific host? Do the bad searches coincide  
> > with chef client runs on the searched-for system? If so, I think this  
> > is a resolved bug on the client, and you can fix it by upgrading the  
> > client on the searched-for box to 0.9.12.
> > 
> > The issue I'm thinking of is that the node is saved at the beginning  
> > of the run so that it has the correct run list when the client asks  
> > the server for the list of cookbooks to sync. However, some of the  
> > attributes have been wiped and not yet rebuilt when this node save  
> > occurs. Sound like your issue?
> > 
> > Thanks,  
> > Dan

---

<div class="post-metadata">

**Author:** ![Jason\_Cook](https://avatars.discourse-cdn.com/v4/letter/j/22d042/32.png) [@Jason\_Cook](https://discourse.chef.io/u/Jason_Cook)\
**Post date:** [January 28, 2011, 11:28pm UTC](https://discourse.chef.io/t/problems-with-search/1671/9 "2011-01-28T23:28:05Z")

</div>

On Fri, Jan 28, 2011 at 1:01 PM, Paul Paradise [paulpar@gmail.com](mailto:paulpar@gmail.com) wrote:

> I've had similar problems in the past where the nagios would fail to  
> restart because a client saved its node[:ipaddress] as nil, and the next  
> chef-client run on my nagios server broke it. Anecdotally, it only happens  
> when I reboot a node. I suspect there's a race condition at startup where  
> chef-client starts running (and collects the ohai attributes) before the  
> node gets a DHCP lease, and ends up writing a nil node[:ipaddress] when the  
> run completes.
> 
> Perhaps you have the same problem, but with the node[:ipaddress]  
> disappearing from your nagios node rather than the monitored nodes?
> 
> -Paul

I've seen it happen on a machine that gets into an oom condition when  
running ohai. The partial data for the run still gets submitted and indexed  
into the search engine. We tend to write very defensive recipes when using  
search to guard against malformed data. We also switched to starting chef  
from cron, though the memory leak issues we saw in daemon mode I think are  
mostly fixed.

Jason