Nodes not showing up in `knife status`

Chef Infra (archive)
1

Hi Chefs,

I’ve recently added some new nodes to our chef-server, but when I run knife status, I don’t see these nodes in the list.

knife node edit $NODE works
knife client edit $NODE works

the nodes show up in the webui.

the name in /etc/chef/client.rb is correct on the nodes.

Is there anything else I should look at?

75 total nodes show up when I run knife status. We’re missing about 6 nodes from the list that I can see so far. I have a full range of last-run-at times; from minutes to over 6000 hours (some nodes I haven’t upgraded the client on for quite a while), so I don’t think it’s just truncating the list.

Thanks in advance!

…spike

2

Hello Spike,
Knife status uses the solr based search mechanism to find all the nodes .
It seems the nodes that are missing don't meet that search criteria .

Can you share the entire knife status command ?

--
Regards, Chirag

On Tuesday, January 29, 2013, Spike Grobstein wrote:

Hi Chefs,

I've recently added some new nodes to our chef-server, but when I run
knife status, I don't see these nodes in the list.

knife node edit $NODE works
knife client edit $NODE works

the nodes show up in the webui.

the name in /etc/chef/client.rb is correct on the nodes.

Is there anything else I should look at?

75 total nodes show up when I run knife status. We're missing about 6
nodes from the list that I can see so far. I have a full range of
last-run-at times; from minutes to over 6000 hours (some nodes I haven't
upgraded the client on for quite a while), so I don't think it's just
truncating the list.

Thanks in advance!

...spike

--
Regards*,*
Chirag Jog
Chief Technology Officer,
Clogeny Technologies | http://clogeny.com
(M) 0091-9766619440 | Skype: chirag.jog

3

Hi,

Are the nodes that are missing either windows nodes or nodes that generate
lots of data from ohai? If so it is probably that the data is not being
indexed correctly. Chef silently fails to index large nodes - you can fix
this by increasing the "maxFieldLength" value. Watch out if you are doing
this under ubuntu as it seems to have two copies of this file installed.

On Tue, Jan 29, 2013 at 8:55 AM, Spike Grobstein
spike@ticketevolution.comwrote:

Hi Chefs,

I've recently added some new nodes to our chef-server, but when I run
knife status, I don't see these nodes in the list.

knife node edit $NODE works
knife client edit $NODE works

the nodes show up in the webui.

the name in /etc/chef/client.rb is correct on the nodes.

Is there anything else I should look at?

75 total nodes show up when I run knife status. We're missing about 6
nodes from the list that I can see so far. I have a full range of
last-run-at times; from minutes to over 6000 hours (some nodes I haven't
upgraded the client on for quite a while), so I don't think it's just
truncating the list.

Thanks in advance!

...spike

--
Cheers,

Peter Donald

4

Hi Peter,

All of my nodes are Ubuntu and pretty much every node in our infrastructure have similar specs. They're all virtualized and nearly all of them are dual CPU, 4GB RAM (with only a couple of exceptions).

I'll look into the maxFieldLength when I get in in the morning and see if that helps.

it definitely looks like it's every server that I've provisioned in the last month or so. Also, I just noticed that all nodes that aren't showing up are Ubuntu 12.04. Could that mean anything?

thanks for the info!

...spike

On Jan 28, 2013, at 9:27 PM, Peter Donald peter@realityforge.org wrote:

Hi,

Are the nodes that are missing either windows nodes or nodes that generate lots of data from ohai? If so it is probably that the data is not being indexed correctly. Chef silently fails to index large nodes - you can fix this by increasing the "maxFieldLength" value. Watch out if you are doing this under ubuntu as it seems to have two copies of this file installed.

On Tue, Jan 29, 2013 at 8:55 AM, Spike Grobstein spike@ticketevolution.com wrote:
Hi Chefs,

I've recently added some new nodes to our chef-server, but when I run knife status, I don't see these nodes in the list.

knife node edit $NODE works
knife client edit $NODE works

the nodes show up in the webui.

the name in /etc/chef/client.rb is correct on the nodes.

Is there anything else I should look at?

75 total nodes show up when I run knife status. We're missing about 6 nodes from the list that I can see so far. I have a full range of last-run-at times; from minutes to over 6000 hours (some nodes I haven't upgraded the client on for quite a while), so I don't think it's just truncating the list.

Thanks in advance!

...spike

--
Cheers,

Peter Donald

5

Hi Chirag,

I'm just running:

knife status

no additional options or anything.

...spike

On Jan 28, 2013, at 9:22 PM, Chirag Jog chirag@clogeny.com wrote:

Hello Spike,
Knife status uses the solr based search mechanism to find all the nodes . It seems the nodes that are missing don't meet that search criteria .

Can you share the entire knife status command ?

--
Regards, Chirag

On Tuesday, January 29, 2013, Spike Grobstein wrote:
Hi Chefs,

I've recently added some new nodes to our chef-server, but when I run knife status, I don't see these nodes in the list.

knife node edit $NODE works
knife client edit $NODE works

the nodes show up in the webui.

the name in /etc/chef/client.rb is correct on the nodes.

Is there anything else I should look at?

75 total nodes show up when I run knife status. We're missing about 6 nodes from the list that I can see so far. I have a full range of last-run-at times; from minutes to over 6000 hours (some nodes I haven't upgraded the client on for quite a while), so I don't think it's just truncating the list.

Thanks in advance!

...spike

--
Regards,
Chirag Jog
Chief Technology Officer,
Clogeny Technologies | http://clogeny.com
(M) 0091-9766619440 | Skype: chirag.jog

6

Hi,

On Tue, Jan 29, 2013 at 1:46 PM, Spike Grobstein
spike@ticketevolution.comwrote:

I'll look into the maxFieldLength when I get in in the morning and see if
that helps.

Be sure to restart solr and run "knife index rebuild" after you change that
value.

it definitely looks like it's every server that I've provisioned in the
last month or so. Also, I just noticed that all nodes that aren't showing
up are Ubuntu 12.04. Could that mean anything?

We run Ubuntu 12.04 so that in itself is unlikely to be the sole factor.

--
Cheers,

Peter Donald

7

Hi Spike,
Another thing to try is that - are those nodes showing up in knife search ?
You could try knife search node 'platform:ubuntu' or knife search node
'name: or name:*. This help identify if they are being
indexed at all.

Regards*,*
Chirag Jog
Chief Technology Officer,
Clogeny Technologies | http://clogeny.com
(M) 0091-9766619440 | Skype: chirag.jog

On Tue, Jan 29, 2013 at 8:19 AM, Spike Grobstein
spike@ticketevolution.comwrote:

Hi Chirag,

I'm just running:

knife status

no additional options or anything.

...spike

On Jan 28, 2013, at 9:22 PM, Chirag Jog chirag@clogeny.com wrote:

Hello Spike,
Knife status uses the solr based search mechanism to find all the nodes .
It seems the nodes that are missing don't meet that search criteria .

Can you share the entire knife status command ?

--
Regards, Chirag

On Tuesday, January 29, 2013, Spike Grobstein wrote:

Hi Chefs,

I've recently added some new nodes to our chef-server, but when I run
knife status, I don't see these nodes in the list.

knife node edit $NODE works
knife client edit $NODE works

the nodes show up in the webui.

the name in /etc/chef/client.rb is correct on the nodes.

Is there anything else I should look at?

75 total nodes show up when I run knife status. We're missing about 6
nodes from the list that I can see so far. I have a full range of
last-run-at times; from minutes to over 6000 hours (some nodes I haven't
upgraded the client on for quite a while), so I don't think it's just
truncating the list.

Thanks in advance!

...spike

--
Regards*,*
Chirag Jog
Chief Technology Officer,
Clogeny Technologies | http://clogeny.com
(M) 0091-9766619440 | Skype: chirag.jog

8

As has previously been said, "knife node show" hits couch, "knife status"
hits the index. If nodes are missing from the index, and they are identical
to your other nodes and have failed to appear within a given time-frame, it
is likely that the expander is not properly forwarding couch updates to the
index.

Try:
http://wiki.opscode.com/display/chef/Chef+Indexer#ChefIndexer-ChefExpanderOperationandTroubleshooting
for
steps to examine the expander state, and also consider interrogating the
length of your queues.

If you don't want to or can't go through the exercise of monitoring all of
the individual back-end components then a a nice basic litmus check is to
check your couchdb ohai_time attribute against the same attribute stored in
solr, and ensure that they match. ohai_time is the time the attributes were
last persisted to the back-end by the client.

Here is a simple example:

#!/usr/bin/env ruby
require 'chef/node'

Assumes Chef::Config is populated with correct values.

hostname = 'host.corp.com'

begin
couch_node = Chef::Node.cdb_load(hostname)
rescue Chef::Exceptions::CouchDBNotFound

I'm probably not managed by Chef.

rescue Exception => e

Something else went terribly wrong

end

begin
index_node = Chef::Search::Query.new.search(:node, "name:#{hostname}")
rescue Exception => e

Something went terribly wrong

end

couchdb_time = couch_node.ohai_time.to_i

begin
index_time = index_node[0][0].ohai_time.to_i
rescue NoMethodError

The node index data is missing. Alert!

end

Now check that neither couchdb_time or index_time are nil, and that they

aren't too

old, and they're identical (or within some acceptable boundary), etc, etc.

--
Sam.

On Tue, Jan 29, 2013 at 2:22 AM, Chirag Jog chirag@clogeny.com wrote:

Hello Spike,
Knife status uses the solr based search mechanism to find all the nodes .
It seems the nodes that are missing don't meet that search criteria .

Can you share the entire knife status command ?

--
Regards, Chirag

On Tuesday, January 29, 2013, Spike Grobstein wrote:

Hi Chefs,

I've recently added some new nodes to our chef-server, but when I run
knife status, I don't see these nodes in the list.

knife node edit $NODE works
knife client edit $NODE works

the nodes show up in the webui.

the name in /etc/chef/client.rb is correct on the nodes.

Is there anything else I should look at?

75 total nodes show up when I run knife status. We're missing about 6
nodes from the list that I can see so far. I have a full range of
last-run-at times; from minutes to over 6000 hours (some nodes I haven't
upgraded the client on for quite a while), so I don't think it's just
truncating the list.

Thanks in advance!

...spike

--
Regards*,*
Chirag Jog
Chief Technology Officer,
Clogeny Technologies | http://clogeny.com
(M) 0091-9766619440 | Skype: chirag.jog

9

ahhh. ok. This is all starting to make sense now. I didn't fully understand how all the components of chef worked together, so I didn't know where to look.

the nodes were not coming up in searches, so it definitely looked like my new nodes weren't being indexed in solr. I tried to run knife index rebuild -y, but got:

$ knife index rebuild -y
ERROR: Server returned error for http://localhost:4000//search/reindex, retrying 1/5 in 4s

(I have an ssh tunnel to access the private network of our infrastructure)

so I took a look at /var/log/chef/solr.log and saw:

Jan 29, 2013 1:52:04 PM org.apache.solr.core.SolrCore execute
INFO: webapp=/solr path=/update params={} status=500 QTime=6
Jan 29, 2013 1:52:04 PM org.apache.solr.common.SolrException log
SEVERE: java.io.FileNotFoundException: /var/cache/chef/solr/data/index/_6xjj.fdt (Permission denied)

I'd had a problem in early january when upgrading my chef-server system and inadvertently deleted the chef user, so I recreated it and went around and had to re-chown everything. I missed the /var/cache/chef directory, so after fixing that, this worked:

$ knife index rebuild -y
Chef::ApiClient: success
Chef::DataBag: success
Chef::Environment: success
Chef::Node: success
Chef::Role: success

now knife status shows all of my nodes, including the new ones.

Thanks for all the help!

...spike

On Jan 29, 2013, at 2:07 AM, Chirag Jog wrote:

Hi Spike,
Another thing to try is that - are those nodes showing up in knife search ?
You could try knife search node 'platform:ubuntu' or knife search node 'name: or name:*. This help identify if they are being indexed at all.

Regards,
Chirag Jog
Chief Technology Officer,
Clogeny Technologies | http://clogeny.com
(M) 0091-9766619440 | Skype: chirag.jog

On Tue, Jan 29, 2013 at 8:19 AM, Spike Grobstein spike@ticketevolution.com wrote:
Hi Chirag,

I'm just running:

knife status

no additional options or anything.

...spike

On Jan 28, 2013, at 9:22 PM, Chirag Jog chirag@clogeny.com wrote:

Hello Spike,
Knife status uses the solr based search mechanism to find all the nodes . It seems the nodes that are missing don't meet that search criteria .

Can you share the entire knife status command ?

--
Regards, Chirag

On Tuesday, January 29, 2013, Spike Grobstein wrote:
Hi Chefs,

I've recently added some new nodes to our chef-server, but when I run knife status, I don't see these nodes in the list.

knife node edit $NODE works
knife client edit $NODE works

the nodes show up in the webui.

the name in /etc/chef/client.rb is correct on the nodes.

Is there anything else I should look at?

75 total nodes show up when I run knife status. We're missing about 6 nodes from the list that I can see so far. I have a full range of last-run-at times; from minutes to over 6000 hours (some nodes I haven't upgraded the client on for quite a while), so I don't think it's just truncating the list.

Thanks in advance!

...spike

--
Regards,
Chirag Jog
Chief Technology Officer,
Clogeny Technologies | http://clogeny.com
(M) 0091-9766619440 | Skype: chirag.jog

10

Your best bet is probably to build chef server using chef solo so you don't run into these manual config problems

http://wiki.opscode.com/display/chef/Installing+Chef+Server+using+Chef+Solo

That's why we're using chef :slight_smile:

On Jan 29, 2013, at 9:00 AM, Spike Grobstein spike@ticketevolution.com wrote:

ahhh. ok. This is all starting to make sense now. I didn't fully understand how all the components of chef worked together, so I didn't know where to look.

the nodes were not coming up in searches, so it definitely looked like my new nodes weren't being indexed in solr. I tried to run knife index rebuild -y, but got:

$ knife index rebuild -y
ERROR: Server returned error for http://localhost:4000//search/reindex, retrying 1/5 in 4s

(I have an ssh tunnel to access the private network of our infrastructure)

so I took a look at /var/log/chef/solr.log and saw:

Jan 29, 2013 1:52:04 PM org.apache.solr.core.SolrCore execute
INFO: webapp=/solr path=/update params={} status=500 QTime=6
Jan 29, 2013 1:52:04 PM org.apache.solr.common.SolrException log
SEVERE: java.io.FileNotFoundException: /var/cache/chef/solr/data/index/_6xjj.fdt (Permission denied)

I'd had a problem in early january when upgrading my chef-server system and inadvertently deleted the chef user, so I recreated it and went around and had to re-chown everything. I missed the /var/cache/chef directory, so after fixing that, this worked:

$ knife index rebuild -y
Chef::ApiClient: success
Chef::DataBag: success
Chef::Environment: success
Chef::Node: success
Chef::Role: success

now knife status shows all of my nodes, including the new ones.

Thanks for all the help!

...spike

On Jan 29, 2013, at 2:07 AM, Chirag Jog wrote:

Hi Spike,
Another thing to try is that - are those nodes showing up in knife search ?
You could try knife search node 'platform:ubuntu' or knife search node 'name: or name:*. This help identify if they are being indexed at all.

Regards,
Chirag Jog
Chief Technology Officer,
Clogeny Technologies | http://clogeny.com
(M) 0091-9766619440 | Skype: chirag.jog

On Tue, Jan 29, 2013 at 8:19 AM, Spike Grobstein spike@ticketevolution.com wrote:
Hi Chirag,

I'm just running:

knife status

no additional options or anything.

...spike

On Jan 28, 2013, at 9:22 PM, Chirag Jog chirag@clogeny.com wrote:

Hello Spike,
Knife status uses the solr based search mechanism to find all the nodes . It seems the nodes that are missing don't meet that search criteria .

Can you share the entire knife status command ?

--
Regards, Chirag

On Tuesday, January 29, 2013, Spike Grobstein wrote:
Hi Chefs,

I've recently added some new nodes to our chef-server, but when I run knife status, I don't see these nodes in the list.

knife node edit $NODE works
knife client edit $NODE works

the nodes show up in the webui.

the name in /etc/chef/client.rb is correct on the nodes.

Is there anything else I should look at?

75 total nodes show up when I run knife status. We're missing about 6 nodes from the list that I can see so far. I have a full range of last-run-at times; from minutes to over 6000 hours (some nodes I haven't upgraded the client on for quite a while), so I don't think it's just truncating the list.

Thanks in advance!

...spike

--
Regards,
Chirag Jog
Chief Technology Officer,
Clogeny Technologies | http://clogeny.com
(M) 0091-9766619440 | Skype: chirag.jog

11

hey guys,
i bootstrapped my chef node,able to see “knife client list” from chefworkstation but “knife node list” and “knife status” shows nothing to me.my ssl connection is fine with chef server.
kindly help

12

@harshblog150 please don’t resurrect a 5 year old thread for your own question.

Please open a new thread and provide the detailed output from the commands. My guess is this is a validator based bootstrap and that the bootstraps first chef run didn’t complete successfully. I am presuming knife client list is only listing the validator or workstation key which would mean this is the case.