Hey folks.
So I discovered that chef-expander had stopped indexing anything on my chef
server as of the 3rd of august. Maybe it was due to an upgrade, or due to a
disk problem, or who knows what.
So I restarted the whole server, and ran chef-server-ctl reindex.
After a few minutes everything was there except for the nodes
Nodes, it seems, only show up in search when chef client runs on them.
And that happens overnight.
And some of my recipes depend on the results of search.
Any ideas? Any command I can run to push at least what is in the pg db for
the nodes into solr?
Thanks!
Hi there,
hikeit@gmail.com writes:
So I restarted the whole server, and ran chef-server-ctl reindex.
After a few minutes everything was there except for the nodes
Nodes, it seems, only show up in search when chef client runs on them.
And that happens overnight.
And some of my recipes depend on the results of search.
Any ideas? Any command I can run to push at least what is in the pg db for
the nodes into solr?
So the way reindex works in Chef 11 Server is that it fetches all
objects out of the db and puts them on the queue to be indexed.
Can you confirm that your nodes are in the db? For example knife node list shows all the nodes you're expecting to see?
Might then be worth looking at some logs of expander and solr. Do you
see logs indicating the reindexing happened? Can you find node data
there or confirm no node data?
How long did you wait? Is it possible that you're expander is working
through the backlog?
--
Seth Falcon | Development Lead | Opscode | @sfalcon
knife node list shows all the nodes, knife search node id:* lists only the
4 nodes that checked in (most of my nodes only check in overnight)
knife search <insert_data_bag> id:* shows all data bag entries, and looking
at the expander log, it definitely reindexed all of the data bag items.
It is now 7 hours later, so i'm thinking it's done and just didn't reindex
the nodes...
-Jesse
On Fri, Aug 9, 2013 at 12:42 PM, Seth Falcon seth@opscode.com wrote:
Hi there,
hikeit@gmail.com writes:
So I restarted the whole server, and ran chef-server-ctl reindex.
After a few minutes everything was there except for the nodes
Nodes, it seems, only show up in search when chef client runs on them.
And that happens overnight.
And some of my recipes depend on the results of search.
Any ideas? Any command I can run to push at least what is in the pg db
for
the nodes into solr?
So the way reindex works in Chef 11 Server is that it fetches all
objects out of the db and puts them on the queue to be indexed.
Can you confirm that your nodes are in the db? For example knife node list shows all the nodes you're expecting to see?
Might then be worth looking at some logs of expander and solr. Do you
see logs indicating the reindexing happened? Can you find node data
there or confirm no node data?
How long did you wait? Is it possible that you're expander is working
through the backlog?
--
Seth Falcon | Development Lead | Opscode | @sfalcon
hikeit@gmail.com writes:
knife node list shows all the nodes, knife search node id:* lists only the
4 nodes that checked in (most of my nodes only check in overnight)
knife search <insert_data_bag> id:* shows all data bag entries, and looking
at the expander log, it definitely reindexed all of the data bag items.
It is now 7 hours later, so i'm thinking it's done and just didn't reindex
the nodes...
Indeed, 7 hours should be quite long enough 
I just did the following experiment with a nightly build of Chef Server
in a vm:
- Created a few nodes and roles using knife
- chef-server-ctl stop chef-solr
- Nuked the solr index:
sudo rm -rf /var/opt/chef-server/chef-solr/data/index
- chef-server-ctl start chef-solr
- Created a node and a role using knife
- Wait 60 sec and verify data created in #5 is searchable. Also
verified that data created in #1 was not found in search
- chef-server-ctl reindex
- Verify all data in search
What you're seeing with missing node data could certainly be a bug. At
this point, I would suggest attempting another c-s-c reindex with a
careful eye on the logs of chef-expander, chef-solr, and erchef.
Can you verify that your node data is hitting the expander during
reindex?
Any crash logs in the erchef logs related in time to reindexing?
Errors or logs of chef-solr?
Also worth looking at rabbitmq logs -- although a problem seems unlikely
to be located there since some data types appear to get reindexed
properly and the queue doesn't know anything about the object types.
Anyhow, hope these suggestions help you track down some more info.
--
Seth Falcon | Development Lead | Opscode | @sfalcon
1 Like
Thanks!
I probably won't reindex our primary again any time soon, but I'll give it
a try in some of our test environments next week. Its possible the reindex
got interrupted or failed, i didn't have a chance to peruse all of the logs.
-Jesse
On Sat, Aug 10, 2013 at 1:10 PM, Seth Falcon seth@opscode.com wrote:
hikeit@gmail.com writes:
knife node list shows all the nodes, knife search node id:* lists only
the
4 nodes that checked in (most of my nodes only check in overnight)
knife search <insert_data_bag> id:* shows all data bag entries, and
looking
at the expander log, it definitely reindexed all of the data bag items.
It is now 7 hours later, so i'm thinking it's done and just didn't
reindex
the nodes...
Indeed, 7 hours should be quite long enough
I just did the following experiment with a nightly build of Chef Server
in a vm:
- Created a few nodes and roles using knife
- chef-server-ctl stop chef-solr
- Nuked the solr index:
sudo rm -rf /var/opt/chef-server/chef-solr/data/index
- chef-server-ctl start chef-solr
- Created a node and a role using knife
- Wait 60 sec and verify data created in #5 is searchable. Also
verified that data created in #1 was not found in search
- chef-server-ctl reindex
- Verify all data in search
What you're seeing with missing node data could certainly be a bug. At
this point, I would suggest attempting another c-s-c reindex with a
careful eye on the logs of chef-expander, chef-solr, and erchef.
Can you verify that your node data is hitting the expander during
reindex?
Any crash logs in the erchef logs related in time to reindexing?
Errors or logs of chef-solr?
Also worth looking at rabbitmq logs -- although a problem seems unlikely
to be located there since some data types appear to get reindexed
properly and the queue doesn't know anything about the object types.
Anyhow, hope these suggestions help you track down some more info.
--
Seth Falcon | Development Lead | Opscode | @sfalcon
The other thing to check is the amount of node data stored per node. If it
exceeds a certain threshold the nodes will silently fail to be indexed. We
have to up the maxKeySize limit for solr every time we install otherwise
our windows nodes will not appear in the index
On Sunday, August 11, 2013, Jesse Campbell wrote:
Thanks!
I probably won't reindex our primary again any time soon, but I'll give it
a try in some of our test environments next week. Its possible the reindex
got interrupted or failed, i didn't have a chance to peruse all of the logs.
-Jesse
On Sat, Aug 10, 2013 at 1:10 PM, Seth Falcon <seth@opscode.com<javascript:_e({}, 'cvml', 'seth@opscode.com');>
wrote:
hikeit@gmail.com <javascript:_e({}, 'cvml', 'hikeit@gmail.com');> writes:
knife node list shows all the nodes, knife search node id:* lists only
the
4 nodes that checked in (most of my nodes only check in overnight)
knife search <insert_data_bag> id:* shows all data bag entries, and
looking
at the expander log, it definitely reindexed all of the data bag items.
It is now 7 hours later, so i'm thinking it's done and just didn't
reindex
the nodes...
Indeed, 7 hours should be quite long enough
I just did the following experiment with a nightly build of Chef Server
in a vm:
- Created a few nodes and roles using knife
- chef-server-ctl stop chef-solr
- Nuked the solr index:
sudo rm -rf /var/opt/chef-server/chef-solr/data/index
- chef-server-ctl start chef-solr
- Created a node and a role using knife
- Wait 60 sec and verify data created in #5 is searchable. Also
verified that data created in #1 was not found in search
- chef-server-ctl reindex
- Verify all data in search
What you're seeing with missing node data could certainly be a bug. At
this point, I would suggest attempting another c-s-c reindex with a
careful eye on the logs of chef-expander, chef-solr, and erchef.
Can you verify that your node data is hitting the expander during
reindex?
Any crash logs in the erchef logs related in time to reindexing?
Errors or logs of chef-solr?
Also worth looking at rabbitmq logs -- although a problem seems unlikely
to be located there since some data types appear to get reindexed
properly and the queue doesn't know anything about the object types.
Anyhow, hope these suggestions help you track down some more info.
--
Seth Falcon | Development Lead | Opscode | @sfalcon
--
Cheers,
Peter Donald