# Latency on search indicies

**URL:** <https://discourse.chef.io/t/latency-on-search-indicies/4130>\
**Category:** Chef Infra (archive)\
**Created:** [June 24, 2013, 10:52am UTC](https://discourse.chef.io/t/latency-on-search-indicies/4130 "2013-06-24T10:52:08Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jeppe\_Nejsum\_Madsen](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/jeppe_nejsum_madsen/32/258_2.png) [@Jeppe\_Nejsum\_Madsen](https://discourse.chef.io/u/Jeppe_Nejsum_Madsen)\
**Post date:** [June 24, 2013, 10:52am UTC](https://discourse.chef.io/t/latency-on-search-indicies/4130/1 "2013-06-24T10:52:08Z")

</div>

Ohai,

What’s the expected latency between executing a command until the search  
index is updated (we’re using hosted chef)?

I’ve seen several times now that it can take 15-20 seconds from moving a  
node from one environment to another before it’s returned by search. Is  
this normal?

/Jeppe

---

<div class="post-metadata">

**Author:** ![Pete\_Cheslock](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/pete_cheslock/32/326_2.png) [@Pete\_Cheslock](https://discourse.chef.io/u/Pete_Cheslock)\
**Post date:** [June 24, 2013, 12:21pm UTC](https://discourse.chef.io/t/latency-on-search-indicies/4130/2 "2013-06-24T12:21:33Z")

</div>

That is probably right as solr needs to reindex the data. I've seen  
similar when using hosted chef.

-Pete

On Mon, Jun 24, 2013 at 3:52 AM, Jeppe Nejsum Madsen [jeppe@ingolfs.dk](mailto:jeppe@ingolfs.dk)wrote:

> Ohai,
> 
> What's the expected latency between executing a command until the search  
> index is updated (we're using hosted chef)?
> 
> I've seen several times now that it can take 15-20 seconds from moving a  
> node from one environment to another before it's returned by search. Is  
> this normal?
> 
> /Jeppe

---

<div class="post-metadata">

**Author:** ![realityforge](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/realityforge/32/395_2.png) [@realityforge](https://discourse.chef.io/u/realityforge)\
**Post date:** [June 24, 2013, 9:20pm UTC](https://discourse.chef.io/t/latency-on-search-indicies/4130/3 "2013-06-24T21:20:57Z")

</div>

Hi,

Just be warned that the lag can be much more significant. We run the  
open source variant of chef for now and have noticed lags of anywhere  
between \<1s to ~120s between the data being inserted and it being  
accessible via search. In most cases we rely on eventual consistency  
to catch up but in the cases of directed changes - particularly in our  
CD pipeline we insert data bag items and then poll until available  
before progressing.

We use search extensively and have a large number of windows nodes  
which seems to place pressure on the indexing system. To reduce the  
impact we have started to strip out lots of windows ohai data and use  
partial search cookbook where possible. That combined wiht a bit more  
memory for the chef box has reduced the lag a little.

Not sure if that helps ...

On Mon, Jun 24, 2013 at 8:52 PM, Jeppe Nejsum Madsen [jeppe@ingolfs.dk](mailto:jeppe@ingolfs.dk) wrote:

> Ohai,
> 
> What's the expected latency between executing a command until the search  
> index is updated (we're using hosted chef)?
> 
> I've seen several times now that it can take 15-20 seconds from moving a  
> node from one environment to another before it's returned by search. Is this  
> normal?
> 
> /Jeppe

--  
Cheers,

Peter Donald

---

<div class="post-metadata">

**Author:** ![BradKnowles](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/bradknowles/32/99_2.png) [@BradKnowles](https://discourse.chef.io/u/BradKnowles)\
**Post date:** [June 25, 2013, 12:46am UTC](https://discourse.chef.io/t/latency-on-search-indicies/4130/4 "2013-06-25T00:46:21Z")

</div>

On Jun 24, 2013, at 3:20 PM, Peter Donald [peter@realityforge.org](mailto:peter@realityforge.org) wrote:

> Just be warned that the lag can be much more significant. We run the  
> open source variant of chef for now and have noticed lags of anywhere  
> between \<1s to ~120s between the data being inserted and it being  
> accessible via search.

What version of chef-server are you running? Is it Chef 11.x, 10.x, or something older?

> We use search extensively and have a large number of windows nodes  
> which seems to place pressure on the indexing system.

Windows nodes put a heavier load on the Chef server, but I would think that chef-server 11.x should be a lot more capable of handling large numbers of clients (even large numbers of Windows nodes) much better than the older 10.x-based versions.

After all, my understanding is that Hosted Chef is basically the world's largest instance of Private Chef 11.x set up in a multi-tiered structure, and I believe that Private Chef has been proven by partners like Facebook, Etsy, Netflix, etc... to scale to at least tens of thousands of nodes on a single Private Chef 11.x cluster.

> To reduce the  
> impact we have started to strip out lots of windows ohai data and use  
> partial search cookbook where possible. That combined wiht a bit more  
> memory for the chef box has reduced the lag a little.

I've always wondered why ohai generates such massive amounts of information per node (regardless of platform), and that all of this information is usually considered "important enough" that all of it should be saved and indexed after every single run. Windows nodes might be worse in this respect, but the problem isn't all that much better on most \*nix nodes.

It seems to me that Ohai data that is going to be saved should be minimized to start with, and then if there are extra bits of information you want/need to be available via search then you should be able to handle those appropriately. Or, at the very least, maybe give us levels of index priority, and some Ohai data should be considered "high priority" and available via search at very low latency, but 90-99% of the rest of the Ohai data should be considered "low priority".

--  
Brad Knowles [brad@shub-internet.org](mailto:brad@shub-internet.org)  
LinkedIn Profile: [http://tinyurl.com/y8kpxu](http://tinyurl.com/y8kpxu)

---

<div class="post-metadata">

**Author:** ![realityforge](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/realityforge/32/395_2.png) [@realityforge](https://discourse.chef.io/u/realityforge)\
**Post date:** [June 25, 2013, 1:42am UTC](https://discourse.chef.io/t/latency-on-search-indicies/4130/5 "2013-06-25T01:42:18Z")

</div>

Hi,

On Tue, Jun 25, 2013 at 10:46 AM, Brad Knowles [brad@shub-internet.org](mailto:brad@shub-internet.org) wrote:

> > Just be warned that the lag can be much more significant. We run the  
> > open source variant of chef for now and have noticed lags of anywhere  
> > between \<1s to ~120s between the data being inserted and it being  
> > accessible via search.
> 
> What version of chef-server are you running? Is it Chef 11.x, 10.x, or something older?

We actually use Chef 11 but to be honest most of the strategies we  
used to address this were developed when we were on 10.x. There is  
still a delay between when a data bag item is inserted/updates and  
when it appears in the search results so I _assume_ the lag is still  
present for node data but have not tested.

> Windows nodes put a heavier load on the Chef server, but I would think that chef-server 11.x should be a lot more capable of handling large numbers of clients (even large numbers of Windows nodes) much better than the older 10.x-based versions.
> 
> After all, my understanding is that Hosted Chef is basically the world's largest instance of Private Chef 11.x set up in a multi-tiered structure, and I believe that Private Chef has been proven by partners like Facebook, Etsy, Netflix, etc... to scale to at least tens of thousands of nodes on a single Private Chef 11.x cluster.

To be fair the chef 11 server does not seem to be particularly loaded  
even when there is lag.

Also most of those other sites can probably afford to rely on eventual  
consistency. However it would be nice to have the parameter to the  
rest endpoints to block until indexing of the changed data is  
complete.

--  
Cheers,

Peter Donald

---

<div class="post-metadata">

**Author:** ![BradKnowles](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/bradknowles/32/99_2.png) [@BradKnowles](https://discourse.chef.io/u/BradKnowles)\
**Post date:** [June 25, 2013, 2:30am UTC](https://discourse.chef.io/t/latency-on-search-indicies/4130/6 "2013-06-25T02:30:00Z")

</div>

On Jun 24, 2013, at 7:42 PM, Peter Donald [peter@realityforge.org](mailto:peter@realityforge.org) wrote:

> We actually use Chef 11 but to be honest most of the strategies we  
> used to address this were developed when we were on 10.x. There is  
> still a delay between when a data bag item is inserted/updates and  
> when it appears in the search results so I _assume_ the lag is still  
> present for node data but have not tested.

Ahh, data bags. I always thought that they were kind of a hack, and maybe not as well implemented as a lot of other parts of Chef.

I would be very interested to find out if you have the same latency with node data as you do with data bags.

> Also most of those other sites can probably afford to rely on eventual  
> consistency. However it would be nice to have the parameter to the  
> rest endpoints to block until indexing of the changed data is  
> complete.

Indeed, that sounds like a knob that would be very nice to be able to tweak.

--  
Brad Knowles [brad@shub-internet.org](mailto:brad@shub-internet.org)  
LinkedIn Profile: [http://tinyurl.com/y8kpxu](http://tinyurl.com/y8kpxu)

---

<div class="post-metadata">

**Author:** ![Jeppe\_Nejsum\_Madsen](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/jeppe_nejsum_madsen/32/258_2.png) [@Jeppe\_Nejsum\_Madsen](https://discourse.chef.io/u/Jeppe_Nejsum_Madsen)\
**Post date:** [June 25, 2013, 8:00am UTC](https://discourse.chef.io/t/latency-on-search-indicies/4130/7 "2013-06-25T08:00:48Z")

</div>

Peter Donald [peter@realityforge.org](mailto:peter@realityforge.org) writes:

> Hi,
> 
> On Tue, Jun 25, 2013 at 10:46 AM, Brad Knowles [brad@shub-internet.org](mailto:brad@shub-internet.org) wrote:
> 
> > > Just be warned that the lag can be much more significant. We run the  
> > > open source variant of chef for now and have noticed lags of anywhere  
> > > between \<1s to ~120s between the data being inserted and it being  
> > > accessible via search.
> > 
> > What version of chef-server are you running? Is it Chef 11.x, 10.x, or something older?
> 
> We actually use Chef 11 but to be honest most of the strategies we  
> used to address this were developed when we were on 10.x.

We're using windows as well. Can you outline what strategies you used?

[...]

> Also most of those other sites can probably afford to rely on eventual  
> consistency.

Probably depends on the workflow....

When we add nodes to e.g. an environment and kick of chef-client, it's  
unfortunate that only some of the nodes get hit.

> However it would be nice to have the parameter to the rest endpoints  
> to block until indexing of the changed data is complete.

Indeed. Otherwise we have to implement this explicitly as you mentioned  
by waiting to make sure indices are updated.

/Jeppe

---

<div class="post-metadata">

**Author:** ![realityforge](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/realityforge/32/395_2.png) [@realityforge](https://discourse.chef.io/u/realityforge)\
**Post date:** [June 25, 2013, 11:45pm UTC](https://discourse.chef.io/t/latency-on-search-indicies/4130/8 "2013-06-25T23:45:27Z")

</div>

On Tue, Jun 25, 2013 at 6:00 PM, Jeppe Nejsum Madsen [jeppe@ingolfs.dk](mailto:jeppe@ingolfs.dk) wrote:

> > > > Just be warned that the lag can be much more significant. We run the  
> > > > open source variant of chef for now and have noticed lags of anywhere  
> > > > between \<1s to ~120s between the data being inserted and it being  
> > > > accessible via search.
> > > 
> > > What version of chef-server are you running? Is it Chef 11.x, 10.x, or something older?
> > 
> > We actually use Chef 11 but to be honest most of the strategies we  
> > used to address this were developed when we were on 10.x.
> 
> We're using windows as well. Can you outline what strategies you used?

Mostly it involved reducing load generated via search, reduce the  
amount of data that went into the search indexes and support explicit  
wait until constructs to wait until search is updated.

In no particular order we did the following;

- Remove all the ohai plugins we weren't using a recipe (See  
[https://gist.github.com/realityforge/5863439](https://gist.github.com/realityforge/5863439))
- Strip out unnecessary ohai data at the end of a run. Conceptually  
similar to [GitHub - chef-boneyard/whitelist-node-attrs-cookbook: DEPRECATED: Development repository for whitelist-node-attrs cookbook](https://github.com/opscode-cookbooks/whitelist-node-attrs)  
except we used a blacklist instead
- Use partial\_search rather than regular search where possible (See  
[GitHub - chef-boneyard/partial\_search\_cookbook: DEPRECATED: Provides a library using the Chef partial search API](https://github.com/opscode-cookbooks/partial_search))
- Avoid broad searching of the node index where possible. Require  
nodes to explicitly publish data that they expect other nodes to use  
in a manner friendly to partial search. i.e. See  
[chef-cutlery/libraries/services.rb at master · realityforge/chef-cutlery · GitHub](https://github.com/realityforge/chef-cutlery/blob/master/libraries/services.rb#L24)
- Add an abstraction layer so that data can be stored either against a  
node or in a data bag item. Sometimes we moved things from a node to a  
databag. See [chef-cutlery/libraries/services.rb at master · realityforge/chef-cutlery · GitHub](https://github.com/realityforge/chef-cutlery/blob/master/libraries/services.rb#L53)
- Our orchestration layer (bunch of ruby libraries using spice that  
are driven by jenkins) has built in primitives to wait until the  
search index has been updated.

We probably did a few other things as well. But most significant  
change was the introduction of the last strategy. We now are driving  
more and more of our config via data bags to try and limit the idea of  
"exported" resources interacting across nodes.

HTH,

Peter Donald

---

<div class="post-metadata">

**Author:** ![kallistec](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/kallistec/32/23_2.png) [@kallistec](https://discourse.chef.io/u/kallistec)\
**Post date:** [June 26, 2013, 4:30pm UTC](https://discourse.chef.io/t/latency-on-search-indicies/4130/9 "2013-06-26T16:30:12Z")

</div>

On Tuesday, June 25, 2013 at 4:45 PM, Peter Donald wrote:

> On Tue, Jun 25, 2013 at 6:00 PM, Jeppe Nejsum Madsen \<jeppe@ingolfs.dk ([mailto:jeppe@ingolfs.dk](mailto:jeppe@ingolfs.dk))\> wrote:
> 
> > > > > Just be warned that the lag can be much more significant. We run the  
> > > > > open source variant of chef for now and have noticed lags of anywhere  
> > > > > between \<1s to ~120s between the data being inserted and it being  
> > > > > accessible via search.

Search is not immediately consistent. The exact latency you experience will depend on the commit frequency configured for Solr.

Newer versions of Solr have a soft commit feature which allows for much more frequent commits without adding disk IO load.Upgrading to Solr 4 is something that's on our radar, but not being actively worked on at the moment.

--  
Daniel DeLeo