Lightweight Search


#1

Hello,

At a recent SV Chef meetup we discussed a problem we have at Wikia related
to the memory requirements of our chef-client runs.

Many folks shared similar concerns and some folks from Opscode expressed
that they are aware of the problem and that a solution may involve search
calls specifying which attributes they need on returned nodes.

I haven’t heard of any movement on this, and we’re considering writing a
patch which implements an optional paramater for search and returns
abbreviated JSON.

I talked to a couple of folks who suggested dropping a note here to see if
we’re potentially duplicating effort and/or if there is any such effort we
can contribute to.

I’ll update as we make progress, thanks in advance for any input!

Justin



#2

Hi Justin - this is high on our roadmap.

Let’s talk about what the high level interface will look like, and then we can dig in to what a particular implementation would entail. Can you write up what you are thinking about at Wikia?


Opscode, Inc.
Adam Jacob, Chief Product Officer
T: (206) 619-7151 E: adam@opscode.com

On Wednesday, August 17, 2011 at 3:40 PM, Justin Alan Ryan wrote:

Hello,

At a recent SV Chef meetup we discussed a problem we have at Wikia related to the memory requirements of our chef-client runs.

Many folks shared similar concerns and some folks from Opscode expressed that they are aware of the problem and that a solution may involve search calls specifying which attributes they need on returned nodes.

I haven’t heard of any movement on this, and we’re considering writing a patch which implements an optional paramater for search and returns abbreviated JSON.

I talked to a couple of folks who suggested dropping a note here to see if we’re potentially duplicating effort and/or if there is any such effort we can contribute to.

I’ll update as we make progress, thanks in advance for any input!

Justin


http://about.me/justizin


#3

Also, I’ve been playing around a bit with getting the size of the node objects saved to be smaller. The current leading contender is to whitelist the node attributes you actually want saved at the end of the run - this way all the attributes from Ohai + Cookbooks are available during the run, but only the ones you actually care about for search are saved. A working prototype is here:

This should help in the near term - slim down the size of the objects to the parts you care about, and the aggregate size of the results will shrink as well. We should still fix the core issue of only returning full objects on search, but this should help in the interim.


Opscode, Inc.
Adam Jacob, Chief Product Officer
T: (206) 619-7151 E: adam@opscode.com

On Wednesday, August 17, 2011 at 4:53 PM, Adam Jacob wrote:

Hi Justin - this is high on our roadmap.

Let’s talk about what the high level interface will look like, and then we can dig in to what a particular implementation would entail. Can you write up what you are thinking about at Wikia?


Opscode, Inc.
Adam Jacob, Chief Product Officer
T: (206) 619-7151 E: adam@opscode.com (mailto:adam@opscode.com)

On Wednesday, August 17, 2011 at 3:40 PM, Justin Alan Ryan wrote:

Hello,

At a recent SV Chef meetup we discussed a problem we have at Wikia related to the memory requirements of our chef-client runs.

Many folks shared similar concerns and some folks from Opscode expressed that they are aware of the problem and that a solution may involve search calls specifying which attributes they need on returned nodes.

I haven’t heard of any movement on this, and we’re considering writing a patch which implements an optional paramater for search and returns abbreviated JSON.

I talked to a couple of folks who suggested dropping a note here to see if we’re potentially duplicating effort and/or if there is any such effort we can contribute to.

I’ll update as we make progress, thanks in advance for any input!

Justin


http://about.me/justizin


#4

On Wed, Aug 17, 2011 at 5:47 PM, Adam Jacob adam@opscode.com wrote:

Also, I’ve been playing around a bit with getting the size of the node
objects saved to be smaller. The current leading contender is to whitelist
the node attributes you actually want saved at the end of the run - this way
all the attributes from Ohai + Cookbooks are available during the run, but
only the ones you actually care about for search are saved. A working
prototype is here:

https://github.com/opscode/whitelist-node-attrs

This should help in the near term - slim down the size of the objects to
the parts you care about, and the aggregate size of the results will shrink
as well. We should still fix the core issue of only returning full objects
on search, but this should help in the interim.

On Wed, Aug 17, 2011 at 4:53 PM, Adam Jacob adam@opscode.com wrote:

Hi Justin - this is high on our roadmap.

Let’s talk about what the high level interface will look like, and then we
can dig in to what a particular implementation would entail. Can you write
up what you are thinking about at Wikia?

Thanks, Adam…

It wasn’t clear how high this was on your roadmap, if you guys can lend some
expertise, we can invest some time to collaborate on and test a solution. I
launched a fancy new chef server today on SSDs. :wink:

A slightly intertwined issue is that we have some inconsistent search
results based on nodes. It’s been suggested this could have to do with 0.9
/ 0.10 mixture in our client base (we still have quite a few karmic hosts,
all lucid are 0.10), but I can’t tell how. To workaround this, in some
places we have searches which grab all nodes and enumerate them.

Obviously this is horrid. :slight_smile:

Another thing I thought of today is, what if each non-self node object were
kept in a central registry in memory, so that if multiple searches in a chef
run return the same nodes, the same item is references in both cases.

Let’s chat soon, this is a high priority for us and source of much
discussion internally.

Thanks again for your enthusiasm in collaborating!



#5

Got a chance to look closer at the whitelist cookbook today,

This is an interesting solution. If I understand correctly:

(a) It stores less data in couch
(b) lucene has to index less data
© the size of the returned node object is smaller, in addition to being
less trouble to handle on the server

Am I correct?

Solutions we’ve been tossing around include:

  • Patch Chef search view to accept optional arguments and filter at that
    layer, which would also require patching the client. We’d prefer not to
    actually launch this solution without a patch being accepted because we
    don’t want to run a bunch of patched stuff, for maintainability sake.
  • Generate data bags of the data we need on the server via some sort of
    cron-ed knife scripts
  • Generate flat files that we can use via remote_file via some sort of
    cron, or in chef recipes running on the chef server.

The more I think about it, the more I like the idea of a whitelist because
we think there may be problems with the accuracy of our search results due
to failed indexing or something, it seems that if we stored and indexed less
data the chef server could do more with less.

I’m not particularly excited about using whitelist as a cookbook which
absolutely must be the last item, because it breaks our workflow and is
likely to result in lots of relatively unproductive conversations.

Is there any roadmap to have whitelist in 0.10 series as a feature? Are
there any other ideas floating about which could address this concern, e.g.
a way to force items to be sticky at the beginning or end of run_list? We
already have a recipe for measuring memory which has to run last, and
adjusting the run_list on each of those handful of nodes has become:

knife node run_list remove "recipe[chef::mem]"
knife node run_list add "recipe[something-new-and-fancy]"
knife node run_list add “recipe[chef::mem]”

That’s annoying, but manageable for 5 nodes. For 290 nodes to ensure that
everyone with knife access knows to keep “recipe[whitelist-node-attrs]” at
the end or their chef run may fail sounds like potentially trading one
headache for another. ;d

It’s not crucial to fix this in one day, but we need to do something soon.
I’m fully inclined for that to be something which involves contributing to
making chef better for everyone.

On Wed, Aug 17, 2011 at 8:31 PM, Justin Alan Ryan <serial.rockstar@gmail.com

wrote:

On Wed, Aug 17, 2011 at 5:47 PM, Adam Jacob adam@opscode.com wrote:

Also, I’ve been playing around a bit with getting the size of the node
objects saved to be smaller. The current leading contender is to whitelist
the node attributes you actually want saved at the end of the run - this way
all the attributes from Ohai + Cookbooks are available during the run, but
only the ones you actually care about for search are saved. A working
prototype is here:

https://github.com/opscode/whitelist-node-attrs

This should help in the near term - slim down the size of the objects to
the parts you care about, and the aggregate size of the results will shrink
as well. We should still fix the core issue of only returning full objects
on search, but this should help in the interim.


http://about.me/justizin


#6

Yep - you got it. It’s not the final answer (which i think involves partial search results, and a change to the client API) but it will go a long way.

To make the whitelist code as it stands part of the chef server, you can probably add a library that patches Chef::Node#save to execute the same code as the whitelist does, while keeping a copy of the 4 original hashes, so it can restore them post-save (in the case of someone calling node.save explicitly.)

Something like this showing up in core chef seems like a good idea.

Adam


Opscode, Inc.
Adam Jacob, Chief Product Officer
T: (206) 619-7151 E: adam@opscode.com

On Thursday, August 18, 2011 at 2:25 PM, Justin Alan Ryan wrote:

Got a chance to look closer at the whitelist cookbook today,

This is an interesting solution. If I understand correctly:

(a) It stores less data in couch
(b) lucene has to index less data
© the size of the returned node object is smaller, in addition to being less trouble to handle on the server

Am I correct?

Solutions we’ve been tossing around include:

  • Patch Chef search view to accept optional arguments and filter at that layer, which would also require patching the client. We’d prefer not to actually launch this solution without a patch being accepted because we don’t want to run a bunch of patched stuff, for maintainability sake.
  • Generate data bags of the data we need on the server via some sort of cron-ed knife scripts
  • Generate flat files that we can use via remote_file via some sort of cron, or in chef recipes running on the chef server.

The more I think about it, the more I like the idea of a whitelist because we think there may be problems with the accuracy of our search results due to failed indexing or something, it seems that if we stored and indexed less data the chef server could do more with less.

I’m not particularly excited about using whitelist as a cookbook which absolutely must be the last item, because it breaks our workflow and is likely to result in lots of relatively unproductive conversations.

Is there any roadmap to have whitelist in 0.10 series as a feature? Are there any other ideas floating about which could address this concern, e.g. a way to force items to be sticky at the beginning or end of run_list? We already have a recipe for measuring memory which has to run last, and adjusting the run_list on each of those handful of nodes has become:

knife node run_list remove "recipe[chef::mem]"
knife node run_list add "recipe[something-new-and-fancy]"
knife node run_list add “recipe[chef::mem]”

That’s annoying, but manageable for 5 nodes. For 290 nodes to ensure that everyone with knife access knows to keep “recipe[whitelist-node-attrs]” at the end or their chef run may fail sounds like potentially trading one headache for another. ;d

It’s not crucial to fix this in one day, but we need to do something soon. I’m fully inclined for that to be something which involves contributing to making chef better for everyone.

On Wed, Aug 17, 2011 at 8:31 PM, Justin Alan Ryan <serial.rockstar@gmail.com (mailto:serial.rockstar@gmail.com)> wrote:

On Wed, Aug 17, 2011 at 5:47 PM, Adam Jacob <adam@opscode.com (mailto:adam@opscode.com)> wrote:

Also, I’ve been playing around a bit with getting the size of the node objects saved to be smaller. The current leading contender is to whitelist the node attributes you actually want saved at the end of the run - this way all the attributes from Ohai + Cookbooks are available during the run, but only the ones you actually care about for search are saved. A working prototype is here:

https://github.com/opscode/whitelist-node-attrs

This should help in the near term - slim down the size of the objects to the parts you care about, and the aggregate size of the results will shrink as well. We should still fix the core issue of only returning full objects on search, but this should help in the interim.

http://about.me/justizin