At some point several months back, I had a discussion with some folks.
Maybe it was at ChefConf. Regardless, we discussed the idea of a
"cached search". Opscoded added (or maybe it hasn’t made it in yet),
the idea of partial search. My suggestion/idea was that the idea of a
cached search be added that was computed once at the beginning of the
run and saved globally for the whole run. The thought was that we
frequently perform the exact same search across different recipes. For
instance in ours we have this at the top of almost every recipe:
distributor = search(:node, “role:distributor AND
chef_environment:#{node.chef_environment}”).first
That’s how we locate our current internal blob distribution point
(nginx serving a bunch of files for Chef in this case).
It would be pretty awesome if we only had to compute that once. I’ve
not tinkered with any specific implementation yet. Does the idea
interest anyone else? I was thinking either an additional parameter to
the existing search (’:cached => true’) or possibly a new LWRP (in the
same way I did encrypted databags before official support was there).
I’ve never looked at the guts of search but I’m pretty sure there are
some possible edge cases where search results would change mid-run
between recipes that could cause some nasty issue to crop up.
Thoughts?
John E. Vincent
@lusis
On Sun, Sep 30, 2012 at 12:23 PM, John E. Vincent (lusis)
lusis.org+chef-list@gmail.com wrote:
run and saved globally for the whole run. The thought was that we
frequently perform the exact same search across different recipes. For
instance in ours we have this at the top of almost every recipe:
It's an interesting idea and there's probably a use case for it. For
your case though, why not just do the search in your base role early
in the run_list and stick the results somewhere other cookbooks can
access it so they don't have to do search at all? That element of your
base role effectively defines an API that other cookbooks you use
internally can use to get that data. You could just stick it in the
node object but you'd probably want to remove it before the node.save
just to ensure that you never accidentally use stale info (unless
that's ok).
KC
There’s been some thought put into a related feature.
Rather than having me speculate here, let’s take this as something that we should all discuss at the upcoming summit.
I totally get the value in caching these results.
-C
From: lusis.org@gmail.com [lusis.org@gmail.com] on behalf of John E. Vincent (lusis) [lusis.org+chef-list@gmail.com]
Sent: Sunday, September 30, 2012 12:23 PM
To: chef@lists.opscode.com
Subject: [chef] Revisting an old idea
At some point several months back, I had a discussion with some folks.
Maybe it was at ChefConf. Regardless, we discussed the idea of a
"cached search". Opscoded added (or maybe it hasn’t made it in yet),
the idea of partial search. My suggestion/idea was that the idea of a
cached search be added that was computed once at the beginning of the
run and saved globally for the whole run. The thought was that we
frequently perform the exact same search across different recipes. For
instance in ours we have this at the top of almost every recipe:
distributor = search(:node, “role:distributor AND
chef_environment:#{node.chef_environment}”).first
That’s how we locate our current internal blob distribution point
(nginx serving a bunch of files for Chef in this case).
It would be pretty awesome if we only had to compute that once. I’ve
not tinkered with any specific implementation yet. Does the idea
interest anyone else? I was thinking either an additional parameter to
the existing search (’:cached => true’) or possibly a new LWRP (in the
same way I did encrypted databags before official support was there).
I’ve never looked at the guts of search but I’m pretty sure there are
some possible edge cases where search results would change mid-run
between recipes that could cause some nasty issue to crop up.
Thoughts?
John E. Vincent
@lusis
On Sep 30, 2012, at 3:23 PM, John E. Vincent (lusis) lusis.org+chef-list@gmail.com wrote:
or possibly a new LWRP (in the
same way I did encrypted databags before official support was there).
We already us a "wrapped" search 90% of the time, so it'd be trivial for us to do this. Do most folks still use search() directly? I've never been above monkey patching…
--Brian
I wrote this internally for our cookbooks in a library cached_search
using moneta mem cache for these exact reasons. I support a :cache
option or new method for caching a search.
On Sun, Sep 30, 2012 at 3:11 PM, Brian Akins brian@akins.org wrote:
On Sep 30, 2012, at 3:23 PM, John E. Vincent (lusis) lusis.org+chef-list@gmail.com wrote:
or possibly a new LWRP (in the
same way I did encrypted databags before official support was there).
We already us a "wrapped" search 90% of the time, so it'd be trivial for us to do this. Do most folks still use search() directly? I've never been above monkey patching…
--Brian