Caching Data for the Duration of a Single Chef Run


#1

I’ve got a question about caching data for the duration of the Chef run.
Let’s assume I have an LWRP that fetches data from a remote API and that API
is relatively slow. In my particular case that data doesn’t change very often,
and in the very off chance it changed mid-run, having stale data would be OK.

So my goal is to have the LWRP fetch the information one time, and cache it for
the duration of the run. That way multiple calls to the LWRP won’t result in
many hits back to the API.

My initial thought was to simply use Chef::Config[:file_cache_path], in
combination with run_status.start_time so that if the cached file was older
than the start time refresh the data. That got me to thinking why not just
write the data directly to a sub key of run_status, and avoid all the logic to
compare time stamps and such. Which was then followed by the question: "I
wonder how everyone else is doing this?

So the question is essentially, what do folks recommend as the best approach to
caching data for the duration of the run. Ideally I want to be sure I do
something that is both 10x and 11x compatible.


#2

Use the node’s run_state to hold your temporal data.

Cheers,

AJ

On 4 June 2013 09:52, davidpetzel@gmail.com wrote:

I’ve got a question about caching data for the duration of the Chef run.
Let’s assume I have an LWRP that fetches data from a remote API and that API
is relatively slow. In my particular case that data doesn’t change very often,
and in the very off chance it changed mid-run, having stale data would be OK.

So my goal is to have the LWRP fetch the information one time, and cache it for
the duration of the run. That way multiple calls to the LWRP won’t result in
many hits back to the API.

My initial thought was to simply use Chef::Config[:file_cache_path], in
combination with run_status.start_time so that if the cached file was older
than the start time refresh the data. That got me to thinking why not just
write the data directly to a sub key of run_status, and avoid all the logic to
compare time stamps and such. Which was then followed by the question: "I
wonder how everyone else is doing this?

So the question is essentially, what do folks recommend as the best approach to
caching data for the duration of the run. Ideally I want to be sure I do
something that is both 10x and 11x compatible.


#3

I’m just using node.run_state for this - data goes into the cookbook-specific keys of that hash. If the data is small enough to be held in memory, I think that’s a perfect solution.

Regards,
Stanisław Pitucha
Cloud Services
Hewlett Packard

-----Original Message-----
From: davidpetzel@gmail.com [mailto:davidpetzel@gmail.com]
Sent: Monday, June 03, 2013 10:53 PM
To: chef@lists.opscode.com
Subject: [chef] Caching Data for the Duration of a Single Chef Run

I’ve got a question about caching data for the duration of the Chef run.
Let’s assume I have an LWRP that fetches data from a remote API and that API is relatively slow. In my particular case that data doesn’t change very often, and in the very off chance it changed mid-run, having stale data would be OK.

So my goal is to have the LWRP fetch the information one time, and cache it for the duration of the run. That way multiple calls to the LWRP won’t result in many hits back to the API.

My initial thought was to simply use Chef::Config[:file_cache_path], in combination with run_status.start_time so that if the cached file was older than the start time refresh the data. That got me to thinking why not just write the data directly to a sub key of run_status, and avoid all the logic to compare time stamps and such. Which was then followed by the question: "I wonder how everyone else is doing this?

So the question is essentially, what do folks recommend as the best approach to caching data for the duration of the run. Ideally I want to be sure I do something that is both 10x and 11x compatible.


#4

AJ and Stanisław, the run_state approach was very clean and easy to
leverage, thanks for the info.

On Tue, Jun 4, 2013 at 7:00 AM, Pitucha, Stanislaw Izaak <
stanislaw.pitucha@hp.com> wrote:

I’m just using node.run_state for this - data goes into the
cookbook-specific keys of that hash. If the data is small enough to be held
in memory, I think that’s a perfect solution.

Regards,
Stanisław Pitucha
Cloud Services
Hewlett Packard

-----Original Message-----
From: davidpetzel@gmail.com [mailto:davidpetzel@gmail.com]
Sent: Monday, June 03, 2013 10:53 PM
To: chef@lists.opscode.com
Subject: [chef] Caching Data for the Duration of a Single Chef Run

I’ve got a question about caching data for the duration of the Chef run.
Let’s assume I have an LWRP that fetches data from a remote API and that
API is relatively slow. In my particular case that data doesn’t change very
often, and in the very off chance it changed mid-run, having stale data
would be OK.

So my goal is to have the LWRP fetch the information one time, and cache
it for the duration of the run. That way multiple calls to the LWRP won’t
result in many hits back to the API.

My initial thought was to simply use Chef::Config[:file_cache_path], in
combination with run_status.start_time so that if the cached file was older
than the start time refresh the data. That got me to thinking why not just
write the data directly to a sub key of run_status, and avoid all the logic
to compare time stamps and such. Which was then followed by the question:
"I wonder how everyone else is doing this?

So the question is essentially, what do folks recommend as the best
approach to caching data for the duration of the run. Ideally I want to be
sure I do something that is both 10x and 11x compatible.


#5

At one point, I had floated the idea of an alternate search functionality
that would be cached for the life of the run. Something like nodes =
cached_search(:node, “blah:blah”). Wonder if that’s worth discussing again.

On Tue, Jun 4, 2013 at 7:00 AM, Pitucha, Stanislaw Izaak <
stanislaw.pitucha@hp.com> wrote:

I’m just using node.run_state for this - data goes into the
cookbook-specific keys of that hash. If the data is small enough to be held
in memory, I think that’s a perfect solution.

Regards,
Stanisław Pitucha
Cloud Services
Hewlett Packard

-----Original Message-----
From: davidpetzel@gmail.com [mailto:davidpetzel@gmail.com]
Sent: Monday, June 03, 2013 10:53 PM
To: chef@lists.opscode.com
Subject: [chef] Caching Data for the Duration of a Single Chef Run

I’ve got a question about caching data for the duration of the Chef run.
Let’s assume I have an LWRP that fetches data from a remote API and that
API is relatively slow. In my particular case that data doesn’t change very
often, and in the very off chance it changed mid-run, having stale data
would be OK.

So my goal is to have the LWRP fetch the information one time, and cache
it for the duration of the run. That way multiple calls to the LWRP won’t
result in many hits back to the API.

My initial thought was to simply use Chef::Config[:file_cache_path], in
combination with run_status.start_time so that if the cached file was older
than the start time refresh the data. That got me to thinking why not just
write the data directly to a sub key of run_status, and avoid all the logic
to compare time stamps and such. Which was then followed by the question:
"I wonder how everyone else is doing this?

So the question is essentially, what do folks recommend as the best
approach to caching data for the duration of the run. Ideally I want to be
sure I do something that is both 10x and 11x compatible.


#6

Wrote this exact thing bout 2 years ago for some internal cookbooks to
cache search results they were using all over the place (looking up the
local repo server and other things). Worked out very nice.

On Tue, Jun 4, 2013 at 6:20 AM, John E. Vincent (lusis) <
lusis.org+chef-list@gmail.com> wrote:

At one point, I had floated the idea of an alternate search functionality
that would be cached for the life of the run. Something like nodes =
cached_search(:node, “blah:blah”). Wonder if that’s worth discussing again.

On Tue, Jun 4, 2013 at 7:00 AM, Pitucha, Stanislaw Izaak <
stanislaw.pitucha@hp.com> wrote:

I’m just using node.run_state for this - data goes into the
cookbook-specific keys of that hash. If the data is small enough to be held
in memory, I think that’s a perfect solution.

Regards,
Stanisław Pitucha
Cloud Services
Hewlett Packard

-----Original Message-----
From: davidpetzel@gmail.com [mailto:davidpetzel@gmail.com]
Sent: Monday, June 03, 2013 10:53 PM
To: chef@lists.opscode.com
Subject: [chef] Caching Data for the Duration of a Single Chef Run

I’ve got a question about caching data for the duration of the Chef run.
Let’s assume I have an LWRP that fetches data from a remote API and that
API is relatively slow. In my particular case that data doesn’t change very
often, and in the very off chance it changed mid-run, having stale data
would be OK.

So my goal is to have the LWRP fetch the information one time, and cache
it for the duration of the run. That way multiple calls to the LWRP won’t
result in many hits back to the API.

My initial thought was to simply use Chef::Config[:file_cache_path], in
combination with run_status.start_time so that if the cached file was older
than the start time refresh the data. That got me to thinking why not just
write the data directly to a sub key of run_status, and avoid all the logic
to compare time stamps and such. Which was then followed by the question:
"I wonder how everyone else is doing this?

So the question is essentially, what do folks recommend as the best
approach to caching data for the duration of the run. Ideally I want to be
sure I do something that is both 10x and 11x compatible.