Delayed evaluation?


#1

This seems like such a simple problem and something that everyone should
come across at one point or another.

Here’s a short snippet of code that illustrates the issue I’m having:

(1…2).each do |count|
ruby_block “ls /tmp” do
block do
node.set[‘test1’][‘lsdir’] = %x[ls /tmp]
end
end

file “/tmp/file#{count}.txt” do
content node[‘test1’][‘lsdir’]
end
end

So, at the end of the run, both /tmp/file1.txt and /tmp/file2.txt are
empty. I had hoped for /tmp/file2.txt to show that '/tmp/file1.txt now
exists.

I know this is because the value of attributes are evaluated during the
compile phase at the beginning of the chef run and at that time, the value
of node[‘test1’][‘lsdir’] is nil. Also, this same issue arises when trying
to use a variable instead of an attribute.

How can I get something like this to execute as I would have hoped? This
has to be a common pattern; set an attribute, use it, modify the attribute,
use the new value. I know there is the new ‘lazy’ delayed evaluator, but I
can only get that to work when used in a value for a resource attribute.

Thanks


John Alberts


#2

value of content is getting evaluated at compile time. its an attributed
value and determined soon as the resource is created. while you are setting
the node attribute via a ruby block, which gets evaluated at the compile
time. some of the options. Im not sure if both assigning, mutating and then
using the attribute is common pattern (or a good pattern at all).

i think if you move the file resource inside a lwrp and then notify that
resource, that might work. because in that case the file resource will be
created inside the provider’s action block (which is evaluated only during
the execution phase), hence u’ll have the node.foo.bar value as you
expected. there are several other ways to do the same trick, but this is
the cleanest i can think of …
cheers
ranjib

On Mon, Dec 16, 2013 at 9:18 PM, John Alberts john.m.alberts@gmail.comwrote:

This seems like such a simple problem and something that everyone should
come across at one point or another.

Here’s a short snippet of code that illustrates the issue I’m having:

(1…2).each do |count|
ruby_block “ls /tmp” do
block do
node.set[‘test1’][‘lsdir’] = %x[ls /tmp]
end
end

file “/tmp/file#{count}.txt” do
content node[‘test1’][‘lsdir’]
end
end

So, at the end of the run, both /tmp/file1.txt and /tmp/file2.txt are
empty. I had hoped for /tmp/file2.txt to show that '/tmp/file1.txt now
exists.

I know this is because the value of attributes are evaluated during the
compile phase at the beginning of the chef run and at that time, the value
of node[‘test1’][‘lsdir’] is nil. Also, this same issue arises when trying
to use a variable instead of an attribute.

How can I get something like this to execute as I would have hoped? This
has to be a common pattern; set an attribute, use it, modify the attribute,
use the new value. I know there is the new ‘lazy’ delayed evaluator, but I
can only get that to work when used in a value for a resource attribute.

Thanks


John Alberts


#3

The more generic solution is to use resources/LWRPs such that the second resource isn’t compiled until after the settings you care about. For example https://github.com/balanced-cookbooks/ci/blob/master/libraries/ci_server.rb#L45-L46

–Noah

On Dec 16, 2013, at 9:18 PM, John Alberts john.m.alberts@gmail.com wrote:

This seems like such a simple problem and something that everyone should come across at one point or another.

Here’s a short snippet of code that illustrates the issue I’m having:

(1…2).each do |count|
ruby_block “ls /tmp” do
block do
node.set[‘test1’][‘lsdir’] = %x[ls /tmp]
end
end

file “/tmp/file#{count}.txt” do
content node[‘test1’][‘lsdir’]
end
end

So, at the end of the run, both /tmp/file1.txt and /tmp/file2.txt are empty. I had hoped for /tmp/file2.txt to show that '/tmp/file1.txt now exists.

I know this is because the value of attributes are evaluated during the compile phase at the beginning of the chef run and at that time, the value of node[‘test1’][‘lsdir’] is nil. Also, this same issue arises when trying to use a variable instead of an attribute.

How can I get something like this to execute as I would have hoped? This has to be a common pattern; set an attribute, use it, modify the attribute, use the new value. I know there is the new ‘lazy’ delayed evaluator, but I can only get that to work when used in a value for a resource attribute.

Thanks


John Alberts


#4

Another possible caveat:

Shouldn’t it be ‘ruby_block “ls /tmp #{count}” do’ ?

-Torben
On Dec 17, 2013 6:39 AM, “Noah Kantrowitz” noah@coderanger.net wrote:

The more generic solution is to use resources/LWRPs such that the second
resource isn’t compiled until after the settings you care about. For
example
https://github.com/balanced-cookbooks/ci/blob/master/libraries/ci_server.rb#L45-L46

–Noah

On Dec 16, 2013, at 9:18 PM, John Alberts john.m.alberts@gmail.com
wrote:

This seems like such a simple problem and something that everyone should
come across at one point or another.

Here’s a short snippet of code that illustrates the issue I’m having:

(1…2).each do |count|
ruby_block “ls /tmp” do
block do
node.set[‘test1’][‘lsdir’] = %x[ls /tmp]
end
end

file “/tmp/file#{count}.txt” do
content node[‘test1’][‘lsdir’]
end
end

So, at the end of the run, both /tmp/file1.txt and /tmp/file2.txt are
empty. I had hoped for /tmp/file2.txt to show that '/tmp/file1.txt now
exists.

I know this is because the value of attributes are evaluated during the
compile phase at the beginning of the chef run and at that time, the value
of node[‘test1’][‘lsdir’] is nil. Also, this same issue arises when trying
to use a variable instead of an attribute.

How can I get something like this to execute as I would have hoped?
This has to be a common pattern; set an attribute, use it, modify the
attribute, use the new value. I know there is the new ‘lazy’ delayed
evaluator, but I can only get that to work when used in a value for a
resource attribute.

Thanks


John Alberts


#5

Thanks everyone for the responses. So… it seems the consensus is that
this is best done in a LWRP. I had considered doing that, it just seemed
like overkill for what should be a simple operation; that being, change the
value of a variable(or attribute) and use that variable again later in the
code.

@Torben, Yes, I would probably get warnings for reusing a resource if I
didn’t give it a unique name in the loop, but that would be the extent of
the impact.

John

On Tue, Dec 17, 2013 at 12:21 AM, Torben Knerr ukio@gmx.de wrote:

Another possible caveat:

Shouldn’t it be ‘ruby_block “ls /tmp #{count}” do’ ?

-Torben
On Dec 17, 2013 6:39 AM, “Noah Kantrowitz” noah@coderanger.net wrote:

The more generic solution is to use resources/LWRPs such that the second
resource isn’t compiled until after the settings you care about. For
example
https://github.com/balanced-cookbooks/ci/blob/master/libraries/ci_server.rb#L45-L46

–Noah

On Dec 16, 2013, at 9:18 PM, John Alberts john.m.alberts@gmail.com
wrote:

This seems like such a simple problem and something that everyone
should come across at one point or another.

Here’s a short snippet of code that illustrates the issue I’m having:

(1…2).each do |count|
ruby_block “ls /tmp” do
block do
node.set[‘test1’][‘lsdir’] = %x[ls /tmp]
end
end

file “/tmp/file#{count}.txt” do
content node[‘test1’][‘lsdir’]
end
end

So, at the end of the run, both /tmp/file1.txt and /tmp/file2.txt are
empty. I had hoped for /tmp/file2.txt to show that '/tmp/file1.txt now
exists.

I know this is because the value of attributes are evaluated during the
compile phase at the beginning of the chef run and at that time, the value
of node[‘test1’][‘lsdir’] is nil. Also, this same issue arises when trying
to use a variable instead of an attribute.

How can I get something like this to execute as I would have hoped?
This has to be a common pattern; set an attribute, use it, modify the
attribute, use the new value. I know there is the new ‘lazy’ delayed
evaluator, but I can only get that to work when used in a value for a
resource attribute.

Thanks


John Alberts


John Alberts


#6

On Dec 17, 2013, at 10:50 AM, John Alberts <john.m.alberts@gmail.commailto:john.m.alberts@gmail.com> wrote:

Thanks everyone for the responses. So… it seems the consensus is that this is best done in a LWRP. I had considered doing that, it just seemed like overkill for what should be a simple operation; that being, change the value of a variable(or attribute) and use that variable again later in the code.

Slightly off-topic, but where are you modifying It may be a style or philosophy thing, but I personally wouldn’t try to modify an attribute once I declare it. I rather that I set my conditions up front in attributes file and maybe override them as necessary within the environment role. Changing them within the recipe strikes me as adding a significant amount of complexity to the code maintenance, as you now have to look at the recipes AND attributes when you modify the cookbook.

Rilindo Foster


#7

On Tue, Dec 17, 2013 at 9:37 AM, Rilindo Foster <
Rilindo.Foster@exlibrisgroup.com> wrote:

Slightly off-topic, but where are you modifying It may be a style or
philosophy thing, but I personally wouldn’t try to modify an attribute once
I declare it. I rather that I set my conditions up front in attributes file
and maybe override them as necessary within the environment role. Changing
them within the recipe strikes me as adding a significant amount of
complexity to the code maintenance, as you now have to look at the recipes
AND attributes when you modify the cookbook.

Yes, I agree. Here’s the actual scenario that I’m dealing with. I’m
creating some new EBS volumes, once they are created, I store those new
device entries in an attribute. Later in the code, I need to use those new
devices to either format, create a raid, etc. So, I need to save those
device values somewhere during the Chef run and be able to use them later
in the code.


John Alberts


#8

I wonder if there is a chef_sugar-ish solution we could build for
this–it’s awfully common. I suspect it is possible to write a
"all_together_resource" resource that let you write:

all_together do
lsdir = 0
(1…2).each do |count|
ruby_block “ls /tmp” do
block do
lsdir = %x[ls /tmp]
end
end

file "/tmp/file#{count}.txt" do
  content lsdir
end

end
end

This would create an all_together_resource resource which will execute the
entire block in its action :run, with each resource running immediately as
soon as the resource is declared.

On Tue, Dec 17, 2013 at 10:03 AM, John Alberts john.m.alberts@gmail.comwrote:

On Tue, Dec 17, 2013 at 9:37 AM, Rilindo Foster <
Rilindo.Foster@exlibrisgroup.com> wrote:

Slightly off-topic, but where are you modifying It may be a style or
philosophy thing, but I personally wouldn’t try to modify an attribute once
I declare it. I rather that I set my conditions up front in attributes file
and maybe override them as necessary within the environment role. Changing
them within the recipe strikes me as adding a significant amount of
complexity to the code maintenance, as you now have to look at the recipes
AND attributes when you modify the cookbook.

Yes, I agree. Here’s the actual scenario that I’m dealing with. I’m
creating some new EBS volumes, once they are created, I store those new
device entries in an attribute. Later in the code, I need to use those new
devices to either format, create a raid, etc. So, I need to save those
device values somewhere during the Chef run and be able to use them later
in the code.


John Alberts


#9

Interesting thought. Sounds like a new weekend project. :slight_smile:

On Tue, Dec 17, 2013 at 11:19 AM, John Keiser jkeiser@getchef.com wrote:

I wonder if there is a chef_sugar-ish solution we could build for
this–it’s awfully common. I suspect it is possible to write a
"all_together_resource" resource that let you write:

all_together do
lsdir = 0
(1…2).each do |count|
ruby_block “ls /tmp” do
block do
lsdir = %x[ls /tmp]
end
end

file "/tmp/file#{count}.txt" do
  content lsdir
end

end
end

This would create an all_together_resource resource which will execute the
entire block in its action :run, with each resource running immediately as
soon as the resource is declared.

On Tue, Dec 17, 2013 at 10:03 AM, John Alberts john.m.alberts@gmail.comwrote:

On Tue, Dec 17, 2013 at 9:37 AM, Rilindo Foster <
Rilindo.Foster@exlibrisgroup.com> wrote:

Slightly off-topic, but where are you modifying It may be a style or
philosophy thing, but I personally wouldn’t try to modify an attribute once
I declare it. I rather that I set my conditions up front in attributes file
and maybe override them as necessary within the environment role. Changing
them within the recipe strikes me as adding a significant amount of
complexity to the code maintenance, as you now have to look at the recipes
AND attributes when you modify the cookbook.

Yes, I agree. Here’s the actual scenario that I’m dealing with. I’m
creating some new EBS volumes, once they are created, I store those new
device entries in an attribute. Later in the code, I need to use those new
devices to either format, create a raid, etc. So, I need to save those
device values somewhere during the Chef run and be able to use them later
in the code.


John Alberts


John Alberts


#10

I see a couple of more “flexible” options. I think node.run_state would
work here but I’m not positive without testing it.

Another option that I’ve always found interesting is creating/using a
databag as part of a recipe. It’s a bit risky in that the databag is just
out there without a commensurate copy in your repo though.

Really this feels like the job of an LWRP. There’s a lot of logic you’re
describing that would really muddy a recipe.

On Tue, Dec 17, 2013 at 1:03 PM, John Alberts john.m.alberts@gmail.comwrote:

On Tue, Dec 17, 2013 at 9:37 AM, Rilindo Foster <
Rilindo.Foster@exlibrisgroup.com> wrote:

Slightly off-topic, but where are you modifying It may be a style or
philosophy thing, but I personally wouldn’t try to modify an attribute once
I declare it. I rather that I set my conditions up front in attributes file
and maybe override them as necessary within the environment role. Changing
them within the recipe strikes me as adding a significant amount of
complexity to the code maintenance, as you now have to look at the recipes
AND attributes when you modify the cookbook.

Yes, I agree. Here’s the actual scenario that I’m dealing with. I’m
creating some new EBS volumes, once they are created, I store those new
device entries in an attribute. Later in the code, I need to use those new
devices to either format, create a raid, etc. So, I need to save those
device values somewhere during the Chef run and be able to use them later
in the code.


John Alberts


#11

Isn’t the issue only that %x[ls /tmp] is being evaluated at compile time?
Avoid that notation and you’ll be able to do what you intend. Use "system"
instead and it will be evaluated as part of the ruby block.

On Tue, Dec 17, 2013 at 10:03 AM, John Alberts john.m.alberts@gmail.comwrote:

On Tue, Dec 17, 2013 at 9:37 AM, Rilindo Foster <
Rilindo.Foster@exlibrisgroup.com> wrote:

Slightly off-topic, but where are you modifying It may be a style or
philosophy thing, but I personally wouldn’t try to modify an attribute once
I declare it. I rather that I set my conditions up front in attributes file
and maybe override them as necessary within the environment role. Changing
them within the recipe strikes me as adding a significant amount of
complexity to the code maintenance, as you now have to look at the recipes
AND attributes when you modify the cookbook.

Yes, I agree. Here’s the actual scenario that I’m dealing with. I’m
creating some new EBS volumes, once they are created, I store those new
device entries in an attribute. Later in the code, I need to use those new
devices to either format, create a raid, etc. So, I need to save those
device values somewhere during the Chef run and be able to use them later
in the code.


John Alberts


#12

There’s also a fundamental misunderstanding about how to use Chef going on
here…

A Chef resource is a test-and-repair datastructure. It lives in a compiled
resource_collection and is meant to describe some policy point about the
system.

This example is trying to use one Chef resource to harvest information from
the system, use the node objetc as global state tracking mechanism, then
use another resource one to ensure its contents of a file.

The best thing to do would be to restate the problem. Start every system
with “I declare”.

"I declare that the file /tmp/file-1 should have the output of ls /tmp as
its content.

There are a couple different ways to solve this, but the easiest would be
to stash the contents of your ls in a compile phase variable and pass it as
a parameters to your file resources.

this is not a resource. It will be discarded after the compile phase

lsdir shell_out!(“ls /tmp”, :env => nil)

these are.

file “/tmp/file1” do
content lsdir
end

file “/tmp/file2” do
content lsdir
end

On Wed, Dec 18, 2013 at 1:17 PM, Dan Razzell danr@activestate.com wrote:

Isn’t the issue only that %x[ls /tmp] is being evaluated at compile time?
Avoid that notation and you’ll be able to do what you intend. Use "system"
instead and it will be evaluated as part of the ruby block.

On Tue, Dec 17, 2013 at 10:03 AM, John Alberts john.m.alberts@gmail.comwrote:

On Tue, Dec 17, 2013 at 9:37 AM, Rilindo Foster <
Rilindo.Foster@exlibrisgroup.com> wrote:

Slightly off-topic, but where are you modifying It may be a style or
philosophy thing, but I personally wouldn’t try to modify an attribute once
I declare it. I rather that I set my conditions up front in attributes file
and maybe override them as necessary within the environment role. Changing
them within the recipe strikes me as adding a significant amount of
complexity to the code maintenance, as you now have to look at the recipes
AND attributes when you modify the cookbook.

Yes, I agree. Here’s the actual scenario that I’m dealing with. I’m
creating some new EBS volumes, once they are created, I store those new
device entries in an attribute. Later in the code, I need to use those new
devices to either format, create a raid, etc. So, I need to save those
device values somewhere during the Chef run and be able to use them later
in the code.


John Alberts


#13

On Wed, Dec 18, 2013 at 2:19 PM, Sean OMeara someara@opscode.com wrote:

There’s also a fundamental misunderstanding about how to use Chef going on
here…

I completely understand what you’re saying and I agree that my original
example is a bad example of what I’m trying to accomplish which is creating
a raid volume from 2 newly created ebs volumes.

So… to use your suggestion of declaration: ‘I declare this system should
have a raid 0 volume created from 2 new EBS volumes.’


John Alberts


#14

On Wed, Dec 18, 2013 at 10:17 AM, Dan Razzell danr@activestate.com wrote:

Isn’t the issue only that %x[ls /tmp] is being evaluated at compile time?
Avoid that notation and you’ll be able to do what you intend. Use "system"
instead and it will be evaluated as part of the ruby block.

I thought any ruby code inside a ruby block will not be evaluated during
compile time and instead during the normal order of the execution phase. I
haven’t tested this yet, but does it really matter if inside that ruby
block I’ve used, %x[ls /tmp] or ls /tmp or system(“ls /tmp”) ?


John Alberts


#15

you are right , when you pass a block inside a ruby_block resource’s block
attribute, its not invoked during the compile phase (unless you do
run_action(:run) hacks). The challenge is how you capture this computed
value and pass it back to another resource. node.run_state is one way of
doing it. but again this is hacky. Also when reusing the coputed value you
have to ensure the consumer ruby code is also evaluated in the execution
phase, not it compile phase. i.e. another ruby_block resource where the
block parameter is accessing node.run_state will work, but file resource’s
content attribute is evaluated during compile time , even if you do content
node.run_state[:value_from_other_ruby_blockl], you are accessing a value
that is not computed yet.

look this is hacky, but this will work:

ruby_block “foo” do
block do
node.run_state[:troll] = ls /tmp
end
end.run_action(:run)

file “/etc/bar” do
content node.run_state[:troll]
end

i think use cases like this shows that we really dont have tow independent
resources, rather they are coupled where one of them is dependent on
other’s attribute value. hence it might be worth to cleanly implement it in
an lwrp where.

On Wed, Dec 18, 2013 at 4:10 PM, John Alberts john.m.alberts@gmail.comwrote:

On Wed, Dec 18, 2013 at 10:17 AM, Dan Razzell danr@activestate.comwrote:

Isn’t the issue only that %x[ls /tmp] is being evaluated at compile
time? Avoid that notation and you’ll be able to do what you intend. Use
"system" instead and it will be evaluated as part of the ruby block.

I thought any ruby code inside a ruby block will not be evaluated during
compile time and instead during the normal order of the execution phase. I
haven’t tested this yet, but does it really matter if inside that ruby
block I’ve used, %x[ls /tmp] or ls /tmp or system(“ls /tmp”) ?


John Alberts


#16

You want to implement this as an atomic operation from within a custom
provider.

On Wed, Dec 18, 2013 at 7:10 PM, John Alberts john.m.alberts@gmail.comwrote:

On Wed, Dec 18, 2013 at 10:17 AM, Dan Razzell danr@activestate.comwrote:

Isn’t the issue only that %x[ls /tmp] is being evaluated at compile
time? Avoid that notation and you’ll be able to do what you intend. Use
"system" instead and it will be evaluated as part of the ruby block.

I thought any ruby code inside a ruby block will not be evaluated during
compile time and instead during the normal order of the execution phase. I
haven’t tested this yet, but does it really matter if inside that ruby
block I’ve used, %x[ls /tmp] or ls /tmp or system(“ls /tmp”) ?


John Alberts


#17

Thanks again to everyone for their input. I think it’s been decided beyond
a shadow of a doubt that I should implement this in a custom provider. :slight_smile:

A question to Ranjib’s example; where can I find more information about
node.run_state? I only see mention of it regarding a windows provider and
really no where else.

On Wed, Dec 18, 2013 at 4:40 PM, Sean OMeara someara@opscode.com wrote:

You want to implement this as an atomic operation from within a custom
provider.

On Wed, Dec 18, 2013 at 7:10 PM, John Alberts john.m.alberts@gmail.comwrote:

On Wed, Dec 18, 2013 at 10:17 AM, Dan Razzell danr@activestate.comwrote:

Isn’t the issue only that %x[ls /tmp] is being evaluated at compile
time? Avoid that notation and you’ll be able to do what you intend. Use
"system" instead and it will be evaluated as part of the ruby block.

I thought any ruby code inside a ruby block will not be evaluated during
compile time and instead during the normal order of the execution phase. I
haven’t tested this yet, but does it really matter if inside that ruby
block I’ve used, %x[ls /tmp] or ls /tmp or system(“ls /tmp”) ?


John Alberts


John Alberts


#18

i cant think of any standard docs, except i some blogs out there that
recommended using it for temporary storage (the dude from nordstorm?) . I
knew its there, but never used it. When i first encountered it in our code
base (it was sumologic cookbook’s sumo_source definition), i had changed it
to an hwrp.

iirc node.run_state was used to detect loaded recipes (remember currently
only recipes that are assigned via run list are present inside node.recipes
not the ones which applied via included_recipe), there was some breaking
changes during chef 10-> 11 migration too,

On Wed, Dec 18, 2013 at 4:48 PM, John Alberts john.m.alberts@gmail.comwrote:

Thanks again to everyone for their input. I think it’s been decided
beyond a shadow of a doubt that I should implement this in a custom
provider. :slight_smile:

A question to Ranjib’s example; where can I find more information about
node.run_state? I only see mention of it regarding a windows provider and
really no where else.

On Wed, Dec 18, 2013 at 4:40 PM, Sean OMeara someara@opscode.com wrote:

You want to implement this as an atomic operation from within a custom
provider.

On Wed, Dec 18, 2013 at 7:10 PM, John Alberts john.m.alberts@gmail.comwrote:

On Wed, Dec 18, 2013 at 10:17 AM, Dan Razzell danr@activestate.comwrote:

Isn’t the issue only that %x[ls /tmp] is being evaluated at compile
time? Avoid that notation and you’ll be able to do what you intend. Use
"system" instead and it will be evaluated as part of the ruby block.

I thought any ruby code inside a ruby block will not be evaluated during
compile time and instead during the normal order of the execution phase. I
haven’t tested this yet, but does it really matter if inside that ruby
block I’ve used, %x[ls /tmp] or ls /tmp or system(“ls /tmp”) ?


John Alberts


John Alberts


#19

May I recommend the ebs_raid LWRP in the aws cookbook?

  • Julian

On Wed, Dec 18, 2013 at 7:07 PM, John Alberts john.m.alberts@gmail.com wrote:

On Wed, Dec 18, 2013 at 2:19 PM, Sean OMeara someara@opscode.com wrote:

There’s also a fundamental misunderstanding about how to use Chef going on
here…

I completely understand what you’re saying and I agree that my original
example is a bad example of what I’m trying to accomplish which is creating
a raid volume from 2 newly created ebs volumes.

So… to use your suggestion of declaration: ‘I declare this system should
have a raid 0 volume created from 2 new EBS volumes.’


John Alberts


[ Julian C. Dunn jdunn@aquezada.com * Sorry, I’m ]
[ WWW: http://www.aquezada.com/staff/julian * only Web 1.0 ]
[ gopher://sdf.org/1/users/keymaker/ * compliant! ]
[ PGP: 91B3 7A9D 683C 7C16 715F 442C 6065 D533 FDC2 05B9 ]


#20

On Wed, Dec 18, 2013 at 7:34 PM, Julian C. Dunn jdunn@aquezada.com wrote:

May I recommend the ebs_raid LWRP in the aws cookbook?

Thanks for reminding me about that. I completely forgot it was added to
the aws cookbook a while back.


John Alberts