Attribute value availability + the implicit dependency graph and chef-deploy patch

Chef Infra (archive)
1

Last night I was seeing an issue where provisioning “apache + passenger
worked” but “apache + passenger + rails app” (deploying the rails app
w/chef-deploy) was failing. In the first scenario, the passenger
cookbook’s attributes was able to access the apache cookbook’s
attributes. In the second scenario, the apache attributes were not
accessible to passenger’s. But in all cases apache attributes are
accessible in the passenger recipe script. In the second scenario I
launched with { “recipes” : [ “railsapp” ] }, since the rails app recipe
has ‘include_recipe “passenger”’ and the passenger recipe has
’include_recipe “apache”’ I expected the dependencies declared via
include_recipe to be traversed to make the attribute values available.
But apparently that doesn’t work, cookbook processing order is random
(?!). This was really frustrating to debug (the symptom was template
names/paths mysteriously null valued and fairly useless error messages
in the ensuing stacktrace); are there docs I’ve overlooked on how
cookbooks relate to each other as far as execution order and data model
life cycles? mentioned plans for something like a
"require_attribute" operator in the conception phase, which sounds
great, though it seems like requiring a recipe should be sufficient to
carry attribute values forward; there’s a dependency graph already
implied by the “require_recipe” operator. If there are docs I’ve
overlooked, forgive the length and thanks for reading this far :slight_smile:

Finally, the chef-deploy resource seemed to be puking on some path
disagreements as well as null valued config elements, diff attached.
thanks!
-Ian

–
Ian Kallen
blog: http://www.arachna.com/roller/spidaman
tweetz: http://twitter.com/spidaman
vox: 415.505.5208

2

Hey-

On Apr 23, 2009, at 9:31 AM, Ian Kallen wrote:

Last night I was seeing an issue where provisioning "apache +
passenger worked" but "apache + passenger + rails app" (deploying
the rails app w/chef-deploy) was failing. In the first scenario, the
passenger cookbook's attributes was able to access the apache
cookbook's attributes. In the second scenario, the apache attributes
were not accessible to passenger's. But in all cases apache
attributes are accessible in the passenger recipe script. In the
second scenario I launched with { "recipes" : [ "railsapp" ] },
since the rails app recipe has 'include_recipe "passenger"' and the
passenger recipe has 'include_recipe "apache"' I expected the
dependencies declared via include_recipe to be traversed to make the
attribute values available. But apparently that doesn't work,
cookbook processing order is random (?!). This was really
frustrating to debug (the symptom was template names/paths
mysteriously null valued and fairly useless error messages in the
ensuing stacktrace); are there docs I've overlooked on how
cookbooks relate to each other as far as execution order and data
model life cycles? mentioned plans for something like a
"require_attribute" operator in the conception phase, which sounds
great, though it seems like requiring a recipe should be sufficient
to carry attribute values forward; there's a dependency graph
already implied by the "require_recipe" operator. If there are docs
I've overlooked, forgive the length and thanks for reading this far :slight_smile:

Finally, the chef-deploy resource seemed to be puking on some path
disagreements as well as null valued config elements, diff attached.
thanks!
-Ian

--
Ian Kallen
blog: What's That Noise?! [Ian Kallen's Weblog]
tweetz: http://twitter.com/spidaman
vox: 415.505.5208

--- a/lib/chef-deploy/cached_deploy.rb
+++ b/lib/chef-deploy/cached_deploy.rb
@@ -24,7 +24,9 @@ class CachedDeploy
end

So this part of the patch is probably ok, i will play with it today.

def restart

  • unless @configuration[:restart_command].empty?
  • #unless @configuration[:restart_command].empty?

  • undefined method `empty?' for nil:NilClass

  • if @configuration.has_key? :restart_command &&
    @configuration[:restart_command]
    Chef::Log.info "restarting app: #{latest_release}"
    Chef::Log.info run(@configuration[:restart_command])
    end
    @@ -63,7 +65,7 @@ class CachedDeploy

This part of the patch is not valid. The general idea is just like a
capistrano deplopy, you keep the database.yml file in /
shared/config/database.yml on the server and then the deploy symlinks
it in , this is so you do not keep database credentails in your source
control.

So you will need to have another part of your chef recipe place the
database.yml file in the right place before you run the deploy
resource and it will work fine.

def migrate
if @configuration[:migrate]

  •  run "ln -nfs #{shared_path}/config/database.yml

#{latest_release}/config/

  •  run "ln -nfs #{shared_path}/cached-copy/config/database.yml

#{latest_rele
Chef::Log.info "Migrating: cd #{latest_release} &&
RAILS_ENV=#{@configura
Chef::Log.info run("cd #{latest_release} &&
RAILS_ENV=#{@configuration[:e
end
@@ -175,4 +177,4 @@ class CachedDeploy
def copy_exclude
@copy_exclude ||= Array(configuration.fetch(:copy_exclude, ))
end
-end
\ No newline at end of file
+end

Cheers-
Ezra Zygmuntowicz
ez@engineyard.com

3

Ezra Zygmuntowicz wrote:

This part of the patch is not valid. The general idea is just like a
capistrano deplopy, you keep the database.yml file in
/shared/config/database.yml on the server and then the
deploy symlinks it in , this is so you do not keep database
credentails in your source control.

So you will need to have another part of your chef recipe place the
database.yml file in the right place before you run the deploy
resource and it will work fine.

  •  run "ln -nfs #{shared_path}/config/database.yml

#{latest_release}/config/

  •  run "ln -nfs #{shared_path}/cached-copy/config/database.yml

#{latest_rele

Ah, this is where my newbness to ruby+rails (+cap) is biting me. I quit
my job last month, until then only looked at ruby in passing; have since
started reading pick axe and online stuffs more vigorously. Probably
TMI. Anyway.

OK, in that context what chef-deploy was doing with the paths and
symlinks makes sense. I'll re-cobble this setup around that.
thanks!
-Ian

--
Ian Kallen
blog: What's That Noise?! [Ian Kallen's Weblog]
tweetz: http://twitter.com/spidaman
vox: 415.505.5208

4

On Thu, Apr 23, 2009 at 9:31 AM, Ian Kallen spidaman.list@gmail.com wrote:

Last night I was seeing an issue where provisioning "apache + passenger
worked" but "apache + passenger + rails app" (deploying the rails app
w/chef-deploy) was failing. In the first scenario, the passenger cookbook's
attributes was able to access the apache cookbook's attributes. In the
second scenario, the apache attributes were not accessible to passenger's.
But in all cases apache attributes are accessible in the passenger recipe
script. In the second scenario I launched with { "recipes" : [ "railsapp" ]
}, since the rails app recipe has 'include_recipe "passenger"' and the
passenger recipe has 'include_recipe "apache"' I expected the dependencies
declared via include_recipe to be traversed to make the attribute values
available. But apparently that doesn't work, cookbook processing order is
random (?!). This was really frustrating to debug (the symptom was template
names/paths mysteriously null valued and fairly useless error messages in
the ensuing stacktrace); are there docs I've overlooked on how cookbooks
relate to each other as far as execution order and data model life cycles?
mentioned plans for something like a "require_attribute" operator in
the conception phase, which sounds great, though it seems like requiring a
recipe should be sufficient to carry attribute values forward; there's a
dependency graph already implied by the "require_recipe" operator. If there
are docs I've overlooked, forgive the length and thanks for reading this far
:slight_smile:

The issue is that Attributes are parsed before the recipes - that
means we you can't use the recipe inclusion for attribute file
ordinality.

http://wiki.opscode.com/display/chef/Anatomy+of+a+Chef+Run

Libraries are loaded first, then Attributes, then Definitions,
followed by Recipes.

(This is because you may have an Attribute that is set in one
cookbook, but utilized in another.)

Adam

--
Opscode, Inc.
Adam Jacob, CTO
T: (206) 508-4759 E: adam@opscode.com

5

Adam Jacob wrote:

http://wiki.opscode.com/display/chef/Anatomy+of+a+Chef+Run

Libraries are loaded first, then Attributes, then Definitions,
followed by Recipes.

(This is because you may have an Attribute that is set in one
cookbook, but utilized in another.
Great, thanks Adam. Let me make sure I'm following the semantics. Given
N cookbooks, the order in which each of the respective attributes loaded
is non-deterministic. Ergo a recipe can depend on values being set for
attributes in any cookbook but having values from one cookbook's
attributes depend on values set in another's is essentially a dice roll.
Therefore in this case (assume elsewhere we've required another
cookbook's recipe where that cookbook's attributes file sets "apache"
but it doesn't matter, cookbook attribute processing isn't ordered in
anyway)

passenger[:apache_load_path] =
"#{apache[:dir]}/mods-available/passenger.load" if attribute?("apache")

whether node[:passenger][:apache_load_path] ever gets set is a matter of
luck; the "apache" attribute may or may not be set when this is being
processed. However, if the cookbook needs information from another, it
must access it in the recipe to rely on it being set. Is that right?

--
Ian Kallen
blog: What's That Noise?! [Ian Kallen's Weblog]
tweetz: http://twitter.com/spidaman
vox: 415.505.5208

6

On Thu, Apr 23, 2009 at 12:46 PM, Ian Kallen spidaman.list@gmail.com wrote:

Great, thanks Adam. Let me make sure I'm following the semantics. Given N
cookbooks, the order in which each of the respective attributes loaded is
non-deterministic. Ergo a recipe can depend on values being set for
attributes in any cookbook but having values from one cookbook's
attributes depend on values set in another's is essentially a dice roll.
Therefore in this case (assume elsewhere we've required another cookbook's
recipe where that cookbook's attributes file sets "apache" but it doesn't
matter, cookbook attribute processing isn't ordered in anyway)

passenger[:apache_load_path] =
"#{apache[:dir]}/mods-available/passenger.load" if attribute?("apache")

whether node[:passenger][:apache_load_path] ever gets set is a matter of
luck; the "apache" attribute may or may not be set when this is being
processed. However, if the cookbook needs information from another, it must
access it in the recipe to rely on it being set. Is that right?

Yep. That is why the solution may very well be include_attribute,
with similar semantics to recipes. It lets us out of that trap.

Adam

--
Opscode, Inc.
Adam Jacob, CTO
T: (206) 508-4759 E: adam@opscode.com