Running chef-client 10.12.0 for the first time on a fresh Ubuntu 12.04 LTS node, the run fails after downloading the cookbooks from Chef Server, because all the files it supposedly downloaded into the /var/cache/chef/cookbooks directory hierarchy are missing!
Taking a look at the debug output, I can see, for example, this portion showing a couple of cookbook files that get downloaded successfully:
[2012-09-07T06:57:57+00:00] DEBUG: Signing the request as ip-10-171-90-254.us-west-1.compute.internal
[2012-09-07T06:57:57+00:00] DEBUG: String to sign: 'Method:GET
Hashed Path:Bsx3zOo6YGnjwbdCT2G3Fb1FewI=
X-Ops-Content-Hash:2jmj7l5rSw0yVb/vlWAYkK/YBwk=
X-Ops-Timestamp:2012-09-07T06:57:57Z
X-Ops-UserId:ip-10-171-90-254.us-west-1.compute.internal’
Header hash: {“X-Ops-Timestamp”=>“2012-09-07T06:57:57Z”, “X-Ops-Content-Hash”=>“2jmj7l5rSw0yVb/vlWAYkK/YBwk=”, “X-Ops-Userid”=>“ip-10-171-90-254.us-west-1.compute.internal”, “X-Ops-Authorization-6”=>“PBvVh0poHpqoG6aqbKXPqoutjHEc5Zgd7XS9fHFhgg==”, “X-Ops-Authorization-5”=>“MnUQDK/FwsbwJW4P6nzDIN3IcCWIX9BWsvEdq4zKQ7VX1X2XH9+eo259GJmB”, “X-Ops-Authorization-4”=>“dbAAW/ulGTgn5wpdO7QsPz1MWOwlEUXb2RVA/mBnwOoz/Y50/mE6UiJos4f8”, “X-Ops-Authorization-3”=>“SgPapXlqI0GhepeyxXNy2GjpaORcvwyWVfj+YHbqcD9V639MNY2T5G2Gj04t”, “X-Ops-Authorization-2”=>“BkwrC1hZ/1naKNasziUTfHgVR5L7N63jVIqbmpaiAs9xrf3yVLNztHILohfu”, “X-Ops-Authorization-1”=>“re6e46EOyrcFCpVU1IHcjsfpFDqXZNKBIIJ10ZMpldx77d8hLLogal+fTZGe”, “X-Ops-Sign”=>“algorithm=sha1;version=1.0;”}
[2012-09-07T06:57:57+00:00] DEBUG: Sending HTTP Request via GET to ec2-184-169-187-137.us-west-1.compute.amazonaws.com:4000/cookbooks/mysql/1.2.6/files/76c6ff4ec2e7ca2f46e9407ed305f6dd
[2012-09-07T06:57:57+00:00] DEBUG: Streaming download from http://ec2-184-169-187-137.us-west-1.compute.amazonaws.com:4000/cookbooks/mysql/1.2.6/files/76c6ff4ec2e7ca2f46e9407ed305f6dd to tempfile /tmp/chef-rest20120907-9045-1a50gu1-0
[2012-09-07T06:57:57+00:00] INFO: Storing updated cookbooks/mysql/libraries/helpers.rb in the cache.
[2012-09-07T06:57:57+00:00] DEBUG: Creating directory /var/cache/chef/cookbooks/mysql/libraries
[2012-09-07T06:57:57+00:00] DEBUG: Signing the request as ip-10-171-90-254.us-west-1.compute.internal
[2012-09-07T06:57:57+00:00] DEBUG: String to sign: 'Method:GET
Hashed Path:QdRxdiZUMx/eQd93YXhPtNmk5oI=
X-Ops-Content-Hash:2jmj7l5rSw0yVb/vlWAYkK/YBwk=
X-Ops-Timestamp:2012-09-07T06:57:57Z
X-Ops-UserId:ip-10-171-90-254.us-west-1.compute.internal’
Header hash: {“X-Ops-Timestamp”=>“2012-09-07T06:57:57Z”, “X-Ops-Content-Hash”=>“2jmj7l5rSw0yVb/vlWAYkK/YBwk=”, “X-Ops-Userid”=>“ip-10-171-90-254.us-west-1.compute.internal”, “X-Ops-Authorization-6”=>“0gRheLKQvX5syxKDzurFIAd4HvsQ/e3X49ewpeSU/A==”, “X-Ops-Authorization-5”=>“LOjc8OQGznPh8pniJz4NPajQjZtHPxqg2/Dd3k5v3enGaGWuGFSp3Tfa8gec”, “X-Ops-Authorization-4”=>“nWDybANYZLRKLHD6NNAGBOxFvl0Dwgl323j5I/5SDAi3pDMwjbDZtvw6aDjD”, “X-Ops-Authorization-3”=>“aI1BefpY9vs4IHB7dmzj0fU4vfsDH7aWXH+Hocyktmdypk+Ab9aIQAfhtyCO”, “X-Ops-Authorization-2”=>“KfLMPzH1OjyidJNpGruNF37MIBD7j7GnxAksI3SQaW8YBBGBIN/+E87tOapl”, “X-Ops-Authorization-1”=>“4zLvH8UEN5PMmNfUVTIQ051Mu8IIb6Qw63lUE6TXwRTc6bMpytyD+e/9E/3o”, “X-Ops-Sign”=>“algorithm=sha1;version=1.0;”}
[2012-09-07T06:57:57+00:00] DEBUG: Sending HTTP Request via GET to ec2-184-169-187-137.us-west-1.compute.amazonaws.com:4000/cookbooks/mysql/1.2.6/files/d05e075954b208e1a15f6ddcd1ab83c0
[2012-09-07T06:57:57+00:00] DEBUG: Streaming download from http://ec2-184-169-187-137.us-west-1.compute.amazonaws.com:4000/cookbooks/mysql/1.2.6/files/d05e075954b208e1a15f6ddcd1ab83c0 to tempfile /tmp/chef-rest20120907-9045-109g3we-0
[2012-09-07T06:57:57+00:00] INFO: Storing updated cookbooks/mysql/libraries/database.rb in the cache.
Given this, you would expect to find helpers.rb and database.rb in /var/cache/chef/cookbooks/mysql/libraries. But they aren’t there. The run fails as soon as it attempts to use one of the cookbook files it has downloaded:
[2012-09-07T06:58:10+00:00] DEBUG: Loading cookbook mysql’s library file: /var/cache/chef/cookbooks/mysql/libraries/helpers.rb
[2012-09-07T06:58:10+00:00] ERROR: Running exception handlers
[2012-09-07T06:58:10+00:00] FATAL: Saving node information to /var/cache/chef/failed-run-data.json
[2012-09-07T06:58:10+00:00] ERROR: Exception handlers complete
[2012-09-07T06:58:10+00:00] DEBUG: Re-raising exception: LoadError - no such file to load – /var/cache/chef/cookbooks/mysql/libraries/helpers.rb
/var/lib/gems/1.8/gems/chef-10.12.0/bin/…/lib/chef/run_context.rb:82:in load' /var/lib/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/run_context.rb:82:inload_libraries’
/var/lib/gems/1.8/gems/chef-10.12.0/bin/…/lib/chef/run_context.rb:120:in call' /var/lib/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/run_context.rb:120:inforeach_cookbook_load_segment’
/var/lib/gems/1.8/gems/chef-10.12.0/bin/…/lib/chef/run_context.rb:119:in each' /var/lib/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/run_context.rb:119:inforeach_cookbook_load_segment’
/var/lib/gems/1.8/gems/chef-10.12.0/bin/…/lib/chef/run_context.rb:117:in each' /var/lib/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/run_context.rb:117:inforeach_cookbook_load_segment’
/var/lib/gems/1.8/gems/chef-10.12.0/bin/…/lib/chef/run_context.rb:80:in load_libraries' /var/lib/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/run_context.rb:57:inload’
/var/lib/gems/1.8/gems/chef-10.12.0/bin/…/lib/chef/client.rb:199:in setup_run_context' /var/lib/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/client.rb:162:inrun’
/var/lib/gems/1.8/gems/chef-10.12.0/bin/…/lib/chef/application/client.rb:254:in run_application' /var/lib/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/application/client.rb:241:inloop’
/var/lib/gems/1.8/gems/chef-10.12.0/bin/…/lib/chef/application/client.rb:241:in run_application' /var/lib/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/application.rb:70:inrun’
/var/lib/gems/1.8/gems/chef-10.12.0/bin/chef-client:26
/usr/local/bin/chef-client:19:in `load’
/usr/local/bin/chef-client:19
[2012-09-07T06:58:10+00:00] FATAL: Stacktrace dumped to /var/cache/chef/chef-stacktrace.out
Investigating /var/cache/chef/cookbooks after this failed run, the full directory hierarchy is there, but it doesn’t contain a single file. /var/cache/chef/cookbooks/mysql/libraries exists, for example, but it’s empty.
Any ideas what could be wrong?
This same bootstrapping process is working perfectly for us on Ubuntu 11.04. I’m working on moving to Ubuntu 12.04, however, and this problem is where I’m stuck.
–
Kevin Yank
Chief Technology Officer, Avalanche Technology Group / AVG (AU/NZ) Pty Ltd
2/101 Tulip Street, Sandringham Victoria 3191 Australia
ph: +61 3 9581 0806