First run on Ubuntu 12 node: /var/cache/chef/cookbooks hierarchy contains no files


#1

Running chef-client 10.12.0 for the first time on a fresh Ubuntu 12.04 LTS node, the run fails after downloading the cookbooks from Chef Server, because all the files it supposedly downloaded into the /var/cache/chef/cookbooks directory hierarchy are missing!

Taking a look at the debug output, I can see, for example, this portion showing a couple of cookbook files that get downloaded successfully:

[2012-09-07T06:57:57+00:00] DEBUG: Signing the request as ip-10-171-90-254.us-west-1.compute.internal
[2012-09-07T06:57:57+00:00] DEBUG: String to sign: 'Method:GET
Hashed Path:Bsx3zOo6YGnjwbdCT2G3Fb1FewI=
X-Ops-Content-Hash:2jmj7l5rSw0yVb/vlWAYkK/YBwk=
X-Ops-Timestamp:2012-09-07T06:57:57Z
X-Ops-UserId:ip-10-171-90-254.us-west-1.compute.internal’
Header hash: {“X-Ops-Timestamp”=>“2012-09-07T06:57:57Z”, “X-Ops-Content-Hash”=>“2jmj7l5rSw0yVb/vlWAYkK/YBwk=”, “X-Ops-Userid”=>“ip-10-171-90-254.us-west-1.compute.internal”, “X-Ops-Authorization-6”=>“PBvVh0poHpqoG6aqbKXPqoutjHEc5Zgd7XS9fHFhgg==”, “X-Ops-Authorization-5”=>“MnUQDK/FwsbwJW4P6nzDIN3IcCWIX9BWsvEdq4zKQ7VX1X2XH9+eo259GJmB”, “X-Ops-Authorization-4”=>“dbAAW/ulGTgn5wpdO7QsPz1MWOwlEUXb2RVA/mBnwOoz/Y50/mE6UiJos4f8”, “X-Ops-Authorization-3”=>“SgPapXlqI0GhepeyxXNy2GjpaORcvwyWVfj+YHbqcD9V639MNY2T5G2Gj04t”, “X-Ops-Authorization-2”=>“BkwrC1hZ/1naKNasziUTfHgVR5L7N63jVIqbmpaiAs9xrf3yVLNztHILohfu”, “X-Ops-Authorization-1”=>“re6e46EOyrcFCpVU1IHcjsfpFDqXZNKBIIJ10ZMpldx77d8hLLogal+fTZGe”, “X-Ops-Sign”=>“algorithm=sha1;version=1.0;”}
[2012-09-07T06:57:57+00:00] DEBUG: Sending HTTP Request via GET to ec2-184-169-187-137.us-west-1.compute.amazonaws.com:4000/cookbooks/mysql/1.2.6/files/76c6ff4ec2e7ca2f46e9407ed305f6dd
[2012-09-07T06:57:57+00:00] DEBUG: Streaming download from http://ec2-184-169-187-137.us-west-1.compute.amazonaws.com:4000/cookbooks/mysql/1.2.6/files/76c6ff4ec2e7ca2f46e9407ed305f6dd to tempfile /tmp/chef-rest20120907-9045-1a50gu1-0
[2012-09-07T06:57:57+00:00] INFO: Storing updated cookbooks/mysql/libraries/helpers.rb in the cache.
[2012-09-07T06:57:57+00:00] DEBUG: Creating directory /var/cache/chef/cookbooks/mysql/libraries
[2012-09-07T06:57:57+00:00] DEBUG: Signing the request as ip-10-171-90-254.us-west-1.compute.internal
[2012-09-07T06:57:57+00:00] DEBUG: String to sign: 'Method:GET
Hashed Path:QdRxdiZUMx/eQd93YXhPtNmk5oI=
X-Ops-Content-Hash:2jmj7l5rSw0yVb/vlWAYkK/YBwk=
X-Ops-Timestamp:2012-09-07T06:57:57Z
X-Ops-UserId:ip-10-171-90-254.us-west-1.compute.internal’
Header hash: {“X-Ops-Timestamp”=>“2012-09-07T06:57:57Z”, “X-Ops-Content-Hash”=>“2jmj7l5rSw0yVb/vlWAYkK/YBwk=”, “X-Ops-Userid”=>“ip-10-171-90-254.us-west-1.compute.internal”, “X-Ops-Authorization-6”=>“0gRheLKQvX5syxKDzurFIAd4HvsQ/e3X49ewpeSU/A==”, “X-Ops-Authorization-5”=>“LOjc8OQGznPh8pniJz4NPajQjZtHPxqg2/Dd3k5v3enGaGWuGFSp3Tfa8gec”, “X-Ops-Authorization-4”=>“nWDybANYZLRKLHD6NNAGBOxFvl0Dwgl323j5I/5SDAi3pDMwjbDZtvw6aDjD”, “X-Ops-Authorization-3”=>“aI1BefpY9vs4IHB7dmzj0fU4vfsDH7aWXH+Hocyktmdypk+Ab9aIQAfhtyCO”, “X-Ops-Authorization-2”=>“KfLMPzH1OjyidJNpGruNF37MIBD7j7GnxAksI3SQaW8YBBGBIN/+E87tOapl”, “X-Ops-Authorization-1”=>“4zLvH8UEN5PMmNfUVTIQ051Mu8IIb6Qw63lUE6TXwRTc6bMpytyD+e/9E/3o”, “X-Ops-Sign”=>“algorithm=sha1;version=1.0;”}
[2012-09-07T06:57:57+00:00] DEBUG: Sending HTTP Request via GET to ec2-184-169-187-137.us-west-1.compute.amazonaws.com:4000/cookbooks/mysql/1.2.6/files/d05e075954b208e1a15f6ddcd1ab83c0
[2012-09-07T06:57:57+00:00] DEBUG: Streaming download from http://ec2-184-169-187-137.us-west-1.compute.amazonaws.com:4000/cookbooks/mysql/1.2.6/files/d05e075954b208e1a15f6ddcd1ab83c0 to tempfile /tmp/chef-rest20120907-9045-109g3we-0
[2012-09-07T06:57:57+00:00] INFO: Storing updated cookbooks/mysql/libraries/database.rb in the cache.

Given this, you would expect to find helpers.rb and database.rb in /var/cache/chef/cookbooks/mysql/libraries. But they aren’t there. The run fails as soon as it attempts to use one of the cookbook files it has downloaded:

[2012-09-07T06:58:10+00:00] DEBUG: Loading cookbook mysql’s library file: /var/cache/chef/cookbooks/mysql/libraries/helpers.rb
[2012-09-07T06:58:10+00:00] ERROR: Running exception handlers
[2012-09-07T06:58:10+00:00] FATAL: Saving node information to /var/cache/chef/failed-run-data.json
[2012-09-07T06:58:10+00:00] ERROR: Exception handlers complete
[2012-09-07T06:58:10+00:00] DEBUG: Re-raising exception: LoadError - no such file to load – /var/cache/chef/cookbooks/mysql/libraries/helpers.rb
/var/lib/gems/1.8/gems/chef-10.12.0/bin/…/lib/chef/run_context.rb:82:in load' /var/lib/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/run_context.rb:82:inload_libraries’
/var/lib/gems/1.8/gems/chef-10.12.0/bin/…/lib/chef/run_context.rb:120:in call' /var/lib/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/run_context.rb:120:inforeach_cookbook_load_segment’
/var/lib/gems/1.8/gems/chef-10.12.0/bin/…/lib/chef/run_context.rb:119:in each' /var/lib/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/run_context.rb:119:inforeach_cookbook_load_segment’
/var/lib/gems/1.8/gems/chef-10.12.0/bin/…/lib/chef/run_context.rb:117:in each' /var/lib/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/run_context.rb:117:inforeach_cookbook_load_segment’
/var/lib/gems/1.8/gems/chef-10.12.0/bin/…/lib/chef/run_context.rb:80:in load_libraries' /var/lib/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/run_context.rb:57:inload’
/var/lib/gems/1.8/gems/chef-10.12.0/bin/…/lib/chef/client.rb:199:in setup_run_context' /var/lib/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/client.rb:162:inrun’
/var/lib/gems/1.8/gems/chef-10.12.0/bin/…/lib/chef/application/client.rb:254:in run_application' /var/lib/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/application/client.rb:241:inloop’
/var/lib/gems/1.8/gems/chef-10.12.0/bin/…/lib/chef/application/client.rb:241:in run_application' /var/lib/gems/1.8/gems/chef-10.12.0/bin/../lib/chef/application.rb:70:inrun’
/var/lib/gems/1.8/gems/chef-10.12.0/bin/chef-client:26
/usr/local/bin/chef-client:19:in `load’
/usr/local/bin/chef-client:19
[2012-09-07T06:58:10+00:00] FATAL: Stacktrace dumped to /var/cache/chef/chef-stacktrace.out

Investigating /var/cache/chef/cookbooks after this failed run, the full directory hierarchy is there, but it doesn’t contain a single file. /var/cache/chef/cookbooks/mysql/libraries exists, for example, but it’s empty.

Any ideas what could be wrong?

This same bootstrapping process is working perfectly for us on Ubuntu 11.04. I’m working on moving to Ubuntu 12.04, however, and this problem is where I’m stuck.


Kevin Yank
Chief Technology Officer, Avalanche Technology Group / AVG (AU/NZ) Pty Ltd

2/101 Tulip Street, Sandringham Victoria 3191 Australia
ph: +61 3 9581 0806


#2

On Friday, September 7, 2012 at 12:56 AM, Kevin Yank wrote:

Running chef-client 10.12.0 for the first time on a fresh Ubuntu 12.04 LTS node, the run fails after downloading the cookbooks from Chef Server, because all the files it supposedly downloaded into the /var/cache/chef/cookbooks directory hierarchy are missing!
Any ideas what could be wrong?

This same bootstrapping process is working perfectly for us on Ubuntu 11.04. I’m working on moving to Ubuntu 12.04, however, and this problem is where I’m stuck.


Kevin Yank
Chief Technology Officer, Avalanche Technology Group / AVG (AU/NZ) Pty Ltd

2/101 Tulip Street, Sandringham Victoria 3191 Australia
ph: +61 3 9581 0806

Is it possible you’re running two chef-client instances at once, perhaps one out of init/cron/upstart/runit/etc. and one in the foreground of the bootstrap? The run_list doesn’t get saved until the end of the bootstrap run, so a second chef-client will see an empty run_list and then purge the cache of “unneeded” files.


Daniel DeLeo


#3

On 08/09/2012, at 1:36 AM, Daniel DeLeo dan@kallistec.com wrote:

Is it possible you’re running two chef-client instances at once, perhaps one out of init/cron/upstart/runit/etc. and one in the foreground of the bootstrap? The run_list doesn’t get saved until the end of the bootstrap run, so a second chef-client will see an empty run_list and then purge the cache of “unneeded” files.

Thanks so much, Daniel! Something like that may well be going on!

I have chef-solo installing chef-client, which gets launched as a daemon; however, the attributes I’m supplying for chef-client’s configuration aren’t being properly applied (a mystery I still need to look into), so I’m then patching /etc/chef/client.rb and running chef-client once to make sure it runs correctly before relaunching the daemon with ‘service chef-client restart’.

So yes, I’d say I have a race condition between the misconfigured chef-client daemon run and the config-patched ‘test run’. I should be able to fix that by killing the daemon before launching the test run.


Kevin Yank
http://kevinyank.com/


#4

On Saturday, September 8, 2012 at 7:40 AM, Kevin Yank wrote:

On 08/09/2012, at 1:36 AM, Daniel DeLeo <dan@kallistec.com (mailto:dan@kallistec.com)> wrote:

Is it possible you’re running two chef-client instances at once, perhaps one out of init/cron/upstart/runit/etc. and one in the foreground of the bootstrap? The run_list doesn’t get saved until the end of the bootstrap run, so a second chef-client will see an empty run_list and then purge the cache of “unneeded” files.

Thanks so much, Daniel! Something like that may well be going on!

I have chef-solo installing chef-client, which gets launched as a daemon; however, the attributes I’m supplying for chef-client’s configuration aren’t being properly applied (a mystery I still need to look into), so I’m then patching /etc/chef/client.rb and running chef-client once to make sure it runs correctly before relaunching the daemon with ‘service chef-client restart’.

So yes, I’d say I have a race condition between the misconfigured chef-client daemon run and the config-patched ‘test run’. I should be able to fix that by killing the daemon before launching the test run.


Kevin Yank
http://kevinyank.com/

Cool, glad to get you sorted out. We’ve added locking around chef-client runs in master, which should prevent this from happening. Since locking implementations and behaviors have historically varied between different operating systems, we’re waiting until Chef 11 to release it so we can be sure it’s well tested.


Daniel DeLeo