Cookbook files/ directory not syncd, chef-client a noop but shef -z "run_chef" works

I’ve got several very odd things, likely unrelated, happening in Chef 0.9.12
– listed here in order of importance:

  • “chef-client” exits without invoking any recipes, whereas “run_chef” in
    shef -z actually tries to run the cookbooks applied to this node.
  • the files/ directory in one of my cookbooks isn’t being pushed to the
    cache on my client
  • the shef command cookbooks.all appears to display “nil” for each cookbook

Pointers on any of these would be appreciated.

cookbooks$ find yumrepo/

yumrepo/files/default/RPM-GPG-KEY-EPEL

cookbooks$ knife cookbook upload -o . yumrepo
INFO: Saving yumrepo
INFO: Validating ruby files
INFO: Validating templates
INFO: Syntax OK
INFO: Generating Metadata
INFO: Uploading files
INFO: Upload complete!

…and over on the client…

chef-client

[Tue, 28 Dec 2010 23:05:51 +0000] INFO: Starting Chef Run (Version 0.9.12)
[Tue, 28 Dec 2010 23:06:09 +0000] INFO: Chef Run complete in 17.457648
seconds
[Tue, 28 Dec 2010 23:06:09 +0000] INFO: cleaning the checksum cache
[Tue, 28 Dec 2010 23:06:09 +0000] INFO: Running report handlers
[Tue, 28 Dec 2010 23:06:09 +0000] INFO: Report handlers complete

shef -z

loading configuration: /etc/chef/client.rb
Session type: solo
Loading…done.

This is shef, the Chef shell.
Chef Version: 0.9.12
http://www.opscode.com/chef
http://wiki.opscode.com/display/chef/Home

run help' for help,exit’ or ^D to quit.

Ohai2u root@staging.example.com!
chef > cookbooks.all
=> [nil, nil, nil, nil, nil]
chef > run_chef
…and things actually run as they should, until…
[Tue, 28 Dec 2010 23:09:22 +0000] ERROR:
cookbook_file[/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL]
(/var/chef/cache/cookbooks/yumrepo/definitions/yumkey.rb:47:in `from_file’)
had an error:
cookbook yumrepo does not contain file files/RPM-GPG-KEY-EPEL

and indeed, /var/chef/cache/cookbooks/yumrepo/ has no files/ directory
whatsoever.

Doing a bit more research --

At least one item was simply a matter of me confusing myself -- my
expectation that cookbook files would be stored in the cache on the client
node appears to have been simply inaccurate. Watching the HTTP PUT by knife
cookbook upload clearly shows the files in question being put, and their
names and signatures are included when the node retrieves its list of
cookbooks. I simply wasn't seeing anything being done with run_client on
account of it being less verbose than shef's default logging level (and
because my run_list was inappropriately constructed, not including the
recipe in which I was inserting extra logging).

It looks like there's only one real issue here, and I'm wondering if it's
simply a limitation of the "solo" mode shef -z runs in:

[Wed, 29 Dec 2010 15:16:33 +0000] ERROR:
template[/etc/yum.repos.d/epel.repo]
(/var/chef/cache/cookbooks/yumrepo/definitions/yumrepo.rb:45:in from_file') had an error: Must call Chef::Cookbook::FileVendor.on_create before calling create_from_manifest factory /usr/lib64/ruby/gems/1.8/gems/chef-0.9.12/lib/chef/cookbook/file_vendor.rb:34:in create_from_manifest'
/usr/lib64/ruby/gems/1.8/gems/chef-0.9.12/lib/chef/cookbook_version.rb:902:in
file_vendor' /usr/lib64/ruby/gems/1.8/gems/chef-0.9.12/lib/chef/cookbook_version.rb:536:in preferred_filename_on_disk_location'
/usr/lib64/ruby/gems/1.8/gems/chef-0.9.12/lib/chef/provider/template.rb:69:in
template_location' /usr/lib64/ruby/gems/1.8/gems/chef-0.9.12/lib/chef/provider/template.rb:39:in action_create'
/usr/lib64/ruby/gems/1.8/gems/chef-0.9.12/lib/chef/resource.rb:395:in send' /usr/lib64/ruby/gems/1.8/gems/chef-0.9.12/lib/chef/resource.rb:395:in run_action'
/usr/lib64/ruby/gems/1.8/gems/chef-0.9.12/lib/chef/runner.rb:53:in
run_action' /usr/lib64/ruby/gems/1.8/gems/chef-0.9.12/lib/chef/runner.rb:89:in converge'
/usr/lib64/ruby/gems/1.8/gems/chef-0.9.12/lib/chef/runner.rb:89:in each' /usr/lib64/ruby/gems/1.8/gems/chef-0.9.12/lib/chef/runner.rb:89:in converge'
/usr/lib64/ruby/gems/1.8/gems/chef-0.9.12/lib/chef/resource_collection.rb:94:in
execute_each_resource' /usr/lib64/ruby/gems/1.8/gems/chef-0.9.12/lib/chef/resource_collection/stepable_iterator.rb:116:in call'
/usr/lib64/ruby/gems/1.8/gems/chef-0.9.12/lib/chef/resource_collection/stepable_iterator.rb:116:in
call_iterator_block' /usr/lib64/ruby/gems/1.8/gems/chef-0.9.12/lib/chef/resource_collection/stepable_iterator.rb:85:in step'
/usr/lib64/ruby/gems/1.8/gems/chef-0.9.12/lib/chef/resource_collection/stepable_iterator.rb:104:in
iterate' /usr/lib64/ruby/gems/1.8/gems/chef-0.9.12/lib/chef/resource_collection/stepable_iterator.rb:55:in each_with_index'
/usr/lib64/ruby/gems/1.8/gems/chef-0.9.12/lib/chef/resource_collection.rb:92:in
execute_each_resource' /usr/lib64/ruby/gems/1.8/gems/chef-0.9.12/lib/chef/runner.rb:84:in converge'
/usr/lib64/ruby/gems/1.8/gems/chef-0.9.12/lib/chef/shef/ext.rb:233:in
run_chef' (irb):1:in irb_binding'

Is cookbook_file expected to work in the context of shef -z?

Beyond that, please accept my apologies for the noise.

On Tue, Dec 28, 2010 at 5:24 PM, Charles Duffy charles@dyfis.net wrote:

I've got several very odd things, likely unrelated, happening in Chef
0.9.12 -- listed here in order of importance:

  • "chef-client" exits without invoking any recipes, whereas "run_chef" in
    shef -z actually tries to run the cookbooks applied to this node.
  • the files/ directory in one of my cookbooks isn't being pushed to the
    cache on my client
  • the shef command cookbooks.all appears to display "nil" for each cookbook

Pointers on any of these would be appreciated.

cookbooks$ find yumrepo/
...
yumrepo/files/default/RPM-GPG-KEY-EPEL
...
cookbooks$ knife cookbook upload -o . yumrepo
INFO: Saving yumrepo
INFO: Validating ruby files
INFO: Validating templates
INFO: Syntax OK
INFO: Generating Metadata
INFO: Uploading files
INFO: Upload complete!

...and over on the client...

chef-client

[Tue, 28 Dec 2010 23:05:51 +0000] INFO: Starting Chef Run (Version 0.9.12)
[Tue, 28 Dec 2010 23:06:09 +0000] INFO: Chef Run complete in 17.457648
seconds
[Tue, 28 Dec 2010 23:06:09 +0000] INFO: cleaning the checksum cache
[Tue, 28 Dec 2010 23:06:09 +0000] INFO: Running report handlers
[Tue, 28 Dec 2010 23:06:09 +0000] INFO: Report handlers complete

shef -z

loading configuration: /etc/chef/client.rb
Session type: solo
Loading........done.

This is shef, the Chef shell.
Chef Version: 0.9.12
Configuration Management System Software - Chef Infra | Chef
http://wiki.opscode.com/display/chef/Home

run help' for help, exit' or ^D to quit.

Ohai2u root@staging.example.com!
chef > cookbooks.all
=> [nil, nil, nil, nil, nil]
chef > run_chef
...and things actually run as they should, until...
[Tue, 28 Dec 2010 23:09:22 +0000] ERROR:
cookbook_file[/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL]
(/var/chef/cache/cookbooks/yumrepo/definitions/yumkey.rb:47:in `from_file')
had an error:
cookbook yumrepo does not contain file files/RPM-GPG-KEY-EPEL

and indeed, /var/chef/cache/cookbooks/yumrepo/ has no files/ directory
whatsoever.

Ohai!

On Wed, Dec 29, 2010 at 8:26 AM, Charles Duffy charles@dyfis.net wrote:

Doing a bit more research --
At least one item was simply a matter of me confusing myself -- my
expectation that cookbook files would be stored in the cache on the client
node appears to have been simply inaccurate. Watching the HTTP PUT by knife
cookbook upload clearly shows the files in question being put, and their
names and signatures are included when the node retrieves its list of
cookbooks. I simply wasn't seeing anything being done with run_client on
account of it being less verbose than shef's default logging level (and
because my run_list was inappropriately constructed, not including the
recipe in which I was inserting extra logging).
It looks like there's only one real issue here, and I'm wondering if it's
simply a limitation of the "solo" mode shef -z runs in:

I think there's possibly two bugs here. One is that even though you
ran shef in "chef-client" mode, it says it's in solo mode. The
offending code is here:

https://github.com/opscode/chef/blob/master/chef/lib/chef/shef/shef_session.rb#L177

This is annoying, but probably unrelated to being unable to use
cookbook files. The root cause of the cookbook file bug is that shef
is not configuring the cookbook fetching system to work as a
chef-client so it doesn't know if it should be getting cookbook files
and templates from disk (like chef-solo) or from the remote server
(like chef-client).

I'll create tickets for these.

Dan

Dan --

Thank you for those extraordinarily detailed bug reports, which were
sufficient for me (as a non-rubyist) to patch my shef into a functional
state (in one case with a minor tweak, "@rest" rather than "rest").

For anyone else interested --
http://tickets.opscode.com/browse/CHEF-1954
http://tickets.opscode.com/browse/CHEF-1955
http://tickets.opscode.com/browse/CHEF-1955
On Wed, Dec 29, 2010 at 5:37 PM, Daniel DeLeo dan@kallistec.com wrote:

Ohai!

On Wed, Dec 29, 2010 at 8:26 AM, Charles Duffy charles@dyfis.net wrote:

Doing a bit more research --
At least one item was simply a matter of me confusing myself -- my
expectation that cookbook files would be stored in the cache on the
client
node appears to have been simply inaccurate. Watching the HTTP PUT by
knife
cookbook upload clearly shows the files in question being put, and their
names and signatures are included when the node retrieves its list of
cookbooks. I simply wasn't seeing anything being done with run_client on
account of it being less verbose than shef's default logging level (and
because my run_list was inappropriately constructed, not including the
recipe in which I was inserting extra logging).
It looks like there's only one real issue here, and I'm wondering if it's
simply a limitation of the "solo" mode shef -z runs in:

I think there's possibly two bugs here. One is that even though you
ran shef in "chef-client" mode, it says it's in solo mode. The
offending code is here:

https://github.com/opscode/chef/blob/master/chef/lib/chef/shef/shef_session.rb#L177

This is annoying, but probably unrelated to being unable to use
cookbook files. The root cause of the cookbook file bug is that shef
is not configuring the cookbook fetching system to work as a
chef-client so it doesn't know if it should be getting cookbook files
and templates from disk (like chef-solo) or from the remote server
(like chef-client).

I'll create tickets for these.

Dan

...well, damnit --

I went back and tried to reproduce this fix, but it looks like I'm hitting a
different failure mode now -- I'm getting a FileNotFound thrown from
preferred_manifest_record, which is called shortly before file_vendor is
referenced.

Like the prior failure, this happens only with 'shef -z' but not using
run_client:

Chef::Exceptions::FileNotFound: cookbook loggly does not contain file
files/loggly.com.crt
from
/usr/lib/ruby/gems/1.8/gems/chef-0.9.12/lib/chef/cookbook_version.rb:527:in
preferred_manifest_record' from /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/lib/chef/cookbook_version.rb:532:in preferred_filename_on_disk_location'
from
/usr/lib/ruby/gems/1.8/gems/chef-0.9.12/lib/chef/provider/cookbook_file.rb:63:in
file_cache_location' from /usr/lib/ruby/gems/1.8/gems/chef-0.9.12/lib/chef/provider/cookbook_file.rb:36:in action_create'

On Wed, Dec 29, 2010 at 6:33 PM, Charles Duffy charles@dyfis.net wrote:

Dan --

Thank you for those extraordinarily detailed bug reports, which were
sufficient for me (as a non-rubyist) to patch my shef into a functional
state (in one case with a minor tweak, "@rest" rather than "rest").

For anyone else interested --
http://tickets.opscode.com/browse/CHEF-1954
http://tickets.opscode.com/browse/CHEF-1955
http://tickets.opscode.com/browse/CHEF-1955
On Wed, Dec 29, 2010 at 5:37 PM, Daniel DeLeo dan@kallistec.com wrote:

Ohai!

On Wed, Dec 29, 2010 at 8:26 AM, Charles Duffy charles@dyfis.net wrote:

Doing a bit more research --
At least one item was simply a matter of me confusing myself -- my
expectation that cookbook files would be stored in the cache on the
client
node appears to have been simply inaccurate. Watching the HTTP PUT by
knife
cookbook upload clearly shows the files in question being put, and their
names and signatures are included when the node retrieves its list of
cookbooks. I simply wasn't seeing anything being done with run_client on
account of it being less verbose than shef's default logging level (and
because my run_list was inappropriately constructed, not including the
recipe in which I was inserting extra logging).
It looks like there's only one real issue here, and I'm wondering if
it's
simply a limitation of the "solo" mode shef -z runs in:

I think there's possibly two bugs here. One is that even though you
ran shef in "chef-client" mode, it says it's in solo mode. The
offending code is here:

https://github.com/opscode/chef/blob/master/chef/lib/chef/shef/shef_session.rb#L177

This is annoying, but probably unrelated to being unable to use
cookbook files. The root cause of the cookbook file bug is that shef
is not configuring the cookbook fetching system to work as a
chef-client so it doesn't know if it should be getting cookbook files
and templates from disk (like chef-solo) or from the remote server
(like chef-client).

I'll create tickets for these.

Dan