[RESOLVED] Chef-client crashing with "NoMethodError: undefined method `each' for nil:NilClass"

Hi, we’ve been using Chef successfully for quite a while but yesterday started having a problem where any chef-client run crashes with:

Error Syncing Cookbooks:

Unexpected Error:
NoMethodError: undefined method `each' for nil:NilClass

The crash site is in synchronizer.rb at line 110:

    def files
      @files ||= cookbooks.inject([]) do |memo, cookbook|
        @eager_segments.each do |segment|
          cookbook.manifest[segment].each do |manifest_record|          # Crash
            memo << CookbookFile.new(cookbook, segment, manifest_record)

For some (but not all) cookbooks, the server always sends a [] as the value of an empty key, but for other cookbooks it doesn’t add the key to the response object (Chef::Cookbook::Metadata) so cookbook.manifest[segment] is nil and cookbook.manifest[segment].each crashes.

It seems strange that this started happening yesterday afternoon. We’ve been using chef-client 12.18.31 for a while and didn’t change it yesterday. Our chef server reports as 12.2.0 and we didn’t explicitly make any changes to it, either.

We’d be grateful for any suggestions as to how to get out of this pickle.

This looks like the error that caused @coderanger to send his email yesterday about cookbooks that had been uploaded using Chef Client 13 (https://github.com/chef/chef/issues/6044).
Have any of the systems that upload cookbooks to your Chef Server been updated to run 13? The solution is to delete any cookbooks that were uploaded that way and then reupload them using Chef Client 12.

Thanks for the speedy and helpful response! Looks like that is indeed my problem. Now to figure out how the version 13 client got installed on my CI server…