Running OpsWorks with chef using CloudFormation gives error , custom cookbooks not found in cache

Hi Guys,

I am getting below error only when I try to run my custom cookbooks , I am using chef_zero as a provisioner to run the custom cookbooks.

My Question is by default which cache is being used by chef_zero ?

:/var/chef/runs/6f707f6d-7be8-48c7-9670-95062536b4c2/local-mode-cache/cache/cookbooks/chef-splunk/recipes/client.rb:46:in rescue in from_file' : /var/chef/runs/6f707f6d-7be8-48c7-9670-95062536b4c2/local-mode-cache/cache/cookbooks/chef-splunk/recipes/service.rb:103:infrom_file’ (CHEF-3694)/var/chef/runs/6f707f6d-7be8-48c7-9670-95062536b4c2/local-mode-cache/cache/cookbooks/chef-splunk/recipes/service.rb:103:in from_file'. Please see https://docs.chef.io/deprecations_resource_cloning.html for further details and information on how to correct this problem. at /opt/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.18.31/lib/chef/event_dispatch/dispatcher.rb:43:incall’
[2017-11-20T15:41:58+00:00] INFO: HTTP Request Returned 404 Not Found: Object not found: chefzero://localhost:8889/data/vault/splunk__default

================================================================================
Recipe Compile Error in /var/chef/runs/6f707f6d-7be8-48c7-9670-95062536b4c2/local-mode-cache/cache/cookbooks/chef-splunk/recipes/default.rb

Net::HTTPServerException
------------------------
404 “Not Found”

Cookbook Trace:

/var/chef/runs/6f707f6d-7be8-48c7-9670-95062536b4c2/local-mode-cache/cache/cookbooks/chef-vault/libraries/helpers.rb:38:in chef_vault_item' /var/chef/runs/6f707f6d-7be8-48c7-9670-95062536b4c2/local-mode-cache/cache/cookbooks/chef-splunk/recipes/setup_auth.rb:21:infrom_file’
/var/chef/runs/6f707f6d-7be8-48c7-9670-95062536b4c2/local-mode-cache/cache/cookbooks/chef-splunk/recipes/client.rb:81:in from_file' /var/chef/runs/6f707f6d-7be8-48c7-9670-95062536b4c2/local-mode-cache/cache/cookbooks/chef-splunk/recipes/default.rb:29:infrom_file’

Relevant File Content:

/var/chef/runs/6f707f6d-7be8-48c7-9670-95062536b4c2/local-mode-cache/cache/cookbooks/chef-vault/libraries/helpers.rb:

31: # +node[‘chef-vault’][‘databag_fallback’]+.
32: # @example
33: # item = chef_vault_item(‘secrets’, ‘bacon’)
34: # log ‘Yeah buddy!’ if item[’_default’][‘type’]
35: # @param [String] bag Name of the data bag to load from.
36: # @param [String] id Identifier of the data bag item to load.
37: def chef_vault_item(bag, id)
38>> if ChefVault::Item.vault?(bag, id)
39: ChefVault::Item.load(bag, id)
40: elsif node[‘chef-vault’][‘databag_fallback’]
41: data_bag_item(bag, id)
42: else
43: raise "Trying to load a regular data bag item #{id} from #{bag}, and databag_fallback is disabled"
44: end
45: end
46:
47: # Helper method which provides an environment wrapper for a data bag.

Platform:

x86_64-linux

[2017-11-20T15:41:58+00:00] ERROR: Running exception handlers
[2017-11-20T15:41:58+00:00] ERROR: Exception handlers complete
[2017-11-20T15:41:58+00:00] FATAL: Stacktrace dumped to /var/chef/runs/6f707f6d-7be8-48c7-9670-95062536b4c2/local-mode-cache/cache/chef-stacktrace.out
[2017-11-20T15:41:58+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2017-11-20T15:41:58+00:00] ERROR: 404 “Not Found”
[2017-11-20T15:41:58+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

@thommay posted the same here !

I figured that this is AWS OpsWorks not supporting the encrypted data bag item so my data bag couldn't be extracted and as a result went to exception and 404.

Just so it help , I have been able to pass through after creating a new custom cookbook which setup the data bags as a first recipe and then run splunk to configure and use the existing node data bags.

Note

AWS OpsWorks Stacks does not support encrypted data bags. To store sensitive data in encrypted form, such as passwords or certificates, we recommend storing it in a private S3 bucket. You can then create a custom recipe that uses the Amazon SDK for Ruby to retrieve the data. For an example, see Using the SDK for Ruby.

Yes, you need to a wrapper cookbook for Splunk including a default data bag, similar to this:

Simply update data_bags/vault/splunk__default.json per your environment and use that in local mode.

You can setup the data bag dynamically especially if your secrets are managed by a separate mechanism other than Chef vault. As mentioned above, you can use Amazon SDK to retrieve these secrets from say AWS KMS or S3 bucket. Here's another example passing down credentials done by Azure Quickstart template for Splunk.

With that said, there's room for improvement, especially if there's a more AWS native way to achieve that in the case of AWS OpsWorks.

Thanks @rarsan , This was helpful !

@rarsan it looks like I might have missed something here , when you said

Simply update data_bags/vault/splunk__default.json per your environment and use that in local mode.

Did you mean that I can just change splunk_{my environment}.json and pass this json directly to chef-client in local mode to get this work without needing to have custom cookbook ?

That’s correct.
I have not tested it with OpsWorks, but this generally works.
As always, you’d want to make sure those credentials don’t leak e.g. locking the machine during bootstrap, deleting data bag file after chef-client runs, etc.