Insufficient Permission attempting to create a directory /var/opt/delivery/workspace/172.29.1.50/test/Development/deliver-customers-rhel/master/acceptance/provision/chef/keys


#1

I’m following along the chef-delivery tutorial and I am stuck on brining up the machine during the provisioning phase. (its about halfway down the page here: https://learn.chef.io/build-a-delivery-pipeline/aws/write-the-build-cookbook/)

The specific error I’m getting through the chef-delivery gui shows:

`

Error executing action 'setup' on resource 'machine[acceptance-deliver-customers-rhel]'
================================================================================

Chef::Exceptions::InsufficientPermissions
-----------------------------------------
aws_key_pair[chef_default] (basic_chef_client::block line 1126) had an error: Chef::Exceptions::InsufficientPermissions: directory[/var/opt/delivery/workspace/172.29.1.50/test/Development/deliver-customers-rhel/master/acceptance/provision/chef/keys] (basic_chef_client::block line 30) had an error: Chef::Exceptions::InsufficientPermissions: Cannot create directory[/var/opt/delivery/workspace/172.29.1.50/test/Development/deliver-customers-rhel/master/acceptance/provision/chef/keys] at /var/opt/delivery/workspace/172.29.1.50/test/Development/deliver-customers-rhel/master/acceptance/provision/chef/keys due to insufficient permissions

Resource Declaration:
---------------------
# In /var/opt/delivery/workspace/172.29.1.50/test/Development/deliver-customers-rhel/master/acceptance/provision/cache/cookbooks/build-cookbook/recipes/provision.rb

 57: machine "#{stage}-#{project}" do
 58:   action [:setup]
 59:   chef_environment delivery_environment
 60:   converge false
 61:   files '/etc/chef/encrypted_data_bag_secret' => File.join(database_passwords_key_path, 'database_passwords_key')
 62:   run_list node[project]['run_list']
 63:   machine_options Cheffish::MergedConfig.new({"bootstrap_options" => {"key_name" => ssh_key['name'], "key_path" => ssh_private_key_path}}, node[project][stage]['aws']['config']['machine_options'])
 64: end

`

What appears to be the issue is that the build node is trying to write to the directory (/var/opt/delivery/workspace/172.29.1.50/test/Development/deliver-customers-rhel/master/acceptance/provision/chef/keys) but can’t for some reason. I don’t understand why not because it created the directory and subdirectories. Also it’s happening on the machine resource which somewhat baffles me. The stack trace is at the bottom of the document.

So I guess my question is how would I further troubleshoot the issue to resolve? Is it possible to run the chef-client run on the build node? What exactly is going on - is the chef-client run on the build node trying to download the database key and attempting to save it to a directory its creating and NOT have permission to do that? If so what’s a good way to troubleshoot? If not please explain.

I’ve tried removing the files attribute from the machine but the error still persisted. I’ve also tried deleting the build node’s provision directory - (/var/opt/delivery/workspace/172.29.1.50/test/Development/deliver-customers-rhel/master/acceptance/provision) no luck. What could I try next?


stack trace:

root@build-node1:~# cat /var/opt/delivery/workspace/172.29.1.50/test/Development/deliver-customers-rhel/master/acceptance/provision/cache/chef-stacktrace.out
Generated at 2016-01-15 22:28:13 +0000
Chef::Exceptions::InsufficientPermissions: machine[acceptance-deliver-customers-rhel] (build-cookbook::provision line 107) had an error: Chef::Exceptions::InsufficientPermissions: aws_key_pair[chef_default] (basic_chef_client::block line 1126) had an error: Chef::Exceptions::InsufficientPermissions: directory[/var/opt/delivery/workspace/172.29.1.50/test/Development/deliver-customers-rhel/master/acceptance/provision/chef/keys] (basic_chef_client::block line 30) had an error: Chef::Exceptions::InsufficientPermissions: Cannot create directory[/var/opt/delivery/workspace/172.29.1.50/test/Development/deliver-customers-rhel/master/acceptance/provision/chef/keys] at /var/opt/delivery/workspace/172.29.1.50/test/Development/deliver-customers-rhel/master/acceptance/provision/chef/keys due to insufficient permissions
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/mixin/why_run.rb:241:in run' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/mixin/why_run.rb:322:inblock in run’
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/mixin/why_run.rb:321:in each' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/mixin/why_run.rb:321:inrun’
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/provider.rb:154:in process_resource_requirements' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/provider.rb:132:inrun_action’
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource.rb:585:in run_action' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/runner.rb:49:inrun_action’
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/runner.rb:81:in block (2 levels) in converge' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/runner.rb:81:ineach’
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/runner.rb:81:in block in converge' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource_collection/resource_list.rb:83:inblock in execute_each_resource’
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource_collection/stepable_iterator.rb:116:in call' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource_collection/stepable_iterator.rb:116:incall_iterator_block’
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource_collection/stepable_iterator.rb:85:in step' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource_collection/stepable_iterator.rb:104:initerate’
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource_collection/stepable_iterator.rb:55:in each_with_index' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource_collection/resource_list.rb:81:inexecute_each_resource’
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/runner.rb:80:in converge' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/cheffish-1.6.0/lib/cheffish/basic_chef_client.rb:69:inblock in converge’
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/cheffish-1.6.0/lib/cheffish/basic_chef_client.rb:133:in call' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/cheffish-1.6.0/lib/cheffish/basic_chef_client.rb:133:inwith_chef_config’
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/cheffish-1.6.0/lib/cheffish/basic_chef_client.rb:68:in converge' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/cheffish-1.6.0/lib/cheffish/basic_chef_client.rb:102:ininline_resource’
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/cheffish-1.6.0/lib/cheffish.rb:5:in inline_resource' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-provisioning-aws-1.6.1/lib/chef/provider/aws_key_pair.rb:29:increate_key’
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-provisioning-aws-1.6.1/lib/chef/provider/aws_key_pair.rb:10:in block in <class:AwsKeyPair>' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/provider.rb:362:inaction_create’
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/provider.rb:144:in run_action' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource.rb:585:inrun_action’
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/runner.rb:49:in run_action' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/runner.rb:81:inblock (2 levels) in converge’
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/runner.rb:81:in each' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/runner.rb:81:inblock in converge’
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource_collection/resource_list.rb:83:in block in execute_each_resource' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource_collection/stepable_iterator.rb:116:incall’
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource_collection/stepable_iterator.rb:116:in call_iterator_block' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource_collection/stepable_iterator.rb:85:instep’
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource_collection/stepable_iterator.rb:104:in iterate' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource_collection/stepable_iterator.rb:55:ineach_with_index’
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource_collection/resource_list.rb:81:in execute_each_resource' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/runner.rb:80:inconverge’
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/cheffish-1.6.0/lib/cheffish/basic_chef_client.rb:69:in block in converge' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/cheffish-1.6.0/lib/cheffish/basic_chef_client.rb:133:incall’
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/cheffish-1.6.0/lib/cheffish/basic_chef_client.rb:133:in with_chef_config' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/cheffish-1.6.0/lib/cheffish/basic_chef_client.rb:68:inconverge’
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/cheffish-1.6.0/lib/cheffish/basic_chef_client.rb:109:in converge_block' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-provisioning-1.5.0/lib/chef/provisioning.rb:12:ininline_resource’
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-provisioning-aws-1.6.1/lib/chef/provisioning/aws_driver/driver.rb:1125:in block in default_aws_keypair' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-provisioning-aws-1.6.1/lib/chef/provisioning/aws_driver/driver.rb:1124:insynchronize’
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-provisioning-aws-1.6.1/lib/chef/provisioning/aws_driver/driver.rb:1124:in default_aws_keypair' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-provisioning-aws-1.6.1/lib/chef/provisioning/aws_driver/driver.rb:755:inbootstrap_options_for’
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-provisioning-aws-1.6.1/lib/chef/provisioning/aws_driver/driver.rb:551:in allocate_machine' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-provisioning-1.5.0/lib/chef/provider/machine.rb:31:inblock in class:Machine
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/provider.rb:362:in action_allocate' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-provisioning-1.5.0/lib/chef/provider/machine.rb:38:inblock in class:Machine
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/provider.rb:362:in action_ready' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-provisioning-1.5.0/lib/chef/provider/machine.rb:45:inblock in class:Machine
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/provider.rb:362:in action_setup' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/provider.rb:144:inrun_action’
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource.rb:585:in run_action' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/runner.rb:49:inrun_action’
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/runner.rb:81:in block (2 levels) in converge' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/runner.rb:81:ineach’
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/runner.rb:81:in block in converge' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource_collection/resource_list.rb:83:inblock in execute_each_resource’
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource_collection/stepable_iterator.rb:116:in call' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource_collection/stepable_iterator.rb:116:incall_iterator_block’
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource_collection/stepable_iterator.rb:85:in step' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource_collection/stepable_iterator.rb:104:initerate’
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource_collection/stepable_iterator.rb:55:in each_with_index' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource_collection/resource_list.rb:81:inexecute_each_resource’
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/runner.rb:80:in converge' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/client.rb:653:inblock in converge’
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/client.rb:648:in catch' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/client.rb:648:inconverge’
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/client.rb:687:in converge_and_save' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/client.rb:269:inrun’
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application.rb:270:in block in fork_chef_client' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application.rb:258:infork’
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application.rb:258:in fork_chef_client' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application.rb:224:inblock in run_chef_client’
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/local_mode.rb:44:in with_server_connectivity' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application.rb:212:inrun_chef_client’
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application/client.rb:408:in block in interval_run_chef_client' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application/client.rb:398:inloop’
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application/client.rb:398:in interval_run_chef_client' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application/client.rb:388:inrun_application’
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application.rb:60:in run' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/bin/chef-client:26:in<top (required)>’
/opt/chefdk/bin/chef-client:54:in load' /opt/chefdk/bin/chef-client:54:in


#2

Hi J Ramos,

Sorry for the late response, we have been working on the tutorial and we actually took it down for a bit while doing some awesome work with it. If you would like to continue with it you can find a PDF doc at the bottom.

About the problem you are facing, I think the FS might have a permission issue somewhere on the path so, I would like you to verify the permissions of:

  • /var
  • /var/opt
  • And if it is possible the rest of the three.

I also want you to show me the umask you have configured on the system. For both root and the dbuild user.

Thank you!

Chef Delivery Tutorial - Public Link
https://s3.amazonaws.com/delivery-packages/Chef+Delivery+tutorial.pdf