Chef Client Run Error without details of the Error

Hi I have a successful chef-client run on my laptop using kitchen converge by using policyfile method. However when I used the chef-client run on my staging environment using the same policyfile it returned error and the error message didn’t say clearly what is the wrong with my policyfile or cookbook.

I have uploaded the error message and could anyone please tell me how to get the root cause of the error?

Starting Chef Client, version 14.2.0

Synchronizing Cookbooks:
  - auditd (2.2.0)
  - chef-sugar (4.0.0)
  - chef-client (10.0.5)
  - compat_resource (12.19.1)
  - cron (6.1.1)
  - firewall (2.6.3)
  - gs_base (0.1.0)
  - gs_crl_host (0.1.0)
  - hostsfile (3.0.1)
  - iptables (4.3.4)
  - line (2.0.1)
  - logrotate (2.2.0)
  - ntp (3.5.6)
  - ohai (5.2.3)
  - openssh (2.6.3)
  - os-hardening (3.0.0)
  - poise-hoist (1.2.1)
  - rsyslog (6.0.4)
  - sudo (5.3.3)
  - ssh-hardening (2.3.1)
  - sysctl (0.9.0)
  - users (5.3.1)
  - windows (4.3.2)
Installing Cookbook Gems:

Running handlers:
[2018-06-21T17:17:55+08:00] ERROR: Running exception handlers
[2018-06-21T17:17:55+08:00] ERROR: Running exception handlers
Running handlers complete
[2018-06-21T17:17:55+08:00] ERROR: Exception handlers complete
[2018-06-21T17:17:55+08:00] ERROR: Exception handlers complete
Chef Client failed. 0 resources updated in 07 seconds
[2018-06-21T17:17:55+08:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2018-06-21T17:17:55+08:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2018-06-21T17:17:55+08:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2018-06-21T17:17:55+08:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2018-06-21T17:17:55+08:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '17'
---- Begin output of bundle install ----
STDOUT: Don't run Bundler as root. Bundler can ask for sudo if it is needed, and
installing your bundle as root will break this application for all non-root
users on this machine.
Fetching source index from https://www.rubygems.org/

Retrying fetcher due to error (2/4): Bundler::HTTPError Could not fetch specs from https://www.rubygems.org/
Retrying fetcher due to error (3/4): Bundler::HTTPError Could not fetch specs from https://www.rubygems.org/
Retrying fetcher due to error (4/4): Bundler::HTTPError Could not fetch specs from https://www.rubygems.org/
Could not fetch specs from https://www.rubygems.org/
STDERR:
---- End output of bundle install ----
Ran bundle install returned 17
[2018-06-21T17:17:55+08:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '17'
---- Begin output of bundle install ----
STDOUT: Don't run Bundler as root. Bundler can ask for sudo if it is needed, and
installing your bundle as root will break this application for all non-root
users on this machine.
Fetching source index from https://www.rubygems.org/

Retrying fetcher due to error (2/4): Bundler::HTTPError Could not fetch specs from https://www.rubygems.org/
Retrying fetcher due to error (3/4): Bundler::HTTPError Could not fetch specs from https://www.rubygems.org/
Retrying fetcher due to error (4/4): Bundler::HTTPError Could not fetch specs from https://www.rubygems.org/
Could not fetch specs from https://www.rubygems.org/
STDERR:
---- End output of bundle install ----
Ran bundle install returned 17
(END)

The same policyfile runs fine on my laptop without error.

The content of the stacktrace.out:

cat /var/chef/cache/chef-stacktrace.out
Generated at 2018-06-21 17:17:55 +0800
Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '17'
---- Begin output of bundle install ----
STDOUT: Don't run Bundler as root. Bundler can ask for sudo if it is needed, and
installing your bundle as root will break this application for all non-root
users on this machine.
Fetching source index from https://www.rubygems.org/

Retrying fetcher due to error (2/4): Bundler::HTTPError Could not fetch specs from https://www.rubygems.org/
Retrying fetcher due to error (3/4): Bundler::HTTPError Could not fetch specs from https://www.rubygems.org/
Retrying fetcher due to error (4/4): Bundler::HTTPError Could not fetch specs from https://www.rubygems.org/
Could not fetch specs from https://www.rubygems.org/
STDERR:
---- End output of bundle install ----
Ran bundle install returned 17
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/mixlib-shellout-2.3.2/lib/mixlib/shellout.rb:293:in `invalid!'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/mixlib-shellout-2.3.2/lib/mixlib/shellout.rb:280:in `error!'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.2.0/lib/chef/mixin/shell_out.rb:124:in `shell_out!'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.2.0/lib/chef/cookbook/gem_installer.rb:62:in `block (2 levels) in install'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.2.0/lib/chef/cookbook/gem_installer.rb:52:in `open'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.2.0/lib/chef/cookbook/gem_installer.rb:52:in `block in install'
/opt/chef/embedded/lib/ruby/2.5.0/tmpdir.rb:89:in `mktmpdir'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.2.0/lib/chef/cookbook/gem_installer.rb:51:in `install'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.2.0/lib/chef/cookbook/cookbook_collection.rb:60:in `install_gems'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.2.0/lib/chef/policy_builder/policyfile.rb:187:in `setup_run_context'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.2.0/lib/chef/client.rb:512:in `setup_run_context'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.2.0/lib/chef/client.rb:278:in `run'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.2.0/lib/chef/application.rb:303:in `run_with_graceful_exit_option'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.2.0/lib/chef/application.rb:279:in `block in run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.2.0/lib/chef/local_mode.rb:44:in `with_server_connectivity'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.2.0/lib/chef/application.rb:261:in `run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.2.0/lib/chef/application/client.rb:441:in `run_application'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.2.0/lib/chef/application.rb:66:in `run'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.2.0/bin/chef-client:26:in `<top (required)>'
/bin/chef-client:75:in `load'
/bin/chef-client:75:in `<main>'

It seems that the server is trying to install gems from rubygems.org but does not have an active internet connection. Do you happen do be airgapped? If so you will need to either configure a proxy to connect to rubygems.org or configure chef to use a local repo: https://docs.chef.io/install_chef_air_gap.html

Yes, my staging environment don’t have internet access but my chef server has internet and it will manage those VMs which don’t have internet.

So there is no way for client which have NO internet access to install cookbooks? However I have other similar servers which cannot access the internet that they can run chef-client successfully without problem. (of course they are using different cookbooks)

Or is it because my specific recipe requires the client server to install gems from the internet? If YES, how to identify such code? Thanks.

@lovelyBlue You cannot install gem from rubygems.org , it does require active internet connection. Alternatively you can download the gem package into cookbook’s file dir and install it from local.

If you are managing many gems and can afford to setup gem server, which is definitely a best path to go. You can deploy local gem server within your vpc like your chef server and deploy it from there.

@suthir, just curious to know how to find out which gem package I need to download by looking at my cookbook? All those downloading process are obscured from the chef-client run, isn’t it?

Or I can run it again in my local laptop and capture the chef-client run process and determine which ruby gem I can install?

@lovelyBlue One of the cookbooks listed at the top of the output is using the gem dependency feature (i.e. has gem 'something' in the metadata.rb file). It’s probably chef-sugar, though others might too. If you want to use that cookbook, you’ll need to provide access to a gem server via the rubygems_url config option, though it can be an internal mirror/proxy/whatever. Otherwise you cannot use that community cookbook or anything that depends on it.

@lovelyBlue If you’re referring any community cookbook, let me know i can find out. If not, its up the way the cookbook have been designed.

Here are the way you can find out.

  1. If they use bundler to manage all the gem, your job is pretty simple. You may find a file called Gemfile. If you run bundle show from the directory where Gemfile resides, it will list you all the gems.
  2. You can search with the keyword gem_package or chef_gem and find out the resource in community cookbook and see what are the gems they are passing in to the recipe.

Hope this helps!

@suthir Neither of those are related at all. The “cookbook gems” phase is specifically related to the gem dependency feature.

@lovelyBlue As @coderanger mentioned, Please check your metadata.rb and find the listed gems.

@suthir and @coderanger, YES I found the ONLY gem dependency in my cookbooks.

It is in the cookbook “line” Gemfile

source 'https://rubygems.org'

gem 'danger'

I am thinking is it possible to install the gem via other means instead of setup local gem repo or local chef supermarket?

Is it possible to include the gem package in my cookbook so the chef client can download and install the gem package from chef server via the cookbook recipe?

Again, the Gemfile has nothing to do with anything, @suthir was incorrect. That error is related to cookbooks with a gem line in their metadata. chef-sugar definitely does, and some of your internal cookbooks might use this feature also. You can use the config option rubygems_url to point the gem installer system at a local Rubygems mirror, or you can stop using chef-sugar.

@lovelyBlue Yes, it is possible, you can keep the gem package in your cookbook’s file dir and use your recipe to download and install gem_package locally.

Thinking aloud, you can also install gem package directly from git repo since you have Gemfile. You can update your Gemfile with the following changes.

gem 'danger',  '5.6.2', :git => 'https://github.com/danger/danger.git'

Here you can use your local git repo. That should work.

Again, no, that has nothing to do with the error. As I said, the solution is to either use a local Rubygems mirror or to not use the gem dependencies feature. Those are the only two options at this time.

Since I have no other options as @coderanger says, I have to setup a local rubygem repo on my chef server and point my knife bootstrap template to this local repo and it worked.

Thanks for the help