Chef-zero OR chef-client -z failing with --recipe-url pointing to a tar.gz file

Chef Infra (archive)
1

Does anyone have any suggestions on how I can solve this issue?

Using chef-client -z --recipe-url <url_with_tar.gz> causes a recipes.tgz file to be created that is not compressed, then it fails to be able to use the recipes.tgz file. I am using a Linux 6.7 OS with gnu tar which also fails a tar -zxf /root/recipes.tgz as it expects a .tgz file to be compressed.

Chef Version

Chef: 12.5.1

Platform Version

2.6.32-573.18.1.el6.x86_64

Replication

chef-client -z --no-color \
  --logfile /tmp/chef-client.log \
  --format doc \
  --log_level debug \
  --override-runlist "recipe[my_cookbook::my_recipe]" \
  --recipe-url http://my.company.com/my_cookbook.tar.gz \
  --json-attributes /tmp/my.json

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/mixlib-shellout-2.2.1/lib/mixlib/shellout.rb:289:in `invalid!': Expected process to exit with [0], but received '2' (Mixlib::ShellOut::ShellCommandFailed)
---- Begin output of tar zxvf /root/recipes.tgz -C /root ----
STDOUT:
STDERR: gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
---- End output of tar zxvf /root/recipes.tgz -C /root ----
Ran tar zxvf /root/recipes.tgz -C /root returned 2
        from /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/mixlib-shellout-2.2.1/lib/mixlib/shellout.rb:276:in `error!'
        from /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/mixin/shell_out.rb:56:in `shell_out!'
        from /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application/client.rb:305:in `reconfigure'
        from /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application.rb:58:in `run'
        from /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/bin/chef-client:26:in `<top (required)>'
        from /usr/bin/chef-client:54:in `load'
        from /usr/bin/chef-client:54:in `<main>'
2

It is the z flag in your command line which is saying the file should be gzipped.

How is created your recipe.tgz file ? can you give the output of command: file /root/recipes.tgz ?

3

The -z in the command line is for chef-client to run as chef zero. It is the --recipe-url which is pulling down my cookbook archive (which is in fact gzipped) and uncompressing it at the same time as creating the recipes.tgz. This results in the recipes.tgz file being created as uncompressed.

file /root/recipes.tgz
/root/recipes.tgz: POSIX tar archive (GNU)
4

I was talking about your tar command line example, it is the z option which tells tar the file is gzipped, nothing to do with the file extension.

I don't get why the download would unzip the file, the code is here and it just copy the file without any transformation called here.

So I highly suspect your orignal file is not gzipped but only a tar ball. Can you give some insight on how your original file is created ?

5

I just checked again and indeed the archive is gzipped. It is created with a tar czvf command.

$ curl -o /tmp/my_cookbook.tar.gz http://my_company.com/my_cookbook.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  526k  100  526k    0     0  2916k      0 --:--:-- --:--:-- --:--:-- 3228k

$ file /tmp/my_cookbook.tar.gz
/tmp/my_cookbook.tar.gz: gzip compressed data, from Unix, last modified: Wed Jul 13 17:59:29 2016

Also, I am able to replicate the code using irb and get the same results:

$ irb
2.1.0 :001 > require 'open-uri'
 => true
2.1.0 :002 > url = 'http://my_company.com/my_cookbook.tar.gz'
 => "http://my_company.com/my_cookbook.tar.gz"
2.1.0 :003 > File.open('/tmp/sample.tgz', "wb") do |f|
2.1.0 :004 >     open(url) do |r|
2.1.0 :005 >       f.write(r.read)
2.1.0 :006?>     end
2.1.0 :007?>   end
 => 2744320
2.1.0 :008 > exit

$ file /tmp/sample.tgz
/tmp/sample.tgz: POSIX tar archive (GNU)

$ tar -ztvf /tmp/sample.tgz

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
6

This may be due to our internal apache server not configured correctly to handle a mime type of application/x-gzip. I’m investigating that angle with our operations team. See the content-type below, looks incorrect.

2.1.0 :007 > page = open(url); page.meta
 => {"date"=>"Thu, 14 Jul 2016 00:35:02 GMT", "server"=>"Apache", "last-modified"=>"Wed, 13 Jul 2016 23:05:27 GMT", "etag"=>"\"1019///my_cookbook.tar.gz\"", "cache-control"=>"max-age=604800", "accept-ranges"=>"bytes", "content-length"=>"538930", "content-type"=>"application/octet-stream"}
7

I configured an apache instance we have for the application/x-gzip mime type. The content-type is now returning application/x-gzip but the results are the same. If I add "Accept-Encoding" => "gzip" to the open, then it produces the file correctly. Do you know of another setting on the server side that would force this?

File.open(out, "wb") do |f|
  open(url, "Accept-Encoding" => "gzip") do |r|
    f.write(r.read)
  end
end
8

I found that the apache configuration also needed not to configure AddEncoding x-gzip .gz .tgz. Once this was removed, the code worked. So to summarize, these were my apache configuration settings:

#AddEncoding x-gzip .gz .tgz
AddType application/x-gzip .gz
9

It's hard to tell without any insight on your apache server configuratio nor it's distribution, etc.

Related Thread on stackOverflow.

I assume your have some kind of configuration around mod_deflate, but I can't tell what without seing the conf.