We have a simple little pipeline of three jenkins jobs that handles 1) uploading the cookbooks to the supermarket; 2) building policy archives from our policy files; 3) pushing policies by version to specific policy groups. This is working swimmingly across probably 25 repos across our company, but one if failing (now consistently, but previously sporadically) on the “chef push-archive” call. The back trace seems to be showing an issue with the configuration path which we are explicitly setting with -c. What is really throwing me is the exact same script succeeds with a different set of policies when running on the exact same build agent. nothing about this seems policy specific though. The failure and success cases are both included below.
Can anyone shed any light on this for me?
Environment info:
- Jenkins 1.651.3 Master
- Windows Server 2012 R2 jenkins agent
- Chef Version:
C:\Program Files\OpenSSH\home\IOSBuild>chef --version
Chef Development Kit Version: 1.3.43
chef-client version: 12.19.36
delivery version: master (dd319aa632c2f550c92a2172b9d1226478fea997)
berks version: 5.6.4
kitchen version: 1.16.0
Failure:
* executing advanced policy file push to specific chef server(s) * downloading policies for version latest of MnMElasticsearchPolicy... * download complete * exploding D:/Jenkins/workspace/18.1.0.0.0-ACPx-MnMElasticsearchPolicy-DEPLOY-NP/_downloads/latest/Policies.zip... * explosion complete * making custom config file in from C:/Program Files/OpenSSH/home/IOSBuild/.chef/knife.rb in order to push to [https://chef.csgidev.com/organizations/csg](https://chef.csgidev.com/organizations/csg) * starting policy archive push from D:/Jenkins/workspace/18.1.0.0.0-ACPx-MnMElasticsearchPolicy-DEPLOY-NP/_downloads/latest... ** pushing policy archive D:/Jenkins/workspace/18.1.0.0.0-ACPx-MnMElasticsearchPolicy-DEPLOY-NP/_downloads/latest/mnm_es_1_7_5_all/mnm_es_1_7_5_all-116a3e3cdd22d72358474c782626b9e57411beb839f525b2202375a218bf25c9.tgz to mnm-onprem-es-aiqa using config D:/Jenkins/workspace/18.1.0.0.0-ACPx-MnMElasticsearchPolicy-DEPLOY-NP/_downloads/latest/chef.csgidev.com.rb ** executing chef push-archive -c 'D:/Jenkins/workspace/18.1.0.0.0-ACPx-MnMElasticsearchPolicy-DEPLOY-NP/_downloads/latest/chef.csgidev.com.rb' 'mnm-onprem-es-aiqa' 'D:/Jenkins/workspace/18.1.0.0.0-ACPx-MnMElasticsearchPolicy-DEPLOY-NP/_downloads/latest/mnm_es_1_7_5_all/mnm_es_1_7_5_all-116a3e3cdd22d72358474c782626b9e57411beb839f525b2202375a218bf25c9.tgz' --debug
Uploading policy to policy group mnm-onprem-es-aiqa
Error: Failed to publish archived policy
Reason: (ArgumentError) string contains null byte C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-config-12.19.36/lib/chef-config/path_helper.rb:115:in `absolute_path'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-config-12.19.36/lib/chef-config/path_helper.rb:115:in `canonical_path'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-config-12.19.36/lib/chef-config/config.rb:109:in `block in <class:Config>'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/mixlib-config-2.2.4/lib/mixlib/config/configurable.rb:48:in `get'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/mixlib-config-2.2.4/lib/mixlib/config.rb:433:in `internal_get'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/mixlib-config-2.2.4/lib/mixlib/config.rb:454:in `internal_get_or_set'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/mixlib-config-2.2.4/lib/mixlib/config.rb:475:in `block in define_attr_accessor_methods'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-config-12.19.36/lib/chef-config/config.rb:537:in `block in <class:Config>'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/mixlib-config-2.2.4/lib/mixlib/config/configurable.rb:48:in `get'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/mixlib-config-2.2.4/lib/mixlib/config.rb:433:in `internal_get'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/mixlib-config-2.2.4/lib/mixlib/config.rb:454:in `internal_get_or_set'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/mixlib-config-2.2.4/lib/mixlib/config.rb:475:in `block in define_attr_accessor_methods'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/http/ssl_policies.rb:79:in `set_custom_certs'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/http/ssl_policies.rb:48:in `apply'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/http/ssl_policies.rb:35:in `apply_to'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/http/basic_client.rb:153:in `configure_ssl'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/http/basic_client.rb:117:in `build_http_client'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/http/basic_client.rb:51:in `http_client'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/http/basic_client.rb:70:in `request'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/http.rb:340:in `block in send_http_request'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/http.rb:372:in `block in retrying_http_errors'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/http.rb:370:in `loop'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/http.rb:370:in `retrying_http_errors'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/http.rb:333:in `send_http_request'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/http.rb:148:in `request'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/http.rb:131:in `post'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/cookbook_uploader.rb:59:in `upload_cookbooks'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-dk-1.3.43/lib/chef-dk/policyfile/uploader.rb:194:in `upload_cookbooks'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-dk-1.3.43/lib/chef-dk/policyfile/uploader.rb:62:in `upload'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-dk-1.3.43/lib/chef-dk/policyfile_services/push_archive.rb:61:in `block in run'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-dk-1.3.43/lib/chef-dk/policyfile_services/push_archive.rb:130:in `block in stage_unpacked_archive'
C:/opscode/chefdk/embedded/lib/ruby/2.3.0/tmpdir.rb:89:in `mktmpdir'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-dk-1.3.43/lib/chef-dk/policyfile_services/push_archive.rb:128:in `stage_unpacked_archive'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-dk-1.3.43/lib/chef-dk/policyfile_services/push_archive.rb:58:in `run'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-dk-1.3.43/lib/chef-dk/command/push_archive.rb:73:in `run'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-dk-1.3.43/lib/chef-dk/command/base.rb:58:in `run_with_default_options'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-dk-1.3.43/lib/chef-dk/cli.rb:73:in `run'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-dk-1.3.43/bin/chef:25:in `<top (required)>'
C:/opscode/chefdk/bin/chef:94:in `load'
C:/opscode/chefdk/bin/chef:94:in `<main>'
ERROR: RuntimeError: Failed to push the policy archive D:/Jenkins/workspace/18.1.0.0.0-ACPx-MnMElasticsearchPolicy-DEPLOY-NP/_downloads/latest/mnm_es_1_7_5_all/mnm_es_1_7_5_all-116a3e3cdd22d72358474c782626b9e57411beb839f525b2202375a218bf25c9.tgz to the chef server d:\Jenkins\workspace\18.1.0.0.0-ACPx-MnMElasticsearchPolicy-DEPLOY-NP>exit 1
Success (different policy archive, same jenkins agent):
* executing standard policy file push * downloading policies for version latest of RunDeckPolicy... * download complete * exploding D:/Jenkins/workspace/18.1.0.0.0-ACPx-RunDeckPolicy-DEPLOY-NP/_downloads/latest/Policies.zip... * explosion complete * starting policy archive push from D:/Jenkins/workspace/18.1.0.0.0-ACPx-RunDeckPolicy-DEPLOY-NP/_downloads/latest... ** pushing policy archive D:/Jenkins/workspace/18.1.0.0.0-ACPx-RunDeckPolicy-DEPLOY-NP/_downloads/latest/rundeck_pro_cluster/rundeck_pro_cluster-caf807af567bd9fa0d36df4b67112d6b888a6cecf79dee49514e756901fa0b57.tgz to rundeck-dev using config C:/Program Files/OpenSSH/home/IOSBuild/.chef/knife.rb ** executing chef push-archive -c 'C:/Program Files/OpenSSH/home/IOSBuild/.chef/knife.rb' 'rundeck-dev' 'D:/Jenkins/workspace/18.1.0.0.0-ACPx-RunDeckPolicy-DEPLOY-NP/_downloads/latest/rundeck_pro_cluster/rundeck_pro_cluster-caf807af567bd9fa0d36df4b67112d6b888a6cecf79dee49514e756901fa0b57.tgz' --debug
Uploading policy to policy group rundeck-dev
Using poise-hoist 1.2.1 (81aa3c86)
Using csg_rundeck_pro 181.0.0 (c2bec542)
Using csg_crowdstrike 181.16.0 (093fc58a)
Using csg_crypto 181.16.0 (c8940ade)
Using csg_core 181.16.0 (6d41ede7)
Using csg_crowdstrike_agent 181.3.0 (cfa5e889)
Using csg_linux_core 181.16.0 (7f2f2a59)
Using csg_win_core 181.16.0 (92a68833)
Using chocolatey 1.2.1 (22ba3af0)
Using audit 4.3.0 (e5a12b26)
Using chef-client 7.2.1 (c47fd5b4)
Using csg_chocolatey 181.16.0 (54c3e5cb)
Using csg_sccm 181.3.0 (c07194e9)
Using csg_summarizer 181.16.0 (2d91985f)
Using csg_windows_2012r2_features_sources 181.4.0 (19469121)
Using windows 3.4.0 (87c1fdbb)
Using compat_resource 12.19.0 (215097ee)
Using ohai 5.2.0 (31667e45)
Using cron 5.0.1 (bf473949)
Using logrotate 2.2.0 (53e09234) ** pushing policy archive D:/Jenkins/workspace/18.1.0.0.0-ACPx-RunDeckPolicy-DEPLOY-NP/_downloads/latest/rundeck_pro_team/rundeck_pro_team-cb936dd7273cf8bd2606982a9cc8a34516a44c1ce26087f2274248fed90922f9.tgz to rundeck-dev using config C:/Program Files/OpenSSH/home/IOSBuild/.chef/knife.rb ** executing chef push-archive -c 'C:/Program Files/OpenSSH/home/IOSBuild/.chef/knife.rb' 'rundeck-dev' 'D:/Jenkins/workspace/18.1.0.0.0-ACPx-RunDeckPolicy-DEPLOY-NP/_downloads/latest/rundeck_pro_team/rundeck_pro_team-cb936dd7273cf8bd2606982a9cc8a34516a44c1ce26087f2274248fed90922f9.tgz' --debug
Uploading policy to policy group rundeck-dev
Using poise-hoist 1.2.1 (81aa3c86)
Using csg_rundeck_pro 181.0.0 (c2bec542)
Using csg_crowdstrike 181.16.0 (093fc58a)
Using csg_crypto 181.16.0 (c8940ade)
Using csg_core 181.16.0 (6d41ede7)
Using csg_crowdstrike_agent 181.3.0 (cfa5e889)
Using csg_linux_core 181.16.0 (7f2f2a59)
Using csg_win_core 181.16.0 (92a68833)
Using chocolatey 1.2.1 (22ba3af0)
Using audit 4.3.0 (e5a12b26)
Using chef-client 7.2.1 (c47fd5b4)
Using csg_chocolatey 181.16.0 (54c3e5cb)
Using csg_sccm 181.3.0 (c07194e9)
Using csg_summarizer 181.16.0 (2d91985f)
Using csg_windows_2012r2_features_sources 181.4.0 (19469121)
Using windows 3.4.0 (87c1fdbb)
Using compat_resource 12.19.0 (215097ee)
Using ohai 5.2.0 (31667e45)
Using cron 5.0.1 (bf473949)
Using logrotate 2.2.0 (53e09234) * policy archive push complete d:\Jenkins\workspace\18.1.0.0.0-ACPx-RunDeckPolicy-DEPLOY-NP>exit 0
For completeness, the (scrubbed) knife.rb file contents:
C:\Program Files\OpenSSH\home\IOSBuild>powershell Get-Content 'C:/Program Files/OpenSSH/home/IOSBuild/.chef/knife.rb'
client_key "C:/Users/IOSBuild/.chef/jenkins_farm_provisioner.pem"
node_name "jenkins_farm_provisioner"
chef_server_url 'https://chef.csgidev.com/organizations/csg'
knife[:vsphere_host] = "omadeviviss01.devcsg.com"
knife[:vsphere_dc] = "ODC_Self_Prov"
knife[:vsphere_insecure] = "true"
knife[:vsphere_user] = 'xxxxxxx'
knife[:vsphere_pass] = 'xxxxxxxxxx'
knife[:supermarket_site] = 'http://xxxxx:xxxxxx@artifactory.csgicorp.com/artifactory/api/chef/csg-18.1.0-chef'