Kitchen-docker + Oracle Linux 5 converge error: closed stream


#1

https://gist.github.com/stormerider/c92325be6cd0931f3e36

Has anyone seen this kind of thing before? I’m really not sure what’s going
on there. As far as I can tell, for some reason during the run, the docker
container seems to just go away. I’m not sure if that’s because chef-solo
failed and it terminated automatically, or if TK reaped it because it lost
communication, or what.

If I restart the container, I’m able to run chef-solo fine:

root@433af12cc045:/tmp/kitchen# chef-solo -c solo.rb -j dna.json
[2015-04-22T16:54:44+00:00] WARN:


SSL validation of HTTPS requests is disabled. HTTPS connections are still
encrypted, but chef is not able to detect forged replies or man in the
middle
attacks.

To fix this issue add an entry like this to your configuration file:

  # Verify all HTTPS connections (recommended)
  ssl_verify_mode :verify_peer

  # OR, Verify only connections to chef-server
  verify_api_cert true

To check your SSL configuration, or troubleshoot errors, you can use the
knife ssl check command like so:

  knife ssl check -c solo.rb

Starting Chef Client, version 11.16.4
Compiling Cookbooks…
Recipe: ohai::default

  • remote_directory[/etc/chef/ohai_plugins for cookbook ohai] action create
    Recipe:

    • cookbook_file[/etc/chef/ohai_plugins/README] action create (up to
      date)
      (up to date)
      Recipe: ohai::default
  • ohai[custom_plugins] action reload[2015-04-22T16:54:45+00:00] WARN:
    Plugin Definition Error: </etc/chef/ohai_plugins/packages.rb>: collect_data
    already defined on platform linux

    • re-run ohai and merge results into node attributes
      [2015-04-22T16:54:46+00:00] WARN: RS-DCS-Inventory::default - This recipe
      normally starts chef-client as a daemon. Skipping while in solo/local mode.
      [2015-04-22T16:54:46+00:00] WARN: RS-DCS-Chef::hoptoad - This recipe uses
      Chef handlers. Skipping while in solo/local mode.
      [2015-04-22T16:54:46+00:00] WARN: RS-DCS-Chef::updater - This recipe uses
      Chef search. Skipping while in solo/local mode.
      [2015-04-22T16:54:46+00:00] WARN: RS-DCS-Base::default - Checking
      node[‘RS’][‘Datacenter’]: ASH
      [2015-04-22T16:54:46+00:00] WARN: Cloning resource attributes for
      package[subversion] from prior resource (CHEF-3694)
      [2015-04-22T16:54:46+00:00] WARN: Previous package[subversion]:
      /tmp/kitchen/cookbooks/RS-DCS-Base/recipes/default.rb:134:in block in from_file' [2015-04-22T16:54:46+00:00] WARN: Current package[subversion]: /tmp/kitchen/cookbooks/RS-DCS-Base/recipes/default.rb:380:infrom_file’
      Recipe: RS-DCS-Base::default
  • package[subversion] action install (up to date)
    Converging 94 resources
    Recipe: ohai::default

  • remote_directory[/etc/chef/ohai_plugins for cookbook ohai] action
    nothing (skipped due to action :nothing)

  • ohai[custom_plugins] action nothing (skipped due to action :nothing)
    Recipe: RS-DCS-Ohai-Plugins::default

  • file[/etc/chef/ohai_plugins/mysql-databases.rb] action delete (up to
    date)

  • ohai[reload] action nothing (skipped due to action :nothing)

  • cookbook_file[/etc/chef/ohai_plugins/databases-mysql.rb] action create
    (up to date)

  • cookbook_file[/etc/chef/ohai_plugins/packages.rb] action create (up to
    date)
    Recipe: logrotate::default

  • package[logrotate] action install (up to date)

  • directory[/etc/logrotate.d] action create (up to date)
    Recipe: chef-client::config

  • template[/etc/logrotate.d/chef-client] action create (up to date)

  • directory[/var/run/chef] action create (up to date)

  • directory[/var/cache/chef] action create (up to date)

  • directory[/var/lib/chef] action create (up to date)

  • directory[/var/log/chef] action create (up to date)

  • directory[/etc/chef] action create (up to date)

  • file[/var/log/chef/client.log] action create (up to date)

  • template[/etc/chef/client.rb] action create (up to date)

  • directory[/etc/chef/client.d] action create (up to date)

  • ruby_block[reload_client_config] action nothing (skipped due to action
    :nothing)
    Recipe: RS-DCS-Base::default

  • sudo[45-nagios-checks] action install

    • template[/etc/sudoers.d/45-nagios-checks] action create (up to date)
      (up to date)
  • template[/etc/sudoers.d/45-nagios-checks] action nothing (skipped due
    to action :nothing)

  • package[nss] action install (up to date)

  • package[pam_krb5] action install (up to date)

  • package[krb5-workstation] action install (up to date)

  • package[nss-tools] action install (up to date)

  • package[pam_passwdqc] action install (up to date)

  • package[cracklib] action install (up to date)

  • package[net-snmp] action install (up to date)

  • package[ntp] action install (up to date)

  • package[nscd] action install (up to date)

  • package[rwho] action install (up to date)

  • package[vixie-cron] action install (up to date)

  • package[anacron] action install (up to date)

  • package[crontabs] action install (up to date)

  • package[authconfig] action install (up to date)

  • package[sudo] action install (up to date)

  • package[screen] action install (up to date)

  • package[less] action install (up to date)

  • package[subversion] action install (up to date)

  • package[strace] action install (up to date)

  • package[resolvconf] action remove (up to date)

  • link[/etc/init.d/ssh] action create (up to date)

  • template[/etc/pam.d/system-auth-ac] action create (up to date)

  • template[/etc/profile.d/bashrc.sh] action create (up to date)

  • template[/etc/krb5.conf] action create (up to date)

  • template[/etc/ldap.conf] action create (up to date)

  • link[/etc/libnss-ldap.conf] action create (up to date)

  • link[/etc/pam_ldap.conf] action create (up to date)

  • cookbook_file[/etc/nsswitch.conf] action create (up to date)

  • template[/etc/resolv.conf] action create (up to date)

  • execute[nssupdate] action nothing (skipped due to action :nothing)

  • template[/etc/snmp/snmpd.conf] action create (up to date)

  • template[/etc/default/snmpd] action create (up to date)

  • service[snmpd] action enable (up to date)

  • cron[ntpdate] action create (up to date)

  • directory[/usr/local/scripts] action create (up to date)

  • package[subversion] action nothing (skipped due to action :nothing)

  • bash[rehome-sysadmin-svn-if-needed] action run

    • execute “bash” “/tmp/chef-script20150422-29-nh1w2k”
  • subversion[SysAdmin] action sync (up to date)

  • ruby_block[svn_config_fix] action run

    • execute the ruby block svn_config_fix
  • group[cacti] action create (up to date)

  • user[cacti] action create (up to date)

  • directory[/home/cacti/.ssh/] action create (up to date)

  • cookbook_file[/home/cacti/.ssh/authorized_keys] action create (up to
    date)

  • cookbook_file[/etc/ssh/sshd_config] action create (skipped due to
    only_if)

  • service[ssh] action nothing (skipped due to action :nothing)

  • execute[delete_ssh_keys_if_needed] action run (skipped due to only_if)

  • execute[generate_new_ssh_host_keys] action nothing (skipped due to
    action :nothing)

  • file[/etc/RosettaStone/FRESH_CLONE] action delete (skipped due to
    only_if)

  • cookbook_file[/etc/update-motd.d/98-chef-role] action create (skipped
    due to only_if)

  • directory[/etc/cron.scripts] action create (up to date)

  • cookbook_file[/etc/cron.scripts/logerror] action create (up to date)

  • service[nscd] action restart

    • restart service service[nscd]
  • service[nscd] action enable (up to date)

  • cookbook_file[/usr/local/bin/chef-boot] action create (up to date)

  • cookbook_file[/usr/local/bin/chef-dryrun] action create (up to date)

  • cookbook_file[/usr/local/bin/chef-run] action create (up to date)

  • cookbook_file[/usr/local/bin/jq] action create (up to date)
    Recipe: RS-DCS-Base::oracle_linux_all_versions

  • cookbook_file[/etc/sudoers] action create (up to date)

  • link[/usr/bin/vi] action create (up to date)
    Recipe: RS-DCS-Base::oracle_linux_5

  • package[gpm] action remove (up to date)

  • package[vim-enhanced] action remove (up to date)

  • package[bluez-gnome] action remove (up to date)

  • package[bluez-libs] action remove (up to date)

  • package[bluez-utils] action remove (up to date)

  • package[xorg-x11-utils] action install (up to date)

  • package[xorg-x11-xinit] action install (up to date)

  • package[xorg-x11-xauth] action install (up to date)

  • package[xorg-x11-apps] action install (up to date)

  • cookbook_file[/etc/modprobe.d/disable-ipv6.conf] action create (up to
    date)

  • ruby_block[append_special_oracle_ldap_auth_string] action run

    • execute the ruby block append_special_oracle_ldap_auth_string
  • file[/etc/openldap/ldap.conf] action delete (skipped due to not_if)

  • link[/etc/openldap/ldap.conf-link] action create (up to date)

  • service[ip6tables] action stop (up to date)

  • service[ip6tables] action disable (up to date)

  • ruby_block[disable-ipv6] action run

    • execute the ruby block disable-ipv6
  • cookbook_file[/etc/selinux/config] action create (up to date)

  • service[iptables] action stop (up to date)

  • service[iptables] action disable (up to date)
    Recipe: RS-DCS-Base::default

  • execute[nssupdate] action run (skipped due to only_if)

  • service[nscd] action restart

    • restart service service[nscd]

Running handlers:
Running handlers complete
Chef Client finished, 8/90 resources updated in 5.434002998 seconds
root@433af12cc045:/tmp/kitchen#

Any thoughts appreciated, I’m very much at a loss as to what’s occurring
here.


~~ StormeRider ~~

“Every world needs its heroes […] They inspire us to be better than we
are. And they protect from the darkness that’s just around the corner.”

(from Smallville Season 6x1: “Zod”)

On why I hate the phrase “that’s so lame”… http://bit.ly/Ps3uSS


#2

Does anyone have any idea how I can debug this issue? Does docker keep a
log of container spinups/commands/etc., beyond what the kitchen log would
show? As you can see in the gist, the Kitchen log doesn’t really show much
more than inside the kitchen verify.

I’m stumped as to what’s going on and could really use some assistance in
how to best go about debugging this, because I’m at a total loss.


~~ StormeRider ~~

“Every world needs its heroes […] They inspire us to be better than we
are. And they protect from the darkness that’s just around the corner.”

(from Smallville Season 6x1: “Zod”)

On why I hate the phrase “that’s so lame”… http://bit.ly/Ps3uSS

On Wed, Apr 22, 2015 at 9:57 AM, Morgan Blackthorne stormerider@gmail.com
wrote:

https://gist.github.com/stormerider/c92325be6cd0931f3e36

Has anyone seen this kind of thing before? I’m really not sure what’s
going on there. As far as I can tell, for some reason during the run, the
docker container seems to just go away. I’m not sure if that’s because
chef-solo failed and it terminated automatically, or if TK reaped it
because it lost communication, or what.

If I restart the container, I’m able to run chef-solo fine:

root@433af12cc045:/tmp/kitchen# chef-solo -c solo.rb -j dna.json
[2015-04-22T16:54:44+00:00] WARN:



SSL validation of HTTPS requests is disabled. HTTPS connections are still
encrypted, but chef is not able to detect forged replies or man in the
middle
attacks.

To fix this issue add an entry like this to your configuration file:

  # Verify all HTTPS connections (recommended)
  ssl_verify_mode :verify_peer

  # OR, Verify only connections to chef-server
  verify_api_cert true

To check your SSL configuration, or troubleshoot errors, you can use the
knife ssl check command like so:

  knife ssl check -c solo.rb


Starting Chef Client, version 11.16.4
Compiling Cookbooks…
Recipe: ohai::default

  • remote_directory[/etc/chef/ohai_plugins for cookbook ohai] action
    create
    Recipe:

    • cookbook_file[/etc/chef/ohai_plugins/README] action create (up to
      date)
      (up to date)
      Recipe: ohai::default
  • ohai[custom_plugins] action reload[2015-04-22T16:54:45+00:00] WARN:
    Plugin Definition Error: </etc/chef/ohai_plugins/packages.rb>: collect_data
    already defined on platform linux

    • re-run ohai and merge results into node attributes
      [2015-04-22T16:54:46+00:00] WARN: RS-DCS-Inventory::default - This recipe
      normally starts chef-client as a daemon. Skipping while in solo/local mode.
      [2015-04-22T16:54:46+00:00] WARN: RS-DCS-Chef::hoptoad - This recipe uses
      Chef handlers. Skipping while in solo/local mode.
      [2015-04-22T16:54:46+00:00] WARN: RS-DCS-Chef::updater - This recipe uses
      Chef search. Skipping while in solo/local mode.
      [2015-04-22T16:54:46+00:00] WARN: RS-DCS-Base::default - Checking
      node[‘RS’][‘Datacenter’]: ASH
      [2015-04-22T16:54:46+00:00] WARN: Cloning resource attributes for
      package[subversion] from prior resource (CHEF-3694)
      [2015-04-22T16:54:46+00:00] WARN: Previous package[subversion]:
      /tmp/kitchen/cookbooks/RS-DCS-Base/recipes/default.rb:134:in block in from_file' [2015-04-22T16:54:46+00:00] WARN: Current package[subversion]: /tmp/kitchen/cookbooks/RS-DCS-Base/recipes/default.rb:380:infrom_file’
      Recipe: RS-DCS-Base::default
  • package[subversion] action install (up to date)
    Converging 94 resources
    Recipe: ohai::default

  • remote_directory[/etc/chef/ohai_plugins for cookbook ohai] action
    nothing (skipped due to action :nothing)

  • ohai[custom_plugins] action nothing (skipped due to action :nothing)
    Recipe: RS-DCS-Ohai-Plugins::default

  • file[/etc/chef/ohai_plugins/mysql-databases.rb] action delete (up to
    date)

  • ohai[reload] action nothing (skipped due to action :nothing)

  • cookbook_file[/etc/chef/ohai_plugins/databases-mysql.rb] action create
    (up to date)

  • cookbook_file[/etc/chef/ohai_plugins/packages.rb] action create (up to
    date)
    Recipe: logrotate::default

  • package[logrotate] action install (up to date)

  • directory[/etc/logrotate.d] action create (up to date)
    Recipe: chef-client::config

  • template[/etc/logrotate.d/chef-client] action create (up to date)

  • directory[/var/run/chef] action create (up to date)

  • directory[/var/cache/chef] action create (up to date)

  • directory[/var/lib/chef] action create (up to date)

  • directory[/var/log/chef] action create (up to date)

  • directory[/etc/chef] action create (up to date)

  • file[/var/log/chef/client.log] action create (up to date)

  • template[/etc/chef/client.rb] action create (up to date)

  • directory[/etc/chef/client.d] action create (up to date)

  • ruby_block[reload_client_config] action nothing (skipped due to action
    :nothing)
    Recipe: RS-DCS-Base::default

  • sudo[45-nagios-checks] action install

    • template[/etc/sudoers.d/45-nagios-checks] action create (up to date)
      (up to date)
  • template[/etc/sudoers.d/45-nagios-checks] action nothing (skipped due
    to action :nothing)

  • package[nss] action install (up to date)

  • package[pam_krb5] action install (up to date)

  • package[krb5-workstation] action install (up to date)

  • package[nss-tools] action install (up to date)

  • package[pam_passwdqc] action install (up to date)

  • package[cracklib] action install (up to date)

  • package[net-snmp] action install (up to date)

  • package[ntp] action install (up to date)

  • package[nscd] action install (up to date)

  • package[rwho] action install (up to date)

  • package[vixie-cron] action install (up to date)

  • package[anacron] action install (up to date)

  • package[crontabs] action install (up to date)

  • package[authconfig] action install (up to date)

  • package[sudo] action install (up to date)

  • package[screen] action install (up to date)

  • package[less] action install (up to date)

  • package[subversion] action install (up to date)

  • package[strace] action install (up to date)

  • package[resolvconf] action remove (up to date)

  • link[/etc/init.d/ssh] action create (up to date)

  • template[/etc/pam.d/system-auth-ac] action create (up to date)

  • template[/etc/profile.d/bashrc.sh] action create (up to date)

  • template[/etc/krb5.conf] action create (up to date)

  • template[/etc/ldap.conf] action create (up to date)

  • link[/etc/libnss-ldap.conf] action create (up to date)

  • link[/etc/pam_ldap.conf] action create (up to date)

  • cookbook_file[/etc/nsswitch.conf] action create (up to date)

  • template[/etc/resolv.conf] action create (up to date)

  • execute[nssupdate] action nothing (skipped due to action :nothing)

  • template[/etc/snmp/snmpd.conf] action create (up to date)

  • template[/etc/default/snmpd] action create (up to date)

  • service[snmpd] action enable (up to date)

  • cron[ntpdate] action create (up to date)

  • directory[/usr/local/scripts] action create (up to date)

  • package[subversion] action nothing (skipped due to action :nothing)

  • bash[rehome-sysadmin-svn-if-needed] action run

    • execute “bash” “/tmp/chef-script20150422-29-nh1w2k”
  • subversion[SysAdmin] action sync (up to date)

  • ruby_block[svn_config_fix] action run

    • execute the ruby block svn_config_fix
  • group[cacti] action create (up to date)

  • user[cacti] action create (up to date)

  • directory[/home/cacti/.ssh/] action create (up to date)

  • cookbook_file[/home/cacti/.ssh/authorized_keys] action create (up to
    date)

  • cookbook_file[/etc/ssh/sshd_config] action create (skipped due to
    only_if)

  • service[ssh] action nothing (skipped due to action :nothing)

  • execute[delete_ssh_keys_if_needed] action run (skipped due to only_if)

  • execute[generate_new_ssh_host_keys] action nothing (skipped due to
    action :nothing)

  • file[/etc/RosettaStone/FRESH_CLONE] action delete (skipped due to
    only_if)

  • cookbook_file[/etc/update-motd.d/98-chef-role] action create (skipped
    due to only_if)

  • directory[/etc/cron.scripts] action create (up to date)

  • cookbook_file[/etc/cron.scripts/logerror] action create (up to date)

  • service[nscd] action restart

    • restart service service[nscd]
  • service[nscd] action enable (up to date)

  • cookbook_file[/usr/local/bin/chef-boot] action create (up to date)

  • cookbook_file[/usr/local/bin/chef-dryrun] action create (up to date)

  • cookbook_file[/usr/local/bin/chef-run] action create (up to date)

  • cookbook_file[/usr/local/bin/jq] action create (up to date)
    Recipe: RS-DCS-Base::oracle_linux_all_versions

  • cookbook_file[/etc/sudoers] action create (up to date)

  • link[/usr/bin/vi] action create (up to date)
    Recipe: RS-DCS-Base::oracle_linux_5

  • package[gpm] action remove (up to date)

  • package[vim-enhanced] action remove (up to date)

  • package[bluez-gnome] action remove (up to date)

  • package[bluez-libs] action remove (up to date)

  • package[bluez-utils] action remove (up to date)

  • package[xorg-x11-utils] action install (up to date)

  • package[xorg-x11-xinit] action install (up to date)

  • package[xorg-x11-xauth] action install (up to date)

  • package[xorg-x11-apps] action install (up to date)

  • cookbook_file[/etc/modprobe.d/disable-ipv6.conf] action create (up to
    date)

  • ruby_block[append_special_oracle_ldap_auth_string] action run

    • execute the ruby block append_special_oracle_ldap_auth_string
  • file[/etc/openldap/ldap.conf] action delete (skipped due to not_if)

  • link[/etc/openldap/ldap.conf-link] action create (up to date)

  • service[ip6tables] action stop (up to date)

  • service[ip6tables] action disable (up to date)

  • ruby_block[disable-ipv6] action run

    • execute the ruby block disable-ipv6
  • cookbook_file[/etc/selinux/config] action create (up to date)

  • service[iptables] action stop (up to date)

  • service[iptables] action disable (up to date)
    Recipe: RS-DCS-Base::default

  • execute[nssupdate] action run (skipped due to only_if)

  • service[nscd] action restart

    • restart service service[nscd]

Running handlers:
Running handlers complete
Chef Client finished, 8/90 resources updated in 5.434002998 seconds
root@433af12cc045:/tmp/kitchen#

Any thoughts appreciated, I’m very much at a loss as to what’s occurring
here.


~~ StormeRider ~~

“Every world needs its heroes […] They inspire us to be better than we
are. And they protect from the darkness that’s just around the corner.”

(from Smallville Season 6x1: “Zod”)

On why I hate the phrase “that’s so lame”… http://bit.ly/Ps3uSS