Ohai-143?

Chef Infra (archive)
1

I am seeing crashes that look like OHAI-143 but what I don’t understand is that I have 30 hosts running the same build. This servers has three extra roles which are place holders and there for empty, but I tried removing them and it did not change anything. Any input to help me figure out the problem?

Thank you,

Joshua

jmiller@srv-101-07:~$ dpkg -l |grep chef
ii chef 0.7.16-1 configuration management system written in R
ii libchef-ruby 0.7.16-1 Ruby libraries for Chef configuration manage
ii libchef-ruby1.8 0.7.16-1 Ruby 1.8 libraries for Chef configuration ma

Starting chef-client in daemon mode crashes

jmiller@srv-101-07:~$ sudo /etc/init.d/chef-client start

  • Starting chef-client chef-client
    …done.
    jmiller@srv-101-07:~$ /usr/lib/ruby/1.8/ohai/plugins/linux/virtualization.rb:58: [BUG] Segmentation fault
    ruby 1.8.7 (2009-06-12 patchlevel 174) [x86_64-linux]

jmiller@srv-101-07:~$

Running chef-client in debug crashes

jmiller@srv-101-07:~$ sudo chef-client -l debug
/usr/lib/ruby/1.8/ohai/plugins/linux/virtualization.rb:58: [BUG] Segmentation fault
ruby 1.8.7 (2009-06-12 patchlevel 174) [x86_64-linux]

Aborted
jmiller@srv-101-07:~ cat /var/log/chef/client.log [Thu, 04 Feb 2010 13:45:53 -0800] INFO: Daemonizing.. [Thu, 04 Feb 2010 13:45:53 -0800] INFO: Daemonizing.. [Thu, 04 Feb 2010 13:45:53 -0800] INFO: Daemonizing.. [Thu, 04 Feb 2010 13:45:53 -0800] INFO: Forked, in 11557. Priveleges: 0 0 [Thu, 04 Feb 2010 13:46:03 -0800] INFO: Starting Chef Run [Thu, 04 Feb 2010 13:46:20 -0800] INFO: Starting Chef Run [Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: Loading plugin os [Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: Loading plugin ruby [Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: Loading plugin languages [Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: Loading plugin kernel [Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: ---- Begin uname -s STDOUT ---- [Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: Linux [Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: ---- End uname -s STDOUT ---- [Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: ---- Begin uname -s STDERR ---- [Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: [Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: ---- End uname -s STDERR ---- [Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: Ran (uname -s) returned 0 [Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: ---- Begin uname -r STDOUT ---- [Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: 2.6.31-16-generic [Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: ---- End uname -r STDOUT ---- [Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: ---- Begin uname -r STDERR ---- [Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: [Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: ---- End uname -r STDERR ---- [Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: Ran (uname -r) returned 0 [Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: ---- Begin uname -v STDOUT ---- [Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: #53-Ubuntu SMP Tue Dec 8 04:02:15 UTC 2009 [Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: ---- End uname -v STDOUT ---- [Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: ---- Begin uname -v STDERR ---- [Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: [Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: ---- End uname -v STDERR ---- [Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: Ran (uname -v) returned 0 [Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: ---- Begin uname -m STDOUT ---- [Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: x86_64 [Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: ---- End uname -m STDOUT ---- [Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: ---- Begin uname -m STDERR ---- [Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: [Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: ---- End uname -m STDERR ---- [Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: Ran (uname -m) returned 0 [Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: Loading plugin virtualization [Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: Loading plugin linux::virtualization jmiller@srv-101-07:~

No debug chef runs manually just fine

jmiller@srv-101-07:~ sudo chef-client jmiller@srv-101-07:~ cat /var/log/chef/client.log
…
[Thu, 04 Feb 2010 13:47:34 -0800] INFO: Starting Chef Run
[Thu, 04 Feb 2010 13:47:39 -0800] INFO: Ran execute[apt-get update] successfully
[Thu, 04 Feb 2010 13:47:39 -0800] WARN: service[nscd]: falling back to process table inspection
[Thu, 04 Feb 2010 13:47:40 -0800] WARN: service[nscd]: attempting to match ‘nscd’ ((?-mix:nscd)) against process list
[Thu, 04 Feb 2010 13:47:40 -0800] WARN: service[nscd]: falling back to process table inspection
[Thu, 04 Feb 2010 13:47:40 -0800] WARN: service[nscd]: attempting to match ‘nscd’ ((?-mix:nscd)) against process list
[Thu, 04 Feb 2010 13:47:41 -0800] WARN: service[apparmor]: falling back to process table inspection
[Thu, 04 Feb 2010 13:47:41 -0800] WARN: service[apparmor]: attempting to match ‘apparmor’ ((?-mix:apparmor)) against process list
[Thu, 04 Feb 2010 13:47:41 -0800] WARN: service[apparmor]: falling back to process table inspection
[Thu, 04 Feb 2010 13:47:41 -0800] WARN: service[apparmor]: attempting to match ‘apparmor’ ((?-mix:apparmor)) against process list
[Thu, 04 Feb 2010 13:47:41 -0800] WARN: service[rsyslog]: falling back to process table inspection
[Thu, 04 Feb 2010 13:47:41 -0800] WARN: service[rsyslog]: attempting to match ‘rsyslog’ ((?-mix:rsyslog)) against process list
[Thu, 04 Feb 2010 13:47:41 -0800] WARN: service[rsyslog]: falling back to process table inspection
[Thu, 04 Feb 2010 13:47:41 -0800] WARN: service[rsyslog]: attempting to match ‘rsyslog’ ((?-mix:rsyslog)) against process list
[Thu, 04 Feb 2010 13:47:42 -0800] WARN: service[snmpd]: falling back to process table inspection
[Thu, 04 Feb 2010 13:47:42 -0800] WARN: service[snmpd]: attempting to match ‘snmpd’ ((?-mix:snmpd)) against process list
[Thu, 04 Feb 2010 13:47:42 -0800] WARN: service[snmpd]: falling back to process table inspection
[Thu, 04 Feb 2010 13:47:42 -0800] WARN: service[snmpd]: attempting to match ‘snmpd’ ((?-mix:snmpd)) against process list
[Thu, 04 Feb 2010 13:47:43 -0800] WARN: service[ssh]: falling back to process table inspection
[Thu, 04 Feb 2010 13:47:43 -0800] WARN: service[ssh]: attempting to match ‘ssh’ ((?-mix:ssh)) against process list
[Thu, 04 Feb 2010 13:47:43 -0800] WARN: service[ssh]: falling back to process table inspection
[Thu, 04 Feb 2010 13:47:43 -0800] WARN: service[ssh]: attempting to match ‘ssh’ ((?-mix:ssh)) against process list
[Thu, 04 Feb 2010 13:47:43 -0800] WARN: service[postfix]: falling back to process table inspection
[Thu, 04 Feb 2010 13:47:43 -0800] WARN: service[postfix]: attempting to match ‘postfix’ ((?-mix:postfix)) against process list
[Thu, 04 Feb 2010 13:47:44 -0800] WARN: service[ntpd]: falling back to process table inspection
[Thu, 04 Feb 2010 13:47:44 -0800] WARN: service[ntpd]: attempting to match ‘ntpd’ ((?-mix:ntpd)) against process list
[Thu, 04 Feb 2010 13:47:47 -0800] INFO: Chef Run complete in 12.70877 seconds
[Thu, 04 Feb 2010 13:48:21 -0800] INFO: Daemonizing…
[Thu, 04 Feb 2010 13:48:21 -0800] INFO: Daemonizing…
[Thu, 04 Feb 2010 13:48:21 -0800] INFO: Daemonizing…
[Thu, 04 Feb 2010 13:48:21 -0800] INFO: Forked, in 11729. Priveleges: 0 0
[Thu, 04 Feb 2010 13:48:29 -0800] INFO: Starting Chef Run
jmiller@srv-101-07:~$

2

Hi Josh,
The segfault is actually a ruby bug, not ohai or Chef. These are
tricky to pin down, but one such bug that was recently resolved is
tracked here: http://redmine.ruby-lang.org/issues/show/2326

Bryan McLellan submitted patches to both Ubuntu and Debian and posted
links to their respective tracking systems:
http://redmine.ruby-lang.org/issues/show/2326#note-7

Can you see if the updated versions of ruby fix the issue for you?

On Thu, Feb 4, 2010 at 1:53 PM, Joshua Miller jassinpain@gmail.com wrote:

I am seeing crashes that look like OHAI-143 but what I don't understand is that I have 30 hosts running the same build. This servers has three extra roles which are place holders and there for empty, but I tried removing them and it did not change anything. Any input to help me figure out the problem?

Thank you,

Joshua

jmiller@srv-101-07:~$ dpkg -l |grep chef
ii chef 0.7.16-1 configuration management system written in R
ii libchef-ruby 0.7.16-1 Ruby libraries for Chef configuration manage
ii libchef-ruby1.8 0.7.16-1 Ruby 1.8 libraries for Chef configuration ma

Starting chef-client in daemon mode crashes

jmiller@srv-101-07:~$ sudo /etc/init.d/chef-client start

  • Starting chef-client chef-client
    ...done.
    jmiller@srv-101-07:~$ /usr/lib/ruby/1.8/ohai/plugins/linux/virtualization.rb:58: [BUG] Segmentation fault
    ruby 1.8.7 (2009-06-12 patchlevel 174) [x86_64-linux]

jmiller@srv-101-07:~$

Running chef-client in debug crashes

jmiller@srv-101-07:~$ sudo chef-client -l debug
/usr/lib/ruby/1.8/ohai/plugins/linux/virtualization.rb:58: [BUG] Segmentation fault
ruby 1.8.7 (2009-06-12 patchlevel 174) [x86_64-linux]

Aborted
jmiller@srv-101-07:~$ cat /var/log/chef/client.log
[Thu, 04 Feb 2010 13:45:53 -0800] INFO: Daemonizing..
[Thu, 04 Feb 2010 13:45:53 -0800] INFO: Daemonizing..
[Thu, 04 Feb 2010 13:45:53 -0800] INFO: Daemonizing..
[Thu, 04 Feb 2010 13:45:53 -0800] INFO: Forked, in 11557. Priveleges: 0 0
[Thu, 04 Feb 2010 13:46:03 -0800] INFO: Starting Chef Run
[Thu, 04 Feb 2010 13:46:20 -0800] INFO: Starting Chef Run
[Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: Loading plugin os
[Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: Loading plugin ruby
[Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: Loading plugin languages
[Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: Loading plugin kernel
[Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: ---- Begin uname -s STDOUT ----
[Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: Linux
[Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: ---- End uname -s STDOUT ----
[Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: ---- Begin uname -s STDERR ----
[Thu, 04 Feb 2010 13:46:20 -0800] DEBUG:
[Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: ---- End uname -s STDERR ----
[Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: Ran (uname -s) returned 0
[Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: ---- Begin uname -r STDOUT ----
[Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: 2.6.31-16-generic
[Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: ---- End uname -r STDOUT ----
[Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: ---- Begin uname -r STDERR ----
[Thu, 04 Feb 2010 13:46:20 -0800] DEBUG:
[Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: ---- End uname -r STDERR ----
[Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: Ran (uname -r) returned 0
[Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: ---- Begin uname -v STDOUT ----
[Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: #53-Ubuntu SMP Tue Dec 8 04:02:15 UTC 2009
[Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: ---- End uname -v STDOUT ----
[Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: ---- Begin uname -v STDERR ----
[Thu, 04 Feb 2010 13:46:20 -0800] DEBUG:
[Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: ---- End uname -v STDERR ----
[Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: Ran (uname -v) returned 0
[Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: ---- Begin uname -m STDOUT ----
[Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: x86_64
[Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: ---- End uname -m STDOUT ----
[Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: ---- Begin uname -m STDERR ----
[Thu, 04 Feb 2010 13:46:20 -0800] DEBUG:
[Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: ---- End uname -m STDERR ----
[Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: Ran (uname -m) returned 0
[Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: Loading plugin virtualization
[Thu, 04 Feb 2010 13:46:20 -0800] DEBUG: Loading plugin linux::virtualization
jmiller@srv-101-07:~$

No debug chef runs manually just fine

jmiller@srv-101-07:~$ sudo chef-client
jmiller@srv-101-07:~$ cat /var/log/chef/client.log
...
[Thu, 04 Feb 2010 13:47:34 -0800] INFO: Starting Chef Run
[Thu, 04 Feb 2010 13:47:39 -0800] INFO: Ran execute[apt-get update] successfully
[Thu, 04 Feb 2010 13:47:39 -0800] WARN: service[nscd]: falling back to process table inspection
[Thu, 04 Feb 2010 13:47:40 -0800] WARN: service[nscd]: attempting to match 'nscd' ((?-mix:nscd)) against process list
[Thu, 04 Feb 2010 13:47:40 -0800] WARN: service[nscd]: falling back to process table inspection
[Thu, 04 Feb 2010 13:47:40 -0800] WARN: service[nscd]: attempting to match 'nscd' ((?-mix:nscd)) against process list
[Thu, 04 Feb 2010 13:47:41 -0800] WARN: service[apparmor]: falling back to process table inspection
[Thu, 04 Feb 2010 13:47:41 -0800] WARN: service[apparmor]: attempting to match 'apparmor' ((?-mix:apparmor)) against process list
[Thu, 04 Feb 2010 13:47:41 -0800] WARN: service[apparmor]: falling back to process table inspection
[Thu, 04 Feb 2010 13:47:41 -0800] WARN: service[apparmor]: attempting to match 'apparmor' ((?-mix:apparmor)) against process list
[Thu, 04 Feb 2010 13:47:41 -0800] WARN: service[rsyslog]: falling back to process table inspection
[Thu, 04 Feb 2010 13:47:41 -0800] WARN: service[rsyslog]: attempting to match 'rsyslog' ((?-mix:rsyslog)) against process list
[Thu, 04 Feb 2010 13:47:41 -0800] WARN: service[rsyslog]: falling back to process table inspection
[Thu, 04 Feb 2010 13:47:41 -0800] WARN: service[rsyslog]: attempting to match 'rsyslog' ((?-mix:rsyslog)) against process list
[Thu, 04 Feb 2010 13:47:42 -0800] WARN: service[snmpd]: falling back to process table inspection
[Thu, 04 Feb 2010 13:47:42 -0800] WARN: service[snmpd]: attempting to match 'snmpd' ((?-mix:snmpd)) against process list
[Thu, 04 Feb 2010 13:47:42 -0800] WARN: service[snmpd]: falling back to process table inspection
[Thu, 04 Feb 2010 13:47:42 -0800] WARN: service[snmpd]: attempting to match 'snmpd' ((?-mix:snmpd)) against process list
[Thu, 04 Feb 2010 13:47:43 -0800] WARN: service[ssh]: falling back to process table inspection
[Thu, 04 Feb 2010 13:47:43 -0800] WARN: service[ssh]: attempting to match 'ssh' ((?-mix:ssh)) against process list
[Thu, 04 Feb 2010 13:47:43 -0800] WARN: service[ssh]: falling back to process table inspection
[Thu, 04 Feb 2010 13:47:43 -0800] WARN: service[ssh]: attempting to match 'ssh' ((?-mix:ssh)) against process list
[Thu, 04 Feb 2010 13:47:43 -0800] WARN: service[postfix]: falling back to process table inspection
[Thu, 04 Feb 2010 13:47:43 -0800] WARN: service[postfix]: attempting to match 'postfix' ((?-mix:postfix)) against process list
[Thu, 04 Feb 2010 13:47:44 -0800] WARN: service[ntpd]: falling back to process table inspection
[Thu, 04 Feb 2010 13:47:44 -0800] WARN: service[ntpd]: attempting to match 'ntpd' ((?-mix:ntpd)) against process list
[Thu, 04 Feb 2010 13:47:47 -0800] INFO: Chef Run complete in 12.70877 seconds
[Thu, 04 Feb 2010 13:48:21 -0800] INFO: Daemonizing..
[Thu, 04 Feb 2010 13:48:21 -0800] INFO: Daemonizing..
[Thu, 04 Feb 2010 13:48:21 -0800] INFO: Daemonizing..
[Thu, 04 Feb 2010 13:48:21 -0800] INFO: Forked, in 11729. Priveleges: 0 0
[Thu, 04 Feb 2010 13:48:29 -0800] INFO: Starting Chef Run
jmiller@srv-101-07:~$