Segfault with CentOS 5.4/5.5 + ruby 1.8.7 + chef 0.10.0

Hi all,

I noticed a couple of email threads in the past about seg faults and didn’t see
any resolution so I wanted to share my experience with the same thing happening
to us on a few CentOS 5.4 and 5.5 servers (we haven’t tested the client on 5.6
yet). I’m consistently able to get our chef-client to segfault whenever it
handles this package resource (haven’t had time to try other packages yet):

[Thu, 02 Jun 2011 17:38:17 -0700] INFO: Processing package[logrotate] action
upgrade (logrotate::default line 20)
/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/…/lib/chef/provider/package/yum.rb:75:
[BUG] Segmentation fault

Thu, 02 Jun 2011 17:38:27 -0700] INFO: Processing package[logrotate] action
upgrade (logrotate::default line 20)
/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/…/lib/chef/provider/package/yum.rb:64:
[BUG] Segmentation fault

[Thu, 02 Jun 2011 17:38:53 -0700] INFO: Processing package[logrotate] action
upgrade (logrotate::default line 20)
/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/…/lib/chef/provider/package/yum.rb:66:
[BUG] Segmentation fault

We were trying this with ruby 1.8.7 p334 from the FrameOS rbel5 i386 repo.
Since it sounded like an issue with ruby, reinstalling with ruby 1.8.6 p420
(and rebuilding the rbel5 rubygem-chef rpm to relax the ruby version
requirement) seemed to solve the problem.

Is anyone else noticing seg fauls with a combination of CentOS 5.x, ruby 1.8.7
and Chef?

Thanks,
Ray

Ray,

Can you provide the means to reproduce, specifically: steps to install from
the RBEL repo, and the AMI / image info? Are these i386 instances?

We maintain packages at rpm.aegisco.com, but I'm happy to try to help debug
the issues you're having. I'm surprised that Ruby 1.8.6 worked, I had
trouble getting 0.10 to work on anything older than 1.8.7 and rubygems
1.6.2.

James

On Thu, Jun 2, 2011 at 6:29 PM, tekkaray@yotowoti.com wrote:

Hi all,

I noticed a couple of email threads in the past about seg faults and didn't
see
any resolution so I wanted to share my experience with the same thing
happening
to us on a few CentOS 5.4 and 5.5 servers (we haven't tested the client on
5.6
yet). I'm consistently able to get our chef-client to segfault whenever it
handles this package resource (haven't had time to try other packages yet):

[Thu, 02 Jun 2011 17:38:17 -0700] INFO: Processing package[logrotate]
action
upgrade (logrotate::default line 20)

/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:75:
[BUG] Segmentation fault

Thu, 02 Jun 2011 17:38:27 -0700] INFO: Processing package[logrotate] action
upgrade (logrotate::default line 20)

/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:64:
[BUG] Segmentation fault

[Thu, 02 Jun 2011 17:38:53 -0700] INFO: Processing package[logrotate]
action
upgrade (logrotate::default line 20)

/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:66:
[BUG] Segmentation fault

We were trying this with ruby 1.8.7 p334 from the FrameOS rbel5 i386 repo.
Since it sounded like an issue with ruby, reinstalling with ruby 1.8.6 p420
(and rebuilding the rbel5 rubygem-chef rpm to relax the ruby version
requirement) seemed to solve the problem.

Is anyone else noticing seg fauls with a combination of CentOS 5.x, ruby
1.8.7
and Chef?

Thanks,
Ray

Hi James,

I originally noticed this happening on our older physical servers running
i386 CentOS 5.4 and 5.5, but I just created an EC2 instance and was able to
reproduce it. Here's what I did:

  1. Created a small instance using RightScale's 5.4 i386 v.5.5.9 AMI
    (ami-31df8e74)
  2. Added RBEL: rpm -Uvh http://rbel.frameos.org/rbel5
  3. Installed rubygem-chef (excluding EPEL so all the ruby packages are from
    RBEL): yum install --disablerepo=epel rubygem-chef
  4. Manually created a /etc/chef/client.rb and copied over validation.pem.
  5. Ran chef-client for the first time to register the node:
    /usr/bin/chef-client
  6. Added recipe[logrotate] to the run list (this is the cookbook from
    opscode)
  7. Re-ran chef-client which picked up the recipe and seg faults trying to
    run it

With regards to the ruby 1.8.6 that ended up working:

  1. I took Fedora 13's ruby-1.8.6.399-1.fc13.src.rpm, dropped in 1.8.6.420
    and repackaged on CentOS 5.5.
  2. Took Fedora 13's rubygems-1.3.6-1.fc13.src.rpm and repackaged it on
    CentOS 5.5 as is (but had to make a minor tweak to one of the patch files)
  3. Excluded ruby-, rubygems-, and rubygem-chef-0.10* in the RBEL yum
    config.
  4. Ran yum install rubygem-chef with my new RPMs.
    It pulled my repackaged RPMs (for ruby, ruby-lib, rubygems, and
    rubygem-chef) and everything else from RBEL.

With regards to my Chef server (if that helps), it's running Chef 0.10 with
all RBEL packages on a custom CentOS 5.6 x86_64 AMI.

Hope that helps!

Thanks,
Ray

On Thu, Jun 2, 2011 at 6:45 PM, James js@aegisco.com wrote:

Ray,

Can you provide the means to reproduce, specifically: steps to install from
the RBEL repo, and the AMI / image info? Are these i386 instances?

We maintain packages at rpm.aegisco.com, but I'm happy to try to help
debug the issues you're having. I'm surprised that Ruby 1.8.6 worked, I had
trouble getting 0.10 to work on anything older than 1.8.7 and rubygems
1.6.2.

James

On Thu, Jun 2, 2011 at 6:29 PM, tekkaray@yotowoti.com wrote:

Hi all,

I noticed a couple of email threads in the past about seg faults and
didn't see
any resolution so I wanted to share my experience with the same thing
happening
to us on a few CentOS 5.4 and 5.5 servers (we haven't tested the client on
5.6
yet). I'm consistently able to get our chef-client to segfault whenever it
handles this package resource (haven't had time to try other packages
yet):

[Thu, 02 Jun 2011 17:38:17 -0700] INFO: Processing package[logrotate]
action
upgrade (logrotate::default line 20)

/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:75:
[BUG] Segmentation fault

Thu, 02 Jun 2011 17:38:27 -0700] INFO: Processing package[logrotate]
action
upgrade (logrotate::default line 20)

/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:64:
[BUG] Segmentation fault

[Thu, 02 Jun 2011 17:38:53 -0700] INFO: Processing package[logrotate]
action
upgrade (logrotate::default line 20)

/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:66:
[BUG] Segmentation fault

We were trying this with ruby 1.8.7 p334 from the FrameOS rbel5 i386 repo.
Since it sounded like an issue with ruby, reinstalling with ruby 1.8.6
p420
(and rebuilding the rbel5 rubygem-chef rpm to relax the ruby version
requirement) seemed to solve the problem.

Is anyone else noticing seg fauls with a combination of CentOS 5.x, ruby
1.8.7
and Chef?

Thanks,
Ray

On Thu, Jun 2, 2011 at 6:29 PM, tekkaray@yotowoti.com wrote:

Hi all,

I noticed a couple of email threads in the past about seg faults and didn't see
any resolution so I wanted to share my experience with the same thing happening
to us on a few CentOS 5.4 and 5.5 servers (we haven't tested the client on 5.6
yet). I'm consistently able to get our chef-client to segfault whenever it
handles this package resource (haven't had time to try other packages yet):

[Thu, 02 Jun 2011 17:38:17 -0700] INFO: Processing package[logrotate] action
upgrade (logrotate::default line 20)
/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:75:
[BUG] Segmentation fault

Thu, 02 Jun 2011 17:38:27 -0700] INFO: Processing package[logrotate] action
upgrade (logrotate::default line 20)
/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:64:
[BUG] Segmentation fault

[Thu, 02 Jun 2011 17:38:53 -0700] INFO: Processing package[logrotate] action
upgrade (logrotate::default line 20)
/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:66:
[BUG] Segmentation fault

We were trying this with ruby 1.8.7 p334 from the FrameOS rbel5 i386 repo.
Since it sounded like an issue with ruby, reinstalling with ruby 1.8.6 p420
(and rebuilding the rbel5 rubygem-chef rpm to relax the ruby version
requirement) seemed to solve the problem.

Is anyone else noticing seg fauls with a combination of CentOS 5.x, ruby 1.8.7
and Chef?

Thanks,
Ray

If you have some time I'd be interested if these segfaults are still
triggered with the rewritten provider coming in 0.10.2. Given the
random nature of the line numbers mentioned it may be a dead end but
details are here

http://lists.opscode.com/sympa/arc/chef-dev/2011-05/msg00014.html

This work was recently merged into chef's master along with further
yum provider improvements, so it's best copied from there.

  • Matt

Ray,

The ami you gave doesn't seem to be public, but I found one which the same
CentOS and RightImage numbers: ami-2342a94a.

Can you double-check that it's a public image, so I can make sure to mirror
you as closely as possible?

Did you also experience this: http://tickets.opscode.com/browse/CHEF-2402

Thanks,

James

On Thu, Jun 2, 2011 at 8:19 PM, Raymond Tham tekkaray@yotowoti.com wrote:

Hi James,

I originally noticed this happening on our older physical servers running
i386 CentOS 5.4 and 5.5, but I just created an EC2 instance and was able to
reproduce it. Here's what I did:

  1. Created a small instance using RightScale's 5.4 i386 v.5.5.9 AMI
    (ami-31df8e74)
  2. Added RBEL: rpm -Uvh http://rbel.frameos.org/rbel5
  3. Installed rubygem-chef (excluding EPEL so all the ruby packages are from
    RBEL): yum install --disablerepo=epel rubygem-chef
  4. Manually created a /etc/chef/client.rb and copied over validation.pem.
  5. Ran chef-client for the first time to register the node:
    /usr/bin/chef-client
  6. Added recipe[logrotate] to the run list (this is the cookbook from
    opscode)
  7. Re-ran chef-client which picked up the recipe and seg faults trying to
    run it

With regards to the ruby 1.8.6 that ended up working:

  1. I took Fedora 13's ruby-1.8.6.399-1.fc13.src.rpm, dropped in 1.8.6.420
    and repackaged on CentOS 5.5.
  2. Took Fedora 13's rubygems-1.3.6-1.fc13.src.rpm and repackaged it on
    CentOS 5.5 as is (but had to make a minor tweak to one of the patch files)
  3. Excluded ruby-, rubygems-, and rubygem-chef-0.10* in the RBEL yum
    config.
  4. Ran yum install rubygem-chef with my new RPMs.
    It pulled my repackaged RPMs (for ruby, ruby-lib, rubygems, and
    rubygem-chef) and everything else from RBEL.

With regards to my Chef server (if that helps), it's running Chef 0.10 with
all RBEL packages on a custom CentOS 5.6 x86_64 AMI.

Hope that helps!

Thanks,
Ray

On Thu, Jun 2, 2011 at 6:45 PM, James js@aegisco.com wrote:

Ray,

Can you provide the means to reproduce, specifically: steps to install
from the RBEL repo, and the AMI / image info? Are these i386 instances?

We maintain packages at rpm.aegisco.com, but I'm happy to try to help
debug the issues you're having. I'm surprised that Ruby 1.8.6 worked, I had
trouble getting 0.10 to work on anything older than 1.8.7 and rubygems
1.6.2.

James

On Thu, Jun 2, 2011 at 6:29 PM, tekkaray@yotowoti.com wrote:

Hi all,

I noticed a couple of email threads in the past about seg faults and
didn't see
any resolution so I wanted to share my experience with the same thing
happening
to us on a few CentOS 5.4 and 5.5 servers (we haven't tested the client
on 5.6
yet). I'm consistently able to get our chef-client to segfault whenever
it
handles this package resource (haven't had time to try other packages
yet):

[Thu, 02 Jun 2011 17:38:17 -0700] INFO: Processing package[logrotate]
action
upgrade (logrotate::default line 20)

/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:75:
[BUG] Segmentation fault

Thu, 02 Jun 2011 17:38:27 -0700] INFO: Processing package[logrotate]
action
upgrade (logrotate::default line 20)

/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:64:
[BUG] Segmentation fault

[Thu, 02 Jun 2011 17:38:53 -0700] INFO: Processing package[logrotate]
action
upgrade (logrotate::default line 20)

/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:66:
[BUG] Segmentation fault

We were trying this with ruby 1.8.7 p334 from the FrameOS rbel5 i386
repo.
Since it sounded like an issue with ruby, reinstalling with ruby 1.8.6
p420
(and rebuilding the rbel5 rubygem-chef rpm to relax the ruby version
requirement) seemed to solve the problem.

Is anyone else noticing seg fauls with a combination of CentOS 5.x, ruby
1.8.7
and Chef?

Thanks,
Ray

Hi James,

Sorry, I should have mentioned that the AMI (ami-31df8e74) is in US West,
not East. It should be an S3 backed RightScale AMI.

I checked /root/.gemrc, and it points to
Index of /rubygems/archive/latest/. However,
the gems should have been installed through yum through the RBEL5 i386 repo.

If it helps, here's the output from "gem list" after installing through yum.

[root@ip-10-168-70-131 ~]# gem list

*** LOCAL GEMS ***

abstract (1.0.0)
activesupport (3.0.3)
allison (2.0.3)
bunny (0.6.0)
chef (0.10.0)
diff-lcs (1.1.2)
echoe (4.5.6)
erubis (2.6.6)
highline (1.6.1)
json (1.4.6)
mime-types (1.16)
mixlib-authentication (1.1.4)
mixlib-cli (1.2.0)
mixlib-config (1.1.2)
mixlib-log (1.3.0)
moneta (0.6.0)
net-sftp (2.0.4)
net-ssh (2.1.4, 2.0.23)
net-ssh-gateway (1.0.1)
net-ssh-multi (1.0.1)
ohai (0.6.4)
polyglot (0.3.1)
rake (0.8.7)
rake-compiler (0.7.5)
rest-client (1.6.1)
rspec (2.5.0)
rspec-core (2.5.1)
rspec-expectations (2.5.0)
rspec-mocks (2.5.0)
systemu (1.2.0)
treetop (1.4.9)
uuidtools (2.1.1)
xml-simple (1.0.12)
yajl-ruby (0.8.2)

On Thu, Jun 2, 2011 at 10:31 PM, James js@aegisco.com wrote:

Ray,

The ami you gave doesn't seem to be public, but I found one which the same
CentOS and RightImage numbers: ami-2342a94a.

Can you double-check that it's a public image, so I can make sure to mirror
you as closely as possible?

Did you also experience this: http://tickets.opscode.com/browse/CHEF-2402

Thanks,

James

On Thu, Jun 2, 2011 at 8:19 PM, Raymond Tham tekkaray@yotowoti.comwrote:

Hi James,

I originally noticed this happening on our older physical servers running
i386 CentOS 5.4 and 5.5, but I just created an EC2 instance and was able to
reproduce it. Here's what I did:

  1. Created a small instance using RightScale's 5.4 i386 v.5.5.9 AMI
    (ami-31df8e74)
  2. Added RBEL: rpm -Uvh http://rbel.frameos.org/rbel5
  3. Installed rubygem-chef (excluding EPEL so all the ruby packages are
    from RBEL): yum install --disablerepo=epel rubygem-chef
  4. Manually created a /etc/chef/client.rb and copied over validation.pem.
  5. Ran chef-client for the first time to register the node:
    /usr/bin/chef-client
  6. Added recipe[logrotate] to the run list (this is the cookbook from
    opscode)
  7. Re-ran chef-client which picked up the recipe and seg faults trying to
    run it

With regards to the ruby 1.8.6 that ended up working:

  1. I took Fedora 13's ruby-1.8.6.399-1.fc13.src.rpm, dropped in 1.8.6.420
    and repackaged on CentOS 5.5.
  2. Took Fedora 13's rubygems-1.3.6-1.fc13.src.rpm and repackaged it on
    CentOS 5.5 as is (but had to make a minor tweak to one of the patch files)
  3. Excluded ruby-, rubygems-, and rubygem-chef-0.10* in the RBEL yum
    config.
  4. Ran yum install rubygem-chef with my new RPMs.
    It pulled my repackaged RPMs (for ruby, ruby-lib, rubygems, and
    rubygem-chef) and everything else from RBEL.

With regards to my Chef server (if that helps), it's running Chef 0.10
with all RBEL packages on a custom CentOS 5.6 x86_64 AMI.

Hope that helps!

Thanks,
Ray

On Thu, Jun 2, 2011 at 6:45 PM, James js@aegisco.com wrote:

Ray,

Can you provide the means to reproduce, specifically: steps to install
from the RBEL repo, and the AMI / image info? Are these i386 instances?

We maintain packages at rpm.aegisco.com, but I'm happy to try to help
debug the issues you're having. I'm surprised that Ruby 1.8.6 worked, I had
trouble getting 0.10 to work on anything older than 1.8.7 and rubygems
1.6.2.

James

On Thu, Jun 2, 2011 at 6:29 PM, tekkaray@yotowoti.com wrote:

Hi all,

I noticed a couple of email threads in the past about seg faults and
didn't see
any resolution so I wanted to share my experience with the same thing
happening
to us on a few CentOS 5.4 and 5.5 servers (we haven't tested the client
on 5.6
yet). I'm consistently able to get our chef-client to segfault whenever
it
handles this package resource (haven't had time to try other packages
yet):

[Thu, 02 Jun 2011 17:38:17 -0700] INFO: Processing package[logrotate]
action
upgrade (logrotate::default line 20)

/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:75:
[BUG] Segmentation fault

Thu, 02 Jun 2011 17:38:27 -0700] INFO: Processing package[logrotate]
action
upgrade (logrotate::default line 20)

/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:64:
[BUG] Segmentation fault

[Thu, 02 Jun 2011 17:38:53 -0700] INFO: Processing package[logrotate]
action
upgrade (logrotate::default line 20)

/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:66:
[BUG] Segmentation fault

We were trying this with ruby 1.8.7 p334 from the FrameOS rbel5 i386
repo.
Since it sounded like an issue with ruby, reinstalling with ruby 1.8.6
p420
(and rebuilding the rbel5 rubygem-chef rpm to relax the ruby version
requirement) seemed to solve the problem.

Is anyone else noticing seg fauls with a combination of CentOS 5.x, ruby
1.8.7
and Chef?

Thanks,
Ray

The problem is in our (aegisco) build of 1.8.7 for i386, which RBEL is
using. We'll work on rebuilding that this weekend. All of your systems are
i386, correct?

Thanks a lot for the report and information!

James

On Thu, Jun 2, 2011 at 11:21 PM, Raymond Tham tekkaray@yotowoti.com wrote:

Hi James,

Sorry, I should have mentioned that the AMI (ami-31df8e74) is in US West,
not East. It should be an S3 backed RightScale AMI.

I checked /root/.gemrc, and it points to
Index of /rubygems/archive/latest/.
However, the gems should have been installed through yum through the RBEL5
i386 repo.

If it helps, here's the output from "gem list" after installing through
yum.

[root@ip-10-168-70-131 ~]# gem list

*** LOCAL GEMS ***

abstract (1.0.0)
activesupport (3.0.3)
allison (2.0.3)
bunny (0.6.0)
chef (0.10.0)
diff-lcs (1.1.2)
echoe (4.5.6)
erubis (2.6.6)
highline (1.6.1)
json (1.4.6)
mime-types (1.16)
mixlib-authentication (1.1.4)
mixlib-cli (1.2.0)
mixlib-config (1.1.2)
mixlib-log (1.3.0)
moneta (0.6.0)
net-sftp (2.0.4)
net-ssh (2.1.4, 2.0.23)
net-ssh-gateway (1.0.1)
net-ssh-multi (1.0.1)
ohai (0.6.4)
polyglot (0.3.1)
rake (0.8.7)
rake-compiler (0.7.5)
rest-client (1.6.1)
rspec (2.5.0)
rspec-core (2.5.1)
rspec-expectations (2.5.0)
rspec-mocks (2.5.0)
systemu (1.2.0)
treetop (1.4.9)
uuidtools (2.1.1)
xml-simple (1.0.12)
yajl-ruby (0.8.2)

On Thu, Jun 2, 2011 at 10:31 PM, James js@aegisco.com wrote:

Ray,

The ami you gave doesn't seem to be public, but I found one which the same
CentOS and RightImage numbers: ami-2342a94a.

Can you double-check that it's a public image, so I can make sure to
mirror you as closely as possible?

Did you also experience this: http://tickets.opscode.com/browse/CHEF-2402

Thanks,

James

On Thu, Jun 2, 2011 at 8:19 PM, Raymond Tham tekkaray@yotowoti.comwrote:

Hi James,

I originally noticed this happening on our older physical servers running
i386 CentOS 5.4 and 5.5, but I just created an EC2 instance and was able to
reproduce it. Here's what I did:

  1. Created a small instance using RightScale's 5.4 i386 v.5.5.9 AMI
    (ami-31df8e74)
  2. Added RBEL: rpm -Uvh http://rbel.frameos.org/rbel5
  3. Installed rubygem-chef (excluding EPEL so all the ruby packages are
    from RBEL): yum install --disablerepo=epel rubygem-chef
  4. Manually created a /etc/chef/client.rb and copied over validation.pem.
  5. Ran chef-client for the first time to register the node:
    /usr/bin/chef-client
  6. Added recipe[logrotate] to the run list (this is the cookbook from
    opscode)
  7. Re-ran chef-client which picked up the recipe and seg faults trying to
    run it

With regards to the ruby 1.8.6 that ended up working:

  1. I took Fedora 13's ruby-1.8.6.399-1.fc13.src.rpm, dropped in 1.8.6.420
    and repackaged on CentOS 5.5.
  2. Took Fedora 13's rubygems-1.3.6-1.fc13.src.rpm and repackaged it on
    CentOS 5.5 as is (but had to make a minor tweak to one of the patch files)
  3. Excluded ruby-, rubygems-, and rubygem-chef-0.10* in the RBEL yum
    config.
  4. Ran yum install rubygem-chef with my new RPMs.
    It pulled my repackaged RPMs (for ruby, ruby-lib, rubygems, and
    rubygem-chef) and everything else from RBEL.

With regards to my Chef server (if that helps), it's running Chef 0.10
with all RBEL packages on a custom CentOS 5.6 x86_64 AMI.

Hope that helps!

Thanks,
Ray

On Thu, Jun 2, 2011 at 6:45 PM, James js@aegisco.com wrote:

Ray,

Can you provide the means to reproduce, specifically: steps to install
from the RBEL repo, and the AMI / image info? Are these i386 instances?

We maintain packages at rpm.aegisco.com, but I'm happy to try to help
debug the issues you're having. I'm surprised that Ruby 1.8.6 worked, I had
trouble getting 0.10 to work on anything older than 1.8.7 and rubygems
1.6.2.

James

On Thu, Jun 2, 2011 at 6:29 PM, tekkaray@yotowoti.com wrote:

Hi all,

I noticed a couple of email threads in the past about seg faults and
didn't see
any resolution so I wanted to share my experience with the same thing
happening
to us on a few CentOS 5.4 and 5.5 servers (we haven't tested the client
on 5.6
yet). I'm consistently able to get our chef-client to segfault whenever
it
handles this package resource (haven't had time to try other packages
yet):

[Thu, 02 Jun 2011 17:38:17 -0700] INFO: Processing package[logrotate]
action
upgrade (logrotate::default line 20)

/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:75:
[BUG] Segmentation fault

Thu, 02 Jun 2011 17:38:27 -0700] INFO: Processing package[logrotate]
action
upgrade (logrotate::default line 20)

/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:64:
[BUG] Segmentation fault

[Thu, 02 Jun 2011 17:38:53 -0700] INFO: Processing package[logrotate]
action
upgrade (logrotate::default line 20)

/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:66:
[BUG] Segmentation fault

We were trying this with ruby 1.8.7 p334 from the FrameOS rbel5 i386
repo.
Since it sounded like an issue with ruby, reinstalling with ruby 1.8.6
p420
(and rebuilding the rbel5 rubygem-chef rpm to relax the ruby version
requirement) seemed to solve the problem.

Is anyone else noticing seg fauls with a combination of CentOS 5.x,
ruby 1.8.7
and Chef?

Thanks,
Ray

Matthew,

I did test the yum updates as well, and they did not resolve the issue. I'm
pretty certain it is our build of 1.8.7 for i386 causing the issues, which
will be affirmed if Ray confirms my assumption about using only i386
servers.

James

On Fri, Jun 3, 2011 at 1:44 AM, James js@aegisco.com wrote:

The problem is in our (aegisco) build of 1.8.7 for i386, which RBEL is
using. We'll work on rebuilding that this weekend. All of your systems are
i386, correct?

Thanks a lot for the report and information!

James

On Thu, Jun 2, 2011 at 11:21 PM, Raymond Tham tekkaray@yotowoti.comwrote:

Hi James,

Sorry, I should have mentioned that the AMI (ami-31df8e74) is in US West,
not East. It should be an S3 backed RightScale AMI.

I checked /root/.gemrc, and it points to
Index of /rubygems/archive/latest/.
However, the gems should have been installed through yum through the RBEL5
i386 repo.

If it helps, here's the output from "gem list" after installing through
yum.

[root@ip-10-168-70-131 ~]# gem list

*** LOCAL GEMS ***

abstract (1.0.0)
activesupport (3.0.3)
allison (2.0.3)
bunny (0.6.0)
chef (0.10.0)
diff-lcs (1.1.2)
echoe (4.5.6)
erubis (2.6.6)
highline (1.6.1)
json (1.4.6)
mime-types (1.16)
mixlib-authentication (1.1.4)
mixlib-cli (1.2.0)
mixlib-config (1.1.2)
mixlib-log (1.3.0)
moneta (0.6.0)
net-sftp (2.0.4)
net-ssh (2.1.4, 2.0.23)
net-ssh-gateway (1.0.1)
net-ssh-multi (1.0.1)
ohai (0.6.4)
polyglot (0.3.1)
rake (0.8.7)
rake-compiler (0.7.5)
rest-client (1.6.1)
rspec (2.5.0)
rspec-core (2.5.1)
rspec-expectations (2.5.0)
rspec-mocks (2.5.0)
systemu (1.2.0)
treetop (1.4.9)
uuidtools (2.1.1)
xml-simple (1.0.12)
yajl-ruby (0.8.2)

On Thu, Jun 2, 2011 at 10:31 PM, James js@aegisco.com wrote:

Ray,

The ami you gave doesn't seem to be public, but I found one which the
same CentOS and RightImage numbers: ami-2342a94a.

Can you double-check that it's a public image, so I can make sure to
mirror you as closely as possible?

Did you also experience this:
http://tickets.opscode.com/browse/CHEF-2402

Thanks,

James

On Thu, Jun 2, 2011 at 8:19 PM, Raymond Tham tekkaray@yotowoti.comwrote:

Hi James,

I originally noticed this happening on our older physical servers
running i386 CentOS 5.4 and 5.5, but I just created an EC2 instance and was
able to reproduce it. Here's what I did:

  1. Created a small instance using RightScale's 5.4 i386 v.5.5.9 AMI
    (ami-31df8e74)
  2. Added RBEL: rpm -Uvh http://rbel.frameos.org/rbel5
  3. Installed rubygem-chef (excluding EPEL so all the ruby packages are
    from RBEL): yum install --disablerepo=epel rubygem-chef
  4. Manually created a /etc/chef/client.rb and copied over
    validation.pem.
  5. Ran chef-client for the first time to register the node:
    /usr/bin/chef-client
  6. Added recipe[logrotate] to the run list (this is the cookbook from
    opscode)
  7. Re-ran chef-client which picked up the recipe and seg faults trying
    to run it

With regards to the ruby 1.8.6 that ended up working:

  1. I took Fedora 13's ruby-1.8.6.399-1.fc13.src.rpm, dropped in
    1.8.6.420 and repackaged on CentOS 5.5.
  2. Took Fedora 13's rubygems-1.3.6-1.fc13.src.rpm and repackaged it on
    CentOS 5.5 as is (but had to make a minor tweak to one of the patch files)
  3. Excluded ruby-, rubygems-, and rubygem-chef-0.10* in the RBEL yum
    config.
  4. Ran yum install rubygem-chef with my new RPMs.
    It pulled my repackaged RPMs (for ruby, ruby-lib, rubygems, and
    rubygem-chef) and everything else from RBEL.

With regards to my Chef server (if that helps), it's running Chef 0.10
with all RBEL packages on a custom CentOS 5.6 x86_64 AMI.

Hope that helps!

Thanks,
Ray

On Thu, Jun 2, 2011 at 6:45 PM, James js@aegisco.com wrote:

Ray,

Can you provide the means to reproduce, specifically: steps to install
from the RBEL repo, and the AMI / image info? Are these i386 instances?

We maintain packages at rpm.aegisco.com, but I'm happy to try to help
debug the issues you're having. I'm surprised that Ruby 1.8.6 worked, I had
trouble getting 0.10 to work on anything older than 1.8.7 and rubygems
1.6.2.

James

On Thu, Jun 2, 2011 at 6:29 PM, tekkaray@yotowoti.com wrote:

Hi all,

I noticed a couple of email threads in the past about seg faults and
didn't see
any resolution so I wanted to share my experience with the same thing
happening
to us on a few CentOS 5.4 and 5.5 servers (we haven't tested the
client on 5.6
yet). I'm consistently able to get our chef-client to segfault
whenever it
handles this package resource (haven't had time to try other packages
yet):

[Thu, 02 Jun 2011 17:38:17 -0700] INFO: Processing package[logrotate]
action
upgrade (logrotate::default line 20)

/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:75:
[BUG] Segmentation fault

Thu, 02 Jun 2011 17:38:27 -0700] INFO: Processing package[logrotate]
action
upgrade (logrotate::default line 20)

/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:64:
[BUG] Segmentation fault

[Thu, 02 Jun 2011 17:38:53 -0700] INFO: Processing package[logrotate]
action
upgrade (logrotate::default line 20)

/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:66:
[BUG] Segmentation fault

We were trying this with ruby 1.8.7 p334 from the FrameOS rbel5 i386
repo.
Since it sounded like an issue with ruby, reinstalling with ruby 1.8.6
p420
(and rebuilding the rbel5 rubygem-chef rpm to relax the ruby version
requirement) seemed to solve the problem.

Is anyone else noticing seg fauls with a combination of CentOS 5.x,
ruby 1.8.7
and Chef?

Thanks,
Ray

That's right, we noticed the seg faults happening on our i386 systems; we
don't have chef-client set up on any x86_64 instances at the moment.

Thanks for looking into this!

On Fri, Jun 3, 2011 at 1:44 AM, James js@aegisco.com wrote:

The problem is in our (aegisco) build of 1.8.7 for i386, which RBEL is
using. We'll work on rebuilding that this weekend. All of your systems are
i386, correct?

Thanks a lot for the report and information!

James

On Thu, Jun 2, 2011 at 11:21 PM, Raymond Tham tekkaray@yotowoti.comwrote:

Hi James,

Sorry, I should have mentioned that the AMI (ami-31df8e74) is in US West,
not East. It should be an S3 backed RightScale AMI.

I checked /root/.gemrc, and it points to
Index of /rubygems/archive/latest/.
However, the gems should have been installed through yum through the RBEL5
i386 repo.

If it helps, here's the output from "gem list" after installing through
yum.

[root@ip-10-168-70-131 ~]# gem list

*** LOCAL GEMS ***

abstract (1.0.0)
activesupport (3.0.3)
allison (2.0.3)
bunny (0.6.0)
chef (0.10.0)
diff-lcs (1.1.2)
echoe (4.5.6)
erubis (2.6.6)
highline (1.6.1)
json (1.4.6)
mime-types (1.16)
mixlib-authentication (1.1.4)
mixlib-cli (1.2.0)
mixlib-config (1.1.2)
mixlib-log (1.3.0)
moneta (0.6.0)
net-sftp (2.0.4)
net-ssh (2.1.4, 2.0.23)
net-ssh-gateway (1.0.1)
net-ssh-multi (1.0.1)
ohai (0.6.4)
polyglot (0.3.1)
rake (0.8.7)
rake-compiler (0.7.5)
rest-client (1.6.1)
rspec (2.5.0)
rspec-core (2.5.1)
rspec-expectations (2.5.0)
rspec-mocks (2.5.0)
systemu (1.2.0)
treetop (1.4.9)
uuidtools (2.1.1)
xml-simple (1.0.12)
yajl-ruby (0.8.2)

On Thu, Jun 2, 2011 at 10:31 PM, James js@aegisco.com wrote:

Ray,

The ami you gave doesn't seem to be public, but I found one which the
same CentOS and RightImage numbers: ami-2342a94a.

Can you double-check that it's a public image, so I can make sure to
mirror you as closely as possible?

Did you also experience this:
http://tickets.opscode.com/browse/CHEF-2402

Thanks,

James

On Thu, Jun 2, 2011 at 8:19 PM, Raymond Tham tekkaray@yotowoti.comwrote:

Hi James,

I originally noticed this happening on our older physical servers
running i386 CentOS 5.4 and 5.5, but I just created an EC2 instance and was
able to reproduce it. Here's what I did:

  1. Created a small instance using RightScale's 5.4 i386 v.5.5.9 AMI
    (ami-31df8e74)
  2. Added RBEL: rpm -Uvh http://rbel.frameos.org/rbel5
  3. Installed rubygem-chef (excluding EPEL so all the ruby packages are
    from RBEL): yum install --disablerepo=epel rubygem-chef
  4. Manually created a /etc/chef/client.rb and copied over
    validation.pem.
  5. Ran chef-client for the first time to register the node:
    /usr/bin/chef-client
  6. Added recipe[logrotate] to the run list (this is the cookbook from
    opscode)
  7. Re-ran chef-client which picked up the recipe and seg faults trying
    to run it

With regards to the ruby 1.8.6 that ended up working:

  1. I took Fedora 13's ruby-1.8.6.399-1.fc13.src.rpm, dropped in
    1.8.6.420 and repackaged on CentOS 5.5.
  2. Took Fedora 13's rubygems-1.3.6-1.fc13.src.rpm and repackaged it on
    CentOS 5.5 as is (but had to make a minor tweak to one of the patch files)
  3. Excluded ruby-, rubygems-, and rubygem-chef-0.10* in the RBEL yum
    config.
  4. Ran yum install rubygem-chef with my new RPMs.
    It pulled my repackaged RPMs (for ruby, ruby-lib, rubygems, and
    rubygem-chef) and everything else from RBEL.

With regards to my Chef server (if that helps), it's running Chef 0.10
with all RBEL packages on a custom CentOS 5.6 x86_64 AMI.

Hope that helps!

Thanks,
Ray

On Thu, Jun 2, 2011 at 6:45 PM, James js@aegisco.com wrote:

Ray,

Can you provide the means to reproduce, specifically: steps to install
from the RBEL repo, and the AMI / image info? Are these i386 instances?

We maintain packages at rpm.aegisco.com, but I'm happy to try to help
debug the issues you're having. I'm surprised that Ruby 1.8.6 worked, I had
trouble getting 0.10 to work on anything older than 1.8.7 and rubygems
1.6.2.

James

On Thu, Jun 2, 2011 at 6:29 PM, tekkaray@yotowoti.com wrote:

Hi all,

I noticed a couple of email threads in the past about seg faults and
didn't see
any resolution so I wanted to share my experience with the same thing
happening
to us on a few CentOS 5.4 and 5.5 servers (we haven't tested the
client on 5.6
yet). I'm consistently able to get our chef-client to segfault
whenever it
handles this package resource (haven't had time to try other packages
yet):

[Thu, 02 Jun 2011 17:38:17 -0700] INFO: Processing package[logrotate]
action
upgrade (logrotate::default line 20)

/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:75:
[BUG] Segmentation fault

Thu, 02 Jun 2011 17:38:27 -0700] INFO: Processing package[logrotate]
action
upgrade (logrotate::default line 20)

/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:64:
[BUG] Segmentation fault

[Thu, 02 Jun 2011 17:38:53 -0700] INFO: Processing package[logrotate]
action
upgrade (logrotate::default line 20)

/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:66:
[BUG] Segmentation fault

We were trying this with ruby 1.8.7 p334 from the FrameOS rbel5 i386
repo.
Since it sounded like an issue with ruby, reinstalling with ruby 1.8.6
p420
(and rebuilding the rbel5 rubygem-chef rpm to relax the ruby version
requirement) seemed to solve the problem.

Is anyone else noticing seg fauls with a combination of CentOS 5.x,
ruby 1.8.7
and Chef?

Thanks,
Ray

Hey guys,

On Fri, Jun 3, 2011 at 11:05 AM, Raymond Tham tekkaray@yotowoti.com wrote:

That's right, we noticed the seg faults happening on our i386 systems; we
don't have chef-client set up on any x86_64 instances at the moment.

I wonder if you made any progress with this stuff. Been swamped lately, but
I closely follow they list and RPM related threads. I'll get some fresh air
soon and I was wondering if testing i386 packages is the best way to help
here right now.

Thanks!

Thanks for looking into this!

On Fri, Jun 3, 2011 at 1:44 AM, James js@aegisco.com wrote:

The problem is in our (aegisco) build of 1.8.7 for i386, which RBEL is
using. We'll work on rebuilding that this weekend. All of your systems are
i386, correct?

Thanks a lot for the report and information!

James

On Thu, Jun 2, 2011 at 11:21 PM, Raymond Tham tekkaray@yotowoti.comwrote:

Hi James,

Sorry, I should have mentioned that the AMI (ami-31df8e74) is in US West,
not East. It should be an S3 backed RightScale AMI.

I checked /root/.gemrc, and it points to
Index of /rubygems/archive/latest/.
However, the gems should have been installed through yum through the RBEL5
i386 repo.

If it helps, here's the output from "gem list" after installing through
yum.

[root@ip-10-168-70-131 ~]# gem list

*** LOCAL GEMS ***

abstract (1.0.0)
activesupport (3.0.3)
allison (2.0.3)
bunny (0.6.0)
chef (0.10.0)
diff-lcs (1.1.2)
echoe (4.5.6)
erubis (2.6.6)
highline (1.6.1)
json (1.4.6)
mime-types (1.16)
mixlib-authentication (1.1.4)
mixlib-cli (1.2.0)
mixlib-config (1.1.2)
mixlib-log (1.3.0)
moneta (0.6.0)
net-sftp (2.0.4)
net-ssh (2.1.4, 2.0.23)
net-ssh-gateway (1.0.1)
net-ssh-multi (1.0.1)
ohai (0.6.4)
polyglot (0.3.1)
rake (0.8.7)
rake-compiler (0.7.5)
rest-client (1.6.1)
rspec (2.5.0)
rspec-core (2.5.1)
rspec-expectations (2.5.0)
rspec-mocks (2.5.0)
systemu (1.2.0)
treetop (1.4.9)
uuidtools (2.1.1)
xml-simple (1.0.12)
yajl-ruby (0.8.2)

On Thu, Jun 2, 2011 at 10:31 PM, James js@aegisco.com wrote:

Ray,

The ami you gave doesn't seem to be public, but I found one which the
same CentOS and RightImage numbers: ami-2342a94a.

Can you double-check that it's a public image, so I can make sure to
mirror you as closely as possible?

Did you also experience this:
http://tickets.opscode.com/browse/CHEF-2402

Thanks,

James

On Thu, Jun 2, 2011 at 8:19 PM, Raymond Tham tekkaray@yotowoti.comwrote:

Hi James,

I originally noticed this happening on our older physical servers
running i386 CentOS 5.4 and 5.5, but I just created an EC2 instance and was
able to reproduce it. Here's what I did:

  1. Created a small instance using RightScale's 5.4 i386 v.5.5.9 AMI
    (ami-31df8e74)
  2. Added RBEL: rpm -Uvh http://rbel.frameos.org/rbel5
  3. Installed rubygem-chef (excluding EPEL so all the ruby packages are
    from RBEL): yum install --disablerepo=epel rubygem-chef
  4. Manually created a /etc/chef/client.rb and copied over
    validation.pem.
  5. Ran chef-client for the first time to register the node:
    /usr/bin/chef-client
  6. Added recipe[logrotate] to the run list (this is the cookbook from
    opscode)
  7. Re-ran chef-client which picked up the recipe and seg faults trying
    to run it

With regards to the ruby 1.8.6 that ended up working:

  1. I took Fedora 13's ruby-1.8.6.399-1.fc13.src.rpm, dropped in
    1.8.6.420 and repackaged on CentOS 5.5.
  2. Took Fedora 13's rubygems-1.3.6-1.fc13.src.rpm and repackaged it on
    CentOS 5.5 as is (but had to make a minor tweak to one of the patch files)
  3. Excluded ruby-, rubygems-, and rubygem-chef-0.10* in the RBEL yum
    config.
  4. Ran yum install rubygem-chef with my new RPMs.
    It pulled my repackaged RPMs (for ruby, ruby-lib, rubygems, and
    rubygem-chef) and everything else from RBEL.

With regards to my Chef server (if that helps), it's running Chef 0.10
with all RBEL packages on a custom CentOS 5.6 x86_64 AMI.

Hope that helps!

Thanks,
Ray

On Thu, Jun 2, 2011 at 6:45 PM, James js@aegisco.com wrote:

Ray,

Can you provide the means to reproduce, specifically: steps to install
from the RBEL repo, and the AMI / image info? Are these i386 instances?

We maintain packages at rpm.aegisco.com, but I'm happy to try to help
debug the issues you're having. I'm surprised that Ruby 1.8.6 worked, I had
trouble getting 0.10 to work on anything older than 1.8.7 and rubygems
1.6.2.

James

On Thu, Jun 2, 2011 at 6:29 PM, tekkaray@yotowoti.com wrote:

Hi all,

I noticed a couple of email threads in the past about seg faults and
didn't see
any resolution so I wanted to share my experience with the same thing
happening
to us on a few CentOS 5.4 and 5.5 servers (we haven't tested the
client on 5.6
yet). I'm consistently able to get our chef-client to segfault
whenever it
handles this package resource (haven't had time to try other packages
yet):

[Thu, 02 Jun 2011 17:38:17 -0700] INFO: Processing package[logrotate]
action
upgrade (logrotate::default line 20)

/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:75:
[BUG] Segmentation fault

Thu, 02 Jun 2011 17:38:27 -0700] INFO: Processing package[logrotate]
action
upgrade (logrotate::default line 20)

/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:64:
[BUG] Segmentation fault

[Thu, 02 Jun 2011 17:38:53 -0700] INFO: Processing package[logrotate]
action
upgrade (logrotate::default line 20)

/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:66:
[BUG] Segmentation fault

We were trying this with ruby 1.8.7 p334 from the FrameOS rbel5 i386
repo.
Since it sounded like an issue with ruby, reinstalling with ruby
1.8.6 p420
(and rebuilding the rbel5 rubygem-chef rpm to relax the ruby version
requirement) seemed to solve the problem.

Is anyone else noticing seg fauls with a combination of CentOS 5.x,
ruby 1.8.7
and Chef?

Thanks,
Ray

These issues seem to keep cropping up. I'm wondering if this really
means that its just time to press on to Ruby 1.9.2 and leave 1.8.x
behind. I also noticed when I got my Velocity registration today that
the recommended steps for installing Chef for use in the Chef training
session suggested using Ruby 1.9.2.

Fortunately for me, I don't run Ruby infrastructure for anything other
than Chef so I don't really care much. Just want it to work and
minimize change. Still looking forward to a fatty installer.

KC

On Thu, Jun 9, 2011 at 12:30 AM, Sergio Rubio rubiojr@frameos.org wrote:

Hey guys,
On Fri, Jun 3, 2011 at 11:05 AM, Raymond Tham tekkaray@yotowoti.com wrote:

That's right, we noticed the seg faults happening on our i386 systems; we
don't have chef-client set up on any x86_64 instances at the moment.

I wonder if you made any progress with this stuff. Been swamped lately, but
I closely follow they list and RPM related threads. I'll get some fresh air
soon and I was wondering if testing i386 packages is the best way to help
here right now.
Thanks!

Thanks for looking into this!

On Fri, Jun 3, 2011 at 1:44 AM, James js@aegisco.com wrote:

The problem is in our (aegisco) build of 1.8.7 for i386, which RBEL is
using. We'll work on rebuilding that this weekend. All of your systems are
i386, correct?
Thanks a lot for the report and information!
James
On Thu, Jun 2, 2011 at 11:21 PM, Raymond Tham tekkaray@yotowoti.com
wrote:

Hi James,
Sorry, I should have mentioned that the AMI (ami-31df8e74) is in US
West, not East. It should be an S3 backed RightScale AMI.
I checked /root/.gemrc, and it points
to Index of /rubygems/archive/latest/.
However, the gems should have been installed through yum through the RBEL5
i386 repo.
If it helps, here's the output from "gem list" after installing through
yum.
[root@ip-10-168-70-131 ~]# gem list
*** LOCAL GEMS ***
abstract (1.0.0)
activesupport (3.0.3)
allison (2.0.3)
bunny (0.6.0)
chef (0.10.0)
diff-lcs (1.1.2)
echoe (4.5.6)
erubis (2.6.6)
highline (1.6.1)
json (1.4.6)
mime-types (1.16)
mixlib-authentication (1.1.4)
mixlib-cli (1.2.0)
mixlib-config (1.1.2)
mixlib-log (1.3.0)
moneta (0.6.0)
net-sftp (2.0.4)
net-ssh (2.1.4, 2.0.23)
net-ssh-gateway (1.0.1)
net-ssh-multi (1.0.1)
ohai (0.6.4)
polyglot (0.3.1)
rake (0.8.7)
rake-compiler (0.7.5)
rest-client (1.6.1)
rspec (2.5.0)
rspec-core (2.5.1)
rspec-expectations (2.5.0)
rspec-mocks (2.5.0)
systemu (1.2.0)
treetop (1.4.9)
uuidtools (2.1.1)
xml-simple (1.0.12)
yajl-ruby (0.8.2)

On Thu, Jun 2, 2011 at 10:31 PM, James js@aegisco.com wrote:

Ray,
The ami you gave doesn't seem to be public, but I found one which the
same CentOS and RightImage numbers: ami-2342a94a.
Can you double-check that it's a public image, so I can make sure to
mirror you as closely as possible?
Did you also experience
this: http://tickets.opscode.com/browse/CHEF-2402
Thanks,
James
On Thu, Jun 2, 2011 at 8:19 PM, Raymond Tham tekkaray@yotowoti.com
wrote:

Hi James,
I originally noticed this happening on our older physical servers
running i386 CentOS 5.4 and 5.5, but I just created an EC2 instance and was
able to reproduce it. Here's what I did:

  1. Created a small instance using RightScale's 5.4 i386 v.5.5.9 AMI
    (ami-31df8e74)
  2. Added RBEL: rpm -Uvh http://rbel.frameos.org/rbel5
  3. Installed rubygem-chef (excluding EPEL so all the ruby packages are
    from RBEL): yum install --disablerepo=epel rubygem-chef
  4. Manually created a /etc/chef/client.rb and copied over
    validation.pem.
  5. Ran chef-client for the first time to register the node:
    /usr/bin/chef-client
  6. Added recipe[logrotate] to the run list (this is the cookbook from
    opscode)
  7. Re-ran chef-client which picked up the recipe and seg faults trying
    to run it
    With regards to the ruby 1.8.6 that ended up working:
  8. I took Fedora 13's ruby-1.8.6.399-1.fc13.src.rpm, dropped in
    1.8.6.420 and repackaged on CentOS 5.5.
  9. Took Fedora 13's rubygems-1.3.6-1.fc13.src.rpm and repackaged it on
    CentOS 5.5 as is (but had to make a minor tweak to one of the patch files)
  10. Excluded ruby-, rubygems-, and rubygem-chef-0.10* in the RBEL yum
    config.
  11. Ran yum install rubygem-chef with my new RPMs.
    It pulled my repackaged RPMs (for ruby, ruby-lib, rubygems, and
    rubygem-chef) and everything else from RBEL.
    With regards to my Chef server (if that helps), it's running Chef 0.10
    with all RBEL packages on a custom CentOS 5.6 x86_64 AMI.
    Hope that helps!
    Thanks,
    Ray
    On Thu, Jun 2, 2011 at 6:45 PM, James js@aegisco.com wrote:

Ray,
Can you provide the means to reproduce, specifically: steps to
install from the RBEL repo, and the AMI / image info? Are these i386
instances?
We maintain packages at rpm.aegisco.com, but I'm happy to try to help
debug the issues you're having. I'm surprised that Ruby 1.8.6 worked, I had
trouble getting 0.10 to work on anything older than 1.8.7 and rubygems
1.6.2.
James

On Thu, Jun 2, 2011 at 6:29 PM, tekkaray@yotowoti.com wrote:

Hi all,

I noticed a couple of email threads in the past about seg faults and
didn't see
any resolution so I wanted to share my experience with the same
thing happening
to us on a few CentOS 5.4 and 5.5 servers (we haven't tested the
client on 5.6
yet). I'm consistently able to get our chef-client to segfault
whenever it
handles this package resource (haven't had time to try other
packages yet):

[Thu, 02 Jun 2011 17:38:17 -0700] INFO: Processing
package[logrotate] action
upgrade (logrotate::default line 20)

/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:75:
[BUG] Segmentation fault

Thu, 02 Jun 2011 17:38:27 -0700] INFO: Processing package[logrotate]
action
upgrade (logrotate::default line 20)

/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:64:
[BUG] Segmentation fault

[Thu, 02 Jun 2011 17:38:53 -0700] INFO: Processing
package[logrotate] action
upgrade (logrotate::default line 20)

/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:66:
[BUG] Segmentation fault

We were trying this with ruby 1.8.7 p334 from the FrameOS rbel5 i386
repo.
Since it sounded like an issue with ruby, reinstalling with ruby
1.8.6 p420
(and rebuilding the rbel5 rubygem-chef rpm to relax the ruby version
requirement) seemed to solve the problem.

Is anyone else noticing seg fauls with a combination of CentOS 5.x,
ruby 1.8.7
and Chef?

Thanks,
Ray

On Thu, Jun 9, 2011 at 9:46 AM, KC Braunschweig kcbraunschweig@gmail.comwrote:

These issues seem to keep cropping up. I'm wondering if this really
means that its just time to press on to Ruby 1.9.2 and leave 1.8.x
behind. I also noticed when I got my Velocity registration today that
the recommended steps for installing Chef for use in the Chef training
session suggested using Ruby 1.9.2.

Fortunately for me, I don't run Ruby infrastructure for anything other
than Chef so I don't really care much. Just want it to work and
minimize change. Still looking forward to a fatty installer.

I'm dumb testing previous patch levels of ruby right now. I've uploaded i386
pacakges to http://rbel.frameos.org/misc if you are interested:

rpm -Uvh --force http://rbel.frameos.org/misc/ruby-1.8.7.302-1.el5.i386.rpm
http://rbel.frameos.org/misc/ruby-libs-1.8.7.302-1.el5.i386.rpm

should do the trick downgrading your ruby version if it's higher. Patch
level 302 was included by F14.
Besides that, RBEL ruby-1.9 is parallel installable with 1.8 if that helps.

Cheers!

KC

On Thu, Jun 9, 2011 at 12:30 AM, Sergio Rubio rubiojr@frameos.org wrote:

Hey guys,
On Fri, Jun 3, 2011 at 11:05 AM, Raymond Tham tekkaray@yotowoti.com
wrote:

That's right, we noticed the seg faults happening on our i386 systems;
we
don't have chef-client set up on any x86_64 instances at the moment.

I wonder if you made any progress with this stuff. Been swamped lately,
but
I closely follow they list and RPM related threads. I'll get some fresh
air
soon and I was wondering if testing i386 packages is the best way to help
here right now.
Thanks!

Thanks for looking into this!

On Fri, Jun 3, 2011 at 1:44 AM, James js@aegisco.com wrote:

The problem is in our (aegisco) build of 1.8.7 for i386, which RBEL is
using. We'll work on rebuilding that this weekend. All of your systems
are
i386, correct?
Thanks a lot for the report and information!
James
On Thu, Jun 2, 2011 at 11:21 PM, Raymond Tham tekkaray@yotowoti.com
wrote:

Hi James,
Sorry, I should have mentioned that the AMI (ami-31df8e74) is in US
West, not East. It should be an S3 backed RightScale AMI.
I checked /root/.gemrc, and it points
to Index of /rubygems/archive/latest/.
However, the gems should have been installed through yum through the
RBEL5
i386 repo.
If it helps, here's the output from "gem list" after installing
through
yum.
[root@ip-10-168-70-131 ~]# gem list
*** LOCAL GEMS ***
abstract (1.0.0)
activesupport (3.0.3)
allison (2.0.3)
bunny (0.6.0)
chef (0.10.0)
diff-lcs (1.1.2)
echoe (4.5.6)
erubis (2.6.6)
highline (1.6.1)
json (1.4.6)
mime-types (1.16)
mixlib-authentication (1.1.4)
mixlib-cli (1.2.0)
mixlib-config (1.1.2)
mixlib-log (1.3.0)
moneta (0.6.0)
net-sftp (2.0.4)
net-ssh (2.1.4, 2.0.23)
net-ssh-gateway (1.0.1)
net-ssh-multi (1.0.1)
ohai (0.6.4)
polyglot (0.3.1)
rake (0.8.7)
rake-compiler (0.7.5)
rest-client (1.6.1)
rspec (2.5.0)
rspec-core (2.5.1)
rspec-expectations (2.5.0)
rspec-mocks (2.5.0)
systemu (1.2.0)
treetop (1.4.9)
uuidtools (2.1.1)
xml-simple (1.0.12)
yajl-ruby (0.8.2)

On Thu, Jun 2, 2011 at 10:31 PM, James js@aegisco.com wrote:

Ray,
The ami you gave doesn't seem to be public, but I found one which the
same CentOS and RightImage numbers: ami-2342a94a.
Can you double-check that it's a public image, so I can make sure to
mirror you as closely as possible?
Did you also experience
this: http://tickets.opscode.com/browse/CHEF-2402
Thanks,
James
On Thu, Jun 2, 2011 at 8:19 PM, Raymond Tham tekkaray@yotowoti.com
wrote:

Hi James,
I originally noticed this happening on our older physical servers
running i386 CentOS 5.4 and 5.5, but I just created an EC2 instance
and was
able to reproduce it. Here's what I did:

  1. Created a small instance using RightScale's 5.4 i386 v.5.5.9 AMI
    (ami-31df8e74)
  2. Added RBEL: rpm -Uvh http://rbel.frameos.org/rbel5
  3. Installed rubygem-chef (excluding EPEL so all the ruby packages
    are
    from RBEL): yum install --disablerepo=epel rubygem-chef
  4. Manually created a /etc/chef/client.rb and copied over
    validation.pem.
  5. Ran chef-client for the first time to register the node:
    /usr/bin/chef-client
  6. Added recipe[logrotate] to the run list (this is the cookbook
    from
    opscode)
  7. Re-ran chef-client which picked up the recipe and seg faults
    trying
    to run it
    With regards to the ruby 1.8.6 that ended up working:
  8. I took Fedora 13's ruby-1.8.6.399-1.fc13.src.rpm, dropped in
    1.8.6.420 and repackaged on CentOS 5.5.
  9. Took Fedora 13's rubygems-1.3.6-1.fc13.src.rpm and repackaged it
    on
    CentOS 5.5 as is (but had to make a minor tweak to one of the patch
    files)
  10. Excluded ruby-, rubygems-, and rubygem-chef-0.10* in the RBEL
    yum
    config.
  11. Ran yum install rubygem-chef with my new RPMs.
    It pulled my repackaged RPMs (for ruby, ruby-lib, rubygems, and
    rubygem-chef) and everything else from RBEL.
    With regards to my Chef server (if that helps), it's running Chef
    0.10
    with all RBEL packages on a custom CentOS 5.6 x86_64 AMI.
    Hope that helps!
    Thanks,
    Ray
    On Thu, Jun 2, 2011 at 6:45 PM, James js@aegisco.com wrote:

Ray,
Can you provide the means to reproduce, specifically: steps to
install from the RBEL repo, and the AMI / image info? Are these
i386
instances?
We maintain packages at rpm.aegisco.com, but I'm happy to try to
help
debug the issues you're having. I'm surprised that Ruby 1.8.6
worked, I had
trouble getting 0.10 to work on anything older than 1.8.7 and
rubygems
1.6.2.
James

On Thu, Jun 2, 2011 at 6:29 PM, tekkaray@yotowoti.com wrote:

Hi all,

I noticed a couple of email threads in the past about seg faults
and
didn't see
any resolution so I wanted to share my experience with the same
thing happening
to us on a few CentOS 5.4 and 5.5 servers (we haven't tested the
client on 5.6
yet). I'm consistently able to get our chef-client to segfault
whenever it
handles this package resource (haven't had time to try other
packages yet):

[Thu, 02 Jun 2011 17:38:17 -0700] INFO: Processing
package[logrotate] action
upgrade (logrotate::default line 20)

/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:75:

[BUG] Segmentation fault

Thu, 02 Jun 2011 17:38:27 -0700] INFO: Processing
package[logrotate]
action
upgrade (logrotate::default line 20)

/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:64:

[BUG] Segmentation fault

[Thu, 02 Jun 2011 17:38:53 -0700] INFO: Processing
package[logrotate] action
upgrade (logrotate::default line 20)

/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:66:

[BUG] Segmentation fault

We were trying this with ruby 1.8.7 p334 from the FrameOS rbel5
i386
repo.
Since it sounded like an issue with ruby, reinstalling with ruby
1.8.6 p420
(and rebuilding the rbel5 rubygem-chef rpm to relax the ruby
version
requirement) seemed to solve the problem.

Is anyone else noticing seg fauls with a combination of CentOS
5.x,
ruby 1.8.7
and Chef?

Thanks,
Ray

Sergio,

The problem is definitely the ruby 1.8.7 build for el5 on i386. If I
remember correctly, we built this using mock, and had a lot of trouble with
it. We have since switched to using native VMs for building packages, and
will be releasing a new round very soon. This round will also resolve
CHEF-2388 http://tickets.opscode.com/browse/CHEF-2388.

I'm not opposed to switching to, or at least including 1.9. One challenge is
that every other ruby package in the CentOS repos use the 1.8 ABI; to run
1.9 you either need to not install any of these packages, or have both 1.8
and 1.9 installed.

James

On Thu, Jun 9, 2011 at 1:25 AM, Sergio Rubio rubiojr@frameos.org wrote:

On Thu, Jun 9, 2011 at 9:46 AM, KC Braunschweig kcbraunschweig@gmail.comwrote:

These issues seem to keep cropping up. I'm wondering if this really
means that its just time to press on to Ruby 1.9.2 and leave 1.8.x
behind. I also noticed when I got my Velocity registration today that
the recommended steps for installing Chef for use in the Chef training
session suggested using Ruby 1.9.2.

Fortunately for me, I don't run Ruby infrastructure for anything other
than Chef so I don't really care much. Just want it to work and
minimize change. Still looking forward to a fatty installer.

I'm dumb testing previous patch levels of ruby right now. I've uploaded
i386 pacakges to http://rbel.frameos.org/misc if you are interested:

rpm -Uvh --force
http://rbel.frameos.org/misc/ruby-1.8.7.302-1.el5.i386.rpm
http://rbel.frameos.org/misc/ruby-libs-1.8.7.302-1.el5.i386.rpm

should do the trick downgrading your ruby version if it's higher. Patch
level 302 was included by F14.
Besides that, RBEL ruby-1.9 is parallel installable with 1.8 if that helps.

Cheers!

KC

On Thu, Jun 9, 2011 at 12:30 AM, Sergio Rubio rubiojr@frameos.org
wrote:

Hey guys,
On Fri, Jun 3, 2011 at 11:05 AM, Raymond Tham tekkaray@yotowoti.com
wrote:

That's right, we noticed the seg faults happening on our i386 systems;
we
don't have chef-client set up on any x86_64 instances at the moment.

I wonder if you made any progress with this stuff. Been swamped lately,
but
I closely follow they list and RPM related threads. I'll get some fresh
air
soon and I was wondering if testing i386 packages is the best way to
help
here right now.
Thanks!

Thanks for looking into this!

On Fri, Jun 3, 2011 at 1:44 AM, James js@aegisco.com wrote:

The problem is in our (aegisco) build of 1.8.7 for i386, which RBEL is
using. We'll work on rebuilding that this weekend. All of your systems
are
i386, correct?
Thanks a lot for the report and information!
James
On Thu, Jun 2, 2011 at 11:21 PM, Raymond Tham tekkaray@yotowoti.com
wrote:

Hi James,
Sorry, I should have mentioned that the AMI (ami-31df8e74) is in US
West, not East. It should be an S3 backed RightScale AMI.
I checked /root/.gemrc, and it points
to Index of /rubygems/archive/latest/
.
However, the gems should have been installed through yum through the
RBEL5
i386 repo.
If it helps, here's the output from "gem list" after installing
through
yum.
[root@ip-10-168-70-131 ~]# gem list
*** LOCAL GEMS ***
abstract (1.0.0)
activesupport (3.0.3)
allison (2.0.3)
bunny (0.6.0)
chef (0.10.0)
diff-lcs (1.1.2)
echoe (4.5.6)
erubis (2.6.6)
highline (1.6.1)
json (1.4.6)
mime-types (1.16)
mixlib-authentication (1.1.4)
mixlib-cli (1.2.0)
mixlib-config (1.1.2)
mixlib-log (1.3.0)
moneta (0.6.0)
net-sftp (2.0.4)
net-ssh (2.1.4, 2.0.23)
net-ssh-gateway (1.0.1)
net-ssh-multi (1.0.1)
ohai (0.6.4)
polyglot (0.3.1)
rake (0.8.7)
rake-compiler (0.7.5)
rest-client (1.6.1)
rspec (2.5.0)
rspec-core (2.5.1)
rspec-expectations (2.5.0)
rspec-mocks (2.5.0)
systemu (1.2.0)
treetop (1.4.9)
uuidtools (2.1.1)
xml-simple (1.0.12)
yajl-ruby (0.8.2)

On Thu, Jun 2, 2011 at 10:31 PM, James js@aegisco.com wrote:

Ray,
The ami you gave doesn't seem to be public, but I found one which
the
same CentOS and RightImage numbers: ami-2342a94a.
Can you double-check that it's a public image, so I can make sure to
mirror you as closely as possible?
Did you also experience
this: http://tickets.opscode.com/browse/CHEF-2402
Thanks,
James
On Thu, Jun 2, 2011 at 8:19 PM, Raymond Tham <tekkaray@yotowoti.com

wrote:

Hi James,
I originally noticed this happening on our older physical servers
running i386 CentOS 5.4 and 5.5, but I just created an EC2 instance
and was
able to reproduce it. Here's what I did:

  1. Created a small instance using RightScale's 5.4 i386 v.5.5.9 AMI
    (ami-31df8e74)
  2. Added RBEL: rpm -Uvh http://rbel.frameos.org/rbel5
  3. Installed rubygem-chef (excluding EPEL so all the ruby packages
    are
    from RBEL): yum install --disablerepo=epel rubygem-chef
  4. Manually created a /etc/chef/client.rb and copied over
    validation.pem.
  5. Ran chef-client for the first time to register the node:
    /usr/bin/chef-client
  6. Added recipe[logrotate] to the run list (this is the cookbook
    from
    opscode)
  7. Re-ran chef-client which picked up the recipe and seg faults
    trying
    to run it
    With regards to the ruby 1.8.6 that ended up working:
  8. I took Fedora 13's ruby-1.8.6.399-1.fc13.src.rpm, dropped in
    1.8.6.420 and repackaged on CentOS 5.5.
  9. Took Fedora 13's rubygems-1.3.6-1.fc13.src.rpm and repackaged it
    on
    CentOS 5.5 as is (but had to make a minor tweak to one of the patch
    files)
  10. Excluded ruby-, rubygems-, and rubygem-chef-0.10* in the RBEL
    yum
    config.
  11. Ran yum install rubygem-chef with my new RPMs.
    It pulled my repackaged RPMs (for ruby, ruby-lib, rubygems, and
    rubygem-chef) and everything else from RBEL.
    With regards to my Chef server (if that helps), it's running Chef
    0.10
    with all RBEL packages on a custom CentOS 5.6 x86_64 AMI.
    Hope that helps!
    Thanks,
    Ray
    On Thu, Jun 2, 2011 at 6:45 PM, James js@aegisco.com wrote:

Ray,
Can you provide the means to reproduce, specifically: steps to
install from the RBEL repo, and the AMI / image info? Are these
i386
instances?
We maintain packages at rpm.aegisco.com, but I'm happy to try to
help
debug the issues you're having. I'm surprised that Ruby 1.8.6
worked, I had
trouble getting 0.10 to work on anything older than 1.8.7 and
rubygems
1.6.2.
James

On Thu, Jun 2, 2011 at 6:29 PM, tekkaray@yotowoti.com wrote:

Hi all,

I noticed a couple of email threads in the past about seg faults
and
didn't see
any resolution so I wanted to share my experience with the same
thing happening
to us on a few CentOS 5.4 and 5.5 servers (we haven't tested the
client on 5.6
yet). I'm consistently able to get our chef-client to segfault
whenever it
handles this package resource (haven't had time to try other
packages yet):

[Thu, 02 Jun 2011 17:38:17 -0700] INFO: Processing
package[logrotate] action
upgrade (logrotate::default line 20)

/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:75:

[BUG] Segmentation fault

Thu, 02 Jun 2011 17:38:27 -0700] INFO: Processing
package[logrotate]
action
upgrade (logrotate::default line 20)

/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:64:

[BUG] Segmentation fault

[Thu, 02 Jun 2011 17:38:53 -0700] INFO: Processing
package[logrotate] action
upgrade (logrotate::default line 20)

/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:66:

[BUG] Segmentation fault

We were trying this with ruby 1.8.7 p334 from the FrameOS rbel5
i386
repo.
Since it sounded like an issue with ruby, reinstalling with ruby
1.8.6 p420
(and rebuilding the rbel5 rubygem-chef rpm to relax the ruby
version
requirement) seemed to solve the problem.

Is anyone else noticing seg fauls with a combination of CentOS
5.x,
ruby 1.8.7
and Chef?

Thanks,
Ray

On Thu, Jun 9, 2011 at 10:56 AM, James js@aegisco.com wrote:

Sergio,

The problem is definitely the ruby 1.8.7 build for el5 on i386. If I
remember correctly, we built this using mock, and had a lot of trouble with
it. We have since switched to using native VMs for building packages, and
will be releasing a new round very soon. This round will also resolve
CHEF-2388 http://tickets.opscode.com/browse/CHEF-2388.

Cool.

All rbel packages are built using mock. Can't really understand why mocking
them make them segfault randomly, but let me know if non-mock builds fix the
issue.

On a side note, ruby 1.8.7 p302 segfaults too from time to time.

I'm not opposed to switching to, or at least including 1.9. One challenge
is that every other ruby package in the CentOS repos use the 1.8 ABI; to run
1.9 you either need to not install any of these packages, or have both 1.8
and 1.9 installed.

Right. Replacing 1.8 with 1.9 means breaking stuff in RHEL. Making 1.9
parallel installable is the right thing to do IMO. We need an alternatives
mechanism like Debian's to switch between different ruby implementations
seamlessly though.

Rgds.

James

On Thu, Jun 9, 2011 at 1:25 AM, Sergio Rubio rubiojr@frameos.org wrote:

On Thu, Jun 9, 2011 at 9:46 AM, KC Braunschweig <kcbraunschweig@gmail.com

wrote:

These issues seem to keep cropping up. I'm wondering if this really
means that its just time to press on to Ruby 1.9.2 and leave 1.8.x
behind. I also noticed when I got my Velocity registration today that
the recommended steps for installing Chef for use in the Chef training
session suggested using Ruby 1.9.2.

Fortunately for me, I don't run Ruby infrastructure for anything other
than Chef so I don't really care much. Just want it to work and
minimize change. Still looking forward to a fatty installer.

I'm dumb testing previous patch levels of ruby right now. I've uploaded
i386 pacakges to http://rbel.frameos.org/misc if you are interested:

rpm -Uvh --force
http://rbel.frameos.org/misc/ruby-1.8.7.302-1.el5.i386.rpm
http://rbel.frameos.org/misc/ruby-libs-1.8.7.302-1.el5.i386.rpm

should do the trick downgrading your ruby version if it's higher. Patch
level 302 was included by F14.
Besides that, RBEL ruby-1.9 is parallel installable with 1.8 if that
helps.

Cheers!

KC

On Thu, Jun 9, 2011 at 12:30 AM, Sergio Rubio rubiojr@frameos.org
wrote:

Hey guys,
On Fri, Jun 3, 2011 at 11:05 AM, Raymond Tham tekkaray@yotowoti.com
wrote:

That's right, we noticed the seg faults happening on our i386 systems;
we
don't have chef-client set up on any x86_64 instances at the moment.

I wonder if you made any progress with this stuff. Been swamped lately,
but
I closely follow they list and RPM related threads. I'll get some fresh
air
soon and I was wondering if testing i386 packages is the best way to
help
here right now.
Thanks!

Thanks for looking into this!

On Fri, Jun 3, 2011 at 1:44 AM, James js@aegisco.com wrote:

The problem is in our (aegisco) build of 1.8.7 for i386, which RBEL
is
using. We'll work on rebuilding that this weekend. All of your
systems are
i386, correct?
Thanks a lot for the report and information!
James
On Thu, Jun 2, 2011 at 11:21 PM, Raymond Tham <tekkaray@yotowoti.com

wrote:

Hi James,
Sorry, I should have mentioned that the AMI (ami-31df8e74) is in US
West, not East. It should be an S3 backed RightScale AMI.
I checked /root/.gemrc, and it points
to
Index of /rubygems/archive/latest/.
However, the gems should have been installed through yum through the
RBEL5
i386 repo.
If it helps, here's the output from "gem list" after installing
through
yum.
[root@ip-10-168-70-131 ~]# gem list
*** LOCAL GEMS ***
abstract (1.0.0)
activesupport (3.0.3)
allison (2.0.3)
bunny (0.6.0)
chef (0.10.0)
diff-lcs (1.1.2)
echoe (4.5.6)
erubis (2.6.6)
highline (1.6.1)
json (1.4.6)
mime-types (1.16)
mixlib-authentication (1.1.4)
mixlib-cli (1.2.0)
mixlib-config (1.1.2)
mixlib-log (1.3.0)
moneta (0.6.0)
net-sftp (2.0.4)
net-ssh (2.1.4, 2.0.23)
net-ssh-gateway (1.0.1)
net-ssh-multi (1.0.1)
ohai (0.6.4)
polyglot (0.3.1)
rake (0.8.7)
rake-compiler (0.7.5)
rest-client (1.6.1)
rspec (2.5.0)
rspec-core (2.5.1)
rspec-expectations (2.5.0)
rspec-mocks (2.5.0)
systemu (1.2.0)
treetop (1.4.9)
uuidtools (2.1.1)
xml-simple (1.0.12)
yajl-ruby (0.8.2)

On Thu, Jun 2, 2011 at 10:31 PM, James js@aegisco.com wrote:

Ray,
The ami you gave doesn't seem to be public, but I found one which
the
same CentOS and RightImage numbers: ami-2342a94a.
Can you double-check that it's a public image, so I can make sure
to
mirror you as closely as possible?
Did you also experience
this: http://tickets.opscode.com/browse/CHEF-2402
Thanks,
James
On Thu, Jun 2, 2011 at 8:19 PM, Raymond Tham <
tekkaray@yotowoti.com>
wrote:

Hi James,
I originally noticed this happening on our older physical servers
running i386 CentOS 5.4 and 5.5, but I just created an EC2
instance and was
able to reproduce it. Here's what I did:

  1. Created a small instance using RightScale's 5.4 i386 v.5.5.9
    AMI
    (ami-31df8e74)
  2. Added RBEL: rpm -Uvh http://rbel.frameos.org/rbel5
  3. Installed rubygem-chef (excluding EPEL so all the ruby packages
    are
    from RBEL): yum install --disablerepo=epel rubygem-chef
  4. Manually created a /etc/chef/client.rb and copied over
    validation.pem.
  5. Ran chef-client for the first time to register the node:
    /usr/bin/chef-client
  6. Added recipe[logrotate] to the run list (this is the cookbook
    from
    opscode)
  7. Re-ran chef-client which picked up the recipe and seg faults
    trying
    to run it
    With regards to the ruby 1.8.6 that ended up working:
  8. I took Fedora 13's ruby-1.8.6.399-1.fc13.src.rpm, dropped in
    1.8.6.420 and repackaged on CentOS 5.5.
  9. Took Fedora 13's rubygems-1.3.6-1.fc13.src.rpm and repackaged
    it on
    CentOS 5.5 as is (but had to make a minor tweak to one of the
    patch files)
  10. Excluded ruby-, rubygems-, and rubygem-chef-0.10* in the RBEL
    yum
    config.
  11. Ran yum install rubygem-chef with my new RPMs.
    It pulled my repackaged RPMs (for ruby, ruby-lib, rubygems, and
    rubygem-chef) and everything else from RBEL.
    With regards to my Chef server (if that helps), it's running Chef
    0.10
    with all RBEL packages on a custom CentOS 5.6 x86_64 AMI.
    Hope that helps!
    Thanks,
    Ray
    On Thu, Jun 2, 2011 at 6:45 PM, James js@aegisco.com wrote:

Ray,
Can you provide the means to reproduce, specifically: steps to
install from the RBEL repo, and the AMI / image info? Are these
i386
instances?
We maintain packages at rpm.aegisco.com, but I'm happy to try to
help
debug the issues you're having. I'm surprised that Ruby 1.8.6
worked, I had
trouble getting 0.10 to work on anything older than 1.8.7 and
rubygems
1.6.2.
James

On Thu, Jun 2, 2011 at 6:29 PM, tekkaray@yotowoti.com wrote:

Hi all,

I noticed a couple of email threads in the past about seg faults
and
didn't see
any resolution so I wanted to share my experience with the same
thing happening
to us on a few CentOS 5.4 and 5.5 servers (we haven't tested the
client on 5.6
yet). I'm consistently able to get our chef-client to segfault
whenever it
handles this package resource (haven't had time to try other
packages yet):

[Thu, 02 Jun 2011 17:38:17 -0700] INFO: Processing
package[logrotate] action
upgrade (logrotate::default line 20)

/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:75:

[BUG] Segmentation fault

Thu, 02 Jun 2011 17:38:27 -0700] INFO: Processing
package[logrotate]
action
upgrade (logrotate::default line 20)

/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:64:

[BUG] Segmentation fault

[Thu, 02 Jun 2011 17:38:53 -0700] INFO: Processing
package[logrotate] action
upgrade (logrotate::default line 20)

/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:66:

[BUG] Segmentation fault

We were trying this with ruby 1.8.7 p334 from the FrameOS rbel5
i386
repo.
Since it sounded like an issue with ruby, reinstalling with ruby
1.8.6 p420
(and rebuilding the rbel5 rubygem-chef rpm to relax the ruby
version
requirement) seemed to solve the problem.

Is anyone else noticing seg fauls with a combination of CentOS
5.x,
ruby 1.8.7
and Chef?

Thanks,
Ray

On Thu, Jun 9, 2011 at 11:27 AM, Sergio Rubio rubiojr@frameos.org wrote:

On a side note, ruby 1.8.7 p302 segfaults too from time to time.

Some more input:

Test run in a CentOS 5.6 i386 node with ruby 1.8.7 p302, Chef 0.10 from
RBEL.

Node has mysql and couchdb (from cookbooks.opscode.com) in the run_list:

Running the following script:


#!/bin/bash
for i in {1..100};do
yum remove -y nginx GeoIP perl mysql-5.0.77-4.el5_6.6.i386
mysql-devel-5.0.77-4.el5_6.6.i386 unixODBC tk erlang >/dev/null 2>&1
echo "running chef..."
chef-client 2>&1 |grep Segmentation
done

yields:

/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:75:
[BUG] Segmentation fault
running chef...
/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:64:
[BUG] Segmentation fault
running chef...
/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:66:
[BUG] Segmentation fault
running chef...
/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:72:
[BUG] Segmentation fault
running chef...
/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:64:
[BUG] Segmentation fault
running chef...
/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:64:
[BUG] Segmentation fault
running chef...
/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:64:
[BUG] Segmentation fault
running chef...
/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:64:
[BUG] Segmentation fault

On Thu, Jun 9, 2011 at 11:52 AM, Sergio Rubio rubiojr@frameos.org wrote:

On Thu, Jun 9, 2011 at 11:27 AM, Sergio Rubio rubiojr@frameos.org wrote:

On a side note, ruby 1.8.7 p302 segfaults too from time to time.

Some more input:

/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:64:

[BUG] Segmentation fault

GDB backtrace:

On Thu, Jun 9, 2011 at 12:16 PM, Sergio Rubio rubiojr@frameos.org wrote:

On Thu, Jun 9, 2011 at 11:52 AM, Sergio Rubio rubiojr@frameos.org wrote:

On Thu, Jun 9, 2011 at 11:27 AM, Sergio Rubio rubiojr@frameos.orgwrote:

On a side note, ruby 1.8.7 p302 segfaults too from time to time.

Some more input:

/usr/lib/ruby/gems/1.8/gems/chef-0.10.0/bin/../lib/chef/provider/package/yum.rb:64:

[BUG] Segmentation fault

GDB backtrace:

ruby chef-client backtrace 1.8.7 CentOS 5 i386 · GitHub

I've opened a bug in case anyone is interested:

Let's hope we can get some assistance from devs.

Rgds.

On Thu, Jun 9, 2011 at 12:44 PM, Sergio Rubio rubiojr@frameos.org wrote:

I've opened a bug in case anyone is interested:

Backport #4856: Random segfaults running opscode chef-client 0.10 - Backport187 - Ruby Issue Tracking System

Let's hope we can get some assistance from devs.

Rgds.

I've made some progress tracing this. I've replaced the popen4 in yum
provider with simple shell quotes and my previous test no longer crashes
ruby.

If you are interested in testing the fix (albeit incomplete), the yum.rb
provider patch is here:

Did not have a chance to test Matthew Kent patches, but I'd like to have a
look at them if time permits. We can patch the RBEL RPMs and do not wait to
the 0.10.2 release if required.

Rgds.

Anyone have a public Centos5 AMI? I'll build you some Fatty.

Here's an el6 build:
http://yum.afistfulofservers.net/affs/fatty/el6/

-s

On Thu, Jun 9, 2011 at 8:55 AM, Sergio Rubio rubiojr@frameos.org wrote:

On Thu, Jun 9, 2011 at 12:44 PM, Sergio Rubio rubiojr@frameos.org wrote:

I've opened a bug in case anyone is interested:
Backport #4856: Random segfaults running opscode chef-client 0.10 - Backport187 - Ruby Issue Tracking System
Let's hope we can get some assistance from devs.
Rgds.

I've made some progress tracing this. I've replaced the popen4 in yum
provider with simple shell quotes and my previous test no longer crashes
ruby.
If you are interested in testing the fix (albeit incomplete), the yum.rb
provider patch is here:
Fixes ruby 1.8.7 segfaulting in CentOS 5.6 i386 · GitHub
Did not have a chance to test Matthew Kent patches, but I'd like to have a
look at them if time permits. We can patch the RBEL RPMs and do not wait to
the 0.10.2 release if required.
Rgds.

On Thursday, June 9, 2011 at 5:55 AM, Sergio Rubio wrote:

On Thu, Jun 9, 2011 at 12:44 PM, Sergio Rubio <rubiojr@frameos.org (mailto:rubiojr@frameos.org)> wrote:

I've opened a bug in case anyone is interested:

Backport #4856: Random segfaults running opscode chef-client 0.10 - Backport187 - Ruby Issue Tracking System

Let's hope we can get some assistance from devs.

Rgds.

I've made some progress tracing this. I've replaced the popen4 in yum provider with simple shell quotes and my previous test no longer crashes ruby.

If you are interested in testing the fix (albeit incomplete), the yum.rb provider patch is here:

Fixes ruby 1.8.7 segfaulting in CentOS 5.6 i386 · GitHub

Did not have a chance to test Matthew Kent patches, but I'd like to have a look at them if time permits. We can patch the RBEL RPMs and do not wait to the 0.10.2 release if required.

Rgds.
I ran into similar issues when I developed Chef::ShellOut, which I found were caused by object allocation during GC. Since I did not have the option of forcing people to upgrade to a ruby without the issue, I disabled GC for the affected portion of the code. You could try replacing popen4 with shell_out and see if this fixes the issue.

https://github.com/opscode/chef/blob/master/chef/lib/chef/shell_out.rb
https://github.com/opscode/chef/blob/master/chef/lib/chef/shell_out/unix.rb

--
Dan DeLeo