Mount resource segfault


#1

The chef-client daemon seems to be dying on all of our boxes (at random) due to
the mount resource:
[Tue, 03 May 2011 08:38:28 -0700] DEBUG: sh(mount)
/usr/lib/ruby/gems/1.8/gems/chef-0.9.14/bin/…/lib/chef/resource/mount.rb:46:
[BUG] Segmentation fault
ruby 1.8.7 (2011-02-18 patchlevel 334) [x86_64-linux]

Has anyone encountered this before?

Thanks,
Mark


#2

This seems to alternate with:
[Tue, 03 May 2011 09:15:01 -0700] DEBUG: sh(mount)
/usr/lib/ruby/gems/1.8/gems/chef-0.9.14/bin/…/lib/chef/mixin/params_validate.rb:83:
[BUG] Segmentation fault
ruby 1.8.7 (2011-02-18 patchlevel 334) [x86_64-linux]

On Tue, May 3, 2011 at 12:05 PM, mrechler@brightcove.com wrote:

The chef-client daemon seems to be dying on all of our boxes (at random)
due to
the mount resource:
[Tue, 03 May 2011 08:38:28 -0700] DEBUG: sh(mount)

/usr/lib/ruby/gems/1.8/gems/chef-0.9.14/bin/…/lib/chef/resource/mount.rb:46:
[BUG] Segmentation fault
ruby 1.8.7 (2011-02-18 patchlevel 334) [x86_64-linux]

Has anyone encountered this before?

Thanks,
Mark


#3

Wow - what operating system and version, and what build of ruby?

Adam

On Tue, May 3, 2011 at 9:17 AM, Mark Rechler mrechler@brightcove.com wrote:

This seems to alternate with:
[Tue, 03 May 2011 09:15:01 -0700] DEBUG: sh(mount)
/usr/lib/ruby/gems/1.8/gems/chef-0.9.14/bin/…/lib/chef/mixin/params_validate.rb:83:
[BUG] Segmentation fault
ruby 1.8.7 (2011-02-18 patchlevel 334) [x86_64-linux]
On Tue, May 3, 2011 at 12:05 PM, mrechler@brightcove.com wrote:

The chef-client daemon seems to be dying on all of our boxes (at random)
due to
the mount resource:
[Tue, 03 May 2011 08:38:28 -0700] DEBUG: sh(mount)

/usr/lib/ruby/gems/1.8/gems/chef-0.9.14/bin/…/lib/chef/resource/mount.rb:46:
[BUG] Segmentation fault
ruby 1.8.7 (2011-02-18 patchlevel 334) [x86_64-linux]

Has anyone encountered this before?

Thanks,
Mark


Opscode, Inc.
Adam Jacob, Chief Product Officer
T: (206) 619-7151 E: adam@opscode.com


#4

This is running on centos 5.6.
ruby 1.8.7 (2011-02-18 patchlevel 334) [x86_64-linux]

On Tue, May 3, 2011 at 1:18 PM, Adam Jacob adam@opscode.com wrote:

Wow - what operating system and version, and what build of ruby?

Adam

On Tue, May 3, 2011 at 9:17 AM, Mark Rechler mrechler@brightcove.com
wrote:

This seems to alternate with:
[Tue, 03 May 2011 09:15:01 -0700] DEBUG: sh(mount)

/usr/lib/ruby/gems/1.8/gems/chef-0.9.14/bin/…/lib/chef/mixin/params_validate.rb:83:

[BUG] Segmentation fault
ruby 1.8.7 (2011-02-18 patchlevel 334) [x86_64-linux]
On Tue, May 3, 2011 at 12:05 PM, mrechler@brightcove.com wrote:

The chef-client daemon seems to be dying on all of our boxes (at random)
due to
the mount resource:
[Tue, 03 May 2011 08:38:28 -0700] DEBUG: sh(mount)

/usr/lib/ruby/gems/1.8/gems/chef-0.9.14/bin/…/lib/chef/resource/mount.rb:46:

[BUG] Segmentation fault
ruby 1.8.7 (2011-02-18 patchlevel 334) [x86_64-linux]

Has anyone encountered this before?

Thanks,
Mark


Opscode, Inc.
Adam Jacob, Chief Product Officer
T: (206) 619-7151 E: adam@opscode.com


#5

Is it consistently just on that mount resource, or is this happening
on other shell commands as well?

Adam

On Tue, May 3, 2011 at 10:22 AM, Mark Rechler mrechler@brightcove.com wrote:

This is running on centos 5.6.
ruby 1.8.7 (2011-02-18 patchlevel 334) [x86_64-linux]

On Tue, May 3, 2011 at 1:18 PM, Adam Jacob adam@opscode.com wrote:

Wow - what operating system and version, and what build of ruby?

Adam

On Tue, May 3, 2011 at 9:17 AM, Mark Rechler mrechler@brightcove.com
wrote:

This seems to alternate with:
[Tue, 03 May 2011 09:15:01 -0700] DEBUG: sh(mount)

/usr/lib/ruby/gems/1.8/gems/chef-0.9.14/bin/…/lib/chef/mixin/params_validate.rb:83:
[BUG] Segmentation fault
ruby 1.8.7 (2011-02-18 patchlevel 334) [x86_64-linux]
On Tue, May 3, 2011 at 12:05 PM, mrechler@brightcove.com wrote:

The chef-client daemon seems to be dying on all of our boxes (at
random)
due to
the mount resource:
[Tue, 03 May 2011 08:38:28 -0700] DEBUG: sh(mount)

/usr/lib/ruby/gems/1.8/gems/chef-0.9.14/bin/…/lib/chef/resource/mount.rb:46:
[BUG] Segmentation fault
ruby 1.8.7 (2011-02-18 patchlevel 334) [x86_64-linux]

Has anyone encountered this before?

Thanks,
Mark


Opscode, Inc.
Adam Jacob, Chief Product Officer
T: (206) 619-7151 E: adam@opscode.com


Opscode, Inc.
Adam Jacob, Chief Product Officer
T: (206) 619-7151 E: adam@opscode.com


#6

The segfault seems to only occur after accessing the mount resource.

On Tue, May 3, 2011 at 1:44 PM, Adam Jacob adam@opscode.com wrote:

Is it consistently just on that mount resource, or is this happening
on other shell commands as well?

Adam

On Tue, May 3, 2011 at 10:22 AM, Mark Rechler mrechler@brightcove.com
wrote:

This is running on centos 5.6.
ruby 1.8.7 (2011-02-18 patchlevel 334) [x86_64-linux]

On Tue, May 3, 2011 at 1:18 PM, Adam Jacob adam@opscode.com wrote:

Wow - what operating system and version, and what build of ruby?

Adam

On Tue, May 3, 2011 at 9:17 AM, Mark Rechler mrechler@brightcove.com
wrote:

This seems to alternate with:
[Tue, 03 May 2011 09:15:01 -0700] DEBUG: sh(mount)

/usr/lib/ruby/gems/1.8/gems/chef-0.9.14/bin/…/lib/chef/mixin/params_validate.rb:83:

[BUG] Segmentation fault
ruby 1.8.7 (2011-02-18 patchlevel 334) [x86_64-linux]
On Tue, May 3, 2011 at 12:05 PM, mrechler@brightcove.com wrote:

The chef-client daemon seems to be dying on all of our boxes (at
random)
due to
the mount resource:
[Tue, 03 May 2011 08:38:28 -0700] DEBUG: sh(mount)

/usr/lib/ruby/gems/1.8/gems/chef-0.9.14/bin/…/lib/chef/resource/mount.rb:46:

[BUG] Segmentation fault
ruby 1.8.7 (2011-02-18 patchlevel 334) [x86_64-linux]

Has anyone encountered this before?

Thanks,
Mark


Opscode, Inc.
Adam Jacob, Chief Product Officer
T: (206) 619-7151 E: adam@opscode.com


Opscode, Inc.
Adam Jacob, Chief Product Officer
T: (206) 619-7151 E: adam@opscode.com


#7

Just thought I would update the thread. Upgrading to ruby 1.9.2 resolved the
segfault issues. Still not sure why they were occurring in 1.8.7 though.

On Tue, May 3, 2011 at 1:52 PM, Mark Rechler mrechler@brightcove.comwrote:

The segfault seems to only occur after accessing the mount resource.

On Tue, May 3, 2011 at 1:44 PM, Adam Jacob adam@opscode.com wrote:

Is it consistently just on that mount resource, or is this happening
on other shell commands as well?

Adam

On Tue, May 3, 2011 at 10:22 AM, Mark Rechler mrechler@brightcove.com
wrote:

This is running on centos 5.6.
ruby 1.8.7 (2011-02-18 patchlevel 334) [x86_64-linux]

On Tue, May 3, 2011 at 1:18 PM, Adam Jacob adam@opscode.com wrote:

Wow - what operating system and version, and what build of ruby?

Adam

On Tue, May 3, 2011 at 9:17 AM, Mark Rechler mrechler@brightcove.com
wrote:

This seems to alternate with:
[Tue, 03 May 2011 09:15:01 -0700] DEBUG: sh(mount)

/usr/lib/ruby/gems/1.8/gems/chef-0.9.14/bin/…/lib/chef/mixin/params_validate.rb:83:

[BUG] Segmentation fault
ruby 1.8.7 (2011-02-18 patchlevel 334) [x86_64-linux]
On Tue, May 3, 2011 at 12:05 PM, mrechler@brightcove.com wrote:

The chef-client daemon seems to be dying on all of our boxes (at
random)
due to
the mount resource:
[Tue, 03 May 2011 08:38:28 -0700] DEBUG: sh(mount)

/usr/lib/ruby/gems/1.8/gems/chef-0.9.14/bin/…/lib/chef/resource/mount.rb:46:

[BUG] Segmentation fault
ruby 1.8.7 (2011-02-18 patchlevel 334) [x86_64-linux]

Has anyone encountered this before?

Thanks,
Mark


Opscode, Inc.
Adam Jacob, Chief Product Officer
T: (206) 619-7151 E: adam@opscode.com


Opscode, Inc.
Adam Jacob, Chief Product Officer
T: (206) 619-7151 E: adam@opscode.com


#8

On Thu, May 5, 2011 at 8:48 AM, Mark Rechler mrechler@brightcove.com wrote:

Just thought I would update the thread. Upgrading to ruby 1.9.2 resolved the
segfault issues. Still not sure why they were occurring in 1.8.7 though.

Was the Ruby 1.8.7p334 installation you were using from the original
CentOS 5.6 RPM installation?

How did you install Ruby 1.9.2? If from RPMs, where did you get them from?

Bryan


#9

The ruby 1.8.7 install was built by tweaking the centos spec file. For 1.9,
this was the specfile used:


This spec file provides everything in 1 rpm, vs splitting them all. I had
tried using FrameOS’s rpms from the rbel repo (http://rbel.frameos.org/),
but they were somewhat broken (I should say just their 1.9 seemed to be
missing some dependencies, the 1.8.7 packages are fine).

On Thu, May 5, 2011 at 12:04 PM, Bryan McLellan btm@loftninjas.org wrote:

On Thu, May 5, 2011 at 8:48 AM, Mark Rechler mrechler@brightcove.com
wrote:

Just thought I would update the thread. Upgrading to ruby 1.9.2 resolved
the
segfault issues. Still not sure why they were occurring in 1.8.7 though.

Was the Ruby 1.8.7p334 installation you were using from the original
CentOS 5.6 RPM installation?

How did you install Ruby 1.9.2? If from RPMs, where did you get them from?

Bryan


#10

Mark,

Did you test the 1.8.7 package in frameos or rpm.aegisco.com against the
segfaulting mount provider?

James

On Thu, May 5, 2011 at 9:12 AM, Mark Rechler mrechler@brightcove.comwrote:

The ruby 1.8.7 install was built by tweaking the centos spec file. For 1.9,
this was the specfile used:
https://github.com/imeyer/ruby-1.9.2-rpm/blob/master/ruby19.spec
This spec file provides everything in 1 rpm, vs splitting them all. I had
tried using FrameOS’s rpms from the rbel repo (http://rbel.frameos.org/),
but they were somewhat broken (I should say just their 1.9 seemed to be
missing some dependencies, the 1.8.7 packages are fine).

On Thu, May 5, 2011 at 12:04 PM, Bryan McLellan btm@loftninjas.orgwrote:

On Thu, May 5, 2011 at 8:48 AM, Mark Rechler mrechler@brightcove.com
wrote:

Just thought I would update the thread. Upgrading to ruby 1.9.2 resolved
the
segfault issues. Still not sure why they were occurring in 1.8.7 though.

Was the Ruby 1.8.7p334 installation you were using from the original
CentOS 5.6 RPM installation?

How did you install Ruby 1.9.2? If from RPMs, where did you get them from?

Bryan


#11

Hi James,

Yes, I tested the frameos package for the segfault and had the same issue
(sorry if I was unclear when I said the package was fine, I meant it was
built correctly/met dependencies for chef). I did not get a chance to test
the aegisco package. Also, it turned out the segfault point was random (just
seemed to be hitting the mount resource more often than not). I’m wondering
if it’s an issue with this particular patchset of ruby 1.8.7.

Mark

On Thu, May 5, 2011 at 12:18 PM, James js@aegisco.com wrote:

Mark,

Did you test the 1.8.7 package in frameos or rpm.aegisco.com against the
segfaulting mount provider?

James

On Thu, May 5, 2011 at 9:12 AM, Mark Rechler mrechler@brightcove.comwrote:

The ruby 1.8.7 install was built by tweaking the centos spec file. For
1.9, this was the specfile used:
https://github.com/imeyer/ruby-1.9.2-rpm/blob/master/ruby19.spec
This spec file provides everything in 1 rpm, vs splitting them all. I had
tried using FrameOS’s rpms from the rbel repo (http://rbel.frameos.org/),
but they were somewhat broken (I should say just their 1.9 seemed to be
missing some dependencies, the 1.8.7 packages are fine).

On Thu, May 5, 2011 at 12:04 PM, Bryan McLellan btm@loftninjas.orgwrote:

On Thu, May 5, 2011 at 8:48 AM, Mark Rechler mrechler@brightcove.com
wrote:

Just thought I would update the thread. Upgrading to ruby 1.9.2
resolved the
segfault issues. Still not sure why they were occurring in 1.8.7
though.

Was the Ruby 1.8.7p334 installation you were using from the original
CentOS 5.6 RPM installation?

How did you install Ruby 1.9.2? If from RPMs, where did you get them
from?

Bryan


#12

On Thu, May 5, 2011 at 9:37 AM, Mark Rechler mrechler@brightcove.com wrote:

I’m wondering if it’s an issue with this particular patchset of ruby 1.8.7.

I bet so. Ruby development is kind of hot and heavy. We had a lot of
problems with p174 and I put a fair amount of work into tracking down
segfaults and backporting fixes. (see 1.8.7.174-1ubuntu1) It isn’t
easy, but saves the community a lot of work when they just get a
patched version through upstream through normal channels.

On Thu, May 5, 2011 at 9:18 AM, James js@aegisco.com wrote:

Did you test the 1.8.7 package in frameos or rpm.aegisco.com against the
segfaulting mount provider?

Let’s be clear that the mount provider isn’t segfaulting, but ruby is.
If we could narrow down exactly when it happened to a particular line
of code, that would be awesome as it makes searching the upstream bug
tracker a lot easier. In some cases it is something more meta though,
like the garbage collector.

Bryan


#13

On Thu, May 5, 2011 at 6:12 PM, Mark Rechler mrechler@brightcove.comwrote:

The ruby 1.8.7 install was built by tweaking the centos spec file. For 1.9,
this was the specfile used:
https://github.com/imeyer/ruby-1.9.2-rpm/blob/master/ruby19.spec
This spec file provides everything in 1 rpm, vs splitting them all. I had
tried using FrameOS’s rpms from the rbel repo (http://rbel.frameos.org/),
but they were somewhat broken (I should say just their 1.9 seemed to be
missing some dependencies, the 1.8.7 packages are fine).

Hi Mark, Sergio from RBEL here.

ruby19 i386 is broken right now but x86_64 should install fine in a clean
CentOS.

About the segfault you are getting, does it happen also in i386 CentOS? ruby
1.8.7 p334 is also in rbel5 i386 repo.

Could you please send me the output from “rpm -qi ruby”?

I’m also trying to reproduce the segfault here.

Thanks!

On Thu, May 5, 2011 at 12:04 PM, Bryan McLellan btm@loftninjas.orgwrote:

On Thu, May 5, 2011 at 8:48 AM, Mark Rechler mrechler@brightcove.com
wrote:

Just thought I would update the thread. Upgrading to ruby 1.9.2 resolved
the
segfault issues. Still not sure why they were occurring in 1.8.7 though.

Was the Ruby 1.8.7p334 installation you were using from the original
CentOS 5.6 RPM installation?

How did you install Ruby 1.9.2? If from RPMs, where did you get them from?

Bryan


Sergio Rubio

FrameOS Linux
http://www.frameos.org
twitter: @rubiojr
blog: http://blog.frameos.org


#14

On Fri, May 6, 2011 at 11:27 AM, Sergio Rubio rubiojr@frameos.org wrote:

On Thu, May 5, 2011 at 6:12 PM, Mark Rechler mrechler@brightcove.comwrote:

The ruby 1.8.7 install was built by tweaking the centos spec file. For
1.9, this was the specfile used:
https://github.com/imeyer/ruby-1.9.2-rpm/blob/master/ruby19.spec
This spec file provides everything in 1 rpm, vs splitting them all. I had
tried using FrameOS’s rpms from the rbel repo (http://rbel.frameos.org/),
but they were somewhat broken (I should say just their 1.9 seemed to be
missing some dependencies, the 1.8.7 packages are fine).

Hi Mark, Sergio from RBEL here.

ruby19 i386 is broken right now but x86_64 should install fine in a clean
CentOS.

About the segfault you are getting, does it happen also in i386 CentOS?
ruby 1.8.7 p334 is also in rbel5 i386 repo.

Could you please send me the output from “rpm -qi ruby”?

I’m also trying to reproduce the segfault here.

Thanks!

Mark, did you have a chance to test the ruby package in a CentOS 5.5 box?
Wondering if rebuilding under CentOS 5.6 will fix something…

Rgds.

On Thu, May 5, 2011 at 12:04 PM, Bryan McLellan btm@loftninjas.orgwrote:

On Thu, May 5, 2011 at 8:48 AM, Mark Rechler mrechler@brightcove.com
wrote:

Just thought I would update the thread. Upgrading to ruby 1.9.2
resolved the
segfault issues. Still not sure why they were occurring in 1.8.7
though.

Was the Ruby 1.8.7p334 installation you were using from the original
CentOS 5.6 RPM installation?

How did you install Ruby 1.9.2? If from RPMs, where did you get them
from?

Bryan


Sergio Rubio

FrameOS Linux
http://www.frameos.org
twitter: @rubiojr
blog: http://blog.frameos.org


Sergio Rubio

FrameOS Linux
http://www.frameos.org
twitter: @rubiojr
blog: http://blog.frameos.org


#15

Hi Sergio,

I have only been able to test on x86_64 CentOS5.4. The first issue I ran
into with the 1.9 spec was the prefix on the binaries (some scripts look for
/usr/bin/ruby vs ruby19), which was easy to change in the spec and rebuild.
Aftwerwards, gem installing chef would berak, I don’t remember the specific
error (apologies). I did not have too much time to attempt reproducing the
issues.

Thanks,
Mark

On Fri, May 6, 2011 at 10:00 AM, Sergio Rubio rubiojr@frameos.org wrote:

On Fri, May 6, 2011 at 11:27 AM, Sergio Rubio rubiojr@frameos.org wrote:

On Thu, May 5, 2011 at 6:12 PM, Mark Rechler mrechler@brightcove.comwrote:

The ruby 1.8.7 install was built by tweaking the centos spec file. For
1.9, this was the specfile used:
https://github.com/imeyer/ruby-1.9.2-rpm/blob/master/ruby19.spec
This spec file provides everything in 1 rpm, vs splitting them all. I had
tried using FrameOS’s rpms from the rbel repo (http://rbel.frameos.org/),
but they were somewhat broken (I should say just their 1.9 seemed to be
missing some dependencies, the 1.8.7 packages are fine).

Hi Mark, Sergio from RBEL here.

ruby19 i386 is broken right now but x86_64 should install fine in a clean
CentOS.

About the segfault you are getting, does it happen also in i386 CentOS?
ruby 1.8.7 p334 is also in rbel5 i386 repo.

Could you please send me the output from “rpm -qi ruby”?

I’m also trying to reproduce the segfault here.

Thanks!

Mark, did you have a chance to test the ruby package in a CentOS 5.5 box?
Wondering if rebuilding under CentOS 5.6 will fix something…

Rgds.

On Thu, May 5, 2011 at 12:04 PM, Bryan McLellan btm@loftninjas.orgwrote:

On Thu, May 5, 2011 at 8:48 AM, Mark Rechler mrechler@brightcove.com
wrote:

Just thought I would update the thread. Upgrading to ruby 1.9.2
resolved the
segfault issues. Still not sure why they were occurring in 1.8.7
though.

Was the Ruby 1.8.7p334 installation you were using from the original
CentOS 5.6 RPM installation?

How did you install Ruby 1.9.2? If from RPMs, where did you get them
from?

Bryan


Sergio Rubio

FrameOS Linux
http://www.frameos.org
twitter: @rubiojr
blog: http://blog.frameos.org


Sergio Rubio

FrameOS Linux
http://www.frameos.org
twitter: @rubiojr
blog: http://blog.frameos.org


#16

On Fri, May 6, 2011 at 10:02 PM, Mark Rechler mrechler@brightcove.comwrote:

Hi Sergio,

I have only been able to test on x86_64 CentOS5.4. The first issue I ran
into with the 1.9 spec was the prefix on the binaries (some scripts look for
/usr/bin/ruby vs ruby19), which was easy to change in the spec and rebuild.
Aftwerwards, gem installing chef would berak, I don’t remember the specific
error (apologies). I did not have too much time to attempt reproducing the
issues.

Sure, no problem. Thank you for the comments.

RBEL ruby19 RPM was designed to be installed in parallel to 1.8 so we don’t
break stuff. Unfortunately, it’s not documented and we don’t have an
’update-alternatives’ mechanism to switch between ruby implementations
easily. We are not there yet. My apologies.

I’ll be stressing the RBEL Chef/Ruby packages soon. I’ll let you know if I
can find something about the segfaulting issues.

Rgds.

Thanks,
Mark

On Fri, May 6, 2011 at 10:00 AM, Sergio Rubio rubiojr@frameos.org wrote:

On Fri, May 6, 2011 at 11:27 AM, Sergio Rubio rubiojr@frameos.orgwrote:

On Thu, May 5, 2011 at 6:12 PM, Mark Rechler mrechler@brightcove.comwrote:

The ruby 1.8.7 install was built by tweaking the centos spec file. For
1.9, this was the specfile used:
https://github.com/imeyer/ruby-1.9.2-rpm/blob/master/ruby19.spec
This spec file provides everything in 1 rpm, vs splitting them all. I
had tried using FrameOS’s rpms from the rbel repo (
http://rbel.frameos.org/), but they were somewhat broken (I should say
just their 1.9 seemed to be missing some dependencies, the 1.8.7 packages
are fine).

Hi Mark, Sergio from RBEL here.

ruby19 i386 is broken right now but x86_64 should install fine in a clean
CentOS.

About the segfault you are getting, does it happen also in i386 CentOS?
ruby 1.8.7 p334 is also in rbel5 i386 repo.

Could you please send me the output from “rpm -qi ruby”?

I’m also trying to reproduce the segfault here.

Thanks!

Mark, did you have a chance to test the ruby package in a CentOS 5.5 box?
Wondering if rebuilding under CentOS 5.6 will fix something…

Rgds.

On Thu, May 5, 2011 at 12:04 PM, Bryan McLellan btm@loftninjas.orgwrote:

On Thu, May 5, 2011 at 8:48 AM, Mark Rechler mrechler@brightcove.com
wrote:

Just thought I would update the thread. Upgrading to ruby 1.9.2
resolved the
segfault issues. Still not sure why they were occurring in 1.8.7
though.

Was the Ruby 1.8.7p334 installation you were using from the original
CentOS 5.6 RPM installation?

How did you install Ruby 1.9.2? If from RPMs, where did you get them
from?

Bryan


Sergio Rubio

FrameOS Linux
http://www.frameos.org
twitter: @rubiojr
blog: http://blog.frameos.org


Sergio Rubio

FrameOS Linux
http://www.frameos.org
twitter: @rubiojr
blog: http://blog.frameos.org


Sergio Rubio

FrameOS Linux
http://www.frameos.org
twitter: @rubiojr
blog: http://blog.frameos.org