TLDR - Chef 11.8.0 --> chef-zero --> Puma forces build tools, looking for a
way to avoid that
Seemingly out of nowhere (between 08:40 and 17:20 pacific), our CI process
for testing cookbooks is no longer worker. The error I’m seeing is:
[2013-10-31T17:38:56-07:00] FATAL: Gem::Installer::ExtensionBuildError:
chef_gem[minitest-chef-handler] (minitest-handler::default line 13) had an
error: Gem::Installer::ExtensionBuildError: ERROR: Failed to build gem
native extension.
/opt/chef/embedded/bin/ruby extconf.rb
creating Makefile
10.42.4.121
make
compiling puma_http11.c
make: gcc: Command not found
make: *** [puma_http11.o] Error 127
My gem skillz are not top notch, but I believe after digging through debug
level logging is that in 11.6.x puma was a development requirement, but not
a runtime requirement, so it was never installed on our nodes
Now with 11.8.0, there is a dependency on chef-zero, which has a runtime
dependency on Puma, so now it is trying to install it.
Where I get confused is that I’m running 10.26.0, so its unclear to me how
that new dependency chain might affect me, but the timing is hard to ignore.
So my question is does that make sense, or am I way off base here? If it
does make sense what options do I have to not have to have build tools on
every box I want to install minitest-chef-handler?
TLDR - Chef 11.8.0 --> chef-zero --> Puma forces build tools, looking for
a way to avoid that
Seemingly out of nowhere (between 08:40 and 17:20 pacific), our CI process
for testing cookbooks is no longer worker. The error I'm seeing is:
[2013-10-31T17:38:56-07:00] FATAL: Gem::Installer::ExtensionBuildError:
chef_gem[minitest-chef-handler] (minitest-handler::default line 13) had an
error: Gem::Installer::ExtensionBuildError: ERROR: Failed to build gem
native extension.
/opt/chef/embedded/bin/ruby extconf.rb
creating Makefile
10.42.4.121
make
compiling puma_http11.c
make: gcc: Command not found
make: *** [puma_http11.o] Error 127
My gem skillz are not top notch, but I believe after digging through debug
level logging is that in 11.6.x puma was a development requirement, but not
a runtime requirement, so it was never installed on our nodes
Now with 11.8.0, there is a dependency on chef-zero, which has a runtime
dependency on Puma, so now it is trying to install it.
Where I get confused is that I'm running 10.26.0, so its unclear to me how
that new dependency chain might affect me, but the timing is hard to ignore.
So my question is does that make sense, or am I way off base here? If it
does make sense what options do I have to not have to have build tools on
every box I want to install minitest-chef-handler?
Thanks Serdar,
Unfortunately, I don't think the Gemfile will help in this case, but let me
give you a little more background on the CI process.
Upon commit to master to a specific cookbooks git repo the following
occurs:
Cookbook source is uploaded to a dedicated testing org on the chef server
A VM is launched and bootstrapped with 10.26.0 omnibus
The run list will include the minitest-handler-cookbook, as well as the
applicable test role/recipe that is specific to that cookbook.
When it passes, its then uploaded to the prod org
So given that we are simply bootstrapping a 10.26.0 client and kicking off
a run, I'm not clear how/where I'd introduce a Gemfile to my environment,
but I'm happy to try it if you can let me know I'd do in my situation.
From what I've been able to debug, I'd be surprised if others are not
seeing this issue as the dependency issues seems to be come from upstream
(unless of course folks make it a standard practice to have build tools on
their CI nodes)?
On Thu, Oct 31, 2013 at 11:58 PM, Serdar Sutay serdar@opscode.com wrote:
Looks like minitest-chef-handler has below dependency on Chef:
gem.add_dependency('chef', '>= 10.12.0')
If you have a Gemfile by which you are controlling your environment you can add:
gem 'chef', '~>10.26'
to it which should move you to Chef 10 and will resolve the issue.
Hope this helps. If not, it would help if you can give a little bit more details about your CI process of testing cookbooks.
Another thing you can do is directly install chef 10.26.0 on your
node--then minitest-handler will just pick that instead of jumping to 11.
As far as the compiling dependency, many folks won't hit that because they
install chef with omnibus--no build tools involved, just install the
.deb/.rpm/etc.
puma was chosen because it was the only webserver I could find that works
everywhere Chef is supported (all Ruby versions and all platforms). This
included webrick (though I can't remember which platform/ruby version it
failed on). To see if I can help with this, I'll take another pass at
finding a pure Ruby one with those characteristics, or some if statements I
can use
Thanks Serdar,
Unfortunately, I don't think the Gemfile will help in this case, but let
me give you a little more background on the CI process.
Upon commit to master to a specific cookbooks git repo the following
occurs:
Cookbook source is uploaded to a dedicated testing org on the chef server
A VM is launched and bootstrapped with 10.26.0 omnibus
The run list will include the minitest-handler-cookbook, as well as the
applicable test role/recipe that is specific to that cookbook.
When it passes, its then uploaded to the prod org
So given that we are simply bootstrapping a 10.26.0 client and kicking off
a run, I'm not clear how/where I'd introduce a Gemfile to my environment,
but I'm happy to try it if you can let me know I'd do in my situation.
From what I've been able to debug, I'd be surprised if others are not
seeing this issue as the dependency issues seems to be come from upstream
(unless of course folks make it a standard practice to have build tools on
their CI nodes)?
On Thu, Oct 31, 2013 at 11:58 PM, Serdar Sutay serdar@opscode.com wrote:
Looks like minitest-chef-handler has below dependency on Chef:
gem.add_dependency('chef', '>= 10.12.0')
If you have a Gemfile by which you are controlling your environment you can add:
gem 'chef', '~>10.26'
to it which should move you to Chef 10 and will resolve the issue.
Hope this helps. If not, it would help if you can give a little bit more details about your CI process of testing cookbooks.
Thanks John,
So I'm not sure I following your comment. I did install via omnibus, so I
don't have build tools installed, which is where the issue seems to stem
from. Here is a GIST of what I'm seeing.
Even when attempting to install manually I'm hitting that error.
Another thing you can do is directly install chef 10.26.0 on your
node--then minitest-handler will just pick that instead of jumping to 11.
As far as the compiling dependency, many folks won't hit that because they
install chef with omnibus--no build tools involved, just install the
.deb/.rpm/etc.
puma was chosen because it was the only webserver I could find that works
everywhere Chef is supported (all Ruby versions and all platforms). This
included webrick (though I can't remember which platform/ruby version it
failed on). To see if I can help with this, I'll take another pass at
finding a pure Ruby one with those characteristics, or some if statements I
can use
Thanks Serdar,
Unfortunately, I don't think the Gemfile will help in this case, but let
me give you a little more background on the CI process.
Upon commit to master to a specific cookbooks git repo the following
occurs:
Cookbook source is uploaded to a dedicated testing org on the chef
server
A VM is launched and bootstrapped with 10.26.0 omnibus
The run list will include the minitest-handler-cookbook, as well as the
applicable test role/recipe that is specific to that cookbook.
When it passes, its then uploaded to the prod org
So given that we are simply bootstrapping a 10.26.0 client and kicking
off a run, I'm not clear how/where I'd introduce a Gemfile to my
environment, but I'm happy to try it if you can let me know I'd do in my
situation.
From what I've been able to debug, I'd be surprised if others are not
seeing this issue as the dependency issues seems to be come from upstream
(unless of course folks make it a standard practice to have build tools on
their CI nodes)?
On Friday, November 1, 2013 at 10:20 AM, David Petzel wrote:
Thanks John,
So I'm not sure I following your comment. I did install via omnibus, so I don't have build tools installed, which is where the issue seems to stem from. Here is a GIST of what I'm seeing.
Even when attempting to install manually I'm hitting that error.
Rubygems, by default, will upgrade every gem it possibly can when installing a new gem. This means that if any “chef plugin” kind of project specifies a dependency on Chef, rubygems will try to upgrade chef when you install it. I don’t know exactly when the option was added, but newish versions of rubygems have a —conservative option that should disable that behavior. I’d need to look more in-depth to see how you should use this with the chef_gem or gem_package resources, but it should work if you give that option as a string to the options parameter.
In the bigger picture, it may be best if plugins do not specify a dependency on Chef. The downside to that is plugins can’t specify a minimum version of chef they support. I don’t have a good solution for that right now.
In the bigger picture, it may be best if plugins do not specify a
dependency on Chef. The downside to that is plugins can’t specify a minimum
version of chef they support. I don’t have a good solution for that right
now.
I recently ran into this very problem.
I've since changed my plugin to have Chef version as a development
dependency instead of a runtime one, since the plugin is only going to run
under Chef anyways, so the runtime is somewhat guaranteed to be in a Chef
version.
The problem arises when the plugin is targeted towards a specific Chef
version(s), and that's currently the only way to constrain things. However,
if you're constraining on a specific X.Y.Z version of Chef, you should be
ok. It's where 11.8.0 introduces new dependencies that were not previously
there - that's where we're all getting stuck.
-M
On Fri, Nov 1, 2013 at 1:33 PM, Daniel DeLeo dan@kallistec.com wrote:
On Friday, November 1, 2013 at 10:20 AM, David Petzel wrote:
Thanks John,
So I'm not sure I following your comment. I did install via omnibus, so I
don't have build tools installed, which is where the issue seems to stem
from. Here is a GIST of what I'm seeing.
Even when attempting to install manually I'm hitting that error.
Rubygems, by default, will upgrade every gem it possibly can when
installing a new gem. This means that if any “chef plugin” kind of project
specifies a dependency on Chef, rubygems will try to upgrade chef when you
install it. I don’t know exactly when the option was added, but newish
versions of rubygems have a —conservative option that should disable that
behavior. I’d need to look more in-depth to see how you should use this
with the chef_gem or gem_package resources, but it should work if you give
that option as a string to the options parameter.
In the bigger picture, it may be best if plugins do not specify a
dependency on Chef. The downside to that is plugins can’t specify a minimum
version of chef they support. I don’t have a good solution for that right
now.
I think I'm fairly clear on the short term workaround (using the —conservative
flag) however I wonder how many plugins out there do this, and thus are
subject to this issue now that 11.8 has been released. Is it acceptable
that 11.8 introduces a dependency on a gem that requires build tools? I ask
because I'm not sure what expectations.
It's where 11.8.0 introduces new dependencies that were not previously
there - that's where we're all getting stuck.
On Friday, November 1, 2013 at 10:39 AM, Mike wrote:
The problem arises when the plugin is targeted towards a specific Chef version(s), and that's currently the only way to constrain things. However, if you're constraining on a specific X.Y.Z version of Chef, you should be ok. It's where 11.8.0 introduces new dependencies that were not previously there - that's where we're all getting stuck.
That’s the trigger that exposes the issue, but there are a number of ways that this could happen. Chef depends on a handful of gems with C extensions, and if any of these were to release a new version that matched Chef’s version constraints, then rubygems would attempt to upgrade them when installing anything that depends on Chef. To borrow (and butcher) a phrase from Allspaw, both conditions are necessary and only jointly sufficient for this problem to occur.
It’s also pretty troubling that you could unknowingly get upgraded to a newer version of chef (including a major version update) when you only wanted to install a test framework.
