Mysql-0.23 cookbook fails on Lucid

Chef Infra (archive)
1

Hello, I’m struggling trying to debug my cookbooks

I have a node with a runlist:

“run_list”: [
“recipe[apt::default]”,
“recipe[nginx::passenger]”,
“recipe[mysql::server]”,
“recipe[php-fpm::default]”,
“recipe[monit::default]”,
“recipe[bodhysite::default]”,
“recipe[mail-system::vdomains]”
],

nginx::passenger starts with include_recipe “build-essential”.

The chef run fails saying this: http://pastebin.com/H7nFSqdy
Looks like mysql::client fails while building mysql gem because mkfs
from ruby-dev is missing. So I added

r1 = package "ruby1.8-dev"
r1.run_action(:install)

to mysql client, but now it still fails with the messages:

[Thu, 15 Jul 2010 09:01:45 +0000] INFO: Installing gem_package[mysql]
version 2.8.1
Building native extensions. This could take a while…
[Thu, 15 Jul 2010 09:01:46 +0000] ERROR: Running exception handlers
[Thu, 15 Jul 2010 09:01:46 +0000] ERROR: Exception handlers complete
[Thu, 15 Jul 2010 09:01:46 +0000] ERROR: Re-raising exception:
Gem::Installer::ExtensionBuildError - ERROR: Failed to build gem native
extension.

/usr/bin/ruby1.8 extconf.rb
checking for mysql_ssl_set()… no
checking for rb_str_set_len()… no
checking for rb_thread_start_timer()… no
checking for mysql.h… no
checking for mysql/mysql.h… no
*** extconf.rb failed ***

Necessary headers are there, the root of problem is what gcc isn’t
installed. But build-essential included as part of nginx::passenger
earlier than mysql. I even tried to add it into run_list as a separate
recipe right after apt::default, it didn’t helped.

A nice person on #chef told me there’s some magic in mysql cookbook, it
makes use of run_action method.

Can someone please clarify it a bit and point me in right direction?

Oh forgot to mention, all this was about mysql-0.23 and
build-essential-0.7 cookbooks.

Thanks,
Dmitry

2

You do need build-essential, that will install gcc for you, I do not see it
in your run_list and just checked the mysql cookbooks for server/client that
is also pulled in by mysql::server and I didn't see an include_recipe.

--sahil

On Thu, Jul 15, 2010 at 2:34 AM, Dmitry V'yal akamaus@gmail.com wrote:

Hello, I'm struggling trying to debug my cookbooks

I have a node with a runlist:

"run_list": [
"recipe[apt::default]",
"recipe[nginx::passenger]",
"recipe[mysql::server]",
"recipe[php-fpm::default]",
"recipe[monit::default]",
"recipe[bodhysite::default]",
"recipe[mail-system::vdomains]"
],

nginx::passenger starts with include_recipe "build-essential".

The chef run fails saying this: http://pastebin.com/H7nFSqdy
Looks like mysql::client fails while building mysql gem because mkfs from
ruby-dev is missing. So I added

r1 = package "ruby1.8-dev"
r1.run_action(:install)

to mysql client, but now it still fails with the messages:

[Thu, 15 Jul 2010 09:01:45 +0000] INFO: Installing gem_package[mysql]
version 2.8.1
Building native extensions. This could take a while...
[Thu, 15 Jul 2010 09:01:46 +0000] ERROR: Running exception handlers
[Thu, 15 Jul 2010 09:01:46 +0000] ERROR: Exception handlers complete
[Thu, 15 Jul 2010 09:01:46 +0000] ERROR: Re-raising exception:
Gem::Installer::ExtensionBuildError - ERROR: Failed to build gem native
extension.

/usr/bin/ruby1.8 extconf.rb
checking for mysql_ssl_set()... no
checking for rb_str_set_len()... no
checking for rb_thread_start_timer()... no
checking for mysql.h... no
checking for mysql/mysql.h... no
*** extconf.rb failed ***

Necessary headers are there, the root of problem is what gcc isn't
installed. But build-essential included as part of nginx::passenger earlier
than mysql. I even tried to add it into run_list as a separate recipe right
after apt::default, it didn't helped.

A nice person on Chef Infra (archive) told me there's some magic in mysql cookbook, it
makes use of run_action method.

Can someone please clarify it a bit and point me in right direction?

Oh forgot to mention, all this was about mysql-0.23 and build-essential-0.7
cookbooks.

Thanks,
Dmitry

3

Hello!

On Jul 15, 2010, at 3:34 AM, Dmitry V'yal wrote:

I have a node with a runlist:

"run_list": [
"recipe[apt::default]",
"recipe[nginx::passenger]",
"recipe[mysql::server]",
"recipe[php-fpm::default]",
"recipe[monit::default]",
"recipe[bodhysite::default]",
"recipe[mail-system::vdomains]"
],

Just to note, you don't need to specify default, it is implied if a recipe isn't specified for the cookbook.

E.g., "recipe[apt]" is the same as "recipe[apt::default]".

That said...

nginx::passenger starts with include_recipe "build-essential".

The chef run fails saying this: http://pastebin.com/H7nFSqdy
Looks like mysql::client fails while building mysql gem because mkfs from ruby-dev is missing. So I added

r1 = package "ruby1.8-dev"
r1.run_action(:install)
...
Necessary headers are there, the root of problem is what gcc isn't installed. But build-essential included as part of nginx::passenger earlier than mysql. I even tried to add it into run_list as a separate recipe right after apt::default, it didn't helped.

Recall that the Chef client run processes recipes in two phases[0] - the compile phase and the execution phase. During the compile phase your Ruby code is processed:

r1 = package "ruby1.8-dev"
r1.run_action(:install)

So this gets evaluated and the resource is created and the package installed. The mysql cookbook uses this 'trick' to ensure the mysql gem can be installed and later used in the mysql cookbook's library in the same Chef client run.

The build-essential and ruby cookbooks do not do this (the Opscode Ruby cookbook installs the ruby dev package, so you can use that). They create the package resources during the compile phase as normal but Chef doesn't actually run the package installation until the execution phase. You should modify your local copy of the build-essential recipe to have the following block for the package installation:

%w{build-essential binutils-doc}.each do |pkg|
p = package pkg do
action :nothing
end
p.run_action(:install)
end

And the ruby recipe to have the following:

extra_packages.each do |pkg|
p = package pkg do
action :nothing
end
p.run_action(:install)
end

Upload the cookbooks to the Chef Server and rerun Chef on the node in question.

Since we (Opscode) first distribute Chef as RubyGems, the recipes are a bit narrow-minded in assuming that by the time Chef runs for the first time the C compiler and Ruby dev packages are available, because that is required to build Gems (like json) with native extensions to get Chef installed in the first place.

It sounds like you have installed Chef from Debian/Ubuntu packages, which is good, but it does mean this adjustment in your local recipe is necessary.

That said, we'll look at how to make these cookbooks better, and the mysql cookbook's readme will be updated to reflect this "requirement".

[0]: Anatomy of a chef run: http://wiki.opscode.com/display/chef/Anatomy+of+a+Chef+Run

--
Opscode, Inc
Joshua Timberman, Senior Solutions Engineer
C: 720.334.RUBY E: joshua@opscode.com

4

On 15.07.2010 14:21, sahil.cooner@gmail.com wrote:

You do need build-essential, that will install gcc for you, I do not see
it in your run_list and just checked the mysql cookbooks for
server/client that is also pulled in by mysql::server and I didn't see
an include_recipe.

--sahil

well, probably I wasn't clear enough. It's included as part of
nginx::passenger recipe.

Still for testing I've included it explicitly

% knife role show webserver
{
"name": "webserver",
"default_attributes": {

},
"json_class": "Chef::Role",
"run_list": [
"recipe[apt::default]",
"recipe[build-essential::default]",
"recipe[nginx::passenger]",
"recipe[mysql::server]",
"recipe[php-fpm::default]",
"recipe[monit::default]",
"recipe[bodhysite::default]",
"recipe[mail-system::vdomains]"
],
"description": "nginx with passenger and mysql",
"chef_type": "role",
"override_attributes": {
"mysql": {
"bind_address": [
"localhost"
]
}
}
}

This gives the following(slightly truncated at top):

[Thu, 15 Jul 2010 16:00:11 +0000] DEBUG: Sending HTTP Request via GET to
chef-server.vpn.bodhy.ru:4000/roles/webserver
[Thu, 15 Jul 2010 16:00:12 +0000] DEBUG: Applying attributes from json file
[Thu, 15 Jul 2010 16:00:12 +0000] DEBUG: Loading Recipe apt::default via
include_recipe
[Thu, 15 Jul 2010 16:00:12 +0000] DEBUG: Found recipe default in
cookbook apt
[Thu, 15 Jul 2010 16:00:12 +0000] DEBUG: Executing apt-get update
[Thu, 15 Jul 2010 16:00:12 +0000] DEBUG: ---- Begin output of apt-get
update ----
[Thu, 15 Jul 2010 16:00:12 +0000] DEBUG: STDOUT: Hit
http://security.ubuntu.com lucid-updates Release.gpg
Hit http://security.ubuntu.com lucid-security Release.gpg
Hit http://us.archive.ubuntu.com lucid Release.gpg
Hit http://us.archive.ubuntu.com lucid-updates Release.gpg
Hit http://security.ubuntu.com lucid-updates Release
Hit http://us.archive.ubuntu.com lucid Release
Hit http://us.archive.ubuntu.com lucid-updates Release
Hit http://security.ubuntu.com lucid-security Release
Hit http://security.ubuntu.com lucid-updates/main Packages
Hit http://security.ubuntu.com lucid-updates/restricted Packages
Hit http://security.ubuntu.com lucid-updates/main Sources
Hit http://security.ubuntu.com lucid-updates/restricted Sources
Hit http://us.archive.ubuntu.com lucid/main Packages
Hit http://us.archive.ubuntu.com lucid/restricted Packages
Hit http://us.archive.ubuntu.com lucid/main Sources
Hit http://us.archive.ubuntu.com lucid/restricted Sources
Hit http://us.archive.ubuntu.com lucid/universe Packages
Hit http://us.archive.ubuntu.com lucid/universe Sources
Hit http://us.archive.ubuntu.com lucid-updates/universe Packages
Hit http://us.archive.ubuntu.com lucid-updates/universe Sources
Hit http://security.ubuntu.com lucid-security/main Packages
Hit http://security.ubuntu.com lucid-security/restricted Packages
Hit http://security.ubuntu.com lucid-security/main Sources
Hit http://security.ubuntu.com lucid-security/restricted Sources
Hit http://security.ubuntu.com lucid-security/universe Packages
Hit http://security.ubuntu.com lucid-security/universe Sources
Hit http://apt.opscode.com lucid Release.gpg
Hit http://apt.opscode.com lucid Release
Ign http://apt.opscode.com lucid/main Packages
Ign http://apt.opscode.com lucid/main Packages
Hit http://apt.opscode.com lucid/main Packages
Reading package lists...
[Thu, 15 Jul 2010 16:00:12 +0000] DEBUG: STDERR:
[Thu, 15 Jul 2010 16:00:12 +0000] DEBUG: ---- End output of apt-get
update ----
[Thu, 15 Jul 2010 16:00:12 +0000] DEBUG: Ran apt-get update returned 0
[Thu, 15 Jul 2010 16:00:12 +0000] INFO: Ran execute[apt-get update]
successfully
[Thu, 15 Jul 2010 16:00:12 +0000] DEBUG: Loading Recipe
build-essential::default via include_recipe
[Thu, 15 Jul 2010 16:00:12 +0000] DEBUG: Found recipe default in
cookbook build-essential
[Thu, 15 Jul 2010 16:00:12 +0000] DEBUG: Loading Recipe nginx::passenger
via include_recipe
[Thu, 15 Jul 2010 16:00:12 +0000] DEBUG: Found recipe passenger in
cookbook nginx
[Thu, 15 Jul 2010 16:00:12 +0000] DEBUG: Loading Recipe build-essential
via include_recipe
[Thu, 15 Jul 2010 16:00:12 +0000] DEBUG: Found recipe default in
cookbook build-essential
[Thu, 15 Jul 2010 16:00:12 +0000] DEBUG: Setting
package[build-essential] to the state of the prior package[build-essential]
[Thu, 15 Jul 2010 16:00:12 +0000] DEBUG: Setting package[binutils-doc]
to the state of the prior package[binutils-doc]
[Thu, 15 Jul 2010 16:00:12 +0000] DEBUG: Setting package[autoconf] to
the state of the prior package[autoconf]
[Thu, 15 Jul 2010 16:00:12 +0000] DEBUG: Setting package[flex] to the
state of the prior package[flex]
[Thu, 15 Jul 2010 16:00:12 +0000] DEBUG: Setting package[bison] to the
state of the prior package[bison]
[Thu, 15 Jul 2010 16:00:13 +0000] DEBUG: Loading Recipe mysql::server
via include_recipe
[Thu, 15 Jul 2010 16:00:13 +0000] DEBUG: Found recipe server in cookbook
mysql
[Thu, 15 Jul 2010 16:00:13 +0000] DEBUG: Loading Recipe mysql::client
via include_recipe
[Thu, 15 Jul 2010 16:00:13 +0000] DEBUG: Found recipe client in cookbook
mysql
[Thu, 15 Jul 2010 16:00:13 +0000] DEBUG: Checking apt-cache policy for
libmysqlclient-dev
[Thu, 15 Jul 2010 16:00:13 +0000] DEBUG: Current version is
5.1.41-3ubuntu12.3
[Thu, 15 Jul 2010 16:00:13 +0000] DEBUG: Current version is
5.1.41-3ubuntu12.3
[Thu, 15 Jul 2010 16:00:13 +0000] DEBUG: Checking apt-cache policy for
mysql-client
[Thu, 15 Jul 2010 16:00:13 +0000] DEBUG: Current version is
5.1.41-3ubuntu12.3
[Thu, 15 Jul 2010 16:00:13 +0000] DEBUG: Current version is
5.1.41-3ubuntu12.3
[Thu, 15 Jul 2010 16:00:13 +0000] DEBUG: Checking apt-cache policy for
ruby1.8-dev
[Thu, 15 Jul 2010 16:00:13 +0000] DEBUG: Current version is 1.8.7.249-2
[Thu, 15 Jul 2010 16:00:13 +0000] DEBUG: Current version is 1.8.7.249-2
[Thu, 15 Jul 2010 16:00:13 +0000] DEBUG: No installed version found for
mysql (>= 0, runtime)
[Thu, 15 Jul 2010 16:00:16 +0000] DEBUG: Found gem mysql version 2.8.1
for platform ruby from http://gems.rubyforge.org/
[Thu, 15 Jul 2010 16:00:16 +0000] INFO: Installing gem_package[mysql]
version 2.8.1
Building native extensions. This could take a while...
[Thu, 15 Jul 2010 16:00:17 +0000] ERROR: Running exception handlers
[Thu, 15 Jul 2010 16:00:17 +0000] ERROR: Exception handlers complete
[Thu, 15 Jul 2010 16:00:17 +0000] ERROR: Re-raising exception:
Gem::Installer::ExtensionBuildError - ERROR: Failed to build gem native
extension.

/usr/bin/ruby1.8 extconf.rb
checking for mysql_ssl_set()... no
checking for rb_str_set_len()... no
checking for rb_thread_start_timer()... no
checking for mysql.h... no
checking for mysql/mysql.h... no
*** extconf.rb failed ***

Tests fail because gcc is not installed. Looks like the mysql cookbook
gets executed just after apt.

Best wishes,
Dmitry

5

... snip ...
Since we (Opscode) first distribute Chef as RubyGems, the recipes are a bit narrow-minded in assuming that by the time Chef runs for the first time the C compiler and Ruby dev packages are available, because that is required to build Gems (like json) with native extensions to get Chef installed in the first place.

It sounds like you have installed Chef from Debian/Ubuntu packages, which is good, but it does mean this adjustment in your local recipe is necessary.

That said, we'll look at how to make these cookbooks better, and the mysql cookbook's readme will be updated to reflect this "requirement".

Every time I do the initial setup of Chef for a new environment I get
bitten by this, and I doubt I'm the only one. Is there any chance of
adding these assumed dependencies to the recommended packages for
Chef? That way everything works as expected, and people who are
disgusted by the thought of having a compiler on their server can
choose to remove it.

6

On 15.07.2010 19:58, Joshua Timberman wrote:

Recall that the Chef client run processes recipes in two phases[0] - the compile phase and the execution phase. During the compile phase your Ruby code is processed:

r1 = package "ruby1.8-dev"
r1.run_action(:install)

So this gets evaluated and the resource is created and the package installed. The mysql cookbook uses this 'trick' to ensure the mysql gem can be installed and later used in the mysql cookbook's library in the same Chef client run.

The build-essential and ruby cookbooks do not do this (the Opscode Ruby cookbook installs the ruby dev package, so you can use that). They create the package resources during the compile phase as normal but Chef doesn't actually run the package installation until the execution phase. You should modify your local copy of the build-essential recipe to have the following block for the package installation:

Thanks a lot, seems I got it. Are there other cookbooks exploiting the
similar idea?

Dmitry