MySQL and PostgreSQL cookbooks, RubyGems and You!


#1

Hey Chefs!

I have a topic I’d like to discuss with the community, with regard to
behavior in the “mysql” and “postgresql” (and related, the “database”)
cookbooks we publish. First, a couple questions about your
expectations of these cookbooks (whether you use them or not):

  1. Do you expect that the “client” recipe for mysql/postgresql will
    install Ruby libraries?
  2. If so, do you expect that because you want to use them within the
    same Chef run? Or because you want to use them with your
    application(s)?

The original intention of the package/gem installation of mysql and pg
Ruby libraries in the client recipes was that they should be installed
because one would want them used for an application. This is a 3+ year
old assumption, and probably not the case now. It seems that people
really only want/need those installed when using the Ruby libraries
through a resource/provider elsewhere in the Chef run, such as with
the “database” cookbook.

There are some tickets about this topic in the COOK project[0], and I
would like to get consensus around resolving the issue for the
broadest use case(s). The behavior between the two client recipes is
inconsistent, and we’d like to resolve that, as well.

Current behavior

In both cookbooks, the “client packages” are installed, such that the
node can interact with the database through the native C libraries and
client-side programs.

In mysql::client:

  • On Red Hat family distributions, the system package “ruby-mysql” is installed.
  • On Debian family distributions, the system package "libmysql-ruby"
    is installed.
  • On all other platforms, the RubyGem “mysql” is installed.

Issues with this:

  • It is inflexible. If you want to use the Gem for your application,
    it won’t be installed on RHEL/Debian[1].
  • Other platforms might have native packages where the library is
    already compiled, and there isn’t the option for the native package.
  • The gem installation uses the “gem_package” resource. This doesn’t
    necessarily make the gem available to be used by Chef, especially
    under the “omnibus” full stack package.

In postgresql::client:

  • The postgresql client libraries and the RubyGem are installed during
    the compile phase of the Chef run.
  • The “pg” gem is installed on all platforms.

Issues with this:

  • COOK-1011 brings up the issue that during the compile phase, the
    package cache on Debian/Ubuntu may not be updated, causing the
    installation to fail.
  • Users who wish to use the native system package don’t have an option
    without modifying the recipe.

Propsal

I have a proposal, generalized for both cookbooks, which I think will
give use more flexibility, and be the least surprising.

  1. Move the list of package names for client installation and server
    installation to attributes in both cookbooks. They would be:

    node[‘mysql’][‘client_packages’]
    node[‘mysql’][‘server_packages’]
    node[‘postgresql’][‘client_packages’]
    node[‘postgresql’][‘server_packages’]

Respectively, and platform specific package names would be selected
using a case statement. This is proposed in part for the ticket
COOK-1236[2]. This replaces the current use of a local variable.

  1. Remove the RubyGem installation from both cookbooks’ client recipes.

  2. Add an attribute that indicates the name of the native package that
    should be installed, and will be false by default, meaning that the
    package wouldn’t be installed. Proposed attribute names:

    node[‘mysql’][‘ruby_package’]
    node[‘postgresql’][‘ruby_package’]

Then, a user can set the attribute (e.g., via a role) to the package
name if they want to install it.

  1. In the “database” cookbook, create “mysql” and “postgresql” recipes
    which handle installing the RubyGems “mysql” and “pg” respectively,
    using the chef_gem resource so they can be used by the database
    resources.

  2. Handle installing build-essential’s packages during recipe compile
    phase so chef_gem works, and also ensure that the package cache is
    updated on Debian family.

I have opened COOK-1441 for tracking the above, and I am working on
migrating related tickets to sub-tasks (or resolving duplicates).

Thoughts? We’d love to hear your feedback here or on the ticket.


Opscode, Inc
Joshua Timberman, Technical Program Manager
IRC, Skype, Twitter, Github: jtimberman

[0]: COOK-1009, COOK-1011, COOK-1384, and others that have been closed
as duplicates.
[1]: this is the general platform families, including centos, fedora,
amazon, ubuntu, etc.
[2]: this ticket in particular allows the use of percona instead of mysql, too.


#2

Ohai again, Chefs!

While we haven’t seen a lot of response on this topic, or on the related tickets[0], the general answers to my questions in the original email are answered thus:

  1. Do you expect that the “client” recipe for mysql/postgresql will
    install Ruby libraries?

Answer: No. A separate recipe is better, and can be included if desired.

  1. If so, do you expect that because you want to use them within the
    same Chef run? Or because you want to use them with your
    application(s)?

Answer: As the above was no, the general use is for Chef, and not for an application deployment. Applications that require Ruby bindings often will use bundler during deployment to get the right gems for the app. This is consistent with Ruby community best practices.

As we’ve not seen obvious objections to the proposal, I think we’ll proceed with the changes outlined below. I do have some notes/updates inline, but nothing dramatic.

  1. Move the list of package names for client installation and server
    installation to attributes in both cookbooks. They will be:

    node[‘mysql’][‘client_packages’]
    node[‘mysql’][‘server_packages’]
    node[‘postgresql’][‘client_packages’]
    node[‘postgresql’][‘server_packages’]

Respectively, and platform specific package names would be selected
using a case statement. This is proposed in part for the ticket
COOK-1236[2]. This replaces the current use of a local variable.

I added COOK-1532 for tracking this for the postgresql cookbook.

  1. Remove the RubyGem installation from both cookbooks’ client recipes.

  2. Add an attribute that indicates the name of the native package that
    should be installed, and will be false by default, meaning that the
    package wouldn’t be installed. Proposed attribute names:

node[‘mysql’][‘ruby_package’]
node[‘postgresql’][‘ruby_package’]

Then, a user can set the attribute (e.g., via a role) to the package
name if they want to install it.

Rather than add a separate attribute, a user will simply add the name of the package for the Ruby bindings to the node attribute for the client packages from #1 above if they want the native package for the database Ruby libraries.

  1. In the “database” cookbook, create “mysql” and “postgresql” recipes
    which handle installing the RubyGems “mysql” and “pg” respectively,
    using the chef_gem resource so they can be used by the database
    resources.

The recipe will actually be “ruby” in the two cookbooks, and the database cookbook’s recipes will include it so you can use “database::mysql” or “database::postgresql”.

  1. Handle installing build-essential’s packages during recipe compile
    phase so chef_gem works, and also ensure that the package cache is
    updated on Debian family.

This will use the new attribute in build-essential, and create a “soft” dependency on the build-essential cookbook. As it will not be the default behavior of the cookbooks to install the Ruby gems, we won’t create a hard dependency on the build-essential cookbook, and document its requirement for those recipes’ use only.

These will be all reflected in sub-task tickets of COOK-1441.

Questions or comments:


Opscode, Inc
Joshua Timberman, Technical Program Manager
IRC, Skype, Twitter, Github: jtimberman


#3

On 07/12/2012 08:14 AM, Joshua Timberman wrote:

Hey Chefs!

I have a topic I’d like to discuss with the community, with regard to
behavior in the “mysql” and “postgresql” (and related, the “database”)
cookbooks we publish. First, a couple questions about your
expectations of these cookbooks (whether you use them or not):

  1. Do you expect that the “client” recipe for mysql/postgresql will
    install Ruby libraries?
  2. If so, do you expect that because you want to use them within the
    same Chef run? Or because you want to use them with your
    application(s)?

Thoughts? We’d love to hear your feedback here or on the ticket.

My predominant thought is: KISS.

The simpler the recipes, the more useful they become for general use.
The only thing I have ruby installed on our servers for is Chef. I gain
no benefits (currently?) from having the Ruby MySQL clients installed.
Same would go for Python I’d hazard a guess. I’d rather such decisions
were handled independently, or maybe split it out into individual
recipes within the cookbook. The Nagios cookbook gives you a variety of
options about how Nagios is installed, be it from source or packages
etc. Maybe something along those lines would be good?

When I first started using Chef I focussed around grabbing cookbooks out
of the central repository and trying to use them on our servers but
after finding myself spending a whole heap of time fighting with
dependencies for things that I didn’t need (and battling CentOS 5&6
quirks), I ultimately gave up and just wrote my own from scratch that
are entirely focused around our use case. The cookbooks have mostly
provided value as demonstrating the capabilities of Chef and various
tricks you can use.

Paul