Introduction and Alternate Rubygems provider


#1

Hi,

I’m quickly introducing myself in the list and will then expose an issue we
have here and a way we found to fix it.

I’m Pierre Baillet, the lead server administrator at fotonauts. I’m in
charge of most of the servers used for the fotopedia website. We use Chef on
all our development, testing and production instances as a way to
standardise our deployments, keep a state of what’s installed on each server
in case of server failure and automate releases. We are quite happy with
Chef, especially when it comes to its powerful Search features that allows
building all our monitoring configuration automatically !

When installing (usually at bootstrap) complex projects, the regular
Rubygems Package Provider ( Chef::Provider::package::Rubygems ) does a lot
of gem calls, hence slowing the installation of gem quite lot. Moreover,
using chef recipes that include “~>” version checking are run at every chef
run and because we run chef every 5 minutes here at fotonauts (mainly as a
deployment helper) we appreciate every spared second during this procedure.

As a workaround for this, we have developed a Rubygems provider (based on a
monkey patched version of the regular provider) which no longer calls the
gem binary every time but:

  • Uses the Gem API to grab the list of available gems,
  • Keeps this list in a cache so that subsequent calls to the Provider are
    much faster

The only parts that use gem .. is the one that actually install the gems
although there is also a way to programmatically install a gem using the Gem
API.

By looking at the original source code of the Provider, I think that the
fact that it’s not using the Gem API is something that’s voluntary and I
guess that it’s to prevent any dependency with the internal Gem API. Am I
wrong ?

If this is the case, I suppose that our patched version will never make it
to the official HEAD (although I’m not sure I understand the interest of
this restriction today as Chef already relies on several external and
sometimes API-changing libraries :D).

However, if this is not the case, I’ll gladly provide a patched version of
the Rubygems provider with our enhancement, for testing first, of course :slight_smile:

Thank you very much for your time,

Pierre.


#2

Pierre,
This is awesome, and something we’ve wanted for a long time. Cf.
http://tickets.opscode.com/browse/CHEF-349

On Fri, May 14, 2010 at 7:19 AM, Pierre Baillet oct@fotonauts.com wrote:

Hi,
I’m quickly introducing myself in the list and will then expose an issue we
have here and a way we found to fix it.
I’m Pierre Baillet, the lead server administrator at fotonauts. I’m in
charge of most of the servers used for the fotopedia website. We use Chef on
all our development, testing and production instances as a way to
standardise our deployments, keep a state of what’s installed on each server
in case of server failure and automate releases. We are quite happy with
Chef, especially when it comes to its powerful Search features that allows
building all our monitoring configuration automatically !

When installing (usually at bootstrap) complex projects, the regular
Rubygems Package Provider ( Chef::Provider::package::Rubygems ) does a lot
of gem calls, hence slowing the installation of gem quite lot. Moreover,
using chef recipes that include “~>” version checking are run at every chef
run and because we run chef every 5 minutes here at fotonauts (mainly as a
deployment helper) we appreciate every spared second during this procedure.
As a workaround for this, we have developed a Rubygems provider (based on a
monkey patched version of the regular provider) which no longer calls the
gem binary every time but:

  • Uses the Gem API to grab the list of available gems,
  • Keeps this list in a cache so that subsequent calls to the Provider are
    much faster
    The only parts that use gem .. is the one that actually install the gems
    although there is also a way to programmatically install a gem using the Gem
    API.
    By looking at the original source code of the Provider, I think that the
    fact that it’s not using the Gem API is something that’s voluntary and I
    guess that it’s to prevent any dependency with the internal Gem API. Am I
    wrong ?

Now, the tricky part is that we support managing a different rubygems
installation than the one used by chef. This is what the gem_binary
parameter is for. I was also playing with converting the gem provider
to use rubygems’ internal API, and I think I have the solution. If you
shell out to gem env gemdir you will get one or more gem
directories, and the gemspecs will be in a subdirectory
$gemdir/specifications. You can then run Gem::SourceIndex.from_gems_in
to get the specs from that gem installation.

If this is the case, I suppose that our patched version will never make it
to the official HEAD (although I’m not sure I understand the interest of
this restriction today as Chef already relies on several external and
sometimes API-changing libraries :D).
However, if this is not the case, I’ll gladly provide a patched version of
the Rubygems provider with our enhancement, for testing first, of course :slight_smile:
Thank you very much for your time,

Pierre.

In summary, this is really cool, and if you can get it to manage
multiple gem installations, we’ll accept it into master. Hope my tip
is helpful.

Dan DeLeo