Ruby Gem Dependencies

Chef Infra (archive)
1

Cookbook Ruby Gem Dependencies:
Extending Chef Client’s Set of Ruby Gems: This document provides background and proposes a solution to topic of extending the Ruby environment provided by the Chef Client package with additional Ruby Gem packages

There are isolated cases in which Chef cookbooks developed by employer need to leverage Ruby libraries that are not bundled in the Chef Client package provided by Chef Software. Since the Gem package format is the means by which libraries and other code are managed in Ruby, any extensions to the Ruby environment provided by the Chef Client package involves use of additional Gem packages.

Current Approach:
We introduced several dependencies on Gems that historically have not been included in the Chef client package. To address these gaps, we packaged necessary Gems and their dependencies in “chefgem-…” RPMs.

Problem:
Since the set of gems embedded in the Chef client changes from version to version, our current chefgem RPMs are not always compatible with what is already embedded. This means the chefgem packages would not always work with a particular version of Chef client and this needs to be addressed.

For example, when we need to ensure the set of employers Gem extensions are added to traditionally managed nodes that have later versions of the Chef Client package, we’re forced to install at least one package with a --nodeps override to get the package installed. We clearly can’t continue applying such a workaround.

Assumptions

Extending the Ruby environment provided by the Chef Client packages needs to be supported
Since we expect the situations to be relatively limited, a pull request to a central management team is an acceptable part of the process
We should avoid implementation approaches where binary Gem packages are bundled in cookbooks so that we:
Avoid duplication of binary packages when multiple cookbooks need to leverage common components
Avoid complicating Chef run lists through the inclusion of cookbooks required to install extension Gems
Approach needs to account for multiple OS platforms
Approach needs to address cases in which the Chef DK is used

solution we are proposing :

We are going to create a single RPM package that is chef-client. This will be reoccurring process so that whenever a new version of chef client introduces we would need to upgrade our chef-client package each time. So that It can bundle the latest chef client along with its required dependent gem versions.Part of it will complement the following build script that eases the process of generating native RPMs/Deb packages given a gem of interest.

for ex. To package chef-client-12.5.1-1 we require chef-12.5.1-1, rest-client & nokogiri-diff. To be more specific below would be the steps to achieve this :
We need to create a single gem RPM and other gem packages
Secondly ,we need to create chef-client package bundling chef-12.5.1-1 and chef-gem altogether

Please review and suggest other workarounds.

2

I am not sure if this really is a solution for you, but are you aware of the new (since 12.5 I think) gem option in the metadta.rb [1]. You can specify gems and their versions in the cookbook which requires the gem and they will be installed before any cookbook is loaded.

You can also specify your own local gem server in the /etc/chef/client.rb with rubygems_url if you require so. This option seems to be undocumented but the code [2] says that it is there.

[1] https://docs.chef.io/config_rb_metadata.html
[2] https://github.com/chef/chef/blob/master/lib/chef/cookbook/gem_installer.rb

3

Hi @sharayumk

If I understand correctly

You have packaged ruby gems as RPM packages that are specifically packaged for a set chef version,

My suggestions would be

Option 1
On upgrade of Chef Client you would need regenerate the mc-chefgem-* packages building each gem-rpm with the target versions of Chef installed. This should negate the need to force package installations.

Option 2
I would see the ideal solution that works across OSs and require no duplicate packaging to be using an internal GEM repository to serve these external dependencies. But you would now have a need to manage the life cycle of this new package format.

How to do this would be to create the internal GEM repository and then update the Chef embedded gem sources by removing the public mirrors and adding only your internal repository.

This way the standard installation and dependency resolution work eg from in cookbooks, metadata.rb with only minimal client configuration required at the point that chef-client is bootstrapped. It is also will only install extra gems where they are actually required for a converge to succeed.

Note: This may introduce new dependencies as during the GEM installations they may require build dependencies on the OS platform being targeted, eg: build-essential , libxml…

4

That's the exact use case of chef_gem resource (maybe with compile_time true) at start of those cookbooks recipes.
How does this (and now the metadata gem keyword) doesn't fulfill your needs ?

Main advantage when cookbooks manage their own dependencies instead of relying on a central cookbook to that is that they don't have to all update at the same time if there's breaking changes.

It is, chef_gem resource is exactly meant to install a gem into chef environment.

This one is really frightening me, which use cases do you have needing chef-dk on nodes ? That sounds heavily counter productive when nodes start to change others desired state, this should be done in another way IMHO.

If you really want to check which gem is included on the nodes's chef environment, a cookbook only responsible of this part, mandatory in each node runlist, and a foodcritic rule to prohibit any chef_gem in other cookbooks would do the trick.
Bonus: you can update a gem version (for security issues for example) without the need to repackage chef-client and redistribute it on all nodes.

My conclusion: Adding a gem into the chef environment is not different than any other configuration management and should not be done outside of Chef control.