Gem in cookbook libraries: what is The Right Way?

Chef Infra (archive)
1

Lately I’m working in an environment without direct access to the external network; this is turning out as a great occasion to find out how many cookbooks (both our private cookbooks and the supermarket ones) do stuff at compile time that they shouldn’t.

Now I’m having trouble using the docker cookbook, as it installs a gem at compile time:

# libraries/_autoload.rb
begin
  gem 'docker-api', '= 1.28.0'
rescue LoadError
  unless defined?(ChefSpec)
    run_context = Chef::RunContext.new(Chef::Node.new, {}, Chef::EventDispatch::Dispatcher.new)

    require 'chef/resource/chef_gem'

    docker = Chef::Resource::ChefGem.new('docker-api', run_context)
    docker.version '= 1.28.0'
    docker.run_action(:install)
  end
end

I’m trying to use this cookbook on a node that cannot reach rubygems.org, so I have a recipe that changes the server to download gems from, but of course my recipe is never run, as everything fails at compile time.

I’d like to submit a pull request to fix this, but I’m not sure of the solution.

The fact is that the docker cookbook doesn’t have any recipes: it has resources and a few helper methods; many of those resources and helpers need the gem to work.

I could change _autoload.rb to install the gem at compile time, thus allowing my recipe to change the gem server, but: how could I change the resources so that they actually use the gem at compile time only?

The first step is obviously to remove the require 'docker' from the global space and from classes/modules body and put it inside methods. This would avoid everything to fail immediately, but would not be enough: I should also ensure that the resource itself works (that is, compiles) without the gem.

In the end, what should happen is this:

  • wrappers of docker should be allowed to optionally prepend recipes that change the way the gem docker-api is installed;
  • then, the gem docker-api should be installed before any other recipe that want to use a docker resource;
  • then, all other recipes should follow.

Could you please give me some advice as for how to proceed?

2

We added official support for this in 12.6 or 12.7 (I forget which). Add gem 'docker-api' , '~> 1.28' to your metadata.rb just like you would depend on a cookbook.

2 Likes
3

@PietroGiorgianni We have gone through a similar process as yourself.

The observation was that Gems fell into two main categories

  • Those required to support the cookbooks
  • Those required to support applications being installed by the cookbooks

Our options were to create a closed/private gem repository paid for (Artifactory) and non paid (gem mirror, gem server) or to vendor any gems with the cookbooks, we also looked at Gem Fury. After a spike we choose the latter which proved to be a little awkward to implement in a generic manner, particularly for trying to provide gems for community cookbooks.

4

@coderanger We also tried adding gems to the metadata.rb but I couldn’t get this to work with Test kitchen. Have you any advice or an article you can point to?

Thanks in advance.

5

Check that your Chef version is new enough. Other than that, it’s a core Chef feature so TK shouldn’t be involved.

6

I’ll :thumbsup: @coderanger’s advice. Use metadata based gem for all library/cookbook needs. Take the omnibus or fpm route for all non-chef app code related code

In your case, you need a handful of extra gems and your target servers does not have rubygems access. I’ll suggest creating a build process, that

  • Install chef in a pristine node,
  • Install all your custom gems (docker in this case?) inside the omnibus gemset
  • Repack the whole /opt/chef using fpm and publish debian/rpm
  • Use the repackaged debian

I dont see any other route if you dont have access to rubygems from the target servers

7

That is just great! I seem to have missed that completely!

8

I also found that having some kind of a “bootstrap” (in our case thats also base) cookbook is quite helpful. I am also working in an environment with no access to the internet from our server and without a certain bootsrapping (setting RPM repos, importing certificates, gem repos etc.) pretty much nothing works. We run this base/bootstrap cookbook with “knife bootstrap” and afterwards all the rest gets included in the run_list.

9

So you do two chef runs, right?

That’s about where we are too: a first run that sets stuff and a second one with the actual run_list.

10

Yes, two chef runs. I know its not a very nice solution, but this first chef run happens only once when bootstrapping. Also, the boostrap/base cookbook only does the really most essential stuff which is needed anyway. For kitchen testing we have a base image with the most important stuff already included.
I would not do this in any other environment, but without internet access I found this to be best solution.