If you’ve seen an error like this Gem::ConflictError: Unable to activate aws-sdk-ec2-1.419.0, because aws-sdk-core-3.119.1 conflicts with aws-sdk-core (~> 3, >= 3.184.0) or any Gem conflict error while working with Chef, this may be helpful to you.
When working with libraries in a cookbook, it’s common to require specific gems. In order to get these gems installed on the node, people often just use the gem setting in the metadata.rb file. This is great but you need to make sure that your gems are pinned so that you don’t run into conflicting gem errors. This is especially important when it comes to any gems that are the same as, or dependent on, gems shipped with Chef Client.
For this example:
- I’ll focus on the
aws-sdk-ec2gem which depends onaws-sdk-corewhich is shipped with Chef Client - I’ll simulate an upgrade from Chef Client v17.4.38 to v18.2.7
After looking at the Chef Client Gemfile.lock for v17.4.38 on Github we can see that the shipped version of aws-sdk-core is v3.119.1. We can see on rubygems that the latest version of aws-sdk-ec2 that will work with this version of core is v1.419.0. So I could just update the gem setting in the metadata.rb file and upgrade Chef Client on my nodes, right? Yes, you could do that, but this does not allow for a safe, rolling update. You’d need to update your cookbook and push out the Chef Client update at the same time, which greatly increases risk.
So how can we do this in a safer way?
First off, we’ll need to move the responsibility of installing gems out of the metadata.rb file and into code somewhere because you can’t really put any code or environment variables into the metadata.rb file. This is where the chef_gem resource comes into play. Using something like the following snippet, we can now install the aws-sdk-ec2 at v1.419.0 in our code.
chef_gem "aws-sdk-ec2" do
compile_time true
version "1.419.0"
end
This brings up another problem: Library files are compiled at compile time, so your library code where you have require 'aws-sdk-ec2’ will be executed before this gem is actually installed by the chef_gem resource (this will result in a cannot load such file -- aws-sdk-ec2 error). We can get around this by adding the require <gem> to a method like so:
example_cookbook/libraries/ec2.rb
# require 'aws-sdk-ec2'
module EC2Helper
def self.require_gem
require 'aws-sdk-ec2'
end
def self.ec2_setup
ec2 = Aws::EC2::Client.new(profile: 'my_profile', region: 'us-east-1')
end
end
Then, in our recipe, we can call this require_gem method after the gems are actually installed.
example_cookbook/recipes/default.rb
chef_gem "aws-sdk-ec2" do
compile_time true
version "1.419.0"
end
EC2Helper.require_gem
This way your libraries can be compiled, and the gem installation can be handled in a recipe.
Now that our gem dependency is fully handled in code, it’s time to make it dynamic, so it can work with multiple versions of Chef Client that ultimately require different gem versions to avoid Gem conflict errors.
You can use this line to determine the version of Chef Client this cookbook is being run on:
client_version = Chef::VERSION
And optionally you can use these lines to double check that the version of aws-sdk-core is the version you expect it to be (if you’re going to do this, make sure gem ‘rubygems’ is added to your metadata.rb file):
require 'rubygems'
installed_gem_core_version = Gem.loaded_specs["aws-sdk-core"].version.to_s
Now, putting it all together with some conditionals, we get something like this:
example_cookbook/recipes/default.rb
# Cookbook:: example
# Recipe:: default
#
# Copyright:: 2023, The Authors, All Rights Reserved.
require 'rubygems'
# In order to avoid gem depedency error we are installing aws-sdk-ssm and aws-sdk-ec2 depending on
# the aws-sdk-core that is shipped with a specific Chef-Client version
client_version = Chef::VERSION
installed_gem_core_version = Gem.loaded_specs["aws-sdk-core"].version.to_s
gems = nil
case client_version
when '17.4.38'
if (installed_gem_core_version == '3.119.1')
# Chef-Client 17.4.38, ships with "aws-sdk-core" => "3.119.1", "aws-sdk-ssm" => "1.110.0", "aws-sdk-secretsmanager" => "1.48.0"
gems = {"aws-sdk-ssm" => "1.110.0", "aws-sdk-ec2" => "1.260.0"}
else
raise "Unsupported aws-sdk-core version got #{installed_gem_core_version}, expected 3.119.1 for chef-client version 17.4.38"
end
when '18.2.7'
if (installed_gem_core_version == '3.171.0')
# Chef-Client 18.2.7, ships with "aws-sdk-core" => "3.171.0", "aws-sdk-ssm" => "1.150.0", "aws-sdk-secretsmanager" => "1.73.0"
gems = {"aws-sdk-ssm" => "1.150.0", "aws-sdk-ec2" => "1.380.0" }
else
raise "Unsupported aws-sdk-core version got #{installed_gem_core_version}, expected 3.171.0 for chef-client version 18.2.7"
end
else
raise "Unsupported Chef Infra Client version installed. Got #{client_version}, expected 17.4.38, 18.2.7"
end
gems.each do |key, value|
chef_gem "#{key}" do
compile_time true
version "#{value}"
end
end
EC2Helper.require_gem
This code will dynamically install versions of gems that don’t conflict with the gems shipped with Chef Client.
Note that you could also do this all in the library file by writing ruby code, but I prefer this method as it’s much easier to find, read, and manage.
This is great for doing rolling upgrades of Chef Client. This also allows for your cookbook code to stay the same, but work with multiple versions of Chef Client at the same time.
I’m sure there are other use cases for this method but using it to cleanly upgrade the Chef Client is a great use case.