Rubygems Issue affecting some ChefDK Users

Hi Chefs,

I’d like to alert those of you that use ChefDK to an issue we discovered after yesterday’s release of ChefDK 0.2.1. If you don’t use ChefDK, you might still want to pay attention since the root of the problem is caused by a change in rubygems 2.2.

This issue will affect any users who’ve installed gems with C extensions into ChefDK’s ruby, but will be most severe for those who started using ChefDK before 0.1.0.

The issue is that rubygems, starting with 2.2, has changed the on-disk layout of gems so that gems can be shared between multiple ruby versions to save disk space. As a consequence, the place where compiled C extensions are located had to change since compiled extensions cannot be shared between different Ruby versions (based on ABI level). To account for this, rubygems 2.2.0 attempts to lazily rebuild gem extensions in the new location after rubygems is upgraded. This causes two problems for ChefDK users:

  1. If you used ChefDK prior to 0.1.0, and you installed gems with C extensions, you will have root-owned gems installed to ChefDK’s ruby. Rubygems will not be able to rebuild the gem without root permissions, so it will emit an error like Error loading RubyGems plugin "/opt/chefdk/embedded/etc…” Permission denied @ dir_s_mkdir
  2. If you have only used ChefDK 0.1.0 and above, and you installed gems with C extensions, those gems’ files will be owned by your user and located in ~/.chefdk/gem. Rubygems should be able to auto-update these gems, but some users have reported errors with the process.

If you’re affected by case #1, your best bet is to completely uninstall ChefDK, remove any leftovers from /opt/chefdk, and re-install. You’ll need to re-run chef gem install for any gems that were installed to the embedded ruby’s gem directory. You may also be able to sudo gem pristine GEM_NAME these gems, but the changes that sudo typically makes to your environment variables means this might not work without some troubleshooting.

If you’re affected by case #2, and the auto-update process fails for some reason, the best way to go is to remove the ~/.chefdk/gem directory and re-install any gems you had installed previously. You can also try to chef gem pristine GEM_NAME these gems if you don’t want to re-install.

I’ve closed the ChefDK issue related to this change since there isn’t really any way for us to make this easier, unfortunately. But feel free to comment on the issue if you’re having trouble with any of the fixes/workarounds.

Finally, I will note that I believe the Rubygems developers made a reasonable effort to maintain compatibility and were caught unaware by some of the edge cases (like ours) that surfaced after the feature was released, even though the outcome is frustrating. Let’s not spew vitriol at them, okay?

Thanks,


Daniel DeLeo

References:

ChefDK bug report: https://github.com/opscode/chef-dk/issues/146

Rubygems layout change PR for 2.2: https://github.com/rubygems/rubygems/pull/596
Rubygems bug when attempting to update root-owned gems: https://github.com/rubygems/rubygems/issues/796
Bundler bug report related to repo layout change: https://github.com/bundler/bundler/issues/2847
Rubygems bug report for things being broken by the 2.2 update: https://github.com/rubygems/rubygems/issues/874

All I read was "spew vitriol at them" brb firing up the Hate Furnace OF DOOM!

loljk

Nice work Dan.

--aj

On Sat, Aug 30, 2014 at 6:46 AM, Daniel DeLeo dan@kallistec.com wrote:

Hi Chefs,

I’d like to alert those of you that use ChefDK to an issue we discovered after yesterday’s release of ChefDK 0.2.1. If you don’t use ChefDK, you might still want to pay attention since the root of the problem is caused by a change in rubygems 2.2.

This issue will affect any users who’ve installed gems with C extensions into ChefDK’s ruby, but will be most severe for those who started using ChefDK before 0.1.0.

The issue is that rubygems, starting with 2.2, has changed the on-disk layout of gems so that gems can be shared between multiple ruby versions to save disk space. As a consequence, the place where compiled C extensions are located had to change since compiled extensions cannot be shared between different Ruby versions (based on ABI level). To account for this, rubygems 2.2.0 attempts to lazily rebuild gem extensions in the new location after rubygems is upgraded. This causes two problems for ChefDK users:

  1. If you used ChefDK prior to 0.1.0, and you installed gems with C extensions, you will have root-owned gems installed to ChefDK’s ruby. Rubygems will not be able to rebuild the gem without root permissions, so it will emit an error like Error loading RubyGems plugin "/opt/chefdk/embedded/etc…” Permission denied @ dir_s_mkdir
  2. If you have only used ChefDK 0.1.0 and above, and you installed gems with C extensions, those gems’ files will be owned by your user and located in ~/.chefdk/gem. Rubygems should be able to auto-update these gems, but some users have reported errors with the process.

If you’re affected by case #1, your best bet is to completely uninstall ChefDK, remove any leftovers from /opt/chefdk, and re-install. You’ll need to re-run chef gem install for any gems that were installed to the embedded ruby’s gem directory. You may also be able to sudo gem pristine GEM_NAME these gems, but the changes that sudo typically makes to your environment variables means this might not work without some troubleshooting.

If you’re affected by case #2, and the auto-update process fails for some reason, the best way to go is to remove the ~/.chefdk/gem directory and re-install any gems you had installed previously. You can also try to chef gem pristine GEM_NAME these gems if you don’t want to re-install.

I’ve closed the ChefDK issue related to this change since there isn’t really any way for us to make this easier, unfortunately. But feel free to comment on the issue if you’re having trouble with any of the fixes/workarounds.

Finally, I will note that I believe the Rubygems developers made a reasonable effort to maintain compatibility and were caught unaware by some of the edge cases (like ours) that surfaced after the feature was released, even though the outcome is frustrating. Let’s not spew vitriol at them, okay?

Thanks,

--
Daniel DeLeo

References:

ChefDK bug report: Error messages when doing `chef gem list` · Issue #146 · chef-boneyard/chef-dk · GitHub

Rubygems layout change PR for 2.2: support for GEM_HOME_SHARED by mpapis · Pull Request #596 · rubygems/rubygems · GitHub
Rubygems bug when attempting to update root-owned gems: Possible wrong condition in `build_extensions' ? · Issue #796 · rubygems/rubygems · GitHub
Bundler bug report related to repo layout change: Bundler should build extensions into a new directory on RubyGems 2.2+ · Issue #2847 · rubygems/bundler · GitHub
Rubygems bug report for things being broken by the 2.2 update: Rubygems 2.2 breaks compatibility with 2.1 gem extensions · Issue #874 · rubygems/rubygems · GitHub