I’m working on getting unit testing for my Ruby application baked into my CI/CD pipeline. I could do that outside the context of habitat, but I’d rather do it within a studio context so I can be sure that everything will be similar to what the app deploys with, and also not have to manage that environment separately.
My first hurdle is that rake is not available in the default studio. To get it in there, I can:
build
hab pkg binlink core/ruby rake
but it still is missing some goodies to be functional:
[6][default:/src:0]# rake
rake aborted!
LoadError: cannot load such file -- bundler/setup
/src/config/boot.rb:3:in `<top (required)>'
/src/config/application.rb:1:in `require_relative'
/src/config/application.rb:1:in `<top (required)>'
/src/Rakefile:4:in `require_relative'
/src/Rakefile:4:in `<top (required)>'
(See full trace by running task with --trace)
I’ve gone a little ways down a rabbithole of binlinking and gem and bundle installs to try to get this functional, but it’s getting deep enough that I’m wondering if this is the right thing to be doing at all. Any thoughts on that? If running my unit tests from within habitat context is a good idea, is there a better way to get the pre-requisites in place to do it?
Playing with this a bit right now.
Found my first rabbit hole (@qhartman you may have found this too)
Since
build
hab pkg binlink core/ruby rake`
returns “LoadError: cannot load such file – bundler/setup” - I thought perhaps we need to run bundle install in the src directory.
I then tried
[28][default:/src:0]# /hab/pkgs/core/bundler/1.16.0/20171129183702/bin/bundle install
which is currently choking on nokogiri
Fetching gem metadata from https://rubygems.org/.........
Using rake 12.3.0
Using concurrent-ruby 1.0.5
Using i18n 0.9.5
Using minitest 5.11.3
Using thread_safe 0.3.6
Using tzinfo 1.2.5
Using activesupport 5.1.5
Using builder 3.2.3
Using erubi 1.7.1
Using mini_portile2 2.3.0
Fetching nokogiri 1.8.2
Installing nokogiri 1.8.2 with native extensions
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.
current directory:
/hab/pkgs/core/bundler/1.16.0/20171129183702/gems/nokogiri-1.8.2/ext/nokogiri
/hab/pkgs/core/ruby/2.4.2/20171014214232/bin/ruby -r
./siteconf20180507-9263-1egkm3u.rb extconf.rb
checking if the C compiler accepts ... *** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers. Check the mkmf.log file for more details. You may
need configuration options.
Provided configuration options:
--with-opt-dir
--without-opt-dir
--with-opt-include
--without-opt-include=${opt-dir}/include
--with-opt-lib
--without-opt-lib=${opt-dir}/lib
--with-make-prog
--without-make-prog
--srcdir=.
--curdir
--ruby=/hab/pkgs/core/ruby/2.4.2/20171014214232/bin/$(RUBY_BASE_NAME)
--help
--clean
/hab/pkgs/core/ruby/2.4.2/20171014214232/lib/ruby/2.4.0/mkmf.rb:457:in `try_do': The
compiler failed to generate an executable file. (RuntimeError)
You have to install development tools first.
from /hab/pkgs/core/ruby/2.4.2/20171014214232/lib/ruby/2.4.0/mkmf.rb:572:in `block
in try_compile'
from /hab/pkgs/core/ruby/2.4.2/20171014214232/lib/ruby/2.4.0/mkmf.rb:523:in
`with_werror'
from /hab/pkgs/core/ruby/2.4.2/20171014214232/lib/ruby/2.4.0/mkmf.rb:572:in
`try_compile'
from extconf.rb:138:in `nokogiri_try_compile'
from extconf.rb:162:in `block in add_cflags'
from /hab/pkgs/core/ruby/2.4.2/20171014214232/lib/ruby/2.4.0/mkmf.rb:630:in
`with_cflags'
from extconf.rb:161:in `add_cflags'
from extconf.rb:410:in `<main>'
To see why this extension failed to compile, please check the mkmf.log which can be
found here:
/hab/pkgs/core/bundler/1.16.0/20171129183702/extensions/x86_64-linux/2.4.0/nokogiri-1.8.2/mkmf.log
extconf failed, exit code 1
Gem files will remain installed in
/hab/pkgs/core/bundler/1.16.0/20171129183702/gems/nokogiri-1.8.2 for inspection.
Results logged to
/hab/pkgs/core/bundler/1.16.0/20171129183702/extensions/x86_64-linux/2.4.0/nokogiri-1.8.2/gem_make.out
An error occurred while installing nokogiri (1.8.2), and Bundler cannot
continue.
Make sure that `gem install nokogiri -v '1.8.2'` succeeds before bundling.
In Gemfile:
rails was resolved to 5.1.5, which depends on
actioncable was resolved to 5.1.5, which depends on
actionpack was resolved to 5.1.5, which depends on
actionview was resolved to 5.1.5, which depends on
rails-dom-testing was resolved to 2.0.3, which depends on
nokogiri
I know we do envision Habitat studios as being development environments, so running tests is definitely something we should be able to do. I’ll keep working at this.
However…gems should already be installed since we already built the application…investigating further
Yeah, that’s basically where I stopped. Once it became apparent that gcc, etc was going to be needed I decided it was time for a reality check.
Completely understandable. Let me see if there is a simpler solution.
I think I may have figured out, but it may not be the solution you are looking for:
Let’s say I have a Rails app called ‘widget_world’ and I create a Habitat plan that uses the ruby scaffolding.
Let’s say I enter a studio, then build the application.
That will put a package in my ./results directory - something along the lines of ./results/origin-widget_world-timestamp.hart
If I go ahead and start that package from the studio with
(studio) $ hab start ./results/origin-widget_world-version-timestamp.hart
And then I run the rake command like this:
(studio) $ hab pkg exec core/widget_world widget_world-rake test
It appears to work as expected.
@qhartman What happens if you ran hab pkg exec core/ruby rake?
My hunch is rake may need some runtime env vars.
Didn’t realize there was a core/bundler.
A few things:
- I was able to do many ruby things with just
core/ruby
[32][default:/src/tmprake:0]# hab pkg exec core/ruby gem install bundler
[33][default:/src/tmprake:0]# hab pkg exec core/ruby bundle install
[34][default:/src/tmprake:0]# hab pkg exec core/ruby bundle install
Don't run Bundler as root. Bundler can ask for sudo if it is needed, and installing your bundle as root will break this application for all
non-root users on this machine.
Using bundler 1.16.1
Using thor 0.20.0
Bundle complete! 1 Gemfile dependency, 2 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.
[35][default:/src/tmprake:0]# hab pkg exec core/ruby thor version
Thor 0.20.0
[36][default:/src/tmprake:0]# cat Rakefile
desc "Do the full build"
task :full => [ :ask, 'life.png', :send ] do
puts "Full Build"
end
[37][default:/src/tmprake:0]# hab pkg exec core/ruby rake --tasks
rake full # Do the full build
- When you use
hab pkg exec core/ruby bundle, the gem’s bins get installed under the $(hab pkg path core/ruby)/bin. IMO this is bad, we shouldn’t pollute anything /hab/pkgs. We might be able to change these paths via env vars:
[37][default:/src/tmprake:0]# hab pkg exec core/ruby bundle env
Environment
Bundler 1.16.1
Platforms ruby, x86_64-linux
Ruby 2.4.2p198 (2017-09-14 revision 59899) [x86_64-linux]
Full Path /hab/pkgs/core/ruby/2.4.2/20171014214232/bin/ruby
Config Dir /hab/pkgs/core/ruby/2.4.2/20171014214232/etc
RubyGems 2.6.14
Gem Home /hab/pkgs/core/ruby/2.4.2/20171014214232/lib/ruby/gems/2.4.0
Gem Path /root/.gem/ruby/2.4.0:/hab/pkgs/core/ruby/2.4.2/20171014214232/lib/ruby/gems/2.4.0
User Path /root/.gem/ruby/2.4.0
Bin Dir /hab/pkgs/core/ruby/2.4.2/20171014214232/bin
Tools
Git not installed
RVM not installed
rbenv not installed
chruby not installed
Bundler Build Metadata
Built At 2017-12-21
Git SHA 0034ef341
Released Version true
@qhartman does this unblock you? Or should we do more investigation?
@nellshamrell I’ll be revisiting this shortly and I’ll let you know!
So far, not working for me. going to keep poking around though and hopefully I’ll come up with some combination of hab pkg exec commands that will get it done.
The biggest trick may be that since this isn’t something I’m going to be running interactively, all the commands I’m running have to work as one-offs and can’t require an interactive studio session.
Ah, I see. We may not have a solution for that at this time, but I would LOVE if you would open a Github Issue around this, it definitely seems like something we want to be able to do! The only reason to move this to a Github issue is so we can make sure it gets on our project boards and prioritized.
Yeah, I’m not getting anywhere with this right now. I’ll definitely create a ticket. For now, I’m rebuilding my docker image that I use as a Jenkins slave to include all the needed ruby bits to run the unit tests outside of habitat. Less efficient, more to maintain, but should work. Hopefully I’ll have some time to revisit, but for now I need shortest path to solution.
Thanks for looking into it!
You are very welcome! Thank you so much for bringing this to our attention!