Habitat uses the free travis-ci.org service (rather than the paid travis-ci.com) one, we’re not able to easily log into a CI instance and debug travis-specific issues. Since spinning up new runs takes awhile and consumes limited resources, debugging travis locally is useful. Travis has documentation for running a container based docker image locally, but it’s rather sparse, so this document describes the rest of the details necessary for debugging habitat CI tests.
Prerequisites
- Docker installed
Travis docker image setup
➤ docker run --privileged --name travis-debug -it -u travis travisci/ci-amethyst:packer-1512508255-986baf0 /bin/bash -l
The --privileged
option is necessary for the studio to successfully bind mount directories such as /dev
. The name travis-debug
is arbitrary and the 6dd8f2496746
identifier is automatically generated. The choice of docker image was a best-guess for what matched the instance
log output for existing travis builds for habitat. For example:
instance: travis-job-b2e18128-2f18-4c14-b371-f199e1b366aa travis-ci-amethyst-trusty-1512508224-986baf0 (via amqp)
The travis docker images can be found on Docker Hub.
Install travis-build
tool
This is broken into chunks to indicate where the commands are expected to generate output.
travis@6dd8f2496746:~$ mkdir .travis
travis@6dd8f2496746:~$ cd .travis/
travis@6dd8f2496746:~/.travis$ git clone https://github.com/travis-ci/travis-build.git
travis@6dd8f2496746:~/.travis$ cd travis-build/
travis@6dd8f2496746:~/.travis/travis-build$ gem install travis
This next set of steps is necessary to work around a travis/support (LoadError)
known issue.
travis@6dd8f2496746:~/.travis/travis-build$ bundle install
travis@6dd8f2496746:~/.travis/travis-build$ bundler add travis
travis@6dd8f2496746:~/.travis/travis-build$ bundler binstubs travis
travis@6dd8f2496746:~/.travis/travis-build$ cd
Set up habitat sources
travis@6dd8f2496746:~$ git clone https://github.com/habitat-sh/habitat.git habitat-sh/habitat
travis@6dd8f2496746:~$ cd habitat-sh/habitat/
travis@6dd8f2496746:~/habitat-sh/habitat$ git checkout branch/commit/to/test
- Modify .travis.yml since
matrix
andenv/global
sections will be ignored - Remove
branches
since we’re running the tests manually - Move the contents of
include
to the top level since we’re explicitly choosing the configuration to run - Remove the
openssl
command which generates thebuilder-github-app.pem
since our environment lacks$encrypted_f0fe831d6b31_key
and$encrypted_f0fe831d6b31_iv
(we’ll install this file manually)
For example:
branches:
only:
- master
- /^sentinel.+$/
- /^acceptance_deploy.+$/
- /^test_development-.*$/
- /^\d+\.\d+\.\d+$/
os: linux
env:
global:
- PATH=$HOME/.cargo/bin:$PATH
# Habitat Rust program components
- _RUST_HAB_BIN_COMPONENTS="components/airlock|components/hab|components/hab-butterfly|components/launcher|components/pkg-export-docker|components/pkg-export-kubernetes|components/pkg-export-helm|components/sup"
matrix:
include:
- language: rust
env:
- COMPONENTS=srv
- AFFECTED_DIRS="Cargo\.lock|$_RUST_BLDR_BIN_COMPONENTS|$_RUST_BLDR_LIB_COMPONENTS"
rust: stable
sudo: required
addons:
apt:
sources:
- kalakris-cmake
packages:
- build-essential
cache:
apt: true
cargo: true
directories:
- "$HOME/pkgs"
before_install:
- source ./support/ci/rust_env.sh
- openssl aes-256-cbc -K $encrypted_f0fe831d6b31_key -iv $encrypted_f0fe831d6b31_iv -in ./support/ci/builder-github-app.pem.enc -out /tmp/builder-github-app.pem -d
script:
- ./test/builder-api/test.sh
becomes
os: linux
language: rust
env:
- COMPONENTS=srv
- AFFECTED_DIRS="Cargo\.lock|$_RUST_BLDR_BIN_COMPONENTS|$_RUST_BLDR_LIB_COMPONENTS"
rust: stable
sudo: required
addons:
apt:
sources:
- kalakris-cmake
packages:
- build-essential
cache:
apt: true
cargo: true
directories:
- "$HOME/pkgs"
before_install:
- source ./support/ci/rust_env.sh
script:
- ./test/builder-api/test.sh
Use travis-build
to turn .travis.yml
into a bash script
travis@6dd8f2496746:~/habitat-sh/habitat$ ~/.travis/travis-build/bin/travis compile > ci.sh
Fix up ci.sh
on account of our .travis.yml
modifications:
- Search for
branch
to find theclone
command and addbranch/commit/to/test
like so:
travis_cmd git\ clone\ --depth\=50\ --branch\=\'branch/commit/to/test\'\ https://github.com/habitat-sh/habitat.git\ habitat-sh/habitat --echo --retry --timing
- Search for
Setting environment variables from .travis.yml
and addtravis_cmd export
commands for anyenv/global
entries removed from the original.travis.yml
. For example:
env:
global:
# Habitat Rust program components
- _RUST_HAB_BIN_COMPONENTS="components/airlock|components/hab|components/hab-butterfly|components/launcher|components/pkg-export-docker|components/pkg-export-kubernetes|components/pkg-export-helm|components/sup"
becomes:
travis_cmd export\ _RUST_HAB_BIN_COMPONENTS\=\"components/airlock\|components/hab\|components/hab-butterfly\|components/launcher\|components/pkg-export-docker\|components/pkg-export-kubernetes\|components/sup\" --echo
Note that characters need to be escaped in the context of travis_cmd
.
- Create
/tmp/builder-github-app.pem
and/or/tmp/habitat-srv-admin
from the credentials in 1Password to account for theopenssl
call that was removed - Optionally replace
sudo apt-get update -qq 2>&1 >/dev/null
withsudo apt-get update
so progress of this slow command is observable
This would be a good time to take a snapshot of your container so that you can run the image from there while
iterating on the branch you’re trying to get passing tests:
docker commit --message="Travis before running ci.sh" 6dd8f2496746 travis-debug-snapshot:tag-name
Where 6dd8f2496746
should be the container ID that shows up in the command prompt (or in docker ps
). travis-debug-snapshot
and tag-name
are arbitrary.
Run ci.sh
For the most readable output, just run bash ci.sh
, but for debugging purposes bash -x ci.sh
is often helpful. If there are failures, you can fix things up, push any necessary changes to your remote branch and then run ci.sh
again. This diverges a bit from the true travis workflow since the workspace is no longer pristine, but it’s much faster to iterate on this way. Once you have things running the way you want, you can start over from the base travis image. Or, if you made a snapshot as suggested above, you can restart it:
➤ docker run --privileged --name=travis-debug-restore -it -u travis travis-debug-snapshot:tag-name /bin/bash -l
And then just run ci.sh
again.