Base Plans Refreshes

Habitat Wiki
1

Base Plans Refresh

Base Plans

Base Plans are a subset of the core plans that are required to build both Habitat and the Studio. Without Habitat and the Studio, you cannot build any Habitat plans. These plans are the “foundation of the Habitat universe” - they are the building blocks upon which every component of Habitat is built upon. They must be built in a very specific order.

For more information and about base plans, see this blog post.

We have divided this process into four stages.

Stage 0

In order build the base plans from scratch, we need a few tools (including busybox, wget, and rq, along with others). At this point, we do NOT use the Habitat plans for these tools, we use binaries custom built for this purpose from the original source files (i.e. the source file for wget comes from https://www.gnu.org/software/wget/) and place them in a tarball.

Stage 1

This stage involves using the tools tarball we created in Stage 0 to build all base plans in a specific order. We create Habitat packages for each base plan using the tools within the Stage 0 tarball.

Stage 2

In this stage we rebuild all the base plans again - but this time we build them using the Habitat artifacts created in stage 1, rather than the tools in the stage 0 tarball. Why do we do this? This makes the Habitat universe we are creating self sustaining - it decouples us from using the tools within the Stage 0 tarball (which, again, are not Habitat packages) and truly builds Habitat using Habitat.

Stage 3

This is the final stage - when we use the base plans artifacts we created in Stage 2 to build all of the other core plans (there are currently 649 of them). This means that all core plans will be built with the new base plans.

Doing the refresh

Check for new releases of Base Plans

All Base Plans are listed here.

I currently check for updates by going to the source url of each plan and seeing if there is a new stable version. It’s currently a tedious, manual process that would be ideal to automate in the future.

I always make a checklist of which plans need to be update, as captured in this issue comment.

Updating and Building the Base Plans

You are going to be doing a LOT of building and need a highly optimized workstation to do this.

Spin up a new instance in our AWS account - I recommend d2.4xlarge (NOTE - make sure to stop this instance (you don’t need to terminate it) when you are not using it to avoid running up our AWS bill).

When it is up, ssh into it and:

  • Create new tmux session
$ tmux new -s base_plans
  • Install Habitat
$ curl https://raw.githubusercontent.com/habitat-sh/habitat/master/components/hab/install.sh | sudo bash
  • Import current secret core key with
$ hab origin key import

(NOTE - after you paste in the key, hit CTRL+D twice to exit out of the input prompt)

  • Clone the core plans repo and create a new branch
$ git clone https://github.com/habitat-sh/habitat.git
$ git clone https://github.com/habitat-sh/core-plans.git
$ (cd core-plans && git checkout -b nshamrell/base-plans-update-aug-2018-2)

Start Updating

These updating instructions are based on this (as of now) still unmerged PR. https://github.com/habitat-sh/habitat/pull/4829

I like to update the plans in order of the base-plans.txt - start with updating linux-headers. I often will update three-four at a time, then run the full build script to ensure that everything builds. If you are building glibc or gcc these are very time consuming.

Stage One

  • Enter Stage 1 studio
$ ./core-plans/bin/bootstrap/stage1-studio.sh enter
  • Build all the base plans in order
(studio) $ ./core-plans/bin/bootstrap/stage1-build-base-plans.sh
  • If you need to build a single plan in this stage 1 studio, you will need to do this:
(studio) $ /src/habitat/components/plan-build/bin/hab-plan-build.sh core-plans/busybox
  • Exit the studio
(studio) $ exit
  • Save the Stage 1 artifacts
$ mv ./results ./results-stage1
  • Install the just built studio
$ sudo hab install ./results-stage1/core-hab-studio-*.hart

Stage Two

  • Get FQDN of hab-backline package
$ ls -1 ./results-stage1/core-hab-backline-*.hart
  • Set env variable for hab-backline studio
$ env HAB_STUDIO_BACKLINE_PKG=core/hab-backline/version/timestamp ./core-plans/bin/bootstrap/stage2-studio.sh new
  • Enter the stage two studio
$ ./core-plans/bin/bootstrap/stage2-studio.sh enter
  • Copy all stage1 packages into the artifact cache
(studio) $ cp ./results-stage1/*.hart /hab/cache/artifacts/
  • Install all stage1 packages
(studio) $ ls -1 /hab/cache/artifacts/*.hart | while read hart; do hab pkg install $hart; done
  • Build all base plans in order
(studio) $ ./core-plans/bin/bootstrap/stage2-build-base-plans.sh
  • Exit the studio
(studio) $ exit
  • Save the Stage 2 artifacts
$ mv ./results ./results-stage2

Stage Three

  • Install Ruby
$ sudo hab pkg install core/ruby
  • Calculate build order of remaining plans and exclude base plans
$ find core-plans habitat/components -name plan.sh \
  | $(hab pkg path core/ruby)/bin/ruby ./core-plans/bin/build-order.rb --without-base \
  | cut -d ' ' -f 2 \
  | grep -v '^$' \
  > stage3_world_order
  • Make a copy of the “master” list and use the working copy for building against
$ cp stage3_world_order stage3_world_order.working
  • Get FQDN of hab-backline package
$ ls -1 ./results-stage2/core-hab-backline-*.hart
  • Set up environmental variables for the stage 3 studio
$ env HAB_STUDIO_BACKLINE_PKG=core/hab-backline/version/timestamp ./core-plans/bin/bootstrap/stage3-studio.sh new
  • Enter the stage 3 studio
$ ./core-plans/bin/bootstrap/stage3-studio.sh enter
  • Copy all stage2 packages into the artifact cache
(studio) $ cp ./results-stage2/*.hart /hab/cache/artifacts/
  • Install all stage2 packages directly from the artifact cache
(studio) $ ls -1 /hab/cache/artifacts/*.hart | while read hart; do hab pkg install $hart; done
  • Build all plans in order OR
(studio) $ ./core-plans/bin/bootstrap/stage3-build-remaining-plans.sh stage3_world_order.working
  • Build individually
(studio) $ export NO_INSTALL_DEPS=true
(studio) $ build ./core-plans/your-plan
  • NOTE: If a plan fails to build, check if it is failing in Builder as well. If it is, I usually skip it. You can skip a plan by removing it from the stage3_world_order.working file
  • NOTE: This script stops unexpectedly sometimes, even if there is not a failure. All you need to do is start it again. You will know the script is really done when the last package in stage3_world_order.working builds successfully.
  • NOTE: If a downstream plan (i.e. elasticsearch5) is not building because it cannot find a locally built/installed package of a failed build (jre8), I usually look at Builder, find the last successful stable build of the plan that is not building, and then install it with
  • NOTE: You may need to pin the downstream plan to an older version of the plan which does not build, and will then likely need to pin other dependencies as well for the downstream plan.
(studio) $ hab install core/plan/version/timestamp
  • Exit the studio
(studio) $ exit
  • Save the stage3 artifacts
$ mv ./results ./results-stage3

Rinse and repeat until all core plans build (it is usually ok to ignore the ones that do not already build on Builder).

Cleanup

$ ./core-plans/bin/bootstrap/stage1-studio.sh rm
$ ./core-plans/bin/bootstrap/stage2-studio.sh rm
$ ./core-plans/bin/bootstrap/stage3-studio.sh rm

Submitting changes in a PR

  • When you have all plans building, submit a PR to the core plans repo with all of your changes

Communicating the Upcoming Refresh

Uploading Updated Packages to Acceptance

Next, you should test the refreshed plans on Acceptance. To do this:

  • Take a snapshot of the live DB and EBS volume and update acceptance with them
  • Turn off autobuild on upload on Acceptance Builder (see Turning off Builder builds)
  • Create base-plans-refresh channel on Acceptance Builder
  • Upload all packages to the base-plans-refresh channel
  • Promote all packages to channel ‘stable’
  • Turn on autobuild on upload on the Acceptance Builder
  • Test some packages that depend on core plans to make sure they build properly

Uploading Updated Packages to Live

Now, it’s time for the big upload to live!

  • Declare a service window for Habitat through status page, Habitat Slack, and the internal Habitat channels
  • Turn off autobuild on upload on Live Builder (see Turning off Builder builds)
  • Create base-plans-refresh channel on Live Builder
  • Upload all packages to the base-plans-refresh channel
  • Promote all packages to channel ‘stable’
  • Turn on autobuild on upload on the Live Builder
  • Test some packages that depend on core plans to make sure they build properly

Afterward

  • Temporarily delete everything in .bldr.toml (to avoid kicking off exponential builds when the pull request is merged) and commit that change
  • Merge the pull request
  • Add content back into .bldr.toml and commit that change
  • If for some reason exponential builds get kicked off, follow these instructions to cancel all builds Canceling all dispatching Builder builds