To composite or not to composite (Vault + Consul)

Habitat Help
1

For anyone not following along, the story so far… I’ve been doing some work over the past couple weeks to learn Habitat and to improve the core/consul plan to allow for server autodiscovery. With a ton of help from everyone especially @Defilan and @adam we’ve successed!!!111

Once that was complete I needed to figure out how to run consul in client (aka “agent”) mode, such that consul could communicate with the master cluster but isn’t participating in the raft consensus, as Hashicorp recommends against communicating directly with the Consul API. (as to why, frankly, I’m not certain, but let’s just push the “I believe button” for now). After a bit of struggle, I think I have it working too! (would love any feedback, maybe I can convince the powers that be to accept it into core! Wouldn’t that be a trip)

So. That brings us to today.

Basically I want to run Vault and Consul and say that "core/vault depends on qubitrenegade/consul-client", I think… The way core/vault is written it uses the consul backend by default and expects there to be a consul endpoint listening on the loopback.

I can conceivably see two ways to run this.

  1. Update the core/consul plan to bind to the consul-client service. I think, however, that this would cause the core/vault package to find ANY consul-client, not necessarily just the consul-client listening on localhost.

  2. Use a composite package that actually installs/starts both qubitrenegade/consul-client and core/vault. But that means that core/vault can’t be run “out of the box” (except for running in “dev” mode), which I don’t know if that’s a good or a bad thing…

Can I modify config files in a composite package of dependent packages? If I create a qubitrenegade/vault-ha package can I create a /hab/user/vault/config/user.toml and override settings in the default.toml? Essentially saying “when you run core/vault alone run it this way, when you run it with other services/hab packages run it that way”. Is it ok to have a package that doesn’t really work by itself and only works as part of a larger/composite package? I feel like the answer is maybe yes…

2

So, we haven’t really done much at all with the “only listen to a service running locally” pattern that you’re describing here, but this is the second time in a week it’s come up (the other one was with running Envoy). I think you might be able to do this by having a bind from vault to consul-client, where the config filters things so you only connect to the conusl-client that’s running on the same IP as your vault service. It’s possible that we might want to support this pattern more formally; I’m not quite sure yet. This bind could be an optional one, so vault could continue to operate normally if you don’t choose to bind it to a consul-client.

As for composite packages, you could theoretically install a composite, and then drop off a user.toml file to tweak the individual service configs as you like. Composites don’t (yet?) have the ability to pull such overrides into themselves as a first-class concept, though that’s an interesting idea to consider (i.e., stand-alone services can have a default.toml; why can’t composites?).

As I say to everyone expressing interest in composites, though, please be aware of their current limitations and “beta” status: https://www.habitat.sh/docs/developing-packages/#limitations

3

Well… I updated the run hook to:

  hab svc start qubitrenegade/consul-client --bind consul-server:consul.default --group $(hostname)
  exec vault server -config={{pkg.svc_config_path}}/settings.hcl

which seems to work… but I can’t figure out how to ascertain the group for the bind… (e.g. in the case of running in a docker container) I also had to add qubitrenegade/consul-client as a package runtime dep, which seems like an ok requirement…?

But when I stop core/vault it doesn’t stop qubitrenegade/consul-client (well if you stop the whole container it stops, but not always gracefully)

Then I cloned vault into its own project and built it as qubitrenegade/vault with a pkg_binds_optional. And created a composite plan found here But I don’t seem to be able to export it as a docker container even though it seems to build ok, but exporting fails with the error:

$ hab pkg export docker qubitrenegade/vault-composite/0.1.0/20180527072816
...
Ω Creating cacerts symlink into /etc
☒ Deleting artifact key symlink
☒ Deleting artifact cache symlink
✗✗✗
✗✗✗ A primary service package could not be determined from: ["qubitrenegade/vault-composite/0.1.0/20180527072816"]. At least one package with a run hook must be provided.
✗✗✗
4

@qubitrenegade Sorry for the late reply… I blame bad email notifications and PTO :frowning:

As you’ve discovered, composites currently don’t work with the Docker exporter… which I now see is conspicuously missing from that list of limitations in the blog post. Terribly sorry for that.