Unable to inject multiple key values sequentially via `hab config apply`

I’m working a package (bixu/certbot) where I need to be able to store SSL certs in a gossiped config. It seems that I can’t sequentially load renewed Certbot certs via hab config apply - when I do this, only the last-applied config seems to ‘stick’, even though my key names are different. For example:

echo "staging_domain_chain_pem = '''$(cat ./chain.pem)'''" | hab config apply nginx.staging $(date +%s)

echo "staging_domain_cert_pem = '''$(cat ./cert.pem)'''" | hab config apply nginx.staging $(date +%s)

curl --silent localhost:9631/services/nginx/staging | jq '.cfg'
  "events": {
    "worker_connections": 1024
  "http": {
    "keepalive_timeout": 60,
    "listen": {
      "port": 80
    "sendfile": "on",
    "tcp_nodelay": "on",
    "tcp_nopush": "on"
  "staging_domain_cert_pem": "-----BEGIN PRIVATE KEY-----
-----END PRIVATE KEY-----",
  "worker_processes": 4

Thinking this might be a race condition, I introduce a sleep of 3 seconds between apply operations, but I still see only one key present at a time (and the key changes depending on which part of the apply loop most recently ran.

That does look strange. Can you file an issue?

It’s my understanding that there is only a single gossiped service “layer” and that it is entirely replaced when running hab config apply. It was explained to me that whichever has the latest version (in this case, timestamp) is what is used as there is no way to completely unset a value if all applied configs are merged (TOML has no null). Is there a reason you can’t use a single command to apply both keys?

printf "staging_domain_chain_pem = '''$(cat ./chain.pem)'''\nstaging_domain_cert_pem = '''$(cat ./cert.pem)'''" | hab config apply nginx.staging $(date +%s)

@HT154, that’s a pretty interesting idea for a workaround, I’ll definitely try that.

What’s interesting about the behavior is that other config properties like http.keepalive_timeout, for example, remain untouched. So it’s not like the entire config is clobbered, but rather that any hab config apply-injected configs are clobbered.

@baumanj: https://github.com/habitat-sh/habitat/issues/6271

Right, config from user.toml, the environment, and default.toml are unaffected. Habitat has four configuration layers: default, environment, user, and gossip that are all merged together before template rendering. Though I’m not sure the precise ordering w.r.t. environment config. Currently, applying new gossip config completely replaces all old gossip config. Documentation around these details really needs to be improved

I think that changing this behavior has severe implications for the gossip mechanism and may not actually be desirable. Like I mentioned, you would never be able to un-set a key set in gossip config because TOML has no notion of null.

Also, I applying gossip config serially is going to cause two service restarts, which seems less than ideal. I think my workaround is really just the only correct approach.

Gotcha. I’m a bit concerned then about how, for example, dynamic changes to “default” configs might be gossiped around (for example, for some packages we set a default IP of to access for a database running in the Studio but override that if we are bound to another service in our ring with {{member.sys.ip}} or similar.

I do agree that your “batch apply” approach is nicer from a triggered-restart perspective (it’s working well in my experiments today), so thanks for raising that. From a design perspective, however, I think Habitat should allow multiple key injections from the CLI (principle of least surprise).

@bixu I agree it’s a bit confusing in this case, but it’s working as designed, and as @HT154 describes.

Would it be possible to use hab file upload as an alternative? It shouldn’t suffer from this limitation.

I’m not questioning what the design is but instead whether the current design choice is going to cause frustration or not. Forbidding a user to add more than one key over time via the CLI is pretty awkward. Consider the example where I have 2 keys, api_token and log_level in my default.toml. As an operator, I might roll a key by hab config applying a new API token, and then, seeing unwanted behavior in my system(s), would want to switch from log_level ="warn" to log_level = "debug" for a brief period.

https://github.com/habitat-sh/habitat/issues/5032 would probably go a long ways towards improving this experience and would also (perhaps?) not require a design change?

As a workaround, I’m concatenating my cert values to a user.toml file and letting the Supervisor pick those up.

@bixu I agree with you, and yes, I think that issue would be very helpful here.