Hello,
I’ve been doing some work with the core/consul plan as it does some awesome things, but I need it to do some other things. Also, as this is my first real foray into Habitat, it provides the perfect vehicle for learning about Habitat!
Which brings me to today’s question.
How do I enable Habitat to autodiscover peers that I want to “consume”? And, can a package be both a “producer” and “Consumer” within the same plan?
Crash course lesson in Consul:
Consul can run in either “server” or “client” mode. The difference being that “servers” participate in gossip, cluster leader election, etc. where “clients” do not. The difference is servers are started with the -server command line parameter. Either way, when run in a cluster, consul needs the -retry-join parameter to tell it at least one other node to connect to. (see #1514 for some context)
What I did there was:
{{#eachAlive svc.members as |member| ~}}
-retry-join {{member.sys.ip}} \
{{/eachAlive ~}}
If my understanding is correct, this looks for other nodes that have been marked as a --peer to habitat, e.g. when I start my hab sup, I do a hab sup --peer 10.10.10.10 and when I start my service as hab svc start core/consul --group production this starts consul in the consul.production group. Thus when my plan is compiled, my members are resolved to IPs based on who’s running the core/consul package in the production group.
This seems to work as I need it (with the exception there seems to be no way to exclude svc.member.self but that’s part of a larger conversation).
Now I want to start consul as a client. So my first thought was, I’ve got my cluster started with the above hook… so I should be able to just add --bind consul:consul.production, right? But it doesn’t seem to work that way…
How about an example:
Consul Cluster:
- Server node 1 - 10.10.10.11
- Server node 2 - 10.10.10.12
- Server node 3 - 10.10.10.13
- Client node 1 -10.10.10.254
Nodes are started with:
# node 1:
hab start core/consul --topology leader --group production --strategy rolling --peer 10.10.10.12 --peer 10.10.10.13
# node 2:
hab start core/consul --topology leader --group production --strategy rolling --peer 10.10.10.11 --peer 10.10.10.13
# node 3:
hab start core/consul --topology leader --group production --strategy rolling --peer 10.10.10.11 --peer 10.10.10.12
This spins up, and generates a run hook that looks like:
#!/bin/bash -xe
exec 2>&1
SERVERMODE=true
if [ "$SERVERMODE" = true ] ; then
exec consul agent -ui \
-retry-join 10.10.10.11 \
-retry-join 10.10.10.12 \
-retry-join 10.10.10.13 \
-server -bootstrap-expect 3 \
-config-file=/hab/svc/consul/config/basic_config.json
else
exec consul agent -dev
fi
At this point, somewhere there’s a service defined as consul and it has three nodes, right? (It would be really nice to be able to access this information via hab sup commands)
So I was thinking, if we updated the run hook, I could have the hook iterate through the bindings, a la:
{{~#if cfg.server.mode ~}} -server -bootstrap-expect {{cfg.bootstrap.expect}}
{{#eachAlive svc.members as |member| ~}}
-retry-join {{member.sys.ip}}
{{/eachAlive ~}}
{{/if ~}}
{{~ #eachAlive bind.consul.members as |member|}}
-retry-join {{member.sys.ip}}
{{~/eachAlive ~}}
And start my client as:
hab start core/consul --bind consul:consul.production --peer 10.10.10.11
This would mean if the following were set:
[server]
mode = false
My stanza that includes -server -bootstrap-expect parameters, those attributes would be excluded from my run hook.
But my bind.consul.memebers don’t seem to resolve during the compile phase…
(The big caveat here is that the client node IP address should never appear as a -retry-join parameter. So there’s never a point where -retry-join 10.10.10.254 would be acceptable as part of the startup command for either servers or clients.)
Does the plan need to define pkg_exports?
Could I do a:
pkg_binds_optional=(
[consul]="sys.ip"
)
Would that allow me to iterate through #eachAlive bind.consul.members and start my consul client as hab start core/consul --bind consul:consul.production --peer 10.10.10.11 or what would that syntax look like?
Thanks!
Edit:
Maybe the answer is that we need a consul and a consul-client plan. But I’m still having trouble groking how I’d have the “client” autodiscover it’s peers/how binding to a service works exactly.
Edit2:
based on this thread I think I’m on the right track here… but that only seems to allow binding to ports?
