This is just to spark some discussion revolving deploying habitat containers into Kubernetes. From just deploying habitat on to anything I see a lot of utility in having an understanding what exists so that the node itself can make lots of intelligent configuration decisions.
Where is my database?
Who is my master if it exists?
Should I be a slave or master?
Who are my neighbors like me?
In the context of Kubernetes, out of the box I can already answer 1 easily as I can specify a Database Service that is either internal or external and I’ll have a dns alias or an environment variable that I can inject into my pods.
However, questions 2 thru 4 are difficult to answer with k8s. To my knowledge, Pods can talk to each other pod in the service if you know exactly the IPs or alias of each pod. This info was scraped manually by interacting with kubectl. I admit I don’t know how to get this info within the Pod.
Now with Habitat, if I were to setup the ring, I will get to answer question 2 thru 4. If I do this in Kubernetes, I have to pass in --peer <PEER> at pod/or container creation time. That peer info is not easily obtained dynamically. One route I took is to establish some single service called the foreman. I take that pod IP and used that to bootstrap. Its not very ideal because I have to setup another service and I worry about pod dying.
Like to know what others are doing with Habitat in Kubernetes. If anyone is exploiting the gossip rings. How are people are bootstrapping their containers?
I’m not using kubernetes yet with habitat, but I’ve tested with docker swarm mode and have a similar situation with the peer in habitat. One thing I’ve tried is passing in the dns name of the current service, which will return any peer. So if I make a service (in kubernetes I think it’s a deployment) called ‘consul’, I could just put ‘–peer consul’ in the start and it would peer with one of the consul containers.
Depending on how things are set up in swarm mode, the DNS query either returns a VIP for the service which will load balance across all containers, or it will return multiple IPs in the DNS response, one for each container.
The problem I ran into is that you have a fairly good chance of that DNS query resulting in you connecting to yourself, and you end up with a node that is disconnected from the rest of the ring. I suspect this could be fixed if habitat can be made to behave such that if it finds itself connecting to itself, it will try the next peer in the list, and will also try connecting to all peers if it gets multiple IPs in response to a DNS lookup. I didn’t get chance to test that behavior properly however as I ran into an unrelated issue, so it’s possible habitat does the right thing here already.
In the meantime, my temporary solution was to make multiple services (e.g. consul1, consul2, consul3) and specify --peer consul2,consul3 when starting the consul1 container. This will work for a service with a fixed amount of nodes, and then that service could act as the foreman for the rest of your services later on.
I’m thinking that is what I need some kind of DNS registry with a bunch of habitat-nodes that can just be added to it so that new nodes can just reference habitat-nodes and that DNS registry/nameserver would just round robin those IPs.
But the problem is I need this DNS registry exposed before the Pod is created/initialized.
The funny thing about Kubernetes is that it does have built in service discovery and a dns registry to allow this. I’m just not entirely sure how to let the ClusterIP be any of the PodIPs returned. Perhaps its not even allowed…