[brainstorming] Building an "HA Loadbalancer" with Nginx and Habitat

Habitat Help
1

Hello,

I’m looking to proof out if there’s a way to replicate existing functionality we currently have in Habitat.

Once common use case I see is using Nginx as a load balancer/reverse proxy in HA mode with Keepalived. It is my hypothesis that using Habitat I can replicate this functionality and make overall configuration easier. But I’m having a bit of trouble getting there…

I started writing a keepalived plan some months ago, but hadn’t really continued as I felt it was replicating some Habitat functionality… (although thinking back, I think there was some issue with a kernel module…? maybe?)

The way we run it, is with two Nginx servers. My understanding is that Habitat requires at least 3 nodes running the same service to be able to form a consensus. I this could easily be overcome by running 3 Nginx nodes instead of the two that we typically run.

The next thing is the floating IP. From an Nginx configuration standpoint, I think just adding a “floating ip” configuration attribute, then writing the nginx config such that only the “leader” configures the floating IP to listen on would probably do the trick…

But where I’m stumped is “how do we tell the network where the FIP lives”? I’m a bit out of my wheelhouse here… It seems that Keepalived uses some implementation of VRRP which is maybe the secret sauce in how the “active” FIP is announced to the world.

Writing this out now (this is the third iteration of this post…) it occurs to me, that I don’t think Habitat has a way of announcing that floating IP, so really we would require the magic of Keepalived for that?

I dunno… maybe the answer is just to use Keepalived. DigitalOcean seems to use Corosync in place of Keepalived…?

Anyway, this is about where I am… I welcome any feedback, comments, questions, thoughts, ideas, insults, jokes, memes, or favorite ice cream flavors…

Thanks!
-Q

2

Oh this is an interesting idea. I haven’t messed around with nginx HA mode so I would probably need to read up and think about this. I think it kind of also depends on what your goal is. Like, are you going to be running HA between 2x different datacenters with the intention of flipping between them in the case that one DC goes down? Or is it single DC multi-node proxies for in case one of the proxies goes down?

3

Well… now THAT is an interesting idea… my initial thought was just multi-node proxies in one DC.

GSLB becomes a whole different problem because now we’re not just talking layer 3 and moving an IP between MAC addresses. (at least based on my understanding, IANANE… (Network Engineer))

4

Have you found any conclusion?

I see two topics, “the load-balanced” service, exposed on hab-sup cluster.

  • Seems to me that at least from an LB perspective, haproxy could do a job and together with " suitability" hook, it could reconfigure whenever underlaying hab-sup cluster change. However today core/haproxy does not count with Leader-Follower topology.

Second addressed issue is how to expose such load-balanced service behind FQDN, here I do not see any means habitat can help. Yes you can use coredns and resolve fqdn against hab-sup cluster, where haproxy runs. But that still needs propagation of the DNS records (which is slow). From this point, keepalived is missing in core-plans.

5

Hey @epicim,

I haven’t really, but it seemed like there wasn’t much interest at the time so I had kinda backburnered this topic in favor of some of my other higher priority projects.

But it’s definitely something I’m still interested in and would like to pursue.

I think the first step is to figure out how to build a keepalived habitat package. There was some issue I was running into… which I cannot seem to remember what it was now… maybe it was just IPv6?…

I think I agree with you that Habitat (in its current form) doesn’t really hook in well with DNS, but I think there might be some potential there… Otherwise, we’ve had some success in using Habitat and Consul together… (basically Consul becomes the DNS server at that point, have a look at my consul-client package).