We’re looking at reviving the old Habitat Kubernetes Operator and are looking for feedback on how to proceed.
For those of you using Habitat with Kubernetes, what are your biggest problems that an Operator could help with? If you used the old Habitat Operator, which of its capabilities would you like to see in a new one? Which design choices made in the old Habitat Operator would you prefer we left out of a new one?
Thanks!
I would love if I could just feed a codebase + plan path into a CRD object and leave it to the operator to deal with building and loading into a supervisor. Maybe with a “pod” notion like Deployments have so a set of services could be defined with affinity to run together and be a mix of built and downloaded packages, I could replace my habitat-compose pattern with something like that potentially
I’ve come to the conclusion that Habitat has almost no place in Kubernetes. I’ve been using them together for nearly two years and over that time period have used less and less of Habitat.
Here’s what’s left:
The binding mechanism introduced an unacceptable amount of instability into my systems (entire clustered services having their config re-templated and getting reloaded/restarted if a single pod drops out and again when the new one spawns) and had to be removed before anything was prod-ready. The config and file gossip system were never useful and always ran orthogonal to the (much better) ConfigMap/Secret mechanism.
I’m still struggling with the inability to have some config template changes restart the service and some not.
At this point, I would recommend against adoption of Habitat for any greenfield project targeting Kubernetes deployment. It is still an extremely valuable tool for migration from traditional VM deployments into containers and Kubernetes, but I’d encourage elimination of the useful bridging features as soon as possible post-migration for the reasons outlined above.
That being said, please see the issues I filed against the operator while I still used it. All of these are still valid: https://github.com/habitat-sh/habitat-operator/issues/created_by/HT154
I agree with @ht154 on vanilla Habitat not being a great fit for Kubernetes. What might be cool is if Habitat could export your habitat plan into a CRD, and subscribe for application updates.
This is not fully thought out, but in Kubernetes, the k8s control plane should handle application lifecycle management. It does this via a custom CRD per service. If someone could write a plan in Habitat, export it to a CRD, and then the Habitat Operator is only responsible for seeing when the Habitat packages & CRDs have updates and pulling those down and coordinating an update (blue/green, rolling, etc.) using Kubernetes, that could be cool. This sticks with the Habitat “build it once, run it anywhere” without getting in Kubernetes’ way.
Hi @tashadrew! 
Yup, that's exactly the approach that I'm looking at right now. Ultimately, I'd like to find a good way to decompose the functionality of the Supervisor in such a way that lets you pick and choose as appropriate for your platform. If you're running on bare metal, let the Supervisor handle everything for you; if you're in Kubernetes, on the other hand, let those capabilities be handled in a native Kubernetes way. Upgrades seemed to be a nice starting point.
Very cool! I can definitely see a lot of value in the Habitat build system, cross platforms.
@tashadrew, you say "the k8s control plane should handle application lifecycle management" and also "Habitat could export your habitat plan into a CRD, and subscribe for application updates", which I interpret as slightly conflicting statements, or do you mean by "application lifecycle" restarts and service binding? If I'm not understanding right, please correct me 
I do agree that Habitat feels like it has a better update mechanism via push to/pull from Channels than K8s does out-of-the-box...
@bixu, I know @tashadrew has her own thoughts on this, but the idea I've been rolling around in my head would basically be using the Operator / CRD as the means to join the Habitat and Kubernetes views of the world. One way might be for the Operator to be listening for channel promotion events, and then it creates a new deployment for that service, or perhaps creates an appropriate commit to a git repository for fans of GitOps workflows, etc. The Supervisor within the container wouldn't be in this update flow at all.
Figuring out the best way to decompose the current Supervisor for this world will take some though, as will the extent to which various Supervisor features can / should be offloaded to Kubernetes. I'm not 100% sure what binds look like in this new world, for instance.
Hope that helps some 