I think there’s a couple important (if not nuanced) distinctions to be made between these two tools though I do think that @metadave did an excellent job of covering the differences above. As a person who has used Chef for 6+ years and now (full transparency) works on the Habitat team I have a slightly different context to share. I also tend to ramble so hopefully the following is useful and not totally incomprehensible
Can you deploy applications with Chef? Totally, most definitely and IF you’ve actually got a reasonable CI/CD system for your applications with Chef then that’s wonderful. You’ve got to use what works for you. Now, if you’re an individual who has had to build a system like that from scratch and maintain it long term, then you’ve very likely cut yourself more than once on the challenges and deficits of using Chef in this manner. The reason being: Chef wasn’t really designed as an application lifecycle management tool. From the ground up it was pretty much targeted and planned as configuration management and infrastructure automation. What we’ve learned over the years is that Chef can do a whole lot of things but really it doesn’t excel at the application side of things. In fact that’s where nearly all of its gnarly monsters are lying in wait!
WRT added complexity - I sort of agree but I also disagree. Compared to similar patterns and workflows I’ve used with Chef in the past - e.g. the immutable + service discovery pattern - It doesn’t add anywhere near the same amount of complexity. IMPO it doesn’t even add near the complexity or brittleness of a home-rolled CI/CD system that only leverages Chef + Chef Server. I’d take that statement one step further and say it sharply decreases complexity in application lifecycle management/automation as well as CI/CD systems by outright obviating a need for as many moving parts. Not to mention minimized complexity in the form of literally shrinking the raw lines of code or rube-goldberg systems required for a Chef cookbook to deploy/manage some applications in a sane way. Especially when we’re talking about things in the context of modern application programming paradigms like the microservices approach.
All of this in mind. Chef is still a great tool, and people should definitely continue to use it. Learning new tools can be really hard and adopting tools that are pre-1.0 can definitely be a risk. The real question at hand though is - “does Habitat add value to most Chef workflows and environments out there today”? To which I think the answer is a resounding - Yes. But, almost as importantly, we (and myself in this as a practitioner) believe it also adds value to environments that use Ansible, Puppet, and Kubernetes too! Hopefully this is helpful context. When it’s all said and done I am wholly in agreement with you on the necessity for people to understand their problems thoroughly so that they can choose the right tool for the job and in my own opinion separate from being a Chef engineer, I would choose Habitat to manage my applications over Chef 100% of the time.