We’re in the process of incorporating Chef into our existing CI pipeline based on Jenkins for application development. During a CI build of our chef-repo’s we’re injecting version constraints into our environment.json file’s stored in version control using a similar approach described here with Berksfile.lock. Those environments and cookbooks are uploaded into a Chef Server that is dedicated towards testing. We call this our “CI Chef Server”. This chef server will be populated with many cookbooks originating from many chef-repo’s. We want to have a suite of automated tests fire off - some based on Test Kitchen and some more application centric. We want to be able to promote a build to a production Chef Server. once the test suite passes We need to have separate Chef Servers for various reasons - PCI compliance and network segmentation being one of the biggest, but also a strong desire to ensure only tested changes touch our production systems.
I think we have a pretty solid approach in mind, but was curious if there were any existing tools that handle the physical promotion of cookbooks and environments (we’re staying clear of roles, and have successfully avoided the need for data bags). It seems like writing something custom to promote these things with calls to knife wouldn’t be too hard, but seems like it might also have some hidden complexities once I get into the thick of it. How are other people doing this?
On the application development side, our pipeline has never hand to worry about this type of thing because applications are made up of a single, atomic package file where promotion is a single step. Chef when used with a Chef Server isn’t that simple sadly.
This is exactly what policyfiles are for. The lockfile creates an "artifact manifest" which is morally equivalent to building and shipping a package. Policyfiles aren't usable with Chef Automate's workflow feature, but it looks like you're building your own pipeline so that is not an issue for you.
Another way to go is to use the "environment cookbook" pattern and have berks generate your environment files. If you do that, you need some checks in Ci to force version number updates when cookbooks are changed.
Or, you could buy Chef Automate and let that be your pipeline. It has all the stuff you're asking for built-in (via the delivery-truck cookbook).
We started down the policyfile route for this very reason, but after consulting with our Chef account rep we were advised against using policyfiles. According to them their implementation is incomplete with an unknown roadmap. We don't want to hitch our wagon to something with an unknown future. Quite sad because policyfile future is a critical one for large shops IMO.
We are using the environment cookbook pattern and our CI server is doing just that.
We're in the process of buying Automate and Workflow looks nice, but it's not that simple for us. We have almost 10 years investment in building a CI pipeline for our application development that is supporting almost 500 developers all developing code in a common CI pattern in a common system. To us Chef is just another type of code and we don't want a separate pipeline from what we already have and love. Visibility and Compliance is what we're really interested in using.
Is there a way to leverage promotion capabilities outside of the context of Workflow?
With what you’re describing, publishing the well tested cookbooks to prod chef server sounds just like a last step…
You already tested your cookbook and uploaded to a test server, just copy paste this upload step to upload to prod server after the functional tests got approved.
The implementation is complete in open source Chef products and will be maintained going forward. Integration with Automate is not currently on the product roadmap.
I recently merged some improvements to Chef Server that make basic integration of policyfiles and Automate's visibility features possible with some small product tweaks. Given that you're not interested in workflow, that would be enough to make Policyfiles supported in Automate for your use cases.
Are you using berks apply to generate your Chef Environments?
I'm not sure if you would be able to re-use that by bringing the cookbook in as a dependency, but you could at least copy the code into your own automation.