how do you practice change management with Chef? Change management meaning can vary a lot and depends on the rest of the whole setup. What I mean is roughly:
How do you know and track the exact state of your deployments/nodes?
Do you version control your chef runs?
For example, for each chef-run do you have the following info on each node:
What packages are installed and which version?
What services are running (containers)?
And which version does the deployed cookbook have? Is it possible to track the exact branch and commit?
Let me outline the setup I have in mind:
Cookbooks tracked in Gitlab (with merge requests, so some form of change management is present)
Gitlab CI is responsible for deployment:
When merged in master, the runner pushes the cookbooks to the respective Chef server and initiates the chef-runs on the nodes (via knife ssh ?)
To get the current state of the node I can run Ohai before chef-client run ?
To get the planned state of the node I can run a Why-Run before chef-client run ?
To get the updated state of the node I can run Ohai after chef-client run**?**
I log all theses things and then create diffs?
Is this change management?
I'm a noob and I don't know if I'm maybe looking at the wrong place or if other tools are better fit for this. But I
already did some research, maybe I lack the correct terminology (not a native speaker).
There are a couple of interpretations of change management. Change management is a set of disciplines to be intentional and reduce risk of operating a complex system.
Change management isn't just being able to prove what will change, (the git diff should show that). If you have properly implemented change management, you won't have surprizes. The way we implemented change management is with code reviews and build promotions. We could make all the changes we wanted to 'dev', but had to get a sign off to promote the change to 'production'.
4 Eyes rule
Put everything in version control, and lock the master branch so you can only merge to the master branch after a code review. That way no 1 person can push changes to production. Automate your pipeline (e.g. travis-ci, jenkins), so any time you make a chagne to the master branch, it automatically syncronizes with your chef server.
Pin your versions
Whenever possible, ensure chef installs a specific version of a package. Never use 'latest'. Otherwise when package maintainers push a breaking change, it will surprize you. You want to control exactly what versions are installed everywhere in your infrastructure
Build Promotion
You should have a tiered deployment process. E.g (dev -> qa -> integration -> production). No changes make it into the next environment without first being validated in the previous environment.
With regards to your plan, it is good to keep track of everything in version control, but running ohai and whyrun before each chef run isn't needed. Instead use the built in chef logging. With small infrastructure it is fine to run 'knife ssh' to trigger the chef run, however once the infrastructure grows you will want to follow the principal of 'eventual consistency' and let nodes check in at a random schedule (every 30 minutes by default is a good place to start).
Thank you spuder for your detailed answer and practical advice.
We already use version control, test with KitchenCI and do code reviews before code gets merged into master to be subsequently automatically deployed.
So this side is covered. However what I want to implement is more of a control, who deployed what on which node. Partly this is covered by the above tooling but has to be extended for the info to be readily accessible.