I’m currently in the process of deploying Chef using Policyfiles. Some background on our environment: we currently have a monolithic repo, because we don’t see there being a large sprawl of cookbooks and recipes. We’re also going to be using different Chef servers / organizations for our infrastructure.
What we’re trying to figure out is the correct usage of chef push, with regards to Policyfiles. The current workflow we’re thinking about is to make changes to the repo and submit all changes through PRs. Part of our CI/CD pipeline would then run chef update on our policies after PR was merged to master. We are thinking of then committing those lock files, directly to the master branch, so they can be used to reliably push the proper files to the Chef server.
However, we found that when running chef push, even from a clean master branch, that it was rewriting the lock file. The source code shows that it’s explicitly being told to do that:
What are we missing when it comes to the Policyfile workflow? If nothing has changed in our repository, is there a way to make files idempotent on a chef push?
Here is our ChefDK information:
Chef Development Kit Version: 0.17.17
chef-client version: 12.13.37
delivery version: master (f68e5c5804cd7d8a76c69b926fbb261e1070751b)
berks version: 4.3.5
kitchen version: 1.11.1
Firstly, the push code has to check for lockfile updates, because cookbooks from local path can be changed at any time (and this is actually ok for a very naive workflow).
You only need to be concerned if the revision_id is changing. Only meaningful fields are included in the computation of the revision_id, so even if some informational fields change, you're ok as long as the revision_id remains constant.
so, the revision_id doesn’t seem to be changing. what we’re settling on for now, for our team (@tim_heckman + myself), is as follows - maybe you can advise a better flow:
development:
branch for feature
work on feature, update locks / use test kitchen / push locks to testing chef-server / converge test nodes
commit site_cookbooks/<the changed cookbooks> and policies/<the policy>.rb (not the lock)
send PR
code review
merge PR
deployment:
operator checks out master branch
operator runs chef push grp1 policies/foo.rb (diffing)
operator commitspolicies/foo.lock.json and pushes the master branch containing the lock json
operator re-runs chef push grp1 policies/foo.rb (diffing)
operator resets hard // checks out HEAD for policies/foo.lock.json (since the scm_info here is unnecessary).
additionally, we seem to be getting “drift” in our scm_info between workstations, for example, my git remote URI is different to @tim_heckman, but refers to the same repository, see diff below:
Your process looks fine to me. You might want to commit the lock during development to ensure you're using the same set of cookbooks in deployment that you use in test, assuming there's not some other part of your process that does that.
There's no way around the lockfile re-writing currently. I'd be open to a patch for a sort of "strict mode" where the chef command would verify no important content changed and either:
Fail if something got updated
Don't update the lock if nothing changed
You already found the relevant part of the code, so it's just a matter of writing the lockfile_needs_update? method and threading the CLI switch through.
I'm guessing your .git/config has the same difference in the remotes, or maybe it's git version difference? chef is just taking the output of git config verbatim:
If there's a sane way to normalize this, that'd be a welcome patch also.