Policyfiles and cookbooks

Chef Infra (archive)
1

Hi,
I’ve got a few questions around how to manage workflow. We are new to chef so don’t have past experience with other work flows, just what we could read about on the interwebs. We started down the single cookbook per git repo plan. We are busy writing cookbooks to get our app/site working by end of the month, but we don’t have a good idea/plan how to manage them all. I started experimenting with policyfiles and provisioning instead of berkshelf. We don’t use environments/roles yet, might not at all.
We want to have a local test env, a qa env, and production env. We are using hosted/managed chef.

What we have now:

  • A chef repo checked into git. This repo started out just having knife.rb file and was meant to be used by anyone to create and configure our website either locally on a laptop or in AWS.
  • This repo has a policies dir that contains a policy file per node:
    policies/webserver.rb policies/database.rb
  • There is a cookbook in this repo called cluster-provision that contains various recipes to create nodes using vagrant or AWS. I haven’t figured out how to make once recipe do either one yet… still learning.

I can run commands like:
chef provision qa --sync policies/webserver.rb -n web-qa --cookbook cookbooks/cluster-provision --recipe setup_vagrant_webserver
or:
chef provision qa --sync policies/database.rb -n web-qa --cookbook cookbooks/cluster-provision --recipe setup_vagrant_webserver
or:
chef provision production --sync policies/database.rb -n web-qa --cookbook cookbooks/cluster-provision --recipe setup_aws_webserver

Great. Question here… I can’t see the policyfile on the Chef Manage website. I can’t see cookbooks either. I can see policies using chef show-policy. Using knife I can’t see anything either. Is this just some missing work/feature preventing it from working? Or are you supposed to take the id shown by show-policy and use it to examine policy lock file to figure out what’s going on?

In the policyfiles, I reference about four of our own cookbooks. Right now I’m using either git reference or a local file path reference. Great. Can I point the policyfile cookbook entries at chef server to get cookbooks? We don’t have a local/private supermarket.

In each of our own cookbooks do we need to have a policyfile to resolve deps and upload them to chef server? Or is that still done using Berksfile/berkshelf?

Some of the sources I’ve used/found so far are:
Chef mailing list: policyfiles-and-chef-provision
Policy Files Guided Tour
I have seen some other topics on chef mailing list about policyfiles but still not clear to me how to setup a workflow. Maybe part of it is not having experience with chef and other workflows is part of it. Thanks in advance.
-Maciej

2

Great. Question here.. I can't see the policyfile on the Chef Manage website.

The manage console doesn't support policyfiles at this time

Using knife I can't see anything either.

There's two parts to this:

  1. Cookbooks that you upload with policyfile commands via chef are stored in a separate API, so you won't see them with knife cookbook list. You can manually poke the APIs via knife raw though.
  2. There aren't any knife commands corresponding to the policyfile APIs at this time. Long-term the goal is that you manage your day-to-day workflow with chef commands that always do the right thing, but you can get lower-level access with knife.

Or are you supposed to take the id shown by show-policy and use it to examine policy lock file to figure out what's going on?

pretty much this. You can pass extra arguments to chef show-policy to see the exact lock that is applied for a given policy name and group with chef show-policy [POLICY_NAME [POLICY_GROUP]]

Can I point the policyfile cookbook entries at chef server to get cookbooks?

No. In order to resolve which versions of cookbooks are compatible with each other, chef needs a way to get the list of all versions and what cookbooks each version depends on. Chef Server doesn't have this functionality right now, though there is an accepted RFC to implement it: https://github.com/chef/chef-rfc/blob/master/rfc014-universe-endpoint.md

We don't have a local/private supermarket.

People have told me good things about mini-mart, but I have not used it myself: MiniMart

Policyfiles also support using a chef repo as a cookbook source.

In each of our own cookbooks do we need to have a policyfile to resolve deps and upload them to chef server? Or is that still done using Berksfile/berkshelf?

Policyfiles are meant to represent a type of machine that you have in your infrastructure, like a load balancer, application server, database server, log host, etc. That said, if you're using a TDI workflow (e.g., test kitchen), it makes sense to have a policyfile for each cookbook, which represents the throwaway hosts you spin up for testing purposes. But the policies that you push to a chef server are the ones the represent kinds of machines you have in your "real" infrastructure.

When you run chef push POLICY_GROUP, that command uploads the locked policy and all the cookbooks (you can see the cookbooks with knife raw cookbook_artifacts). Any nodes which have the same policy_name as the policy that you uploaded, and the same policy_group that you gave as the argument to that command will start using the policy on their next chef-client run. Thus the simplest workflow, assuming you have a staging environment and production environment, is to do chef push staging, run chef-client on the relevant hosts in staging, then chef push production. As you get more sophisticated, you can add a Ci/CD pipeline into the mix to automatically test changes and promote them.

Or is that still done using Berksfile/berkshelf?

To be extra clear on this point, no. chef push handles uploading your cookbooks using the new Cookbook Artifact API in addition to uploading the policy to the server.

3

Thanks for replying Dan. I re-read the POLICYFILE_README.md and saw it mention chef_server and RFC you referred to, guess I just missed it.

We are trying to follow a TDD process, so one cookbook per git repo and unit/integration tests for each cookbook. Don’t yet have a CI env setup so we just run everything manually for now. We use test-kitchen with chef_zero/Berkshelf provisioner and vagrant(or docker). So far it’s been working well. I have not yet tried replacing chef_zero/Berkshelf with policyfile_zero/Policyfiles.

My thought was to just use git as the source for each of our cookbooks and tags to get specific versions. I’ll take a look at mini-mart, thanks for the pointer. We are following single cookbook/git repo model. Actually, I think we will put a ‘provision’ cookbook in this chef repo and put all the recipes related to creating nodes/machines in it. So you lost me when you suggested we could use chef_repo as a source in Policyfile.rb. How does that work if we don’t keep all our cookbooks in a chef repo?

-Maciej

4

My thought was to just use git as the source for each of our cookbooks and tags to get specific versions.

That'll work fine, it'll just be a bit tedious. Eventually we'll have a way to pull in "partial policyfiles" that might help, but for now you'll end up doing a fair bit of duplicated work to update the tags on all your policies when you release a new tag. You could maybe use a little ruby-fu in your policyfiles to set all your tags from a single JSON or YAML file to automate this for now.

So you lost me when you suggested we could use chef_repo as a source in Policyfile.rb. How does that work if we don't keep all our cookbooks in a chef repo?

If you're happy with the one repo per cookbook setup then the chef repo cookbook source isn't relevant to you. Just pointing out that it's an option.

HTH,
Dan