# Policyfile workflow/mindset questions

**URL:** <https://discourse.chef.io/t/policyfile-workflow-mindset-questions/7568>\
**Category:** Chef Infra (archive)\
**Created:** [January 7, 2016, 10:01pm UTC](https://discourse.chef.io/t/policyfile-workflow-mindset-questions/7568 "2016-01-07T22:01:46Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![shanesveller](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/shanesveller/32/1178_2.png) [@shanesveller](https://discourse.chef.io/u/shanesveller)\
**Post date:** [January 7, 2016, 10:01pm UTC](https://discourse.chef.io/t/policyfile-workflow-mindset-questions/7568/1 "2016-01-07T22:01:47Z")

</div>

I’ve made a few casual, aborted attempts at updating our monolithic Chef repo to  
be able to use Policyfiles, and I’m ready to question whether our Chef usage  
patterns would be compatible with Policyfiles as they are currently implemented,  
or whether I’m just approaching Policyfiles in the “wrong” way.

I’ve read multiple times in multiple places that Policyfiles are meant to enable  
an opinionated workflow, and I’m willing to accept if that workflow just doesn’t  
jive with how things are currently done around here, or equally willing to check  
my assumptions at the door and adopt a new mindset if the resulting workflow  
truly is better.

# Current Chef usage

- Team of one (me), first introduced infrastructure-as-code/automation at this  
company last July - previous infrastructure was all ad-hoc
- Worked ~15 months with Chef at previous job and carried many (hopefully good)  
practices forward
- Merging/splitting businesses over company history, along with varying levels  
of security and performance requirements, led to multiple AWS accounts with no  
peering or external-to-external connectivity allowed (at this time) as well as  
some bare-metal servers at a local data center
- Currently have three separate Chef servers spread across these AWS accounts  
with their own single orgs and their own node inventory
- Each Chef org/business division has some common needs and some unique needs
- Each Chef org/business division has some combination of dev/staging/production  
nodes (both in actual real-world purpose and Chef node-\>environment assignments)
- Majority of Chef recipe logic for the common needs can be shared as-is between  
nodes belonging to different Chef orgs, with small variances that are mostly  
done via attributes
- Each Chef Server receives the full complement of our internal cookbooks, and  
most of the same data bags - to be quite honest, there are probably better  
isolation practices to be had here
- Chef environments are currently respected via conditional logic in our  
internal cookbooks, rather than applying attributes, roles or run lists  
directly themselves
- Internal cookbooks are version-pinned in production Chef environment for  
controlled roll-out
- All existing Chef nodes are single purpose (no sharing a host between DB and  
app, etc.)
- Nodes are provisioned for their intended purpose by composing base/shared  
recipes and single-purpose recipes via the run list
- No Chef roles are used at this time (my mentor in Chef believed they were an  
anti-pattern, for many of the reasons which led to the development of  
Policyfiles as far as I understand)
- Almost no community cookbooks are used as-is, but are wrapped first - some of  
the wrappers are merely setting attributes and then including the community  
cookbook
- Behavioral variances between organizations that are not merely “change this  
attribute to another value” are done via recipes that extend or replace the  
base recipe
- Merges to the `master` branch of our monolith Chef repo trigger an upload of  
cookbooks to all three Chef servers, via a root Berksfile/Berksfile.lock that  
walks the cookbooks directory for our internal cookbooks

# Policyfile-related questions and thoughts

_Some of the above makes Policyfiles seem appealing to me, but I am experiencing  
some hurdles for adoption and am trying to determine how many of them are  
technical and how many are due to incorrect or incomplete assumptions on my  
part, or my continuing use of bad or outdated habits within a newer paradigm._

- Are Policyfiles, as implemented, required to be the single entrypoint and  
source of truth for what Chef does to a given node? Is this likely to  
continue to be so?

- If yes to the above, are there any ways to DRY up the inclusion of the  
base/shared recipes which do things like set up our OS user accounts, package  
manager defaults, etc. when using Policyfiles?

- I currently deploy the same cookbook multiple times with  
dev/staging/production variants, often multiplied by the three Chef  
organizations we have that require slightly different configuration via  
attributes. Am I likely going to need a separate Policyfile per org and/or per  
environment?

- What’s the workflow look like for promoting a given cookbook version from dev  
to staging to production, or whatever their equivalent is at your organization?

- If you’ve implemented Policyfiles at your organization, could you describe  
what some of your project’s file structure and workflow looks like? I’m  
considering adopting a `policies/$org/$purpose/Policyfile.rb` structure or  
similar and storing them as part of our monolithic Chef repo if I pursue this  
seriously

- Last I checked, it was not possible to add or override arbitrary attributes in  
TestKitchen when using the `policyfile_zero` provisioner, like you could with  
the other provisioners, but that might be added eventually. What about in  
"real-world" usage?

- Has the story about other third-party tools (i.e. Terraform) gotten any better  
yet?

## Closing Comments

In particular, I think a few of our simpler wrapper cookbooks could disappear  
entirely in favor of Policyfiles.

The main things blocking my adoption, other than inexperience, are my widespread  
use of `chef-sugar` in multiple internal cookbooks, which will require either an  
update to that cookbook or a rewrite to use within a Policyfile due to  
[this issue](https://github.com/sethvargo/chef-sugar/issues/114).

The monolithic repo model is also proving to make it difficult if not impossible  
to do any kind of gradual update and rollout of a shift to Policyfiles. This has  
also caused me headaches with our trial of Delivery, so it may be time to beg  
the powers that be for permission to create some 30-40 additional private Git  
repos.

---

<div class="post-metadata">

**Author:** ![kallistec](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/kallistec/32/23_2.png) [@kallistec](https://discourse.chef.io/u/kallistec)\
**Post date:** [January 8, 2016, 12:50am UTC](https://discourse.chef.io/t/policyfile-workflow-mindset-questions/7568/2 "2016-01-08T00:50:56Z")

</div>

> [@shanesveller](#):
>
> Are Policyfiles, as implemented, required to be the single entrypoint andsource of truth for what Chef does to a given node? Is this likely tocontinue to be so?

I'm not entirely sure what you mean by this, but I can say that the Policyfile.lock.json that gets generated is the only place that chef-client will look for a run list, the attributes at the role-level precedence, and the cookbook version set. Currently, the only input into the process that creates lock JSON is one ruby file. In the future we're planning to have some kind of "snippet" mechanism where you can have multiple files as inputs to that process. That will be a post-1.0 kind of thing though.

> [@shanesveller](#):
>
> If yes to the above, are there any ways to DRY up the inclusion of thebase/shared recipes which do things like set up our OS user accounts, packagemanager defaults, etc. when using Policyfiles?

Right now you would have to either accept the duplication or use role cookbook type approaches.

> [@shanesveller](#):
>
> I currently deploy the same cookbook multiple times with dev/staging/production variants, often multiplied by the three Chef organizations we have that require slightly different configuration via attributes. Am I likely going to need a separate Policyfile per org and/or per environment?

You would have a policy group for dev/stage/prod; the point of the policy group is you can have a different revision of your policy active in each group. What I've done to replace environment attributes is stick those in a data bag item and then copy that to the node via a cookbook. The other way to do that is to put the full matrix of attributes in your policyfile and then select the right set for the environment with logic in a cookbook.

> [@shanesveller](#):
>
> If you've implemented Policyfiles at your organization, could you describe what some of your project's file structure and workflow looks like? I'm considering adopting a policies/$org/$purpose/Policyfile.rb structure orsimilar and storing them as part of our monolithic Chef repo if I pursue this seriously

I have a customer who uses a monolithic repo, with a single policies/ directory. You can name the policies whatever you want (e.g., policies/some-app-frontend.rb) and all the `chef` commands support an extra argument to reference the policyfile (e.g., `chef install policies/some-app-frontend.rb`). Though if you would be happier with shorter command lines and using `cd` to change contexts, what you've described is also ok.

> [@shanesveller](#):
>
> Last I checked, it was not possible to add or override arbitrary attributes in TestKitchen when using the policyfile\_zero provisioner, like you could with the other provisioners, but that might be added eventually. What about in"real-world" usage?

TestKitchen will soon natively support policyfiles the same way it does berks (no custom provisioner required), and we'll address that then.

> [@shanesveller](#):
>
> Has the story about other third-party tools (i.e. Terraform) gotten any betteryet?

I'm not personally aware of the problem, but if you have the latest chef-client and chef server, you can set the policy name and policy group in the node json, if that helps.

> [@shanesveller](#):
>
> The main things blocking my adoption, other than inexperience, are my widespreaduse of chef-sugar in multiple internal cookbooks, which will require either anupdate to that cookbook or a rewrite to use within a Policyfile due tothis issue.

That issue should only apply if you are using the "compatibility mode" which is where cookbooks get uploaded with zany version numbers to emulate the behavior of the cookbook\_artifact endpoint that now exists in Chef Server. Unfortunately, Chef Zero doesn't support that API just yet, and Chef Zero is what you use in test kitchen (via the chef-client local mode feature), but that is also coming soon. That bug should go away once Chef Zero supports the new policyfile and cookbook artifact APIs.

> [@shanesveller](#):
>
> The monolithic repo model is also proving to make it difficult if not impossible to do any kind of gradual update and rollout of a shift to Policyfiles. This hasalso caused me headaches with our trial of Delivery, so it may be time to beg the powers that be for permission to create some 30-40 additional private Git repos.

You should be able to do full verticals by server type. As long as you don't turn on the compatibility mode, you can have some nodes using policyfiles and some not using them in the same Chef Server and organization. Likewise, storing your Policyfiles in a monolithic repo is an explicitly supported thing in Policyfiles, and you can use the `default_source :chef_repo, "path"` directive to pull cookbooks from a monolithic repo.

---

<div class="post-metadata">

**Author:** ![shanesveller](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/shanesveller/32/1178_2.png) [@shanesveller](https://discourse.chef.io/u/shanesveller)\
**Post date:** [January 8, 2016, 1:00am UTC](https://discourse.chef.io/t/policyfile-workflow-mindset-questions/7568/3 "2016-01-08T01:00:49Z")

</div>

Thanks for the very thorough reply, Daniel!

---

<div class="post-metadata">

**Author:** ![yoshiwaan](https://avatars.discourse-cdn.com/v4/letter/y/ba9def/32.png) [@yoshiwaan](https://discourse.chef.io/u/yoshiwaan)\
**Post date:** [January 8, 2016, 9:35pm UTC](https://discourse.chef.io/t/policyfile-workflow-mindset-questions/7568/4 "2016-01-08T21:35:47Z")

</div>

_I currently deploy the same cookbook multiple times with  
dev/staging/production variants, often multiplied by the three Chef  
organizations we have that require slightly different configuration via  
attributes_

Different variants or different versions of the same cookbook? If it is  
actually different variants then I feel this is better handled by  
environments (or some equivalent like a structure in your policyfile to  
supply environment variables) and let the logic in the cookbook code make  
the changes for the environment based on those attributes. That way you’re  
controlling a common set of cookbooks for all environments (per org).

I feel like this might simplify your implementation (it will at least  
eliminate the need for having a separate Policyfile per environment, which  
seems counter-intuitive to me), especially if the desire is to test a  
Policyfile on an environment and then promote that same policy file to the  
next environment as a way of controlling release/change.

The last time I looked at policyfiles (which was admittedly a while ago) it  
seemed like environment support was a little lacking/difficult, but it also  
isn’t necessarily something that is directly related to Policyfiles.  
I think it could solve a few problems if the Policyfile structure could  
somehow reference per environment (per policy group?) attributes, either  
from a data structure in the policyfile or from an external file, but with  
the output lock file containing all static values so it can still be a  
versionable object.

---

<div class="post-metadata">

**Author:** ![kallistec](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/kallistec/32/23_2.png) [@kallistec](https://discourse.chef.io/u/kallistec)\
**Post date:** [January 9, 2016, 12:12am UTC](https://discourse.chef.io/t/policyfile-workflow-mindset-questions/7568/5 "2016-01-09T00:12:30Z")

</div>

> [@yoshiwaan](#):
>
> The last time I looked at policyfiles (which was admittedly a while ago) it  
> seemed like environment support was a little lacking/difficult, but it also  
> isn't necessarily something that is directly related to Policyfiles.  
> I think it could solve a few problems if the Policyfile structure could  
> somehow reference per environment (per policy group?) attributes, either  
> from a data structure in the policyfile or from an external file, but with  
> the output lock file containing all static values so it can still be a  
> versionable object.

I've done something like this with data bags:

```auto
  env_data = Chef::DataBagItem.load('application_environment_APPNAME', LIFECYCLE_STAGE).raw_data
  env_data.delete('id')
  # Use `env_default` because it will be unused in policyfile mode
  attributes.env_default = env_data

```

As you say, the data bag item could be updated out of band, but in this case that wasn't a concern because the data bag item is owned by the same CD system that is managing the cookbooks and policyfiles.

Open to hearing what others have done and adding sugar in Chef core for the most successful patterns.