Hello Chef Experts!
i do apology for the lengthy post.
recent job switch caused me to move away from the configuration manager competition and into Chef… well Opsworks to be precise but i’m leaving out the aws part for now…
All good and well and chef is pretty awesome but i can’t help to notice some key differences between the two products as for running stuff into production and since i’ve add some success managing my old infrastructure i would very much like to find again that type of stability and flow.
I’ve tried to find some documentation but nothing yet that covers everything.
Repos?
We had multiple repos for handling roles/profiles and hiera. That way we could separate the data from the code. How can this be done via Chef? I know about chef roles but would love something more global and something more separated! Do people use a single repo and do sub-modules to individually manage different parts?
chef-repo/ <- Main repository
- system-tunning <- Part of main repo
- roles <- Part of main repo
- chef-splunk <- Git Submodule from supermarket
- cassandra <- Git submodule from supermarket
- nginx <- Git submodule from supermarket
- SuperSoftware <- Internal Git submodule
- splunk-config <- wrapper cookbook?
- nginx-config <- wrapper cookbook?
- cassandra-config <-wrapper cookbook?
I could find dozens of ways a single big repo could fail and yet the company i work for now uses this… with branches for features and then merging everything to master… i smell catastrophe in the making so would love your ideas on NOT doing a single big repository.
Data?
Speaking of data… I know about databags or precedence or attributes but what is used in production??? I have yet to find a standard like hiera. The idea is not much about getting all my data in ONE place but as to find a easy way to manage my data.
Cookbook management?
Module management was dynamically handle via r10k meaning it was easy to either keep the version of the module we wanted or upgrade to use the new functionality without touching the code of the module. I would love to do this with cookbooks.
chef-repo/
- chef-splunk <- don’t touch this
- splunk-config <- could be modified. Like a wrapper-cookbook (don’t know if that term is used)
I know about Berkshelf but i would like something more global… something that used environnement instead of single cookbook.
Rolling out?
Rolling out was easy. Commit your code via Gitlab-CI and dev/stage was automatically tested (linted, style guide, tests) and updated with R10k launched from the puppet server. Don’t even have to log in the server it was all automated. Would very much like that to find a git flow that would replicate that. Lint and Style guide included 
I already use test-kitchen but since we have a few dev that use the environnement i would again like something more global.
Not sure if i make sense or people use a totally different setup that has nothing to do with my speech but i’m open to every solution at the moment!
Thanks all!!
I'll start by the end for what is sure:
Commit your code via Gitlab-CI and dev/stage was automatically tested (linted, style guide, tests) and updated with R10k launched from the puppet server. Don't even have to log in the server it was all automated.
My .gitlab-ci.yml for linting (with chef-dk pre installed on the runner/docker image):
before_script:
- /opt/chefdk/bin/chef shell-init bash
Linting:
stage: test
script:
- /opt/chefdk/embedded/bin/chef --version
- /opt/chefdk/embedded/bin/cookstyle --version
- /opt/chefdk/embedded/bin/cookstyle
- /opt/chefdk/embedded/bin/foodcritic --version
- /opt/chefdk/embedded/bin/foodcritic . --exclude spec -P
# Unit test (commented for this one as not applicable
#- /opt/chefdk/embedded/bin/rspec spec
Now for the roll out, depends on the workflow you're following, could be a berks upload or taking advantage of knife spork.
All in all the "recommended" way is one repo per cookbook, usually along with berkshelf or policyfiles to handle the whole dependency resolution and ensure everything is in line. Sub modules are not that used and the tooling doesn't support it for what I know.
wrapper-cookbook is exactly a pattern, used in the sense that it include a community or 'base' cookbook and it overrides values to drive it or the 'base' only provide resources and the wrapper make use of them in it's specific way.
For example, the mysql cookbook provides resources only, you'll make a company wrapper for basic install according to your needs and probably use another wrapper around the company's cookbook to tweak different installs.
That said you did mention Opsworks as target, which has it's own structure with layers, it is different from what we do with a chef-server and some options won't work or won't fit in an Opsworks environment from what I did saw from it.
Hope I did shed some light, I probably forgot parts when writing this, hopefully someone else will write something better, but feel free to ping me if you wish some details on some parts.
1 Like
Thanks for all your answers! It’s exactly what i’m looking for.
I will check out knife spork. Looks good.
Yeah i know about the different layers of the opsworks stack but with a feel for the chef architecture i will be able to adapt. I’m so far behind in terms of Chef deployement that for now everyone’s input is gold.
Regarding wrappers… taking your mysql example for the cookbook-wrapper is there a naming convention for those types of wrappers? Like -config or -mycompany_mysql or mysql-profile… etc… i know it’s just a name but when you have a big repo with tons of stuff in it, it helps put things in the right place.
Oh and last thing… data Taking again your mysql cookbook wrapper… lets say you have supercompany_mysql and you want to change the install dir… you would put this in the default recipe, install recipe or attributes? I know it doesn’t change the fact that it will work but again i’m looking to clean up the code and every styling guide / suggestion i can get my hands into i’ll take 
Again thank you very much for all your help and time!
For my part I prefix with mycompany every internal cookbook.
then for where to make the changes, when it’s attributes I do supersede them at same level in attribute file as much as possible, for edge cases I use node.default in the wrapper recipe, and for some others cases I use rewind to modify a community cookbook behavior in the wrapper recipe.
I do a really low use of databags as they are prompt to break when multiples person try to update them at the same time.
I do use some role attributes, but really lightly, usually to use the role and/or attribute as a search key (for clusters in the same environment). (A role named mysql-server-cl1 with attribute cluster-name = "cl1" for example so I can search within the chef_environment the partners in this cluster)
It worth to say the role could be emulated with some more control with a cookbook which have as sole purpose defining attributes (and maybe a runlist with depends and include_recipe), bringing a version limitation possibility at the environment level for this role, which legacy roles doesn’t offer.
1 Like
I also did the mycompany-mysql type of naming. Glad i’m in the right direction at least regarding that
i’ll get used to the level i guess for the attributes. Haven’t had an issue where i needed to really specify for now.
Well i could see role attributes being usefull at least in a couple of cases:
- domain specific stuff (used 4,5 domains at my old job so setting those individually would have been a mess)
- Like you said cluster attributes in the same environment
- generic users (but people shouldn’t use those… ever)
- software users… just to have a homogenized park.
- Default system settings (tcp, kernel, fs options, etc) for architecture type (video streaming box, web box, cluster node)
If i have to set every one of these every time i won’t be happy! That is why we loved hiera so much so that you could set domain, specific node type of settings and be done with it!
Well any “default” attribute can be in it’s own cookbook, the main reason is propagation between environments, an update to a role is environment wide, where a cookbook can be version bumped and be used on a fixed version in each environment.
The goal is the same, don’t repeat yourself
For information I’ve two “base_infrastructure” cookbooks, one for windows boxes, one for linux boxes, each include specific platform cookbooks and some OS tweaks/configuration, each node gets one of them as first in the runlist when provisionning/bootstrapping it.
Side note for cookbook naming: ruby doesn’t like the - in class names, as the resources are class instances named cookbook_resource, they’ll be modified into a _, this could be strange for newcomers, so the recomendation is to avoid - in cookbook names, I personnaly use them to differentiate between attributes wrapper cookbooks and “library” cookbooks (the later defining custom resources when the former only define attributes)
1 Like