We're doing exactly what Torben suggests not to do, use environments to lock down the cookbook versions. We've gone a few steps further than that, let me explain the workflow:
-we've decided our chef artifacts should be treated the same as our code, and are essentially part of our product. they even live in the same git repository.
-workflow: developers push to git, jenkins publishes to chef.
-when jenkins runs a build, it takes the build number (i.e.: 1234) and updates the cookbook minor version, such that chef-client cookbook version is 2.0.1234.
-environments are named as 'production-0_1234', and have the corresponding cookbook versions with the 1234 minor number set to the same number.
-roles are also named -0_1234.
-we have plans to do something similar with data bags but aren't there yet.
-our environments have nothing in them except for the cookbook versions, the software (debian package) version that we just built, and four attributes that are global.
-jenkins uploads all the cookbooks, roles and environments, moves the test servers from to the new environment and roles, and runs the tests.
-jenkins has a daily job to clean out old unused cookbooks, roles and environments.
When we decide to activate/publish/deploy a build, we use a custom tool to switch each node to the new environment and roles (since from chef's perspective the roles change) plus add some logging about who made the change.
Future plans:
-do versioning around some of the data bags
-publish builds daily and eventually publish every successful build.
On our dev laptops, we generate a somewhat random build number (based on a hash of the username etc) which is tied to the developer, and we use chef-client on our dev systems. That way if I need to do some testing on a specific node i can switch the node to my build number, test it, and the switch it back again. I want to eventually get to what Facebook mentioned at ChefConf, where you essentially 'checkout' a node and if you don't touch anything after an hour it gets automatically put back to the production version, but we're a bit away from that yet.
Why do it this way? We have some tough requirements to fill around versioning and testing, partially because we're managing remote appliances which. If we lose the ability to manage those appliances the only way to recover is to have the customer ship the appliance back to us, resulting in cost and embarrassment. This seemed the sanest way to ensure that we always get the exact version we expect to get.
I'd love to get some feedback on what we've done.
cheers
mike
--
Michael Hart
Arctic Wolf Networks
M: 226.388.4773
On 2013-07-17, at 2:16 AM, Torben Knerr <ukio@gmx.demailto:ukio@gmx.de> wrote:
Adding my 5 cents below... 
On Jul 16, 2013 6:50 PM, "Matthew Moretti" <werebus@gmail.commailto:werebus@gmail.com> wrote:
I'll take a stab at a few of these. Answers inline.
On Tue, Jul 16, 2013 at 5:02 AM, Florian Hehlen <Florian.Hehlen@mri-group.commailto:Florian.Hehlen@mri-group.com> wrote:
Hi all,
I am looking for some tips on chef best practices and having a hard time finding anything other than highly focused docs and tutorials on the web. Here are some of the issues I am struggling with:
· What granularity is best for recipes? When does it make sense to use some of the more complex ideas like depends, replaces, recommends/suggest, etc
Granularity is really up to you, but I think it's better to not have a given cookbook or recipe do too much. As far as the various cookbook metadata fields, I don't think I've ever seen any of them other than "depends" used. Do declare your dependencies, though.
As of Chef 11 you really must declare your dependencies via 'depends' in metadata, otherwise recipes will not be loaded (this was different in Chef 10).
+1 for fine-grained cookbooks, i.e. on the granularity level of apache, mysql etc.. - pretty much what you find on the community side. Having this level of granularity makes them reusable and composable.
When it comes to assembling your specific application by reusing / composing and configuring a set of these cookbooks (which I tend to call library cookbooks) then I would create a socalled "application cookbook" rather than using roles (the benefit of an "application cookbook" is that it can be properly versioned and dependency managed). My rule of thumb is that one application cookbook represents a whole node. These kinds of cookbooks are at a different level of granularity because they describe your whole application rather than the single parts it is composed of. This also makes it much less reusable than library cookbooks because they are so specific to your use case / application (i.e. they tend to live in a corporate git repo rather than on the community site)
· How to handle uninstallation? As different cookbooks, or as a different recipe, or as something that will be done under the covers by a replacement recipe…
It kind of depends on what your end goal is. I've never had the need to specifically remove a package in order to get to by desired end-state. If you do need that, I'd put
· How to structure roles in a generic and useful way for workstations and servers.
Again, this depends on your needs. Roles are generally used to group a set of related recipes, but some folks don't use them at all.
Some folks = proponents of the application cookbook pattern (see above)
See this blog post for a nice summary:
http://alluvium.com/blog/2013/05/03/the-application-cookbook-pattern-berkshelf-and-team-chef-workflow/
· I find that there is a certain amount of overlap in definition between node, client, and user. In fact bootstrapping creates a node and a client for each machine. Is there a good way to think about how these 3 relate to each other.
So, nodes are "the thing you're configuring" - they generally correspond to "servers" or "ec2 instances". Clients are "how you authenticate to the Chef Server", and you're right: if you're using a Chef server, they'll pretty much always be one-to-one (i think it's technically possible to do otherwise, but I wouldn't). Users are really just a special kind of client that also has a password for the WebUI - they generally correspond to people (like you).
· How to best manage recipe versioning using versions, environments, version constraints, etc. I am specifically wondering about being able to upload recipes that are being developed and tested without the fear that they will end up in production. I would like to stay away from having to manage 2 instances of chef.
I have a very specific opinion on that as I believe that you are doing it wrong if you are using environments to lock your complete cookbook dependency graph.
So what should you do instead?
- use application cookbooks in favor of roles (one app cookbook represents one node)
- lock your dependency versions in your application cookbook's metadata.rb (e.g. "depends 'apache2', '=1.0.0'"). When you release a version of your application cookbook lock the transitive dependencies as well so the complete graph of cookbook dependencies for that app cookbook / node is locked.
- in your environment files, only lock your application cookbook versions (their dependencies are locked in their metadata)
This approach gives you a clean dependency management, does not clutter your environments with implementation details (locking transitive cookbook dependencies), and makes it really easy to promote applications from one environment to another.
I'm aware that many folks (in my opinion) misuse environments for the sole purpose of locking the complete cookbook dependency graphs, and some tools are even encouraging this style of work (e.g. 'berks apply ENVIRONMENT').
Well, if you use roles, then you don't have a choice, but if you use application cookbooks, then this is really an antipattern (and I'm very opinionated about this :-))
P.S.: I was assuming you are using some kind of cookbook dependency management. I would recommend to use Berkshelf. You can also look for "The Berkshelf Way" - you should find some slides from ChefConf 2013...
HTH,
Torben
I realise that this is a whole lot of stuff in one email… this is a list of questions that has been growing in the last few weeks as I explore and plan for chef in our environment.
Cheers,
Flo
LEGAL DISCLAIMER
This communication and any attached documents are strictly confidential and/or legally privileged and they may not be used or disclosed by someone who is not a named recipient. If you have received this electronic communication in error please notify the sender by replying to this electronic communication inserting the word "misdirected" as the subject and delete this communication from your system.
Hope that helps,
Matt Moretti
