So, application cookbooks are one of those things that I think need more
discussion as a community -- there's a lot to be discussed here and this
really isn't the forum. I'm going to just not address this and explain how
chef-workflow relates to this, but will explain that I don't write
application cookbooks in the most accepted definition of the term.
Roles are an intrinsic part of chef workflow right now, unfortunately.
They're the "key" the test system uses to provision machines and is
generally how chef-workflow partitions groups of servers. I plan on
breaking this eventually, allowing you to define groups of machines based
on arbitrary conditions.
Buuuuut this doesn't really matter for this case because chef-workflow
really isn't about maintaining or testing cookbooks, it's about maintaining
and testing infrastructure -- what I mean by that is that you wouldn't use
chef-workflow to test a single cookbook in most instances, but instead use
chef-workflow to maintain machine profiles that actually exist on your
network -- roles are the easiest way to express this in chef generally.
If you wish to focus on specific cookbooks, I think you'd be better served
by test-kitchen -- chef-workflow 0.3 intends to integrate Fletcher's work
on jamie-ci (which will be test-kitchen 1.0) and for now has a rather crude
recipe testing system based on minitest-chef-handler to try and meet this
need, but ultimately chef-workflow has very little concern about specific
cookbooks and spends its energy focusing on simulating a network of
machines you'd actually put in production instead.
On Wed, Feb 13, 2013 at 2:29 AM, Torben Knerr ukio@gmx.de wrote:
Hi Erik,
it's the former problem that you mentioned, i.e. taking a developed
cookbook that's release ready and pushing it to chef servers.
Thanks for sharing your ideas on how to handle this, I haven't thought of
the first option with Private Chef yet. The third option is what I want to
avoid.
The second option is what sounds most friendly to me, but I'd probably use application
cookbookshttp://devopsanywhere.blogspot.de/2012/11/how-to-write-reusable-chef-cookbooks.html instead
of roles, with all transitive dependencies locked in the application
cookbook and then in the environment locking only the application
cookbooks' versions. This requires a cookbook resolver for sure. The
"release process" I had in mind was something like:
- make sure cookbook has only stable dependencies (only for application
cookbooks - library cookbooks should be lax imho)
- update cookbook version to release version
- pass tests
- create tag in vcs
- upload w/ freeze cookbook to chef server
- update cookbook version to next development version
I'm sure that with chef-workflow you could implement this process as a set
of rake tasks, but I was more interested how you deal with it currently
(which I have a lots of ideas now, so thanks! :-)).
One of my major pain points still is how to differentiate release versions
from development versions. As chef has a fixed versioning scheme of the
form "x.y.z" for cookbooks you can not use something like "1.1.0.dev" or
similar. For your own cookbooks you could define a rule like even version
numbers indicate stable/released versions while uneven version numbers
indicate in-development versions (in your chef-workflow tasks you would
enforce this).
While this works for your own cookbooks, you can't use it for forked
community cookbooks, e.g. for a fork of the apache2 1.5.0 cookbook I'd
ideally use 1.5.0.patch1, but that's not possible at the moment. So
currently I keep the version at 1.5.0 and use the Cheffile to specify the
git location of my forked version, but this happens on a sidetrack not
visible to chef itself - I'd rather specify 1.5.0.patch1 in metadata.rb if
that would be possible...
That's my ideas (and pains) so far...
Cheers, Torben
On Tue, Feb 12, 2013 at 2:53 PM, Erik Hollensbe erik@hollensbe.orgwrote:
I'm going to presume you're referring to the act of taking a developed
cookbook that's release ready and pushing it to chef servers that control
production as opposed to releasing it to the community. If what you really
want is the latter, I imagine there are few people more qualified to answer
this question than opscode's Joshua Timberman.
As for the former, there are a few approaches. Talk to your team and get
an idea of what they want to do. This could easily get very long, so I'll
do my best to keep it as brief as possible. These are just the ones I've
done, they have different advantages and they are not the only options.
Despite writing a lot of code surrounding workflow, I'm learning new (and
better) ways all the time too.
-
Multiple chef servers and vcs branches -- this probably lines up best
with chef-workflow from an "least amount of friction" standpoint (as c-w
centers around the idea of having isolated chef servers for development
cycles) but is not strictly necessary or the best idea. The idea is that
you keep a release branch and one or more integration branches and tags,
and the act of a release is merging to release, tagging HEAD and pushing
the whole thing out. Trivial to automate and your vcs lets you reason
pretty clearly about what's changing. Given private chef's recent changes
in licensing, it would be trivial to do this without a lot of
infrastructure and still keep everything in-house.
-
Full environment locks allow you to basically do the above on a single
chef server, but the environment is responsible for maintaining the
cookbook versions in use. This has other problems if you need to deal with
non-versioned artifacts such as roles or databags, but is arguably the most
"chef-ish" solution. This is a little more dependent on custom tooling to
get right -- a cookbook resolver such as Berkshelf or Librarian is pretty
important here, and I believe infochimps has some excellent tooling around
determining the changes in metadata that will happen as the result of an
upload. The big advantage here is the partitioning of cookbooks and
metadata (something the first solution needs a lot more work to do
properly) which is a big advantage for those looking to the community to
deal with a lot of their infrastructure concerns. It's also a little more
tolerant of one-offs simply because it will explode violently if you make
no attempt to be tolerant. 
-
The third is the obvious but less desirable "push it all and let $diety
sort it out", which while seeming absolutely scary is a real thing for lots
of places and is close enough for government work, making this a real
possibility for those who don't have a lot of energy to spend on the
problem or won't see significant gains by formalizing things. Don't
discount it because it's not actually a process is all I'm saying.
Chef-Workflow can work well with all three of these -- largely because it
mandates none of them. Your release process is your own.
On Tue, Feb 12, 2013 at 3:47 AM, Torben Knerr ukio@gmx.de wrote:
Hi Erik,
nice work! I have read the wiki with delight and it will definitely be
on the list for my next chef project.
Concerning workflow, I'm still looking for good way of handling cookbook
releases and release vs. in-development version schemes. Do you have any
ideas or suggestions about this as well?
Cheers,
Torben
On Fri, Feb 8, 2013 at 11:32 PM, Erik Hollensbe erik@hollensbe.orgwrote:
I'm pleased to announce the next major release of Chef-Workflow, 0.2.0.
Chef-Workflow is a toolkit for unifying your infrastructure management
with Chef. It aims to provide the tools you need to define your own
workflow and testing needs with powerful, sensible defaults, with minimal
dogma. The system is built as an "advanced tool for people with advanced
needs", giving you what you need to coordinate an operations team around a
set of in-house formal practices.
It's split into 2 major components with a unifying toolkit library.
-
Chef-Workflow Tasklib is a suite of rake tasks and supporting rake
toolkit to do common things you'd need to do with a chef server. The tasks
are malleable so you can compose your own method of working within your
team, and bring in optional included tasks to enhance your workflow,
already integrated with the system.
-
Chef-Workflow Testlib is an integration testing system that
fully orchestrates a network of chef managed machines and allows you to
test interactions between built systems.
Our Documentation is here:
https://github.com/chef-workflow/chef-workflow/wiki
If you wish for more detail on the release, there's a blog post that
goes into detail on the changes and there are changelogs available as well:
http://erik.hollensbe.org/2013/02/08/chef-workflow-0-dot-2-0-released/
Thanks for your time,
-Erik