Hi,
after adapting more than enough community cookbooks to our local setup,
I published a blog entry yesterday to describe my idea of improving the
status quo wrt/ reusable cookbooks:
http://blog.nistu.de/2012/03/04/reusability-in-configuration-management-systems/
I would love to discuss these ideas and see what people think about it.
cheers,
Nikolay
–
“It’s all part of my Can’t-Do approach to life.” Wally
On Mar 5, 2012, at 9:54 AM, Nikolay Sturm wrote:
Hi,
after adapting more than enough community cookbooks to our local setup,
I published a blog entry yesterday to describe my idea of improving the
status quo wrt/ reusable cookbooks:http://blog.nistu.de/2012/03/04/reusability-in-configuration-management-systems/
I would love to discuss these ideas and see what people think about it.
cheers,
Nikolay
--
"It's all part of my Can't-Do approach to life." Wally
Here here! I couldn't agree more with your suggestions.
In addition, I think we need better tools to manage, share, and collaborate on our cookbooks. So, for example, if I write a cookbook that's tightly-coupled to my infrastructure and you come along and improve it along the lines you describe in your post, there should be a way for you to put your version forward to the community and then for everyone to collectively communicate that yours is the new best version and submit their own improvements, etc.
So this could be a layer on top of GitHub (and we'd need to make the knife-GitHub integration more official), or it could be a vastly-improved community.opscode.com, or something else entirely. But right now there is just too much friction between "I've made this cookbook more general" and "this is now the primary version of this cookbook."
I made this comment on the blog post and haven't had time to respond
to its reponse.
"cookbooks by default have to be opinionated"
I stand by that for this reason (with a few caveats):
Having said that, yes, folks who publish cookbooks should consider
making as much as possible as attributes. The reason here is obvious.
It's much easier to override attributes than to override an entire
recipe. As a counter though, if the attitude is that folks have to
have a fully configurable cross-platform cookbook before they can
release it, then I don't want that. I'd rather have an unnecessarily
opinionated cookbook than no cookbook at all. In fact, I'm working
with someone at the new company who has very little chef experience
and my advice was to hard code everything. We can refactor into
attributes later.
Here's an example of a cookbook we used at my last gig where I made
some things overly configurable even though we weren't ready for it
yet:
default[:portal][:container] = "jetty"
default[:portal][:base_dir] = "/usr/local/portal/"
default[:portal][:user] = "portal"
default[:portal][:group] = "portal"
default[:portal][:context_path] = "/portal"
default[:portal][:warfile] = "portal.war"
We were not even remotely ready to run on anything other than jetty
and the context root HAD to be portal, however I went ahead and made
them attributes just in case we ever got to the point where we weren't
tied so heavily to jetty.
I think cookbook authors could do at least the following as attributes
since these areas are probably the most common:
user
group
path
(maybe) init script
use 'conf.d' setups where available instead of owning a file wholesale.
The main reason for not doing initscript is that it litters the
codebase with ugly-ass conditionals which actually HURT readability.
I'm considering a proposal to make init scripts more intelligent like
the runit LWRP. Yes, you can sort of do it with file specificity in
the templates directory but upstart files don't go in the same places
as SYSV init scripts so you've still got some conditionals.
Also, the post is awesome and very much appreciated. I just think that
in a few areas it goes to far to the extreme. With proper attributes,
there's no reason you need a distinct recipe JUST for group creation
(and at worst case you've created an ordering issue that might not be
readily apparent.
On Mon, Mar 5, 2012 at 10:25 AM, Wes Morgan cap10morgan@gmail.com wrote:
On Mar 5, 2012, at 9:54 AM, Nikolay Sturm wrote:
Hi,
after adapting more than enough community cookbooks to our local setup,
I published a blog entry yesterday to describe my idea of improving the
status quo wrt/ reusable cookbooks:http://blog.nistu.de/2012/03/04/reusability-in-configuration-management-systems/
I would love to discuss these ideas and see what people think about it.
cheers,
Nikolay
--
"It's all part of my Can't-Do approach to life." WallyHere here! I couldn't agree more with your suggestions.
In addition, I think we need better tools to manage, share, and collaborate on our cookbooks. So, for example, if I write a cookbook that's tightly-coupled to my infrastructure and you come along and improve it along the lines you describe in your post, there should be a way for you to put your version forward to the community and then for everyone to collectively communicate that yours is the new best version and submit their own improvements, etc.
So this could be a layer on top of GitHub (and we'd need to make the knife-GitHub integration more official), or it could be a vastly-improved community.opscode.com, or something else entirely. But right now there is just too much friction between "I've made this cookbook more general" and "this is now the primary version of this cookbook."
On Mar 5, 2012, at 9:25 AM, Wes Morgan wrote:
So this could be a layer on top of GitHub (and we'd need to make the knife-GitHub integration more official), or it could be a vastly-improved community.opscode.com, or something else entirely. But right now there is just too much friction between "I've made this cookbook more general" and "this is now the primary version of this cookbook."
As Opscode splits their cookbooks into individual repos, they will be able to more easily do this. So long as everyone does a proper fork of the project on github, the branches should all remain visible and accessible, and then it's up to the folks who run the repo as to whose changes they actually choose to incorporate.
I think the Cookbooks project on github is effectively already doing this kind of thing for the various repos they track.
IMO, the real trick is getting everyone to properly fork and track their projects, and then getting people who are willing and able to apply the generalizations to the site-specific cookbooks that are released.
I like Nikolay's idea, but I'm not a real programmer and I don't apply the methods he describes. It would be up to someone else to come along and take whatever I've rigged together with spit, bailing wire, chewing gum, and a whole lot of prayer, and then turn that into something more general and useful.
For example, when people hear that I've written a cookbook for the Zerigo dynamic DNS service, they might be interested to see what I've got -- that is, until they actually do see it, and realize just how horrible it really is. But, it actually works for us, which is something that I can't say for the other Zerigo cookbook we found. And what I really care about is that something works, not how pretty it is.
--
Brad Knowles bknowles@ihiji.com
SAGE Level IV, Chef Level 0.0.1
I have to disagree with you john, I do think that there can be value into
moving components as granular as group creation into separate recipes.
There certainly are downsides as you point out:
But the upsides are huge
I don't think anyone thinks that your cookbook should start out looking
like Nikolay's example but it can be the ultimate refactored version.
In my work I often have to model an existing configuration because that is
what the customer wants, there are lots of arbitrary config details that
are not relevant to other parts of the cookbook, further i have many
similar servers w/ minor variations.
Splitting parts of recipes into lots of small individual recipes could make
cookbooks much more testable.
We're churning out tons of cookbooks w/ tons of overlap. If we don't figure
out to better reuse code, well, we won't be automating as much as we should
be 
congrats to Nikolay for the great article. It is important data point in a
larger discussion.
On Mon, Mar 5, 2012 at 4:40 PM, John E. Vincent (lusis) <lusis.org+
chef-list@gmail.com> wrote:
I made this comment on the blog post and haven't had time to respond
to its reponse."cookbooks by default have to be opinionated"
I stand by that for this reason (with a few caveats):
- You must have defaults. You can't lay bits out on disk without some
opinion of where they need to go- Some things don't make sense as configurable (I'm not going to
deploy a php app to rails and I'm I might HAVE to use a specific
version of php)- In cases where cookbooks have interdependencies, you must be able to
express that (take the opscode users cookbook and nagios cookbook -
both want an explcitly named admin group)Having said that, yes, folks who publish cookbooks should consider
making as much as possible as attributes. The reason here is obvious.
It's much easier to override attributes than to override an entire
recipe. As a counter though, if the attitude is that folks have to
have a fully configurable cross-platform cookbook before they can
release it, then I don't want that. I'd rather have an unnecessarily
opinionated cookbook than no cookbook at all. In fact, I'm working
with someone at the new company who has very little chef experience
and my advice was to hard code everything. We can refactor into
attributes later.Here's an example of a cookbook we used at my last gig where I made
some things overly configurable even though we weren't ready for it
yet:default[:portal][:container] = "jetty"
default[:portal][:base_dir] = "/usr/local/portal/"
default[:portal][:user] = "portal"
default[:portal][:group] = "portal"
default[:portal][:context_path] = "/portal"
default[:portal][:warfile] = "portal.war"We were not even remotely ready to run on anything other than jetty
and the context root HAD to be portal, however I went ahead and made
them attributes just in case we ever got to the point where we weren't
tied so heavily to jetty.I think cookbook authors could do at least the following as attributes
since these areas are probably the most common:user
group
path
(maybe) init script
use 'conf.d' setups where available instead of owning a file wholesale.The main reason for not doing initscript is that it litters the
codebase with ugly-ass conditionals which actually HURT readability.
I'm considering a proposal to make init scripts more intelligent like
the runit LWRP. Yes, you can sort of do it with file specificity in
the templates directory but upstart files don't go in the same places
as SYSV init scripts so you've still got some conditionals.Also, the post is awesome and very much appreciated. I just think that
in a few areas it goes to far to the extreme. With proper attributes,
there's no reason you need a distinct recipe JUST for group creation
(and at worst case you've created an ordering issue that might not be
readily apparent.On Mon, Mar 5, 2012 at 10:25 AM, Wes Morgan cap10morgan@gmail.com wrote:
On Mar 5, 2012, at 9:54 AM, Nikolay Sturm wrote:
Hi,
after adapting more than enough community cookbooks to our local setup,
I published a blog entry yesterday to describe my idea of improving the
status quo wrt/ reusable cookbooks:http://blog.nistu.de/2012/03/04/reusability-in-configuration-management-systems/
I would love to discuss these ideas and see what people think about it.
cheers,
Nikolay
--
"It's all part of my Can't-Do approach to life." WallyHere here! I couldn't agree more with your suggestions.
In addition, I think we need better tools to manage, share, and
collaborate on our cookbooks. So, for example, if I write a cookbook that's
tightly-coupled to my infrastructure and you come along and improve it
along the lines you describe in your post, there should be a way for you to
put your version forward to the community and then for everyone to
collectively communicate that yours is the new best version and submit
their own improvements, etc.So this could be a layer on top of GitHub (and we'd need to make the
knife-GitHub integration more official), or it could be a vastly-improved
community.opscode.com, or something else entirely. But right now there is
just too much friction between "I've made this cookbook more general" and
"this is now the primary version of this cookbook."
On Tue, Mar 6, 2012 at 4:59 AM, Bryan Berry bryan.berry@gmail.com wrote:
I have to disagree with you john, I do think that there can be value into
moving components as granular as group creation into separate recipes.
Disagreement is not allowed. We need group think consensus
There certainly are downsides as you point out:
- less readable
- can hide or create ordering problems
But the upsides are huge
I don't think anyone thinks that your cookbook should start out looking like
Nikolay's example but it can be the ultimate refactored version.In my work I often have to model an existing configuration because that is
what the customer wants, there are lots of arbitrary config details that are
not relevant to other parts of the cookbook, further i have many similar
servers w/ minor variations.Splitting parts of recipes into lots of small individual recipes could make
cookbooks much more testable.We're churning out tons of cookbooks w/ tons of overlap. If we don't figure
out to better reuse code, well, we won't be automating as much as we should
be
The problem here is that a code reuse system already exists in Chef -
attributes. This is the mechanism by which we define
variable...variables It is much easier to me to simply be able to
say "production environment uses X for a variable while development
uses Y" via environment level attributes than to have to modify a
recipe (and in this case hunt down which one it the value is being
used).
The problem here is that going down the route of millions of recipes
that do one thing is just as bad as overdoing attributes. The only
REAL variation that chef does not yet address is abstracting out
certain things that are hard problems (init scripts, package naming
differences).
congrats to Nikolay for the great article. It is important data point in a
larger discussion.
Indeed. Great post.
Also, the post is awesome and very much appreciated. I just think that
in a few areas it goes to far to the extreme. With proper attributes,
there's no reason you need a distinct recipe JUST for group creation
(and at worst case you've created an ordering issue that might not be
readily apparent.
John, I agree with most of your comments in this thread, I would argue,
however, that my approach to code reuse (composable recipes) solves a
different problem than using attributes does.
IMHO attributes are primarily used to vary specific data points: which
username to use, which IP to bind to, which directory to install to...
While you can vary these things, you still have to agree to the
policies(?) set in place by the cookbook author: setup a local user and
group with the same name, linking executables to /usr/local/bin, ...
Composable recipes allow you to adapt these policy decisions without
much hassle.
After reading the responses to my post, I see other problems as possibly
more urgent for the community at large (comming up with clean cookbooks,
improving existing cookbooks instead of reinventing the wheel). I myself
will further explore composable recipes and see if the policies I talk
about are a real problem, or just a perceived one. ATM I honestly don't
know.
I would like to thank everyone for their feedback, it was really helpful
to me.
cheers,
Nikolay