If anyone out there is taking advantage of the !merge logic in deep
merge (http://wiki.opscode.com/display/chef/Deep+Merge) I’d appreciate
your opinion on the additional functionality I’d like to add in
http://tickets.opscode.com/browse/CHEF-2737
Any use cases you could envision for it would be appreciated as well.
Matthew Kent | http://magoazul.com
On Wed, Dec 7, 2011 at 7:58 PM, Matthew Kent
mkent+chef-users@magoazul.com wrote:
If anyone out there is taking advantage of the !merge logic in deep
merge (http://wiki.opscode.com/display/chef/Deep+Merge) I'd appreciate
your opinion on the additional functionality I'd like to add inhttp://tickets.opscode.com/browse/CHEF-2737
Any use cases you could envision for it would be appreciated as well.
I first think of situations where you would want something to happen
everywhere by default, but not in a handful of places.
There have been times when I have wanted to put some kind of logic in
roles. Take the hardware cookbook framework for instance.
You deploy to three locations. Development happens to be on Dell 1U
servers, production is on an HP blade chassis and in EC2. You monitor
all of this hardware differently. Notable your Dell and HP clusters
run OMSA and SMH respectively, the vendors software packages. How do
you automate the installation of this software where it should be, and
not where it should not?
Because by design we only send cookbooks to a node when it has that
cookbook in its run list, or there is a dependency between the
cookbooks, I strive to avoid sending cookbooks to nodes that don't
need them. I think I worry about this too much.
The last time I was faced with this problem I created a hardware
cookbook which depended on the specific hardware related cookbooks.
The default recipe for this cookbook is in the base role which is
applied to all nodes. This recipe then reviews the DMI manufacturer
information and includes the relevant recipes in the other
hardware-specific cookbooks.
The downside to this is you've got one cookbook which just makes some
simple choices to include other cookbooks. You wonder why you need a
whole cookbook for this. Yet, it succeeds in simplicity because you
already understand cookbooks and you don't have to learn another
concept to leverage it.
This is where I worry about deep merge of role attributes. The
precedence level of attributes is already hard enough for most people
and there is a tendency to abuse this when something is happening that
we don't understand; we just use an override. Eventually this becomes
an arms race; people have asked how to override and override
attribute.
Use cases often seem like something that should be recipe logic. But
if you're trying to abstract business out of your recipes, role
attributes could be the place to do it.
Imagine setting the number of workers for a daemon in a role. You
could adjust this value with role attribute deep merge. If the default
number of works in the recipe is too low for you, you could set a
higher one in your base role. Maybe another node needs even more. But
you could do this with override attributes.
Maybe the ports a server should listen to? Default apache to 80, and
ssl servers use deep merge to add 443 while intranet servers add 8080?
This should really be done in cookbook logic though.
Perhaps a multiplier for the the amount of memory to use for an
application based on the amount available? Still seems like recipe
logic to me.
Deep merge seems like it could solve a really neat problem for
someone, but I don't know what that problem is. Hopefully I've
inspired someone else to have more problems than me.
Bryan
On Wed, Dec 7, 2011 at 6:11 PM, Bryan McLellan btm@loftninjas.org wrote:
<snipped some great and interesting thoughts I'll leave for others to comment>
Deep merge seems like it could solve a really neat problem for
someone, but I don't know what that problem is. Hopefully I've
inspired someone else to have more problems than me.
To be clear about what the ticket concerns is only the merging of
arrays during the deep merge process. The merging of hashes in deep
merge has proven itself useful in role inheritance (I think), allowing
for greater role reuse and organization.
As for solving a problem I think "someone" is definitely the key here
Deep merging may not seem terribly helpful when it comes to single-purpose,
highly specific, custom cookbooks which basically know what their
configurations should be.
But it is helpful when it comes to reusing third-party cookbooks: it
permits writing custom roles to control and integrate with complex generic
third-party cookbooks which look at deep attributes structures to decide
what to do.
There is the alternative strategy that such cookbooks might instead provide
definitions and resources which may then be called by a custom cookbook, so
that instead of writing a custom role to control the third-party cookbook
one writes a custom cookbook to control the third-party cookbook. But that
requires much greater effort on the part of the author of the third-party
cookbook. So we still need the ability to control and integrate with
third-party cookbooks by mangling the attributes.
On Thu, Dec 8, 2011 at 1:47 AM, Matthew Kent
mkent+chef-users@magoazul.comwrote:
On Wed, Dec 7, 2011 at 6:11 PM, Bryan McLellan btm@loftninjas.org wrote:
<snipped some great and interesting thoughts I'll leave for others to
comment>Deep merge seems like it could solve a really neat problem for
someone, but I don't know what that problem is. Hopefully I've
inspired someone else to have more problems than me.To be clear about what the ticket concerns is only the merging of
arrays during the deep merge process. The merging of hashes in deep
merge has proven itself useful in role inheritance (I think), allowing
for greater role reuse and organization.As for solving a problem I think "someone" is definitely the key here
- the control over the merging of arrays outlined in CHEF-2737 is only
one extra layer of flexibility that, like say default attributes in
roles, you can skip over and be just fine. It could even be made clear
in the deep merge documentation that there are other ways to achieve
this functionality. To me Chef is already very flexible and this makes
it slightly more so. Although if the eventual goal is to pare down
some of the flexibility in favour of making Chef simpler than I agree
this probably goes in the wrong direction.
--
Matthew Kent | http://magoazul.com
On Thu, Dec 8, 2011 at 10:03 AM, Jay Feldblum y_feldblum@yahoo.com wrote:
But it is helpful when it comes to reusing third-party cookbooks: it permits
writing custom roles to control and integrate with complex generic
third-party cookbooks which look at deep attributes structures to decide
what to do.
Could you provide a specific example for others to follow to understand?
Bryan
Imagine a cookbook for implementing firewall rules, which inspected the
node attributes to discover the rules, and was expected to be configured
via roles. One role might want to append a rule, where each rule can
include multiple source and destination interfaces, ip-addresses,
protocols, port ranges, etc. Another node might want to append another
rule, and a third role might want to update the first rule by appending
e.g. an additional port range.
Imagine the application or database cookbooks were implemented differently,
where they inspected node attributes rather than searched data bags. The
configurations for these are rather complex. Then one role might want to
add a some data, while another role might want to update it; the first role
providing blanket/default data, the second role being application-specific.
Imagine fnichol's chef-rvm cookbook had default attributes that you don't
like. You want your role to add the cookbook to the run-list but delete
those default attributes, for example, delete the nested attribute that
says ruby vX should be installed by default or gem Y should be installed by
default in each ruby.
On Thu, Dec 8, 2011 at 10:45 AM, Bryan McLellan btm@loftninjas.org wrote:
On Thu, Dec 8, 2011 at 10:03 AM, Jay Feldblum y_feldblum@yahoo.com
wrote:But it is helpful when it comes to reusing third-party cookbooks: it
permits
writing custom roles to control and integrate with complex generic
third-party cookbooks which look at deep attributes structures to decide
what to do.Could you provide a specific example for others to follow to understand?
Bryan
You just described how the firewall and ufw cookbooks work, they
depend on merging of attributes.
Thanks,
Matt Ray
Senior Technical Evangelist | Opscode Inc.
matt@opscode.com | (512) 731-2218
Twitter, IRC, GitHub: mattray
On Thu, Dec 8, 2011 at 4:22 PM, Jay Feldblum y_feldblum@yahoo.com wrote:
Imagine a cookbook for implementing firewall rules, which inspected the node
attributes to discover the rules, and was expected to be configured via
roles. One role might want to append a rule, where each rule can include
multiple source and destination interfaces, ip-addresses, protocols, port
ranges, etc. Another node might want to append another rule, and a third
role might want to update the first rule by appending e.g. an additional
port range.Imagine the application or database cookbooks were implemented differently,
where they inspected node attributes rather than searched data bags. The
configurations for these are rather complex. Then one role might want to add
a some data, while another role might want to update it; the first role
providing blanket/default data, the second role being application-specific.Imagine fnichol's chef-rvm cookbook had default attributes that you don't
like. You want your role to add the cookbook to the run-list but delete
those default attributes, for example, delete the nested attribute that says
ruby vX should be installed by default or gem Y should be installed by
default in each ruby.
- Jay Feldblum
On Thu, Dec 8, 2011 at 10:45 AM, Bryan McLellan btm@loftninjas.org wrote:
On Thu, Dec 8, 2011 at 10:03 AM, Jay Feldblum y_feldblum@yahoo.com
wrote:But it is helpful when it comes to reusing third-party cookbooks: it
permits
writing custom roles to control and integrate with complex generic
third-party cookbooks which look at deep attributes structures to decide
what to do.Could you provide a specific example for others to follow to understand?
Bryan
On Thursday, December 8, 2011 at 9:20 AM, Matt Ray wrote:
You just described how the firewall and ufw cookbooks work, they
depend on merging of attributes.Thanks,
Matt Ray
Senior Technical Evangelist | Opscode Inc.
matt@opscode.com (mailto:matt@opscode.com) | (512) 731-2218
Twitter, IRC, GitHub: mattrayOn Thu, Dec 8, 2011 at 4:22 PM, Jay Feldblum <y_feldblum@yahoo.com (mailto:y_feldblum@yahoo.com)> wrote:
Imagine a cookbook for implementing firewall rules, which inspected the node
attributes to discover the rules, and was expected to be configured via
roles. One role might want to append a rule, where each rule can include
multiple source and destination interfaces, ip-addresses, protocols, port
ranges, etc. Another node might want to append another rule, and a third
role might want to update the first rule by appending e.g. an additional
port range.Imagine the application or database cookbooks were implemented differently,
where they inspected node attributes rather than searched data bags. The
configurations for these are rather complex. Then one role might want to add
a some data, while another role might want to update it; the first role
providing blanket/default data, the second role being application-specific.Imagine fnichol's chef-rvm cookbook had default attributes that you don't
like. You want your role to add the cookbook to the run-list but delete
those default attributes, for example, delete the nested attribute that says
ruby vX should be installed by default or gem Y should be installed by
default in each ruby.
- Jay Feldblum
First, a little background on attributes and !merge:
There are no plans to remove attribute merging. While it's unfortunate that the user interface for attributes is quite complex, the ability to customize inputs to recipes based on combinations of roles and (very infrequently, I hope) node-specific attributes is required for infrastructures of even modest scale.
Attribute merging, as implied by the name, is a generally additive process: when merging two Hashes (dicts, JSON "Objects") the keys in the result are the set union of the keys in the original Hashes. Likewise, when merging two Arrays, you get the set union.
It's easy to imagine a case where you'd want to subtract values, say, removing port 80 from an Array specifying which ports httpd should listen on. And in fact, Chef's deep merging has some support for this, by inserting special "!merge:" values in your attributes.
While one can imagine uses for this "subtractive merging," in practice no one uses it, and simply by its existence, the computational and memory requirements of the merging algorithm are increased substantially. This is a big deal when you're, say, iterating over a large set of nodes returned from a search result--a much more common use case.
The alternative existing mechanisms you could use to subtract values from attributes (modify the cookbook, use a "proxy" cookbook to forcibly set the attributes correctly) aren't very pretty either, but I think they'd at least lead you to a more declarative result, and be easier for everyone on your team to follow.
--
Dan DeLeo
Thanks for this summary and positioning statement - sounds like
there’s not much interesting in merging CHEF-2737 so I’ve closed it
for now.
I’ll file another ticket for cleaning up the existing 0.10 role
!merge: behaviour and I’ll update the deep merge doc for
clarification.
Matthew Kent | http://magoazul.com
On Monday, December 12, 2011 at 3:25 PM, Matthew Kent wrote:
I'm guessing in 0.11 you'd like the !merge subtraction removed
entirely? Only so I can mention this in the code/docs as well.
Certainly would reduce the code gymnastics in the deep merge mixin and
specs...
Yeah, we haven't talked about an explicit timeline but that's probably a good time to make it happen.
--
Dan DeLeo
--
Matthew Kent | http://magoazul.com