How to represent nested attributes


#1

Something that could maybe do with a bit of standardization. The issue is given some nested hashes, how do you indicate a specific key. The attribute declaration and the new attribute whitelists use a “/” separator, most knife command line options use a “.” and partial searches use none and require a real array. Thoughts?

–Noah


#2

On Jul 7, 2014, at 2:09 PM, Noah Kantrowitz noah@coderanger.net wrote:

Something that could maybe do with a bit of standardization. The issue is given some nested hashes, how do you indicate a specific key. The attribute declaration and the new attribute whitelists use a “/” separator, most knife command line options use a “.” and partial searches use none and require a real array. Thoughts?

Oh, and another another comes to mind, search itself uses "_"s, though in a lossy conversion.

–Noah


#3

My tooling takes attribute selectors with . separators and drills into things accordingly. Because JSON.

eg https://github.com/josephholsten/discoteq/blob/master/chef/service.go#L128

On Jul 7, 2014, at 2:09 PM, Noah Kantrowitz noah@coderanger.net wrote:

Something that could maybe do with a bit of standardization. The issue is given some nested hashes, how do you indicate a specific key. The attribute declaration and the new attribute whitelists use a “/” separator, most knife command line options use a “.” and partial searches use none and require a real array. Thoughts?

–Noah


#4

On Monday, July 7, 2014 at 2:10 PM, Noah Kantrowitz wrote:

On Jul 7, 2014, at 2:09 PM, Noah Kantrowitz <noah@coderanger.net (mailto:noah@coderanger.net)> wrote:

Something that could maybe do with a bit of standardization. The issue is given some nested hashes, how do you indicate a specific key. The attribute declaration and the new attribute whitelists use a “/” separator, most knife command line options use a “.” and partial searches use none and require a real array. Thoughts?
And ohai on the command line uses a slash I think.

Oh, and another another comes to mind, search itself uses "_"s, though in a lossy conversion.

–Noah
This annoys me to no end. I don’t really have a favorite. Both “/“ and “.” show up in actual attributes (that’ll be true for any character we pick though) so in any case we need some form of escaping or a way to specify a real array that isn’t terrible on the command line, though I think having a separator character is reasonable to make the 80+% use case less cumbersome.


Daniel DeLeo


#5

On 08/07/14 07:09, Noah Kantrowitz wrote:

Something that could maybe do with a bit of standardization. The issue is given some nested hashes, how do you indicate a specific key. The attribute declaration and the new attribute whitelists use a “/” separator, most knife command line options use a “.” and partial searches use none and require a real array. Thoughts?

I really wish this was standardized. The fact that there are at least
three ways of doing this is a real problem, and so are characters like
’.’ and ‘_’, that are frequently included in key names. My vote would
be for ‘/’ separated, or ‘[array][ish][notation]’ because that largely
does away with the conflicting separator problem, but if we could at
least be consistent within knife, that would be way less insane than
what we’re doing now.

When I wrote knife-attribute[1][2] I used the dot notation because it’s
common across all the core knife commands except search, and because
that’s what’s supported by GenericPresenter#extract_nested_value, but
#extract_nested_value doesn’t even support escaping, so if you encounter
something with periods in the key, you’re completely boned. It may be
worth my while to submit a PR to at least allow escaping there, now that
it comes to mind again.

Pete

[1] https://github.com/pdf/knife-attribute
[2] OT: It might be a good idea to have a ‘knife’ section on the
supermarket to act as a central repository for discovering knife plugins
(beyond gem search knife-).


#6

Okay, sounds like everyone wants standardization and we’re all okay with using period (“.”) as the separator. To get where we want to go, we need to get to consensus on some things:

  1. How do we handle attributes that contain the separator character? I propose that Arrays should be acceptable in Ruby DSLs, e.g.:

[ “an_attribute”, “that.contains.dots” ]

Refers to

node[“an_attribute”][“that.contains.dots”]

Any good ideas on how we can make this usable on the command line? Give up and require people to “alias” their attributes in cookbooks if they hit this case?

  1. What’s the scope, and/or what parts of the problem do we want to fix at what times? As Noah mentions, this affects search (specifying the key in your query), partial search, attribute whitelisting, and ohai filtering (command line and internal). We (Chef Software) can modify the whitelist feature[1] to use the period character since it hasn’t shipped in an official release yet. The others require either breaking changes or some kind of interface versioning. Changing search (e.g., knife search node ‘a.nested.key.in.search.query:value’) could potentially require a data migration[2]. How do we see this happening, and when?

To be clear, we’re planning to go ahead and switch whitelisting to use period and array separators since the next chef release is imminent and we don’t want to add additional back compat hassles. For everything else, we should go through the chef-rfc process (https://github.com/opscode/chef-rfc/pull/17/files). Noah, do you want to write that up?

References:

  1. https://github.com/opscode/chef/blob/master/lib/chef/whitelist.rb
  2. The format of documents stored in Solr is given https://github.com/opscode/chef-expander/blob/master/lib/chef/expander/solrizer.rb#L175-188 and
    https://github.com/opscode/chef-expander/blob/master/lib/chef/expander/flattener.rb


Daniel DeLeo

On Monday, July 7, 2014 at 11:18 PM, Peter Fern wrote:

On 08/07/14 07:09, Noah Kantrowitz wrote:

Something that could maybe do with a bit of standardization. The issue is given some nested hashes, how do you indicate a specific key. The attribute declaration and the new attribute whitelists use a “/” separator, most knife command line options use a “.” and partial searches use none and require a real array. Thoughts?

I really wish this was standardized. The fact that there are at least
three ways of doing this is a real problem, and so are characters like
’.’ and ‘_’, that are frequently included in key names. My vote would
be for ‘/’ separated, or ‘[array][ish][notation]’ because that largely
does away with the conflicting separator problem, but if we could at
least be consistent within knife, that would be way less insane than
what we’re doing now.

When I wrote knife-attribute[1][2] I used the dot notation because it’s
common across all the core knife commands except search, and because
that’s what’s supported by GenericPresenter#extract_nested_value, but
#extract_nested_value doesn’t even support escaping, so if you encounter
something with periods in the key, you’re completely boned. It may be
worth my while to submit a PR to at least allow escaping there, now that
it comes to mind again.

Pete

[1] https://github.com/pdf/knife-attribute
[2] OT: It might be a good idea to have a ‘knife’ section on the
supermarket to act as a central repository for discovering knife plugins
(beyond gem search knife-).


#7

For search, it’s not actually a big deal that the dots don’t ‘mean’ what
you think. foo.bar.baz where the actual key was ‘.bar.baz’ is the same as
not, from a full text search point of view. If it was acceptable, you could
even support both _ and . as separators in search queries.

Adam

On Fri, Jul 11, 2014 at 12:10 PM, Daniel DeLeo dan@kallistec.com wrote:

Okay, sounds like everyone wants standardization and we’re all okay with
using period (“.”) as the separator. To get where we want to go, we need to
get to consensus on some things:

  1. How do we handle attributes that contain the separator character? I
    propose that Arrays should be acceptable in Ruby DSLs, e.g.:

[ “an_attribute”, “that.contains.dots” ]

Refers to

node[“an_attribute”][“that.contains.dots”]

Any good ideas on how we can make this usable on the command line? Give up
and require people to “alias” their attributes in cookbooks if they hit
this case?

  1. What’s the scope, and/or what parts of the problem do we want to fix at
    what times? As Noah mentions, this affects search (specifying the key in
    your query), partial search, attribute whitelisting, and ohai filtering
    (command line and internal). We (Chef Software) can modify the whitelist
    feature[1] to use the period character since it hasn’t shipped in an
    official release yet. The others require either breaking changes or some
    kind of interface versioning. Changing search (e.g., knife search node ‘a.nested.key.in.search.query:value’) could potentially require a data
    migration[2]. How do we see this happening, and when?

To be clear, we’re planning to go ahead and switch whitelisting to use
period and array separators since the next chef release is imminent and we
don’t want to add additional back compat hassles. For everything else, we
should go through the chef-rfc process (
https://github.com/opscode/chef-rfc/pull/17/files). Noah, do you want to
write that up?

References:

  1. https://github.com/opscode/chef/blob/master/lib/chef/whitelist.rb
  2. The format of documents stored in Solr is given
    https://github.com/opscode/chef-expander/blob/master/lib/chef/expander/solrizer.rb#L175-188
    and

https://github.com/opscode/chef-expander/blob/master/lib/chef/expander/flattener.rb


Daniel DeLeo

On Monday, July 7, 2014 at 11:18 PM, Peter Fern wrote:

On 08/07/14 07:09, Noah Kantrowitz wrote:

Something that could maybe do with a bit of standardization. The issue
is given some nested hashes, how do you indicate a specific key. The
attribute declaration and the new attribute whitelists use a “/” separator,
most knife command line options use a “.” and partial searches use none and
require a real array. Thoughts?

I really wish this was standardized. The fact that there are at least
three ways of doing this is a real problem, and so are characters like
’.’ and ‘_’, that are frequently included in key names. My vote would
be for ‘/’ separated, or ‘[array][ish][notation]’ because that largely
does away with the conflicting separator problem, but if we could at
least be consistent within knife, that would be way less insane than
what we’re doing now.

When I wrote knife-attribute[1][2] I used the dot notation because it’s
common across all the core knife commands except search, and because
that’s what’s supported by GenericPresenter#extract_nested_value, but
#extract_nested_value doesn’t even support escaping, so if you encounter
something with periods in the key, you’re completely boned. It may be
worth my while to submit a PR to at least allow escaping there, now that
it comes to mind again.

Pete

[1] https://github.com/pdf/knife-attribute
[2] OT: It might be a good idea to have a ‘knife’ section on the
supermarket to act as a central repository for discovering knife plugins
(beyond gem search knife-).