Dialect support and loading enhancements


#1

At long last I’ve pushed up a completed-enough-to-use version of my dialects proposal. I won’t repeat the whole document here, but some highlights:

  1. Hooks to load Chef recipes and attribute files from formats other than .rb
  2. JSON and Yaml dialects for attribute files.
  3. Root shortcuts for cookbook source files of which you commonly have only one.
    cookbooks/
    apache2/
    attributes.rb
    recipe.rb
    library.rb
  4. Removal of the need for the default/ path segment under files/ and templates/.
  5. Ability to pass an array to #source on cookbook_file and template to define your own lookup path similar to the current implicit one.

Personally I think all of these put together are an order of magnitude improvement in ease of learning Chef and writing new cookbooks while not losing any of the expressive power you want on the high end. This branch should present no backwards incompatibilities, though I think it should kick off a discussion separately about deprecating the implicit search path on files/ and templates/ in favor of people using the new explicit search path syntax for the rare occasions where such things are desired (and even in those cases, it is generally just a few of the options in the current search path). Any such deprecation would have to be over a very long time though. I have opened https://tickets.opscode.com/browse/CHEF-4559 to track this patch. If anyone has any feedback please don’t hesitate to shoot me an email or comment on the pull request/ticket.

–Noah


#2

On 2013-09-19, at 20:37, Noah Kantrowitz noah@coderanger.net wrote:

https://github.com/opscode/chef/pull/997

At long last I’ve pushed up a completed-enough-to-use version of my dialects proposal. I won’t repeat the whole document here, but some highlights:

  1. Hooks to load Chef recipes and attribute files from formats other than .rb
  2. JSON and Yaml dialects for attribute files.
  3. Root shortcuts for cookbook source files of which you commonly have only one.
    cookbooks/
    apache2/
    attributes.rb
    recipe.rb
    library.rb
  4. Removal of the need for the default/ path segment under files/ and templates/.
  5. Ability to pass an array to #source on cookbook_file and template to define your own lookup path similar to the current implicit one.

Personally I think all of these put together are an order of magnitude improvement in ease of learning Chef and writing new cookbooks while not losing any of the expressive power you want on the high end. This branch should present no backwards incompatibilities, though I think it should kick off a discussion separately about deprecating the implicit search path on files/ and templates/ in favor of people using the new explicit search path syntax for the rare occasions where such things are desired (and even in those cases, it is generally just a few of the options in the current search path). Any such deprecation would have to be over a very long time though. I have opened https://tickets.opscode.com/browse/CHEF-4559 to track this patch. If anyone has any feedback please don’t hesitate to shoot me an email or comment on the pull request/ticket.

So much <3.

For those interested in how alternate formats can work well, here’s the doc for saltstack’s pydsl (closest to chef dsl today):
http://docs.saltstack.com/ref/renderers/all/salt.renderers.pydsl.html
compared to their jinja + yaml format (similar to mustache + yaml from the proposal):
http://docs.saltstack.com/ref/renderers/all/salt.renderers.jinja.html

I’m not sure we could ever write libraries or lwrps in non-ruby, but I definitely support it in recipes, attributes, metadata & templates.

~j


#3

On Sep 19, 2013, at 3:57 PM, Joseph Holsten joseph@josephholsten.com wrote:

If anyone has any feedback please don’t hesitate to shoot me an email or comment on the pull request/ticket.

So much <3.

Same here!

For those interested in how alternate formats can work well, here’s the doc for saltstack’s pydsl (closest to chef dsl today):
http://docs.saltstack.com/ref/renderers/all/salt.renderers.pydsl.html
compared to their jinja + yaml format (similar to mustache + yaml from the proposal):
http://docs.saltstack.com/ref/renderers/all/salt.renderers.jinja.html

I’ve been looking at saltstack, as well as ansible, and I’m really starting to like jinja2+yaml. As another point of comparison, here’s the main ansible page for their playbooks: http://ansibleworks.com/docs/playbooks.html.

I’m not sure we could ever write libraries or lwrps in non-ruby, but I definitely support it in recipes, attributes, metadata & templates.

That, plus collapsing unnecessary directories – I see so much winnage.

In comparison to Puppet, I like that Chef has such a complete REST API that is so easy for our programmers to learn. In comparison to ansible, I like the fact that the REST API with Chef comes out-of-the-box, instead of requiring that you download their commercial AWX tool (even if it is free for managing up to ten hosts).

As we know, tools like Omnibus make it so much easier to manage the tools you’re using to manage the rest of the infrastructure, and that lesson seems to be sorely lacking with regards to the other systems I’ve been looking at lately.

Oh, and then there is support for a wide variety of platforms.

If we can simplify Chef enough to make it easier for non-programmers to learn, but still maintain the underlying power, I think the company and the community will be in an excellent position.


Brad Knowles brad@shub-internet.org
LinkedIn Profile: http://tinyurl.com/y8kpxu


#4

First off, thanks for thinking this through so well, and having the patch
to match. Awesome.

I have a couple of questions/statements/observations:

  1. Do other people actually think the directory structure of a cookbook is
    the biggest barrier to learning? This confuses the crap out of me - but I’m
    the one that inflicted the directory structure on you, so perhaps I’m an
    awful judge.

  2. It feels like this could be accepted in pieces, depending on how we all
    feel about it. For example, 3 and 4/5 are to separate features, and how I
    feel about them is totally not influenced by how I feel about dialects. It
    makes sense to me that 4/5 should be adopted - we cause more confusion with
    the single copy pattern than we do delight, and nobody looses out - it’s
    easy enough to implement it if it turns out you need it.

  3. My concern about dialects is really that, in the end, you wind up
    hitting a wall. In the grand tradition of manipulexity and whipuptude, we
    know there is a path that can provide both (the ruby dsl), but that we
    think there are some barriers to it’s full whipuptidunal potential (too
    many directories, etc.) Is it really simpler to learn the syntax from yaml,
    and all the gotchas that might be implied there, than to learn the basic
    syntax that you know scales moving forward? I think ansible and salt both
    provide a delightful experience of immediacy to the consumer, and that
    experience is a great one to mimic - but I’m not sure I think that path is
    because you’re writing YAML rather than ruby. Is this simply because I know
    more than one programming language and have no inherent fear of learning
    another one anymore?

Best,
Adam

On Thu, Sep 19, 2013 at 1:37 PM, Noah Kantrowitz noah@coderanger.netwrote:

https://github.com/opscode/chef/pull/997

At long last I’ve pushed up a completed-enough-to-use version of my
dialects proposal. I won’t repeat
the whole document here, but some highlights:

  1. Hooks to load Chef recipes and attribute files from formats other than
    .rb
  2. JSON and Yaml dialects for attribute files.
  3. Root shortcuts for cookbook source files of which you commonly have
    only one.
    cookbooks/
    apache2/
    attributes.rb
    recipe.rb
    library.rb
  4. Removal of the need for the default/ path segment under files/ and
    templates/.
  5. Ability to pass an array to #source on cookbook_file and template to
    define your own lookup path similar to the current implicit one.

Personally I think all of these put together are an order of magnitude
improvement in ease of learning Chef and writing new cookbooks while not
losing any of the expressive power you want on the high end. This branch
should present no backwards incompatibilities, though I think it should
kick off a discussion separately about deprecating the implicit search path
on files/ and templates/ in favor of people using the new explicit search
path syntax for the rare occasions where such things are desired (and even
in those cases, it is generally just a few of the options in the current
search path). Any such deprecation would have to be over a very long time
though. I have opened https://tickets.opscode.com/browse/CHEF-4559 to
track this patch. If anyone has any feedback please don’t hesitate to shoot
me an email or comment on the pull request/ticket.

–Noah


Opscode, Inc.
Adam Jacob, Chief Dev Officer
T: (206) 619-7151 E: adam@opscode.com


#5

Hi,

Overall, I’m +1 on 3, 4, and 5. I’m unsure on 1 and 2.

  1. Do other people actually think the directory structure of a cookbook is the biggest barrier to learning? This confuses the crap out of me - but I’m the one that inflicted the directory structure on you, so perhaps I’m an awful judge.

It certainly isn’t the biggest barrier to learning. However, I have
helped more than a few people who were confused by this. Also, in my
own cookbook writing, I’m using default 90% of the time. Given how
rarely I use directories other than default, having the ability to
forgo it entirely and handle the other 10% of the cases via passing an
array to the source attribute would be nice.

Cheers,

Steven

On Thu, Sep 19, 2013 at 3:54 PM, Adam Jacob adam@opscode.com wrote:

First off, thanks for thinking this through so well, and having the patch to
match. Awesome.

I have a couple of questions/statements/observations:

  1. Do other people actually think the directory structure of a cookbook is
    the biggest barrier to learning? This confuses the crap out of me - but I’m
    the one that inflicted the directory structure on you, so perhaps I’m an
    awful judge.

  2. It feels like this could be accepted in pieces, depending on how we all
    feel about it. For example, 3 and 4/5 are to separate features, and how I
    feel about them is totally not influenced by how I feel about dialects. It
    makes sense to me that 4/5 should be adopted - we cause more confusion with
    the single copy pattern than we do delight, and nobody looses out - it’s
    easy enough to implement it if it turns out you need it.

  3. My concern about dialects is really that, in the end, you wind up hitting
    a wall. In the grand tradition of manipulexity and whipuptude, we know there
    is a path that can provide both (the ruby dsl), but that we think there are
    some barriers to it’s full whipuptidunal potential (too many directories,
    etc.) Is it really simpler to learn the syntax from yaml, and all the
    gotchas that might be implied there, than to learn the basic syntax that you
    know scales moving forward? I think ansible and salt both provide a
    delightful experience of immediacy to the consumer, and that experience is a
    great one to mimic - but I’m not sure I think that path is because you’re
    writing YAML rather than ruby. Is this simply because I know more than one
    programming language and have no inherent fear of learning another one
    anymore?

Best,
Adam

On Thu, Sep 19, 2013 at 1:37 PM, Noah Kantrowitz noah@coderanger.net
wrote:

https://github.com/opscode/chef/pull/997

At long last I’ve pushed up a completed-enough-to-use version of my
dialects
proposal
. I won’t
repeat the whole document here, but some highlights:

  1. Hooks to load Chef recipes and attribute files from formats other than
    .rb
  2. JSON and Yaml dialects for attribute files.
  3. Root shortcuts for cookbook source files of which you commonly have
    only one.
    cookbooks/
    apache2/
    attributes.rb
    recipe.rb
    library.rb
  4. Removal of the need for the default/ path segment under files/ and
    templates/.
  5. Ability to pass an array to #source on cookbook_file and template to
    define your own lookup path similar to the current implicit one.

Personally I think all of these put together are an order of magnitude
improvement in ease of learning Chef and writing new cookbooks while not
losing any of the expressive power you want on the high end. This branch
should present no backwards incompatibilities, though I think it should kick
off a discussion separately about deprecating the implicit search path on
files/ and templates/ in favor of people using the new explicit search path
syntax for the rare occasions where such things are desired (and even in
those cases, it is generally just a few of the options in the current search
path). Any such deprecation would have to be over a very long time though. I
have opened https://tickets.opscode.com/browse/CHEF-4559 to track this
patch. If anyone has any feedback please don’t hesitate to shoot me an email
or comment on the pull request/ticket.

–Noah


Opscode, Inc.
Adam Jacob, Chief Dev Officer
T: (206) 619-7151 E: adam@opscode.com


#6

On Sep 19, 2013, at 3:54 PM, Adam Jacob adam@opscode.com wrote:

First off, thanks for thinking this through so well, and having the patch to match. Awesome.

I have a couple of questions/statements/observations:

  1. Do other people actually think the directory structure of a cookbook is the biggest barrier to learning? This confuses the crap out of me - but I’m the one that inflicted the directory structure on you, so perhaps I’m an awful judge.

Not the biggest by far, but every time we can avoid explaining why “default.rb” is a special filename I think that saves a little bit of cognitive overhead and ditto on “just put it in default/” on file/templates.

  1. It feels like this could be accepted in pieces, depending on how we all feel about it. For example, 3 and 4/5 are to separate features, and how I feel about them is totally not influenced by how I feel about dialects. It makes sense to me that 4/5 should be adopted - we cause more confusion with the single copy pattern than we do delight, and nobody looses out - it’s easy enough to implement it if it turns out you need it.

Definitely, the different bits were just implemented sequentially, would be pretty straightforward to break the patch up more if people feel there is need.

  1. My concern about dialects is really that, in the end, you wind up hitting a wall. In the grand tradition of manipulexity and whipuptude, we know there is a path that can provide both (the ruby dsl), but that we think there are some barriers to it’s full whipuptidunal potential (too many directories, etc.) Is it really simpler to learn the syntax from yaml, and all the gotchas that might be implied there, than to learn the basic syntax that you know scales moving forward? I think ansible and salt both provide a delightful experience of immediacy to the consumer, and that experience is a great one to mimic - but I’m not sure I think that path is because you’re writing YAML rather than ruby. Is this simply because I know more than one programming language and have no inherent fear of learning another one anymore?

My concern isn’t fear of learning, but just reducing barriers to entry. Anyone that has talked to me about cookbooks for more than 15 minutes has heard my diatribe about writing everything as resources/providers etc etc, which are definitely not something that can be (easily) done in anything but Ruby, so 100% on board with “learn Ruby” being a core goal if someone is going to stick around the Chef community for a while. The counter to this is that the amount you need to learn to get started with Chef if you aren’t a Ruby dev is pretty high and I think the dialects system can provide more intermediary stepping stones on a power/complexity slope.

–Noah


#7

On Sep 19, 2013, at 4:43 PM, Noah Kantrowitz noah@coderanger.net wrote:

On Sep 19, 2013, at 3:54 PM, Adam Jacob adam@opscode.com wrote:

First off, thanks for thinking this through so well, and having the patch to match. Awesome.

I have a couple of questions/statements/observations:

  1. Do other people actually think the directory structure of a cookbook is the biggest barrier to learning? This confuses the crap out of me - but I’m the one that inflicted the directory structure on you, so perhaps I’m an awful judge.

Not the biggest by far, but every time we can avoid explaining why “default.rb” is a special filename I think that saves a little bit of cognitive overhead and ditto on “just put it in default/” on file/templates.

  1. It feels like this could be accepted in pieces, depending on how we all feel about it. For example, 3 and 4/5 are to separate features, and how I feel about them is totally not influenced by how I feel about dialects. It makes sense to me that 4/5 should be adopted - we cause more confusion with the single copy pattern than we do delight, and nobody looses out - it’s easy enough to implement it if it turns out you need it.

Definitely, the different bits were just implemented sequentially, would be pretty straightforward to break the patch up more if people feel there is need.

  1. My concern about dialects is really that, in the end, you wind up hitting a wall. In the grand tradition of manipulexity and whipuptude, we know there is a path that can provide both (the ruby dsl), but that we think there are some barriers to it’s full whipuptidunal potential (too many directories, etc.) Is it really simpler to learn the syntax from yaml, and all the gotchas that might be implied there, than to learn the basic syntax that you know scales moving forward? I think ansible and salt both provide a delightful experience of immediacy to the consumer, and that experience is a great one to mimic - but I’m not sure I think that path is because you’re writing YAML rather than ruby. Is this simply because I know more than one programming language and have no inherent fear of learning another one anymore?

My concern isn’t fear of learning, but just reducing barriers to entry. Anyone that has talked to me about cookbooks for more than 15 minutes has heard my diatribe about writing everything as resources/providers etc etc, which are definitely not something that can be (easily) done in anything but Ruby, so 100% on board with “learn Ruby” being a core goal if someone is going to stick around the Chef community for a while. The counter to this is that the amount you need to learn to get started with Chef if you aren’t a Ruby dev is pretty high and I think the dialects system can provide more intermediary stepping stones on a power/complexity slope.

Just to expand on this a bit further, I think we will have to have some care to make sure the dialects “line up”. By that I mean that we shouldn’t (only) go and implement all of Salt or Puppet as a dialect, we want to make sure that skills and knowledge in one dialect carry over as you move up the curve.

As an example, most cookbooks have pretty simple, declarative attributes files with just a few default values. For these the yaml format works very nicely and cuts down on syntax to learn. When the day comes that you want to add an if/else or a call to value_for_platform to your attributes you do have learn some new syntax as you transcribe to Ruby, but all the core knowledge about attributes, precedence levels, etc all transfers over smoothly. As long as we can find these same kind of mappings, I think the dialects system has value as an intermediary learning tool.

I’m sure that other kinds of dialects will exist, and in fact I think a dialect that can directly consume puppet manifests and just calls the puppet gem internally would be both hilarious and probably useful to people doing migrations, and thats fine as long as thats not how we start people off in the community.

–Noah


#8

I don’t know; what little I understand about language learning says it only
holds for dialects you know. If you struggle to learn English, I don’t
propose you switch to French as a simpler glide path.

If you know of French, you may well find places where expressing yourself
that way makes everything easier. What I’m most not sold on here is that we
are better off doing dialects than really tuning the dsl, and dropping
other cognitive barriers.

I know it’s not zero sum - we can do both. But as soon as the recipe
dialects ship in chef, it’s going to need to feel first class: if it
doesn’t, it will never gain traction.

If you come in alone, starting with YAML for recipe syntax is a mistake -
you need your baby language skills to grow with you, and the complexity of
the holy trinity is not high in chef. How much of your simplification do
you really think comes from dialects, and how much from cookbook structure
simplification?

Adam
On Sep 19, 2013 5:03 PM, “Noah Kantrowitz” noah@coderanger.net wrote:

On Sep 19, 2013, at 4:43 PM, Noah Kantrowitz noah@coderanger.net wrote:

On Sep 19, 2013, at 3:54 PM, Adam Jacob adam@opscode.com wrote:

First off, thanks for thinking this through so well, and having the
patch to match. Awesome.

I have a couple of questions/statements/observations:

  1. Do other people actually think the directory structure of a cookbook
    is the biggest barrier to learning? This confuses the crap out of me - but
    I’m the one that inflicted the directory structure on you, so perhaps I’m
    an awful judge.

Not the biggest by far, but every time we can avoid explaining why
"default.rb" is a special filename I think that saves a little bit of
cognitive overhead and ditto on “just put it in default/” on file/templates.

  1. It feels like this could be accepted in pieces, depending on how we
    all feel about it. For example, 3 and 4/5 are to separate features, and how
    I feel about them is totally not influenced by how I feel about dialects.
    It makes sense to me that 4/5 should be adopted - we cause more confusion
    with the single copy pattern than we do delight, and nobody looses out -
    it’s easy enough to implement it if it turns out you need it.

Definitely, the different bits were just implemented sequentially, would
be pretty straightforward to break the patch up more if people feel there
is need.

  1. My concern about dialects is really that, in the end, you wind up
    hitting a wall. In the grand tradition of manipulexity and whipuptude, we
    know there is a path that can provide both (the ruby dsl), but that we
    think there are some barriers to it’s full whipuptidunal potential (too
    many directories, etc.) Is it really simpler to learn the syntax from yaml,
    and all the gotchas that might be implied there, than to learn the basic
    syntax that you know scales moving forward? I think ansible and salt both
    provide a delightful experience of immediacy to the consumer, and that
    experience is a great one to mimic - but I’m not sure I think that path is
    because you’re writing YAML rather than ruby. Is this simply because I know
    more than one programming language and have no inherent fear of learning
    another one anymore?

My concern isn’t fear of learning, but just reducing barriers to entry.
Anyone that has talked to me about cookbooks for more than 15 minutes has
heard my diatribe about writing everything as resources/providers etc etc,
which are definitely not something that can be (easily) done in anything
but Ruby, so 100% on board with “learn Ruby” being a core goal if someone
is going to stick around the Chef community for a while. The counter to
this is that the amount you need to learn to get started with Chef if you
aren’t a Ruby dev is pretty high and I think the dialects system can
provide more intermediary stepping stones on a power/complexity slope.

Just to expand on this a bit further, I think we will have to have some
care to make sure the dialects “line up”. By that I mean that we shouldn’t
(only) go and implement all of Salt or Puppet as a dialect, we want to make
sure that skills and knowledge in one dialect carry over as you move up the
curve.

As an example, most cookbooks have pretty simple, declarative attributes
files with just a few default values. For these the yaml format works very
nicely and cuts down on syntax to learn. When the day comes that you want
to add an if/else or a call to value_for_platform to your attributes you do
have learn some new syntax as you transcribe to Ruby, but all the core
knowledge about attributes, precedence levels, etc all transfers over
smoothly. As long as we can find these same kind of mappings, I think the
dialects system has value as an intermediary learning tool.

I’m sure that other kinds of dialects will exist, and in fact I think a
dialect that can directly consume puppet manifests and just calls the
puppet gem internally would be both hilarious and probably useful to people
doing migrations, and thats fine as long as thats not how we start people
off in the community.

–Noah


#9

On Sep 19, 2013, at 5:54 PM, Adam Jacob adam@opscode.com wrote:

  1. My concern about dialects is really that, in the end, you wind up hitting a wall. In the grand tradition of manipulexity and whipuptude, we know there is a path that can provide both (the ruby dsl), but that we think there are some barriers to it’s full whipuptidunal potential (too many directories, etc.) Is it really simpler to learn the syntax from yaml, and all the gotchas that might be implied there, than to learn the basic syntax that you know scales moving forward? I think ansible and salt both provide a delightful experience of immediacy to the consumer, and that experience is a great one to mimic - but I’m not sure I think that path is because you’re writing YAML rather than ruby. Is this simply because I know more than one programming language and have no inherent fear of learning another one anymore?

I’m not a programmer/developer. I may have encountered a variety of languages, but I’ve never really been a programmer/developer. I’m an ops guy who can do a little light scripting.

At this stage, I think you have pretty much reached the entire audience of people who are already programmers (and especially already Ruby programmers), and the next raft of customers are going to be more and more like me – largely self-taught operations people. Sure, you can give them reference material to help guide them through Ruby enough to do a little simple Chef, but that’s like giving someone a simplified guide “Just enough French to survive in Paris”.

Either are okay for tourists who will be there for a few days, but the people who are going to be living there long-term really need to properly learn French – or, in our case, Ruby. And I’m sorry, I’m just not a natural linguist – I tried to learn French during the seven-plus years I lived in Belgium, and it literally went in one ear and out the other. Ruby is and will always be an alien thing for me. For that matter, the same is true for Python, or C, or Cobol, or any of the other programming languages I’ve been exposed to over the thirty-plus years I’ve been mucking about with computers. The closest I can come to any native computer language is Bourne shell (not bash), and even that’s just due to relatively intense exposure over a long period of time – since 1984.

If we can change the rules of the game so that people don’t have to learn Ruby to handle the easy 80% of the work, then they can deal in something more closely resembling a language they can understand, and leave the hard 20% stuff for someone who has the necessary talents and aptitudes.

The other analogy I would make is to music – just because you give someone a musical instrument and you force them through the torture of music lessons for a few years, doesn’t necessarily mean that they will be a talented musician that should perform in front of a crowd of thousands.

So, yes – to me, YAML is a game changer. Even if that’s not your “native” language for the system, it’s close enough to English (and other “natural” languages) that people can get up to speed much faster with it, and they can be more proficient and retain more proficiency over time.


Brad Knowles brad@shub-internet.org
LinkedIn Profile: http://tinyurl.com/y8kpxu


#10

On Sep 19, 2013, at 5:16 PM, Adam Jacob adam@opscode.com wrote:

I don’t know; what little I understand about language learning says it only holds for dialects you know. If you struggle to learn English, I don’t propose you switch to French as a simpler glide path.

If you know of French, you may well find places where expressing yourself that way makes everything easier. What I’m most not sold on here is that we are better off doing dialects than really tuning the dsl, and dropping other cognitive barriers.

We could definitely improve the various DSLs, though even with that a side benefit of this structure is that it can be extended to roles/data bags and hooked in to knife upload, as that currently can’t handle non-JSON formats. Even if we remove recipes from the dialect system I think that part is worth keeping.

I know it’s not zero sum - we can do both. But as soon as the recipe dialects ship in chef, it’s going to need to feel first class: if it doesn’t, it will never gain traction.

I’ve specifically punted on having anything but Ruby for a recipe dialect so the community can sort out if its a good idea and what kinds of things it should be used for. I will definitely migrate my Javascript and Python proto-dialects from chef-funnel.

If you come in alone, starting with YAML for recipe syntax is a mistake - you need your baby language skills to grow with you, and the complexity of the holy trinity is not high in chef. How much of your simplification do you really think comes from dialects, and how much from cookbook structure simplification?

I think you underestimate the perceived barrier of entry for the Python community and other similar places. These are not “baby language skills”, these are very good programmers that just need to be led along a path and Python syntax helps them absorb the ideas behind resources and idempotency and all the good Chef-y core concepts. I don’t think people are avoiding Chef because they can’t learn Ruby, its really not that hard, but it adds considerably to the feeling of difficulty.

–Noah


#11

On Sep 19, 2013, at 7:16 PM, Adam Jacob adam@opscode.com wrote:

I don’t know; what little I understand about language learning says it only holds for dialects you know. If you struggle to learn English, I don’t propose you switch to French as a simpler glide path.

Some languages are simply easier than others. My wife and I lived almost eight years in Belgium, and despite multiple attempts to the contrary, I never managed more than just a smattering of French – just enough to get me in trouble with taxi drivers. In contrast, my wife and I spent a sum total of about three weeks in Italy (spread over multiple occasions), and to this day I still swear that I learned and used more Italian in those three weeks than I used French in almost eight years in Belgium.

For me, Italian is relatively easy. German is relatively easy. Dutch and Flemish sound like a cat hacking up a hairball, but they’re close enough to German and English that what I’ve learned of them came fairly easy. What little I know of Spanish, came pretty easily. But French? French was one of the hardest things I ever tried to learn.

If you come in alone, starting with YAML for recipe syntax is a mistake - you need your baby language skills to grow with you, and the complexity of the holy trinity is not high in chef. How much of your simplification do you really think comes from dialects, and how much from cookbook structure simplification?

Cookbook structure is one thing that I think would be relatively easy to simplify and would positively impact almost everyone.

YAML support would not be of much help to anyone who is already a knowledgeable and talented programmer, whether they know Ruby or not. But for those of us who are NOT talented in that area, YAML support could be a godsend.


Brad Knowles brad@shub-internet.org
LinkedIn Profile: http://tinyurl.com/y8kpxu


#12

On 9/19/13 3:54 PM, Adam Jacob wrote:

  1. Do other people actually think the directory structure of a
    cookbook is the biggest barrier to learning? This confuses the crap
    out of me - but I’m the one that inflicted the directory structure on
    you, so perhaps I’m an awful judge.

I made another proposal to deprecate file specificity and deprecate the
templates directory as part of unifying/simplifying the file
providers… Its complexity that doesn’t often get used and is probably
better moved into the recipe. I’ve gotten tired of doing “mkdir
templates/default” and “cd templates/default” all the time when i never
use file specificity… The sharks may be circling this one…


#13

We have not even scratched the surface of people who are already
programmers, from an opscode customer base point of view. Of course, Chef
isn’t about Opscode, not really - but god bless you for thinking about it.
:slight_smile:

My concern is you won’t get 80% of what you want out of it - you’re going
to get 20% of it. What’s capable with a dialect vs the native ruby is
hugely different - and it’s going to drift naturally. I think Noah is smart
to say that the low risk choice here is something like YAML - it’s
essentially just serialization, so it might be able to stretch relatively
naturally with a serialized resource catalog. But if you compare it to what
you will experience with a tool for whom the template+YAML approach is
truly first class, the delight won’t be the same.

It’s interesting to hear you say you feel like you don’t know how to
program, when i’ve seen you be an effective member of our community for a
long time now. Is it completely out of school to say that perhaps you’re
arguing for your limitations? You might not be a CS major - lord knows, I’m
not. But you do know how to program, and you’re doing it in ruby. :slight_smile: The
closest you’ve come to a real programming language is Ruby: you’re
programming in it, and have been the entire time you’ve rocked the Chef, an
you’ve done so successfully.

My concern really boil down to this: I don’t want to say that there will be
a simpler path that turns out to just be a ghetto - an underserved, less
loved corner of Chef. I think we’ve made similar mistakes in the past, and
they’ve hurt us (for example, the unintentional impacts Opscode’s focus has
had on the chef solo experience being improved upon) as a community.

Having someone like Noah who is committed to seeing it through goes a long
way to solving that concern, in that I totally believe he would be an
active maintainer of that functionality.

Thinking out loud: what if we wrapped it in a feature flag and marked it
experimental?

Adam

On Thu, Sep 19, 2013 at 5:25 PM, Brad Knowles brad@shub-internet.orgwrote:

On Sep 19, 2013, at 5:54 PM, Adam Jacob adam@opscode.com wrote:

  1. My concern about dialects is really that, in the end, you wind up
    hitting a wall. In the grand tradition of manipulexity and whipuptude, we
    know there is a path that can provide both (the ruby dsl), but that we
    think there are some barriers to it’s full whipuptidunal potential (too
    many directories, etc.) Is it really simpler to learn the syntax from yaml,
    and all the gotchas that might be implied there, than to learn the basic
    syntax that you know scales moving forward? I think ansible and salt both
    provide a delightful experience of immediacy to the consumer, and that
    experience is a great one to mimic - but I’m not sure I think that path is
    because you’re writing YAML rather than ruby. Is this simply because I know
    more than one programming language and have no inherent fear of learning
    another one anymore?

I’m not a programmer/developer. I may have encountered a variety of
languages, but I’ve never really been a programmer/developer. I’m an ops
guy who can do a little light scripting.

At this stage, I think you have pretty much reached the entire audience of
people who are already programmers (and especially already Ruby
programmers), and the next raft of customers are going to be more and more
like me – largely self-taught operations people. Sure, you can give them
reference material to help guide them through Ruby enough to do a little
simple Chef, but that’s like giving someone a simplified guide “Just enough
French to survive in Paris”.

Either are okay for tourists who will be there for a few days, but the
people who are going to be living there long-term really need to properly
learn French – or, in our case, Ruby. And I’m sorry, I’m just not a
natural linguist – I tried to learn French during the seven-plus years I
lived in Belgium, and it literally went in one ear and out the other. Ruby
is and will always be an alien thing for me. For that matter, the same is
true for Python, or C, or Cobol, or any of the other programming languages
I’ve been exposed to over the thirty-plus years I’ve been mucking about
with computers. The closest I can come to any native computer language is
Bourne shell (not bash), and even that’s just due to relatively intense
exposure over a long period of time – since 1984.

If we can change the rules of the game so that people don’t have to learn
Ruby to handle the easy 80% of the work, then they can deal in something
more closely resembling a language they can understand, and leave the hard
20% stuff for someone who has the necessary talents and aptitudes.

The other analogy I would make is to music – just because you give
someone a musical instrument and you force them through the torture of
music lessons for a few years, doesn’t necessarily mean that they will be a
talented musician that should perform in front of a crowd of thousands.

So, yes – to me, YAML is a game changer. Even if that’s not your
"native" language for the system, it’s close enough to English (and other
"natural" languages) that people can get up to speed much faster with it,
and they can be more proficient and retain more proficiency over time.


Brad Knowles brad@shub-internet.org
LinkedIn Profile: http://tinyurl.com/y8kpxu


Opscode, Inc.
Adam Jacob, Chief Dev Officer
T: (206) 619-7151 E: adam@opscode.com


#14

On Sep 19, 2013, at 7:39 PM, Adam Jacob adam@opscode.com wrote:

We have not even scratched the surface of people who are already programmers, from an opscode customer base point of view. Of course, Chef isn’t about Opscode, not really - but god bless you for thinking about it. :slight_smile:

My worry isn’t about customers per se, honestly its pretty selfish. I like Chef, I want to write my fix-the-world stuffs in or with Chef, and as such I need to get more people (in my case a lot of Python people) to use Chef so that I don’t have to go duplicate my work in Ansible or whatever is The Thing at that moment.

My concern is you won’t get 80% of what you want out of it - you’re going to get 20% of it. What’s capable with a dialect vs the native ruby is hugely different - and it’s going to drift naturally. I think Noah is smart to say that the low risk choice here is something like YAML - it’s essentially just serialization, so it might be able to stretch relatively naturally with a serialized resource catalog. But if you compare it to what you will experience with a tool for whom the template+YAML approach is truly first class, the delight won’t be the same.

I just want to be clear, I haven’t said that. I added JSON and Yaml for attributes files, not recipes. I don’t think Yaml for recipes belongs in core for exactly the reasons I mentioned re: difficulty translating knowledge. I could certainly imagine a mapping, but needing non-declarative stuff in recipes is pretty close to a given, whereas in attr files it only crops up in very advanced cookbooks, usually in service of cross-platform unification. I do think the JS and Python dialects will be able to maintain a good portion of the expressive power of the Ruby DSL, but I don’t think those belong in core just due to complexity of maintenance and general long road to stability (you can currently segfault Ruby from your Python code if you try and recursively load libffi for example). If a fully declarative (JSON/Yaml) dialect is added for recipes I think the target wouldn’t be humans, but auto generated recipe code. I don’t see why you would do that though, so I left it out. Other than that it would only work well for suuuuuuper simple cases, which do happen, but I don’t have a clear picture of how thats evolving. I could certainly see a future where more and more stuff moves in to LWRPs and app-ish cookbooks really could be fully declarative invocations of those LWRPs, but we are a long way from that.

It’s interesting to hear you say you feel like you don’t know how to program, when i’ve seen you be an effective member of our community for a long time now. Is it completely out of school to say that perhaps you’re arguing for your limitations? You might not be a CS major - lord knows, I’m not. But you do know how to program, and you’re doing it in ruby. :slight_smile: The closest you’ve come to a real programming language is Ruby: you’re programming in it, and have been the entire time you’ve rocked the Chef, an you’ve done so successfully.

My concern really boil down to this: I don’t want to say that there will be a simpler path that turns out to just be a ghetto - an underserved, less loved corner of Chef. I think we’ve made similar mistakes in the past, and they’ve hurt us (for example, the unintentional impacts Opscode’s focus has had on the chef solo experience being improved upon) as a community.

Agreed, it would mean a lot of documentation work to ensure mobility between different formats. In my head I’m imagining something like how multi-lang SDK docs work sometimes with a format radio button at the top of the page which switches things within the page so you can see how the syntaxes interact, but maybe that is overkill.

Having someone like Noah who is committed to seeing it through goes a long way to solving that concern, in that I totally believe he would be an active maintainer of that functionality.

Thinking out loud: what if we wrapped it in a feature flag and marked it experimental?

I think because its already opt-in and there is no change to the Ruby formats this might not accomplish much. The only way a new user will find out this exists is via docs and blog posts and the like, and I think the “here be dragons” would belong at that level, rather than someone trying to use the new feature and either getting a message of “please pass --dialects=yaml,py to continue” (which is one of those just-do-what-I-mean rage inducing situations) or silent failure.

–Noah


#15

I think it’s clear the sharks have one. I haven’t used it in years.

On Thu, Sep 19, 2013 at 5:54 PM, Lamont Granquist lamont@opscode.comwrote:

On 9/19/13 3:54 PM, Adam Jacob wrote:

  1. Do other people actually think the directory structure of a cookbook
    is the biggest barrier to learning? This confuses the crap out of me - but
    I’m the one that inflicted the directory structure on you, so perhaps I’m
    an awful judge.

I made another proposal to deprecate file specificity and deprecate the
templates directory as part of unifying/simplifying the file providers…
Its complexity that doesn’t often get used and is probably better moved
into the recipe. I’ve gotten tired of doing “mkdir templates/default” and
"cd templates/default" all the time when i never use file specificity…
The sharks may be circling this one…


Opscode, Inc.
Adam Jacob, Chief Dev Officer
T: (206) 619-7151 E: adam@opscode.com


#16

To me Chef’s biggest power was its flexibility; i can easily monkey
patch/wrap whatever cookbook/internal chef behavior, and i’m not the only
doing this. just look at the number of cookbook that use rubygems or
include internal chef mixins.

As someone working with various CM tools (chef, puppet, salt) and multiple
programming languages, i always found chef to be more coherent and easy
because it’s all ruby. most programmers will agree that the biggest
cognitive overhead is not the syntax of a programming language but rather
the behavior of its constructs which are 95% standard library and
frameworks. salt will not behave like chef even if both accept the same
yaml catalog, ditto for puppet. adding another syntax will slightly
increase cognitive overhead but as “crippled chef” user base groes it will
lead to different cookbook patterns and different behaviors - and that will
be a big cognitive overhead.
On top of this, consider that templating is applying logic to data, aka
programming. If i have to program, i want tools, tests and libraries -
which are the hallmarks of an evolved eco-system; if a significant number
of users will use “cripple chef” you will have less eco-system, not more.
In many ways this reminds me of the age old “lisp vs everyone else” debate:
at some point, the data IS the program.

my 2cents.

On Fri, Sep 20, 2013 at 6:10 AM, Noah Kantrowitz noah@coderanger.netwrote:

On Sep 19, 2013, at 7:39 PM, Adam Jacob adam@opscode.com wrote:

We have not even scratched the surface of people who are already
programmers, from an opscode customer base point of view. Of course, Chef
isn’t about Opscode, not really - but god bless you for thinking about it.
:slight_smile:

My worry isn’t about customers per se, honestly its pretty selfish. I like
Chef, I want to write my fix-the-world stuffs in or with Chef, and as such
I need to get more people (in my case a lot of Python people) to use Chef
so that I don’t have to go duplicate my work in Ansible or whatever is The
Thing at that moment.

My concern is you won’t get 80% of what you want out of it - you’re
going to get 20% of it. What’s capable with a dialect vs the native ruby is
hugely different - and it’s going to drift naturally. I think Noah is smart
to say that the low risk choice here is something like YAML - it’s
essentially just serialization, so it might be able to stretch relatively
naturally with a serialized resource catalog. But if you compare it to what
you will experience with a tool for whom the template+YAML approach is
truly first class, the delight won’t be the same.

I just want to be clear, I haven’t said that. I added JSON and Yaml for
attributes files, not recipes. I don’t think Yaml for recipes belongs in
core for exactly the reasons I mentioned re: difficulty translating
knowledge. I could certainly imagine a mapping, but needing non-declarative
stuff in recipes is pretty close to a given, whereas in attr files it only
crops up in very advanced cookbooks, usually in service of cross-platform
unification. I do think the JS and Python dialects will be able to maintain
a good portion of the expressive power of the Ruby DSL, but I don’t think
those belong in core just due to complexity of maintenance and general long
road to stability (you can currently segfault Ruby from your Python code if
you try and recursively load libffi for example). If a fully declarative
(JSON/Yaml) dialect is added for recipes I think the target wouldn’t be
humans, but auto generated recipe code. I don’t see why you would do that
though, so I left it out. Other than that it would only work well for
suuuuuuper simple cases, which do happen, but I don’t have a clear picture
of how thats evolving. I could certainly see a future where more and more
stuff moves in to LWRPs and app-ish cookbooks really could be fully
declarative invocations of those LWRPs, but we are a long way from that.

It’s interesting to hear you say you feel like you don’t know how to
program, when i’ve seen you be an effective member of our community for a
long time now. Is it completely out of school to say that perhaps you’re
arguing for your limitations? You might not be a CS major - lord knows, I’m
not. But you do know how to program, and you’re doing it in ruby. :slight_smile: The
closest you’ve come to a real programming language is Ruby: you’re
programming in it, and have been the entire time you’ve rocked the Chef, an
you’ve done so successfully.

My concern really boil down to this: I don’t want to say that there will
be a simpler path that turns out to just be a ghetto - an underserved, less
loved corner of Chef. I think we’ve made similar mistakes in the past, and
they’ve hurt us (for example, the unintentional impacts Opscode’s focus has
had on the chef solo experience being improved upon) as a community.

Agreed, it would mean a lot of documentation work to ensure mobility
between different formats. In my head I’m imagining something like how
multi-lang SDK docs work sometimes with a format radio button at the top of
the page which switches things within the page so you can see how the
syntaxes interact, but maybe that is overkill.

Having someone like Noah who is committed to seeing it through goes a
long way to solving that concern, in that I totally believe he would be an
active maintainer of that functionality.

Thinking out loud: what if we wrapped it in a feature flag and marked it
experimental?

I think because its already opt-in and there is no change to the Ruby
formats this might not accomplish much. The only way a new user will find
out this exists is via docs and blog posts and the like, and I think the
"here be dragons" would belong at that level, rather than someone trying to
use the new feature and either getting a message of “please pass
–dialects=yaml,py to continue” (which is one of those just-do-what-I-mean
rage inducing situations) or silent failure.

–Noah


#17

On Sep 20, 2013, at 2:51 AM, Avishai Ish-Shalom avishai@fewbytes.com wrote:

To me Chef’s biggest power was its flexibility; i can easily monkey patch/wrap whatever cookbook/internal chef behavior, and i’m not the only doing this. just look at the number of cookbook that use rubygems or include internal chef mixins.

It is worth noting that at least with the dialects I’ve written so far, this would continue to work just fine. The JSON/Yaml attributes loading just calls the same things you would to set the attributes, and the JS/Python stuff from chef-funnel cleanly calls the same methods as the normal Chef DSL. There are definitely ways to make dialects where this is not the case, but it goes back to the same underlying structure issue, any such dialect would have to either be for novelty value or have some other reason to be Not Chef™. In short I think as long as it is we who ar e creating the patterns, we can ensure the result is not crippled and follows the same patterns and meshes with the same tools.

–Noah


#18

As someone who regularly has to train people in Chef, dialects scares the
living crap out of me.

-s

On Thu, Sep 19, 2013 at 1:37 PM, Noah Kantrowitz noah@coderanger.netwrote:

https://github.com/opscode/chef/pull/997

At long last I’ve pushed up a completed-enough-to-use version of my
dialects proposal. I won’t repeat
the whole document here, but some highlights:

  1. Hooks to load Chef recipes and attribute files from formats other than
    .rb
  2. JSON and Yaml dialects for attribute files.
  3. Root shortcuts for cookbook source files of which you commonly have
    only one.
    cookbooks/
    apache2/
    attributes.rb
    recipe.rb
    library.rb
  4. Removal of the need for the default/ path segment under files/ and
    templates/.
  5. Ability to pass an array to #source on cookbook_file and template to
    define your own lookup path similar to the current implicit one.

Personally I think all of these put together are an order of magnitude
improvement in ease of learning Chef and writing new cookbooks while not
losing any of the expressive power you want on the high end. This branch
should present no backwards incompatibilities, though I think it should
kick off a discussion separately about deprecating the implicit search path
on files/ and templates/ in favor of people using the new explicit search
path syntax for the rare occasions where such things are desired (and even
in those cases, it is generally just a few of the options in the current
search path). Any such deprecation would have to be over a very long time
though. I have opened https://tickets.opscode.com/browse/CHEF-4559 to
track this patch. If anyone has any feedback please don’t hesitate to shoot
me an email or comment on the pull request/ticket.

–Noah


#19

Ha! I second that emotion.

On Friday, September 20, 2013, Sean OMeara wrote:

As someone who regularly has to train people in Chef, dialects scares the
living crap out of me.

-s

On Thu, Sep 19, 2013 at 1:37 PM, Noah Kantrowitz <noah@coderanger.net<javascript:_e({}, ‘cvml’, ‘noah@coderanger.net’);>

wrote:

https://github.com/opscode/chef/pull/997

At long last I’ve pushed up a completed-enough-to-use version of my
dialects proposal. I won’t repeat
the whole document here, but some highlights:

  1. Hooks to load Chef recipes and attribute files from formats other than
    .rb
  2. JSON and Yaml dialects for attribute files.
  3. Root shortcuts for cookbook source files of which you commonly have
    only one.
    cookbooks/
    apache2/
    attributes.rb
    recipe.rb
    library.rb
  4. Removal of the need for the default/ path segment under files/ and
    templates/.
  5. Ability to pass an array to #source on cookbook_file and template to
    define your own lookup path similar to the current implicit one.

Personally I think all of these put together are an order of magnitude
improvement in ease of learning Chef and writing new cookbooks while not
losing any of the expressive power you want on the high end. This branch
should present no backwards incompatibilities, though I think it should
kick off a discussion separately about deprecating the implicit search path
on files/ and templates/ in favor of people using the new explicit search
path syntax for the rare occasions where such things are desired (and even
in those cases, it is generally just a few of the options in the current
search path). Any such deprecation would have to be over a very long time
though. I have opened https://tickets.opscode.com/browse/CHEF-4559 to
track this patch. If anyone has any feedback please don’t hesitate to shoot
me an email or comment on the pull request/ticket.

–Noah


Mandi Walls
mandi.walls@gmail.com

@lnxchk


#20

Can you be more specific? Is the worry that people will demand training in 3rd party tools? Or that people will already know a non-ruby thing and it will be harder to get them going on the opscode training materials since the might have to “unlearn” some stuff. If the latter, I think the same argment of knowledge mapping applies. It means care must be taken is designing new dialects, but that’s a thing we can do.

–Noah

mandi walls mandi.walls@gmail.com wrote:

Ha! I second that emotion.

On Friday, September 20, 2013, Sean OMeara wrote:

As someone who regularly has to train people in Chef, dialects scares
the
living crap out of me.

-s

On Thu, Sep 19, 2013 at 1:37 PM, Noah Kantrowitz
<noah@coderanger.net<javascript:_e({}, ‘cvml’, ‘noah@coderanger.net’);>

wrote:

https://github.com/opscode/chef/pull/997

At long last I’ve pushed up a completed-enough-to-use version of my
dialects proposal. I won’t
repeat

the whole document here, but some highlights:

  1. Hooks to load Chef recipes and attribute files from formats other
    than

.rb
2. JSON and Yaml dialects for attribute files.
3. Root shortcuts for cookbook source files of which you commonly
have

only one.
cookbooks/
apache2/
attributes.rb
recipe.rb
library.rb
4. Removal of the need for the default/ path segment under files/
and

templates/.
5. Ability to pass an array to #source on cookbook_file and template
to

define your own lookup path similar to the current implicit one.

Personally I think all of these put together are an order of
magnitude

improvement in ease of learning Chef and writing new cookbooks while
not

losing any of the expressive power you want on the high end. This
branch

should present no backwards incompatibilities, though I think it
should

kick off a discussion separately about deprecating the implicit
search path

on files/ and templates/ in favor of people using the new explicit
search

path syntax for the rare occasions where such things are desired
(and even

in those cases, it is generally just a few of the options in the
current

search path). Any such deprecation would have to be over a very long
time

though. I have opened https://tickets.opscode.com/browse/CHEF-4559
to

track this patch. If anyone has any feedback please don’t hesitate
to shoot

me an email or comment on the pull request/ticket.

–Noah


Mandi Walls
mandi.walls@gmail.com

@lnxchk