An deep topic

[John_E_Vincent_lusis]

Okay so I’m afraid I’m going to regret doing this but Dan was right that
this is the best place for it. I thought about doing it on the dev list but
figured there would be more (appropriate?) feedback here.

I don’t think this fits with a blog post because (Dan is right here as
well) that’s dictation not discussion. I realize I’m an overly vocal critic
of this topic but I’ve heard in private from quite a few folks about these
same frustrations. I’ll leave it up to them if they want to speak up. Some
folks aren’t comfortable with that though and feel like it would be burning
a bridge. Please keep that in mind. Also please be aware that I’m going to
point to specific statements by people but I’m not trying to call those
people out and this should not be construed as a personal attack. I
understand I can’t stop people from feeling that way but I wanted to be
clear about this.

So this is partly about Berkshelf and more about communication and workflow.

I originally posted a gist right around ChefConf this year about Berkshelf (
https://gist.github.com/lusis/10696647). My original criticisms were borne
out of using omnibus but they apply to the chef toolchain as a whole. Some
of those concerns have been partly addressed but some have not and in fact
some have gotten worse.

I really think this whole issue boiled down to a lack of communication from
Chef in some official capacity. I realize the reasons for NOT wanting to go
on the record but at some point silence is complicity.

I understand that much of these changes have come out of the various
community summits but I don’t think it’s unfair to say that the community
summit was/is largely the vocal minority. And maybe that’s okay but again
based on my conversations with people, they feel entirely thrown for loop
after loop.

So I’m going to frame these questions again (at least the ones that haven’t
been clarified for ME):

1. Is Berkshelf becoming a core chef dependency?
   I heard no but let’s be clear when ALL of the tooling coming out of Chef
   proper has a dependency on Berkshelf, that’s no different. Also ChefDK
   doesn’t solve that problem (especially since it’s not actually released
   yet). Related is “The Berkshelf Way”/model the prescribed workflow for
   chef repos (note that I’m not saying cookbooks here for a reason).

2. What problem is Berkshelf trying to solve?
   This is part of my frustration and other folks as well. The running joke is
   that Berkshelf solved a problem I didn’t have. And it’s true. Again I can
   totally accept that some people had a problem and found that Berkshelf
   solved it but is that reason enough to bake it into EVERYTHING?

FWIW I didn’t have a problem with Berkshelf originally. One: I didn’t use
it and didn’t HAVE to use it. Two: It didn’t have this entirely new service
it depended on.

Now I can’t avoid it and it requires a separate internet service (though
now apparently rolled into the supermarket) just to be used.

But this ties into bigger workflow questions. From almost every corner of
the Chef proper universe (rough paraphrases):

“Berkshelf won”
“Stop doing devodd. Stop uploading unreleased artifacts to your Chef
Servers”
“the separate repos per cookbook voice + Berkshelf is the loudest and with
good reason”
“basically (in response to ‘so individual cookbooks can be treated like
district pieces of software?’)”
“You don’t have to change your workflow… You have to
change your workflow a bit”

And yet the Chef documentation is all over the place:
“The chef-dk defines a common workflow for cookbook development, including
unit and integration testing, identifying lint-like behavior, dedicated
tooling, and more”
“The chef-repo is located on a workstation and should be synchronized with
a version control system, such as git. All of the data in the chef-repo
should be treated like source code.”

There is conflicting information here in that the docs describe one thing
and yet there is semi-official direction/prescription/statements that you
should use a
single-cookbook-artifact-berkshelf-some-tool-that-isn’t-actually-knife
workflow.

But since this toolchain using this new workflow is so convoluted right
now, there’s ChefDK but it’s not released yet and even if it was it’s
running an ENTIRELY different version of Ruby than the Omnibus chef-client.

And in the end there’s, and I hesitated to mention this but it’s a part of
it, a profound level of arrogance about this workflow as a whole laced with
"but you can keep doing it the OLD way" or “you’ll eventually figure out
the right workflow” or “Berkshelf fits every use case”. But just a year ago
there was ANOTHER way that was named the same thing but done differently
and it’s not the right way anymore.

As someone who is largely a run of the mill chef end-user (you can disagree
with that. I have a larger than normal “voice” for better or worse), you
have to appreciate the confusion here at the mixed messaging and “new
shiny” every time I look up. As I’ve said before, Chef is a means to an end
for me and not the defining thing. I don’t like to fight my tools and when
they keep fighting back I have to weigh the balance of how much time I
invest with the tool versus getting actual work done. At some point, you
figure out that you’re fighting the wrong things.

And I realize that Chef is going through some transformative things right
now and trying to get back to certain roots.

So what am I asking here? What’s the point of this rambling?

Where’s the communication? What’s the intention? The post on Omnibus was
awesome because it was clear. A line was drawn that said “look, omnibus was
built for this and we think it’s awesome that people are using it for this
other thing but there’s a conflict that we can’t maintain”

I’m totally cool with that. What’s hard for me is that when it comes to
workflow, there’s a defacto standard (driven by a vocal subset of the
community and unofficially by Chef employees - my interpretation) that is
infinitely more complicated than the one officially documented. And it’s
not just a question of workflow, it changes the entire model of how you use
Chef when writing cookbooks (again in conflict with the official
documentation).

I realize the desire to straddle both ways but at some point you can’t when
they diverge so dramatically. And really one of them is in largely direct
opposition to the way of working that’s even possible for a subset of the
community. I feel like there’s an entire chef middle class being cut out of
this entire topic.

So am I asking for an “official workflow”? Part of me says no but part of
me says “yes already. And then update the docs so I have somewhere to point
new users that isn’t being contradicted everywhere else”. And frankly for
me I almost want it to be done so I can make the difficult decisions around
"is it time to move to another tool or not" because in the end my goal is
pretty simple. I need to manage my servers and the tool that I have to
fight the least to do that is the one I’m going to use.

Maybe I’m wrong about this. I could be the only person who feels this way
but I don’t think so.

Thanks for reading,
John E. Vincent

[adamhjk]

I'm going to hop in on top, but my intent isn't to stifle or ignore the
resulting conversation (which has been great).

First off, one of the things that has gotten off in the way we work as a
community is understanding what space we're in. You can see it in this
email, and in resulting threads - if you post to this list from a Chef
Software email address, are you speaking for yourself, or on behalf of the
company?

Since I'm probably the only person who can clear this up short of Barry
Crist (who is awesome, but I don't think is on this thread,) lets go ahead
and create a new rule. If you want to know the Official Position (tm) of
Chef Software, you can go ahead and ask directly - and I, or someone the
company designates, will respond with an Official Clarification (tm).
Otherwise, all communication on this list, and in IRC, is the opinion of
it's author, and not a representation of the Company. This is, I believe,
the only tenable position for this space to remain one that is actually
community driven, and not simply a channel for communicating with/around/to
Chef Software. I am adding this to the upcoming RFC on how the project is
maintained - keep your eyes peeled for that before our first community
meeting on Friday.

Now, on to your email.

On Mon, Jul 7, 2014 at 8:05 PM, John E. Vincent (lusis) <
lusis.org+chef-list@gmail.com> wrote:

Okay so I'm afraid I'm going to regret doing this but Dan was right that
this is the best place for it. I thought about doing it on the dev list but
figured there would be more (appropriate?) feedback here.

Why are you afraid you will regret it? Either you want to find solutions to
your problems, or you don't - and this is the place for us to find
solutions. Don't regret it - participate in it.

I don't think this fits with a blog post because (Dan is right here as
well) that's dictation not discussion. I realize I'm an overly vocal critic
of this topic but I've heard in private from quite a few folks about these
same frustrations. I'll leave it up to them if they want to speak up. Some
folks aren't comfortable with that though and feel like it would be burning
a bridge. Please keep that in mind. Also please be aware that I'm going to
point to specific statements by people but I'm not trying to call those
people out and this should not be construed as a personal attack. I
understand I can't stop people from feeling that way but I wanted to be
clear about this.

I understand not wanting to break confidences, or people being concerned
about hurting the feelings of others. At the same time, I'm wary of
arguments that involve the specter of broad-scale assent - at several
points in this email, you make reference to "how other people feel" and
"the chef middle class" - all of which creates a self-defined rhetorical
division within the community that, if it does exist, certainly can't be
quantified easily without more discussion. As a rhetorical device, it is
super effective - easy to think of a silent middle being abused by elites,
which, as one of what you probably define as the upper class, I resent on a
personal level, and I think my actual participation as a human being
refutes it consistently. The thread on this list that ensues proves both
the need for the discussion and the lack of actual division caused by
anything other than lack of participation.

So I'm going to frame these questions again (at least the ones that
haven't been clarified for ME):

1. Is Berkshelf becoming a core chef dependency?
   I heard no but let's be clear when ALL of the tooling coming out of Chef
   proper has a dependency on Berkshelf, that's no different. Also ChefDK
   doesn't solve that problem (especially since it's not actually released
   yet). Related is "The Berkshelf Way"/model the prescribed workflow for
   chef repos (note that I'm not saying cookbooks here for a reason).

As the CTO of Chef Software, let the next few paragraphs serve as an
Official Clarification (tm). (You request it in a piece I snipped) As we
move forward, I would like to see these conversations not require official
clarification from Chef Software, because improvements in the transparency
of how decisions are made and the roadmap is communicated obviates the
need. Since we don't live in that world yet, here we go.

There are currently no plans for the "chef-client" omnibus package and
rubygem to include Berkshelf as a dependency. It makes no sense to do so.
There are conversations about doing fully client-side dependency
resolution, as it would be more efficient for all concerned.

The Chef DK is our attempt to bring a single, vetted, consistent user
experience to those developing infrastructure with Chef. That includes
having (at least one) blessed workflow that allows users to go from very
little tooling to a fully built, best practice way of managing your
infrastructure at scale. Step one of that is simply getting all the tooling
in one place, released often, and vetted. Step two is creating and fixing
those workflows, so that we can then talk about them more broadly. There
are many more steps. Chef DK includes Berkshelf as its dependency resolver,
and the workflows it brings to bear (test driven infrastructure primarily)
use Berkshelf.

Chef Software is committed to ensuring that every tool shipped in the Chef
DK works well with the rest of the DK, is vetted and tested. In many cases,
that means working directly with the communities and authors of those
tools, which we haven't taken away from those authors (and couldn't if we
wanted to - they weren't "ours" to begin with.) That includes Berkshelf,
ChefSpec, Test Kitchen, and more - all of which we participate in and
support, but do not 'own'.

End of Official Clarification (tm), and back to just plain old Adam.

2. What problem is Berkshelf trying to solve?
   This is part of my frustration and other folks as well. The running joke
   is that Berkshelf solved a problem I didn't have. And it's true. Again I
   can totally accept that some people had a problem and found that Berkshelf
   solved it but is that reason enough to bake it into EVERYTHING?

Julian answered this well.

FWIW I didn't have a problem with Berkshelf originally. One: I didn't use
it and didn't HAVE to use it. Two: It didn't have this entirely new service
it depended on.

Now I can't avoid it and it requires a separate internet service (though
now apparently rolled into the supermarket) just to be used.

I understand your frustration here, but you want to have your cake and eat
it to. You want to stay up to date with an evolving, naturally leading edge
of development. On the one hand, you don't have a problem that required
client side dependency solving, because you're happy with the single
repository, vendor branch approach. For the record, so was I - I created it
because it worked for me, and I didn't struggle with the issues other
people had. I was also perfectly happy to have my workflow be logging in to
a machine and running chef client to observe if my shit worked. I was very
proficient at it.

On the other hand, you want to be able to use tools like Test Kitchen, or
the new generators in the Chef DK. The creators of those tools, over the
course of the last several years, all think that having dependency
resolution at the level of individual cookbooks is essential to its good
functioning. After using those workflows (which I didn't want to do,
because I didn't have those problems) - I agree with them, on a personal
level. I also think the results in practice speak for themselves - it
really does make it faster, and safer, to develop infrastructure this way.

You still don't have to use it. Not using it means opting out of the things
that, as the authors of those tools, people decided was necessary. That's
cool, because, as you say - you don't have those problems anyway, right?

But this ties into bigger workflow questions. From almost every corner of
the Chef proper universe (rough paraphrases):

"Berkshelf won"
"Stop doing devodd. Stop uploading unreleased artifacts to your Chef
Servers"
"the separate repos per cookbook voice + Berkshelf is the loudest and with
good reason"
"basically (in response to 'so individual cookbooks can be treated like
district pieces of software?')"
"You don't have to change your workflow.. You have to
change your workflow a bit"

And yet the Chef documentation is all over the place:
"The chef-dk defines a common workflow for cookbook development, including
unit and integration testing, identifying lint-like behavior, dedicated
tooling, and more"
"The chef-repo is located on a workstation and should be synchronized with
a version control system, such as git. All of the data in the chef-repo
should be treated like source code."

There is conflicting information here in that the docs describe one thing
and yet there is semi-official direction/prescription/statements that you
should use a
single-cookbook-artifact-berkshelf-some-tool-that-isn't-actually-knife
workflow.

But since this toolchain using this new workflow is so convoluted right
now, there's ChefDK but it's not released yet and even if it was it's
running an ENTIRELY different version of Ruby than the Omnibus chef-client.

We haven't ported the universe to a new workflow precisely because, in many
cases, it just isn't ready for it yet. The difficulties in tying those
tools together are sometimes deep, and pushing it out in front of the world
doesn't help. I agree we need to get to a place where the best practice
workflow is both documented and simple, and goes from using nothing but
chef-apply on a single recipe all the way up to managing complex
infrastructure with integrated testing and servers. The experience should
be delightful and natural at each part of the way.

It isn't right now, because we're in transition.

And in the end there's, and I hesitated to mention this but it's a part of
it, a profound level of arrogance about this workflow as a whole laced with
"but you can keep doing it the OLD way" or "you'll eventually figure out
the right workflow" or "Berkshelf fits every use case". But just a year ago
there was ANOTHER way that was named the same thing but done differently
and it's not the right way anymore.

As someone who is largely a run of the mill chef end-user (you can
disagree with that. I have a larger than normal "voice" for better or
worse), you have to appreciate the confusion here at the mixed messaging
and "new shiny" every time I look up. As I've said before, Chef is a means
to an end for me and not the defining thing. I don't like to fight my tools
and when they keep fighting back I have to weigh the balance of how much
time I invest with the tool versus getting actual work done. At some point,
you figure out that you're fighting the wrong things.

And I realize that Chef is going through some transformative things right
now and trying to get back to certain roots.

The arrogance exists on all sides of what, until this moment, hasn't even
really been a debate. Just so we're being clear, cloaking yourself in a
populist mantle (just a regular chef user, the chef middle class) and
calling those who think differently than you 'arrogant' is, itself, pretty
damn arrogant.

Lets all drop that from the dialogue, and just focus on getting the mixed
messaging and confusion that's at the heart of your message cleaned up.

mSo what am I asking here? What's the point of this rambling?

Where's the communication? What's the intention? The post on Omnibus was
awesome because it was clear. A line was drawn that said "look, omnibus was
built for this and we think it's awesome that people are using it for this
other thing but there's a conflict that we can't maintain"

I'm totally cool with that. What's hard for me is that when it comes to
workflow, there's a defacto standard (driven by a vocal subset of the
community and unofficially by Chef employees - my interpretation) that is
infinitely more complicated than the one officially documented. And it's
not just a question of workflow, it changes the entire model of how you use
Chef when writing cookbooks (again in conflict with the official
documentation).

Here is a proposal. Lets have two officially supported workflows.

1. The 'classic' workflow. This has a single chef-repo directory, that
   includes every artifact you need in your infrastructure. It its managed as
   a single git repository, uses vendor branches for importing external
   cookbooks via 'knife cookbook site', and either uploads directly to a Chef
   server or is bundled together for use in solo or chef-zero. The Chef DK
   should have generators that support this model as a first class citizen.

2. The 'TDI' workflow. This workflow involves breaking your single
   chef-repo into a git repository per cookbook, with all other assets tracked
   in a single common repository. It uses Berkshelf and Test-Kitchen to
   support test-driven infrastructure, and either uploads directly to a chef
   server or is bundled together for use in solo or chef-zero. The Chef DK
   should have generators that support this model as a first class citizen.

In either case, the glide path to either should be easy and basically
identical, until you decide to start writing tests.

I realize the desire to straddle both ways but at some point you can't
when they diverge so dramatically. And really one of them is in largely
direct opposition to the way of working that's even possible for a subset
of the community. I feel like there's an entire chef middle class being cut
out of this entire topic.

Divergence isn't a bad thing. If you don't want, or don't see, the benefits
of the TDI workflow - you shouldn't use it.

For many of us, that workflow is consistently producing better results than
the 'classic' one did.

So am I asking for an "official workflow"? Part of me says no but part of
me says "yes already. And then update the docs so I have somewhere to point
new users that isn't being contradicted everywhere else". And frankly for
me I almost want it to be done so I can make the difficult decisions around
"is it time to move to another tool or not" because in the end my goal is
pretty simple. I need to manage my servers and the tool that I have to
fight the least to do that is the one I'm going to use.

Maybe I'm wrong about this. I could be the only person who feels this way
but I don't think so.

As of today, I don't think you can say the TDI workflow is ready to be the
default workflow for new users. My opinion is we should continue to develop
it, and even recommend it for those whose problems it solves today and can
handle being early adopters. We shouldn't put it in front of new users (see
the revamped LearnChef content as an example.)
Adam

[Sean_OMeara1]

Dammit Adam! I spent all day composing a response and you went and posted a
much better one before I could.
Oh well. Here it is anyway.

Here is the world as I see it.

Once upon a time, in the beginning, there were no workflows.

The early Chef community was made up of early adopters. Tool smiths.
Builders. People on the edges of the Puppet and CFEngine communities. As
such, we brought our workflows with us, and the chef-repo workflow was born.

A single point of version control for All The Things made a lot of
intuitive sense. It still does. It is easy to wrap your mind around. It is
easy to teach to newcomers. It is unsophisticated and gets the job done. As
a matter of fact, we still use that approach in our Fundamentals Training,
because it lets us explain all the moving parts of Chef without
complicating things.

However, it quickly breaks down. Questions quickly arise around scaling
teams, SCM workflows, code re-use and modularity, and "best practices".
Answers to those questions were generally vauge and non-prescriptive
because we did not have the answer. TIMTOWTDI was the only possible
answer, and I personally believe it was the
right one for the time.

I still think its often a valid answer because I'm not entirely even sure
what "it" is. The best way to do what? Manage an infrastructure we've never
seen with a team of people we've never met? My epistemological problem
detector just went off. We also just jumped the shark while riding the
workflow pony for a tool designed to ensure directory permissions are
correct and landed on the moon.

So, lets get back to earth.

Given this lack of guidance on how to solve unknowable problems, the Chef
community stepped up to fill the void. The result was ecosystem tools,
created to solve problems specific to their workflows. Here are two web
shops in NYC that baked Chef into their existing operational workflows by
creating tools around it.
One made it into the wider community, and the other is baked so far into
existing processes, that it would not be useful for anyone outside the
organization.

Etsy chef-workflow

Paperless Ops Chef Workflow

This is usage of Chef as a building block. Making a custom space ship is
great if you're already a master builder. However, for the majority of
people, it is easy to end up with Lego explosions on the first few
attempts. That, I think is where the angst lies. Chef is a terrifyingly
powerful tool, and without guidance people get hurt.

If we're going to start prescribing "The Chef Way" of doing things, We need
to be absolutely clear about what we're talking about. As far as I can
tell, this falls into two major categories. The first, would be "how to
develop cookbooks", and the second being "how to operate cookbooks", which
is another animal entirely.

Now, on to Berkshelf.

The Berkshelf Way focuses on the "how to develop cookbooks" part of the
equation. It is result of a community member with a software engineering
background needing to coordinate hundreds of developers across the world.
To operate at that scale, you need modularity. You need re-use. You need
artifacts. You need to let subject matter experts be experts on their
subjects, and it all needs to be tested to make sure it works.

And so, inspiration was taken from the wider software world. CPAN. Maven.
PyPI. Rubygems. Npm. The "classic" chef-repo vendor-branch workflow made
this very difficult, because its its focus was on
infrastructure-as-a-project, instead of cookbook-as-a-project.

By focusing on "how to develop a cookbook" while leaving the "how to
operate your specific infrastructure" off the table, it is possible to
start being generally prescriptive. Perform integration tests. Write specs
to guard against regressions. Make well defined scopes. Publish to an
artifact repository (chef-server/ supermarket), use semver, etc.

To a large number of community members (myself included), pretty much all
of this was alien "software engineering" talk. As recently as 1.5 years ago
Icouldn't explain the difference between an integration test and a unit
test if you held a gun to my head. It was the Chef community that helped me
learn all these things. A major thing to remember is that as these
practices evolve, we have to learn them too. We're often in uncharted
territory.

As the lead on COOK, here's what I can do.

First, explicitly define the relationship between the cookbook ecosystem on
Supermarket, the Chef Software sponsored COOK project, Chef. I'll do this
in a separate post, in order to keep this one short Second, point to the
"shining cookbooks on a hill" as Dylan suggested. And third, give guidance
to how to develop cookbooks "The Sean OMeara Way" at any given moment. Keep
in mind these ways will change over time, as they always do.

-s

On Tue, Jul 8, 2014 at 5:00 PM, Adam Jacob adam@getchef.com wrote:

I'm going to hop in on top, but my intent isn't to stifle or ignore the
resulting conversation (which has been great).

First off, one of the things that has gotten off in the way we work as a
community is understanding what space we're in. You can see it in this
email, and in resulting threads - if you post to this list from a Chef
Software email address, are you speaking for yourself, or on behalf of the
company?

Since I'm probably the only person who can clear this up short of Barry
Crist (who is awesome, but I don't think is on this thread,) lets go ahead
and create a new rule. If you want to know the Official Position (tm) of
Chef Software, you can go ahead and ask directly - and I, or someone the
company designates, will respond with an Official Clarification (tm).
Otherwise, all communication on this list, and in IRC, is the opinion of
it's author, and not a representation of the Company. This is, I believe,
the only tenable position for this space to remain one that is actually
community driven, and not simply a channel for communicating with/around/to
Chef Software. I am adding this to the upcoming RFC on how the project is
maintained - keep your eyes peeled for that before our first community
meeting on Friday.

Now, on to your email.

On Mon, Jul 7, 2014 at 8:05 PM, John E. Vincent (lusis) <
lusis.org+chef-list@gmail.com> wrote:

Okay so I'm afraid I'm going to regret doing this but Dan was right that

this is the best place for it. I thought about doing it on the dev list but
figured there would be more (appropriate?) feedback here.

Why are you afraid you will regret it? Either you want to find solutions
to your problems, or you don't - and this is the place for us to find
solutions. Don't regret it - participate in it.

I don't think this fits with a blog post because (Dan is right here as
well) that's dictation not discussion. I realize I'm an overly vocal critic
of this topic but I've heard in private from quite a few folks about these
same frustrations. I'll leave it up to them if they want to speak up. Some
folks aren't comfortable with that though and feel like it would be burning
a bridge. Please keep that in mind. Also please be aware that I'm going to
point to specific statements by people but I'm not trying to call those
people out and this should not be construed as a personal attack. I
understand I can't stop people from feeling that way but I wanted to be
clear about this.

I understand not wanting to break confidences, or people being concerned
about hurting the feelings of others. At the same time, I'm wary of
arguments that involve the specter of broad-scale assent - at several
points in this email, you make reference to "how other people feel" and
"the chef middle class" - all of which creates a self-defined rhetorical
division within the community that, if it does exist, certainly can't be
quantified easily without more discussion. As a rhetorical device, it is
super effective - easy to think of a silent middle being abused by elites,
which, as one of what you probably define as the upper class, I resent on a
personal level, and I think my actual participation as a human being
refutes it consistently. The thread on this list that ensues proves both
the need for the discussion and the lack of actual division caused by
anything other than lack of participation.

So I'm going to frame these questions again (at least the ones that
haven't been clarified for ME):

1. Is Berkshelf becoming a core chef dependency?
   I heard no but let's be clear when ALL of the tooling coming out of Chef
   proper has a dependency on Berkshelf, that's no different. Also ChefDK
   doesn't solve that problem (especially since it's not actually released
   yet). Related is "The Berkshelf Way"/model the prescribed workflow for
   chef repos (note that I'm not saying cookbooks here for a reason).

As the CTO of Chef Software, let the next few paragraphs serve as an
Official Clarification (tm). (You request it in a piece I snipped) As we
move forward, I would like to see these conversations not require official
clarification from Chef Software, because improvements in the transparency
of how decisions are made and the roadmap is communicated obviates the
need. Since we don't live in that world yet, here we go.

There are currently no plans for the "chef-client" omnibus package and
rubygem to include Berkshelf as a dependency. It makes no sense to do so.
There are conversations about doing fully client-side dependency
resolution, as it would be more efficient for all concerned.

The Chef DK is our attempt to bring a single, vetted, consistent user
experience to those developing infrastructure with Chef. That includes
having (at least one) blessed workflow that allows users to go from very
little tooling to a fully built, best practice way of managing your
infrastructure at scale. Step one of that is simply getting all the tooling
in one place, released often, and vetted. Step two is creating and fixing
those workflows, so that we can then talk about them more broadly. There
are many more steps. Chef DK includes Berkshelf as its dependency resolver,
and the workflows it brings to bear (test driven infrastructure primarily)
use Berkshelf.

Chef Software is committed to ensuring that every tool shipped in the Chef
DK works well with the rest of the DK, is vetted and tested. In many cases,
that means working directly with the communities and authors of those
tools, which we haven't taken away from those authors (and couldn't if we
wanted to - they weren't "ours" to begin with.) That includes Berkshelf,
ChefSpec, Test Kitchen, and more - all of which we participate in and
support, but do not 'own'.

End of Official Clarification (tm), and back to just plain old Adam.

2. What problem is Berkshelf trying to solve?
   This is part of my frustration and other folks as well. The running joke
   is that Berkshelf solved a problem I didn't have. And it's true. Again I
   can totally accept that some people had a problem and found that Berkshelf
   solved it but is that reason enough to bake it into EVERYTHING?

Julian answered this well.

FWIW I didn't have a problem with Berkshelf originally. One: I didn't use
it and didn't HAVE to use it. Two: It didn't have this entirely new service
it depended on.

Now I can't avoid it and it requires a separate internet service (though
now apparently rolled into the supermarket) just to be used.

I understand your frustration here, but you want to have your cake and eat
it to. You want to stay up to date with an evolving, naturally leading edge
of development. On the one hand, you don't have a problem that required
client side dependency solving, because you're happy with the single
repository, vendor branch approach. For the record, so was I - I created it
because it worked for me, and I didn't struggle with the issues other
people had. I was also perfectly happy to have my workflow be logging in to
a machine and running chef client to observe if my shit worked. I was very
proficient at it.

On the other hand, you want to be able to use tools like Test Kitchen, or
the new generators in the Chef DK. The creators of those tools, over the
course of the last several years, all think that having dependency
resolution at the level of individual cookbooks is essential to its good
functioning. After using those workflows (which I didn't want to do,
because I didn't have those problems) - I agree with them, on a personal
level. I also think the results in practice speak for themselves - it
really does make it faster, and safer, to develop infrastructure this way.

You still don't have to use it. Not using it means opting out of the
things that, as the authors of those tools, people decided was necessary.
That's cool, because, as you say - you don't have those problems anyway,
right?

But this ties into bigger workflow questions. From almost every corner of
the Chef proper universe (rough paraphrases):

"Berkshelf won"
"Stop doing devodd. Stop uploading unreleased artifacts to your Chef
Servers"
"the separate repos per cookbook voice + Berkshelf is the loudest and
with good reason"
"basically (in response to 'so individual cookbooks can be treated like
district pieces of software?')"
"You don't have to change your workflow.. You have
to change your workflow a bit"

And yet the Chef documentation is all over the place:
"The chef-dk defines a common workflow for cookbook development,
including unit and integration testing, identifying lint-like behavior,
dedicated tooling, and more"
"The chef-repo is located on a workstation and should be synchronized
with a version control system, such as git. All of the data in the
chef-repo should be treated like source code."

There is conflicting information here in that the docs describe one thing
and yet there is semi-official direction/prescription/statements that you
should use a
single-cookbook-artifact-berkshelf-some-tool-that-isn't-actually-knife
workflow.

But since this toolchain using this new workflow is so convoluted right
now, there's ChefDK but it's not released yet and even if it was it's
running an ENTIRELY different version of Ruby than the Omnibus chef-client.

We haven't ported the universe to a new workflow precisely because, in
many cases, it just isn't ready for it yet. The difficulties in tying those
tools together are sometimes deep, and pushing it out in front of the world
doesn't help. I agree we need to get to a place where the best practice
workflow is both documented and simple, and goes from using nothing but
chef-apply on a single recipe all the way up to managing complex
infrastructure with integrated testing and servers. The experience should
be delightful and natural at each part of the way.

It isn't right now, because we're in transition.

And in the end there's, and I hesitated to mention this but it's a part
of it, a profound level of arrogance about this workflow as a whole laced
with "but you can keep doing it the OLD way" or "you'll eventually figure
out the right workflow" or "Berkshelf fits every use case". But just a year
ago there was ANOTHER way that was named the same thing but done
differently and it's not the right way anymore.

As someone who is largely a run of the mill chef end-user (you can
disagree with that. I have a larger than normal "voice" for better or
worse), you have to appreciate the confusion here at the mixed messaging
and "new shiny" every time I look up. As I've said before, Chef is a means
to an end for me and not the defining thing. I don't like to fight my tools
and when they keep fighting back I have to weigh the balance of how much
time I invest with the tool versus getting actual work done. At some point,
you figure out that you're fighting the wrong things.

And I realize that Chef is going through some transformative things right
now and trying to get back to certain roots.

The arrogance exists on all sides of what, until this moment, hasn't even
really been a debate. Just so we're being clear, cloaking yourself in a
populist mantle (just a regular chef user, the chef middle class) and
calling those who think differently than you 'arrogant' is, itself, pretty
damn arrogant.

Lets all drop that from the dialogue, and just focus on getting the mixed
messaging and confusion that's at the heart of your message cleaned up.

mSo what am I asking here? What's the point of this rambling?

Where's the communication? What's the intention? The post on Omnibus was
awesome because it was clear. A line was drawn that said "look, omnibus was
built for this and we think it's awesome that people are using it for this
other thing but there's a conflict that we can't maintain"

I'm totally cool with that. What's hard for me is that when it comes to
workflow, there's a defacto standard (driven by a vocal subset of the
community and unofficially by Chef employees - my interpretation) that is
infinitely more complicated than the one officially documented. And it's
not just a question of workflow, it changes the entire model of how you use
Chef when writing cookbooks (again in conflict with the official
documentation).

Here is a proposal. Lets have two officially supported workflows.

1. The 'classic' workflow. This has a single chef-repo directory, that
   includes every artifact you need in your infrastructure. It its managed as
   a single git repository, uses vendor branches for importing external
   cookbooks via 'knife cookbook site', and either uploads directly to a Chef
   server or is bundled together for use in solo or chef-zero. The Chef DK
   should have generators that support this model as a first class citizen.

2. The 'TDI' workflow. This workflow involves breaking your single
   chef-repo into a git repository per cookbook, with all other assets tracked
   in a single common repository. It uses Berkshelf and Test-Kitchen to
   support test-driven infrastructure, and either uploads directly to a chef
   server or is bundled together for use in solo or chef-zero. The Chef DK
   should have generators that support this model as a first class citizen.

In either case, the glide path to either should be easy and basically
identical, until you decide to start writing tests.

I realize the desire to straddle both ways but at some point you can't
when they diverge so dramatically. And really one of them is in largely
direct opposition to the way of working that's even possible for a subset
of the community. I feel like there's an entire chef middle class being cut
out of this entire topic.

Divergence isn't a bad thing. If you don't want, or don't see, the
benefits of the TDI workflow - you shouldn't use it.

For many of us, that workflow is consistently producing better results
than the 'classic' one did.

So am I asking for an "official workflow"? Part of me says no but part of
me says "yes already. And then update the docs so I have somewhere to point
new users that isn't being contradicted everywhere else". And frankly for
me I almost want it to be done so I can make the difficult decisions around
"is it time to move to another tool or not" because in the end my goal is
pretty simple. I need to manage my servers and the tool that I have to
fight the least to do that is the one I'm going to use.

Maybe I'm wrong about this. I could be the only person who feels this way
but I don't think so.

As of today, I don't think you can say the TDI workflow is ready to be the
default workflow for new users. My opinion is we should continue to develop
it, and even recommend it for those whose problems it solves today and can
handle being early adopters. We shouldn't put it in front of new users (see
the revamped LearnChef content as an example.)
Adam

[John_E_Vincent_lusis]

On Tue, Jul 8, 2014 at 5:00 PM, Adam Jacob adam@getchef.com wrote:

I'm going to hop in on top, but my intent isn't to stifle or ignore the
resulting conversation (which has been great).

First off, one of the things that has gotten off in the way we work as a
community is understanding what space we're in. You can see it in this
email, and in resulting threads - if you post to this list from a Chef
Software email address, are you speaking for yourself, or on behalf of the
company?

Since I'm probably the only person who can clear this up short of Barry
Crist (who is awesome, but I don't think is on this thread,) lets go ahead
and create a new rule. If you want to know the Official Position (tm) of
Chef Software, you can go ahead and ask directly - and I, or someone the
company designates, will respond with an Official Clarification (tm).
Otherwise, all communication on this list, and in IRC, is the opinion of
it's author, and not a representation of the Company. This is, I believe,
the only tenable position for this space to remain one that is actually
community driven, and not simply a channel for communicating with/around/to
Chef Software. I am adding this to the upcoming RFC on how the project is
maintained - keep your eyes peeled for that before our first community
meeting on Friday.

Now, on to your email.

On Mon, Jul 7, 2014 at 8:05 PM, John E. Vincent (lusis) <
lusis.org+chef-list@gmail.com> wrote:

Okay so I'm afraid I'm going to regret doing this but Dan was right that

this is the best place for it. I thought about doing it on the dev list but
figured there would be more (appropriate?) feedback here.

Why are you afraid you will regret it? Either you want to find solutions
to your problems, or you don't - and this is the place for us to find
solutions. Don't regret it - participate in it.

I don't think this fits with a blog post because (Dan is right here as
well) that's dictation not discussion. I realize I'm an overly vocal critic
of this topic but I've heard in private from quite a few folks about these
same frustrations. I'll leave it up to them if they want to speak up. Some
folks aren't comfortable with that though and feel like it would be burning
a bridge. Please keep that in mind. Also please be aware that I'm going to
point to specific statements by people but I'm not trying to call those
people out and this should not be construed as a personal attack. I
understand I can't stop people from feeling that way but I wanted to be
clear about this.

I understand not wanting to break confidences, or people being concerned
about hurting the feelings of others. At the same time, I'm wary of
arguments that involve the specter of broad-scale assent - at several
points in this email, you make reference to "how other people feel" and
"the chef middle class" - all of which creates a self-defined rhetorical
division within the community that, if it does exist, certainly can't be
quantified easily without more discussion. As a rhetorical device, it is
super effective - easy to think of a silent middle being abused by elites,
which, as one of what you probably define as the upper class, I resent on a
personal level, and I think my actual participation as a human being
refutes it consistently. The thread on this list that ensues proves both
the need for the discussion and the lack of actual division caused by
anything other than lack of participation.

That was definitely never my intent but you are right. It's hard for anyone
to prove. I don't know how better to communicate but suffice to say because
I've been a vocal critic (not in any negative or even positive sense of the
word), I've had quite a bit of private feed back mostly via twitter DMs.
I'm not really comfortable discussing that particular point anymore and I
never intended it as some sort of rhetorical "I win" button.

To clarify the "middle class" bit, though, it wasn't even a matter of
"abuse by elites". I was simply talking about the group of people in the
middle between "Chef New Users" and "Chef Expert Level Users". Somewhere in
that "Expert" range there are also people who have a freedom to work on
chef ecosystem as part of their actual jobs. I'm not bitter about that and
I don't look down on them (or up really) - we're talking horizontal not
vertical. If that came off wrong I'm definitely sorry.

So I'm going to frame these questions again (at least the ones that
haven't been clarified for ME):

1. Is Berkshelf becoming a core chef dependency?
   I heard no but let's be clear when ALL of the tooling coming out of Chef
   proper has a dependency on Berkshelf, that's no different. Also ChefDK
   doesn't solve that problem (especially since it's not actually released
   yet). Related is "The Berkshelf Way"/model the prescribed workflow for
   chef repos (note that I'm not saying cookbooks here for a reason).

As the CTO of Chef Software, let the next few paragraphs serve as an
Official Clarification (tm). (You request it in a piece I snipped) As we
move forward, I would like to see these conversations not require official
clarification from Chef Software, because improvements in the transparency
of how decisions are made and the roadmap is communicated obviates the
need. Since we don't live in that world yet, here we go.

There are currently no plans for the "chef-client" omnibus package and
rubygem to include Berkshelf as a dependency. It makes no sense to do so.
There are conversations about doing fully client-side dependency
resolution, as it would be more efficient for all concerned.

The Chef DK is our attempt to bring a single, vetted, consistent user
experience to those developing infrastructure with Chef. That includes
having (at least one) blessed workflow that allows users to go from very
little tooling to a fully built, best practice way of managing your
infrastructure at scale. Step one of that is simply getting all the tooling
in one place, released often, and vetted. Step two is creating and fixing
those workflows, so that we can then talk about them more broadly. There
are many more steps. Chef DK includes Berkshelf as its dependency resolver,
and the workflows it brings to bear (test driven infrastructure primarily)
use Berkshelf.

Chef Software is committed to ensuring that every tool shipped in the Chef
DK works well with the rest of the DK, is vetted and tested. In many cases,
that means working directly with the communities and authors of those
tools, which we haven't taken away from those authors (and couldn't if we
wanted to - they weren't "ours" to begin with.) That includes Berkshelf,
ChefSpec, Test Kitchen, and more - all of which we participate in and
support, but do not 'own'.

End of Official Clarification (tm), and back to just plain old Adam.

Fair enough. official clarification was probably the wrong phrasing to get
my request across.

2. What problem is Berkshelf trying to solve?
   This is part of my frustration and other folks as well. The running joke
   is that Berkshelf solved a problem I didn't have. And it's true. Again I
   can totally accept that some people had a problem and found that Berkshelf
   solved it but is that reason enough to bake it into EVERYTHING?

Julian answered this well.

FWIW I didn't have a problem with Berkshelf originally. One: I didn't use
it and didn't HAVE to use it. Two: It didn't have this entirely new service
it depended on.

Now I can't avoid it and it requires a separate internet service (though
now apparently rolled into the supermarket) just to be used.

I understand your frustration here, but you want to have your cake and eat
it to. You want to stay up to date with an evolving, naturally leading edge
of development. On the one hand, you don't have a problem that required
client side dependency solving, because you're happy with the single
repository, vendor branch approach. For the record, so was I - I created it
because it worked for me, and I didn't struggle with the issues other
people had. I was also perfectly happy to have my workflow be logging in to
a machine and running chef client to observe if my shit worked. I was very
proficient at it.

I don't think that's entirely a fair statement. Honestly I'm NOT trying to
stay on the leading edge of development. That's not really a part of my
job. I think if anything I get caught behind on changes because I'm not
staying up to date. From my perspective it's pretty difficult to stay up to
date because things are changing so frequently. Like I said, I'm in a
middle ground here as a chef user. I use chef as a means to and end. It
works well but I have to weigh the needs of our usage with the (maybe
perceived) volatility in the community. It very much feels to me. as I
said, that when I come up for air to grab a new cookbook or update some
knife plugin, I spend a day just dealing with "what's broken now". It feels
very much like fighting Vagrant releases did.

It's probably a fair statement that I should be better about tracking these
changes but again what ends up happening is we want to use a cookbook from
opscode-cookbook's repo or get a bug fix and everything has changed yet
again. Perception? Maybe but that doesn't make it any less valid.

On the other hand, you want to be able to use tools like Test Kitchen, or
the new generators in the Chef DK. The creators of those tools, over the
course of the last several years, all think that having dependency
resolution at the level of individual cookbooks is essential to its good
functioning. After using those workflows (which I didn't want to do,
because I didn't have those problems) - I agree with them, on a personal
level. I also think the results in practice speak for themselves - it
really does make it faster, and safer, to develop infrastructure this way.

You still don't have to use it. Not using it means opting out of the things

that, as the authors of those tools, people decided was necessary. That's
cool, because, as you say - you don't have those problems anyway, right?

I think this is an area I take issue with. The implication here is that we
don't test our cookbooks or that we want to use test kitchen or that our
workflow is somehow suboptimal. Are there things I would change? Sure but
the implication still feels like "you'll come around. you just aren't
REALLY using things yet. Wait till you have to do X".

But this ties into bigger workflow questions. From almost every corner of

the Chef proper universe (rough paraphrases):

"Berkshelf won"
"Stop doing devodd. Stop uploading unreleased artifacts to your Chef
Servers"
"the separate repos per cookbook voice + Berkshelf is the loudest and
with good reason"
"basically (in response to 'so individual cookbooks can be treated like
district pieces of software?')"
"You don't have to change your workflow.. You have
to change your workflow a bit"

And yet the Chef documentation is all over the place:
"The chef-dk defines a common workflow for cookbook development,
including unit and integration testing, identifying lint-like behavior,
dedicated tooling, and more"
"The chef-repo is located on a workstation and should be synchronized
with a version control system, such as git. All of the data in the
chef-repo should be treated like source code."

There is conflicting information here in that the docs describe one thing
and yet there is semi-official direction/prescription/statements that you
should use a
single-cookbook-artifact-berkshelf-some-tool-that-isn't-actually-knife
workflow.

But since this toolchain using this new workflow is so convoluted right
now, there's ChefDK but it's not released yet and even if it was it's
running an ENTIRELY different version of Ruby than the Omnibus chef-client.

We haven't ported the universe to a new workflow precisely because, in
many cases, it just isn't ready for it yet. The difficulties in tying those
tools together are sometimes deep, and pushing it out in front of the world
doesn't help. I agree we need to get to a place where the best practice
workflow is both documented and simple, and goes from using nothing but
chef-apply on a single recipe all the way up to managing complex
infrastructure with integrated testing and servers. The experience should
be delightful and natural at each part of the way.

It isn't right now, because we're in transition.

And in the end there's, and I hesitated to mention this but it's a part
of it, a profound level of arrogance about this workflow as a whole laced
with "but you can keep doing it the OLD way" or "you'll eventually figure
out the right workflow" or "Berkshelf fits every use case". But just a year
ago there was ANOTHER way that was named the same thing but done
differently and it's not the right way anymore.

As someone who is largely a run of the mill chef end-user (you can
disagree with that. I have a larger than normal "voice" for better or
worse), you have to appreciate the confusion here at the mixed messaging
and "new shiny" every time I look up. As I've said before, Chef is a means
to an end for me and not the defining thing. I don't like to fight my tools
and when they keep fighting back I have to weigh the balance of how much
time I invest with the tool versus getting actual work done. At some point,
you figure out that you're fighting the wrong things.

And I realize that Chef is going through some transformative things right
now and trying to get back to certain roots.

The arrogance exists on all sides of what, until this moment, hasn't even
really been a debate.

Just so we're being clear, cloaking yourself in a populist mantle (just a

regular chef user, the chef middle class) and calling those who think
differently than you 'arrogant' is, itself, pretty damn arrogant.

Lets all drop that from the dialogue, and just focus on getting the mixed
messaging and confusion that's at the heart of your message cleaned up.

Just a small comment here (unrelated to the arrogance bit). Just because
nobody has spoken up doesn't mean there hasn't been discussion. I don't
know what internal discussions/debates were had. I'm only speaking to what
people have confided in me and my personal experiences. This isn't the
first time this has come up from me for sure.

mSo what am I asking here? What's the point of this rambling?

Where's the communication? What's the intention? The post on Omnibus was
awesome because it was clear. A line was drawn that said "look, omnibus was
built for this and we think it's awesome that people are using it for this
other thing but there's a conflict that we can't maintain"

I'm totally cool with that. What's hard for me is that when it comes to
workflow, there's a defacto standard (driven by a vocal subset of the
community and unofficially by Chef employees - my interpretation) that is
infinitely more complicated than the one officially documented. And it's
not just a question of workflow, it changes the entire model of how you use
Chef when writing cookbooks (again in conflict with the official
documentation).

Here is a proposal. Lets have two officially supported workflows.

1. The 'classic' workflow. This has a single chef-repo directory, that
   includes every artifact you need in your infrastructure. It its managed as
   a single git repository, uses vendor branches for importing external
   cookbooks via 'knife cookbook site', and either uploads directly to a Chef
   server or is bundled together for use in solo or chef-zero. The Chef DK
   should have generators that support this model as a first class citizen.

2. The 'TDI' workflow. This workflow involves breaking your single
   chef-repo into a git repository per cookbook, with all other assets tracked
   in a single common repository. It uses Berkshelf and Test-Kitchen to
   support test-driven infrastructure, and either uploads directly to a chef
   server or is bundled together for use in solo or chef-zero. The Chef DK
   should have generators that support this model as a first class citizen.

In either case, the glide path to either should be easy and basically
identical, until you decide to start writing tests.

I don't want to bikeshed names here but again the implication is that
someone using the single chef-repo isn't doing any testing. The only
difference between these two boils down to monolithic repo vs.
per-cookbook.

I realize the desire to straddle both ways but at some point you can't
when they diverge so dramatically. And really one of them is in largely
direct opposition to the way of working that's even possible for a subset
of the community. I feel like there's an entire chef middle class being cut
out of this entire topic.

Divergence isn't a bad thing. If you don't want, or don't see, the
benefits of the TDI workflow - you shouldn't use it.

For many of us, that workflow is consistently producing better results
than the 'classic' one did.

So am I asking for an "official workflow"? Part of me says no but part of
me says "yes already. And then update the docs so I have somewhere to point
new users that isn't being contradicted everywhere else". And frankly for
me I almost want it to be done so I can make the difficult decisions around
"is it time to move to another tool or not" because in the end my goal is
pretty simple. I need to manage my servers and the tool that I have to
fight the least to do that is the one I'm going to use.

Maybe I'm wrong about this. I could be the only person who feels this way
but I don't think so.

As of today, I don't think you can say the TDI workflow is ready to be the
default workflow for new users. My opinion is we should continue to develop
it, and even recommend it for those whose problems it solves today and can
handle being early adopters. We shouldn't put it in front of new users (see
the revamped LearnChef content as an example.)

On another note, I'm glad you got involved in the discussion not from a
"official chef policy" point but as an end-user. You've always been good
about that and this is definitely no exception. For me personally it means
a boatload.

Adam

[adamhjk]

On Tue, Jul 8, 2014 at 10:40 PM, John E. Vincent (lusis) <
lusis.org+chef-list@gmail.com> wrote:

That was definitely never my intent but you are right. It's hard for
anyone to prove. I don't know how better to communicate but suffice to say
because I've been a vocal critic (not in any negative or even positive
sense of the word), I've had quite a bit of private feed back mostly via
twitter DMs. I'm not really comfortable discussing that particular point
anymore and I never intended it as some sort of rhetorical "I win" button.

To clarify the "middle class" bit, though, it wasn't even a matter of
"abuse by elites". I was simply talking about the group of people in the
middle between "Chef New Users" and "Chef Expert Level Users". Somewhere in
that "Expert" range there are also people who have a freedom to work on
chef ecosystem as part of their actual jobs. I'm not bitter about that and
I don't look down on them (or up really) - we're talking horizontal not
vertical. If that came off wrong I'm definitely sorry.

It's fine to talk about the fact that you've heard similar feedback from
others - I have too. I don't think you need to break confidences, or
anything even remotely like that. I do think you need to be careful when
you use language like you did, particularly in a community. Regardless of
putting a disclaimer on the front about how non-offensive you want to be,
you absolutely framed this as an us-vs-them way, which makes the starting
point for discourse at a net negative.

Less blame, more fixes.

Fair enough. official clarification was probably the wrong phrasing to get
my request across.

I dunno, it felt super healthy to me.

I don't think that's entirely a fair statement. Honestly I'm NOT trying to
stay on the leading edge of development. That's not really a part of my
job. I think if anything I get caught behind on changes because I'm not
staying up to date. From my perspective it's pretty difficult to stay up to
date because things are changing so frequently. Like I said, I'm in a
middle ground here as a chef user. I use chef as a means to and end. It
works well but I have to weigh the needs of our usage with the (maybe
perceived) volatility in the community. It very much feels to me. as I
said, that when I come up for air to grab a new cookbook or update some
knife plugin, I spend a day just dealing with "what's broken now". It feels
very much like fighting Vagrant releases did.

It's probably a fair statement that I should be better about tracking
these changes but again what ends up happening is we want to use a cookbook
from opscode-cookbook's repo or get a bug fix and everything has changed
yet again. Perception? Maybe but that doesn't make it any less valid.

It doesn't make it less valid, but it does reflect on some of the
difficulties inherent in your situation. A great example is a cookbook from
the opscode-cookbooks repo - they are maintained pieces of software, by
folks that are employed to maintain them. They try and keep them up to
date, to accept pull requests, and to make them high quality. That work
involves tools that you don't like or want, so when you (admittedly) go to
use them after a long absence, they cause you pain.

This isn't unique to Chef - this is the same conversation everyone has when
they talk about pinning a dependency. It's great because you stay stable,
but you are paying a tax the whole time - you either pay that tax a little
bit all the time (to stay current) or a whole lot on the day your taxes are
due (you decide you need an upgrade for whatever reason). That's the price
you pay for re-use, in any situation.

I'm not saying we shouldn't be using semver, or communicating more clearly

• folks should be. At the same time, I think the lack of sympathy you pick
  up on is a little bit real: it sucks to have that problem, but them's the
  breaks, and you make your bed, to some extent. There is no free lunch.

At the last community summit, I talked about the fact that, from my point
of view, the idea of having community standard abstractions at the level of
something like, say, "apache" actually turned out to be a failed
experiment. If you come in alone, and you have to configure apache, you
probably need less than 10 configuration statements to be perfect. It might
take you 20 minutes to figure it out, say. Now, lets say you decide to
automate your apache configuration (yay!) and you pop open the Apache 2
cookbook. What you get hit with is all the complexity of apache (cause that
cookbook does everything) plus all the complexity of Chef (which is not
small). Maybe you find your way to the couple statements in chef you need
to get the result, and move on, maybe you don't. God help you if you
actually find a bug, or need to edit the source - you'll be hit in the face
with what is actually a complex software project, managing software on 10+
different platforms, and you'll be brutalized.

The alternative would have been maintaining your own apache cookbook that
just does what you need. It's probably 5 resources to begin with. It might
always be. Which situation is better? I know what I would choose. The whole
world doesn't agree with me, but I really think the sweet spot for re-use
is at the level of resources, providers, and libraries rather than recipes
for most cases.

I think this is an area I take issue with. The implication here is that we
don't test our cookbooks or that we want to use test kitchen or that our
workflow is somehow suboptimal. Are there things I would change? Sure but
the implication still feels like "you'll come around. you just aren't
REALLY using things yet. Wait till you have to do X".

That's not what I'm saying. I'm saying if you want to test your cookbooks
using those tools, their authors have some opinions based on their
experience about how your workflow needs to change. If you hate that, you
still have choices - it just doesn't involve reusing other peoples work,
because you opted out of that.

Just a small comment here (unrelated to the arrogance bit). Just because
nobody has spoken up doesn't mean there hasn't been discussion. I don't
know what internal discussions/debates were had. I'm only speaking to what
people have confided in me and my personal experiences. This isn't the
first time this has come up from me for sure.

Of course. I would wager this is the first conversation that's happened as
a community, in public, about what decisions we should make to better
accommodate people in your situation.

I don't want to bikeshed names here but again the implication is that
someone using the single chef-repo isn't doing any testing. The only
difference between these two boils down to monolithic repo vs.
per-cookbook.

Sure. You are otherwise in favor?

The second thing we'll need here is someone who wants to take on the role
of shepherding the experience for both workflows. Asking folks who have
moved away from the monolothic repo approach (for what, to them, are valid
reasons) to maintain the other standard at parity is probably a loosing
proposition. For example, the majority of the testing workflows assume that
you use berkshelf to inject dependencies into the vm before you converge.
That doesn't mean you have to be outside a monolithic repo, but it does
change your Berkshelf configuration (for example, you could just have a
loop that injects every cookbook in your repo). You could also write
another way of injecting your resources.

On another note, I'm glad you got involved in the discussion not from a

"official chef policy" point but as an end-user. You've always been good
about that and this is definitely no exception. For me personally it means
a boatload.

It's how it should actually have always worked.

Adam

[Sean_OMeara1]

I'm trying to understand why you think you can't avoid Berkshelf...

Just because cookbooks are dropping Gemfiles and testing harnesses
(.kitchen.yml) into their individual source repositories... what's
stopping you from doing a "knife vendor" into your chef-repo and
testing using whatever methodology you always have?

-s

On Wed, Jul 9, 2014 at 1:25 PM, Adam Jacob adam@getchef.com wrote:

On Tue, Jul 8, 2014 at 10:40 PM, John E. Vincent (lusis)
lusis.org+chef-list@gmail.com wrote:

That was definitely never my intent but you are right. It's hard for
anyone to prove. I don't know how better to communicate but suffice to say
because I've been a vocal critic (not in any negative or even positive sense
of the word), I've had quite a bit of private feed back mostly via twitter
DMs. I'm not really comfortable discussing that particular point anymore and
I never intended it as some sort of rhetorical "I win" button.

To clarify the "middle class" bit, though, it wasn't even a matter of
"abuse by elites". I was simply talking about the group of people in the
middle between "Chef New Users" and "Chef Expert Level Users". Somewhere in
that "Expert" range there are also people who have a freedom to work on chef
ecosystem as part of their actual jobs. I'm not bitter about that and I
don't look down on them (or up really) - we're talking horizontal not
vertical. If that came off wrong I'm definitely sorry.

It's fine to talk about the fact that you've heard similar feedback from
others - I have too. I don't think you need to break confidences, or
anything even remotely like that. I do think you need to be careful when you
use language like you did, particularly in a community. Regardless of
putting a disclaimer on the front about how non-offensive you want to be,
you absolutely framed this as an us-vs-them way, which makes the starting
point for discourse at a net negative.

Less blame, more fixes.

Fair enough. official clarification was probably the wrong phrasing to get
my request across.

I dunno, it felt super healthy to me.

I don't think that's entirely a fair statement. Honestly I'm NOT trying to
stay on the leading edge of development. That's not really a part of my job.
I think if anything I get caught behind on changes because I'm not staying
up to date. From my perspective it's pretty difficult to stay up to date
because things are changing so frequently. Like I said, I'm in a middle
ground here as a chef user. I use chef as a means to and end. It works well
but I have to weigh the needs of our usage with the (maybe perceived)
volatility in the community. It very much feels to me. as I said, that when
I come up for air to grab a new cookbook or update some knife plugin, I
spend a day just dealing with "what's broken now". It feels very much like
fighting Vagrant releases did.

It's probably a fair statement that I should be better about tracking
these changes but again what ends up happening is we want to use a cookbook
from opscode-cookbook's repo or get a bug fix and everything has changed yet
again. Perception? Maybe but that doesn't make it any less valid.

It doesn't make it less valid, but it does reflect on some of the
difficulties inherent in your situation. A great example is a cookbook from
the opscode-cookbooks repo - they are maintained pieces of software, by
folks that are employed to maintain them. They try and keep them up to date,
to accept pull requests, and to make them high quality. That work involves
tools that you don't like or want, so when you (admittedly) go to use them
after a long absence, they cause you pain.

This isn't unique to Chef - this is the same conversation everyone has when
they talk about pinning a dependency. It's great because you stay stable,
but you are paying a tax the whole time - you either pay that tax a little
bit all the time (to stay current) or a whole lot on the day your taxes are
due (you decide you need an upgrade for whatever reason). That's the price
you pay for re-use, in any situation.

I'm not saying we shouldn't be using semver, or communicating more clearly -
folks should be. At the same time, I think the lack of sympathy you pick up
on is a little bit real: it sucks to have that problem, but them's the
breaks, and you make your bed, to some extent. There is no free lunch.

At the last community summit, I talked about the fact that, from my point of
view, the idea of having community standard abstractions at the level of
something like, say, "apache" actually turned out to be a failed experiment.
If you come in alone, and you have to configure apache, you probably need
less than 10 configuration statements to be perfect. It might take you 20
minutes to figure it out, say. Now, lets say you decide to automate your
apache configuration (yay!) and you pop open the Apache 2 cookbook. What you
get hit with is all the complexity of apache (cause that cookbook does
everything) plus all the complexity of Chef (which is not small). Maybe
you find your way to the couple statements in chef you need to get the
result, and move on, maybe you don't. God help you if you actually find a
bug, or need to edit the source - you'll be hit in the face with what is
actually a complex software project, managing software on 10+ different
platforms, and you'll be brutalized.

The alternative would have been maintaining your own apache cookbook that
just does what you need. It's probably 5 resources to begin with. It might
always be. Which situation is better? I know what I would choose. The whole
world doesn't agree with me, but I really think the sweet spot for re-use is
at the level of resources, providers, and libraries rather than recipes for
most cases.

I think this is an area I take issue with. The implication here is that we
don't test our cookbooks or that we want to use test kitchen or that our
workflow is somehow suboptimal. Are there things I would change? Sure but
the implication still feels like "you'll come around. you just aren't REALLY
using things yet. Wait till you have to do X".

That's not what I'm saying. I'm saying if you want to test your cookbooks
using those tools, their authors have some opinions based on their
experience about how your workflow needs to change. If you hate that, you
still have choices - it just doesn't involve reusing other peoples work,
because you opted out of that.

Just a small comment here (unrelated to the arrogance bit). Just because
nobody has spoken up doesn't mean there hasn't been discussion. I don't know
what internal discussions/debates were had. I'm only speaking to what people
have confided in me and my personal experiences. This isn't the first time
this has come up from me for sure.

Of course. I would wager this is the first conversation that's happened as a
community, in public, about what decisions we should make to better
accommodate people in your situation.

I don't want to bikeshed names here but again the implication is that
someone using the single chef-repo isn't doing any testing. The only
difference between these two boils down to monolithic repo vs. per-cookbook.

Sure. You are otherwise in favor?

The second thing we'll need here is someone who wants to take on the role of
shepherding the experience for both workflows. Asking folks who have moved
away from the monolothic repo approach (for what, to them, are valid
reasons) to maintain the other standard at parity is probably a loosing
proposition. For example, the majority of the testing workflows assume that
you use berkshelf to inject dependencies into the vm before you converge.
That doesn't mean you have to be outside a monolithic repo, but it does
change your Berkshelf configuration (for example, you could just have a loop
that injects every cookbook in your repo). You could also write another way
of injecting your resources.

On another note, I'm glad you got involved in the discussion not from a
"official chef policy" point but as an end-user. You've always been good
about that and this is definitely no exception. For me personally it means a
boatload.

It's how it should actually have always worked.

Adam

[Brian_Akins]

Coming late to this thread, but I am one of the folks who had an offline
discussion/rant with lusis.

When Heartbleed (#1) happened, I needed to rebuild a ton of omnibus things
and omnibus was just horribly broken. It had worked fine just a few days
before then it just didn’t. In a fury of vendoring things and doing
horrible hacks to get through the ordeal, I vented privately to a few folks
who were having to do the same and also had a few public “wtf?!” comments.
In this instance, it was just incredible bad timing and had the “breaking
features” happened a week later, it probably would not have been a big
deal. It was a stressful 48-72 hours to put it mildly, and the rants were
just me blowing off some steam. Then a few weeks later, omnibus broke
again, then again, etc. I do appreciated the statement ChefCo made
concerning omnibus and what would and would not be “supported” - it was
just a little late for me. Similar things have happened with other parts
of the toolchain.

I am (or was) just an end user of the tool chain. I’ll admit I was enamored
with “oh, shiny new workflow/tools” for a while, but then I needed to get
stuff done. Chef is an awesome tool, but at the end of that day that’s all
it is to me. It’s great that an “ecosystem” is developing around it, but
ultimately I just need to get stuff done. From the outside looking in, it
does seem that the Chef community forgets this sometimes and becomes
enamored with the tools themselves. I include myself in that. Same could
be said about a bunch of open source communities as well.

I’ve taken a different career path – for a while at least – so, I don’t
necessarily have to think about such things every day. A lot of the
"venom" in my original rants was because of 48-72 hours of dealing with
Heartbleed. Heck, I ranted about pretty much every piece of tooling I had
to deal with that day and everyone just assumed all the complaints were
about them However, there was some truth to the rant - “why can’t this
crap just work???”

Hugs to every one. I feel like this should be a discussion at a bar or
something.

As a former co-worker stated: “there are two types of software: software
that @bakins hates and software that @bakins will hate”

–Brian

[Joseph_Bowman]

On the flip side my bash chef cookbook for compiling apache was
incredibly helpful when heartbleed hit and saved me a ton of time. But
yea, that was an extremely stressful time and you wouldn't believe the
wtf moments I had when analyzing vendor provided stacks. Ugh...

On Fri, Jul 11, 2014 at 9:56 AM, Brian Akins brian@akins.org wrote:

Coming late to this thread, but I am one of the folks who had an offline
discussion/rant with lusis.

When Heartbleed (#1) happened, I needed to rebuild a ton of omnibus things
and omnibus was just horribly broken. It had worked fine just a few days
before then it just didn't. In a fury of vendoring things and doing horrible
hacks to get through the ordeal, I vented privately to a few folks who were
having to do the same and also had a few public "wtf?!" comments. In this
instance, it was just incredible bad timing and had the "breaking features"
happened a week later, it probably would not have been a big deal. It was a
stressful 48-72 hours to put it mildly, and the rants were just me blowing
off some steam. Then a few weeks later, omnibus broke again, then again,
etc. I do appreciated the statement ChefCo made concerning omnibus and what
would and would not be "supported" - it was just a little late for me.
Similar things have happened with other parts of the toolchain.

I am (or was) just an end user of the tool chain. I'll admit I was enamored
with "oh, shiny new workflow/tools" for a while, but then I needed to get
stuff done. Chef is an awesome tool, but at the end of that day that's all
it is to me. It's great that an "ecosystem" is developing around it, but
ultimately I just need to get stuff done. From the outside looking in, it
does seem that the Chef community forgets this sometimes and becomes
enamored with the tools themselves. I include myself in that. Same could be
said about a bunch of open source communities as well.

I've taken a different career path -- for a while at least -- so, I don't
necessarily have to think about such things every day. A lot of the "venom"
in my original rants was because of 48-72 hours of dealing with Heartbleed.
Heck, I ranted about pretty much every piece of tooling I had to deal with
that day and everyone just assumed all the complaints were about them
However, there was some truth to the rant - "why can't this crap just
work???"

Hugs to every one. I feel like this should be a discussion at a bar or
something.

As a former co-worker stated: "there are two types of software: software
that @bakins hates and software that @bakins will hate"

--Brian

[mhenrixon]

My experiences are the opposite of Joseph's! It took me around 20 minutes to fix the heart bleed bug and when people in the company started posting about it in the chat rooms I could say it's been fixed for a week already and this is thanks to the amazing tooling that chef brings to the table (berkshelf and friends).

On Fri, Jul 11, 2014 at 3:56 PM, Brian Akins brian@akins.org wrote:

Coming late to this thread, but I am one of the folks who had an offline
discussion/rant with lusis.
When Heartbleed (#1) happened, I needed to rebuild a ton of omnibus things
and omnibus was just horribly broken. It had worked fine just a few days
before then it just didn't. In a fury of vendoring things and doing
horrible hacks to get through the ordeal, I vented privately to a few folks
who were having to do the same and also had a few public "wtf?!" comments.
In this instance, it was just incredible bad timing and had the "breaking
features" happened a week later, it probably would not have been a big
deal. It was a stressful 48-72 hours to put it mildly, and the rants were
just me blowing off some steam. Then a few weeks later, omnibus broke
again, then again, etc. I do appreciated the statement ChefCo made
concerning omnibus and what would and would not be "supported" - it was
just a little late for me. Similar things have happened with other parts
of the toolchain.
I am (or was) just an end user of the tool chain. I'll admit I was enamored
with "oh, shiny new workflow/tools" for a while, but then I needed to get
stuff done. Chef is an awesome tool, but at the end of that day that's all
it is to me. It's great that an "ecosystem" is developing around it, but
ultimately I just need to get stuff done. From the outside looking in, it
does seem that the Chef community forgets this sometimes and becomes
enamored with the tools themselves. I include myself in that. Same could
be said about a bunch of open source communities as well.
I've taken a different career path -- for a while at least -- so, I don't
necessarily have to think about such things every day. A lot of the
"venom" in my original rants was because of 48-72 hours of dealing with
Heartbleed. Heck, I ranted about pretty much every piece of tooling I had
to deal with that day and everyone just assumed all the complaints were
about them However, there was some truth to the rant - "why can't this
crap just work???"
Hugs to every one. I feel like this should be a discussion at a bar or
something.
As a former co-worker stated: "there are two types of software: software
that @bakins hates and software that @bakins will hate"
--Brian

[Joseph_Bowman]

No, chef saved me a ton of time fixing heartbleed where I used chef,
sorry I wasn't clear. My pain was in the stacks that were not managed
by chef.

On 7/11/14, Mikael Henriksson mikael@zoolutions.se wrote:

My experiences are the opposite of Joseph's! It took me around 20 minutes to
fix the heart bleed bug and when people in the company started posting about
it in the chat rooms I could say it's been fixed for a week already and this
is thanks to the amazing tooling that chef brings to the table (berkshelf
and friends).

On Fri, Jul 11, 2014 at 3:56 PM, Brian Akins brian@akins.org wrote:

Coming late to this thread, but I am one of the folks who had an offline
discussion/rant with lusis.
When Heartbleed (#1) happened, I needed to rebuild a ton of omnibus
things
and omnibus was just horribly broken. It had worked fine just a few days
before then it just didn't. In a fury of vendoring things and doing
horrible hacks to get through the ordeal, I vented privately to a few
folks
who were having to do the same and also had a few public "wtf?!"
comments.
In this instance, it was just incredible bad timing and had the "breaking
features" happened a week later, it probably would not have been a big
deal. It was a stressful 48-72 hours to put it mildly, and the rants
were
just me blowing off some steam. Then a few weeks later, omnibus broke
again, then again, etc. I do appreciated the statement ChefCo made
concerning omnibus and what would and would not be "supported" - it was
just a little late for me. Similar things have happened with other parts
of the toolchain.
I am (or was) just an end user of the tool chain. I'll admit I was
enamored
with "oh, shiny new workflow/tools" for a while, but then I needed to get
stuff done. Chef is an awesome tool, but at the end of that day that's
all
it is to me. It's great that an "ecosystem" is developing around it, but
ultimately I just need to get stuff done. From the outside looking in,
it
does seem that the Chef community forgets this sometimes and becomes
enamored with the tools themselves. I include myself in that. Same could
be said about a bunch of open source communities as well.
I've taken a different career path -- for a while at least -- so, I don't
necessarily have to think about such things every day. A lot of the
"venom" in my original rants was because of 48-72 hours of dealing with
Heartbleed. Heck, I ranted about pretty much every piece of tooling I had
to deal with that day and everyone just assumed all the complaints were
about them However, there was some truth to the rant - "why can't
this
crap just work???"
Hugs to every one. I feel like this should be a discussion at a bar or
something.
As a former co-worker stated: "there are two types of software: software
that @bakins hates and software that @bakins will hate"
--Brian

[mhenrixon]

Ah that explains a few things, hugs back at ya and I agree. This discussion would have been amazing at a bar

On Fri, Jul 11, 2014 at 4:10 PM, Joseph Bowman bowman.joseph@gmail.com
wrote:

No, chef saved me a ton of time fixing heartbleed where I used chef,
sorry I wasn't clear. My pain was in the stacks that were not managed
by chef.
On 7/11/14, Mikael Henriksson mikael@zoolutions.se wrote:

My experiences are the opposite of Joseph's! It took me around 20 minutes to
fix the heart bleed bug and when people in the company started posting about
it in the chat rooms I could say it's been fixed for a week already and this
is thanks to the amazing tooling that chef brings to the table (berkshelf
and friends).

On Fri, Jul 11, 2014 at 3:56 PM, Brian Akins brian@akins.org wrote:

Coming late to this thread, but I am one of the folks who had an offline
discussion/rant with lusis.
When Heartbleed (#1) happened, I needed to rebuild a ton of omnibus
things
and omnibus was just horribly broken. It had worked fine just a few days
before then it just didn't. In a fury of vendoring things and doing
horrible hacks to get through the ordeal, I vented privately to a few
folks
who were having to do the same and also had a few public "wtf?!"
comments.
In this instance, it was just incredible bad timing and had the "breaking
features" happened a week later, it probably would not have been a big
deal. It was a stressful 48-72 hours to put it mildly, and the rants
were
just me blowing off some steam. Then a few weeks later, omnibus broke
again, then again, etc. I do appreciated the statement ChefCo made
concerning omnibus and what would and would not be "supported" - it was
just a little late for me. Similar things have happened with other parts
of the toolchain.
I am (or was) just an end user of the tool chain. I'll admit I was
enamored
with "oh, shiny new workflow/tools" for a while, but then I needed to get
stuff done. Chef is an awesome tool, but at the end of that day that's
all
it is to me. It's great that an "ecosystem" is developing around it, but
ultimately I just need to get stuff done. From the outside looking in,
it
does seem that the Chef community forgets this sometimes and becomes
enamored with the tools themselves. I include myself in that. Same could
be said about a bunch of open source communities as well.
I've taken a different career path -- for a while at least -- so, I don't
necessarily have to think about such things every day. A lot of the
"venom" in my original rants was because of 48-72 hours of dealing with
Heartbleed. Heck, I ranted about pretty much every piece of tooling I had
to deal with that day and everyone just assumed all the complaints were
about them However, there was some truth to the rant - "why can't
this
crap just work???"
Hugs to every one. I feel like this should be a discussion at a bar or
something.
As a former co-worker stated: "there are two types of software: software
that @bakins hates and software that @bakins will hate"
--Brian

[adamhjk]

Still can be - come to the summit.
On Jul 11, 2014 7:14 AM, "Mikael Henriksson" mikael@zoolutions.se wrote:

Ah that explains a few things, hugs back at ya and I agree. This
discussion would have been amazing at a bar

On Fri, Jul 11, 2014 at 4:10 PM, Joseph Bowman bowman.joseph@gmail.com
wrote:

No, chef saved me a ton of time fixing heartbleed where I used chef,
sorry I wasn't clear. My pain was in the stacks that were not managed
by chef.

On 7/11/14, Mikael Henriksson mikael@zoolutions.se wrote:

My experiences are the opposite of Joseph's! It took me around 20
minutes to
fix the heart bleed bug and when people in the company started posting
about
it in the chat rooms I could say it's been fixed for a week already and
this
is thanks to the amazing tooling that chef brings to the table
(berkshelf
and friends).

On Fri, Jul 11, 2014 at 3:56 PM, Brian Akins brian@akins.org wrote:

Coming late to this thread, but I am one of the folks who had an
offline
discussion/rant with lusis.
When Heartbleed (#1) happened, I needed to rebuild a ton of omnibus
things
and omnibus was just horribly broken. It had worked fine just a few
days
before then it just didn't. In a fury of vendoring things and doing
horrible hacks to get through the ordeal, I vented privately to a few
folks
who were having to do the same and also had a few public "wtf?!"
comments.
In this instance, it was just incredible bad timing and had the
"breaking
features" happened a week later, it probably would not have been a big
deal. It was a stressful 48-72 hours to put it mildly, and the rants
were
just me blowing off some steam. Then a few weeks later, omnibus broke
again, then again, etc. I do appreciated the statement ChefCo made
concerning omnibus and what would and would not be "supported" - it
was
just a little late for me. Similar things have happened with other
parts
of the toolchain.
I am (or was) just an end user of the tool chain. I'll admit I was
enamored
with "oh, shiny new workflow/tools" for a while, but then I needed to
get
stuff done. Chef is an awesome tool, but at the end of that day that's
all
it is to me. It's great that an "ecosystem" is developing around it,
but
ultimately I just need to get stuff done. From the outside looking in,
it
does seem that the Chef community forgets this sometimes and becomes
enamored with the tools themselves. I include myself in that. Same
could
be said about a bunch of open source communities as well.
I've taken a different career path -- for a while at least -- so, I
don't
necessarily have to think about such things every day. A lot of the
"venom" in my original rants was because of 48-72 hours of dealing
with
Heartbleed. Heck, I ranted about pretty much every piece of tooling I
had
to deal with that day and everyone just assumed all the complaints
were
about them However, there was some truth to the rant - "why can't
this
crap just work???"
Hugs to every one. I feel like this should be a discussion at a bar or
something.
As a former co-worker stated: "there are two types of software:
software
that @bakins hates and software that @bakins will hate"
--Brian

[kallistec]

This is a bit more about omnibus than berkshelf, straying maybe a bit off topic, but here goes.

On Friday, July 11, 2014 at 6:56 AM, Brian Akins wrote:

Coming late to this thread, but I am one of the folks who had an offline discussion/rant with lusis.

When Heartbleed (#1) happened, I needed to rebuild a ton of omnibus things and omnibus was just horribly broken. It had worked fine just a few days before then it just didn't. In a fury of vendoring things and doing horrible hacks to get through the ordeal, I vented privately to a few folks who were having to do the same and also had a few public "wtf?!" comments. In this instance, it was just incredible bad timing and had the "breaking features" happened a week later, it probably would not have been a big deal. It was a stressful 48-72 hours to put it mildly, and the rants were just me blowing off some steam. Then a few weeks later, omnibus broke again, then again, etc. I do appreciated the statement ChefCo made concerning omnibus and what would and would not be "supported" - it was just a little late for me. Similar things have happened with other parts of the toolchain.

This has always been the most confusing part of this (for lack of a better word) disagreement to me. My initial reaction to this is, “why would you go through a major breaking change just to get a newer version of OpenSSL?”. All of the breaking changes in omnibus followed the semver guidelines, why not just pin to “~> 1.0” ?

I have to imagine that you would’ve done that if you thought you could, though. So I’m guessing that you pulled down the latest omnibus-software, that project had a dependency on new omnibus, and you felt like that was your best or maybe only option to accomplish your goal of shipping you project with a patched OpenSSL. Clearly, that sucks.

I think there’s a couple of things we should consider to make this better in the future:

1. We announced on our blog that we’re only going to support a very narrow scope of functionality in omnibus-software, but I think it’s the right thing to do for us to make an exception for severe security issues like heart bleed. For example, we could have provided a branch of omnibus-software that was omnibus 1.0 compatible that had just the heartbleed patch. This is something we did anyway, because we have a lot of omnibus projects and we don’t keep them all on the bleeding edge.

2. There’s a huge difference in how we at Chef Software think about the “build lab” feature of omnibus and how a lot of people use it. To build our own products, we have a jenkins cluster with long-lived build slaves (on a mishmash of clouds and some physical hardware) that run the omnibus builds and move the artifacts around. The “build lab” stuff is only used for development. Lots of people outside Chef Software seem to be using the build lab as the thing that builds the artifacts they release to their users. Given that understanding, we’ll be more conscientious about changing the build lab stuff in the future. Also, if you want to have an optional different configuration for the build lab that uses different tools, then submit it as a patch.

3. Instead of seething privately or subtweeting us, find us on IRC, post to this list, create a bug, or non-sub-tweet at us (or all four). We could’ve gotten you a minimal patch for omnibus-software that worked with your existing tools (we did exactly that for our own needs) to solve the immediate problem with a lot less hassle.

Hugs to every one. I feel like this should be a discussion at a bar or something.

--Brian
Like Adam said, come to the summit and we can

--
Daniel DeLeo

[Lamont_Granquist_OLD]

On 7/7/14, 8:05 PM, John E. Vincent (lusis) wrote:

FWIW I didn't have a problem with Berkshelf originally. One: I didn't
use it and didn't HAVE to use it. Two: It didn't have this entirely
new service it depended on.

Now I can't avoid it and it requires a separate internet service
(though now apparently rolled into the supermarket) just to be used.
Just addressing this one issue...

I hated the Berkshelf API with a passion initially. I still think its a
hack, but its a hack around problems that needed to be solved. The
first problem is just speed. Berkshelf needs to depsolve against all
the cookbooks on the community site, and doing that with a single
request for every cookbook was slow. So the /universe endpoint was
created to slurp it all down. That endpoint is now integrated with the
community site, and what is going on is essentially exactly what happens
when you use 'gem install' and it goes and hits rubygems to pull down
the list of current gems so that the rubygems depsolver can figure out
what gems to install. Its that problem, and its now solved in a very
similar fashion. There's another obvious source of cookbooks -- the
chef server -- that you should be able to depsolve against as well, and
we should add a universe endpoint (probably
/organizations//cookbook_universe or something) to the chef-server
to solve it.

The other problem is one dealing with metadata.rb and metadata.json that
has its roots in configuration-as-code. The problem is that if you ship
around metadata.rb and it has pure-ruby code in it that references
external data you will not necessarily get the same metadata.json result
when you evaluate it. There are two chef tickets on this:

https://tickets.opscode.com/browse/CHEF-4810
https://tickets.opscode.com/browse/CHEF-4811

There's also Jamie's blog post on it:

http://blog.vialstudios.com/the-importance-of-compiled-metadata/

And there's my issue that I opened against berkshelf where you can see
it tied all together:

Respect transitive dependencies on git branches over community site

The result of that leads to the lack of being able to use transitive
github dependencies and the need to serve your own API server. The use
most important case which is driving it is an important one which is
that people are using ruby code in metadata.rb to drive their version
numbers off of CI/CD (e.g. jenkins) and in that case the artifacts need
to have compiled metadata.json so that all the actors involved in the
system agree on the version number in the metadata. Since CI/CD is
important, we don't want to break that.

Eliminating the need for 'finalized artifacts' and API servers may not
be a use case that you care about, but we can't just break the users
that do care about it because you don't. My initial reaction to the
problem is just to do a (tableflip) on configuration-as-code and think
to myself that we should deprecate metadata.rb and mandate using
metadata.json (back-to-the-future time). There might be a better way to
address the problem, though, which looks at the CD/CI use case and fixes
their version numbering use case a better way. I'm somewhat optimistic
that in the future that the berks API server will have completely gone
away.

I'd also be interested in solutions like baking supermarket into every
chef server deploy. Then everyone would have a local repo to push
cookbooks to and a /universe endpoint. Then that just becomes your
local berks API server.

So, if I've got any point here its that I'm very grouchy about the berks
API server because I see it as something which is a barrier to entry of
berkshelf adoption and its a problem that need to get solved. But
being grouchy about it without understanding why its there (Jamie is
pretty smart, he didn't just write it because he felt like writing some
server code) is not being productive. We need to understand the reasons
why things are there, and the use cases that drive them and come up with
solutions to the problems, rather than just complain. Just complaining
about the berks API is bad signalling since it just causes flak to the
people working on berkshelf. What you want to signal is the chef
server team that they should integrate the /universe endpoint, or that
supermarket should be bundled with chef server, or other solutions like
that. What the chef server team hears right now is that Berkshelf sucks,
which sounds entirely like a Berkshelf problem. That's not useful in
helping them to determine what they could do better with the server code
or packaging to help solve that problem -- and I'm fairly convinced that
this is an ecosystem problem, and that Berkshelf is what it is because
its been developed in isolation from work going on in the client and
server, and that needs to change in order to make it better.

[jeffbyrnes]

It’d be absolutely stellar if the Chef Server also provided a universe endpoint; that would obviate our need for a Berkshelf API server entirely, especially since we don’t have any particular need for our own Supermarket right now. The alternative of having Supermarket baked-in to the Chef Server would also be welcome.

On the larger topic of Chef providing us with The Way to author cookbooks, I agree with Dylan’s points that we don’t necessarily need a One True Way, but some more direction on a Few Good Ways would be very welcome. And I wholeheartedly agree that the website examples should be solid examples.

--
Jeff Byrnes
@berkleebassist
Operations Engineer
EverTrue
704.516.4628

On July 8, 2014 at 1:20:57 PM, Lamont Granquist (lamont@opscode.com) wrote:

On 7/7/14, 8:05 PM, John E. Vincent (lusis) wrote:

FWIW I didn't have a problem with Berkshelf originally. One: I didn't
use it and didn't HAVE to use it. Two: It didn't have this entirely
new service it depended on.

Now I can't avoid it and it requires a separate internet service
(though now apparently rolled into the supermarket) just to be used.
Just addressing this one issue...

I hated the Berkshelf API with a passion initially. I still think its a
hack, but its a hack around problems that needed to be solved. The
first problem is just speed. Berkshelf needs to depsolve against all
the cookbooks on the community site, and doing that with a single
request for every cookbook was slow. So the /universe endpoint was
created to slurp it all down. That endpoint is now integrated with the
community site, and what is going on is essentially exactly what happens
when you use 'gem install' and it goes and hits rubygems to pull down
the list of current gems so that the rubygems depsolver can figure out
what gems to install. Its that problem, and its now solved in a very
similar fashion. There's another obvious source of cookbooks -- the
chef server -- that you should be able to depsolve against as well, and
we should add a universe endpoint (probably
/organizations//cookbook_universe or something) to the chef-server
to solve it.

The other problem is one dealing with metadata.rb and metadata.json that
has its roots in configuration-as-code. The problem is that if you ship
around metadata.rb and it has pure-ruby code in it that references
external data you will not necessarily get the same metadata.json result
when you evaluate it. There are two chef tickets on this:

https://tickets.opscode.com/browse/CHEF-4810
https://tickets.opscode.com/browse/CHEF-4811

There's also Jamie's blog post on it:

http://blog.vialstudios.com/the-importance-of-compiled-metadata/

And there's my issue that I opened against berkshelf where you can see
it tied all together:

Respect transitive dependencies on git branches over community site

The result of that leads to the lack of being able to use transitive
github dependencies and the need to serve your own API server. The use
most important case which is driving it is an important one which is
that people are using ruby code in metadata.rb to drive their version
numbers off of CI/CD (e.g. jenkins) and in that case the artifacts need
to have compiled metadata.json so that all the actors involved in the
system agree on the version number in the metadata. Since CI/CD is
important, we don't want to break that.

Eliminating the need for 'finalized artifacts' and API servers may not
be a use case that you care about, but we can't just break the users
that do care about it because you don't. My initial reaction to the
problem is just to do a (tableflip) on configuration-as-code and think
to myself that we should deprecate metadata.rb and mandate using
metadata.json (back-to-the-future time). There might be a better way to
address the problem, though, which looks at the CD/CI use case and fixes
their version numbering use case a better way. I'm somewhat optimistic
that in the future that the berks API server will have completely gone
away.

I'd also be interested in solutions like baking supermarket into every
chef server deploy. Then everyone would have a local repo to push
cookbooks to and a /universe endpoint. Then that just becomes your
local berks API server.

So, if I've got any point here its that I'm very grouchy about the berks
API server because I see it as something which is a barrier to entry of
berkshelf adoption and its a problem that need to get solved. But
being grouchy about it without understanding why its there (Jamie is
pretty smart, he didn't just write it because he felt like writing some
server code) is not being productive. We need to understand the reasons
why things are there, and the use cases that drive them and come up with
solutions to the problems, rather than just complain. Just complaining
about the berks API is bad signalling since it just causes flak to the
people working on berkshelf. What you want to signal is the chef
server team that they should integrate the /universe endpoint, or that
supermarket should be bundled with chef server, or other solutions like
that. What the chef server team hears right now is that Berkshelf sucks,
which sounds entirely like a Berkshelf problem. That's not useful in
helping them to determine what they could do better with the server code
or packaging to help solve that problem -- and I'm fairly convinced that
this is an ecosystem problem, and that Berkshelf is what it is because
its been developed in isolation from work going on in the client and
server, and that needs to change in order to make it better.

[jdunn]

On Tue, Jul 8, 2014 at 1:51 PM, Jeff Byrnes jeff@evertrue.com wrote:

On the larger topic of Chef providing us with The Way to author cookbooks, I
agree with Dylan’s points that we don’t necessarily need a One True Way, but
some more direction on a Few Good Ways would be very welcome. And I
wholeheartedly agree that the website examples should be solid examples.

I totally agree with you. Our barrier is often that engineers hate
writing, so we have people on board now like Thomas Petchel (in charge
of the recent rebuild of learnchef) to make things more consumable for
end users. (One of the complaints I had about learnchef v1 is that it
made you eat the whole bowl on day 1, and now it is definitely more
crawl-walk-run oriented.)

But I would like to see something like Lamont's recent adventures in
modern cookbook authoring [1] turned into a scenario on learnchef.com
or something similar.

• Julian

[1] getting_started.md · GitHub

--
[ Julian C. Dunn jdunn@aquezada.com * Sorry, I'm ]
[ WWW: Julian Dunn's Blog - Commentary on media, technology, and everything in between. * only Web 1.0 ]
[ gopher://sdf.org/1/users/keymaker/ * compliant! ]
[ PGP: 91B3 7A9D 683C 7C16 715F 442C 6065 D533 FDC2 05B9 ]

[John_E_Vincent_lusis]

So that post from Lamont is awesome but it kinda points to the thing I was
bringing up. This section right here:

"This is the correct way to start building Chef Cookbooks"

And then

"Not that we're not using Test Kitchen to actually test our cookbook."

Wait what? Then why is it "Test" Kitchen?

When I asked all the why questions, this is a good example. Why shouldn't
I use the documented knife cookbook create? Is it not acceptable to the
job of creating a cookbook? What makes this more confusing is that this is
coming from someone who works for Chef.

On Tue, Jul 8, 2014 at 2:01 PM, Julian C. Dunn jdunn@aquezada.com wrote:

On Tue, Jul 8, 2014 at 1:51 PM, Jeff Byrnes jeff@evertrue.com wrote:

On the larger topic of Chef providing us with The Way to author
cookbooks, I
agree with Dylan’s points that we don’t necessarily need a One True Way,
but
some more direction on a Few Good Ways would be very welcome. And I
wholeheartedly agree that the website examples should be solid examples.

I totally agree with you. Our barrier is often that engineers hate
writing, so we have people on board now like Thomas Petchel (in charge
of the recent rebuild of learnchef) to make things more consumable for
end users. (One of the complaints I had about learnchef v1 is that it
made you eat the whole bowl on day 1, and now it is definitely more
crawl-walk-run oriented.)

But I would like to see something like Lamont's recent adventures in
modern cookbook authoring [1] turned into a scenario on learnchef.com
or something similar.

• Julian

[1] getting_started.md · GitHub

--
[ Julian C. Dunn jdunn@aquezada.com * Sorry, I'm ]
[ WWW: Julian Dunn's Blog - Commentary on media, technology, and everything in between. * only Web 1.0 ]
[ gopher://sdf.org/1/users/keymaker/ * compliant! ]
[ PGP: 91B3 7A9D 683C 7C16 715F 442C 6065 D533 FDC2 05B9 ]

[John_E_Vincent_lusis]

Something I wanted to bring up here is that much of the Berks API could
have been clarified with a communicated roadmap.

"Hey folks, we're going to start pushing the Berkshelf API as the canonical
depsolver for cookbooks. The plan is to run a copy on Heroku, then run a
copy as part of supermarket and eventually move it into Chef Server"

I think I would have been much "happier" with that.

But instead, I'm trying to figure out why I need this thing that's being
pushed that isn't a part of Chef. Note that my concerns about the Berkshelf
API endpoint is that it's yet another service I need to run myself since I
need it to be available all the time. I have a Chef Server. Shouldn't I be
able to use that? I know the answer to that now but this was part of my
original complaint/concern. To be clear, we're actively moving to NOT go to
the internet for anything at this point. We mirror all artifacts,
cookbooks, python packages, rubygems, java deps, go deps - everything. Why?
Because the internet sucks as a source of truth and I get woken up at 3AM
because a chef run failed because pypi was offline or the Berkshelf API is
down again or any number of reasons. So every dependency matters.

On Tue, Jul 8, 2014 at 1:20 PM, Lamont Granquist lamont@opscode.com wrote:

On 7/7/14, 8:05 PM, John E. Vincent (lusis) wrote:

FWIW I didn't have a problem with Berkshelf originally. One: I didn't use
it and didn't HAVE to use it. Two: It didn't have this entirely new service
it depended on.

Now I can't avoid it and it requires a separate internet service (though
now apparently rolled into the supermarket) just to be used.

Just addressing this one issue...

I hated the Berkshelf API with a passion initially. I still think its a
hack, but its a hack around problems that needed to be solved. The first
problem is just speed. Berkshelf needs to depsolve against all the
cookbooks on the community site, and doing that with a single request for
every cookbook was slow. So the /universe endpoint was created to slurp
it all down. That endpoint is now integrated with the community site, and
what is going on is essentially exactly what happens when you use 'gem
install' and it goes and hits rubygems to pull down the list of current
gems so that the rubygems depsolver can figure out what gems to install.
Its that problem, and its now solved in a very similar fashion. There's
another obvious source of cookbooks -- the chef server -- that you should
be able to depsolve against as well, and we should add a universe endpoint
(probably /organizations//cookbook_universe or something) to the
chef-server to solve it.

The other problem is one dealing with metadata.rb and metadata.json that
has its roots in configuration-as-code. The problem is that if you ship
around metadata.rb and it has pure-ruby code in it that references external
data you will not necessarily get the same metadata.json result when you
evaluate it. There are two chef tickets on this:

https://tickets.opscode.com/browse/CHEF-4810
https://tickets.opscode.com/browse/CHEF-4811

There's also Jamie's blog post on it:

http://blog.vialstudios.com/the-importance-of-compiled-metadata/

And there's my issue that I opened against berkshelf where you can see it
tied all together:

Respect transitive dependencies on git branches over community site · Issue #1126 · berkshelf/berkshelf · GitHub

The result of that leads to the lack of being able to use transitive
github dependencies and the need to serve your own API server. The use
most important case which is driving it is an important one which is that
people are using ruby code in metadata.rb to drive their version numbers
off of CI/CD (e.g. jenkins) and in that case the artifacts need to have
compiled metadata.json so that all the actors involved in the system agree
on the version number in the metadata. Since CI/CD is important, we don't
want to break that.

Eliminating the need for 'finalized artifacts' and API servers may not be
a use case that you care about, but we can't just break the users that do
care about it because you don't. My initial reaction to the problem is
just to do a (tableflip) on configuration-as-code and think to myself that
we should deprecate metadata.rb and mandate using metadata.json
(back-to-the-future time). There might be a better way to address the
problem, though, which looks at the CD/CI use case and fixes their version
numbering use case a better way. I'm somewhat optimistic that in the future
that the berks API server will have completely gone away.

I'd also be interested in solutions like baking supermarket into every
chef server deploy. Then everyone would have a local repo to push
cookbooks to and a /universe endpoint. Then that just becomes your local
berks API server.

So, if I've got any point here its that I'm very grouchy about the berks
API server because I see it as something which is a barrier to entry of
berkshelf adoption and its a problem that need to get solved. But being
grouchy about it without understanding why its there (Jamie is pretty
smart, he didn't just write it because he felt like writing some server
code) is not being productive. We need to understand the reasons why
things are there, and the use cases that drive them and come up with
solutions to the problems, rather than just complain. Just complaining
about the berks API is bad signalling since it just causes flak to the
people working on berkshelf. What you want to signal is the chef server
team that they should integrate the /universe endpoint, or that supermarket
should be bundled with chef server, or other solutions like that. What the
chef server team hears right now is that Berkshelf sucks, which sounds
entirely like a Berkshelf problem. That's not useful in helping them to
determine what they could do better with the server code or packaging to
help solve that problem -- and I'm fairly convinced that this is an
ecosystem problem, and that Berkshelf is what it is because its been
developed in isolation from work going on in the client and server, and
that needs to change in order to make it better.

[zts]

On Tue, Jul 8, 2014 at 6:20 PM, Lamont Granquist lamont@opscode.com wrote:

Just complaining about the berks API is bad signalling since it just
causes flak to the people working on berkshelf. What you want to signal
is the chef server team that they should integrate the /universe endpoint,
or that supermarket should be bundled with chef server, or other solutions
like that. What the chef server team hears right now is that Berkshelf
sucks, which sounds entirely like a Berkshelf problem. That's not useful
in helping them to determine what they could do better with the server code
or packaging to help solve that problem -- and I'm fairly convinced that
this is an ecosystem problem, and that Berkshelf is what it is because its
been developed in isolation from work going on in the client and server,
and that needs to change in order to make it better.

I've refrained from saying much publicly about the berks API precisely
because I don't want to give flak to the people working on Berkshelf - that
said, I would very much like to give a little flack to Chef (the company)
about it. Although the PR[1] announcing corporate stewardship of the
project didn't say much about what that actually meant, I figured that it
would mean a little less itch-scratching and a little more thought about
how users would be adversely affected by changes.

Then the vagrant-berkshelf plugin was deprecated[2]. I understand why an
open source team would do so, and I don't fault them for it - on the other
hand, I would criticise Chef (the company) for that same decision. I can
only assume that Chef felt that their stewardship of Berkshelf didn't
extend to vagrant-berkshelf - which is a shame, since test-kitchen isn't a
replacement for every use case.

Onto Berkshelf 3, the first major release under Chef's stewardship.
Previously, I was perfectly happy using Hosted Chef as an artifact
repository in my Berkshelf workflow. With Berks 3, I now have to run my
own berks API server to preserve that functionality. I chose Hosted Chef
to avoid running my own Chef server, so this really grates[3]. Again, I
understand this decision by an open source team, but I'm left wondering
where the corporate stewardship comes into it.

Here's the thing: Berkshelf 3 is great upgrade. I put off updating to it
for ages because of the amount of work involved, I still grumble about the
API server every other day, but it's a net win all the same. The
individuals working on it deserve a great big round of applause.

So, here's my message to the Chef Server team (and Chef, the company, at
large): Berkshelf 3 is a great tool, but it's let down by the fact that the
API server as a separate dependency. Please roll the API server into Chef
Server - especially Hosted Chef - so that I can stop saying "Berkshelf 3 is
a great too, but..." and start saying "Berkshelf 3 is a great tool!"

Thankyou,

Zac

1: Opscode To Steward Berkshelf - Chef Blog | Chef
2: The future of Vagrant Berkshelf | Seth Vargo
3: for the moment, I only use Hosted Chef as an artifact repository - and
pay nothing for the privilege. I'd have made a lot more noise about this
if I felt entitled to complain.

[jdunn]

So I'll let Lamont defend himself too but let's be clear that this is
just a thing he wrote to talk about the state of the union & his
experiments in it. If it were more than that, it'd be more than a
Gist.

This (the m/l) is the right forum for such questions & criticisms though.

On Tue, Jul 8, 2014 at 2:16 PM, John E. Vincent (lusis)
lusis.org+chef-list@gmail.com wrote:

So that post from Lamont is awesome but it kinda points to the thing I was
bringing up. This section right here:

"This is the correct way to start building Chef Cookbooks"

The text reads "This is the correct way to start building Chef
Cookbooks and leverage ChefDK and Test Kitchen as of this writing." so
it could also be construed as "here is how you use ChefDK and Test
Kitchen to build modern cookbooks". I'm not sure that the use of the
word "correct" carries some official endorsement or
"speaking-on-behalf-of-ChefCo" weight to it.

And then

"Not that we're not using Test Kitchen to actually test our cookbook."

Wait what? Then why is it "Test" Kitchen?

This statement reflects that there are no formal integration tests
written as part of the exercise. The document does walk you through
converging the node with the cookbook as written, so that could be
construed as enough of a "test" depending on your needs.

When I asked all the why questions, this is a good example. Why shouldn't
I use the documented knife cookbook create? Is it not acceptable to the
job of creating a cookbook?

"knife cookbook create" doesn't include all the unit and integration
test harnesses that make up a modern cookbook today. It also doesn't
allow you to customize the template from which a cookbook is
generated. 'chef generate' does.

What makes this more confusing is that this is coming from someone who works for Chef.

How is it more confusing? I'm not sure it's realistic to assume that
every email coming from an @opscode.com or @getchef.com domain
represents some kind of all-encompassing ChefCo position about the
toolchain.

• Julian

--
[ Julian C. Dunn jdunn@aquezada.com * Sorry, I'm ]
[ WWW: http://www.aquezada.com/staff/julian * only Web 1.0 ]
[ gopher://sdf.org/1/users/keymaker/ * compliant! ]
[ PGP: 91B3 7A9D 683C 7C16 715F 442C 6065 D533 FDC2 05B9 ]