Releases, github milestones, etc


#1

Heya devs,

This has come up tangentially in a few PRs, but I think it deserves a real
discussion.

How are we handling github milestones and releases - because it seems a lot
like milestones are not actually paid attention to in releases.

As an example, I tagged https://github.com/chef/chef/pull/3235 with 12.3.0
because it’s a bugfix to 12.2.x and 12.3.0 came out today sans that.

So if we don’t pull in all things with a milestone of X into an X release -
how do we actually as a team know what’s going into a release? Shouldn’t
everything with a milestone either be merged or someone post and say “I don’t
think this is necessary for release X because of reason Y, can we schedule for
release X+1?” and then we change the milestone it’s tagged with?


Phil Dibowitz phil@ipom.com
Open Source software and tech docs Insanity Palace of Metallica
http://www.phildev.net/ http://www.ipom.com/

“Be who you are and say what you feel, because those who mind don’t matter
and those who matter don’t mind.”

  • Dr. Seuss

#2

Hey Phil,
that one didn’t make it to 12.3.0 because we’d already cut the RC last
week. I’ve not had a chance to look at it fully yet, but if it’s urgent
we’ll either pick it up in 12.3.1 or 12.4 in a couple of weeks.
In general, we’re using the issue milestones “Accepted Minor” for work that
we think aren’t breaking changes, “Accepted Major”, for things that we
think will be breaking changes or major work, and “Help Wanted”, for issues
that we’d like help from the community with. We’ve been publishing the list
of Help Wanted issues to the chef-dev list as we assign things to that
bucket.
Hope this helps,
-Thom

On Mon, Apr 27, 2015 at 9:15 PM, Phil Dibowitz phil@ipom.com wrote:

Heya devs,

This has come up tangentially in a few PRs, but I think it deserves a real
discussion.

How are we handling github milestones and releases - because it seems a lot
like milestones are not actually paid attention to in releases.

As an example, I tagged https://github.com/chef/chef/pull/3235 with 12.3.0
because it’s a bugfix to 12.2.x and 12.3.0 came out today sans that.

So if we don’t pull in all things with a milestone of X into an X release -
how do we actually as a team know what’s going into a release? Shouldn’t
everything with a milestone either be merged or someone post and say “I
don’t
think this is necessary for release X because of reason Y, can we schedule
for
release X+1?” and then we change the milestone it’s tagged with?


Phil Dibowitz phil@ipom.com
Open Source software and tech docs Insanity Palace of Metallica
http://www.phildev.net/ http://www.ipom.com/

“Be who you are and say what you feel, because those who mind don’t matter
and those who matter don’t mind.”

  • Dr. Seuss

#3

On 04/28/2015 01:06 AM, Thom May wrote:

Hey Phil,
that one didn’t make it to 12.3.0 because we’d already cut the RC last week.
I’ve not had a chance to look at it fully yet, but if it’s urgent we’ll either
pick it up in 12.3.1 or 12.4 in a couple of weeks.

Well - it’s a breakage, so it’s important in that sense, even though it’s a
relatively new feature not many people are using yet.

But I’m trying to discuss the more general problem here, not this specific PR.
And to be clear I’m not blaming anyone here. I think we have a process problem.

In general, we’re using the issue milestones “Accepted Minor” for work that we
think aren’t breaking changes, “Accepted Major”, for things that we think will
be breaking changes or major work, and “Help Wanted”, for issues that we’d
like help from the community with. We’ve been publishing the list of Help
Wanted issues to the chef-dev list as we assign things to that bucket.
Hope this helps,

I’m assuming your using Issue here interchangably with PR and not expecting
contributors to have to make an issue for every PR? (that would seem
unnecessarily onerous for contributors).

https://github.com/chef/chef-rfc/pull/36 doesn’t actually mention how to get a
PR into a release, but since Chef doesn’t actually release from HEAD (cough
delivery-keynote couch), which is totally fine, we need a defined way to
denote stuff is desired for the next major/minor/bugfix release. Upon previous
discussions with btm, serdar, and others, I was lead to believe milestones on
PRs was that way… but that doesn’t actually seem to be the case.

“Accepted Major” and “Accepted Minor” are not - afaict - sufficient. They
don’t tell me what release my thing is going to be in.

To put it in RFC terms:

As a contributor I should be able to say “I believe this belongs in 12.3”

As a maintainer I should be able to say “I’m sorry, but we’re not going to put
it in 12.3 because it’s big and scary”

As a contributor I should get the chance to know that and not suddenly see a
release without the thing that I added the milestone to.

As a contributor I don’t expect to be able to add a Milestone to a PR that is
already past the freeze date.

As a contributor I don’t expect a release to go out the door with any pending
PRs tagged for it (either the Milestone should have been changed or I should
have been unable to add it).

It’s not about this particular PR, this isn’t an uncommon occurance. In quite
a few cases the only reason I’ve gotten my PR into a specific release is
because I’m badgering Lamont, Adam, Serdar or someone else via email/phonecall
or whatever (or it’s not that important to me and it falls into a release
eventually) enough to make sure someone sees it.

As a maintainer I understand there is a time we cut an RC and want to get a
release out the door, but from the perspective of both an end-user and a
contributor, we’re making a bad experience for people waiting on their
fix/feature/whatever. We need to have a very clear way of a contributor to
(optionally) specify what release they are expecting it to get in and a very
clear way for us to let them know if that’s going to happen.


Phil Dibowitz phil@ipom.com
Open Source software and tech docs Insanity Palace of Metallica
http://www.phildev.net/ http://www.ipom.com/

“Be who you are and say what you feel, because those who mind don’t matter
and those who matter don’t mind.”

  • Dr. Seuss

#4

On Tuesday, April 28, 2015 at 2:13 AM, Phil Dibowitz wrote:

On 04/28/2015 01:06 AM, Thom May wrote:

Hey Phil,
that one didn’t make it to 12.3.0 because we’d already cut the RC last week.
I’ve not had a chance to look at it fully yet, but if it’s urgent we’ll either
pick it up in 12.3.1 or 12.4 in a couple of weeks.

Well - it’s a breakage, so it’s important in that sense, even though it’s a
relatively new feature not many people are using yet.

But I’m trying to discuss the more general problem here, not this specific PR.
And to be clear I’m not blaming anyone here. I think we have a process problem.

In general, we’re using the issue milestones “Accepted Minor” for work that we
think aren’t breaking changes, “Accepted Major”, for things that we think will
be breaking changes or major work, and “Help Wanted”, for issues that we’d
like help from the community with. We’ve been publishing the list of Help
Wanted issues to the chef-dev list as we assign things to that bucket.
Hope this helps,

I’m assuming your using Issue here interchangably with PR and not expecting
contributors to have to make an issue for every PR? (that would seem
unnecessarily onerous for contributors).

https://github.com/chef/chef-rfc/pull/36 doesn’t actually mention how to get a
PR into a release, but since Chef doesn’t actually release from HEAD (cough
delivery-keynote couch), which is totally fine, we need a defined way to
denote stuff is desired for the next major/minor/bugfix release. Upon previous
discussions with btm, serdar, and others, I was lead to believe milestones on
PRs was that way… but that doesn’t actually seem to be the case.

“Accepted Major” and “Accepted Minor” are not - afaict - sufficient. They
don’t tell me what release my thing is going to be in.

To put it in RFC terms:

As a contributor I should be able to say “I believe this belongs in 12.3”

As a maintainer I should be able to say “I’m sorry, but we’re not going to put
it in 12.3 because it’s big and scary”

As a contributor I should get the chance to know that and not suddenly see a
release without the thing that I added the milestone to.

As a contributor I don’t expect to be able to add a Milestone to a PR that is
already past the freeze date.

As a contributor I don’t expect a release to go out the door with any pending
PRs tagged for it (either the Milestone should have been changed or I should
have been unable to add it).

It’s not about this particular PR, this isn’t an uncommon occurance. In quite
a few cases the only reason I’ve gotten my PR into a specific release is
because I’m badgering Lamont, Adam, Serdar or someone else via email/phonecall
or whatever (or it’s not that important to me and it falls into a release
eventually) enough to make sure someone sees it.

As a maintainer I understand there is a time we cut an RC and want to get a
release out the door, but from the perspective of both an end-user and a
contributor, we’re making a bad experience for people waiting on their
fix/feature/whatever. We need to have a very clear way of a contributor to
(optionally) specify what release they are expecting it to get in and a very
clear way for us to let them know if that’s going to happen.

Absolutely agree. In the past we announced RC freezes on the dev list (at least) with some advance notice, usually a week or so. If we just started doing that again (at least until we’re shipping from master), would that be enough?


Phil Dibowitz phil@ipom.com (mailto:phil@ipom.com)
Open Source software and tech docs Insanity Palace of Metallica
http://www.phildev.net/ http://www.ipom.com/

“Be who you are and say what you feel, because those who mind don’t matter
and those who matter don’t mind.”

  • Dr. Seuss


Daniel DeLeo


#5

On 04/30/2015 07:39 AM, Daniel DeLeo wrote:

Absolutely agree. In the past we announced RC freezes on the dev list (at
least) with some advance notice, usually a week or so. If we just started
doing that again (at least until we’re shipping from master), would that be
enough?

I don’t think so. For active developers, that’ll help, but if I’m a user who
writes a diff, the information I need should be in the PR - I shouldn’t have
to correlate the date of the PR with emails. I think the source of truth
should be what milestones I can/did a contributor put on a PR (or some
equivalent).


Phil Dibowitz phil@ipom.com
Open Source software and tech docs Insanity Palace of Metallica
http://www.phildev.net/ http://www.ipom.com/

“Be who you are and say what you feel, because those who mind don’t matter
and those who matter don’t mind.”

  • Dr. Seuss

#6

I don’t think that’s realistic though; that hypothetical contributor can’t
make a good judgement on whether their contribution is gonna go into a
patch release or has to be held until the next major version.
That said, I think it is reasonable to have a tag that allows the
contributor to nominate regressions.

Semi-relatedly, 12.3.0 is just a point in time on master, with two very
trivial changes taken after that fact to ensure we didn’t change a public
api in the release.
So I think it’s reasonable that people know that once their code is merged
to master they’ll get released in the very next minor version we do.

On Thu, Apr 30, 2015 at 5:19 PM, Phil Dibowitz phil@ipom.com wrote:

On 04/30/2015 07:39 AM, Daniel DeLeo wrote:

Absolutely agree. In the past we announced RC freezes on the dev list (at
least) with some advance notice, usually a week or so. If we just started
doing that again (at least until we’re shipping from master), would that
be
enough?

I don’t think so. For active developers, that’ll help, but if I’m a user
who
writes a diff, the information I need should be in the PR - I shouldn’t
have
to correlate the date of the PR with emails. I think the source of truth
should be what milestones I can/did a contributor put on a PR (or some
equivalent).


Phil Dibowitz phil@ipom.com
Open Source software and tech docs Insanity Palace of Metallica
http://www.phildev.net/ http://www.ipom.com/

“Be who you are and say what you feel, because those who mind don’t matter
and those who matter don’t mind.”

  • Dr. Seuss

#7

On Thu, Apr 30, 2015 at 05:49:55PM +0100, Thom May wrote:

I don’t think that’s realistic though; that hypothetical contributor can’t
make a good judgement on whether their contribution is gonna go into a
patch release or has to be held until the next major version.
That said, I think it is reasonable to have a tag that allows the
contributor to nominate regressions.

Many contributors won’t… but if you’re not even looking at milestones, how
are you even sure you’re merging in the things you want to merge to a given
release. We need a consistent way to tag stuff for a release so that Chef can
do a search, ensure everything is merged.

Let’s say there was a regression in the RC and I sent a PR, merge it, and tag
it for release, but you don’t merge it. Oops, now we just released a
regression we didn’t have to. The argument of “oh, but we’ll know about the
regression so we’ll cherry pick it ourselves” isn’t acceptable. The whole
point of having maintainers and open development system is so there’s
transparency and consistency. EIther we’re using milestones in GH or we’re
not, but we need a system that’s consistent.


Phil Dibowitz phil@ipom.com
Open Source software and tech docs Insanity Palace of Metallica
http://www.phildev.net/ http://www.ipom.com/

“Be who you are and say what you feel, because those who mind don’t matter
and those who matter don’t mind.”

  • Dr. Seuss

#8

On 05/01/2015 05:54, Phil Dibowitz wrote:

On Thu, Apr 30, 2015 at 05:49:55PM +0100, Thom May wrote:

I don’t think that’s realistic though; that hypothetical contributor can’t
make a good judgement on whether their contribution is gonna go into a
patch release or has to be held until the next major version.
That said, I think it is reasonable to have a tag that allows the
contributor to nominate regressions.
Many contributors won’t… but if you’re not even looking at milestones, how
are you even sure you’re merging in the things you want to merge to a given
release. We need a consistent way to tag stuff for a release so that Chef can
do a search, ensure everything is merged.

Let’s say there was a regression in the RC and I sent a PR, merge it, and tag
it for release, but you don’t merge it. Oops, now we just released a
regression we didn’t have to. The argument of “oh, but we’ll know about the
regression so we’ll cherry pick it ourselves” isn’t acceptable. The whole
point of having maintainers and open development system is so there’s
transparency and consistency. EIther we’re using milestones in GH or we’re
not, but we need a system that’s consistent.

I don’t think it’s sane to for every contributor to be allowed to decide

  • based on their own value judgment, that may or may not be sound - when
    their PR should get merged. This is likely to result in at best noise,
    and at worst chaos, since everyone wants their stuff to go in as soon as
    possible, and often only the project maintainers have any broad sense of
    the state of the projects, and the impact of other items in the merge queue.

I do agree that milestones add useful visibility for all involved
though. Contributor settable tags for enhancement, bugfix and
regression do make sense - the review team can use them to prioritize
regressions and then /they/ should bin issues into a milestone.


#9

On Fri, May 01, 2015 at 11:00:53AM +1000, Peter Fern wrote:

On 05/01/2015 05:54, Phil Dibowitz wrote:

On Thu, Apr 30, 2015 at 05:49:55PM +0100, Thom May wrote:

I don’t think that’s realistic though; that hypothetical contributor can’t
make a good judgement on whether their contribution is gonna go into a
patch release or has to be held until the next major version.
That said, I think it is reasonable to have a tag that allows the
contributor to nominate regressions.
Many contributors won’t… but if you’re not even looking at milestones, how
are you even sure you’re merging in the things you want to merge to a given
release. We need a consistent way to tag stuff for a release so that Chef can
do a search, ensure everything is merged.

Let’s say there was a regression in the RC and I sent a PR, merge it, and tag
it for release, but you don’t merge it. Oops, now we just released a
regression we didn’t have to. The argument of “oh, but we’ll know about the
regression so we’ll cherry pick it ourselves” isn’t acceptable. The whole
point of having maintainers and open development system is so there’s
transparency and consistency. EIther we’re using milestones in GH or we’re
not, but we need a system that’s consistent.

I don’t think it’s sane to for every contributor to be allowed to decide

  • based on their own value judgment, that may or may not be sound - when
    their PR should get merged. This is likely to result in at best noise,
    and at worst chaos, since everyone wants their stuff to go in as soon as
    possible, and often only the project maintainers have any broad sense of
    the state of the projects, and the impact of other items in the merge queue.

That’s fair, but there should be a clear way of requesting it be in a
release (most people probably won’t), and all requests should be accepted or
denied before the release so there’s no surprises. I shouldn’t have to
guess.

I do agree that milestones add useful visibility for all involved
though. Contributor settable tags for enhancement, bugfix and
regression do make sense - the review team can use them to prioritize
regressions and then /they/ should bin issues into a milestone.

If we say only maintainers can add them… well I am and I did.

If we say only chef-employees can add them, fine, but then (1) lets make that
clear and (2) then they should always be added immediately and can be
changed in the event that a PR takes less/more or becomes less/more
complicated, etc.


Phil Dibowitz phil@ipom.com
Open Source software and tech docs Insanity Palace of Metallica
http://www.phildev.net/ http://www.ipom.com/

“Be who you are and say what you feel, because those who mind don’t matter
and those who matter don’t mind.”

  • Dr. Seuss

#10

On Tue, Apr 28, 2015 at 5:13 AM, Phil Dibowitz phil@ipom.com wrote:

But I’m trying to discuss the more general problem here, not this specific
PR.
And to be clear I’m not blaming anyone here. I think we have a process
problem.

<3. Ticketing process, so dear to me.

All emotions are my own, as corporations aren’t people and can’t have
emotions. :wink:

In general, we’re using the issue milestones “Accepted Minor” for work
that we
think aren’t breaking changes, “Accepted Major”, for things that we
think will
be breaking changes or major work, and “Help Wanted”, for issues that
we’d
like help from the community with. We’ve been publishing the list of Help
Wanted issues to the chef-dev list as we assign things to that bucket.
Hope this helps,

I’m assuming your using Issue here interchangably with PR and not expecting
contributors to have to make an issue for every PR? (that would seem
unnecessarily onerous for contributors).

https://github.com/chef/chef-rfc/pull/36 doesn’t actually mention how to
get a
PR into a release, but since Chef doesn’t actually release from HEAD
(cough
delivery-keynote couch), which is totally fine, we need a defined way to
denote stuff is desired for the next major/minor/bugfix release. Upon
previous
discussions with btm, serdar, and others, I was lead to believe milestones
on
PRs was that way… but that doesn’t actually seem to be the case.

Historically, how this works has varied over the years. Of course, it ends
up being “whatever is in the branch the release is cut from” which I’ll get
back to. When we had JIRA and ticket workflows (btm swoons), committers
were “forced” to set Fixed Version when they marked a ticket as merged.
That at least told you what went into the release. You could also set Fix
Version for things you wanted to go into the release. It wasn’t a
guarantee, but we used it when we need that functionality.

“Accepted Major” and “Accepted Minor” are not - afaict - sufficient. They

don’t tell me what release my thing is going to be in.

I’m super sad I have to go use git to figure out what release something is
in . I want to be able to ask people “are you using at least 12.x.y?
because it was fixed there” rather than “you should always be upgrading to
the latest release, because we fixed that at some point.” I think “Accepted
…” is more for triaging work that we think should be done than tracking
what you’re looking for.

To put it in RFC terms:

As a contributor I should be able to say “I believe this belongs in 12.3”

As a maintainer I should be able to say “I’m sorry, but we’re not going to
put
it in 12.3 because it’s big and scary”

As a contributor I should get the chance to know that and not suddenly see
a
release without the thing that I added the milestone to.

As a contributor I don’t expect to be able to add a Milestone to a PR that
is
already past the freeze date.

As a contributor I don’t expect a release to go out the door with any
pending
PRs tagged for it (either the Milestone should have been changed or I
should
have been unable to add it).

+600,203,123.02, you’ve been awarded Process Hero.

So, I want to dump some context here for visibility for those outside of
Chef Software.

For the “next while”, Chef Software is going to keep being responsible for
the releases because it requires a build system that’s internal. We’ve
always wanted to expose that in some way, and we’re working towards doing
so. The client just moved to a “new” build cluster that’s completely Chef
[the software] managed. We’ve been running that cluster for a while for
other projects, but the client has the most platforms to support so getting
things like Solaris and FreeBSD going took some work. 12.3.0 was the first
release done on it. One huge thing is that this cluster is more way
reliable. For instance, the guests run on EC2 rather than your typical
outdated openstack skunkworks project. Really soon (month? I hope) we’re
going to be getting this cluster to the point where it produces an
"integration" build at least daily, if not for every merge to master. This
means that non-Chef Software maintainers can merge code and get resulting
builds.

In the long, long ago, we had months between releases. They were really
hard and not our priority. At various points in time it was my job to care
about that, all other times I just care about it anyway. We’re definitely
continuing to improve on that. At Chef Software we’ve moved toward
"functional teams" this year in Chef Engineering (oh names). Which means
there isn’t a “client team” anymore, and part of that effort is ensuring
that any of the teams can make a release. My team has been doing the
releases since then (H/T Jay), but Community Engineering just did the
12.3.0 release. Which is AWESOME, and a milestone in getting to that place
internally. Part of this goal is having a more frequent cadence of smaller
releases with less risk in each of them.

If we “release all the time,” the parts of the process that aren’t the
build become the heavy parts and there’s going to be a tendency to skip out
on those. (Sidebar: Kind of like how cloud computing drove configuration
management). I think you bring up the exact points we need to focus on to
make sure we don’t lose the ticketing parts of that process that expose
value to contributors, maintainers, and users in telling whats in a release
and what will be in a release without having to muster to gusto to ask all
the time.

We’ve been using “x-stable” branches for releases for a while and the
"master" branch for the next major version development, loosely. I think
most of us want to get away from that and just use master. In particular it
caused some regressions in the last few releases because we had some
commits on the 12-stable branch that weren’t in master, and then we created
a separate branch for 12.3 and it was as hairy as would be expected. Just
using master does kick the can down the road in terms of what we do when we
want to start working on breaking changes for the next release. Maybe those
go in a branch, which we have to regularly rebase against master? If we get
rid of release branches, then we basically ship everything that’s in master
for a minor release, every time. Maintainers who merge stuff to master get
it shipped. I think that solves your concern about merging stuff as a
Maintainer but not having it shipped. If we have to do a patch release off
a previous release to fix a regression, we can checkout the last release
tag and do that, and it is that persons responsibility to ensure master
gets it too. This should be written up.

I’m waiting on a copyright comment from Serdar before merging chef-rfc#36
(Github Workflow Process). I think the “RFC terms” above split between that
process and the RFC047 release process.There’s also chef#3205, which is a
medium hurdle to getting integration builds of the client easily, and has
some forward looking thoughts on future versioning that need to keep this
kind of discussion in mind.


Anyway, this conversation should absolutely result in PRs against RFCs.

It’s probably “as easy as” writing in process for using milestones. In the
Github Workflow Process the next unreleased milestone gets set to something
when it’s merged to track that it’ll be in that release. In the RFC047
release process, we basically say that we search for something like
"is:open milestone:12.2.1" if that’s the release we want to make and we
have to have a discussion/resolve anything that’s still open (e.g. this is
too big, has to be in the next release, this release is just about getting
a regression fixed). The hard part is probably convincing everyone who has
to do it’s that it is worth the time so we don’t lose any potential release
cadence (like it’s hard convincing people that keeping a clean git history
is worth the effort :] ).

Bryan


#11

On Friday, May 1, 2015 at 9:38 AM, Bryan McLellan wrote:

On Tue, Apr 28, 2015 at 5:13 AM, Phil Dibowitz <phil@ipom.com (mailto:phil@ipom.com)> wrote:

But I’m trying to discuss the more general problem here, not this specific PR.
And to be clear I’m not blaming anyone here. I think we have a process problem.

<3. Ticketing process, so dear to me.

All emotions are my own, as corporations aren’t people and can’t have emotions. :wink:

In general, we’re using the issue milestones “Accepted Minor” for work that we
think aren’t breaking changes, “Accepted Major”, for things that we think will
be breaking changes or major work, and “Help Wanted”, for issues that we’d
like help from the community with. We’ve been publishing the list of Help
Wanted issues to the chef-dev list as we assign things to that bucket.
Hope this helps,

I’m assuming your using Issue here interchangably with PR and not expecting
contributors to have to make an issue for every PR? (that would seem
unnecessarily onerous for contributors).

https://github.com/chef/chef-rfc/pull/36 doesn’t actually mention how to get a
PR into a release, but since Chef doesn’t actually release from HEAD (cough
delivery-keynote couch), which is totally fine, we need a defined way to
denote stuff is desired for the next major/minor/bugfix release. Upon previous
discussions with btm, serdar, and others, I was lead to believe milestones on
PRs was that way… but that doesn’t actually seem to be the case.

Historically, how this works has varied over the years. Of course, it ends up being “whatever is in the branch the release is cut from” which I’ll get back to. When we had JIRA and ticket workflows (btm swoons), committers were “forced” to set Fixed Version when they marked a ticket as merged. That at least told you what went into the release. You could also set Fix Version for things you wanted to go into the release. It wasn’t a guarantee, but we used it when we need that functionality.

“Accepted Major” and “Accepted Minor” are not - afaict - sufficient. They
don’t tell me what release my thing is going to be in.

I’m super sad I have to go use git to figure out what release something is in . I want to be able to ask people “are you using at least 12.x.y? because it was fixed there” rather than “you should always be upgrading to the latest release, because we fixed that at some point.” I think “Accepted …” is more for triaging work that we think should be done than tracking what you’re looking for.

To put it in RFC terms:

As a contributor I should be able to say “I believe this belongs in 12.3”

As a maintainer I should be able to say “I’m sorry, but we’re not going to put
it in 12.3 because it’s big and scary”

As a contributor I should get the chance to know that and not suddenly see a
release without the thing that I added the milestone to.

As a contributor I don’t expect to be able to add a Milestone to a PR that is
already past the freeze date.

As a contributor I don’t expect a release to go out the door with any pending
PRs tagged for it (either the Milestone should have been changed or I should
have been unable to add it).

+600,203,123.02, you’ve been awarded Process Hero.

So, I want to dump some context here for visibility for those outside of Chef Software.

For the “next while”, Chef Software is going to keep being responsible for the releases because it requires a build system that’s internal. We’ve always wanted to expose that in some way, and we’re working towards doing so. The client just moved to a “new” build cluster that’s completely Chef [the software] managed. We’ve been running that cluster for a while for other projects, but the client has the most platforms to support so getting things like Solaris and FreeBSD going took some work. 12.3.0 was the first release done on it. One huge thing is that this cluster is more way reliable. For instance, the guests run on EC2 rather than your typical outdated openstack skunkworks project. Really soon (month? I hope) we’re going to be getting this cluster to the point where it produces an “integration” build at least daily, if not for every merge to master. This means that non-Chef Software maintainers can merge code and get resulting builds.

In the long, long ago, we had months between releases. They were really hard and not our priority. At various points in time it was my job to care about that, all other times I just care about it anyway. We’re definitely continuing to improve on that. At Chef Software we’ve moved toward “functional teams” this year in Chef Engineering (oh names). Which means there isn’t a “client team” anymore, and part of that effort is ensuring that any of the teams can make a release. My team has been doing the releases since then (H/T Jay), but Community Engineering just did the 12.3.0 release. Which is AWESOME, and a milestone in getting to that place internally. Part of this goal is having a more frequent cadence of smaller releases with less risk in each of them.

If we “release all the time,” the parts of the process that aren’t the build become the heavy parts and there’s going to be a tendency to skip out on those. (Sidebar: Kind of like how cloud computing drove configuration management). I think you bring up the exact points we need to focus on to make sure we don’t lose the ticketing parts of that process that expose value to contributors, maintainers, and users in telling whats in a release and what will be in a release without having to muster to gusto to ask all the time.

We’ve been using “x-stable” branches for releases for a while and the “master” branch for the next major version development, loosely. I think most of us want to get away from that and just use master. In particular it caused some regressions in the last few releases because we had some commits on the 12-stable branch that weren’t in master, and then we created a separate branch for 12.3 and it was as hairy as would be expected. Just using master does kick the can down the road in terms of what we do when we want to start working on breaking changes for the next release. Maybe those go in a branch, which we have to regularly rebase against master?
IMO we have to use feature flags for this stuff for the rest of time. Even if we have to copy and paste a bunch of functionality to keep the old thing working, it’s not that bad because the old thing shouldn’t change much once we decide we want to get rid of it at some future time. Then major version bumps are where we change the default feature flag and/or remove the old thing.

This is also what we should do for code we think is “too scary to merge right before a release.” For example, if we want to merge a thing, we’re a little unsure of it because it’s a big change or whatever, and we plan to ship a minor release in 2 days, then we should rework the patch to be able to toggle it off. In that case, we’d change the toggle (or remove it) after the release.

If we get rid of release branches, then we basically ship everything that’s in master for a minor release, every time. Maintainers who merge stuff to master get it shipped. I think that solves your concern about merging stuff as a Maintainer but not having it shipped. If we have to do a patch release off a previous release to fix a regression, we can checkout the last release tag and do that, and it is that persons responsibility to ensure master gets it too. This should be written up.

I’m waiting on a copyright comment from Serdar before merging chef-rfc#36 (Github Workflow Process). I think the “RFC terms” above split between that process and the RFC047 release process.There’s also chef#3205, which is a medium hurdle to getting integration builds of the client easily, and has some forward looking thoughts on future versioning that need to keep this kind of discussion in mind.

https://github.com/chef/chef-rfc/blob/master/rfc047-release-process.md
https://github.com/chef/chef/issues/3205

Anyway, this conversation should absolutely result in PRs against RFCs.

It’s probably “as easy as” writing in process for using milestones. In the Github Workflow Process the next unreleased milestone gets set to something when it’s merged to track that it’ll be in that release. In the RFC047 release process, we basically say that we search for something like “is:open milestone:12.2.1” if that’s the release we want to make and we have to have a discussion/resolve anything that’s still open (e.g. this is too big, has to be in the next release, this release is just about getting a regression fixed). The hard part is probably convincing everyone who has to do it’s that it is worth the time so we don’t lose any potential release cadence (like it’s hard convincing people that keeping a clean git history is worth the effort :] ).

I hate “digital paperwork” more than anyone I know, but it’s a necessary evil. Of course the best thing to do is automate it as much as possible.

Bryan

I broadly agree with the general points here, especially with what Phil said “in RFC terms."


Daniel DeLeo


#12

On 05/01/2015 09:38 AM, Bryan McLellan wrote:

Anyway, this conversation should absolutely result in PRs against RFCs.

It’s probably “as easy as” writing in process for using milestones. In the
Github Workflow Process the next unreleased milestone gets set to something
when it’s merged to track that it’ll be in that release. In the RFC047 release
process, we basically say that we search for something like “is:open
milestone:12.2.1” if that’s the release we want to make and we have to have a
discussion/resolve anything that’s still open (e.g. this is too big, has to be
in the next release, this release is just about getting a regression fixed).
The hard part is probably convincing everyone who has to do it’s that it is
worth the time so we don’t lose any potential release cadence (like it’s hard
convincing people that keeping a clean git history is worth the effort :] ).

Thanks Bryan! Yes I was probably assuming more widespread knowledge of some of
that context then was reasonable, I’m glad you laid that out.

I LOVE the idea of living on master, but I realize we’ve got work to do before
we get there.

While I’m always happy to write RFCs (or PRs against RFCs), I don’t think it
makes any sense for a non-Chef-employee to write one around
release-engineering since (a) only Chef-employees will be doing it
(justifiably) for the forseeable future and (b) it has to fit into your guys’
workflow and tools.

But I’m happy to help figure out a process. How about something like this -
please feel free to modify/hackup/etc.:

  • At freeze-date, releaser must:
  • search for “is:closed milestone:X.Y.Z” and either:
    • cherry-pick to release branch OR
    • comment that we’re not pulling it in for reason Q (and add a later milestone)
  • search for “is:open milestone:X.Y.Z” and either:
    • merge to master & merge to to release branch IF it was ready already OR
    • change the milestone
  • IF the release is X.Y.0 then repeat the above 2 steps for
    "milestone:X.Y-1.Z+1"… i.e if the last release was 12.3.1 and we’re
    releasing 12.4, we should check for anything tagged for 12.3.2.
  • Close the aforementioned milestones
  • Add milestones for the next anticipated minor and bugfix release (assuming
    a release of, say 12.3.1, he or she would add 12.3.2 and 12.4.0).
  • Add a milestone for -rcfix
  • Do build

Then…

  • At final-release time (or additional-RC time), the releaser must do the same
    thing for -rcfix - not everything has to be cherry-picked, but it
    must EITHER be cherry-picked OR moved to a later milestone

The goals here being:

  • When an RC is built, everything LEFT with that milestone is included (not
    everything that had that milestone)
  • When a final is built, anything with RC-fix was either picked or explicitly
    denied.

Thoughts?


Phil Dibowitz phil@ipom.com
Open Source software and tech docs Insanity Palace of Metallica
http://www.phildev.net/ http://www.ipom.com/

“Be who you are and say what you feel, because those who mind don’t matter
and those who matter don’t mind.”

  • Dr. Seuss

#13

On 5/1/15 11:01 AM, Phil Dibowitz wrote:

On 05/01/2015 09:38 AM, Bryan McLellan wrote:

Anyway, this conversation should absolutely result in PRs against RFCs.

It’s probably “as easy as” writing in process for using milestones. In the
Github Workflow Process the next unreleased milestone gets set to something
when it’s merged to track that it’ll be in that release. In the RFC047 release
process, we basically say that we search for something like “is:open
milestone:12.2.1” if that’s the release we want to make and we have to have a
discussion/resolve anything that’s still open (e.g. this is too big, has to be
in the next release, this release is just about getting a regression fixed).
The hard part is probably convincing everyone who has to do it’s that it is
worth the time so we don’t lose any potential release cadence (like it’s hard
convincing people that keeping a clean git history is worth the effort :] ).
Thanks Bryan! Yes I was probably assuming more widespread knowledge of some of
that context then was reasonable, I’m glad you laid that out.

I LOVE the idea of living on master, but I realize we’ve got work to do before
we get there.

While I’m always happy to write RFCs (or PRs against RFCs), I don’t think it
makes any sense for a non-Chef-employee to write one around
release-engineering since (a) only Chef-employees will be doing it
(justifiably) for the forseeable future and (b) it has to fit into your guys’
workflow and tools.

But I’m happy to help figure out a process. How about something like this -
please feel free to modify/hackup/etc.:

  • At freeze-date, releaser must:
    • search for “is:closed milestone:X.Y.Z” and either:
      • cherry-pick to release branch OR
      • comment that we’re not pulling it in for reason Q (and add a later milestone)
    • search for “is:open milestone:X.Y.Z” and either:
      • merge to master & merge to to release branch IF it was ready already OR
      • change the milestone
    • IF the release is X.Y.0 then repeat the above 2 steps for
      "milestone:X.Y-1.Z+1"… i.e if the last release was 12.3.1 and we’re
      releasing 12.4, we should check for anything tagged for 12.3.2.
    • Close the aforementioned milestones
    • Add milestones for the next anticipated minor and bugfix release (assuming
      a release of, say 12.3.1, he or she would add 12.3.2 and 12.4.0).
    • Add a milestone for -rcfix
    • Do build

Then…

  • At final-release time (or additional-RC time), the releaser must do the same
    thing for -rcfix - not everything has to be cherry-picked, but it
    must EITHER be cherry-picked OR moved to a later milestone

The goals here being:

  • When an RC is built, everything LEFT with that milestone is included (not
    everything that had that milestone)
  • When a final is built, anything with RC-fix was either picked or explicitly
    denied.

Thoughts?

I’m more in favor of the rc being cut off of master and that’s the
freeze. Anything that was in the minor milestone that wasn’t merged to
master gets bumped to the next milestone. The only thing that gets
cherry-picked to the rc branch after that is bugfixes.

Releases take so much time that I don’t think any more burden should be
placed on the person shepherding the release. I also think that any
more process than this simply won’t get followed correctly and that
people doing the release will just continually make mistakes. We need to
make it easy on that person to get their job done because the release
job already sucks hard enough as it is.

I think this also is by far the safest approach for anyone trying to get
a patch in. Whatever process we come up with, I’m quite certain that if
you’ve gotten your patch into master before the dot-0 rc is forked off
that it’ll make it into the release.

There’s also the issue that the rc is supposed to reflect the stable
code that is going to get shipped. Any code which is cherry-picked from
master after the rc goes is code which got no testing in the rc. To
make the rc process produce a more stable shipped release we shouldn’t
be cherry-picking anything other than regressions we find in the rc.


#14

On 05/01/2015 02:29 PM, Lamont Granquist wrote:

I’m more in favor of the rc being cut off of master and that’s the freeze.
Anything that was in the minor milestone that wasn’t merged to master gets
bumped to the next milestone. The only thing that gets cherry-picked to the
rc branch after that is bugfixes.

I’d love that. Are we OK with that? I got the impression we weren’t quite
ready for that yet.

There’s also the issue that the rc is supposed to reflect the stable code that
is going to get shipped. Any code which is cherry-picked from master after
the rc goes is code which got no testing in the rc. To make the rc process
produce a more stable shipped release we shouldn’t be cherry-picking anything
other than regressions we find in the rc.

Agreed.


Phil Dibowitz phil@ipom.com
Open Source software and tech docs Insanity Palace of Metallica
http://www.phildev.net/ http://www.ipom.com/

“Be who you are and say what you feel, because those who mind don’t matter
and those who matter don’t mind.”

  • Dr. Seuss

#15

On Fri, May 1, 2015 at 2:01 PM, Phil Dibowitz phil@ipom.com wrote:

While I’m always happy to write RFCs (or PRs against RFCs), I don’t think
it
makes any sense for a non-Chef-employee to write one around
release-engineering since (a) only Chef-employees will be doing it
(justifiably) for the forseeable future and (b) it has to fit into your
guys’
workflow and tools.

Our internal tools are already pretty well reduced to a single line in
RFC047. I like the perspective of someone on the outside adding context
like “as someone that isn’t you, I like knowing thing”

But I’m happy to help figure out a process. How about something like this -

please feel free to modify/hackup/etc.:

  • At freeze-date, releaser must:
  • search for “is:closed milestone:X.Y.Z” and either:
    • cherry-pick to release branch OR
    • comment that we’re not pulling it in for reason Q (and add a later
      milestone)
  • search for “is:open milestone:X.Y.Z” and either:
    • merge to master & merge to to release branch IF it was ready
      already OR
    • change the milestone
  • IF the release is X.Y.0 then repeat the above 2 steps for
    "milestone:X.Y-1.Z+1"… i.e if the last release was 12.3.1 and we’re
    releasing 12.4, we should check for anything tagged for 12.3.2.
  • Close the aforementioned milestones
  • Add milestones for the next anticipated minor and bugfix release
    (assuming
    a release of, say 12.3.1, he or she would add 12.3.2 and 12.4.0).
  • Add a milestone for -rcfix
  • Do build

I think we just ship off master or near master like Lamont is talking
about. Let’s work that out. But that removes the first part of the process
where we look for closed things and do anything with them. They’re already
merged to the place that they ship from. Ideally, we don’t have to merge
anything for the release because the Maintainers are doing it periodically
(which also lets people use that code in the nightly builds)

  • At freeze-date, releaser must:
  • search for “is:open milestone:X.Y.Z” and either:
    • merge to master
    • comment that we’re not pulling it in for reason Q (and add a later
      milestone)
  • if shipping a major or minor release, repeat the previous step for any
    lesser milestones
  • Close the aforementioned milestones
  • Add milestones for the next release versions as applicable
  • Build/ship RC

Wait one week per RFC047. Watch for regressions filed.

  • tag/ship final build.

I’d like to avoid the rcfix milestones. It’s on the Maintainers to watch
for regressions between an rc and final get that stuff fixed and merged
into the release branch already. While we don’t all triage issues
completely, we all have our ears to the ground in different places and hear
about these. I think the rcfix milestone creation is going to mostly be
busywork and more likely to be skipped/forgotten/noisy than help us find an
issue/pr we wouldn’t find anyway.

On Fri, May 1, 2015 at 5:29 PM, Lamont Granquist lamont@chef.io wrote:

I’m more in favor of the rc being cut off of master and that’s the
freeze. Anything that was in the minor milestone that wasn’t merged to
master gets bumped to the next milestone. The only thing that gets
cherry-picked to the rc branch after that is bugfixes.

Yeah I don’t like freezing master itself. It’s been annoying having our
release tags out on release branches, but I think that’s actually okay if
our release branches are short dead ends. Which is what I think we’re
talking about. But I’m not sure I want a bunch of branches lying around.

git checkout -b 12-release master
git tag 12.x.y.rc.0 -a
git push origin 12.x.y.rc.0 && git push origin 12-release

build and ship off the 12.x.y.rc.0 tag

development can continue off master

regression is found

git checkout 12-release
git commit -a -m "fix regression"
git tag 12.x.y.rc.1 -a
git checkout master
git cherry-pick some-sha
git push origin 12.x.y.rc.1 && git push origin 12-release && git push
origin master

… release happens on 12-release

git checkout master
git merge 12-release

It seems like you’re going to have problems with the cherry-pick on that
final merge, but I’d like those branches to get merged back in so they can
"go away." Am I missing something about git here? Is there a better way to
do that?

Releases take so much time that I don’t think any more burden should be

placed on the person shepherding the release. I also think that any more
process than this simply won’t get followed correctly and that people doing
the release will just continually make mistakes. We need to make it easy on
that person to get their job done because the release job already sucks
hard enough as it is.

Releases are getting faster + easier. Eventually the talk is that the
builds happen automatically and we will “promote a build” to stable when we
want to. 12.3.0 happened fairly magically using manhattan. I thinking
milestones are really worthwhile to anyone who isn’t the person making the
release. We need this even internally now since we’re trying to have more
teams writing client code so they can understand whats happening with their
PRs.

We could have a Jenkins step before chef-release that takes the Chef
Version and searches the github issues API for open issues for that
version, and fails if there are any. I think that’d make more people rage
than it would help though.


Bryan McLellan | chef | engineering lead
© 206.607.7108 | (t) @btmspox | (www) http://chef.io


#16

On 05/01/2015 03:24 PM, Bryan McLellan wrote:

On Fri, May 1, 2015 at 2:01 PM, Phil Dibowitz <phil@ipom.com
mailto:phil@ipom.com> wrote:

While I'm always happy to write RFCs (or PRs against RFCs), I don't think it
makes any sense for a non-Chef-employee to write one around
release-engineering since (a) only Chef-employees will be doing it
(justifiably) for the forseeable future and (b) it has to fit into your guys'
workflow and tools.

Our internal tools are already pretty well reduced to a single line in RFC047.
I like the perspective of someone on the outside adding context like “as
someone that isn’t you, I like knowing thing”

https://github.com/chef/chef-rfc/blob/master/rfc047-release-process.md#chef-client-release-process-1

I’m loving this discussion. :slight_smile:

I think we just ship off master or near master like Lamont is talking about.
Let’s work that out. But that removes the first part of the process where we
look for closed things and do anything with them. They’re already merged to
the place that they ship from. Ideally, we don’t have to merge anything for
the release because the Maintainers are doing it periodically (which also lets
people use that code in the nightly builds)

Yup. And in fact, the other half goes away too - no one needs to tag anything
for any release… it’s either in when we do the release or it goes to the
next one. Unless you want the tag to mean “blocking this release” in which
case I think we should be very careful with it - maybe only maintainers
can/should add them or something.

Yeah I don’t like freezing master itself. It’s been annoying having our
release tags out on release branches, but I think that’s actually okay if our
release branches are short dead ends. Which is what I think we’re talking
about. But I’m not sure I want a bunch of branches lying around.

I like where you’re going, I have minor changes I’d suggest, because…

It seems like you’re going to have problems with the cherry-pick on that final
merge, but I’d like those branches to get merged back in so they can “go
away.” Am I missing something about git here? Is there a better way to do that?

Right, so I would instead not cherry pick that way. I’d always cherry-pick
from master INTO the rc instead of the other way around. Master should never
ever get behind. So:

git checkout -b 12.x.y-release master
git tag 12.x.y.rc.0 -a
git push origin 12.x.y.rc.0 && git push origin 12.x.y-release

build and ship off the 12.x.y.rc.0 tag

development can continue off master

regression is found

git co master

fix thing

git commit -a -m "fix regression"
git co 12.x.y-release
git cherry-pick
git tag 12.x.y.rc.1 -a
git push origin 12.x.y.rc.1 && git push origin 12-release && git push origin
master

ready to release final, no new changes

git co 12.x.y-release
git tag 12.x.y -a

build, release

git checkout master
git merge 12-release


Phil Dibowitz phil@ipom.com
Open Source software and tech docs Insanity Palace of Metallica
http://www.phildev.net/ http://www.ipom.com/

“Be who you are and say what you feel, because those who mind don’t matter
and those who matter don’t mind.”

  • Dr. Seuss

#17

On 5/1/15 3:24 PM, Bryan McLellan wrote:

Yeah I don’t like freezing master itself. It’s been annoying having
our release tags out on release branches, but I think that’s actually
okay if our release branches are short dead ends. Which is what I
think we’re talking about. But I’m not sure I want a bunch of branches
lying around.
I’m not sure I see the problem with old x.y-stable branches. Its just
old history, like old tags, old commits, etc. Useful if you need to
check something old out, completely ignorable by everyone else.


#18

On 5/1/15 4:11 PM, Phil Dibowitz wrote:

I think we just ship off master or near master like Lamont is talking about.
Let’s work that out. But that removes the first part of the process where we
look for closed things and do anything with them. They’re already merged to
the place that they ship from. Ideally, we don’t have to merge anything for
the release because the Maintainers are doing it periodically (which also lets
people use that code in the nightly builds)
Yup. And in fact, the other half goes away too - no one needs to tag anything
for any release… it’s either in when we do the release or it goes to the
next one. Unless you want the tag to mean “blocking this release” in which
case I think we should be very careful with it - maybe only maintainers
can/should add them or something.

I think the person doing the release needs to do the decision as to if
its blocking or not.

In the future this hopefully goes away, we just pick a nightly to
promote. In the present, I think the person doing the work needs to be
the decider. Once they’re satisfied with what is going into the
release, the unfinished stuff in the milestone are bumped and the rc
branch is cut.

That keeps it so that we don’t have maintainers expecting to be able to
hold up releases for pet fixes that aren’t done yet.


#19

On 05/02/2015 08:18 AM, Lamont Granquist wrote:

On 5/1/15 4:11 PM, Phil Dibowitz wrote:

I think we just ship off master or near master like Lamont is talking about.
Let’s work that out. But that removes the first part of the process where we
look for closed things and do anything with them. They’re already merged to
the place that they ship from. Ideally, we don’t have to merge anything for
the release because the Maintainers are doing it periodically (which also lets
people use that code in the nightly builds)
Yup. And in fact, the other half goes away too - no one needs to tag anything
for any release… it’s either in when we do the release or it goes to the
next one. Unless you want the tag to mean “blocking this release” in which
case I think we should be very careful with it - maybe only maintainers
can/should add them or something.

I think the person doing the release needs to do the decision as to if its
blocking or not.

In the future this hopefully goes away, we just pick a nightly to promote. In
the present, I think the person doing the work needs to be the decider. Once
they’re satisfied with what is going into the release, the unfinished stuff in
the milestone are bumped and the rc branch is cut.

That keeps it so that we don’t have maintainers expecting to be able to hold
up releases for pet fixes that aren’t done yet.

I’m fine with that if ChefCo doesn’t need to track things think “have” to be
in the next release.

So… what needs to happen to start doing releases off of master?


Phil Dibowitz phil@ipom.com
Open Source software and tech docs Insanity Palace of Metallica
http://www.phildev.net/ http://www.ipom.com/

“Be who you are and say what you feel, because those who mind don’t matter
and those who matter don’t mind.”

  • Dr. Seuss

#20

On Sat, May 2, 2015 at 6:17 PM, Phil Dibowitz phil@ipom.com wrote:

On 05/02/2015 08:18 AM, Lamont Granquist wrote:

On 5/1/15 4:11 PM, Phil Dibowitz wrote:

I think we just ship off master or near master like Lamont is talking
about.

Let’s work that out. But that removes the first part of the process
where we

look for closed things and do anything with them. They’re already
merged to

the place that they ship from. Ideally, we don’t have to merge
anything for

the release because the Maintainers are doing it periodically (which
also lets

people use that code in the nightly builds)
Yup. And in fact, the other half goes away too - no one needs to tag
anything

for any release… it’s either in when we do the release or it goes to
the

next one. Unless you want the tag to mean “blocking this release” in
which

case I think we should be very careful with it - maybe only maintainers
can/should add them or something.

I think the person doing the release needs to do the decision as to if
its
blocking or not.

In the future this hopefully goes away, we just pick a nightly to
promote. In
the present, I think the person doing the work needs to be the decider.
Once
they’re satisfied with what is going into the release, the unfinished
stuff in
the milestone are bumped and the rc branch is cut.

That keeps it so that we don’t have maintainers expecting to be able to
hold
up releases for pet fixes that aren’t done yet.

I’m fine with that if ChefCo doesn’t need to track things think “have” to
be
in the next release.

So… what needs to happen to start doing releases off of master?

12.3 came off master. (well, I cut a 12.3 branch off master and released
from there,
but that’s just noise, frankly. Getting the versions right and tagging
master is equivalent).
There’s no reason right now for 12.4 not to come off master in a couple of
weeks time.
What we do need to figure out is what Dan mentioned - feature flags to
manage
changes that would constitute major versions.
-Thom