Assuming a dependency tree like A => B => C, if we build a new (unstable) version of C, a reverse dependency build will ensure that A and B are rebuilt to depend on this new, unstable C.
However, if we trigger a build of A, that build will only pull versions of B and C that are stable, and these may have been built with much older code revisions. So we’ve now taken 1 step forward and 2 steps back.
We can get a list of packages for a service from the Supervisor API, but promoting them to stable will only help us until there is a newer version of these packages, at which point A will again be building against a version of B and/or C that is now out of date.
If we set HAB_BLDR_CHANNEL to, say, unstable, then we end up pulling all kinds of broken stuff from core and other origins and are unlikely to get a working build.
Mulling this over, are we just assuming a workflow with a central CI server but should instead be triggering Builder job groups with a Git commit hook or something similar?
I think using git hooks (ie, relying on builder’s github integration for example) will simplify greatly your pipeline indeed.
When doing so, builder will rebuild packages for every commits created, ensuring that all reverse deps will be rebuilt too. This means that unstable a package should always use the latest b and c.
I think you should use the auto-promote feature of builder, to a staging channel maybe? this way your sups could listen to staging instead of unstable. (however, I don’t know if this is a wip or if it’s actually usable)
As far as we know auto-promotion doesn’t work right now. But this is exactly why we need this for our own packages (hopefully we could set HAB_BLDR_CHANNEL to <our channel> and then Builder would reach out to the core origin and the stable channel for any packages not in <our channel>?
(We are already using Git-triggered automatic builds in Builder, by the way.)
Correct, auto promotion is not currently available on Builder, so for now you’d need to check the job group status and promote the group yourself. If you promote to something like a “staging” channel, then you could use HAB_BLDR_CHANNEL to pull in packages from there without pulling in random things from unstable. Builder will fall back to stable if it doesn’t find what its looking for in the specified HAB_BLDR_CHANNEL.
So I think one remaining problems is that we don’t have a way to tell Builder what channel to promote a built package to. Currently any built package goes to unstable, and there doesn’t appear to be a way to override this, which means your advice about using a channel like staging is a bit hard to follow.
The only workaround I can see for now is to go back to using Travis or Jenkins, but then we are back in a situation where we have lost reverse-dependency builds.
As for the alternatives, using a CI to do the promote seems to be the only way. Maybe we can try to have a simple shell script that anyone can reuse to check a package latest build and allow promotion to staging only if all rev dep builds are successful?
For example, when I merge modifications in core-plans, I still get the bldr job group by hand, and then have a little command that allows me to check how rev deps rebuilds are doing. This might be scriptable.
Sure, but then we lose the ability to leverage automatic reverse-dependency builds. In fact, we also then need to explicitly disable any autobuild integration to avoid the “Polka Problem” (we ran into this in Production today, where a 2nd-tier service had been rebuilt and a 1st-tier service was then pointing at the wrong version of 2nd-tier service code that it depended on )
Hmm, I don’t think you do actually lose the ability, on the contrary I’d say The jog group id will allow you to promote all the rebuilt packages at once actually (this is what we do for core-plans).
Also, another idea. You could explicitly start a job with hab bldr job start -g <origin>/<package> package name being the deepest dependency, when a modification happens on any project, and start from there (promote the created job group after checking all went well).
)
The jog group id will allow you to promote all the rebuilt packages at once actually (this is what we do for core-plans).