How to revert a published package with incorrect version number?

Habitat Help
1

Due to a human error version 3.6.x has been published instead of the correct 0.3.6 version number. This is an issue as now the test setup listening to the unstable channel always installs the 3.6.x version instead of any new version with a lower version number (e.g. 0.3.7).

Issues:

  • There is no way to delete a package or demote it to an unused channel

  • There is no way to create a channel via public builder

  • the usage of cli (0.6.21) is not actually very clear, as described below and still leads to the wrong package being installed

    hab bldr channel create -o myorigin deprecated
    ✗✗✗
    ✗✗✗ No auth token specified
    ✗✗✗

    hab bldr channel create -o myorigin --auth my token deprecated
    error: Found argument ‘–auth’ which wasn’t expected, or isn’t valid in this context

    hab bldr channel create --help
    hab-bldr-channel-create 0.62.1/20180905002037
    Creates a new channel

    USAGE:
    hab bldr channel create [OPTIONS]

    FLAGS:
    -h, --help Prints help information
    -V, --version Prints version information

    OPTIONS:
    -u, --url <BLDR_URL> Specify an alternate Builder endpoint. If not specified, the value will be taken from the
    HAB_BLDR_URL environment variable if defined. (default: https://bldr.habitat.sh)
    -o, --origin Sets the origin to which the channel will belong. Default is from ‘HAB_ORIGIN’ or cli.toml

    ARGS:
    The channel name

It seems like there is no way to specify the token with the channel command. The only way to fix this is to provide the token via an environment variable.

Now we can finally move on to promoting the offending packages to the deprecated channel one at a time via the cli. However, if now I run hab svc unload mypkg and then hab svc load --channel unstable mypkg, we end up with the old package from the deprecated channel. How does one correctly fix a situation like that?

2

That certainly is confusing that hab bldr channel create doesn’t take the necessary auth arguments, but it does look in the HAB_AUTH_TOKEN environment variable. Either way, we should create an issue to address that, since it’s not very intuitive.

However, you don’t need to use hab bldr channel create to assign a particular channel to a package. You can hab pkg promote <PKG_IDENT> <CHANNEL> and if that channel doesn’t exist, it will be created.

Unfortunately, there’s no way that I know of to remove a package from the unstable channel, so if you don’t want it installed, you’ll need to promote the package you want to a different channel and specify it in your hab pkg install or hab svc load commands.

3

Looks like we already have an issue for that: https://github.com/habitat-sh/habitat/issues/5210

1 Like
4

Thanks for your reply. However, I think that is what I am actually trying to do right now. I promoted the faulty packages to the deprecated channel. However they still remain installed on the node, even if I issue the following two commands:

hab svc unload mypkg
hab svc load --channel unstable mypkg

I even tried to manually delete the package from /hab/pkg/myorg/mypkg/faultyVersion. However after invoking hab svc load --channel unstable mypkg I still see the package from deprecated channel loaded. Maybe it's pulled from a local chache without checking that the channel has changed @builder. I am not really sure what is going on. However, there is no pkg uninstall command or similar, so I am a little at a loss here how to correctly fix that.

Hmm, after rereading... do you actually suggest to no longer use the unstable channel and promote every package I need installed on the test environment to a different channel? That would be quite inconvenient. However, it kind of looks like even promoting the package to a different channel does not remove it from the stable channel, so probably that will be the only way to go. is that correct?

5

Yeah, there’s currently no way to delete a package entirely from builder (other packages could depend on it, and then it could cause cascading breakages). And since every package is a member of the unstable channel, if you want to avoid that bad one, you’ll either have to use a higher version number, or do your installs from a different channel.

With regards to clearing out the bad one, you’ll need to remove the installed versions from both /hab/pkg and /hab/cache/artifacts.

6

Thank you. Then a new channel it is
 :sunglasses:

7

whoa
 there is something involved, that I currently can not fully understand.

consider these two packages:

0.3.6/20180831111102
channel: test, unstable
updated (date shown in builder): 2018-08-31

3.6.5/20180824095820
channel: deprecated, unstable
updated: 2018-08-24

Now, if I unload everything installed AND delete those packages from the following locations:
/hab/pkg , /hab/svc and /hab/cache/artifacts

hab pkg install --channel test mypkg:
★ Install of myorigin/mypkg/0.3.6/20180831111102 complete with 1 new packages installed.
-> all good

hab svc load myorigin/mypkg

hab svc status
myorigin/mypkg/3.5.6/20180824095820
-> oh why?!?!?

Then I unloaded the service, deleted all the artifacts in the /hab folder again and now things were getting interesting

hab pkg install --channel test myorigin/mypkg
» Installing myorigin/mypkg
:cloud: Determining latest version of myorigin/mypkg in the ‘test’ channel
→ Found newer installed version (myorigin/mypkg/3.5.6/20180824095820) than remote version (myorigin/mypkg/0.3.6/20180831111102)
→ Using myorigin/mypkg/3.5.6/20180824095820
★ Install of myorigin/mypkg/3.5.6/20180824095820 complete with 0 new packages installed.

Habitat seems to consider a package which lives in a different channel, but was already installed newer, if the version number is higher. In my opinion this seems to be a bug. But where does it find that version locally?

I am very sure I do not have any revision of that package in the hab folder, as a find command does not show anything that matches my pkg name.

Next I’ll go see what happens if I start with a fresh node.

8

That is really strange. Can you show what hab svc status showed right before the hab pkg install --channel test mypkg and hab svc load myorigin/mypkg?

9

Well, I reinstalled that node yesterday, as it was pretty useless in that state. However, I am very sure that hab svc status did only list another package that was installed (no dependencies between the two packages). The package I was having problems with no longer was listed at all. I was running through that scenario a couple of times yesterday to make sure it is not an error on my side.

However, in case that service would have still been loaded, the hab svc load command usually just reports an error that the service is already loaded, afaik.

10

Did you see unexpected behavior again on the fresh node?

11

No. I think to make this show up, there needs to be a change in the used channel involved (which still brings up the question where habitat has been caching that package). Most likely habitat does not track the channel after a package has been downloaded. Additionally habitat seems to get confused about what package is newer in case the version number and the actual date of the package don’t make sense. (In case we can not delete packages, it would probably make sense to reject a package if the date is older, but the version number higher - at least at a semver major version change. but does hab even recommend semver? even though, that would probably just cause different issues
)
Additionally, an incorrectly published package causes a bunch of UX issues with builder, which I have added to the appropriate GitHub issue about unlisting a package. It would be extremely helpful if there would be a way to unlist or delete a package in case something goes wrong. Most package managing systems provide a way to do something similar which proves to be incredibly useful from time to time.

12

If you can come up with a repo for the strange behavior, I definitely think that’s worth an issue of its own, because I don’t see how that was occurring either.

While we on the habitat team are generally in favor of semver, we also have to support well-know packages that don’t use it, so we’re in a bit of a bind there. Maybe an explicit plan setting that indicates whether or not semver is being used could be helpful. Feel free to file an issue there.

Thanks for your comment on Origin support to unlist packages. I do think we need an official path for redressing appropriate needs for deletion. We just need to also be careful that it can either be safely automated without breaking dependent packages or is a rare enough occurrence that it’s not overwhelming to support manually.

13

Well, all issues described here could at least be avoided or easily recovered from if there would be a way of manually last-resort-delete that incorrect package. Even if it would just be “yeah, you need to contact someone who is allowed to do that on public builder”, or an http endpoint like the one to make public packages private. I think it should be a rare occasion that someone needs to delete a package. I don’t think having to tell people they need to switch to a different channel is a very good solution to work around a single mistake that was made when a package was published.
Everything else I suggested is at least from my viewpoint just another workaround to make the situation where packages can not be deleted less dreadful.

The issue I was encountering here in the end very likely is a rare edge case as you need to switch the channel a node is subscribed to and you need to have a package from a previous channel installed that has a higher version number.

I would like to dig deeper into this issue, but honestly there is a pile of stuff where I would like to dig deeper and I just don’t have the time to do so right now :frowning: