Binlinking step failing during docker export

Habitat Help
1

Starting this morning, exporting to docker has started failing for me when it gets to the near-final binlinking step:

» Binlinking ld from my_org/my_package/0.1.2/20180425152711 into /tmp/hab-pkg-export-docker.UqvxA957nu6y/rootfs/bin
✗✗✗
✗✗✗ `ld' was not found under any 'PATH' directories in the my_org/my_package/0.1.2/20180425152711 package
✗✗✗

I’m not sure where to even begin troubleshooting this, as the problem would seem to be rooted somewhere in the docker export or core packages, which have always “just worked”.

2

Aaaaaand I just noticed that another build fired about thirty minutes after this failure, and it seems to have worked without me changing anything. Might this somehow be related to the gmp issue that was just rolled back?

EDIT - Actually no it’s not, that was something else that succeeded, this is still failing for me.

3

Thats super strange. This is a bug we had seen before but I thought it had been resolved? I’ll try to find the issue and solution, it could be possible that one of our build workers just didnt get the fix or something. I don’t think it would be related to the gmp update.

4

This is happening on my local machine and on my containerized build slaves, not on bldr. I’m running hab 0.55.0/20180321220925

5

Thats even weirder. You could check out https://github.com/habitat-sh/builder/issues/240 this is the issue where we tracked the thing we were seeing on our builders. I’m not quite sure why you’d be seeing that locally but if it pops up again you could update your build to include a export RUST_LOG=debug and grab the logs that might be helpful!

6

To clarify a little bit, it manages to binlink a bunch of stuff before failing on ld specifically.

7

Ran with Debug. I didn’t see anything that helped. It feels like whatever component provides “ld” is missing it for some reason, but I don’t even know if that makes sense.

Full output is at https://pastebin.com/CwRMerUA

The few lines around the failure are:

» Binlinking objdump from finalze/active-oversight-api/0.1.2/20180425152711 into /tmp/hab-pkg-export-docker.BIwFCoIzTQFI/rootfs/bin
★ Binlinked objdump from finalze/active-oversight-api/0.1.2/20180425152711 to /tmp/hab-pkg-export-docker.BIwFCoIzTQFI/rootfs/bin/objdump
» Binlinking ld from finalze/active-oversight-api/0.1.2/20180425152711 into /tmp/hab-pkg-export-docker.BIwFCoIzTQFI/rootfs/bin
✗✗✗
✗✗✗ `ld' was not found under any 'PATH' directories in the finalze/active-oversight-api/0.1.2/20180425152711 package
✗✗✗
8

I just noticed that this only happens if I do “hab pkg export docker the_pkg.hart” from the cli. If I do it from within a studio it seems to work. I’ve tested this with versions 0.53.0 - 0.55.0

My build automation though relies on running this stuff from the cli, so my ci/cd is still dead in the water.

9

I’ll try to take a more in-depth look at this momentarily. My apologies I’ve been on calls since getting started in here.

10

Great news! This magically started working again!

I still don’t know why though. I didn’t change anything.

EDIT: Grepped through the build output, and it’s not trying to binlink “ld” anymore.

Second EDIT: I found that this seems to related to variable in the creation of the hart file. Old files still throw this error when trying to create docker images from them, but new ones do not.

11

That is super weird. I think some of this has to do with pathing and so older version of packages could run into this but I wouldn’t expect anyone to see it anymore. I’m now about to take a further look to see if I can deduce anything. Thanks for the updates and your diligence!

12

Cool. Let me know if there’s anything else I can add that might be useful.

13

Welp, this problem has re-appeared. This is all on fresh hart files too. Let me know if there’s any information I can provide that might help troubleshoot. I’ll be working on this pretty much all day today.

QH

14

It’s still working from within the studio, but failing when I run from the CLI. ex -

hab studio enter
hab pkg export docker ./results/the_hart

works, but this doesn’t:

hab pkg export docker ./results/the_hart

EDIT: Just realized I wasn’t capturing the DEBUG output in the log files I was creating so the below isn’t really true. Re-running and re-comparing logs…

I’ve enabled debugging in both places and saved the output. After replacing the UUID-ish section of the directory structure that is different between the two runs, they are identical until it gets to the part about binlinking ld, at which point the cli-run bails out with the above errors, and the in-studio run continues to success.

I’m going to see if I can figure out how to run the export from within a studio context in my Jenkins pipeline to work around this so I’m not having to manually produce builds until a real solution is figured out…

15

I’ve found that hab studio run hab pkg export docker ./results/the.hart seems to work, so I’m running with that for now.

Is this more or less idiomatic than just a direct hab pkg export docker ./results/the.hart? Seems like a weird end-around.

16

We have two open binlinking issues, I'm not sure if they are related but thought I'd share just in case:

17

@qhartman That’s definitely how I do it, yes (though I’m on a Mac) — either I export from within an interactive Studio, or I script out the export using hab studio run and last_build.env:

hab studio run "build" && 
  source results/last_build.env && \
  hab studio run "hab pkg export docker results/$pkg_artifact"
18

I’m also told we have a fix for this that should be in the next release. :smile: