Adding dependency binaries to PATH

Habitat Help
#1

I’ve run into another new problem, that I don’t think existed previously. In my plan I have a few dependencies that provide binaries:

pkg_scaffolding="core/scaffolding-ruby"
pkg_deps=(core/curl finalze/imagemagick core/file)

and I SWEAR they used to be runnable from an interactive session within the resulting container when I exported to Docker, as if they were in the path. Now though, they are not, and they do not appear in the path. For example:

~ # curl
/bin/sh: curl: not found
~ # /hab/pkgs/core/curl/7.54.1/20180329185356/bin/curl
curl: try 'curl --help' for more information
~ # which curl
~ # echo $PATH
/bin
~ # ls /bin/curl
ls: /bin/curl: No such file or directory
~ #

I’ve looked at my app_env.sh and that’s not doing anything relevant, and I’ve poked around in all the startup and initialization scripts I can think of and haven’t found a place where these binaries are made available to my app. However, my app seems to be able to find them as it is working correctly, but some ancillary support scripts we have do not. They rely on some Ruby gems to do their work, and those gems in turn assume these binaries will be in the path. In the short term, I’ve worked around the problem by symlinking the binaries I know I need into /bin when I need them, but that’s not a sustainable solution.

Is this a change in behavior, or is my memory faulty? I can conceive of a number of ways I could work around this with custom bash profiles for root and all sorts of other nonsense, but it feels like the correct behavior would be for Habitat to add the binaries provided by deps to the path to ensure they are available.

#2

This is a change in behavior. We are no longer binlinking transitive dependency binaries. You should be able to exec them like hab pkg exec origin/my_app curl

#3

ok. That’s helpful when I need to run them by hand, but it doesn’t help my gems that are blowing up find them at all.

Are you guys set on this behavior going forward, or is there a chance that I could convince you to revert? What prompted the change?

#4

The change waas made because binutils was getting binlinked over some existing binaries as an unintended side-effect. You should be able to binlink them yourself in the init hook with hab pkg binlink origin/myapp curl --force

#5

Cool, I’ll give that a shot. Thanks!

#6

OK, I worked around this by adding the following to my init hook

{{#each pkg.deps}}
hab pkg binlink {{this.origin}}/{{this.name}}
{{/each}}

This is doing a lot more than I need it to since it goes through the whole transitive dependency tree and I only need my explicit deps to be binlinked, but it makes it one less thing to maintain when deps change.

Thanks for the pointer.

#7

:construction: I’m not sure if this idea will work :construction:

you might be able to iterate through each entry in the DEPS file and just binlink those. Also, if you know which binaries you need you can specify them as an arg to hab pkg binlink

#8

in my testing this produces a list in my init that looks like:


hab pkg binlink core/sed

hab pkg binlink core/xz

hab pkg binlink core/zlib

hab pkg binlink finalze/imagemagick

and in manually testing each of those, it goes through and binlinks all the bins that exist in that package. So it is doing a lot more than what I want.

I do mostly know what bins I need to binlink, but I’m trying to avoid needing to maintain dependency information in multiple places.

#9

On closer reading you reference a piece I’m not familiar with. The DEPS file? I’m going to go poke at that…

ok, found it. Yeah, that will create a much shorter list. I’ll switch that over to a shell loop…

Yeah, this is less intense:

while read dep; do
  hab pkg binlink $dep
done < {{pkg.path}}/DEPS

Adding that to the init hook gets my deps binlinked.

#10

I was referred to this post in the slack channel. I’m having a similar issue with uwsgi not running since the latest cli release. Unfortunately, when I add your binlink code to my init I just get a permission denied. The only way I have found to get around the uwsgi ‘not found’ error is to move core/gcc/5.2.0 from pkg_build_deps to pkg_deps. Does anyone have any suggestions or should I just go with what’s working now? :slight_smile:

#11

hm, if you’re getting permission denied, I would guess that you are doing something to change the default user setup? I would check the ownership in the /bin and make sure the hab user can write there. But, there is something to be said for going with what’s working…

#12

Thanks for your reply. I am not altering the users/groups at all, they are the default hab.

#13

hm, can you paste in your init logs?

#14

hab-sup(MR): Supervisor Member-ID 96025ac2200b431ab65f025a112e511a
hab-sup(AG): The f1yers/mib service was successfully loaded
hab-sup(MR): Starting f1yers/mib
mib.default(UCW): Watching user.toml
hab-sup(MR): Starting gossip-listener on 0.0.0.0:9638
hab-sup(MR): Starting ctl-gateway on 127.0.0.1:9632
hab-sup(MR): Starting http-gateway on 0.0.0.0:9631
mib.default(HK): init, compiled to /hab/svc/mib/hooks/init
mib.default(HK): run, compiled to /hab/svc/mib/hooks/run
mib.default(HK): Hooks compiled
mib.default(SR): Hooks recompiled
default(CF): Updated mime.types 50fb39ed5095e84b2043630bd0448d292ac96af69b2b91561c6fedbe6d6d7d8c
default(CF): Updated config.sh 30ae8b5f44475d6818eae8d157972c1e3e870b57aae900722692d113fa3b00d2
mib.default(SR): Configuration recompiled
mib.default(SR): Initializing
mib.default hook[init]:(HK): + read dep
mib.default hook[init]:(HK): + hab pkg binlink core/pcre/8.38/20170513213423
mib.default hook[init]:(HK): » Binlinking pcre-config from core/pcre/8.38/20170513213423 into /bin
mib.default hook[init]:(HK): ✗✗✗
mib.default hook[init]:(HK): ✗✗✗ Permission denied (os error 13)
mib.default hook[init]:(HK): ✗✗✗
mib.default hook[init]:(HK): + exec
mib.default(HK): Initialization failed! ‘init’ exited with status code 1

Binlinking says it's linking bins, but isn't
#15

Though I don’t think I need those binaries. I am installing uwsgi with pip in do_install(). It requires gcc but prior to 0.57.0 I was able to only specify core/gcc in my pkg_build_dep. I am going to roll with it for now since it works unless anyone can offer a better pattern. Appreciate your help!