Dependency conflicts between different versions of glibc

Habitat Help
1

Posting on behalf of @bixu, who's currently having issues logging in to the forums

[2] api(do_before)> ldd $(hab pkg path core/pcre)/lib/libpcre.so.1
    linux-vdso.so.1 (0x00007fff955c6000)
    libpthread.so.0 => /hab/pkgs/core/glibc/2.27/20180608041157/lib/libpthread.so.0 (0x00007fd1ca979000)
    libc.so.6 => /hab/pkgs/core/glibc/2.27/20180608041157/lib/libc.so.6 (0x00007fd1ca5c5000)
    /hab/pkgs/core/glibc/2.22/20170513201042/lib64/ld-linux-x86-64.so.2 (0x00007fd1cae1e000)

The issues around core/pcre and core/glibc are completely blocking us at the moment so I'm trying to figure out the root cause/how to disentangle...

We ended up removing PCRE from our uWSGI build config. This is a workaround, not a fix (if we needed regular expressions in uWSGI we'd be back to a broken build. Does anyone have any ideas about how to solve the core problem, which seems to be around conflicting glibc versions?

@ssd responded:

I’ll likely have to solve similar problems soon. I think your ldd output might be a bit misleading though, the version of ld-linux is just based on whatever version of ldd you have binlinked, not baked into the so

@bixu Is this an issue you are getting in Bldr or in your own build process or both?
And, if it is a build process outside of bldr, do you happen to be using a custom HAB_BLDR_CHANNEL?

@bixu said:

we see it in both Builder and in Studio builds.
We are using 0.58.0

@ssd:

Heya hab-team. I think we need to figure out some advice for people about what to do about the kind of build issues @bixu is hitting, at least until the work that allows us to unpin the backline packages is done.

The crux of his issue seems to be this:

  1. His build requires setting LD_LIBRARY_PATH
  2. Some of the libraries in the LD_LIBRARY_PATH includes libraries used by grep
  3. The grep used by hab-plan-build is pinned to the old packages and thus incompatible with the library in LD_LIBRARY_PATH.

This isn’t a grep-specific problem. It can occur with any of the tools required by hab-plan-build if you happen to set LD_LIBRARY_PATH or other variables via your build.

Anyway, that is the summary of the issue. It is the same basic issue as people were seeing with attach() and 0.58 didn’t fix it because we had to pin backline packages because of the breaking changes in glibc.

I believe the attach thread @ssd is referring to is https://discourse.chef.io/t/cannot-attach-in-a-plan-where-cflags-library-path-or-ld-library-path-are-modified/650

2

@ssd and I discussed the possibility that directly depending on core/grep might have helped here, but I’ve just tried this without success - we still get the relocation errors.

3

Why does your build need LD_LIBRARY_PATH? Is it possible to find the offending bin/lib and patchelf it to use the specific shared library?

4

In addition to @bdangit’s suggestion, it seems like perhaps you could either:

  1. Pin all dependencies to the pre-core-plans-refresh versions as we’ve done with https://github.com/habitat-sh/habitat/pull/5238
  2. If you don’t need pre 3.2 kernel support, perhaps we can set something up to build everything post core-plans refresh.
5

It’s a hacky solution - but the only one I’ve seen work in this situation is to pin all dependencies to the pre-core-plans-refresh versions if you have a dependency that needs per 3.2 kernel support.

6

Actually…let me look at this a bit deeper.

7

I wonder if we can rebuild core/hab-plan-build to use the updated core/grep?

Looking into this.

8

Interestingly, although hab-plan-build is a core plan, it is not in the core plans github repo, which means it was not rebuilt as part of the base plans refresh, which may be why it is using an older version of core/grep, which uses the older glibc, seeing if we can rebuild this…

9

Ah - I see that hab-plan-build is pinned to the older version of core/grep https://github.com/habitat-sh/habitat/blob/master/components/plan-build/plan.sh

Investigating further.

10

@bixu - can you try pinning core/pcre to core/pcre/8.38/20170513213423? That is the latest stable version prior to the base plans refresh.

11

I agree that there is some amount of pinning that will work for this case. I think that is probably a fine work-around if we think we will be able to unpin in the near future (long-lived pins tend to live much longer than the reason that generated them). If we do think that is the best alternative right now, perhaps we can put together a list of libraries that may need to be pinned currently since this could happen with just about any library used by executables called by hab-plan-build.

12

@ssd indeed - this is a short term solution until the multi-architecture work is complete. I will put together a wiki post listing troublesome libraries and instructions on what to pin them to.

13

Just added a wiki post with a guide - if anyone knows of other packages that need to be pinned, please feel free to add them!

https://discourse.chef.io/t/temporary-workaround-for-glibc-dependency-errors/692