Issues with dependency conflicts after core-plans rebuild

smartb-pair · June 20, 2018, 10:05am

We are running into a lot of issues with Builder and transitive dependency conflicts today. We’re assuming this is because of the core plans refresh. Here’s an example of the conflict list:

smartb-private/api/0.1.0/20180620095545
    core/bash/4.4.19/20180608092913 (*)
        core/glibc/2.27/20180608041157 (*)
            core/linux-headers/4.15.9/20180608041107 (*)
        core/ncurses/6.1/20180608091810 (*)
            core/gcc-libs/7.3.0/20180608091701 (*)
                core/glibc/2.27/20180608041157 (*)
                    core/linux-headers/4.15.9/20180608041107 (*)
            core/glibc/2.27/20180608041157 (*)
                core/linux-headers/4.15.9/20180608041107 (*)
        core/readline/7.0.3/20180608092900 (*)
            core/glibc/2.27/20180608041157 (*)
                core/linux-headers/4.15.9/20180608041107 (*)
            core/ncurses/6.1/20180608091810 (*)
                core/gcc-libs/7.3.0/20180608091701 (*)
                    core/glibc/2.27/20180608041157 (*)
                        core/linux-headers/4.15.9/20180608041107 (*)
                core/glibc/2.27/20180608041157 (*)
                    core/linux-headers/4.15.9/20180608041107 (*)
    core/bc/1.07.1/20180608093241
        core/flex/2.6.4/20180608092657
            core/glibc/2.27/20180608041157 (*)
                core/linux-headers/4.15.9/20180608041107 (*)
        core/glibc/2.27/20180608041157 (*)
            core/linux-headers/4.15.9/20180608041107 (*)
        core/ncurses/6.1/20180608091810 (*)
            core/gcc-libs/7.3.0/20180608091701 (*)
                core/glibc/2.27/20180608041157 (*)
                    core/linux-headers/4.15.9/20180608041107 (*)
            core/glibc/2.27/20180608041157 (*)
                core/linux-headers/4.15.9/20180608041107 (*)
        core/readline/7.0.3/20180608092900 (*)
            core/glibc/2.27/20180608041157 (*)
                core/linux-headers/4.15.9/20180608041107 (*)
            core/ncurses/6.1/20180608091810 (*)
                core/gcc-libs/7.3.0/20180608091701 (*)
                    core/glibc/2.27/20180608041157 (*)
                        core/linux-headers/4.15.9/20180608041107 (*)
                core/glibc/2.27/20180608041157 (*)
                    core/linux-headers/4.15.9/20180608041107 (*)
    core/coreutils/8.29/20180608092141 (*)
        core/acl/2.2.52/20180608091922 (*)
            core/attr/2.4.47/20180608091914 (*)
                core/glibc/2.27/20180608041157 (*)
                    core/linux-headers/4.15.9/20180608041107 (*)
            core/glibc/2.27/20180608041157 (*)
                core/linux-headers/4.15.9/20180608041107 (*)
        core/attr/2.4.47/20180608091914 (*)
            core/glibc/2.27/20180608041157 (*)
                core/linux-headers/4.15.9/20180608041107 (*)
        core/glibc/2.27/20180608041157 (*)
            core/linux-headers/4.15.9/20180608041107 (*)
        core/gmp/6.1.2/20180608051426 (*)
            core/glibc/2.27/20180608041157 (*)
                core/linux-headers/4.15.9/20180608041107 (*)
        core/libcap/2.25/20180608091936 (*)
            core/attr/2.4.47/20180608091914 (*)
                core/glibc/2.27/20180608041157 (*)
                    core/linux-headers/4.15.9/20180608041107 (*)
            core/glibc/2.27/20180608041157 (*)
                core/linux-headers/4.15.9/20180608041107 (*)
    core/gcc/7.3.0/20180608051919
        core/binutils/2.30/20180608050633
            core/glibc/2.27/20180608041157 (*)
                core/linux-headers/4.15.9/20180608041107 (*)
            core/zlib/1.2.11/20180608050617 (*)
                core/glibc/2.27/20180608041157 (*)
                    core/linux-headers/4.15.9/20180608041107 (*)
        core/glibc/2.27/20180608041157 (*)
            core/linux-headers/4.15.9/20180608041107 (*)
        core/gmp/6.1.2/20180608051426 (*)
            core/glibc/2.27/20180608041157 (*)
                core/linux-headers/4.15.9/20180608041107 (*)
        core/libmpc/1.1.0/20180608051824
            core/glibc/2.27/20180608041157 (*)
                core/linux-headers/4.15.9/20180608041107 (*)
            core/gmp/6.1.2/20180608051426 (*)
                core/glibc/2.27/20180608041157 (*)
                    core/linux-headers/4.15.9/20180608041107 (*)
            core/mpfr/4.0.1/20180608051629
                core/glibc/2.27/20180608041157 (*)
                    core/linux-headers/4.15.9/20180608041107 (*)
                core/gmp/6.1.2/20180608051426 (*)
                    core/glibc/2.27/20180608041157 (*)
                        core/linux-headers/4.15.9/20180608041107 (*)
        core/mpfr/4.0.1/20180608051629
            core/glibc/2.27/20180608041157 (*)
                core/linux-headers/4.15.9/20180608041107 (*)
            core/gmp/6.1.2/20180608051426 (*)
                core/glibc/2.27/20180608041157 (*)
                    core/linux-headers/4.15.9/20180608041107 (*)
        core/zlib/1.2.11/20180608050617 (*)
            core/glibc/2.27/20180608041157 (*)
                core/linux-headers/4.15.9/20180608041107 (*)
    core/gcc-libs/7.3.0/20180608091701 (*)
        core/glibc/2.27/20180608041157 (*)
            core/linux-headers/4.15.9/20180608041107 (*)
    core/libffi/3.2.1/20180608141245
        core/glibc/2.27/20180608041157 (*)
            core/linux-headers/4.15.9/20180608041107 (*)
        core/libtool/2.4.6/20180608093951
            core/binutils/2.30/20180608050633
                core/glibc/2.27/20180608041157 (*)
                    core/linux-headers/4.15.9/20180608041107 (*)
                core/zlib/1.2.11/20180608050617 (*)
                    core/glibc/2.27/20180608041157 (*)
                        core/linux-headers/4.15.9/20180608041107 (*)
            core/coreutils/8.29/20180608092141 (*)
                core/acl/2.2.52/20180608091922 (*)
                    core/attr/2.4.47/20180608091914 (*)
                        core/glibc/2.27/20180608041157 (*)
                            core/linux-headers/4.15.9/20180608041107 (*)
                    core/glibc/2.27/20180608041157 (*)
                        core/linux-headers/4.15.9/20180608041107 (*)
                core/attr/2.4.47/20180608091914 (*)
                    core/glibc/2.27/20180608041157 (*)
                        core/linux-headers/4.15.9/20180608041107 (*)
                core/glibc/2.27/20180608041157 (*)
                    core/linux-headers/4.15.9/20180608041107 (*)
                core/gmp/6.1.2/20180608051426 (*)
                    core/glibc/2.27/20180608041157 (*)
                        core/linux-headers/4.15.9/20180608041107 (*)
                core/libcap/2.25/20180608091936 (*)
                    core/attr/2.4.47/20180608091914 (*)
                        core/glibc/2.27/20180608041157 (*)
                            core/linux-headers/4.15.9/20180608041107 (*)
                    core/glibc/2.27/20180608041157 (*)
                        core/linux-headers/4.15.9/20180608041107 (*)
            core/glibc/2.27/20180608041157 (*)
                core/linux-headers/4.15.9/20180608041107 (*)
            core/grep/3.1/20180608092809
                core/glibc/2.27/20180608041157 (*)
                    core/linux-headers/4.15.9/20180608041107 (*)
                core/pcre/8.41/20180608092740 (*)
                    core/gcc-libs/7.3.0/20180608091701 (*)
                        core/glibc/2.27/20180608041157 (*)
                            core/linux-headers/4.15.9/20180608041107 (*)
                    core/glibc/2.27/20180608041157 (*)
                        core/linux-headers/4.15.9/20180608041107 (*)
            core/sed/4.4/20180608091938
                core/acl/2.2.52/20180608091922 (*)
                    core/attr/2.4.47/20180608091914 (*)
                        core/glibc/2.27/20180608041157 (*)
                            core/linux-headers/4.15.9/20180608041107 (*)
                    core/glibc/2.27/20180608041157 (*)
                        core/linux-headers/4.15.9/20180608041107 (*)
                core/glibc/2.27/20180608041157 (*)
                    core/linux-headers/4.15.9/20180608041107 (*)
    core/pcre/8.41/20180608092740 (*)
        core/gcc-libs/7.3.0/20180608091701 (*)
            core/glibc/2.27/20180608041157 (*)
                core/linux-headers/4.15.9/20180608041107 (*)
        core/glibc/2.27/20180608041157 (*)
            core/linux-headers/4.15.9/20180608041107 (*)
    core/procps-ng/3.3.12/20180608092056
        core/glibc/2.27/20180608041157 (*)
            core/linux-headers/4.15.9/20180608041107 (*)
        core/ncurses/6.1/20180608091810 (*)
            core/gcc-libs/7.3.0/20180608091701 (*)
                core/glibc/2.27/20180608041157 (*)
                    core/linux-headers/4.15.9/20180608041107 (*)
            core/glibc/2.27/20180608041157 (*)
                core/linux-headers/4.15.9/20180608041107 (*)
    core/python/3.6.3/20180608141331
        core/bzip2/1.0.6/20180608091727 (*)
            core/glibc/2.27/20180608041157 (*)
                core/linux-headers/4.15.9/20180608041107 (*)
        core/gcc-libs/7.3.0/20180608091701 (*)
            core/glibc/2.27/20180608041157 (*)
                core/linux-headers/4.15.9/20180608041107 (*)
        core/gdbm/1.14.1/20180608094002 (*)
            core/glibc/2.27/20180608041157 (*)
                core/linux-headers/4.15.9/20180608041107 (*)
        core/glibc/2.27/20180608041157 (*)
            core/linux-headers/4.15.9/20180608041107 (*)
        core/ncurses/6.1/20180608091810 (*)
            core/gcc-libs/7.3.0/20180608091701 (*)
                core/glibc/2.27/20180608041157 (*)
                    core/linux-headers/4.15.9/20180608041107 (*)
            core/glibc/2.27/20180608041157 (*)
                core/linux-headers/4.15.9/20180608041107 (*)
        core/openssl/1.0.2n/20180608102213 (*)
            core/cacerts/2018.03.07/20180608102212 (*)
            core/glibc/2.27/20180608041157 (*)
                core/linux-headers/4.15.9/20180608041107 (*)
            core/zlib/1.2.11/20180608050617 (*)
                core/glibc/2.27/20180608041157 (*)
                    core/linux-headers/4.15.9/20180608041107 (*)
        core/readline/7.0.3/20180608092900 (*)
            core/glibc/2.27/20180608041157 (*)
                core/linux-headers/4.15.9/20180608041107 (*)
            core/ncurses/6.1/20180608091810 (*)
                core/gcc-libs/7.3.0/20180608091701 (*)
                    core/glibc/2.27/20180608041157 (*)
                        core/linux-headers/4.15.9/20180608041107 (*)
                core/glibc/2.27/20180608041157 (*)
                    core/linux-headers/4.15.9/20180608041107 (*)
        core/sqlite/3130000/20180608141313
            core/glibc/2.27/20180608041157 (*)
                core/linux-headers/4.15.9/20180608041107 (*)
            core/readline/7.0.3/20180608092900 (*)
                core/glibc/2.27/20180608041157 (*)
                    core/linux-headers/4.15.9/20180608041107 (*)
                core/ncurses/6.1/20180608091810 (*)
                    core/gcc-libs/7.3.0/20180608091701 (*)
                        core/glibc/2.27/20180608041157 (*)
                            core/linux-headers/4.15.9/20180608041107 (*)
                    core/glibc/2.27/20180608041157 (*)
                        core/linux-headers/4.15.9/20180608041107 (*)
        core/zlib/1.2.11/20180608050617 (*)
            core/glibc/2.27/20180608041157 (*)
                core/linux-headers/4.15.9/20180608041107 (*)
    core/runit/2.1.2/20180609180740
        core/glibc/2.27/20180608041157 (*)
            core/linux-headers/4.15.9/20180608041107 (*)
    smartb/pgbouncer/1.7.2/20180502160435
        core/bash/4.3.42/20170513213519 (*)
            core/glibc/2.22/20170513201042 (*)
                core/linux-headers/4.3/20170513200956 (*)
            core/ncurses/6.0/20170513213009 (*)
                core/glibc/2.22/20170513201042 (*)
                    core/linux-headers/4.3/20170513200956 (*)
                core/gcc-libs/5.2.0/20170513212920 (*)
                    core/glibc/2.22/20170513201042 (*)
                        core/linux-headers/4.3/20170513200956 (*)
            core/readline/6.3.8/20170513213506 (*)
                core/glibc/2.22/20170513201042 (*)
                    core/linux-headers/4.3/20170513200956 (*)
                core/ncurses/6.0/20170513213009 (*)
                    core/glibc/2.22/20170513201042 (*)
                        core/linux-headers/4.3/20170513200956 (*)
                    core/gcc-libs/5.2.0/20170513212920 (*)
                        core/glibc/2.22/20170513201042 (*)
                            core/linux-headers/4.3/20170513200956 (*)
        core/glibc/2.22/20170513201042 (*)
            core/linux-headers/4.3/20170513200956 (*)
        core/libevent/2.0.22/20170514151137
            core/glibc/2.22/20170513201042 (*)
                core/linux-headers/4.3/20170513200956 (*)
        core/openssl/1.0.2l/20171014213633 (*)
            core/glibc/2.22/20170513201042 (*)
                core/linux-headers/4.3/20170513200956 (*)
            core/zlib/1.2.8/20170513201911 (*)
                core/glibc/2.22/20170513201042 (*)
                    core/linux-headers/4.3/20170513200956 (*)
            core/cacerts/2017.09.20/20171014212239 (*)
        jonathantron/c-ares/1.12.0/20170526185528
            core/glibc/2.22/20170513201042 (*)
                core/linux-headers/4.3/20170513200956 (*)
    smartb/postgresql/9.4.14/20171222140635
        core/bash/4.3.42/20170513213519 (*)
            core/glibc/2.22/20170513201042 (*)
                core/linux-headers/4.3/20170513200956 (*)
            core/ncurses/6.0/20170513213009 (*)
                core/glibc/2.22/20170513201042 (*)
                    core/linux-headers/4.3/20170513200956 (*)
                core/gcc-libs/5.2.0/20170513212920 (*)
                    core/glibc/2.22/20170513201042 (*)
                        core/linux-headers/4.3/20170513200956 (*)
            core/readline/6.3.8/20170513213506 (*)
                core/glibc/2.22/20170513201042 (*)
                    core/linux-headers/4.3/20170513200956 (*)
                core/ncurses/6.0/20170513213009 (*)
                    core/glibc/2.22/20170513201042 (*)
                        core/linux-headers/4.3/20170513200956 (*)
                    core/gcc-libs/5.2.0/20170513212920 (*)
                        core/glibc/2.22/20170513201042 (*)
                            core/linux-headers/4.3/20170513200956 (*)
        core/glibc/2.22/20170513201042 (*)
            core/linux-headers/4.3/20170513200956 (*)
        core/libossp-uuid/1.6.2/20170514001348
            core/glibc/2.22/20170513201042 (*)
                core/linux-headers/4.3/20170513200956 (*)
        core/openssl/1.0.2l/20171014213633 (*)
            core/glibc/2.22/20170513201042 (*)
                core/linux-headers/4.3/20170513200956 (*)
            core/zlib/1.2.8/20170513201911 (*)
                core/glibc/2.22/20170513201042 (*)
                    core/linux-headers/4.3/20170513200956 (*)
            core/cacerts/2017.09.20/20171014212239 (*)
        core/perl/5.22.1/20170513213942
            core/glibc/2.22/20170513201042 (*)
                core/linux-headers/4.3/20170513200956 (*)
            core/zlib/1.2.8/20170513201911 (*)
                core/glibc/2.22/20170513201042 (*)
                    core/linux-headers/4.3/20170513200956 (*)
            core/bzip2/1.0.6/20170513212938 (*)
                core/glibc/2.22/20170513201042 (*)
                    core/linux-headers/4.3/20170513200956 (*)
            core/gdbm/1.11/20170513213716 (*)
                core/glibc/2.22/20170513201042 (*)
                    core/linux-headers/4.3/20170513200956 (*)
            core/db/5.3.28/20170513213734
                core/glibc/2.22/20170513201042 (*)
                    core/linux-headers/4.3/20170513200956 (*)
                core/gcc-libs/5.2.0/20170513212920 (*)
                    core/glibc/2.22/20170513201042 (*)
                        core/linux-headers/4.3/20170513200956 (*)
            core/coreutils/8.25/20170513213226 (*)
                core/glibc/2.22/20170513201042 (*)
                    core/linux-headers/4.3/20170513200956 (*)
                core/acl/2.2.52/20170513213108 (*)
                    core/glibc/2.22/20170513201042 (*)
                        core/linux-headers/4.3/20170513200956 (*)
                    core/attr/2.4.47/20170513213059 (*)
                        core/glibc/2.22/20170513201042 (*)
                            core/linux-headers/4.3/20170513200956 (*)
                core/attr/2.4.47/20170513213059 (*)
                    core/glibc/2.22/20170513201042 (*)
                        core/linux-headers/4.3/20170513200956 (*)
                core/gmp/6.1.0/20170513202112 (*)
                    core/glibc/2.22/20170513201042 (*)
                        core/linux-headers/4.3/20170513200956 (*)
                core/libcap/2.24/20170513213120 (*)
                    core/glibc/2.22/20170513201042 (*)
                        core/linux-headers/4.3/20170513200956 (*)
                    core/attr/2.4.47/20170513213059 (*)
                        core/glibc/2.22/20170513201042 (*)
                            core/linux-headers/4.3/20170513200956 (*)
            core/less/481/20170513213936
                core/glibc/2.22/20170513201042 (*)
                    core/linux-headers/4.3/20170513200956 (*)
                core/ncurses/6.0/20170513213009 (*)
                    core/glibc/2.22/20170513201042 (*)
                        core/linux-headers/4.3/20170513200956 (*)
                    core/gcc-libs/5.2.0/20170513212920 (*)
                        core/glibc/2.22/20170513201042 (*)
                            core/linux-headers/4.3/20170513200956 (*)
                core/pcre/8.38/20170513213423 (*)
                    core/glibc/2.22/20170513201042 (*)
                        core/linux-headers/4.3/20170513200956 (*)
                    core/gcc-libs/5.2.0/20170513212920 (*)
                        core/glibc/2.22/20170513201042 (*)
                            core/linux-headers/4.3/20170513200956 (*)
        core/readline/6.3.8/20170513213506 (*)
            core/glibc/2.22/20170513201042 (*)
                core/linux-headers/4.3/20170513200956 (*)
            core/ncurses/6.0/20170513213009 (*)
                core/glibc/2.22/20170513201042 (*)
                    core/linux-headers/4.3/20170513200956 (*)
                core/gcc-libs/5.2.0/20170513212920 (*)
                    core/glibc/2.22/20170513201042 (*)
                        core/linux-headers/4.3/20170513200956 (*)
        core/zlib/1.2.8/20170513201911 (*)
            core/glibc/2.22/20170513201042 (*)
                core/linux-headers/4.3/20170513200956 (*)
    smartb/botocore/1.8.2/20180413085304
    smartb/Cython/0.27.3/20180413085427
    smartb/numpy/1.13.1/20180413085546
    smartb/pandas/0.22.0/20180413085744
    smartb/scipy/0.19.1/20180413090041
    smartb/tensorflow/1.4.1/20180503093745

How can we avoid this issue in the future? We had hoped reverse-dependency builds would keep this from happening, but apparently they do not?

cc: @bixu

smartb-pair · June 20, 2018, 12:02pm

It looks like this is related to a library package we maintain, smartb/tensorflow. We currently don’t have a good way to auto-promote this package to the stable channel…

elliott-davis · June 20, 2018, 12:03pm

In this case the reverse dep builds didn’t happen since we essentially uploaded a new base set. To fix this particular issue you need to trip the reverse deps yourself (pgbouncer, postgres, etc)

smartb-pair · June 20, 2018, 12:04pm

Is there a way we can automate this? Even the Habitat ‘experts’ on our team found this behavior difficult to understand/debug.

smartb-pair · June 20, 2018, 12:15pm

Additionally, we aren’t able to trigger manual rebuilds for the following library packages:

smartb/scipy
smartb/pandas

smartb-pair · June 20, 2018, 12:18pm

We’ve tried looking at our current Builder jobs from the CLI (hab bldr job status) but that command fails like:

 $ RUST_BACKTRACE=1 hab bldr job status
thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', src/libcore/option.rs:335:21
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
   1: std::panicking::default_hook::{{closure}}
   2: std::panicking::default_hook
   3: std::panicking::rust_panic_with_hook
   4: std::panicking::begin_panic
   5: std::panicking::begin_panic_fmt
   6: rust_begin_unwind
   7: core::panicking::panic_fmt
   8: core::panicking::panic
   9: hab::command::bldr::job::status::start
  10: hab::start
  11: hab::main
  12: std::rt::lang_start::{{closure}}
  13: std::panicking::try::do_call
  14: __rust_maybe_catch_panic
  15: std::rt::lang_start_internal
  16: main

smartb-pair · June 20, 2018, 12:21pm

We were able to get job status by upgrading to version 0.57.0, but we still don’t see Builder jobs for either smartb/scipy or smartb/pandas:

$ hab bldr job status --origin smartb
☁ Determining status of job groups in smartb origin

CREATED AT                        GROUP ID             STATUS    IDENT
2018-06-20T12:10:57.562994+00:00  1008566253659316224  Complete  smartb/numpy
2018-06-20T12:10:20.612386+00:00  1008565943691853824  Complete  smartb/botocore
2018-06-20T12:09:58.815592+00:00  1008565760862134272  Complete  smartb/Cython
2018-06-20T11:34:57.116929+00:00  1008548130524676096  Complete  smartb/pgbouncer
2018-06-20T11:30:03.032569+00:00  1008545663577661440  Complete  smartb/pgbouncer
2018-06-20T11:25:28.310004+00:00  1008543359042486272  Complete  smartb/c-ares
2018-06-20T10:01:47.968402+00:00  1008501245344628736  Complete  smartb/pgbouncer
2018-06-20T09:57:17.686505+00:00  1008498978071650304  Complete  smartb/tensorflow
2018-06-15T14:59:15.463877+00:00  1005027082359685120  Complete  smartb/irondb
2018-06-14T17:35:06.613999+00:00  1004380749751492608  Complete  smartb/irondb

smartb-pair · June 20, 2018, 12:38pm

We are also noticing that the output of hab bldr job status --origin smartb doesn’t show some jobs that have been started via the Builder UI.

smartb-pair · June 20, 2018, 12:48pm

Final resolution to this was to rebuild our smartb/postgresql package. Is there a faster way to identify which dependencies are at the root of large conflict trees like the one we had? Debugging this took several hours

elliott-davis · June 20, 2018, 1:11pm

It’s mostly eyeballing it and knowing what your dependency graph looks like. You might be able to use the build-dependent-order.rb script from core-plans if all of your plans are in the same repo. I haven’t ever leveraged it outside of core-plans though so your mileage may vary.

holoway · June 20, 2018, 2:09pm

We definitely need to get the reverse dep builds working universally.

bixu · July 10, 2018, 2:11pm

By “working universally” do we mean both DEPS and TDEPS triggering a build when they update?

Do we also mean reverse deps triggering across more than one level of dependency?

holoway · July 16, 2018, 5:02pm

Yes, that’s what I mean

Topic		Replies	Views
Temporary workaround for glibc dependency errors Habitat Wiki	0	669	July 10, 2018
Dependency conflicts between different versions of glibc Habitat Help	12	1669	July 10, 2018
Can we expect reverse-dependency builds in Habitat Builder with multiple levels of dependencies? Habitat Help	3	441	April 13, 2018
Hardening habitat against upstream changes / failures Habitat Help	7	649	December 6, 2018
Zlib, The Universe and You! Chef Habitat	1	644	October 31, 2016

Issues with dependency conflicts after core-plans rebuild

Related topics