For context I’ve done this already Howto - Upload package output from a CI/CD build to private builder
When I try to install my package hab pkg install -u https://yourcompany my/awesomepackage it finds my package but then fails on a core/acl dependency
The API and builder UI both seem to suggest that core/acl is present.
Walls of text to follow…
curl https://hab-store.icts.uiowa.edu/v1/depot/pkgs/core/acl
{"range_start":0,"range_end":1,"total_count":2,"data":[{"channels":["stable","unstable"],"name":"acl","origin":"core","platforms":["x86_64-linux"],"release":"20180608091922","version":"2.2.52"},{"channels":["stable","unstable"],"name":"acl","origin":"core","platforms":["x86_64-linux"],"release":"20170513213108","version":"2.2.52"}]}
[root@localhost ~]# RUST_LOG=debug hab pkg install -u https://hab-store.icts.uiowa.edu -c stable core/acl
DEBUG 2018-07-26T21:05:37Z: habitat_common::ui: UI { shell: Shell { input: InputStream { isatty: true }, out: OutputStream { coloring: Auto, isatty: true, is_colored(): true, supports_color(): true }, err: OutputStream { coloring: Auto, isatty: true, is_colored(): true, supports_color(): true } } }
DEBUG 2018-07-26T21:05:37Z: hab: clap cli args: ["hab", "pkg", "install", "-u", "https://hab-store.icts.uiowa.edu", "-c", "stable", "core/acl"]
DEBUG 2018-07-26T21:05:37Z: hab: remaining cli args: []
DEBUG 2018-07-26T21:05:37Z: hab::config: No CLI config found, loading defaults
DEBUG 2018-07-26T21:05:37Z: habitat_common::command::package::install: install key_cache_path: /hab/cache/keys
DEBUG 2018-07-26T21:05:37Z: habitat_http_client::ssl: Setting CA file for SSL context to: /hab/cache/ssl/cert.pem
DEBUG 2018-07-26T21:05:37Z: habitat_http_client::api_client: Client socket timeout: 120 secs
DEBUG 2018-07-26T21:05:37Z: habitat_http_client::api_client: User-Agent: hab/0.59.0/20180712155441 (x86_64-linux; 3.10.0-693.21.1.el7.x86_64)
» Installing core/acl
☁ Determining latest version of core/acl in the 'stable' channel
DEBUG 2018-07-26T21:05:37Z: habitat_http_client::api_client: GET https://hab-store.icts.uiowa.edu/v1/depot/channels/core/stable/pkgs/acl/latest with ApiClient { endpoint: "https://hab-store.icts.uiowa.edu/v1", inner: Client { redirect_policy: FollowAll, read_timeout: Some(Duration { secs: 120, nanos: 0 }), write_timeout: Some(Duration { secs: 120, nanos: 0 }), proxy: None }, proxy: None, target_scheme: "https", user_agent_header: UserAgent("hab/0.59.0/20180712155441 (x86_64-linux; 3.10.0-693.21.1.el7.x86_64)") }
DEBUG 2018-07-26T21:05:37Z: hyper::net: http scheme
DEBUG 2018-07-26T21:05:37Z: hyper::net: https scheme
DEBUG 2018-07-26T21:05:37Z: hyper::http::h1: request line: Get "/v1/depot/channels/core/stable/pkgs/acl/latest" Http11
DEBUG 2018-07-26T21:05:37Z: hyper::http::h1: headers=Headers { Host: hab-store.icts.uiowa.edu
, User-Agent: hab/0.59.0/20180712155441 (x86_64-linux; 3.10.0-693.21.1.el7.x86_64)
, }
DEBUG 2018-07-26T21:05:37Z: hyper::client::response: version=Http11, status=Ok
DEBUG 2018-07-26T21:05:37Z: hyper::client::response: headers=Headers { Server: nginx/1.13.10
, Date: Thu, 26 Jul 2018 21:05:37 GMT
, Content-Type: application/json; charset=utf-8
, Content-Length: 4994
, Connection: keep-alive
, Vary: Accept-Encoding
, ETag: a574a45003163e1d3b2cab070c1b8e231b9ee1791d2c3b0f1719bbf2f8a18cb1
, Cache-Control: private, no-cache, no-store
Cache-Control: private, no-cache, no-store
, Access-Control-Allow-Origin: *
, Access-Control-Allow-Headers: authorization, range
, Access-Control-Allow-Methods: PUT, DELETE, PATCH
, Access-Control-Expose-Headers: content-disposition
, }
DEBUG 2018-07-26T21:05:37Z: habitat_depot_client: Body: "{\"channels\":[\"stable\",\"unstable\"],\"checksum\":\"a574a45003163e1d3b2cab070c1b8e231b9ee1791d2c3b0f1719bbf2f8a18cb1\",\"config\":\"\",\"deps\":[{\"name\":\"attr\",\"origin\":\"core\",\"release\":\"20180608091914\",\"version\":\"2.4.47\"},{\"name\":\"glibc\",\"origin\":\"core\",\"release\":\"20180608041157\",\"version\":\"2.27\"}],\"exposes\":[],\"ident\":{\"name\":\"acl\",\"origin\":\"core\",\"release\":\"20180608091922\",\"version\":\"2.2.52\"},\"is_a_service\":false,\"manifest\":\"# core / acl\\nCommands for Manipulating POSIX Access Control Lists.\\n\\n* __Maintainer__: The Habitat Maintainers <humans@habitat.sh>\\n* __Version__: 2.2.52\\n* __Release__: 20180608091922\\n* __Architecture__: x86_64\\n* __System__: linux\\n* __Target__: x86_64-linux\\n* __Upstream URL__: [https://savannah.nongnu.org/projects/acl](https://savannah.nongnu.org/projects/acl)\\n* __License__: lgpl \\n* __Source__: [http://download.savannah.gnu.org/releases/acl/acl-2.2.52.src.tar.gz](http://download.savannah.gnu.org/releases/acl/acl-2.2.52.src.tar.gz)\\n* __SHA__: `179074bb0580c06c4b4137be4c5a92a701583277967acdb5546043c7874e0d23`\\n* __Path__: `/hab/pkgs/core/acl/2.2.52/20180608091922`\\n* __Build Dependencies__: `core/coreutils core/diffutils core/patch core/make core/file core/gcc core/gettext `\\n* __Dependencies__: `core/glibc core/attr `\\n* __Interpreters__: no interpreters or undefined\\n\\n# Plan\\n\\n## Build Flags\\n\\n```bash\\nCFLAGS: -I/hab/pkgs/core/glibc/2.27/20180608041157/include -I/hab/pkgs/core/attr/2.4.47/20180608091914/include -I/hab/pkgs/core/make/4.2.1/20180608031320/include -I/hab/pkgs/core/file/5.32/20180608050620/include -I/hab/pkgs/core/gcc/7.3.0/20180608051919/include -I/hab/pkgs/core/gettext/0.19.8/20180608030824/include\\nCPPFLAGS: -I/hab/pkgs/core/glibc/2.27/20180608041157/include -I/hab/pkgs/core/attr/2.4.47/20180608091914/include -I/hab/pkgs/core/make/4.2.1/20180608031320/include -I/hab/pkgs/core/file/5.32/20180608050620/include -I/hab/pkgs/core/gcc/7.3.0/20180608051919/include -I/hab/pkgs/core/gettext/0.19.8/20180608030824/include\\nCXXFLAGS: -I/hab/pkgs/core/glibc/2.27/20180608041157/include -I/hab/pkgs/core/attr/2.4.47/20180608091914/include -I/hab/pkgs/core/make/4.2.1/20180608031320/include -I/hab/pkgs/core/file/5.32/20180608050620/include -I/hab/pkgs/core/gcc/7.3.0/20180608051919/include -I/hab/pkgs/core/gettext/0.19.8/20180608030824/include\\nLDFLAGS: -L/hab/pkgs/core/glibc/2.27/20180608041157/lib -L/hab/pkgs/core/attr/2.4.47/20180608091914/lib -L/hab/pkgs/core/file/5.32/20180608050620/lib -L/hab/pkgs/core/gcc/7.3.0/20180608051919/lib -L/hab/pkgs/core/gettext/0.19.8/20180608030824/lib\\nLD_RUN_PATH: /hab/pkgs/core/acl/2.2.52/20180608091922/lib:/hab/pkgs/core/glibc/2.27/20180608041157/lib:/hab/pkgs/core/attr/2.4.47/20180608091914/lib\\n```\\n\\n## Plan Source\\n\\n```bash\\npkg_name=acl\\npkg_origin=core\\npkg_version=2.2.52\\npkg_maintainer=\\\"The Habitat Maintainers <humans@habitat.sh>\\\"\\npkg_description=\\\"Commands for Manipulating POSIX Access Control Lists.\\\"\\npkg_upstream_url=\\\"https://savannah.nongnu.org/projects/acl\\\"\\npkg_license=(\'lgpl\')\\npkg_source=\\\"http://download.savannah.gnu.org/releases/$pkg_name/$pkg_name-${pkg_version}.src.tar.gz\\\"\\npkg_shasum=\\\"179074bb0580c06c4b4137be4c5a92a701583277967acdb5546043c7874e0d23\\\"\\npkg_deps=(\\n core/glibc\\n core/attr\\n)\\npkg_build_deps=(\\n core/coreutils\\n core/diffutils\\n core/patch\\n core/make\\n core/file\\n core/gcc\\n core/gettext\\n)\\npkg_bin_dirs=(bin)\\npkg_include_dirs=(include)\\npkg_lib_dirs=(lib)\\n\\ndo_prepare() {\\n # Fix a bug that causes `getfacl -e` to segfault on overly long group name.\\n #\\n # Thanks to: http://www.linuxfromscratch.org/lfs/view/stable/chapter06/acl.html\\n sed -i -e \\\"/TABS-1;/a if (x > (TABS-1)) x = (TABS-1);\\\" \\\\\\n libacl/__acl_to_any_text.c\\n\\n # Update all references to the `/usr/bin/file` absolute path with `file`\\n # which will be on `$PATH` due to file being a build dependency.\\n grep -lr /usr/bin/file ./* | while read -r f; do\\n sed -i -e \\\"s,/usr/bin/file,file,g\\\" \\\"$f\\\"\\n done\\n}\\n\\ndo_install() {\\n make install install-dev install-lib\\n chmod -v 755 \\\"${pkg_prefix}/lib/libacl.so\\\"\\n}\\n\\n\\n# ----------------------------------------------------------------------------\\n# **NOTICE:** What follows are implementation details required for building a\\n# first-pass, \\\"stage1\\\" toolchain and environment. It is only used when running\\n# in a \\\"stage1\\\" Studio and can be safely ignored by almost everyone. Having\\n# said that, it performs a vital bootstrapping process and cannot be removed or\\n# significantly altered. Thank you!\\n# ----------------------------------------------------------------------------\\nif [[ \\\"$STUDIO_TYPE\\\" = \\\"stage1\\\" ]]; then\\n pkg_build_deps=(\\n core/gcc\\n )\\nfi\\n```\",\"target\":\"x86_64-linux\",\"tdeps\":[{\"name\":\"attr\",\"origin\":\"core\",\"release\":\"20180608091914\",\"version\":\"2.4.47\"},{\"name\":\"glibc\",\"origin\":\"core\",\"release\":\"20180608041157\",\"version\":\"2.27\"},{\"name\":\"linux-headers\",\"origin\":\"core\",\"release\":\"20180608041107\",\"version\":\"4.15.9\"}],\"visibility\":\"public\"}"
↓ Downloading core/acl/2.2.52/20180608091922
DEBUG 2018-07-26T21:05:37Z: habitat_http_client::api_client: GET https://hab-store.icts.uiowa.edu/v1/depot/pkgs/core/acl/2.2.52/20180608091922/download with ApiClient { endpoint: "https://hab-store.icts.uiowa.edu/v1", inner: Client { redirect_policy: FollowAll, read_timeout: Some(Duration { secs: 120, nanos: 0 }), write_timeout: Some(Duration { secs: 120, nanos: 0 }), proxy: None }, proxy: None, target_scheme: "https", user_agent_header: UserAgent("hab/0.59.0/20180712155441 (x86_64-linux; 3.10.0-693.21.1.el7.x86_64)") }
DEBUG 2018-07-26T21:05:37Z: hyper::http::h1: request line: Get "/v1/depot/pkgs/core/acl/2.2.52/20180608091922/download" Http11
DEBUG 2018-07-26T21:05:37Z: hyper::http::h1: headers=Headers { Host: hab-store.icts.uiowa.edu
, User-Agent: hab/0.59.0/20180712155441 (x86_64-linux; 3.10.0-693.21.1.el7.x86_64)
, }
DEBUG 2018-07-26T21:05:37Z: hyper::client::response: version=Http11, status=NotFound
DEBUG 2018-07-26T21:05:37Z: hyper::client::response: headers=Headers { Server: nginx/1.13.10
, Date: Thu, 26 Jul 2018 21:05:37 GMT
, Content-Length: 0
, Connection: keep-alive
, Access-Control-Allow-Origin: *
, Access-Control-Allow-Headers: authorization, range
, Access-Control-Allow-Methods: PUT, DELETE, PATCH
, Access-Control-Expose-Headers: content-disposition
, }
DEBUG 2018-07-26T21:05:37Z: habitat_depot_client: Response: Response { status: NotFound, headers: Headers { Server: nginx/1.13.10
, Date: Thu, 26 Jul 2018 21:05:37 GMT
, Content-Length: 0
, Connection: keep-alive
, Access-Control-Allow-Origin: *
, Access-Control-Allow-Headers: authorization, range
, Access-Control-Allow-Methods: PUT, DELETE, PATCH
, Access-Control-Expose-Headers: content-disposition
, }, version: Http11, url: "https://hab-store.icts.uiowa.edu/v1/depot/pkgs/core/acl/2.2.52/20180608091922/download", status_raw: RawStatus(404, "Not Found"), message: Http11Message { is_proxied: false, method: None, stream: Wrapper { obj: Some(Reading(SizedReader(remaining=0))) } } }
↓ Downloading core/acl/2.2.52/20180608091922
DEBUG 2018-07-26T21:05:40Z: habitat_http_client::api_client: GET https://hab-store.icts.uiowa.edu/v1/depot/pkgs/core/acl/2.2.52/20180608091922/download with ApiClient { endpoint: "https://hab-store.icts.uiowa.edu/v1", inner: Client { redirect_policy: FollowAll, read_timeout: Some(Duration { secs: 120, nanos: 0 }), write_timeout: Some(Duration { secs: 120, nanos: 0 }), proxy: None }, proxy: None, target_scheme: "https", user_agent_header: UserAgent("hab/0.59.0/20180712155441 (x86_64-linux; 3.10.0-693.21.1.el7.x86_64)") }
DEBUG 2018-07-26T21:05:40Z: hyper::http::h1: request line: Get "/v1/depot/pkgs/core/acl/2.2.52/20180608091922/download" Http11
DEBUG 2018-07-26T21:05:40Z: hyper::http::h1: headers=Headers { Host: hab-store.icts.uiowa.edu
, User-Agent: hab/0.59.0/20180712155441 (x86_64-linux; 3.10.0-693.21.1.el7.x86_64)
, }
DEBUG 2018-07-26T21:05:41Z: hyper::client::response: version=Http11, status=NotFound
DEBUG 2018-07-26T21:05:41Z: hyper::client::response: headers=Headers { Server: nginx/1.13.10
, Date: Thu, 26 Jul 2018 21:05:40 GMT
, Content-Length: 0
, Connection: keep-alive
, Access-Control-Allow-Origin: *
, Access-Control-Allow-Headers: authorization, range
, Access-Control-Allow-Methods: PUT, DELETE, PATCH
, Access-Control-Expose-Headers: content-disposition
, }
DEBUG 2018-07-26T21:05:41Z: habitat_depot_client: Response: Response { status: NotFound, headers: Headers { Server: nginx/1.13.10
, Date: Thu, 26 Jul 2018 21:05:40 GMT
, Content-Length: 0
, Connection: keep-alive
, Access-Control-Allow-Origin: *
, Access-Control-Allow-Headers: authorization, range
, Access-Control-Allow-Methods: PUT, DELETE, PATCH
, Access-Control-Expose-Headers: content-disposition
, }, version: Http11, url: "https://hab-store.icts.uiowa.edu/v1/depot/pkgs/core/acl/2.2.52/20180608091922/download", status_raw: RawStatus(404, "Not Found"), message: Http11Message { is_proxied: false, method: None, stream: Wrapper { obj: Some(Reading(SizedReader(remaining=0))) } } }
↓ Downloading core/acl/2.2.52/20180608091922
DEBUG 2018-07-26T21:05:44Z: habitat_http_client::api_client: GET https://hab-store.icts.uiowa.edu/v1/depot/pkgs/core/acl/2.2.52/20180608091922/download with ApiClient { endpoint: "https://hab-store.icts.uiowa.edu/v1", inner: Client { redirect_policy: FollowAll, read_timeout: Some(Duration { secs: 120, nanos: 0 }), write_timeout: Some(Duration { secs: 120, nanos: 0 }), proxy: None }, proxy: None, target_scheme: "https", user_agent_header: UserAgent("hab/0.59.0/20180712155441 (x86_64-linux; 3.10.0-693.21.1.el7.x86_64)") }
DEBUG 2018-07-26T21:05:44Z: hyper::http::h1: request line: Get "/v1/depot/pkgs/core/acl/2.2.52/20180608091922/download" Http11
DEBUG 2018-07-26T21:05:44Z: hyper::http::h1: headers=Headers { Host: hab-store.icts.uiowa.edu
, User-Agent: hab/0.59.0/20180712155441 (x86_64-linux; 3.10.0-693.21.1.el7.x86_64)
, }
DEBUG 2018-07-26T21:05:44Z: hyper::client::response: version=Http11, status=NotFound
DEBUG 2018-07-26T21:05:44Z: hyper::client::response: headers=Headers { Server: nginx/1.13.10
, Date: Thu, 26 Jul 2018 21:05:44 GMT
, Content-Length: 0
, Connection: keep-alive
, Access-Control-Allow-Origin: *
, Access-Control-Allow-Headers: authorization, range
, Access-Control-Allow-Methods: PUT, DELETE, PATCH
, Access-Control-Expose-Headers: content-disposition
, }
DEBUG 2018-07-26T21:05:44Z: habitat_depot_client: Response: Response { status: NotFound, headers: Headers { Server: nginx/1.13.10
, Date: Thu, 26 Jul 2018 21:05:44 GMT
, Content-Length: 0
, Connection: keep-alive
, Access-Control-Allow-Origin: *
, Access-Control-Allow-Headers: authorization, range
, Access-Control-Allow-Methods: PUT, DELETE, PATCH
, Access-Control-Expose-Headers: content-disposition
, }, version: Http11, url: "https://hab-store.icts.uiowa.edu/v1/depot/pkgs/core/acl/2.2.52/20180608091922/download", status_raw: RawStatus(404, "Not Found"), message: Http11Message { is_proxied: false, method: None, stream: Wrapper { obj: Some(Reading(SizedReader(remaining=0))) } } }
↓ Downloading core/acl/2.2.52/20180608091922
DEBUG 2018-07-26T21:05:47Z: habitat_http_client::api_client: GET https://hab-store.icts.uiowa.edu/v1/depot/pkgs/core/acl/2.2.52/20180608091922/download with ApiClient { endpoint: "https://hab-store.icts.uiowa.edu/v1", inner: Client { redirect_policy: FollowAll, read_timeout: Some(Duration { secs: 120, nanos: 0 }), write_timeout: Some(Duration { secs: 120, nanos: 0 }), proxy: None }, proxy: None, target_scheme: "https", user_agent_header: UserAgent("hab/0.59.0/20180712155441 (x86_64-linux; 3.10.0-693.21.1.el7.x86_64)") }
DEBUG 2018-07-26T21:05:47Z: hyper::http::h1: request line: Get "/v1/depot/pkgs/core/acl/2.2.52/20180608091922/download" Http11
DEBUG 2018-07-26T21:05:47Z: hyper::http::h1: headers=Headers { Host: hab-store.icts.uiowa.edu
, User-Agent: hab/0.59.0/20180712155441 (x86_64-linux; 3.10.0-693.21.1.el7.x86_64)
, }
DEBUG 2018-07-26T21:05:47Z: hyper::client::response: version=Http11, status=NotFound
DEBUG 2018-07-26T21:05:47Z: hyper::client::response: headers=Headers { Server: nginx/1.13.10
, Date: Thu, 26 Jul 2018 21:05:47 GMT
, Content-Length: 0
, Connection: keep-alive
, Access-Control-Allow-Origin: *
, Access-Control-Allow-Headers: authorization, range
, Access-Control-Allow-Methods: PUT, DELETE, PATCH
, Access-Control-Expose-Headers: content-disposition
, }
DEBUG 2018-07-26T21:05:47Z: habitat_depot_client: Response: Response { status: NotFound, headers: Headers { Server: nginx/1.13.10
, Date: Thu, 26 Jul 2018 21:05:47 GMT
, Content-Length: 0
, Connection: keep-alive
, Access-Control-Allow-Origin: *
, Access-Control-Allow-Headers: authorization, range
, Access-Control-Allow-Methods: PUT, DELETE, PATCH
, Access-Control-Expose-Headers: content-disposition
, }, version: Http11, url: "https://hab-store.icts.uiowa.edu/v1/depot/pkgs/core/acl/2.2.52/20180608091922/download", status_raw: RawStatus(404, "Not Found"), message: Http11Message { is_proxied: false, method: None, stream: Wrapper { obj: Some(Reading(SizedReader(remaining=0))) } } }
↓ Downloading core/acl/2.2.52/20180608091922
DEBUG 2018-07-26T21:05:50Z: habitat_http_client::api_client: GET https://hab-store.icts.uiowa.edu/v1/depot/pkgs/core/acl/2.2.52/20180608091922/download with ApiClient { endpoint: "https://hab-store.icts.uiowa.edu/v1", inner: Client { redirect_policy: FollowAll, read_timeout: Some(Duration { secs: 120, nanos: 0 }), write_timeout: Some(Duration { secs: 120, nanos: 0 }), proxy: None }, proxy: None, target_scheme: "https", user_agent_header: UserAgent("hab/0.59.0/20180712155441 (x86_64-linux; 3.10.0-693.21.1.el7.x86_64)") }
DEBUG 2018-07-26T21:05:50Z: hyper::http::h1: request line: Get "/v1/depot/pkgs/core/acl/2.2.52/20180608091922/download" Http11
DEBUG 2018-07-26T21:05:50Z: hyper::http::h1: headers=Headers { Host: hab-store.icts.uiowa.edu
, User-Agent: hab/0.59.0/20180712155441 (x86_64-linux; 3.10.0-693.21.1.el7.x86_64)
, }
DEBUG 2018-07-26T21:05:50Z: hyper::client::response: version=Http11, status=NotFound
DEBUG 2018-07-26T21:05:50Z: hyper::client::response: headers=Headers { Server: nginx/1.13.10
, Date: Thu, 26 Jul 2018 21:05:50 GMT
, Content-Length: 0
, Connection: keep-alive
, Access-Control-Allow-Origin: *
, Access-Control-Allow-Headers: authorization, range
, Access-Control-Allow-Methods: PUT, DELETE, PATCH
, Access-Control-Expose-Headers: content-disposition
, }
DEBUG 2018-07-26T21:05:50Z: habitat_depot_client: Response: Response { status: NotFound, headers: Headers { Server: nginx/1.13.10
, Date: Thu, 26 Jul 2018 21:05:50 GMT
, Content-Length: 0
, Connection: keep-alive
, Access-Control-Allow-Origin: *
, Access-Control-Allow-Headers: authorization, range
, Access-Control-Allow-Methods: PUT, DELETE, PATCH
, Access-Control-Expose-Headers: content-disposition
, }, version: Http11, url: "https://hab-store.icts.uiowa.edu/v1/depot/pkgs/core/acl/2.2.52/20180608091922/download", status_raw: RawStatus(404, "Not Found"), message: Http11Message { is_proxied: false, method: None, stream: Wrapper { obj: Some(Reading(SizedReader(remaining=0))) } } }
✗✗✗
✗✗✗ We tried 5 times but could not download core/acl/2.2.52/20180608091922. Giving up.
✗✗✗
[root@localhost ~]#I connected directly to minio and this particular version of the core/acl package definitely isn’t there. I have to think this has to do with how I build & upload
From what I can tell it looks like the S3 (minio) save happens before the database write which should prevent this sort of scenario.
Are there any logs or queues where minio would have said OK but really failed later?
I think I have tracked this down, am going to move it to a github issue
Ok good news is we’ve resolved this issue and the upstream uploading bug will be resolved in the next release! We’ve also added more validation on uploads and I’m working up a PR that adds a --force flag and once that’s in we’ll have a blog post on getting these resolved for deployments seeing these issues!