On-prem builder core origin package download failure

We just set up an on-prem builder and imported the core origin (latest build as of June 22, 1 am et). Any time we attempt to install a package from the core origin it fails. Here’s what we know:

  • hab pkg install core/jre8 will always fail after attempting to download 5 times. Both windows and linux packages
  • hab pkg search core/jre8 is successful
  • we can upload and download without an issue on other non-core origins
  • Upstream functionality works: if we ask for a windows package that doesn’t exist, the upstream will fetch it, but it cannot be download after it is fetched. The packages show up with valid version numbers in the core origin in the ui

Any ideas? or ideas on what log file I should be inspecting for more info?

@raskchanky do you have any thoughts on this?

builder proxy logs:

10.255.1.250 - - [22/Jun/2018:15:08:02 +0000] "GET /v1/depot/channels/core/stable/pkgs/jre8/latest HTTP/1.1" 200 8757 0.025 "-" "hab/0.57.0/20180614230004 (x86_64-linux; 4.9.87-linuxkit-aufs)" - -
10.255.1.250 - - [22/Jun/2018:15:08:02 +0000] "GET /v1/depot/pkgs/core/jre8/8.172.0/20180608184654/download HTTP/1.1" 404 0 0.013 "-" "hab/0.57.0/20180614230004 (x86_64-linux; 4.9.87-linuxkit-aufs)" - -
10.255.1.250 - - [22/Jun/2018:15:08:05 +0000] "GET /v1/depot/pkgs/core/jre8/8.172.0/20180608184654/download HTTP/1.1" 404 0 0.012 "-" "hab/0.57.0/20180614230004 (x86_64-linux; 4.9.87-linuxkit-aufs)" - -
10.255.1.250 - - [22/Jun/2018:15:08:08 +0000] "GET /v1/depot/pkgs/core/jre8/8.172.0/20180608184654/download HTTP/1.1" 404 0 0.013 "-" "hab/0.57.0/20180614230004 (x86_64-linux; 4.9.87-linuxkit-aufs)" - -
10.255.1.250 - - [22/Jun/2018:15:08:11 +0000] "GET /v1/depot/pkgs/core/jre8/8.172.0/20180608184654/download HTTP/1.1" 404 0 0.012 "-" "hab/0.57.0/20180614230004 (x86_64-linux; 4.9.87-linuxkit-aufs)" - -
10.255.1.250 - - [22/Jun/2018:15:08:14 +0000] "GET /v1/depot/pkgs/core/jre8/8.172.0/20180608184654/download HTTP/1.1" 404 0 0.013 "-" "hab/0.57.0/20180614230004 (x86_64-linux; 4.9.87-linuxkit-aufs)" - -

What happens if you:

curl -v https://your.internal.depot.domain.here/v1/depot/pkgs/core/jre8/8.172.0/20180608184654

What response do you get?

Assuming the above curl request fails, I’d be curious to know what response you get from:

curl -v https://your.internal.depot.domain.here/v1/depot/pkgs/core/jre8/latest

$ curl -v https://builder.ncrsaas.com/v1/depot/pkgs/core/jre8/8.172.0/20180608184654

* About to connect() to builder.ncrsaas.com port 443 (#0)
*   Trying 10.255.5.119...
* Connected to builder.ncrsaas.com (10.255.5.119) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
* 	subject: CN=*.ncrsaas.com,OU=SaaS Operations,O=NCR Corporation,L=Atlanta,ST=Georgia,C=US
* 	start date: Jan 29 00:00:00 2018 GMT
* 	expire date: Jan 29 12:00:00 2019 GMT
* 	common name: *.ncrsaas.com
* 	issuer: CN=DigiCert Global CA G2,O=DigiCert Inc,C=US
> GET /v1/depot/pkgs/core/jre8/8.172.0/20180608184654 HTTP/1.1
> User-Agent: curl/7.29.0
> Host: builder.ncrsaas.com
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: nginx/1.13.10
< Date: Fri, 22 Jun 2018 16:24:54 GMT
< Content-Type: application/json; charset=utf-8
< Content-Length: 8757
< Connection: keep-alive
< Vary: Accept-Encoding
< ETag: b12ae22f0752b3769d8aec79e25210646879d4f5f9d2797b4d9e4caf3fe92f45
< Cache-Control: public, max-age=31536000
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Headers: authorization, range
< Access-Control-Allow-Methods: PUT, DELETE, PATCH
< Access-Control-Expose-Headers: content-disposition
< X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Alternate-Protocol: 443:npn-spdy/3
< X-UA-Compatible: IE=Edge
< Strict-Transport-Security: max-age=31536000; includeSubDomains;
< 
{"channels":["stable","unstable"],"checksum":"b12ae22f0752b3769d8aec79e25210646879d4f5f9d2797b4d9e4caf3fe92f45","config":"","deps":[{"name":"gcc-libs","origin":"core","release":"20180608091701","version":"7.3.0"},{"name":"glibc","origin":"core","release":"20180608041157","version":"2.27"},{"name":"libxext","origin":"core","release":"20180608172617","version":"1.3.3"},{"name":"libxi","origin":"core","release":"20180608174730","version":"1.7.9"},{"name":"libxrender","origin":"core","release":"20180608174757","version":"0.9.10"},{"name":"libxtst","origin":"core","release":"20180608174807","version":"1.2.3"},{"name":"xlib","origin":"core","release":"20180608172326","version":"1.6.5"}],"exposes":[],"ident":{"name":"jre8","origin":"core","release":"20180608184654","version":"8.172.0"},"is_a_service":false,"manifest":"# core / jre8\nOracle Java Runtime Environment. This package is made available to you to allow you to run your applications as provided in and subject to the terms of the Oracle Binary Code License Agreement for the Java SE Platform Products and JavaFX, found at http://www.oracle.com/technetwork/java/javase/terms/license/index.html\n\n* __Maintainer__: The Habitat Maintainers <humans@habitat.sh>\n* __Version__: 8.172.0\n* __Release__: 20180608184654\n* __Architecture__: x86_64\n* __System__: linux\n* __Target__: x86_64-linux\n* __Upstream URL__: [http://www.oracle.com/technetwork/java/javase/overview/index.html](http://www.oracle.com/technetwork/java/javase/overview/index.html)\n* __License__: Oracle Binary Code License Agreement for the Java SE Platform Products and JavaFX \n* __Source__: [http://download.oracle.com/otn-pub/java/jdk/8u172-b11/a58eab1ec242421181065cdc37240b08/jre-8u172-linux-x64.tar.gz](http://download.oracle.com/otn-pub/java/jdk/8u172-b11/a58eab1ec242421181065cdc37240b08/jre-8u172-linux-x64.tar.gz)\n* __SHA__: `f08f25aec2bdc86138ccba8fd5b904451e3afa1d24a88c85f28c2d84bfd45bad`\n* __Path__: `/hab/pkgs/core/jre8/8.172.0/20180608184654`\n* __Build Dependencies__: `core/patchelf `\n* __Dependencies__: `core/glibc core/gcc-libs core/xlib core/libxi core/libxext core/libxrender core/libxtst `\n* __Interpreters__: no interpreters or undefined\n\n# Plan\n\n## Build Flags\n\n```bash\nCFLAGS: -I/hab/pkgs/core/glibc/2.27/20180608041157/include -I/hab/pkgs/core/xlib/1.6.5/20180608172326/include -I/hab/pkgs/core/libxi/1.7.9/20180608174730/include -I/hab/pkgs/core/libxext/1.3.3/20180608172617/include -I/hab/pkgs/core/libxrender/0.9.10/20180608174757/include -I/hab/pkgs/core/libxtst/1.2.3/20180608174807/include\nCPPFLAGS: -I/hab/pkgs/core/glibc/2.27/20180608041157/include -I/hab/pkgs/core/xlib/1.6.5/20180608172326/include -I/hab/pkgs/core/libxi/1.7.9/20180608174730/include -I/hab/pkgs/core/libxext/1.3.3/20180608172617/include -I/hab/pkgs/core/libxrender/0.9.10/20180608174757/include -I/hab/pkgs/core/libxtst/1.2.3/20180608174807/include\nCXXFLAGS: -I/hab/pkgs/core/glibc/2.27/20180608041157/include -I/hab/pkgs/core/xlib/1.6.5/20180608172326/include -I/hab/pkgs/core/libxi/1.7.9/20180608174730/include -I/hab/pkgs/core/libxext/1.3.3/20180608172617/include -I/hab/pkgs/core/libxrender/0.9.10/20180608174757/include -I/hab/pkgs/core/libxtst/1.2.3/20180608174807/include\nLDFLAGS: -L/hab/pkgs/core/glibc/2.27/20180608041157/lib -L/hab/pkgs/core/gcc-libs/7.3.0/20180608091701/lib -L/hab/pkgs/core/xlib/1.6.5/20180608172326/lib -L/hab/pkgs/core/libxi/1.7.9/20180608174730/lib -L/hab/pkgs/core/libxext/1.3.3/20180608172617/lib -L/hab/pkgs/core/libxrender/0.9.10/20180608174757/lib -L/hab/pkgs/core/libxtst/1.2.3/20180608174807/lib\nLD_RUN_PATH: /hab/pkgs/core/jre8/8.172.0/20180608184654/lib:/hab/pkgs/core/glibc/2.27/20180608041157/lib:/hab/pkgs/core/gcc-libs/7.3.0/20180608091701/lib:/hab/pkgs/core/xlib/1.6.5/20180608172326/lib:/hab/pkgs/core/libxi/1.7.9/20180608174730/lib:/hab/pkgs/core/libxext/1.3.3/20180608172617/lib:/hab/pkgs/core/libxrender/0.9.10/20180608174757/lib:/hab/pkgs/core/libxtst/1.2.3/20180608174807/lib:/hab/pkgs/core/jre8/8.172.0/20180608184654/lib/amd64/jli:/hab/pkgs/core/jre8/8.172.0/20180608184654/lib/amd64/server:/hab/pkgs/core/jre8/8.172.0/20180608184654/lib/amd64\n```\n\n## Plan Source\n\n```bash\npkg_origin=core\npkg_maintainer=\"The Habitat Maintainers <humans@habitat.sh>\"\npkg_name=jre8\npkg_version=8.172.0\npkg_upstream_version=8u172\npkg_source=\"http://download.oracle.com/otn-pub/java/jdk/${pkg_upstream_version}-b11/a58eab1ec242421181065cdc37240b08/jre-${pkg_upstream_version}-linux-x64.tar.gz\"\npkg_shasum=\"f08f25aec2bdc86138ccba8fd5b904451e3afa1d24a88c85f28c2d84bfd45bad\"\npkg_filename=\"jre-${pkg_upstream_version}-linux-x64.tar.gz\"\npkg_license=('Oracle Binary Code License Agreement for the Java SE Platform Products and JavaFX')\npkg_description=('Oracle Java Runtime Environment. This package is made available to you to allow you to run your applications as provided in and subject to the terms of the Oracle Binary Code License Agreement for the Java SE Platform Products and JavaFX, found at http://www.oracle.com/technetwork/java/javase/terms/license/index.html')\npkg_upstream_url=\"http://www.oracle.com/technetwork/java/javase/overview/index.html\"\npkg_deps=(core/glibc core/gcc-libs core/xlib core/libxi core/libxext core/libxrender core/libxtst)\npkg_build_deps=(core/patchelf)\npkg_bin_dirs=(bin)\npkg_lib_dirs=(lib)\npkg_include_dirs=(include)\n\nsource_dir=$HAB_CACHE_SRC_PATH/${pkg_name}-${pkg_upstream_version}\n\ndo_setup_environment() {\n set_runtime_env JAVA_HOME \"$pkg_prefix\"\n}\n\n## Refer to habitat/components/plan-build/bin/hab-plan-build.sh for help\n\n# Customomized download_file() to work around the Oracle EULA Cookie-wall\n#  See: http://stackoverflow.com/questions/10268583/downloading-java-jdk-on-linux-via-wget-is-shown-license-page-instead\ndownload_file() {\n  local url=\"$1\"\n  local dst=\"$2\"\n  local sha=\"$3\"\n\n  build_line \"By including the JRE, you accept the terms of the Oracle Binary Code License Agreement for the Java SE Platform Products and JavaFX, which can be found at http://www.oracle.com/technetwork/java/javase/terms/license/index.html\"\n\n  pushd \"$HAB_CACHE_SRC_PATH\" > /dev/null\n  if [[ -f $dst && -n \"$sha\" ]]; then\n    build_line \"Found previous file '$dst', attempting to re-use\"\n    if verify_file \"$dst\" \"$sha\"; then\n      build_line \"Using cached and verified '$dst'\"\n      return 0\n    else\n      build_line \"Clearing previous '$dst' file and re-attempting download\"\n      rm -fv \"$dst\"\n    fi\n  fi\n\n  build_line \"Downloading '$url' to '$dst'\"\n  $_wget_cmd --no-check-certificate --no-cookies --header \"Cookie: oraclelicense=accept-securebackup-cookie\" \"$url\" -O \"$dst\"\n  build_line \"Downloaded '$dst'\";\n  popd > /dev/null\n}\n\ndo_unpack() {\n  local unpack_file=\"$HAB_CACHE_SRC_PATH/$pkg_filename\"\n  mkdir \"$source_dir\"\n  pushd \"$source_dir\" >/dev/null\n  tar xz --strip-components=1 -f \"$unpack_file\"\n\n  popd > /dev/null\n  return 0\n}\n\ndo_build() {\n  return 0\n}\n\ndo_install() {\n  cd \"$source_dir\" || exit\n  cp -r ./* \"$pkg_prefix\"\n\n  build_line \"Setting interpreter for '${pkg_prefix}/bin/java' '$(pkg_path_for glibc)/lib/ld-linux-x86-64.so.2'\"\n  build_line \"Setting rpath for '${pkg_prefix}/bin/java' to '$LD_RUN_PATH'\"\n\n  export LD_RUN_PATH=$LD_RUN_PATH:$pkg_prefix/lib/amd64/jli:$pkg_prefix/lib/amd64/server:$pkg_prefix/lib/amd64\n\n  find \"$pkg_prefix\"/bin -type f -executable \\\n    -exec sh -c 'file -i \"$1\" | grep -q \"x-executable; charset=binary\"' _ {} \\; \\\n    -exec patchelf --interpreter \"$(pkg_path_for glibc)/lib/ld-linux-x86-64.so.2\" --set-rpath \"${LD_RUN_PATH}\" {} \\;\n\n  find \"$pkg_prefix/lib/amd64\" -type f -name \"*.so\" \\\n    -exec patchelf --set-rpath \"${LD_RUN_PATH}\" {} \\;\n}\n\ndo_strip() {\n  return 0\n}\n```","target":"x86_64-linux","tdeps":[{"name":"gcc-libs","origin":"core","release":"20180608091701","version":"7.3.0"},{"name":"glibc","origin":"core","release":"20180608041157","version":"2.27"},{"name":"libxau","origin":"core","release":"20180608172221","version":"1.0.8"},{"name":"libxcb","origin":"core","release":"20180608172233","version":"1.12"},{"name":"libxdmcp","origi* Connection #0 to host builder.ncrsaas.com left intact
n":"core","release":"20180608172228","version":"1.1.2"},{"name":"libxext","origin":"core","release":"20180608172617","version":"1.3.3"},{"name":"libxi","origin":"core","release":"20180608174730","version":"1.7.9"},{"name":"libxrender","origin":"core","release":"20180608174757","version":"0.9.10"},{"name":"libxtst","origin":"core","release":"20180608174807","version":"1.2.3"},{"name":"linux-headers","origin":"core","release":"20180608041107","version":"4.15.9"},{"name":"xlib","origin":"core","release":"20180608172326","version":"1.6.5"}],"visibility":"public"}

Any other ideas @raskchanky? I could blow up the core origin and try a re-bootstrap but that seems a little heavy handed

Wow, I’m a little blown away that you can curl the details of that package, and yet downloading it 404s.

Can you try one more thing, just to be extra sure?

curl -v -o core-jre8-8.172.0-20180608184654-x86_64-linux.hart https://builder.ncrsaas.com/v1/depot/pkgs/core/jre8/8.172.0/20180608184654/download

That link is obviously linux specific, so if this is a windows only package, it will need to be tweaked slightly. I’m assuming since the first curl you ran was successful, that download should also work.

Beyond that, I would also try running the hab pkg install with RUST_LOG=debug set, which should provide a lot of verbose output about what’s happening behind the scenes and what’s failing. Off the cuff, I wonder if the target/architecture is getting crossed somehow, where it’s trying to find a package for an architecture that isn’t there.

404!

curl -v -o core-jre8-8.172.0-20180608184654-x86_64-linux.hart https://builder.ncrsaas.com/v1/depot/pkgs/core/jre8/8.172.0/20180608184654/download
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 23.96.177.164...
* TCP_NODELAY set
* Connected to builder.ncrsaas.com (23.96.177.164) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
  CAfile: /opt/chefdk/embedded/ssl/certs/cacert.pem
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [85 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2776 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-SHA
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=US; ST=Georgia; L=Atlanta; O=NCR Corporation; OU=SaaS Operations; CN=*.ncrsaas.com
*  start date: Jan 29 00:00:00 2018 GMT
*  expire date: Jan 29 12:00:00 2019 GMT
*  subjectAltName: host "builder.ncrsaas.com" matched cert's "*.ncrsaas.com"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert Global CA G2
*  SSL certificate verify ok.
} [5 bytes data]
> GET /v1/depot/pkgs/core/jre8/8.172.0/20180608184654/download HTTP/1.1
> Host: builder.ncrsaas.com
> User-Agent: curl/7.59.0
> Accept: */*
> 
{ [5 bytes data]
< HTTP/1.1 404 Not Found
< Server: nginx/1.13.10
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Headers: authorization, range
< Access-Control-Allow-Methods: PUT, DELETE, PATCH
< Access-Control-Expose-Headers: content-disposition
< Date: Fri, 22 Jun 2018 19:54:06 GMT
< Content-Length: 0
< 
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Connection #0 to host builder.ncrsaas.com left intact

Huh, ok. So I’m guessing it’s a windows package? Which means this should succeed:

curl -v -o core-jre8-8.172.0-20180608184654-x86_64-windows.hart https://builder.ncrsaas.com/v1/depot/pkgs/core/jre8/8.172.0/20180608184654/download?target=x86_64-windows

If that ^ works, then somehow hab is getting confused about the target/platform. Any chance you’re running hab pkg install on a linux host for a windows package, or vice-versa?

it’s a linux package. there’s a windows variant too. I suspect it will fail similarly

yep :frowning:

curl -v -o core-jre8-8.172.0-20180608184654-x86_64-windows.hart https://builder.ncrsaas.com/v1/depot/pkgs/core/jre8/8.172.0/20180608184654/download?target=x86_64-windows
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 23.96.177.164...
* TCP_NODELAY set
* Connected to builder.ncrsaas.com (23.96.177.164) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
  CAfile: /opt/chefdk/embedded/ssl/certs/cacert.pem
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [85 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2776 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* TLSv1.2 (IN), TLS change cipher, Client hello (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-SHA
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=US; ST=Georgia; L=Atlanta; O=NCR Corporation; OU=SaaS Operations; CN=*.ncrsaas.com
*  start date: Jan 29 00:00:00 2018 GMT
*  expire date: Jan 29 12:00:00 2019 GMT
*  subjectAltName: host "builder.ncrsaas.com" matched cert's "*.ncrsaas.com"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert Global CA G2
*  SSL certificate verify ok.
} [5 bytes data]
> GET /v1/depot/pkgs/core/jre8/8.172.0/20180608184654/download?target=x86_64-windows HTTP/1.1
> Host: builder.ncrsaas.com
> User-Agent: curl/7.59.0
> Accept: */*
> 
{ [5 bytes data]
< HTTP/1.1 404 Not Found
< Server: nginx/1.13.10
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Headers: authorization, range
< Access-Control-Allow-Methods: PUT, DELETE, PATCH
< Access-Control-Expose-Headers: content-disposition
< Date: Fri, 22 Jun 2018 20:06:39 GMT
< Content-Length: 0
< 
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Connection #0 to host builder.ncrsaas.com left intact

OH!

One thing that occurs to me here is that we just completed a feature where now packages are kept in S3/Minio instead of on the filesystem of the builder-api node. I believe the default in the configuration for the on-prem depot is to use Minio.

Is it possible you’ve consumed the newest code/configuration but haven’t migrated packages from the filesystem to Minio?

I set up minio last night as a clean install and then imported the core origin using the bootstrap script. we have 2 custom origins and packages uploaded with hab pkg upload are able to download from the depot without any of these issues. Maybe it’s something with the minio setup + the core origin bootstrapper?

The fact that new packages uploaded via hab pkg upload work fine and existing ones don’t definitely points to minio not having your other packages. I think there was a tool written to migrate packages. Let me dig it up.

Have you tried this?

I'll try it tonight. The core origin was a fresh import after setting up a fresh builder install, so if this works then we need an issue open to make the initial core import play nice with the new minio function. I'll report back here.
15297139517121853639772

This did not help @raskchanky. I’m going to try to blow away the core origin and re-bootstrap

Actually, I might have lied. It didn’t help right after I ran the minio migration but it appears to be in a happy state now. Running tests to verify

Yeah, that did it @raskchanky. I suspect the initial core origin bootstrap is not minio compatible or I stepped into the twilight zone halfway through the deployment. We’re all good!

I’m glad you got it sorted out!