We run most of our nodes in a context where public web access is not allowed. We’re making heavy use of JFrog’s Artifactory’s “Remote Repo” for providing a smart cache/mirror for well known repository types (ruby gems, npm, rpm, nuget/chocolately, etc.). I haven’t found a way to accomplish something similar for the chef packages we pull from omnitruck API during the bootstrap process and during the chef-client runs of the cookbooks that manage our Chef Server and Chef Automate servers. We started down the path of creating a sync script that pulls from Omnitruck and pushes to a custom Artifactory repo, but this is starting to explode in complexity as we have to support more platforms and manage versions. I fear we’re on a path of creating our own Omnitruck which I want to avoid. So i’m going back and reviewing my initial analysis and thinking.
As best I can tell, Onitruck API is a simple 2-endpoint API that offers metadata lookup + download url discovery and the download itself. I wasn’t able to find a way front this with Artifactory natively. It wants a package manager repo/protocol, which this is not.
This must be a common problem - what are other people doing to solve this?
That could potentially solve part of the problem, but I don’t think it could solve it completely. We really have two course grained cases: 1) chef-client and ChefDK packages on standard nodes; 2) Chef Server and Chef Automate packages on the nodes running this Chef infrastructure.
The first gap that comes to mind is Windows chef-client packages in the #1 case. Our approach to the #2 case has been to use chef_ingredient cookbook. Perhaps there is a way to force this to work off of a package repo, but presently at least, this wants to curl things down from Omnitruck API and install from a local RPM.
@jakauppila we ended up writing a manual sync script. We wired it up to our CI pipeline that builds and publishes cookbooks. We have a control file that declares the versions of the packages we care about, the script reads that file, hits OmniTruck API and uploads it into a generic local repo we have in our Artifactory server. We mirror the directory structure of OmniTruck and therefore it works in all of our cases across Windows and Linux. It’s crude and not really the way I want to do it, but it works. There is a chocolatey package out on chocolatey.org for the chef-client and other things, and you could use a product like Artifactory to mirror it… but like most choco packages, it just does a call back to the public web, so you’re kinda screwed. We ended up creating our own dedicated choco packages for this reason. Doing that, might be your easiest path.