CI-friendly chefdk?

Chef Infra (archive)
1

I know that ChefDK meant for dev machines, I can find quite a few statements here in this mail lists that confirms that, but all of it pretty old (back from 2014).

ChefDK has evolved over the years, so as we, and probably it is time to have a fresh look on the issue.

ChefDK provides a sustainable baseline. In my humble opinion - people should be using ChefDK on CI machines too. It gives an upgrade path, it focusing community on using stable versions of gems and eliminates the struggle with resolving all that ‘dependency hell’ we all had to deal with before ChefDK. Having to cook your CI server for Chef on your own leaves so many room for mistakes, especially that there are no (as far as I know) ‘official’ HowTo doc for this. Something tells me these days pretty much everyone (except probably some legacy setups) are using ChefDK on CI servers, at least it sounds like a common sense to me.

However, as of now ChefDK is pretty much locked down in terms of it’s ingredients. I can see why though, having people (especially newcomers) able to mess up with their /opt/chefdk locally was causing many false-negative feedback and was leading to spending time investigating not real issues. However, for managed CI-environment it is not the case. Either it is a chef-controlled machine or a container, there is no way to mess up with chefdk folder to a degree to not be able to recover, as these would not be any manual actions. In the same time, having the ability to change versions of some gems in the chefdk is crucial at a times to unblock the progress. To get a fix into ChefDK as of now can easily take months, because it involves having the fix done, having a new gem released and only then having that gem shipped into ChefDK. Agile teams these days cannot afford waiting for months, they want to move fast (and break things hehe). There is more background on my specific case here https://github.com/chef/chef-dk/issues/1559 - I started it as an issue and then being told mailing list is the right place to have a conversation like that.

Instead of cooking my own CI env with bundler, it sounds much more safe to have ChefDK installed as a baseline and then overriding some versions of some gems. It is safe in a managed environments and also provides a way to manage technical debt - I can leave a comment in my recipe or a Dockerfile with the link to the issue we’re waiting a fix for and later on be safely able to remove that ‘patch’ override and turn back to clean ChefDK installation. In opposite, having your own bundler-based setup leaves you on your own to find out a way to keep your CI environment as close to ChefDK and community effort as possible (IF you are smart enough to understand that you’d better be close).

To sum up - we have a problem statement: no ability to override gems in a managed CI env.

I don’t know what the actual solution would be, but just to start some discussion here are a few options from the top of my head:

  1. Having ChefDK version pins somewhat relaxed. Maybe getting rid of appbundler leaving with just a Gemfile.lock, or some other option that does not lock you down with /opt/chefdk/bin overrides.
  2. Having a new tool or improvements to appbundle-updater so that I can feed a single source of truth to it (like a Gemfile) and having it to rebuild /opt/chefdk for me. Having a docs for that if this is something already possible with a reasonable effort…
  3. Having a separate ChefDK bundle for CI servers, somewhat aligned with or generated from ChefDK, maybe in a form of Dockerfile (but not just a Dockerfile that installs the package since it will not solve a problem statement above).

/discuss

Best regards,
Dee

2

There's a lot to discuss in your post but I just want to briefly say this is not a good option as appbundler provides a huge performance improvement in the CLI experience. Rubygems does a lot of runtime dependency solving which can result in situations where knife help takes 10s or more to run. Part of the point of appbundler is to feed a working dependency solution into rubygems ahead of time to avoid this issue. The way bundler works, it doesn't provide the same benefit.

3

Well maybe a convenient one-liner to rebuild the whole thing with feeding it a different set of dependencies, preferably not the whole thing (like Gemfile) but only these I want to override. I don’t think any of solutions mentioned above are perfect, but that is the best I was able to come up with in a short notice. I don’t know much about chefdk internals.

4

There's no halfway. Either you're rebuilding the whole ChefDK Gemfile to generate the locks or you're not. If we had magic to make upgrading gems inside of ChefDK easier we'd use that to make maintaining ChefDK itself easier.

To make it easier to user the bundler that comes with ChefDK I fixed this bug:

That means if you bundle install; bundle exec <something> and hit the appbundled binstubs that they detect they're running under bundler and don't apply the stubs and just use your bundle instead.

5

Hi

I needed some input , I am trying to setup chefdk on my my VMs.

While checking knife ssl check , I am getting the erros.

As far as I understood it is considering the different ceritcicate for IP and hostname which bellong to chefserver.

[root@chefdk .chef]#

[root@chefdk .chef]# knife ssl fetch

WARNING: Certificates from 9.126.144.73 **(IP of the chef server )**will be fetched and placed in your trusted_cert

directory (/root/chef-repo/.chef/trusted_certs).

Knife has no means to verify these are the correct certificates. You should

verify the authenticity of these certificates after downloading.

Adding certificate for chefserver in /root/chef-repo/.chef/trusted_certs/chefserver.crt

[root@chefdk .chef]# knife ssl check

Connecting to host 9.126.144.73:443

ERROR: The SSL cert is signed by a trusted authority but is not valid for the given hostname

ERROR: You are attempting to connect to: ‘9.126.144.73’

ERROR: The server’s certificate belongs to ‘chefserver’ <<< This is confusing why it is treating the certificate differently as this IP belongs to chefserver only the hostname -f works fine for both chefserver and chefdk)

TO FIX THIS ERROR:

The solution for this issue depends on your networking configuration. If you

are able to connect to this server using the hostname chefserver

instead of 9.126.144.73, then you can resolve this issue by updating chef_server_url

in your configuration file.

If you are not able to connect to the server using the hostname chefserver

you will have to update the certificate on the server to use the correct hostname.

[root@chefdk .chef]# cat /etc/hosts

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4

::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

9.126.144.73 chefserver

9.126.144.76 chefdk

Thanks

Prerna

6

If we had magic to make upgrading gems inside of ChefDK easier we’d use that to make maintaining ChefDK itself easier.

Well that's why I'm opening this discussion. Let's create some magic.