Chef-Provisioning Future Plans

Chef Infra (archive)
1

Ohai!

I’m currently investigating possibility to use chef-provisioning + chef to bootstrap complicated environments. There are many competitors like ansible, terraform, etc. So in 2018 I’m asking myself, am I right that I’m trying to use such tools, like chef-provisioning? Currently they are looking young and not maintained well, aren’t they?

There is still not clear about feature plans, but I want to believe that chef-provisioning will become a standard for chef stack.

I like Chef used pure language like ruby for automation instead of a DSL, which limits us.

Now, when I started my investigations on chef provisioning I see number of issues. Annoying one: most common use-cases aren’t documented well, so I have to ready sources, blogs and watch videos. We need a tutorial here: how to start from scratch:

  1. How to create repo
  2. Which settings to specify in .chef/config.rb. Put all stuff under version control or ignore something
  3. When/How to run chef-client -z - local mode deprecated. What will be next?
  4. Run against existing chef-server

Another and most important for me - lack of flexibility of bootstrapping process with chef-provisioning.

  1. convergence_strategy is hardcoded by a driver and can not be manipulated and overwritten
  2. converge is mixed and responsible for several tasks, like choose transport (winrm, ssh, whatever), install chef-client via mixlib, prepare chef objects and actual converge. All hardcoded. Need different actions?
  3. Chef-client installation is fully depends on mixlib-install, but mixlib-install doesn’t allow customization from my side. I must provide only url to custom install script.

To be clear, here is my use-case: I’m trying to bootstrap environment offline using the following plan:

  1. Create temporary bootstrap-repo and cookbook for provisioning
  2. Create number of artifacts locally (like chef-client, chef-server, etc). Here is mixlib-install helps.
  3. Go offline
  4. Use bootstrap-repo to provision chef-server with local chef-zero: here are the dragons:
    a. Chef always tries to install chef-client and I can’t control this
    b. Specifying custom install_sh_url is not possible because I haven’t infrastructure yet to specify any url. chef-install.sh scripts forces me to do that and checks that I have http/https in the url.
    c. There is a trick: if /opt/chef folder exists chef-install.sh tries to validate installed version and skips further installation. But it is hack and I can’t relay on it.
    d. I must maintain locally set of artifacts for Debian/Ubuntu/CentOs in case hybrid infrastructure
    e. I though to use habitat exported chef.tar.gz to bootstrap any sever which works nice, but chef-install.sh knows nothing about that. What are you plans on habitat?
    f. machine_file is not idempotent, not sure how to properly deal with this. not_if/only_if section? But I have to execute commands remotely? use machine_file.machine.execute()?
    g. machine_exec - same issue.
    h. I see, that chef-provisioning uses native ruby net/ssh. This cool but huge amount of work already done in https://github.com/capistrano/sshkit. I successfully used this library to produce a “farbirc”/“ansible” like tool based on ruby: remote execs, tests, upload, downloads, batches, etc. It also has different backends, like ssh - execute remotely, local - execute locally, printer - just prints command locally, useful for dryrun, you can implement custom backends, like docker, winrm.

I really think that Chef is the best one tool for configuration and I want to help, contribute, and, of course, solve my particular case. But it is not clear for me what to do?

2

Regarding Chef Provisioning this blog article might give you a little more insight about the status of this project: https://coderanger.net/provisioning/ . It is quite old but I think not too much has changed since then.

3

I saw these notes. Yep, it is quite old, that’s why I’m asking community.
Need to understand is there any point to invest development on this tools or look for something else?

4

Personally I have never used chef provisioning due to the mentioned blog post. About one and a half years ago I decided to give Terraform a try and it was a great decision for me. It interacts perfectly well with Chef and Management loves that we could switch from AWS to almost any other provider rather easily just because everything is already done and thus documented in terraform.
I am also working without Internet access (chef calls this airgapped if you want to look at the docs) and in the End everything changed for us once we started using Artifactory. Right now I am really looking forward to what happens with Chef Workstation and chef-run since this tool really fills a gap that otherwise only big environments and tools like ansible could fill.

5

Hi Yauhen :slight_smile:

From my personal experience, currently chef-provisioning is 90% useful on private clouds and bare metal servers, where you should not think about SG`s, A[E]LB rules, VPC`s and so on.

At the moment you facing production AWS project, especially when info-sec team appears, chef-provisioning is almost useless and it’s became too complicated and expensive to develop and support its code, Terraform is much better from any perspective. Infrastructure (routers, LB’s, networks etc.) really became part of code, and CP not ready for it.

Joerg is right, killer feature of CP - you can see chef-run log and don’t need to check cloud-init log of every instance in ASG.

About transport lib, you can check train gem, used at least in Chef 14.

Had similar issues with install_sh_url, only solution - hack lib for your needs.

As you need all-in-one solution, mb it will be good idea for you to build your own chef-client with pre-installed gems, artifacts and so on with Omnibus