Sourcing ingredients from your local store for use within your recipes

hippiehacker · March 7, 2013, 6:31am

I wrote some knife plugins that download the latest version of
chef_client, chef_server, ubuntu, windows, and vagrant, into the
:file_cache_path and then populate data_bags with semantic versions,
architecture, checksums, os ver supported etc:

I then do searches in my recipes to grab the latest, but have the
desired version able to be set as an attribute.
.

I thought about putting some of the resultant cached files into
files/default for various cookbooks before uploading to chef server so
end nodes need not have internet, only connections to the chef_server.
Or for chef-solo repos/cookbooks that would have batteries included as
this works with chef-solo if you add chef-search-solo to your runlist:

github.com

hh/cloud-kitchen/blob/master/.chef/create-usb-solo.rb#L17


file_cache_path "#{current_dir}/cache"
file_backup_path "#{ENV['HOME']}/.chef/backup"
cache_options( :path => "#{ENV['HOME']}/.chef/checksums")
verbose_logging false


solo_json_file = "#{current_dir}/create-usb-solo.json"
open(solo_json_file,'w+') do |f|
f.write(
  {
    "run_list" => [
      "recipe[chef-solo-search]",
      "recipe[ii-chef-server::cache-files]",
      "recipe[ii-fileserver::ingredients]",
      "recipe[ii-usb::create-usb-solo]"
      ],
    'ii-usb' => {
      'src-chef-repo' => "#{current_dir}/..", # for now we'll just copy ourselves... I need to figure out caching
      'target-device' => ENV['TARGETUSB'], # We do this to force setting it at runtime
      'partition-size'=> '-1' # set this to the number of megabytes you want the partition to be, -1 means as big as the usb
    },
    'private_chef' => {

These sourced ingredients / dynamic data bags backed by third party
cached files seems interesting to me and I wanted wider feedback /
audience.

Would a set of ingredients repos be useful for the community at large
beyond myself?

https://github.com/easybake-ingredients

Anyone see any interesting use cases for this?

could cookbooks depend on versioned data bags created this way?
the ingredient repos could be code reviewed separately from your cookbooks!
the populated cache could be easily copied to usb or a file share etc

hippiehacker · March 7, 2013, 4:46pm

For some ingredients, we might be able to use something modeled after
http://wiki.debian.org/debian/watch/ to auto check for new releases.

On Wed, Mar 6, 2013 at 10:31 PM, Chris McClimans chef@hippiehacker.org wrote:

I wrote some knife plugins that download the latest version of
chef_client, chef_server, ubuntu, windows, and vagrant, into the
:file_cache_path and then populate data_bags with semantic versions,
architecture, checksums, os ver supported etc:

cloud-kitchen/.chef/plugins/knife at master · hh/cloud-kitchen · GitHub

cloud-kitchen/data_bags at master · hh/cloud-kitchen · GitHub

I then do searches in my recipes to grab the latest, but have the
desired version able to be set as an attribute.
.

ii-fileserver/recipes/cache-files.rb at master · ii-cookbooks/ii-fileserver · GitHub

ii-chef-server/recipes/lxc-osc-download.rb at master · ii-cookbooks/ii-chef-server · GitHub

I thought about putting some of the resultant cached files into
files/default for various cookbooks before uploading to chef server so
end nodes need not have internet, only connections to the chef_server.
Or for chef-solo repos/cookbooks that would have batteries included as
this works with chef-solo if you add chef-search-solo to your runlist:

cloud-kitchen/.chef/create-usb-solo.rb at master · hh/cloud-kitchen · GitHub

These sourced ingredients / dynamic data bags backed by third party
cached files seems interesting to me and I wanted wider feedback /
audience.

Would a set of ingredients repos be useful for the community at large
beyond myself?

easybake-ingredients · GitHub

Anyone see any interesting use cases for this?

could cookbooks depend on versioned data bags created this way?

the ingredient repos could be code reviewed separately from your cookbooks!

the populated cache could be easily copied to usb or a file share etc

hippiehacker · March 7, 2013, 5:17pm

This pattern of enumerating through and downloading versions of
software has been solved before in debian.

Which results in automated monitoring of new releases of upstream
software, which is used to notify and eventually build debian packages
around them.

If we could do something similar with easybake ingredients and put it
into a continuous delivery pipeline, we could have tested/vetted
combinations of ingredients.

ingredients.easybake.cd

Looks like uscan is written in perl:

http://anonscm.debian.org/gitweb/?p=devscripts/devscripts.git;a=blob;f=scripts/uscan.pl;h=8723fb45dfd25df36ebc2aa06fd045df8ee75416;hb=HEAD

My next target is Virtualbox, which has a debian/watch file in the
debian sources and I’m digging the automated approach they took to
checking for new upstream releases.

hippiehacker · March 10, 2013, 6:27am

I haven't rewritten uscan in ruby, and they are all still separate
knife plugins, but I updated the easybake ingredients with some
common external dependencies for chef/devops work. Next is to put
together a devops-workstation cookbook that supports OSX, Windows, and
Ubuntu hosts with 'Vagrant+Windows+Ubuntu' and ability to build
baseboxes based on pulling these files from the file_cache_path. If we
run chef-solo from a USB stick, it could feasibly not require a
network connection. If we booted the USB stick as an automated OS
installer (that included a chef-solo run at the end), it would deploy
an devops workstation on hardware with all of the below:

Any blobs people use that I might be missing here?

Ubuntu desktop ISO 12.04.2 for ubuntu desktop 12.04.2
Ubuntu server ISO 12.04.2 for ubuntu server 12.04.2
Windows GRMSXEVAL_EN_DVD ISO 7601 for windows Windows GRMSXEVAL_EN_DVD
ISO 2008r2
VirtualBox Extension Pack 4.2.8
VirtualBox Guest Additions ISO 4.2.8
VirtualBox 4.2.8 for el 6
VirtualBox 4.2.8 for mac_os_x 10.7, 10.8
VirtualBox 4.2.8 for windows 2008r2, 2012, 7, 8
VirtualBox 4.2.8 for ubuntu 12.04
VirtualBox 4.2.8 for ubuntu 12.10
Chef Omnibus Client 11.4.0-18-gdf096fa-1 for debian 6
Chef Omnibus Client 11.4.0-18-gdf096fa-1 for el 5; for sles 11.2
Chef Omnibus Client 11.4.0-18-gdf096fa-1 for el 6; for suse 12.1
Chef Omnibus Client 11.4.0-18-gdf096fa-1 for mac_os_x 10.6
Chef Omnibus Client 11.4.0-18-gdf096fa-1 for mac_os_x 10.7
Chef Omnibus Client 11.4.0-18-gdf096fa-1 for ubuntu 10.04, 10.10
Chef Omnibus Client 11.4.0-18-gdf096fa-1 for ubuntu 11.04, 11.10, 12.04, 12.10
Chef Omnibus Client 11.4.0-18-gdf096fa-1 for windows 2008, 2003r2, 2008r2, 2012
Chef Server 11.0.6 for el 5
Chef Server 11.0.6 for el 6
Chef Server 11.0.6 for ubuntu 10.04, 10.10
Chef Server 11.0.6 for ubuntu 11.04, 11.10
Chef Server 11.0.6 for ubuntu 12.10, 12.04
Vagrant v1.0.6 for mac_os_x 10.7, 10.8
Vagrant v1.0.6 for windows 2008r2, 2012, 7, 8
Vagrant v1.0.6 for i686 ubuntu 10.04, 10.10, 11.04, 11.10, 12.04,
12.10; for i686 debian 6
Vagrant v1.0.6 for i686 el 5, 6; for i686 sles 11.2, 12.2
Vagrant v1.0.6 for ubuntu 10.04, 10.10, 11.04, 11.10, 12.04, 12.10; for debian 6
Vagrant v1.0.6 for el 5, 6; for sles 11.2, 12.2
Emacs 24.2 for i386 windows 2008r2, 2012, 7, 8
Emacs 24.2 for osx 10.6, 10.7, 10.8
GVim 7.3.46 for i386 windows 2008r2, 2012, 7, 8
Sublime Text 2.0.1 for osx 10.6, 10.7, 10.8
Sublime Text 2.0.1 for windows 2008r2, 2012, 7, 8
Sublime Text 2.0.1 for debian 6; for el 5, 6; for sles 11.2, 12.1; for
ubuntu 10.04, 10.10, 11.04, 11.10, 12.04, 12.10
Sublime Text 2.0.1 for i386 windows 2008r2, 2012, 7, 8
Sublime Text 2.0.1 for i386 debian 6; for i386 el 5, 6; for i386 sles
11.2, 12.1; for i386 ubuntu 10.04, 10.10, 11.04, 11.10, 12.04, 12.10
Git 1.8.1.2 for windows 2008r2, 2012, 7, 8
Git 1.8.1.2 for osx 10.6, 10.7, 10.8

On Thu, Mar 7, 2013 at 9:17 AM, Chris McClimans chef@hippiehacker.org wrote:

This pattern of enumerating through and downloading versions of
software has been solved before in debian.

http://wiki.debian.org/debian/watch/

uscan(1) — devscripts — Debian bookworm — Debian Manpages

http://anonscm.debian.org/gitweb/?p=pkg-virtualbox/virtualbox-ose.git;a=blob_plain;f=debian/watch;hb=HEAD

Which results in automated monitoring of new releases of upstream
software, which is used to notify and eventually build debian packages
around them.

If we could do something similar with easybake ingredients and put it
into a continuous delivery pipeline, we could have tested/vetted
combinations of ingredients.

ingredients.easybake.cd

Looks like uscan is written in perl:

http://anonscm.debian.org/gitweb/?p=devscripts/devscripts.git;a=blob;f=scripts/uscan.pl;h=8723fb45dfd25df36ebc2aa06fd045df8ee75416;hb=HEAD

My next target is Virtualbox, which has a debian/watch file in the
debian sources and I'm digging the automated approach they took to
checking for new upstream releases.

Topic		Replies	Views
Using Chef offline Chef Infra (archive)	11	1618	May 20, 2015
Chef supermarket and chef-solo Chef Infra (archive)	2	765	March 8, 2018
Chef-server Cookbook Chef Infra (archive)	1	303	February 25, 2015
Chef-solo + cookbooks development Chef Infra (archive)	8	368	October 21, 2010
How to copy cookbook versions, data bags and policyfiles from one Chef organization to another Chef Workstation	0	663	February 13, 2023

Sourcing ingredients from your local store for use within your recipes

Related Topics