Best approach for one time jobs

Chef Infra (archive)
1

Hello,

I’m new here, so pls bear with me if this is known and solved prb.

I’m using hosted Chef to manage mail server nodes. chef-client is configured to run in daemon mode on the nodes.

I want to run many one time jobs like say migration of mailstore format from maildir to mbox. Chef provides a good way to execute the jobs without actually doing login to the server. I want to retain that benefit. But when the cookbook is added to runlist, it will execute each time the chef-client executes.

The standard way for one time jobs is Push Jobs, but it is not supported on hosted chef.

So I’ve thought of following approaches to implement the one time jobs :

  1. Admin adds cookbook to runlist using Chef gui. Waits for execution to finish. Removes the cookbook from runlist. The cookbooks finish execution within 10 mins. chef-client is configured to run every 30 mins.

  2. Admin only adds cookbook. It is removed at some later date. A defensive check is present in the one time cookbooks to check whether it has executed before. I’m little unsure about how can I implement a generic defensive check. I’ve not tested this.

  3. Each cookbook has code to automatically remove itself from runlist. I’ve tested this. It works, but I’m little apprehensive that if a later date someone changes it or removes it, the cookbook behaviour will change.

  4. Jenkins has a chef plugin to execute cookbooks on given server. We use Jenkins too. So we can do this. But data bags cannot be updated from Jenkins.

I was wondering if I was on a totally wrong path. Is there a simpler way to do this ?
Anybody has used any of the above 4 approaches ?

Thanks in advance,
Amol

2

If you use option number 1 or 2, the admin will pull their hair out and quit, if you have a large number of nodes to update.

If you use option number 3, you will need to keep track of which nodes have had the cookbook run against them, so you can be sure all required nodes were touched.

For option 4, I’m not familiar with the Jenkins plugin, but you could do the same thing without a plugin.

I’m assuming you have a management server of some sort setup in such a way that it has ssh access to all your chef nodes, you could write something to search the chef server for the nodes you’re interested in and loop through them via SSH to execute a chef-client run with an override run list.

I’m going to assume you have some specific criteria for which nodes require the update. It could be by node name, nodes with a given role, tag or attribute, or maybe a specific platform. I’m going to also assume that you can trace this change back to some sort of change request process. This affords you the opportunity to name the single-use, single-purpose cookbook after the change request (e.g. CR_42), and the ability to associate nodes with the CR number to filter out already completed nodes. This could be triggered from a single job on the Jenkins server that accepts a single parameter which is the CR number.

Write a new cookbook named CR_42. The cookbook will have two or three recipes, the first will be a search-and-execute recipe run on the management machine, the second would be the default recipe that does the actual change. You may even be able to include the audit mode features to ensure that the change was actually applied successfully.

Then you just need to search for applicable nodes, and apply your one-time cookbook’s changes to each target node, and then mark(or unmark?) the node in some way with the CR number. You could append it to an attribute of CR numbers (e.g. node['company']['change_requests']['applied'] << 'CR_42'), or just tag the node with CR_42.

# recipes/deploy.rb
#
# find nodes 
# * with fqdn matching *.qa.company.biz
# * with the attribute node['reason']['for']['change'] = 'valid'
# * without the tag 'CR_42'
search_query = "fqdn:*.qa.company.biz AND reason_for_change:valid NOT tags:CR_42"
target_nodes = search(:node, search_query)
target_nodes.each do |node|
  command = "chef-client --audit-mode -o #{cookbook_name}"
  
  # code for establishing an ssh connection goes here
  ssh.exec command

  # tag the node with the CR number if successful and save the update
  node.tag "CR_42"
  node.save
end

Then in the default recipe

# recipes/default.rb

# whatever the change request needs
log cookbook_name
execute "Applying #{cookbook_name} to #{node} (#{node['ipaddress']}) do
  command 'foobar'
end

# validate the change
include_recipe "#{cookbook_name}::audit"

I don’t know if this is the best approach, but it is one approach that you could take.

3

The best approach is #1 with a local file used for idempotence:

script 'do a thing' do
  code <<-EOH
mything --whatever
touch /etc/mything
EOH
  creates '/etc/mything'
end

Alternative answer, use a tool designed for procedural stuff or rethink your tasks in terms of convergent behavior and resources.

4

Thanks Brandon and coderanger.

It seems that implementing the stuff may take some effort.
But maintaining it will be a bigger and continuous effort.

So rethinking the options, I was wondering about Chef Push Jobs.
Does Chef Push Jobs have GUI/WebUI ?