A case for 'run once' recipes

Chef Infra (archive)
1

Dear Chef’s

I’ve been giving chef a two-day crash course and have to admit to two
things: 1) it is insanely powerful, and 2) it lacks what I (and many others)
would label as a critical feature: “run once” recipes.

Some background to my claim first. I’m the architect of a wholesale ISP
platform [1] here in South Africa. We have a fully automated service
provisioning platform in place, built on Rails, ruote [2] and daemon-kit
[3]. I was looking at chef specifically to replace parts of the existing
shared hosting provisioning processes and leverage the strength of the
community behind chef instead of fighting my own battle and duplicating
significant efforts.

Joshua Timberman suggested I make my case here and solicit some feedback on
the topic from other chefs.

I would love to help chef gain the ability to support ‘run once’ recipes in
client/server mode. When running a recipe the attributes for that specific
run are defined, overwriting any attributes present in the recipe be
default. Basically what the JSON parameters do with chef-solo.

Spawning new clouds repaidly with chef is only one of its possible use
cases, maintaining them is another. Chef would gain a lot of traction and
support in the greater ISP arena if these ‘run once’ features are
implemented.

What do you think?

Best

–
Kenneth Kalmer
kenneth.kalmer@gmail.com
http://opensourcery.co.za
@kennethkalmer

2

Spawning new clouds repaidly with chef is only one of its possible
use cases, maintaining them is another. Chef would gain a lot of
traction and support in the greater ISP arena if these 'run once'
features are implemented.

Are you looking for something more complicated than adding a not_if
into your current recipe sets? I have a lot of "run once" recipes in
this fashion.

Or, you can just put a big if() around the recipe blocks. The whole
thing is just interpreted ruby.

Caleb

3

On Sep 23, 2009, at 8:10 AM, Caleb Tennis wrote:

Spawning new clouds repaidly with chef is only one of its possible
use cases, maintaining them is another. Chef would gain a lot of
traction and support in the greater ISP arena if these 'run once'
features are implemented.

Are you looking for something more complicated than adding a not_if
into your current recipe sets? I have a lot of "run once" recipes
in this fashion.

Or, you can just put a big if() around the recipe blocks. The whole
thing is just interpreted ruby.

Caleb

I agree with Caleb, the whole point of chef is to make your recipes

idempotent. For the run once recipes that I have needed you can have
the resource touch a file called .alreadydone or something and add a
not_if { File.exists?('.alreadydone')} to the resource to emulate run
once recipes. Perhaps you could wrap up this functionality into the
resources themselves with a little work to have certain resources do
this automatcially for you?

Cheers-

Ezra Zygmuntowicz
ez@engineyard.com

4

On Wed, Sep 23, 2009 at 8:40 PM, Ezra Zygmuntowicz ez@engineyard.comwrote:

On Sep 23, 2009, at 8:10 AM, Caleb Tennis wrote:

Spawning new clouds repaidly with chef is only one of its possible use
cases, maintaining them is another. Chef would gain a lot of traction and
support in the greater ISP arena if these 'run once' features are
implemented.

Are you looking for something more complicated than adding a not_if into
your current recipe sets? I have a lot of "run once" recipes in this
fashion.

Or, you can just put a big if() around the recipe blocks. The whole thing
is just interpreted ruby.

Caleb

   I agree with Caleb, the whole point of chef is to make your recipes

idempotent. For the run once recipes that I have needed you can have the
resource touch a file called .alreadydone or something and add a not_if {
File.exists?('.alreadydone')} to the resource to emulate run once recipes.
Perhaps you could wrap up this functionality into the resources themselves
with a little work to have certain resources do this automatcially for you?

Well, I should have spent more time in the original message, including
actually linking to the footnotes :frowning:

To be more verbose, we'll commission a new server with chef and once
commissioned we'll need to be able to run the same recipes once off with
different attributes each time. For instance, we'll have a recipe that adds
a user with their quotas, another for setting up the virtual host, stats,
etc, and another for setting up their database. For each hosting account
created I want to be able to run the same recipe, loading the individual
recipes, but a custom set of attributes for that run.

I could perfectly well do this with chef-solo and use a JSON file, but then
I still have my agents on the nodes that need to run chef-solo, where they
do it themselves currently.

Arjuna made an example in the wiki,
http://wiki.opscode.com/display/chef/Libraries, for an ISP scenario, that I
still have to test. With my limited knowledge of chef at this stage it seems
that it would rebuild everything on every run for every user. I will
probably stand corrected here.

Apologies if I'm being a pain in the behind on this one, but on the surface
this seems like such a great fit.

Best

--
Kenneth Kalmer
kenneth.kalmer@gmail.com
http://opensourcery.co.za
@kennethkalmer

5

Arjuna made an example in the wiki, http://wiki.opscode.com/display/chef/Libraries
, for an ISP scenario, that I still have to test. With my limited
knowledge of chef at this stage it seems that it would rebuild
everything on every run for every user. I will probably stand
corrected here.

I think you just need to spend more time with it - it'll do exactly
what you're looking for, and if you need something that's edge-case,
you can coerce it to do that too.

In this case, it doesn't rebuild everything on every run for every
user. It just ensures that what's there is as you've specified. If
it hasn't changed, it won't touch it.

I think you'll find it'll do exactly what you need.

Caleb

6

On Wed, Sep 23, 2009 at 9:05 PM, Caleb Tennis caleb.tennis@gmail.com wrote:

Arjuna made an example in the wiki,
http://wiki.opscode.com/display/chef/Libraries, for an ISP scenario, that I
still have to test. With my limited knowledge of chef at this stage it seems
that it would rebuild everything on every run for every user. I will
probably stand corrected here.

I think you just need to spend more time with it - it'll do exactly what
you're looking for, and if you need something that's edge-case, you can
coerce it to do that too.
In this case, it doesn't rebuild everything on every run for every user. It
just ensures that what's there is as you've specified. If it hasn't
changed, it won't touch it.
I think you'll find it'll do exactly what you need.
Caleb

I've not looked deeply into it, but would it not be possible to have a
flag which says if the account has been provisioned or not if you need
to do something exotic. I would have thought that for the most part
though, they should generally be run once

--
$ echo "kpfmAdpoofdufevq/dp/vl" | perl -pe 's/(.)/chr(ord($1)-1)/ge'

7 
   I agree with Caleb, the whole point of chef is to make your recipes

idempotent. For the run once recipes that I have needed you can have the
resource touch a file called .alreadydone or something and add a not_if {
File.exists?('.alreadydone')} to the resource to emulate run once recipes.
Perhaps you could wrap up this functionality into the resources themselves
with a little work to have certain resources do this automatcially for you?

Idempotence is the prevailing paradigm amongst tools like chef but
personally I think there is a case for a "migrations" paradigm. IMHO
building and maintaining a server is not that different than building
and maintaining a database. You start with the initial schema and it
evolves as time goes on.

Once good example of this are the installation of packages. Once a
package is installed there is no need to check every hour whether or
not it's installed is there? What if you want to remove a package (or
a user). You modify the configuration to remove the package and then
for ever after check once an hour to make sure it's not installed.

To me the idea of having server migrations makes a lot of sense.

I wonder what you all think.

8

On Wed, Sep 23, 2009 at 5:10 PM, Tim Uckun timuckun@gmail.com wrote:

Idempotence is the prevailing paradigm amongst tools like chef but
personally I think there is a case for a "migrations" paradigm. IMHO
building and maintaining a server is not that different than building
and maintaining a database. You start with the initial schema and it
evolves as time goes on.

The Rails-style database migrations you're referring to are an effort to
effect idempotence for database schema changes. You migrate your schema
once, and the changes that need to be applied are applied. When you run
'migrate' on a fully up-to-date schema, nothing happens because your
migrations have already been run. Idempotent system changes are not at odds
with systems evolving over time, nor are they at odds with processes being
run only once (but you are responsible for keeping track of what's been run,
at least at present).

--Brian P O'Rourke

9

Yo,

On 24/09/2009, at 12:48 PM, Brian O'Rourke wrote:

On Wed, Sep 23, 2009 at 5:10 PM, Tim Uckun timuckun@gmail.com wrote:
Idempotence is the prevailing paradigm amongst tools like chef but
personally I think there is a case for a "migrations" paradigm. IMHO
building and maintaining a server is not that different than building
and maintaining a database. You start with the initial schema and it
evolves as time goes on.

The Rails-style database migrations you're referring to are an
effort to effect idempotence for database schema changes. You
migrate your schema once, and the changes that need to be applied
are applied. When you run 'migrate' on a fully up-to-date schema,
nothing happens because your migrations have already been run.
Idempotent system changes are not at odds with systems evolving over
time, nor are they at odds with processes being run only once (but
you are responsible for keeping track of what's been run, at least
at present).

Technically under the hood the majority of our resources and providers
are migration-style, yet the DSL you declare desired state in has no
concept of a migration - it is just a target.

service "foo" do
action: start
end

On a system with the service 'foo' stopped, this will start it; while
running, it will not. File and Template compare checksums of content
[when applicable]. Other resources do not have any built in
idempotency - guards must be added via only_if or not_if.

If you really think you need this behaviour, leveraging a simple
bool node attribute will suffice:

if node.crazy_app.do_crazy_stuff
all_vhosts.each do |vhost|
remote_file "/etc/apache2/sites-enabled/#{vhost[:server_name]" do
source vhost[:server_name
end
...
end

Toggle it via attributes, a Role, the webUI/API or a JSON override file

Regards, HTH!

--
Arjuna Christensen, Software Development Engineer
Opscode, Inc.
E: aj@opscode.com