# Why does chef have a two-pass execution flow?

**URL:** <https://discourse.chef.io/t/why-does-chef-have-a-two-pass-execution-flow/3542>\
**Category:** Chef Infra (archive)\
**Created:** [January 29, 2013, 2:16am UTC](https://discourse.chef.io/t/why-does-chef-have-a-two-pass-execution-flow/3542 "2013-01-29T02:16:24Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ian\_MacLeod](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/ian_macleod/32/609_2.png) [@Ian\_MacLeod](https://discourse.chef.io/u/Ian_MacLeod)\
**Post date:** [January 29, 2013, 2:16am UTC](https://discourse.chef.io/t/why-does-chef-have-a-two-pass-execution-flow/3542/1 "2013-01-29T02:16:24Z")

</div>

Sorry if this is (most likely) a repost; I wasn’t able to find any dupes in the list archive, though ☹

Chef executes recipes with a two pass model: First it compiles each resource/provider by walking through the run list. Then it converges each of the compiled resource/providers to get the host to the desired state.

While I (finally) understand how it works; I don’t understand the why. It seems like there are quite a few pitfalls w/ this model:

- It is very easy to execute tasks out of order by accidentally defining work outside of a resource (forgetting to use execute/ruby\_block providers, dropping to lower level Ruby, etc)

- LWRPs that declare inline resources have a confusingly different execution order (their actions occur in the convergence phase, but any resources they declare get run after the current provider)

- It is not very obvious to newcomers; they seem to expect that recipes are executed immediately.

What are the benefits of this two pass architecture? Why was Chef built this way? (I’m not finding obvious docs on it)

Thanks,  
-Ian

---

<div class="post-metadata">

**Author:** ![Sean\_OMeara](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/sean_omeara/32/435_2.png) [@Sean\_OMeara](https://discourse.chef.io/u/Sean_OMeara)\
**Post date:** [January 29, 2013, 3:01am UTC](https://discourse.chef.io/t/why-does-chef-have-a-two-pass-execution-flow/3542/2 "2013-01-29T03:01:57Z")

</div>

Hi Ian.

This is one of the biggest conceptual stumbling blocks with new Chef users.

Chef is very much not "just imperative programming" (despite what  
you'll read at certain places on the internet). Chef is about building  
a set of convergent resource statements called a Resource Collection.  
By doing this, it allows the you to create subscribe/notify relations  
between resources, so you can restart a service when you fix a  
configuration file, for example. Having a compilation phase allows you  
to make decisions about what to put on the collection.

It allows for things like this:

1.upto(99) { |i|  
beer\_bottle "#{i}" do  
action [:take\_down, :pass\_around]  
end  
}

That would place 99 uniquely named convergent beer\_bottles on the  
resource collection.

More usefully, it lets you take advantage of ruby libraries (sql,  
chef, anything) to drive data about what to declare.

mah\_nodes = search(:node, "role:webserver")  
template "lb\_config" do  
variables( :members =\> mah\_nodes)  
end

Chef was designed with a larger infrastructure in mind, not just  
single nodes. Spinning up a new node can automatically be integrated  
without having to manually track topology information, which is very  
hard, if not impossible to do on IaaS providers.

As to your concerns:

- "It is very easy to execute tasks out of order by accidentally  
defining work outside of a resource"

With great power comes great responsibility. There's also nothing  
stopping you from dd'ing /dev/urandom into your boot sector. After  
some practice with Chef and an understanding of the resource  
collection, spotting instances of "doing work in the compile phase"  
stand out very sorely at a glance.

Repeat: "Chef is Not Just Ruby"

- "LWRPs that declare inline resource execute in the compilation phase"

Actually, no, LWRPs (or "custom types", as I've starting calling them)  
are evaluated in the compilation phase, and added to the resource  
collection just like any core resource type. Currently, custom\_type  
with nested\_resource1 and nested\_resource2 will appear as three  
resources on the collection. In Chef11, there will be the option to  
have it appear as a single resource, through the magic of  
run\_contexts.

- "It is not very obvious to newcomers"

I could not agree more. We make a point to spend a bit of time  
explaining this and making it as clear as possible in our training  
sessions, but the docs could definitely be more clear about it. Making  
this more widely understood is a bit of a personal mission of mine.  
Unfortunately, the "Chef is just pure Ruby" myth remains widespread.

Hope that helps,

-s

On Mon, Jan 28, 2013 at 9:16 PM, Ian MacLeod [ian@wavii.com](mailto:ian@wavii.com) wrote:

> Sorry if this is (most likely) a repost; I wasn't able to find any dupes in  
> the list archive, though ☹
> 
> Chef executes recipes with a two pass model: First it compiles each  
> resource/provider by walking through the run list. Then it converges each  
> of the compiled resource/providers to get the host to the desired state.
> 
> While I (finally) understand how it works; I don't understand the why. It  
> seems like there are quite a few pitfalls w/ this model:
> 
> - It is very easy to execute tasks out of order by accidentally defining  
> work outside of a resource (forgetting to use execute/ruby\_block providers,  
> dropping to lower level Ruby, etc)
> 
> - LWRPs that declare inline resources have a confusingly different execution  
> order (their actions occur in the convergence phase, but any resources they  
> declare get run after the current provider)
> 
> - It is not very obvious to newcomers; they seem to expect that recipes are  
> executed immediately.
> 
> What are the benefits of this two pass architecture? Why was Chef built  
> this way? (I'm not finding obvious docs on it)
> 
> Thanks,  
> -Ian

---

<div class="post-metadata">

**Author:** ![Dan\_Razzell](https://avatars.discourse-cdn.com/v4/letter/d/ed655f/32.png) [@Dan\_Razzell](https://discourse.chef.io/u/Dan_Razzell)\
**Post date:** [January 29, 2013, 3:24am UTC](https://discourse.chef.io/t/why-does-chef-have-a-two-pass-execution-flow/3542/3 "2013-01-29T03:24:55Z")

</div>

I guess the /why/ makes more sense intuitively if you have some  
background in configuration management.

The basic idea of configuration management is to specify how a given  
class of system /should/ be configured, where "configured" means being  
in a measurable state.

Some mechanism separate from this specification then comes along,  
compares a given system to its specification, and applies changes  
intended bring the system into line with its specification. In Chef  
this activity is called "convergence".

Specifications are fundamentally /declarative/. They represent  
/policy/, not mechanism. They're supposed to be entirely agnostic to  
/how/ the convergence takes place.

What you're proposing is to treat specifications as if they were  
/procedural/, not declarative. That's fine, but then you have lost the  
point of configuration management. You're just programming again.

Mark Burgess has done some interesting work in this area. Look for his  
name against "computer immunology".

You're not alone in finding yourself tempted with Chef to add "just a  
bit" of procedural content to your specification. I know the feeling  
well. It starts out innocently at first. You think, well, I have to  
define a group of similar resources. I'll just iterate over them neatly  
like this. Then you realize that one of the resources is really a bit  
of a special case. So you thrown in a conditional. A bit of this here  
and that there... at some point, you forget that it's a specification.  
It's starting to look more like an algorithm. And now you're frustrated  
by this whole compile/runtime thing. It's /getting in the way/ of your  
algorithm!

No, actually. What's happened is, you've let scope creep break your  
specification. That's all, really.

On 13-01-28 06:16 PM, Ian MacLeod wrote:

> Sorry if this is (most likely) a repost; I wasn't able to find any  
> dupes in the list archive, though ☹
> 
> Chef executes recipes with a two pass model: First it /compiles/ each  
> resource/provider by walking through the run list. Then it  
> /converges/ each of the compiled resource/providers to get the host to  
> the desired state.
> 
> While I (finally) understand /how/ it works; I don't understand the  
> /why/. It seems like there are quite a few pitfalls w/ this model:
> 
> - It is very easy to execute tasks out of order by accidentally  
> defining work outside of a resource (forgetting to use  
> execute/ruby\_block providers, dropping to lower level Ruby, etc)
> 
> - LWRPs that declare inline resources have a confusingly different  
> execution order (their actions occur in the convergence phase, but any  
> resources they declare get run /after/ the current provider)
> 
> - It is not very obvious to newcomers; they seem to expect that  
> recipes are executed immediately.
> 
> What are the benefits of this two pass architecture? Why was Chef  
> built this way? (I'm not finding obvious docs on it)
> 
> Thanks,  
> -Ian

---

<div class="post-metadata">

**Author:** ![Ian\_MacLeod](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/ian_macleod/32/609_2.png) [@Ian\_MacLeod](https://discourse.chef.io/u/Ian_MacLeod)\
**Post date:** [January 29, 2013, 4:28am UTC](https://discourse.chef.io/t/why-does-chef-have-a-two-pass-execution-flow/3542/4 "2013-01-29T04:28:22Z")

</div>

Heh, fantastic example resource. I’m still a bit fuzzy, though!

Even though we’re collecting a bunch of resources and their desired states through the compilation phase - In practice it seems that for the most part, that mirrors the ordering and conditionals put forth by a given recipe.

By “in practice”, I’m referring to my fractured understanding and that gleaned off of tutorials/docs; not necessarily The Right Way 😛

How often do recipes converge out of their declarative ordering (due to triggers, or other recipes)? Am I missing a point here, but that seems to be the main benefit of the two-phase execution? Most examples I see are are pretty sequential in nature. Say, for example, the openssh cookbook[https://github.com/opscode-cookbooks/openssh/blob/master/recipes/default.rb](https://github.com/opscode-cookbooks/openssh/blob/master/recipes/default.rb):

- Install the openssh package
- Configure it as a service
- Lay down the configuration file (declared twice?)
- (Re)start the server if the config file was touched

Wouldn’t that cookbook (and many/most others) be just as successful if they converged immediately?

Apologies if I’m coming off as dogmatic/combative - I’m mostly just trying to understand the rationales here; I’ve got a few hundred hosts and a myriad of roles to manage with chef, and I want to make sure I do it properly.

LWRPs that declare inline resource execute in the compilation phase

Sorry, let me clarify. Say I have a recipe:

evil\_cat “mr-bigglesworth” do  
action :stroke  
variant "persian"  
end

And the following evil\_cat provider:

use\_inline\_resources

action :stroke do  
converge\_by(“ensure that we have an evil #{new\_resource.variant} cat on hand”) do  
unless have\_henchman? "cat-wrangler"  
shell\_out! "hire-henchman cat-wrangler"  
end

```
henchman "cat-wrangler" do
  action :dispatch
  destination "animal-shelter"
  not_if { have_cat_variant? new_resource.variant }
end

```

end

converge\_by(“stroke cat in a most evil manner”) do  
Chef::Log.info "#{new\_resource.name} is most pleased."  
end  
end

I’ll get the following ordering:

Starting Chef Client, version 11.0.0.beta.0  
Compiling Cookbooks…  
Converging 1 resources  
Recipe: world-domination::default

- evil-cat[mr-bigglesworth] action stroke
  - ensure that we have an evil persian cat on hand
  - stroke cat in a most evil mannerRecipe: 

- henchman[cat-wrangler] action dispatch
  - Dispatching cat-wrangler to animal-shelter

The inlined henchman resource ends up executing after the action block returns. This is good in that it’s consistent with how recipes behave. However, it’s confusing because most examples of LWRPs out there perform work directly in the action block (frequently calls to shell\_out!) You’re forced into an all-or-nothing of either all resources, or cheating a bit and doing the action :nothing, run\_action(…) trick.

-Ian

On Jan 28, 2013, at 7:01 PM, Sean OMeara \<[someara@gmail.com](mailto:someara@gmail.com)[mailto:someara@gmail.com](mailto:someara@gmail.com)\> wrote:

Hi Ian.

This is one of the biggest conceptual stumbling blocks with new Chef users.

Chef is very much not “just imperative programming” (despite what  
you’ll read at certain places on the internet). Chef is about building  
a set of convergent resource statements called a Resource Collection.  
By doing this, it allows the you to create subscribe/notify relations  
between resources, so you can restart a service when you fix a  
configuration file, for example. Having a compilation phase allows you  
to make decisions about what to put on the collection.

It allows for things like this:

1.upto(99) { |i|  
beer\_bottle “#{i}” do  
action [:take\_down, :pass\_around]  
end  
}

That would place 99 uniquely named convergent beer\_bottles on the  
resource collection.

More usefully, it lets you take advantage of ruby libraries (sql,  
chef, anything) to drive data about what to declare.

mah\_nodes = search(:node, “role:webserver”)  
template “lb\_config” do  
variables( :members =\> mah\_nodes)  
end

Chef was designed with a larger infrastructure in mind, not just  
single nodes. Spinning up a new node can automatically be integrated  
without having to manually track topology information, which is very  
hard, if not impossible to do on IaaS providers.

As to your concerns:

- “It is very easy to execute tasks out of order by accidentally  
defining work outside of a resource”

With great power comes great responsibility. There’s also nothing  
stopping you from dd’ing /dev/urandom into your boot sector. After  
some practice with Chef and an understanding of the resource  
collection, spotting instances of "doing work in the compile phase"  
stand out very sorely at a glance.

Repeat: “Chef is Not Just Ruby”

- “LWRPs that declare inline resource execute in the compilation phase”

Actually, no, LWRPs (or “custom types”, as I’ve starting calling them)  
are evaluated in the compilation phase, and added to the resource  
collection just like any core resource type. Currently, custom\_type  
with nested\_resource1 and nested\_resource2 will appear as three  
resources on the collection. In Chef11, there will be the option to  
have it appear as a single resource, through the magic of  
run\_contexts.

- “It is not very obvious to newcomers”

I could not agree more. We make a point to spend a bit of time  
explaining this and making it as clear as possible in our training  
sessions, but the docs could definitely be more clear about it. Making  
this more widely understood is a bit of a personal mission of mine.  
Unfortunately, the “Chef is just pure Ruby” myth remains widespread.

Hope that helps,

-s

On Mon, Jan 28, 2013 at 9:16 PM, Ian MacLeod \<[ian@wavii.com](mailto:ian@wavii.com)[mailto:ian@wavii.com](mailto:ian@wavii.com)\> wrote:  
Sorry if this is (most likely) a repost; I wasn’t able to find any dupes in  
the list archive, though ☹

Chef executes recipes with a two pass model: First it compiles each  
resource/provider by walking through the run list. Then it converges each  
of the compiled resource/providers to get the host to the desired state.

While I (finally) understand how it works; I don’t understand the why. It  
seems like there are quite a few pitfalls w/ this model:

- It is very easy to execute tasks out of order by accidentally defining  
work outside of a resource (forgetting to use execute/ruby\_block providers,  
dropping to lower level Ruby, etc)

- LWRPs that declare inline resources have a confusingly different execution  
order (their actions occur in the convergence phase, but any resources they  
declare get run after the current provider)

- It is not very obvious to newcomers; they seem to expect that recipes are  
executed immediately.

What are the benefits of this two pass architecture? Why was Chef built  
this way? (I’m not finding obvious docs on it)

Thanks,  
-Ian