Re: CHEF-3747 - Compile Time Packages


Hi! I have some work in progress to update this pull request to be more
generic rather than package specific. The discussion we had in the code
review made me rethink this a bit about the impact it has on resources at
large, not just the specific resources where I was running into it. That
said, I feel the current ChefPackage cookbook (which the pull request
originated from) provides a good example of the problem as well as one
approach to the solution.

So, to break the problem down, lets say I have the postgresql::ruby recipe
in my run list because I need the pg gem for some of my other recipes. The
postgresql::ruby recipe will then install the packages it requires during
the compile phase so the pg gem can be properly built. Up to this point,
everybody is happy.

Now, lets say in some custom recipe, I have something like this:

package ‘postgresql-client’ do
notifies :create, ‘ruby_block[log pg install]’, :immediately

ruby_block ‘log pg install’ do
block do
Chef::Log.warn "*** Postgresql Client has been installed!"
action :nothing

What happens here is that postgresql::ruby breaks the expected behavior.
Since the postgresql-client package was installed during compile time, no
notifications will be processed, and the ruby_block resource will never be
executed. I was experiencing this issue explicitly with package resources,
which is why I whipped up the chef_package cookbook. However, as was pointed
out in the code review, and as I have been starting to run into in some
recent work, this issue affects more than just package resources. It affects
all resources and their notifications.

In the cookbook, and the current pull request, I solved this for packages by
creating ChefPackage resources for all Package resources. When a ChefPackage is
discovered in the compile phase, a corresponding Package resource would be
created based on the ChefPackage resource (ensuring notifications were in
tact) and added to the resource collection. These resources would be
processed regularly by the runner, but instead of attempting to run the action,
it would simply check the resource if it had been updated by the last action
and if so, apply notifications.

I am currently updating the pull request to provide this behavior for all
resources by using a flag to mark it as having been updated during the
compile phase, so the runner will know its status and if the notifications should
be processed. One of the big questions is is that the expected behavior. In the
code review, it was mentioned that immediate notifications might imply that
the notified resource be executed during the compile phase. I don’t feel
that would be the expected behavior and don’t think it should be the
behavior. However, with regards to expectations, I think having it defined
in the documentation would be enough to make it clear.

The change that I am proposing is flagging resources that are run during
compile time. This allows the updated_by_last_action value to be propogated
to the execution phase. It also allows the runner to know the resource’s
action should not be attempted, only the notifications processed if its
updated_by_last_action? is returning true.

While I would be really happy with built in tools for adding resources to
arbitrary points in the resource collection, I don’t see it providing a
solution for cases like postgresql::ruby where recipes will be depending on
the pg gem during the compile phase rather than the execution phase.

So, this change is targeting a very edge case, since compile time resource
execution doesn’t (and shouldn’t) happen that often. But I feel that it’s
important for things to behave as expected when that edge case is


  • Chris

Daniel DeLeo writes:

« HTML content follows »

Hi Chefs,

Recently we’ve been reviewing CHEF-3747 and wanted to get outside input.

In short, this adds a shortcut for installing a package at compile time, but
it has an interesting twist: It creates a corresponding package resource that
will trigger notifications at converge time where the resource “would have
been” if it had been a converge-time resource.

We wanted to hear more from Chris (the patch’s author) about what exactly
he’s using this behavior to do. As I understand it, there are two

  1. These packages are prerequisites for gems that are installed and loaded
    via chef_gem;
  2. When installed, additional resources need to run via notifications.

We also want to hear from you. This behavior seems quite complex; is it
something you think you would use? Is the complexity worth the benefit?

I’ve also come across some discussions arguing that moving resources to
compile time is an anti pattern. Personally I think its fine for one-off
hacks, but I’m wary of baking it deeper into core Chef. Considering that the
compile/converge distinction is pretty much required to implement
notifications, those obviously act weirdly when everything gets moved to
compile time. I personally also think Chef’s strict ordering of run list
items and the resource collection is the best way to get changes made in the
desired order. On the other hand, this may necessitate splitting cookbooks
into smaller pieces and sprinkling them across the run_list, which can be

One alternative we’ve been discussing here at Opscode is a refactor of the
way that the recipe DSL is implemented such that you’d be able to prepend
resources to the resource collection, or insert them at an arbitrary point,
but this isn’t something that would be implemented soon.


Daniel DeLeo