CHEF-3747 - Compile Time Packages


#1

Hi Chefs,

Recently we’ve been reviewing CHEF-3747 and wanted to get outside input.

http://tickets.opscode.com/browse/CHEF-3747

In short, this adds a shortcut for installing a package at compile time, but it has an interesting twist: It creates a corresponding package resource that will trigger notifications at converge time where the resource “would have been” if it had been a converge-time resource.

We wanted to hear more from Chris (the patch’s author) about what exactly he’s using this behavior to do. As I understand it, there are two constraints:

  1. These packages are prerequisites for gems that are installed and loaded via chef_gem;
  2. When installed, additional resources need to run via notifications.

We also want to hear from you. This behavior seems quite complex; is it something you think you would use? Is the complexity worth the benefit?

I’ve also come across some discussions arguing that moving resources to compile time is an anti pattern. Personally I think its fine for one-off hacks, but I’m wary of baking it deeper into core Chef. Considering that the compile/converge distinction is pretty much required to implement notifications, those obviously act weirdly when everything gets moved to compile time. I personally also think Chef’s strict ordering of run list items and the resource collection is the best way to get changes made in the desired order. On the other hand, this may necessitate splitting cookbooks into smaller pieces and sprinkling them across the run_list, which can be awkward.

One alternative we’ve been discussing here at Opscode is a refactor of the way that the recipe DSL is implemented such that you’d be able to prepend resources to the resource collection, or insert them at an arbitrary point, but this isn’t something that would be implemented soon.

Thoughts?


Daniel DeLeo


#2

Compile time modification is a hack that should very much be discouraged.

I’d rather see something akin to “run stages” where you could force
the convergence of a recipe before the compile phase of another.

-s

On Mon, Jan 28, 2013 at 7:02 PM, Daniel DeLeo dan@kallistec.com wrote:

Hi Chefs,

Recently we’ve been reviewing CHEF-3747 and wanted to get outside input.

http://tickets.opscode.com/browse/CHEF-3747

In short, this adds a shortcut for installing a package at compile time, but
it has an interesting twist: It creates a corresponding package resource
that will trigger notifications at converge time where the resource “would
have been” if it had been a converge-time resource.

We wanted to hear more from Chris (the patch’s author) about what exactly
he’s using this behavior to do. As I understand it, there are two
constraints:

  1. These packages are prerequisites for gems that are installed and loaded
    via chef_gem;
  2. When installed, additional resources need to run via notifications.

We also want to hear from you. This behavior seems quite complex; is it
something you think you would use? Is the complexity worth the benefit?

I’ve also come across some discussions arguing that moving resources to
compile time is an anti pattern. Personally I think its fine for one-off
hacks, but I’m wary of baking it deeper into core Chef. Considering that the
compile/converge distinction is pretty much required to implement
notifications, those obviously act weirdly when everything gets moved to
compile time. I personally also think Chef’s strict ordering of run list
items and the resource collection is the best way to get changes made in the
desired order. On the other hand, this may necessitate splitting cookbooks
into smaller pieces and sprinkling them across the run_list, which can be
awkward.

One alternative we’ve been discussing here at Opscode is a refactor of the
way that the recipe DSL is implemented such that you’d be able to prepend
resources to the resource collection, or insert them at an arbitrary point,
but this isn’t something that would be implemented soon.

Thoughts?


Daniel DeLeo


#3

I’m with Sean. Just wrote a blog post about how compile-time execution
results in an arms race where you end up just pulling in more and more
things.

  • Julian

Sent from my smartphone. Sorry about any typos.

On Jan 28, 2013, at 7:21 PM, Sean OMeara someara@gmail.com wrote:

Compile time modification is a hack that should very much be discouraged.

I’d rather see something akin to “run stages” where you could force
the convergence of a recipe before the compile phase of another.

-s

On Mon, Jan 28, 2013 at 7:02 PM, Daniel DeLeo dan@kallistec.com wrote:

Hi Chefs,

Recently we’ve been reviewing CHEF-3747 and wanted to get outside input.

http://tickets.opscode.com/browse/CHEF-3747

In short, this adds a shortcut for installing a package at compile time, but

it has an interesting twist: It creates a corresponding package resource

that will trigger notifications at converge time where the resource "would

have been" if it had been a converge-time resource.

We wanted to hear more from Chris (the patch’s author) about what exactly

he’s using this behavior to do. As I understand it, there are two

constraints:

  1. These packages are prerequisites for gems that are installed and loaded

via chef_gem;

  1. When installed, additional resources need to run via notifications.

We also want to hear from you. This behavior seems quite complex; is it

something you think you would use? Is the complexity worth the benefit?

I’ve also come across some discussions arguing that moving resources to

compile time is an anti pattern. Personally I think its fine for one-off

hacks, but I’m wary of baking it deeper into core Chef. Considering that the

compile/converge distinction is pretty much required to implement

notifications, those obviously act weirdly when everything gets moved to

compile time. I personally also think Chef’s strict ordering of run list

items and the resource collection is the best way to get changes made in the

desired order. On the other hand, this may necessitate splitting cookbooks

into smaller pieces and sprinkling them across the run_list, which can be

awkward.

One alternative we’ve been discussing here at Opscode is a refactor of the

way that the recipe DSL is implemented such that you’d be able to prepend

resources to the resource collection, or insert them at an arbitrary point,

but this isn’t something that would be implemented soon.

Thoughts?

Daniel DeLeo


#4

Sascha Bates
| sascha.bates@gmail.com | 612 850 0444 | sascha.bates@skype |
sascha_bates@yahoo |
Almost every time I’ve arranged to run installs/configs at compile
time, I usually find that I made a mistake somewhere along the
line in my work and can usually unravel a boatload of compile time
activities once I’ve identified the error. I do believe there are
valid reasons for doing it sometimes, but as Chef has matured and
my programming skills along with it, these are def exceptions and
not rules.

  I don't want to belittle anyone's issues that they are solving
  with compile-time because I know I have been frustrated and used
  that method to solve problems, but I think that generally some
  careful coding can address most compile time issues. I think that
  what we often see is people generously contributing their
  internally developed cookbooks that were used to solve specific
  problems but are not always entirely universal in application. 
  Maybe we can make a community workshop/hackday where we try to
  take some of the contributed cookbooks and work on making them
  models of ideal behavior?  There might even be an additional
  readme section that elaborates on the problem that was solved with
  a hack and how the contributor solved it long-term more gracefully
  as a chef programming lesson as well. "Advanced Chef Problem
  Solving" or something :)




  Sascha




  On 1/28/13 6:52 PM, Julian Dunn wrote:
I'm with Sean. Just wrote a blog post about how compile-time execution results in an arms race where you end up just pulling in more and more things.

http://www.juliandunn.net/2013/01/23/when-application-and-library-cookbooks-fail/

    - Julian

Sent from my smartphone. Sorry about any typos.

    On Jan 28, 2013, at 7:21 PM, Sean OMeara &lt;<a moz-do-not-send="true" href="mailto:someara@gmail.com">someara@gmail.com</a>&gt;
    wrote:
Compile time modification is a hack that should very much be discouraged.

I’d rather see something akin to “run stages” where you
could force

the convergence of a recipe before the compile phase of
another.

-s

On Mon, Jan 28, 2013 at 7:02 PM, Daniel DeLeo <dan@kallistec.com>
wrote:

Hi Chefs,
Recently we've been reviewing CHEF-3747 and wanted to get outside input.
http://tickets.opscode.com/browse/CHEF-3747
In short, this adds a shortcut for installing a package at compile time, but
it has an interesting twist: It creates a corresponding package resource
that will trigger notifications at converge time where the resource "would
have been" if it had been a converge-time resource.
We wanted to hear more from Chris (the patch's author) about what exactly
he's using this behavior to do. As I understand it, there are two
constraints:
1. These packages are prerequisites for gems that are installed and loaded
via chef_gem;
2. When installed, additional resources need to run via notifications.
We also want to hear from you. This behavior seems quite complex; is it
something you think you would use? Is the complexity worth the benefit?
I've also come across some discussions arguing that moving resources to
compile time is an anti pattern. Personally I think its fine for one-off
hacks, but I'm wary of baking it deeper into core Chef. Considering that the
compile/converge distinction is pretty much required to implement
notifications, those obviously act weirdly when everything gets moved to
compile time. I personally also think Chef's strict ordering of run list
items and the resource collection is the best way to get changes made in the
desired order. On the other hand, this may necessitate splitting cookbooks
into smaller pieces and sprinkling them across the run_list, which can be
awkward.
One alternative we've been discussing here at Opscode is a refactor of the
way that the recipe DSL is implemented such that you'd be able to prepend
resources to the resource collection, or insert them at an arbitrary point,
but this isn't something that would be implemented soon.
Thoughts?
--
Daniel DeLeo

#5

+1. I always find compile time resource execution a painful affair.
Wherever we have updated resource based notifications (using report
handlers), compile time resource execution screw them.
It will be amazing to have a multistage convergence, with cross stage
notification ability. If not multistage, something like initializers (as
rails have) where dependencies can be expressed in chef resource dsl would
work.

On Mon, Jan 28, 2013 at 4:21 PM, Sean OMeara someara@gmail.com wrote:

Compile time modification is a hack that should very much be discouraged.

I’d rather see something akin to “run stages” where you could force
the convergence of a recipe before the compile phase of another.

-s

On Mon, Jan 28, 2013 at 7:02 PM, Daniel DeLeo dan@kallistec.com wrote:

Hi Chefs,

Recently we’ve been reviewing CHEF-3747 and wanted to get outside input.

http://tickets.opscode.com/browse/CHEF-3747

In short, this adds a shortcut for installing a package at compile time,
but
it has an interesting twist: It creates a corresponding package resource
that will trigger notifications at converge time where the resource
"would
have been" if it had been a converge-time resource.

We wanted to hear more from Chris (the patch’s author) about what exactly
he’s using this behavior to do. As I understand it, there are two
constraints:

  1. These packages are prerequisites for gems that are installed and
    loaded
    via chef_gem;
  2. When installed, additional resources need to run via notifications.

We also want to hear from you. This behavior seems quite complex; is it
something you think you would use? Is the complexity worth the benefit?

I’ve also come across some discussions arguing that moving resources to
compile time is an anti pattern. Personally I think its fine for one-off
hacks, but I’m wary of baking it deeper into core Chef. Considering that
the
compile/converge distinction is pretty much required to implement
notifications, those obviously act weirdly when everything gets moved to
compile time. I personally also think Chef’s strict ordering of run list
items and the resource collection is the best way to get changes made in
the
desired order. On the other hand, this may necessitate splitting
cookbooks
into smaller pieces and sprinkling them across the run_list, which can be
awkward.

One alternative we’ve been discussing here at Opscode is a refactor of
the
way that the recipe DSL is implemented such that you’d be able to prepend
resources to the resource collection, or insert them at an arbitrary
point,
but this isn’t something that would be implemented soon.

Thoughts?


Daniel DeLeo


#6

there were some uses cases in the cookbooks previous where at
compile-time gems were being required. a lot of the cookbooks have been
refactored so that not_if/only_if blocks that use gems now lazily
require the gem and other instances of lazy loading – the wordpress
cookbook used to be a notorious offender which just did a requires
’mysql’ right up front in compile-time. i think it was felt that it was
easier to make installing a chef_gem and then requiring it to
JustWork™ rather than having to refactor requires into
evaluation-time blocks.

looking at the cleanup that has been done in the cookbooks it’s possible
that this is really an anti-pattern and that the work should have been
put into fixing the cookbooks instead. one thing to do would be to test
all the cookbooks against a patched chef-client which took out the
compile-time forcing of chef_gem and see what cookbooks are still broken
(e.g. aws cookbook is definitely, but trivially, broken).

On 1/28/13 4:52 PM, Julian Dunn wrote:

I’m with Sean. Just wrote a blog post about how compile-time execution
results in an arms race where you end up just pulling in more and more
things.

http://www.juliandunn.net/2013/01/23/when-application-and-library-cookbooks-fail/

  • Julian

Sent from my smartphone. Sorry about any typos.

On Jan 28, 2013, at 7:21 PM, Sean OMeara <someara@gmail.com
mailto:someara@gmail.com> wrote:

Compile time modification is a hack that should very much be discouraged.

I’d rather see something akin to “run stages” where you could force
the convergence of a recipe before the compile phase of another.

-s

On Mon, Jan 28, 2013 at 7:02 PM, Daniel DeLeo <dan@kallistec.com
mailto:dan@kallistec.com> wrote:

Hi Chefs,

Recently we’ve been reviewing CHEF-3747 and wanted to get outside input.

http://tickets.opscode.com/browse/CHEF-3747

In short, this adds a shortcut for installing a package at compile
time, but
it has an interesting twist: It creates a corresponding package resource
that will trigger notifications at converge time where the resource
"would
have been" if it had been a converge-time resource.

We wanted to hear more from Chris (the patch’s author) about what
exactly
he’s using this behavior to do. As I understand it, there are two
constraints:

  1. These packages are prerequisites for gems that are installed and
    loaded
    via chef_gem;
  2. When installed, additional resources need to run via notifications.

We also want to hear from you. This behavior seems quite complex; is it
something you think you would use? Is the complexity worth the benefit?

I’ve also come across some discussions arguing that moving resources to
compile time is an anti pattern. Personally I think its fine for one-off
hacks, but I’m wary of baking it deeper into core Chef. Considering
that the
compile/converge distinction is pretty much required to implement
notifications, those obviously act weirdly when everything gets moved to
compile time. I personally also think Chef’s strict ordering of run list
items and the resource collection is the best way to get changes
made in the
desired order. On the other hand, this may necessitate splitting
cookbooks
into smaller pieces and sprinkling them across the run_list, which
can be
awkward.

One alternative we’ve been discussing here at Opscode is a refactor
of the
way that the recipe DSL is implemented such that you’d be able to
prepend
resources to the resource collection, or insert them at an arbitrary
point,
but this isn’t something that would be implemented soon.

Thoughts?


Daniel DeLeo


#7

]] Daniel DeLeo

I’ve also come across some discussions arguing that moving resources
to compile time is an anti pattern. Personally I think its fine for
one-off hacks, but I’m wary of baking it deeper into core
Chef.

Agreed, one of the wonderful features of chef is that it allows you to
do those hacks when you need to.

One alternative we’ve been discussing here at Opscode is a refactor of
the way that the recipe DSL is implemented such that you’d be able to
prepend resources to the resource collection, or insert them at an
arbitrary point, but this isn’t something that would be implemented
soon.

IIRC this works already, but you’re poking around in internals. Having
the collection exposed more explicitly would be useful, I think.


Tollef Fog Heen
UNIX is user friendly, it’s just picky about who its friends are


#8

On Tue, Jan 29, 2013 at 11:21 AM, Sean OMeara someara@gmail.com wrote:

Compile time modification is a hack that should very much be discouraged.

I’d rather see something akin to “run stages” where you could force
the convergence of a recipe before the compile phase of another.

+1

If a single chef run could be multiple stages, where each stage was a
runlist and each stage compiled and converged before the next stage, this
would make my life a lot easier. For bonus points you could make it
possible to run stages individually and suddenly application deploys would
be much easier in chef.


Cheers,

Peter Donald


#9

big +1 for “phases” or “stages”

forcing resources to execute during compile-time ultimately breaks
cookbook reusability

I think puppet may already have stages and I am curious how well those
work in practice. I have never used them.
http://docs.puppetlabs.com/puppet/2.7/reference/lang_run_stages.html

On Tue, Jan 29, 2013 at 11:53 AM, Peter Donald peter@realityforge.org wrote:

On Tue, Jan 29, 2013 at 11:21 AM, Sean OMeara someara@gmail.com wrote:

Compile time modification is a hack that should very much be discouraged.

I’d rather see something akin to “run stages” where you could force
the convergence of a recipe before the compile phase of another.

+1

If a single chef run could be multiple stages, where each stage was a
runlist and each stage compiled and converged before the next stage, this
would make my life a lot easier. For bonus points you could make it possible
to run stages individually and suddenly application deploys would be much
easier in chef.


Cheers,

Peter Donald


#10

On 1/29/13 3:40 AM, “Bryan Berry” bryan.berry@gmail.com wrote:

big +1 for “phases” or “stages”

forcing resources to execute during compile-time ultimately breaks
cookbook reusability

I think puppet may already have stages and I am curious how well those
work in practice. I have never used them.
http://docs.puppetlabs.com/puppet/2.7/reference/lang_run_stages.html

There are use cases where compile time is necessary. The big one is
"install this library so I can make a call that needs it".

One side effect of stages is that you’ll get the same kind of false
escalation you get with compile time - once someone moves into an earlier
stage forcibly, lots of things might be drug along.

I think it might make more sense to talk about putting in better support
for manipulating where a resource appears on the existing resource
collection, along with an abstraction for doing the “install this early so
I can use it now” case. Let’s see if it makes a difference.

Adam


#11

I think this helps with isolation, but I think it might make order much less predictable. This is an escalation of early ordering, not an easing of the primitives that allow you to control final ordering.

Partial run-lists are a good idea, but you would need to include the idea that previous portions of the complete list have converged before. Otherwise, you’re going to see the scope of each recipe artificially increasing, and you’re going to have lots of difficulty predicting what side-effects are going to produce (for example, user ids from installed packages who order may be different based on the staged-run timing.)

What are the use-cases we are trying to solve with these features? Partial run-list application is clear. What else?

Adam

From: Peter Donald <peter@realityforge.orgmailto:peter@realityforge.org>
Date: Tuesday, January 29, 2013 2:53 AM
Cc: Chef Dev <chef-dev@lists.opscode.commailto:chef-dev@lists.opscode.com>
Subject: [chef-dev] Re: Re: CHEF-3747 - Compile Time Packages

On Tue, Jan 29, 2013 at 11:21 AM, Sean OMeara <someara@gmail.commailto:someara@gmail.com> wrote:
Compile time modification is a hack that should very much be discouraged.

I’d rather see something akin to “run stages” where you could force
the convergence of a recipe before the compile phase of another.

+1

If a single chef run could be multiple stages, where each stage was a runlist and each stage compiled and converged before the next stage, this would make my life a lot easier. For bonus points you could make it possible to run stages individually and suddenly application deploys would be much easier in chef.


Cheers,

Peter Donald


#12

Hi,

On Wed, Jan 30, 2013 at 3:27 AM, Adam Jacob adam@opscode.com wrote:

I think this helps with isolation, but I think it might make order much
less predictable. This is an escalation of early ordering, not an easing of
the primitives that allow you to control final ordering.

Partial run-lists are a good idea, but you would need to include the
idea that previous portions of the complete list have converged before.
Otherwise, you’re going to see the scope of each recipe artificially
increasing, and you’re going to have lots of difficulty predicting what
side-effects are going to produce (for example, user ids from installed
packages who order may be different based on the staged-run timing.)

That is unlikely to be an issue with the way we build our infrastructure.
Typically we break our convergence into 3 separate “stages”.

  • The first stage is basically setting up the OS level guff. Usually fairly
    consistent across hosts.
  • The second stage tends to be application server bits. i.e. Install the
    app server, install the web server, install the database server, install
    the message broker etc.
  • The third stage tends to be creating an instance of the application
    server installing resources into the application server. i.e. Define the
    destinations in the message broker, create the database in db server,
    install the application/resource config into the app server, add the site
    to the web server etc.

There is no cross stage ordering requirements. (Other than the first stage
has to occur before the second). There is almost no cross stage
notifications and no cross stage ordering constraints*. So at least in all
our use cases I think the introduction of stages would simplify rather than
increase the complexity of understanding a chef run.

  • We often use the “notifying_action” LWRP pattern [1] to separate out a
    stage and on a few occasions have had to define a resource again within an
    LWRP so as to trigger an action (typically to restart a service).

What are the use-cases we are trying to solve with these features? Partial

run-list application is clear. What

A new timing mechanism for notifications. I want to be able to say that all
the notifications with a particular tag will be resolved at a particular
point in the chef converge phase. If several config files are updated, the
service should only be retstarted once but you should be able to identify
the point during the converge by which it must be restarted

We have also talked about the ability to export resources from a LWRP so
that they could be notified even if it is using the “notifying_action” LWRP
pattern. Currently we have to occasionally redefine resources which just
sucks. However I am not even sure how this would work as it would not be
added to the resource collection until after the LWRP ran.

Partial run list application would probably be sufficient (if done right)
when combined with the enhancement to notifications.

[1]
https://github.com/realityforge/chef-cutlery/blob/master/libraries/notifying_action.rb


Cheers,

Peter Donald


#13

I see you’re implementing this from the perspective of the postgresql gem
requiring the postgresql client… I run into the same thing with mysql.

Right now we have to have phases of deploy, which to us means running
bootstrap twice or in some cases 3 or 4 times, adding roles as we go.

the simple case for us:

  1. mysql gem is required for the mysql cookbook.
  2. mysql-devel package is required to build mysql gem
  3. yum repositories need to be configured for mysql-devel to install (its a
    custom package for us)

#1 must be compile time, #2 could be either, and #3 is converge time (using
the opscode provided yum resources)

stages would fix this.

-jesse

On Tue, Jan 29, 2013 at 6:40 AM, Bryan Berry bryan.berry@gmail.com wrote:

big +1 for “phases” or “stages”

forcing resources to execute during compile-time ultimately breaks
cookbook reusability

I think puppet may already have stages and I am curious how well those
work in practice. I have never used them.
http://docs.puppetlabs.com/puppet/2.7/reference/lang_run_stages.html

On Tue, Jan 29, 2013 at 11:53 AM, Peter Donald peter@realityforge.org
wrote:

On Tue, Jan 29, 2013 at 11:21 AM, Sean OMeara someara@gmail.com wrote:

Compile time modification is a hack that should very much be
discouraged.

I’d rather see something akin to “run stages” where you could force
the convergence of a recipe before the compile phase of another.

+1

If a single chef run could be multiple stages, where each stage was a
runlist and each stage compiled and converged before the next stage,
this
would make my life a lot easier. For bonus points you could make it
possible
to run stages individually and suddenly application deploys would be much
easier in chef.


Cheers,

Peter Donald