CHEF-2988 allowed_recipes, restricted_recipes, and override_recipes


#1

Let’s talk about this patch and what the design gives us versus what
it locks us into.


http://tickets.opscode.com/browse/CHEF-2988 - Run List Modifiers

Provides three new options to modify run lists:

–allowed-recipes: Restricts what recipes are allowed in the run list
to only those provided. This restriction is not applied to recipe
dependencies.
–restricted-recipes: Restricts provided recipes from running. If a
restricted recipe is a dependency of another recipe, neither are
allowed to run.
–override-runlist: Replaces the current run list with provided run
list. This override is only applied for the current run.


Bryan McLellan | opscode | technical program manager, open source
© 206.607.7108 | (t) @btmspox | (b) http://blog.loftninjas.org


#2

On Wed, Mar 21, 2012 at 3:06 PM, Bryan McLellan btm@opscode.com wrote:

Let’s talk about this patch and what the design gives us versus what
it locks us into.


http://tickets.opscode.com/browse/CHEF-2988 - Run List Modifiers

Provides three new options to modify run lists:

–allowed-recipes: Restricts what recipes are allowed in the run list
to only those provided. This restriction is not applied to recipe
dependencies.
–restricted-recipes: Restricts provided recipes from running. If a
restricted recipe is a dependency of another recipe, neither are
allowed to run.
–override-runlist: Replaces the current run list with provided run
list. This override is only applied for the current run.

My personal opinion is that it’s a hack and encourages bad practice.
If your client runs are slow, fix that. If you don’t trust your runs,
fix that. This is a path of madness and surprises all the way around.

Again, personal opinion.


#3

On Mar 21, 2012, at 12:22 PM, John Vincent wrote:

On Wed, Mar 21, 2012 at 3:06 PM, Bryan McLellan btm@opscode.com wrote:

Let’s talk about this patch and what the design gives us versus what
it locks us into.


http://tickets.opscode.com/browse/CHEF-2988 - Run List Modifiers

Provides three new options to modify run lists:

–allowed-recipes: Restricts what recipes are allowed in the run list
to only those provided. This restriction is not applied to recipe
dependencies.
–restricted-recipes: Restricts provided recipes from running. If a
restricted recipe is a dependency of another recipe, neither are
allowed to run.
–override-runlist: Replaces the current run list with provided run
list. This override is only applied for the current run.

My personal opinion is that it’s a hack and encourages bad practice.
If your client runs are slow, fix that. If you don’t trust your runs,
fix that. This is a path of madness and surprises all the way around.

+1, the only other reason I can think of is handled by https://github.com/mattray/mattray-cookbooks/tree/master/one-shot


#4

On Mar 21, 2012, at 12:06 PM, Bryan McLellan wrote:

Let’s talk about this patch and what the design gives us versus what
it locks us into.


http://tickets.opscode.com/browse/CHEF-2988 - Run List Modifiers

Provides three new options to modify run lists:

–allowed-recipes: Restricts what recipes are allowed in the run list
to only those provided. This restriction is not applied to recipe
dependencies.
–restricted-recipes: Restricts provided recipes from running. If a
restricted recipe is a dependency of another recipe, neither are
allowed to run.
–override-runlist: Replaces the current run list with provided run
list. This override is only applied for the current run.


Bryan McLellan | opscode | technical program manager, open source
© 206.607.7108 | (t) @btmspox | (b) http://blog.loftninjas.org

I could see this being pretty useful for bootstrapping. Outside of that, I agree with the other comments.

-Erik


#5

My gut agrees w lusis.

Avoiding unwanted behavior should really be dealt with via disciplined use
if git. If enterprise customers have big chef servers with too many
sysadmins and too many cookbooks uploaded, I think they might be better
served by having multiple chef servers
On Mar 21, 2012 8:22 PM, “John Vincent” lusis.org+chef-dev@gmail.com
wrote:

On Wed, Mar 21, 2012 at 3:06 PM, Bryan McLellan btm@opscode.com wrote:

Let’s talk about this patch and what the design gives us versus what
it locks us into.


http://tickets.opscode.com/browse/CHEF-2988 - Run List Modifiers

Provides three new options to modify run lists:

–allowed-recipes: Restricts what recipes are allowed in the run list
to only those provided. This restriction is not applied to recipe
dependencies.
–restricted-recipes: Restricts provided recipes from running. If a
restricted recipe is a dependency of another recipe, neither are
allowed to run.
–override-runlist: Replaces the current run list with provided run
list. This override is only applied for the current run.

My personal opinion is that it’s a hack and encourages bad practice.
If your client runs are slow, fix that. If you don’t trust your runs,
fix that. This is a path of madness and surprises all the way around.

Again, personal opinion.


#6

Sorry for top post, pre-coffee email;

Just wanted to suggest that you guys actually test the functionality
before getting evangelical – it was trivial for me to overlook this
as hogwash without building a gem and actually testing the different
modes as I couldn’t conjure a use case out of thin air – mostly came
up with "what the hell is this crap why would anyone want to do this?"
the first time I looked at the feature request.

Some cool behaviour that I hadn’t considered was actually possible
e.g.: an approximation of multi-platform cookbook dependencies with
components disabled for particular suites (with client/solo config)…
This can easily solve the age-old apt/yum/…/pacman? multiple
cookbook dependency, only run the right one. So too could another
level of abstraction designed specifically for this purpose, if that
is what we are solving for.

I am querying our client who requested CHEF-2988 to weigh on in the
mailing list with their original requirements that lead to it being
developed. Hopefully that provides a little context.

I can’t imagine this hurting a new user – if you restrict a recipe,
the including, and included recipes are not run. It’s basic. Shit just
runs or it doesn’t, and it is noisy about it. You don’t have to guess.
I can see some complexity when a new user is restricting parts of his
environment from running without considering the implications, which I
guess is what everyone is worried about – dumb people doing dumb
shit.

Said new user would receive multiple warnings and in the event she
were to seek help this would be obvious to a skilled operator.

[I have cc’d some of the users involved in the internal testing of the functionality]

–AJ

On 22 March 2012 08:53, Bryan Berry bryan.berry@gmail.com wrote:

My gut agrees w lusis.

Avoiding unwanted behavior should really be dealt with via disciplined use
if git. If enterprise customers have big chef servers with too many
sysadmins and too many cookbooks uploaded, I think they might be better
served by having multiple chef servers

On Mar 21, 2012 8:22 PM, “John Vincent” lusis.org+chef-dev@gmail.com
wrote:

On Wed, Mar 21, 2012 at 3:06 PM, Bryan McLellan btm@opscode.com wrote:

Let’s talk about this patch and what the design gives us versus what
it locks us into.


http://tickets.opscode.com/browse/CHEF-2988 - Run List Modifiers

Provides three new options to modify run lists:

–allowed-recipes: Restricts what recipes are allowed in the run list
to only those provided. This restriction is not applied to recipe
dependencies.
–restricted-recipes: Restricts provided recipes from running. If a
restricted recipe is a dependency of another recipe, neither are
allowed to run.
–override-runlist: Replaces the current run list with provided run
list. This override is only applied for the current run.

My personal opinion is that it’s a hack and encourages bad practice.
If your client runs are slow, fix that. If you don’t trust your runs,
fix that. This is a path of madness and surprises all the way around.

Again, personal opinion.


#7

On Wed, Mar 21, 2012 at 4:38 PM, AJ Christensen aj@junglist.gen.nz wrote:

Sorry for top post, pre-coffee email;

Just wanted to suggest that you guys actually test the functionality
before getting evangelical – it was trivial for me to overlook this
as hogwash without building a gem and actually testing the different
modes as I couldn’t conjure a use case out of thin air – mostly came
up with "what the hell is this crap why would anyone want to do this?"
the first time I looked at the feature request.

Some cool behaviour that I hadn’t considered was actually possible
e.g.: an approximation of multi-platform cookbook dependencies with
components disabled for particular suites (with client/solo config)…
This can easily solve the age-old apt/yum/…/pacman? multiple
cookbook dependency, only run the right one. So too could another
level of abstraction designed specifically for this purpose, if that
is what we are solving for.

IMHO that problem should be resolved with support inside chef. We have
file and template locality. That should be moved up the stack somehow.

I’m a huge hater of needing conditionals but we have a process that,
while under utillized by most folks, is pretty damn powerful. It just
needs to be expanded conceptually.

I am querying our client who requested CHEF-2988 to weigh on in the
mailing list with their original requirements that lead to it being
developed. Hopefully that provides a little context.

I can’t imagine this hurting a new user – if you restrict a recipe,
the including, and included recipes are not run. It’s basic. Shit just
runs or it doesn’t, and it is noisy about it. You don’t have to guess.
I can see some complexity when a new user is restricting parts of his
environment from running without considering the implications, which I
guess is what everyone is worried about – dumb people doing dumb
shit.

Said new user would receive multiple warnings and in the event she
were to seek help this would be obvious to a skilled operator.

Imho someone shouldn’t need to seek help for something like this. The
only justifications I’ve heard thus far are:

  1. my runs take too long
  2. I want to do a one off thing

Again, the conditional aspect is nice but I’d much rather it be
formalized in different way akin to how locality works now.

[I have cc’d some of the users involved in the internal testing of the functionality]

–AJ

On 22 March 2012 08:53, Bryan Berry bryan.berry@gmail.com wrote:

My gut agrees w lusis.

Avoiding unwanted behavior should really be dealt with via disciplined use
if git. If enterprise customers have big chef servers with too many
sysadmins and too many cookbooks uploaded, I think they might be better
served by having multiple chef servers

On Mar 21, 2012 8:22 PM, “John Vincent” lusis.org+chef-dev@gmail.com
wrote:

On Wed, Mar 21, 2012 at 3:06 PM, Bryan McLellan btm@opscode.com wrote:

Let’s talk about this patch and what the design gives us versus what
it locks us into.


http://tickets.opscode.com/browse/CHEF-2988 - Run List Modifiers

Provides three new options to modify run lists:

–allowed-recipes: Restricts what recipes are allowed in the run list
to only those provided. This restriction is not applied to recipe
dependencies.
–restricted-recipes: Restricts provided recipes from running. If a
restricted recipe is a dependency of another recipe, neither are
allowed to run.
–override-runlist: Replaces the current run list with provided run
list. This override is only applied for the current run.

My personal opinion is that it’s a hack and encourages bad practice.
If your client runs are slow, fix that. If you don’t trust your runs,
fix that. This is a path of madness and surprises all the way around.

Again, personal opinion.


#8

On Wed, Mar 21, 2012 at 1:44 PM, John Vincent
lusis.org+chef-dev@gmail.com wrote:

  1. my runs take too long
  2. I want to do a one off thing

So, besides the fact that some users want to do what you feel like is
a bad thing, is there a reason not to include it?

I think it’s a bad idea ™ to do one-off runs without having done a
complete converge in the near-term past (the side effects demand it!)

  • but I can see the value in doing something like this for things like
    triggered application deployments, where I know I want to trigger a
    single subset of the run list.

My feeling is we shouldn’t dismiss functionality some set of us find
useful simply out of dogmatic adherence to a model another (probably
majority) of us do like, unless it’s really damaging to Chef. I
don’t see how this is, other than being a little afraid of what
happens in infrastructure that relies on it a lot.

Nothing stops us from shipping it and making it obsolete by having a
more awesome way to do it.

Adam


Opscode, Inc.
Adam Jacob, Chief Customer Officer
T: (206) 619-7151 E: adam@opscode.com


#9

On Wed, Mar 21, 2012 at 9:55 PM, Adam Jacob adam@opscode.com wrote:

On Wed, Mar 21, 2012 at 1:44 PM, John Vincent
lusis.org+chef-dev@gmail.com wrote:

  1. my runs take too long
  2. I want to do a one off thing

So, besides the fact that some users want to do what you feel like is
a bad thing, is there a reason not to include it?

Fair question. The easy answer is no. There is absolutely no reason
technically not to include it.

I think it’s a bad idea ™ to do one-off runs without having done a
complete converge in the near-term past (the side effects demand it!)

  • but I can see the value in doing something like this for things like
    triggered application deployments, where I know I want to trigger a
    single subset of the run list.

My feeling is we shouldn’t dismiss functionality some set of us find
useful simply out of dogmatic adherence to a model another (probably
majority) of us do like, unless it’s really damaging to Chef. I
don’t see how this is, other than being a little afraid of what
happens in infrastructure that relies on it a lot.

I see it a bit different than dogmatic adherence. I see it as setting
an example of best practices.
There’s a reason Chef doesn’t (or rather didn’t depending) have a dry
run. This is sort of the same thing. I’m not saying not to listen to
the community and Jupiter knows there are more people contributing
code right now than me. My personal opinion is that, regardless of the
tool in question, it should espouse what it feels are best practices.
I hate Maven with the fire of 20 million suns but one thing Maven does
is stick to what it considers best practices.

Maybe that’s MORE of an argument for allowing it than not. Maven makes
it impossible to step outside its world.

Nothing stops us from shipping it and making it obsolete by having a
more awesome way to do it.

This is true but it’s harder to remove a feature than add one. Like I
said, I’d rather see the “root” problem addressed instead of adding on
a possible dangerous short-term fix.

Adam


Opscode, Inc.
Adam Jacob, Chief Customer Officer
T: (206) 619-7151 E: adam@opscode.com


#10

On 21 March 2012 20:06, Bryan McLellan btm@opscode.com wrote:

Let’s talk about this patch and what the design gives us versus what
it locks us into.


http://tickets.opscode.com/browse/CHEF-2988 - Run List Modifiers

Provides three new options to modify run lists:

–allowed-recipes: Restricts what recipes are allowed in the run list
to only those provided. This restriction is not applied to recipe
dependencies.
–restricted-recipes: Restricts provided recipes from running. If a
restricted recipe is a dependency of another recipe, neither are
allowed to run.
–override-runlist: Replaces the current run list with provided run
list. This override is only applied for the current run.

Nowhere in the ticket I can read what problem is that Chris Roberts tries
to solve with these options. Personally, I would like to see Chris step in
first and describe the original problem he had. Only then can we check if
the proposed solution is the best fit for the problem and within the
current Chef architecture.

Ringo


#11

On Thu, Mar 22, 2012 at 7:50 AM, Ringo De Smet ringo.desmet@gmail.comwrote:

On 21 March 2012 20:06, Bryan McLellan btm@opscode.com wrote:

Let’s talk about this patch and what the design gives us versus what
it locks us into.


http://tickets.opscode.com/browse/CHEF-2988 - Run List Modifiers

Provides three new options to modify run lists:

–allowed-recipes: Restricts what recipes are allowed in the run list
to only those provided. This restriction is not applied to recipe
dependencies.
–restricted-recipes: Restricts provided recipes from running. If a
restricted recipe is a dependency of another recipe, neither are
allowed to run.
–override-runlist: Replaces the current run list with provided run
list. This override is only applied for the current run.

Nowhere in the ticket I can read what problem is that Chris Roberts tries
to solve with these options. Personally, I would like to see Chris step in
first and describe the original problem he had. Only then can we check if
the proposed solution is the best fit for the problem and within the
current Chef architecture.

Ringo

These options were added as an expert tool to be used as a situation
requires. Triggering an application deployment which provides only a subset
of the run list (as previously described) can be one situation. Another
situation could be where a recipe encounters an unexpected state leading to
a failed run. To provide an example scenario:

  • An emergency application update needs to be deployed
  • During the chef run an unrelated recipe (to the application deployment)
    fails
  • Chef run has now halted and the application update has not completed
  • Failing recipe must be fixed before the application update may be deployed

These options provide the ability to ignore the failing recipe (or
explicitly specify the application deployment recipe) and continue on with
the run. This lets the application update be applied while allowing a fix
to the broken recipe to be pushed out once it has been completed and
properly tested.

While ideally all recipes in all cookbooks should work properly, it may not
always be the case. They may encounter an unexpected state on the system or
make a bad assumption about data they are processing. If/when this happens,
especially within a third party recipe, tracking down and fixing the
problem may not be trivial and adds to the time required to get an update
applied. These options provide a tool to handle these situations.

  • Chris

#12

Hi,

It seems like this is an attempt to jury-rig something onto the chef
model rather than changing the model. It seems like the use case is
actually adhoc execution of commands/resource installation using the
chef syntax? If so why not add the ability to distribute a one-shot
command to chef clients when needed. Then these application deploys
and/or apt-get upgrades or whatever could be run through this system
and regular infrastructure maintenance can continue to use the
existing approach. I think trying to merge the two approaches may lead
to a bit of complexity when adopting chef.

The one place where I can see a useful to have an include/exclude is
in the resolution of cookbook dependencies. I can quiet easily see
that it is unnecessary to have a apt cookbook in a chef server if you
are only deploying to redhat infrastructure. It would be nice to add
an exclude and be done with it.


Cheers,

Peter Donald


#13

Peter,
Good point, and I’d echo the same. We are working internally on design for just that feature (ad-hoc push execution of partial / alternative run lists) and want to share our ideas with the community once we have a better handle on it.

-Chris

On Mar 22, 2012, at 3:28 PM, Peter Donald wrote:

Hi,

It seems like this is an attempt to jury-rig something onto the chef
model rather than changing the model. It seems like the use case is
actually adhoc execution of commands/resource installation using the
chef syntax? If so why not add the ability to distribute a one-shot
command to chef clients when needed. Then these application deploys
and/or apt-get upgrades or whatever could be run through this system
and regular infrastructure maintenance can continue to use the
existing approach. I think trying to merge the two approaches may lead
to a bit of complexity when adopting chef.

The one place where I can see a useful to have an include/exclude is
in the resolution of cookbook dependencies. I can quiet easily see
that it is unnecessary to have a apt cookbook in a chef server if you
are only deploying to redhat infrastructure. It would be nice to add
an exclude and be done with it.


Cheers,

Peter Donald


#14

Hey gang,

Finally getting around to chiming in on this, as the client of Heavywater that commissioned them to build this feature (to some of you this will be of no surprise :slight_smile: here are my thoughts and experiences.

Here’s the main idea, we want regular, periodic converges to deal with all aspects of the system except for specific applications. Our application deployments are surprisingly similar to the idea of a Heroku slug. Everything the application needs bundled up in a tidy package. Any change to the that package is a new version of the application, this includes configuration changes. Not to get metaphysical but a configuration change is just the next iteration in a long line of changes in the life of an application. A configuration change changes what the application is just like changing code but we rarely version them that way. As a result we only want changes to configurations to happen on deploy, we never want our configs to change to happen due to a search index update when a new node comes online. Search is awesome and I want to make good use of it but we deploy often enough that it can wait until we ship more code and tell chef to update the config. So the end result is a steady
state run_list that everyone can expect to keep the system in check and updated and then when developers do their deploys a single application recipe chef-client run with only that recipe in the run_list. Additionally, as someone else mentioned, this has the added benefit of avoiding any issues, bugs, whatever that my occur in recipes before/after in the run_list. I want my developers to deploy as often and as quickly as possible, they won’t do that if they run into bugs I wrote in recipes causing their deploys to fail.

In the end this is an attempt at isolation of recipe converges and consolidation of any application changes to only deploy time. If there is a better way lets do that but this seemed like the least invasive and lowest risk to get the functionality I wanted.

-Joe


Name: Joseph A. Williams
Email: williams.joe@gmail.com

On Thursday, March 22, 2012 at 3:50 PM, Christopher Brown wrote:

Peter,
Good point, and I’d echo the same. We are working internally on design for just that feature (ad-hoc push execution of partial / alternative run lists) and want to share our ideas with the community once we have a better handle on it.

-Chris

On Mar 22, 2012, at 3:28 PM, Peter Donald wrote:

Hi,

It seems like this is an attempt to jury-rig something onto the chef
model rather than changing the model. It seems like the use case is
actually adhoc execution of commands/resource installation using the
chef syntax? If so why not add the ability to distribute a one-shot
command to chef clients when needed. Then these application deploys
and/or apt-get upgrades or whatever could be run through this system
and regular infrastructure maintenance can continue to use the
existing approach. I think trying to merge the two approaches may lead
to a bit of complexity when adopting chef.

The one place where I can see a useful to have an include/exclude is
in the resolution of cookbook dependencies. I can quiet easily see
that it is unnecessary to have a apt cookbook in a chef server if you
are only deploying to redhat infrastructure. It would be nice to add
an exclude and be done with it.


Cheers,

Peter Donald


#15

On Mon, Mar 26, 2012 at 12:43 PM, Joe Williams williams.joe@gmail.com wrote:

Hey gang,

Finally getting around to chiming in on this, as the client of Heavywater
that commissioned them to build this feature (to some of you this will be of
no surprise :slight_smile: here are my thoughts and experiences.

Honestly I’m surprised that it was you :wink:

Here’s the main idea, we want regular, periodic converges to deal with all
aspects of the system except for specific applications. Our application
deployments are surprisingly similar to the idea of a Heroku slug.
Everything the application needs bundled up in a tidy package. Any change to
the that package is a new version of the application, this includes
configuration changes. Not to get metaphysical but a configuration change is
just the next iteration in a long line of changes in the life of an
application. A configuration change changes what the application is just
like changing code but we rarely version them that way. As a result we only
want changes to configurations to happen on deploy, we never want our
configs to change to happen due to a search index update when a new node
comes online. Search is awesome and I want to make good use of it but we
deploy often enough that it can wait until we ship more code and tell chef
to update the config. So the end result is a steady state run_list that
everyone can expect to keep the system in check and updated and then when
developers do their deploys a single application recipe chef-client run with
only that recipe in the run_list. Additionally, as someone else mentioned,
this has the added benefit of avoiding any issues, bugs, whatever that my
occur in recipes before/after in the run_list. I want my developers to
deploy as often and as quickly as possible, they won’t do that if they run
into bugs I wrote in recipes causing their deploys to fail.

In all honestly, this feels like a case where doing the deploys with
Chef feels like it doesn’t fit. I mean I’ve seen the deploy code
you’ve written before and it’s awesome.
Does any of the application configuration get handled by chef at all?
I’m guessing not. Or are there system level configurations that impact
HOW the application is launched (env vars or whatnot?). I mean I get
that Chef manages the layout of base bits on disk (and maybe the
initial structure needed by the applications)?

Honestly, this is why at the previous company we used Jenkins for all
the deploys. We had hooks into our jobs to get information from Chef
(in our case via Noah - not acceptable for you guys obviously).
Initial bootstraps of new nodes would pull down the latest released
code artifact (tarball) so that it could come up ready to serve but
those artifacts were created by Jenkins.

In the end this is an attempt at isolation of recipe converges and
consolidation of any application changes to only deploy time. If there is a
better way lets do that but this seemed like the least invasive and lowest
risk to get the functionality I wanted.

-Joe

I totally see the need but at some point it feels like shoehorning in
to Chef. Maybe adding it provides enough flexibility to stave off the
need to “Herokuize” your world. I’m a big fan of either doing
something all the way with Chef or using a tool that’s a better fit.
If application deployments can’t afford the whole Chef run (and I can
totally see why that might be the case), then I wouldn’t use Chef to
do them.

Thanks for the clarification though. I do have a much better picture internally.

On Thursday, March 22, 2012 at 3:50 PM, Christopher Brown wrote:

Peter,
Good point, and I’d echo the same. We are working internally on design for
just that feature (ad-hoc push execution of partial / alternative run lists)
and want to share our ideas with the community once we have a better handle
on it.

-Chris

On Mar 22, 2012, at 3:28 PM, Peter Donald wrote:

Hi,

It seems like this is an attempt to jury-rig something onto the chef
model rather than changing the model. It seems like the use case is
actually adhoc execution of commands/resource installation using the
chef syntax? If so why not add the ability to distribute a one-shot
command to chef clients when needed. Then these application deploys
and/or apt-get upgrades or whatever could be run through this system
and regular infrastructure maintenance can continue to use the
existing approach. I think trying to merge the two approaches may lead
to a bit of complexity when adopting chef.

The one place where I can see a useful to have an include/exclude is
in the resolution of cookbook dependencies. I can quiet easily see
that it is unnecessary to have a apt cookbook in a chef server if you
are only deploying to redhat infrastructure. It would be nice to add
an exclude and be done with it.


Cheers,

Peter Donald


#16

I want to reopen the original discussion with a response to your
queries to try and get this moving some more:

On 22 March 2012 08:06, Bryan McLellan btm@opscode.com wrote:

Let’s talk about this patch and what the design gives us versus what
it locks us into.

What the design gives us:

  • The ability to implement via local policy a non-ideal pragmatic
    approach to restricting run list components during various convergence
    phases
  • A “straw man” fix to non-compatible recipe inclusion in the case of
    multi platform cookbook metadata dependency, e.g. apt & yum
  • Easily deprecated or refactored later on for feature more accurately
    satisfying ad-hoc, orthogonal component failure zone separation

Locks us into:

  • Clear test coverage of affected features. Code is clear and I have
    personally reviewed it and tested many times.
  • Potential problems for new users who do not anticipate the correct
    behaviour of the run list modifiers or use them by error, e.g. typo,
    or cargo-culted configuration template
  • Some documentation overhead. Heavywater would be willing to assist
    with this, of course.

Anyone else?

–AJ


http://tickets.opscode.com/browse/CHEF-2988 - Run List Modifiers

Provides three new options to modify run lists:

–allowed-recipes: Restricts what recipes are allowed in the run list
to only those provided. This restriction is not applied to recipe
dependencies.
–restricted-recipes: Restricts provided recipes from running. If a
restricted recipe is a dependency of another recipe, neither are
allowed to run.
–override-runlist: Replaces the current run list with provided run
list. This override is only applied for the current run.


Bryan McLellan | opscode | technical program manager, open source
© 206.607.7108 | (t) @btmspox | (b) http://blog.loftninjas.org


#17

I’m allowing myself to think out loud, and I’m mixing design concerns
with implementation concerns. Sorry.

On Mon, Mar 26, 2012 at 1:29 PM, Jay Feldblum y_feldblum@yahoo.com wrote:

I think the key issue is that:

application deployment =/= node convergence

Configuring a system and deploying an application are different
things. Why? Experience from a world where different people managed
systems and applications? Because one often requires more
orchestration than the other, such as when updating a database scheme
with an application deployment?

We used to make a point about Chef being a systems integration
framework rather than a configuration management language because we
wanted to bring all the pieces of building infrastructure together. I
still think this is possible and we’re getting there.

On Thu, Mar 22, 2012 at 6:28 PM, Peter Donald peter@realityforge.org wrote:

It seems like this is an attempt to jury-rig something onto the chef
model rather than changing the model. It seems like the use case is
actually adhoc execution of commands/resource installation using the
chef syntax? If so why not add the ability to distribute a one-shot
command to chef clients when needed.

Peter and CB are right that this needs to work ad-hoc runs in fully,
not just a one-off run on the client. While this could be a step in
that direction, we need to be sure we’re not creating a hurdle for
that fuller implementation.

I don’t like that in this patch we deal with allowed recipes and
restricted recipes in different parts of the code. That’s not
necessarily the patches fault, but I think requires that if we went
down this road we did some refactoring.

A more easily correctable issue is that the logs regarding allowed and
restricted recipes both call them restricted recipes, making the
interaction between these features unclear.

On Mon, Mar 26, 2012 at 5:18 PM, AJ Christensen aj@junglist.gen.nz wrote:

I want to reopen the original discussion with a response to your
queries to try and get this moving some more:

What the design gives us:

  • The ability to implement via local policy a non-ideal pragmatic
    approach to restricting run list components during various convergence
    phases

How would this functionality work with a larger policy? Would this
extend well into setting allowed and restricted cookbooks in a role?

  • A “straw man” fix to non-compatible recipe inclusion in the case of
    multi platform cookbook metadata dependency, e.g. apt & yum

It seems that the fix would be not including those cookbooks, and
rather including them based on node[‘platform’].

  • Easily deprecated or refactored later on for feature more accurately
    satisfying ad-hoc, orthogonal component failure zone separation

This gives the user more control over their run, which is good. It
might encourage them into bad habits, which is bad, but we must be
cautious not to judge their badness prematurely.

Locks us into:

  • Clear test coverage of affected features. Code is clear and I have
    personally reviewed it and tested many times.

I think this belongs in the other column. :slight_smile:

  • Potential problems for new users who do not anticipate the correct
    behaviour of the run list modifiers or use them by error, e.g. typo,
    or cargo-culted configuration template

We’ve got some sanity checking coming in from CHEF-1398 [1] for
RunListItems. Since this implementation of override run list sets
node.run_list, this should jump through that hoop properly.

This patch removes a run list item from the nodes run list, but what
if we want to restrict a dependent recipe that isn’t in the run list?
Given a run list of “recipe[check_sl]” where the check_sl cookbook
depends on the install_sl cookbook, if we run “sudo chef-client -r
recipe[install_sl]” we still run the install_sl::default recipe
because it was not in the run list but rather was in the expanded run
list. This is pretty confusing because it doesn’t do what you just
asked it to do, which was prevent that cookbook from running.
Shouldn’t this fail from the start?

The dependency selector has already run on the server and given us a
set of cookbook versions that match the provided dependencies, but
what are the consequences of restricting a cookbook now? I think we’re
likely to have recipes fail and it is going to be unclear why, a
situation that is prevented by the chef-server preventing a set of
cookbook versions that don’t resolve the dependencies. It seems that
we would want to pass the run list by the server for a thumbs up
before going too far down a road that could lead to peril.

(Thinking about and testing how this affects node attributes reminds
me that attributes are still non-deterministically loaded. Ugh.)

This does lock us into the allowed/restricted features. While I think
there is general consensus that ad-hoc runs are a good thing, I
haven’t heard anyone propose these before. My gut feeling is that the
unnecessarily complicate the creation of the run_list. If we decide we
want these, we need to ensure we’ve designed them correctly. Allowing
these features to be used via a role would likely change the final
composition of the run_list as the decisions could be made on the
server. This would change the result of a particular run between Chef
versions which is always of consequence to our design decisions.

I think that since this implementation came with an assumed design,
merging this would require getting some time in a engineering sprint
at Opscode where we could design the requirements and then see if this
patch would fit them.

Bryan

[1] http://tickets.opscode.com/browse/CHEF-1398


#18

Hi Bryan,

Just wanted to make a few responses to your points:

On Mon, Mar 26, 2012 at 3:58 PM, Bryan McLellan btm@loftninjas.org wrote:

I don’t like that in this patch we deal with allowed recipes and
restricted recipes in different parts of the code. That’s not
necessarily the patches fault, but I think requires that if we went
down this road we did some refactoring.

The restricted/allowed recipes are dealt with in the same area of the code
(run_context and language_include_recipe). It’s the run list override that
is dealt with in a separate area of the code (due to it’s nature).

A more easily correctable issue is that the logs regarding allowed and
restricted recipes both call them restricted recipes, making the
interaction between these features unclear.

When the allowed recipes option is used, the logs will output a notice with
the list of allowed recipes. From that point, any recipe provided in the
run list that is not within the allowed recipes is implicitly a restricted
recipe at the run list level, which is then reported on attempted loads.
Changing the output to specify that a recipe is restricted due to the fact
that it is not within the allowed recipes list would be trivial to add and
may provide more useful context.

How would this functionality work with a larger policy? Would this
extend well into setting allowed and restricted cookbooks in a role?

With regards to the allowed/restricted recipes, it would be very easy to
allow this to be set via role attributes. After the attribute expansion
within the #load on RunContext, the node could be checked for
allowed/restricted recipes and set them. However, I would see any values
already set having precedence as the command line call should have the
final say.

  • A “straw man” fix to non-compatible recipe inclusion in the case of
    multi platform cookbook metadata dependency, e.g. apt & yum

It seems that the fix would be not including those cookbooks, and
rather including them based on node[‘platform’].

I would agree with this and personally would lean towards either a layout
much like the templates/files use or separate platform specific recipes
that can be dynamically loaded via a “base” recipe dependent on the
currently detected platform. However, the latter falls back to cookbook
design rather then providing platform specific structuring via the cookbook
layout.

This patch removes a run list item from the nodes run list, but what
if we want to restrict a dependent recipe that isn’t in the run list?
Given a run list of “recipe[check_sl]” where the check_sl cookbook
depends on the install_sl cookbook, if we run “sudo chef-client -r
recipe[install_sl]” we still run the install_sl::default recipe
because it was not in the run list but rather was in the expanded run
list. This is pretty confusing because it doesn’t do what you just
asked it to do, which was prevent that cookbook from running.
Shouldn’t this fail from the start?

The dependency selector has already run on the server and given us a
set of cookbook versions that match the provided dependencies, but
what are the consequences of restricting a cookbook now? I think we’re
likely to have recipes fail and it is going to be unclear why, a
situation that is prevented by the chef-server preventing a set of
cookbook versions that don’t resolve the dependencies. It seems that
we would want to pass the run list by the server for a thumbs up
before going too far down a road that could lead to peril.

When a recipe is specified within the restricted-recipes option it is
simply not allowed to run, either explicitly on the run list or implicitly
via dependency. If recipe[check_sl] is dependent on recipe[install_sl] and
install_sl is a restricted recipe, neither recipe will be loaded.

Thanks,

  • Chris

#19

With regards to the allowed/restricted recipes, it would be very easy to
allow this to be set via role attributes. After the attribute expansion
within the #load on RunContext, the node could be checked for
allowed/restricted recipes and set them. However, I would see any values
already set having precedence as the command line call should have the
final say.

Still requires some specs to be added for coverage, but this would be the
gist of it:


#20

On 22 March 2012 23:28, Peter Donald peter@realityforge.org wrote:

It seems like this is an attempt to jury-rig something onto the chef
model rather than changing the model. It seems like the use case is
actually adhoc execution of commands/resource installation using the
chef syntax? If so why not add the ability to distribute a one-shot
command to chef clients when needed. Then these application deploys
and/or apt-get upgrades or whatever could be run through this system
and regular infrastructure maintenance can continue to use the
existing approach. I think trying to merge the two approaches may lead
to a bit of complexity when adopting chef.

Tools to perform one-shot commands over a set of machines:

The latter can read Chef or Ohai metadata.

No need to change Chef for that.

Ringo