Application cookbook "annoyances"

Hey guys,

in the process of converting my roles to application cookbooks I
encountered some “annoyances” which I don’t know how to handle properly yet.

  1. some cookbooks rely on roles, e.g. nagios::client or munin::client find
    their server counterpart using a search for role:monitoring

I’m thinking of keeping the roles as “marker roles” which act like tags
basically and keep backwards compatibility for cookbooks using role-based
searches.

How do you handle this?

  1. some cookbooks search for recipes in the runlist, e.g.
    apt::cacher-client searches for servers with recipe:apt\:\:cacher-ng

Now that the only item in my run_list is my-app-cookbook and the
apt::cacher-ng is include_reciped it can not be found anymore via that
search.

How do you handle such cases?

I believe there are a lot more recipes using these kinds of searches which
would break when following the application cookbook pattern.

I could live with “marker roles” to make these cookbooks happy, but for the
cookbook searches I have no clue.

Is it true that with Chef 11 include_reciped recipes show up in the
expanded run_list? Can you search for them using recipe:apt\:\:cacher-ng
for example?

Cheers,
Torben

On Wed, Feb 6, 2013 at 1:25 AM, Torben Knerr ukio@gmx.de wrote:

Hey guys,

in the process of converting my roles to application cookbooks I encountered
some "annoyances" which I don't know how to handle properly yet.

  1. some cookbooks rely on roles, e.g. nagios::client or munin::client find
    their server counterpart using a search for role:monitoring

I'm thinking of keeping the roles as "marker roles" which act like tags
basically and keep backwards compatibility for cookbooks using role-based
searches.

How do you handle this?

  1. some cookbooks search for recipes in the runlist, e.g. apt::cacher-client
    searches for servers with recipe:apt\:\:cacher-ng

Now that the only item in my run_list is my-app-cookbook and the
apt::cacher-ng is include_reciped it can not be found anymore via that
search.

How do you handle such cases?

In most cases we either patch the cookbook locally or try and get the
changes in upstream. We have yet to find any other good alternative.

--
Cheers,

Peter Donald

Ohai,

On 2/5/13 7:25 AM, "Torben Knerr" ukio@gmx.de wrote:

  1. some cookbooks rely on roles, e.g. nagios::client or munin::client
    find their server counterpart using a search for role:monitoring

I'm thinking of keeping the roles as "marker roles" which act like tags
basically and keep backwards compatibility for cookbooks using role-based
searches.

How do you handle this?

One thing we've talked about in the past is having the role set as an
attribute. The nagios cookbook currently allows this. IMO this is a great
use case for roles anyway, and one of their purposes.

An argument could be made that the search should be for a node with the
appropriate recipe applied, but only recipes in the expanded run list (not
those from 'include_recipe' in another recipe)...

  1. some cookbooks search for recipes in the runlist, e.g.
    apt::cacher-client searches for servers with recipe:apt\:\:cacher-ng

Now that the only item in my run_list is my-app-cookbook and the
apt::cacher-ng is include_reciped it can not be found anymore via
that search.

How do you handle such cases?

I believe there are a lot more recipes using these kinds of searches
which would break when following the application cookbook pattern.

I could live with "marker roles" to make these cookbooks happy, but for
the cookbook searches I have no clue.

Is it true that with Chef 11 include_reciped recipes show up in the
expanded run_list? Can you search for them using
recipe:apt\:\:cacher-ng for example?

They do not get appended to the "recipes" attribute. That has also been
discussed in the past:

I thought there was another ticket for this, but I couldn't find it.

Cheers,
Joshua

Am 06.02.2013 03:41 schrieb "Joshua Timberman" joshua@opscode.com:

Ohai,

On 2/5/13 7:25 AM, "Torben Knerr" ukio@gmx.de wrote:

  1. some cookbooks rely on roles, e.g. nagios::client or munin::client
    find their server counterpart using a search for role:monitoring

I'm thinking of keeping the roles as "marker roles" which act like tags
basically and keep backwards compatibility for cookbooks using role-based
searches.

How do you handle this?

One thing we've talked about in the past is having the role set as an
attribute. The nagios cookbook currently allows this. IMO this is a great
use case for roles anyway, and one of their purposes.

The nagios cookbook allows to set role to search for in the search query
(default is "monitoring"), but it doesn't set the role of the node, does it?

If so, I wouldn't have to keep around marker roles as .rb files but would
set the node's role in the application cookbook instead.

An argument could be made that the search should be for a node with the
appropriate recipe applied, but only recipes in the expanded run list (not
those from 'include_recipe' in another recipe)...

Agree, but this having the problems described in 2) below...

I believe that there are valid use cases for both searches, so we'd ideally
find a way to handle both.

  1. some cookbooks search for recipes in the runlist, e.g.
    apt::cacher-client searches for servers with recipe:apt\:\:cacher-ng

Now that the only item in my run_list is my-app-cookbook and the
apt::cacher-ng is include_reciped it can not be found anymore via
that search.

How do you handle such cases?

I believe there are a lot more recipes using these kinds of searches
which would break when following the application cookbook pattern.

I could live with "marker roles" to make these cookbooks happy, but for
the cookbook searches I have no clue.

Is it true that with Chef 11 include_reciped recipes show up in the
expanded run_list? Can you search for them using
recipe:apt\:\:cacher-ng for example?

They do not get appended to the "recipes" attribute. That has also been
discussed in the past:

That's currently the dealbreaker for me. I added a comment and upvoted the
ticket.

In the meantime, is there any viable workaround other than patching the
cookbooks that use search 'recipe:...'?

E.g. would it be possible to set the "recipes" attribute from within the
application cookbook and would that be indexed for search then?

I thought there was another ticket for this, but I couldn't find it.

Cheers,
Joshua

Am 06.02.2013 03:41 schrieb "Joshua Timberman" joshua@opscode.com:

Ohai,

On 2/5/13 7:25 AM, "Torben Knerr" ukio@gmx.de wrote:

  1. some cookbooks rely on roles, e.g. nagios::client or munin::client
    find their server counterpart using a search for role:monitoring

I'm thinking of keeping the roles as "marker roles" which act like tags
basically and keep backwards compatibility for cookbooks using role-based
searches.

How do you handle this?

One thing we've talked about in the past is having the role set as an
attribute. The nagios cookbook currently allows this. IMO this is a great
use case for roles anyway, and one of their purposes.

The nagios cookbook allows to set role to search for in the search query
(default is "monitoring"), but it doesn't set the role of the node, does it?

If so, I wouldn't have to keep around marker roles as .rb files but would
set the node's role in the application cookbook instead.

An argument could be made that the search should be for a node with the
appropriate recipe applied, but only recipes in the expanded run list (not
those from 'include_recipe' in another recipe)...

Agree, but this having the problems described in 2) below...

I believe that there are valid use cases for both searches, so we'd ideally
find a way to handle both.

  1. some cookbooks search for recipes in the runlist, e.g.
    apt::cacher-client searches for servers with recipe:apt\:\:cacher-ng

Now that the only item in my run_list is my-app-cookbook and the
apt::cacher-ng is include_reciped it can not be found anymore via
that search.

How do you handle such cases?

I believe there are a lot more recipes using these kinds of searches
which would break when following the application cookbook pattern.

I could live with "marker roles" to make these cookbooks happy, but for
the cookbook searches I have no clue.

Is it true that with Chef 11 include_reciped recipes show up in the
expanded run_list? Can you search for them using
recipe:apt\:\:cacher-ng for example?

They do not get appended to the "recipes" attribute. That has also been
discussed in the past:

That's currently the dealbreaker for me. I added a comment and upvoted the
ticket.

In the meantime, is there any viable workaround other than patching the
cookbooks that use search 'recipe:...'?

E.g. would it be possible to set the "recipes" attribute from within the
application cookbook and would that be indexed for search then?

I thought there was another ticket for this, but I couldn't find it.

Cheers,
Joshua

On Feb 5, 2013, at 10:54 PM, Torben Knerr ukio@gmx.de wrote:

The nagios cookbook allows to set role to search for in the search query (default is "monitoring"), but it doesn't set the role of the node, does it?

If so, I wouldn't have to keep around marker roles as .rb files but would set the node's role in the application cookbook instead.

No, it only sets what the query is going to use. Essentially:

search(:node, "role:monitoring")

Or replace with whatever the attribute gets set, e.g.:

node.set['nagios']['server_role'] = 'nagios_thing'

Results in this query:

search(:node, "role:nagios_thing")

An argument could be made that the search should be for a node with the
appropriate recipe applied, but only recipes in the expanded run list (not
those from 'include_recipe' in another recipe)...

Agree, but this having the problems described in 2) below...

I believe that there are valid use cases for both searches, so we'd ideally find a way to handle both.

They do not get appended to the "recipes" attribute. That has also been
discussed in the past:

That's currently the dealbreaker for me. I added a comment and upvoted the ticket.

In the meantime, is there any viable workaround other than patching the cookbooks that use search 'recipe:...'?

E.g. would it be possible to set the "recipes" attribute from within the application cookbook and would that be indexed for search then?

What about using an attribute on the server set via its appropriate recipe that indicates that it is the "server" component for clients to search?

We use this in our rsyslog cookbook.

Default value:

Server recipe sets the attribute:

Client searches for a node with the attribute:

This may not require any modification of the upstream cookbook on your part either. In your "application" cookbook, have a "server" recipe that sets the attribute. In the same cookbook, have a "client" recipe that performs the search. Using the Nagios example, the "server_role" is used for the nrpe.cfg file, so you could re-open the template and set mon_host appropriately.

recipe[my-nagios::server]

node.set['nagios']['server'] = true

recipe[my-nagios::client]

mon_host = search(:node, "nagios_server:true")
nrpecfg = resources("template[#{node['nagios']['nrpe']['conf_dir']}/nrpe.cfg]")
nrpecfg.variables(:mon_host => mon_host)

Cheers,
Joshua

Thanks Joshua,

I like the approach of the rsyslog cookbook more as it does not rely on
roles but rather uses a dedicated attribute to distinguish client & server.

For the other cookbooks doing "recipe:" or "role:" searches, doing an
alternative search and re-opening template in the application cookbook
would be a workaround. Still, it's a workaround and relies on the
implementation details of the worked-around cookbooks.

Someone mentioned that with Chef 11 the include_reciped recipes would be
visible in the run_list (or some expanded form of it). Is that true? I can
hardly imagine this as the include_recipes are dynamic and thus can only
be determined after the recipes have been evaluated but not in advance,
isn't it?

At least I have some ideas now how to go further...

Cheers,
Torben

On Sun, Feb 10, 2013 at 12:30 AM, Joshua Timberman joshua@opscode.comwrote:

On Feb 5, 2013, at 10:54 PM, Torben Knerr ukio@gmx.de wrote:

The nagios cookbook allows to set role to search for in the search query
(default is "monitoring"), but it doesn't set the role of the node, does it?

If so, I wouldn't have to keep around marker roles as .rb files but
would set the node's role in the application cookbook instead.

No, it only sets what the query is going to use. Essentially:

search(:node, "role:monitoring")

Or replace with whatever the attribute gets set, e.g.:

node.set['nagios']['server_role'] = 'nagios_thing'

Results in this query:

search(:node, "role:nagios_thing")

An argument could be made that the search should be for a node with the
appropriate recipe applied, but only recipes in the expanded run list
(not
those from 'include_recipe' in another recipe)...

Agree, but this having the problems described in 2) below...

I believe that there are valid use cases for both searches, so we'd
ideally find a way to handle both.

They do not get appended to the "recipes" attribute. That has also been
discussed in the past:

That's currently the dealbreaker for me. I added a comment and upvoted
the ticket.

In the meantime, is there any viable workaround other than patching the
cookbooks that use search 'recipe:...'?

E.g. would it be possible to set the "recipes" attribute from within the
application cookbook and would that be indexed for search then?

What about using an attribute on the server set via its appropriate recipe
that indicates that it is the "server" component for clients to search?

We use this in our rsyslog cookbook.

Default value:

https://github.com/opscode-cookbooks/rsyslog/blob/master/attributes/default.rb#L21

Server recipe sets the attribute:

https://github.com/opscode-cookbooks/rsyslog/blob/master/recipes/server.rb#L22

Client searches for a node with the attribute:

https://github.com/opscode-cookbooks/rsyslog/blob/master/recipes/client.rb#L25-L26

This may not require any modification of the upstream cookbook on your
part either. In your "application" cookbook, have a "server" recipe that
sets the attribute. In the same cookbook, have a "client" recipe that
performs the search. Using the Nagios example, the "server_role" is used
for the nrpe.cfg file, so you could re-open the template and set mon_host
appropriately.

recipe[my-nagios::server]

node.set['nagios']['server'] = true

recipe[my-nagios::client]

mon_host = search(:node, "nagios_server:true")
nrpecfg =
resources("template[#{node['nagios']['nrpe']['conf_dir']}/nrpe.cfg]")
nrpecfg.variables(:mon_host => mon_host)

Cheers,
Joshua

It seems to me that both patterns have their place. Roughly speaking, a dedicated attribute seems the best pattern to find a server, and a search is best when you are looking for clients.

Searching for a particular role or recipe breaks when you set up several of the same kind of server, but want a particular client to connect to a specific one. Examples are that you might have several database servers, several backup servers, or several rsyslog servers. In fact, there were one or two community recipes I couldn't use for this reason (can't remember which ones off the top of my head).

On the other hand, when the server needs a list of clients, you can often best use a search to find all of them automatically.

I sometimes go a step further, and when I create a recipe, I have a single recipe for both client and server, and an attribute that holds the fqdn of the server. The recipe uses it to identify whether it should act as client or as a server, and then if it's a client, it uses the same attribute to find the correct server among multiple ones.

-----Original message-----

From:Torben Knerr ukio@gmx.de
Sent:Tue 02-12-2013 03:36 am
Subject:[chef] Re: Re: application cookbook "annoyances"
To:chef@lists.opscode.com;

Thanks Joshua,
I like the approach of the rsyslog cookbook more as it does not rely on roles but rather uses a dedicated attribute to distinguish client & server.
For the other cookbooks doing "recipe:" or "role:" searches, doing an alternative search and re-opening template in the application cookbook would be a workaround. Still, it's a workaround and relies on the implementation details of the worked-around cookbooks.

Someone mentioned that with Chef 11 the include_reciped recipes would be visible in the run_list (or some expanded form of it). Is that true? I can hardly imagine this as the include_recipes are dynamic and thus can only be determined after the recipes have been evaluated but not in advance, isn't it?

At least I have some ideas now how to go further...
Cheers,
Torben

On Sun, Feb 10, 2013 at 12:30 AM, Joshua Timberman joshua@opscode.com wrote:

On Feb 5, 2013, at 10:54 PM, Torben Knerr ukio@gmx.de wrote:

The nagios cookbook allows to set role to search for in the search query (default is "monitoring"), but it doesn't set the role of the node, does it?

If so, I wouldn't have to keep around marker roles as .rb files but would set the node's role in the application cookbook instead.

No, it only sets what the query is going to use. Essentially:

 search(:node, "role:monitoring")

Or replace with whatever the attribute gets set, e.g.:

 node.set['nagios']['server_role'] = 'nagios_thing'

Results in this query:

 search(:node, "role:nagios_thing")

An argument could be made that the search should be for a node with the
appropriate recipe applied, but only recipes in the expanded run list (not
those from 'include_recipe' in another recipe)...

Agree, but this having the problems described in 2) below...

I believe that there are valid use cases for both searches, so we'd ideally find a way to handle both.

They do not get appended to the "recipes" attribute. That has also been
discussed in the past:

That's currently the dealbreaker for me. I added a comment and upvoted the ticket.

In the meantime, is there any viable workaround other than patching the cookbooks that use search 'recipe:...'?

E.g. would it be possible to set the "recipes" attribute from within the application cookbook and would that be indexed for search then?

What about using an attribute on the server set via its appropriate recipe that indicates that it is the "server" component for clients to search?

We use this in our rsyslog cookbook.

Default value:

https://github.com/opscode-cookbooks/rsyslog/blob/master/attributes/default.rb#L21

Server recipe sets the attribute:

https://github.com/opscode-cookbooks/rsyslog/blob/master/recipes/server.rb#L22

Client searches for a node with the attribute:

https://github.com/opscode-cookbooks/rsyslog/blob/master/recipes/client.rb#L25-L26

This may not require any modification of the upstream cookbook on your part either. In your "application" cookbook, have a "server" recipe that sets the attribute. In the same cookbook, have a "client" recipe that performs the search. Using the Nagios example, the "server_role" is used for the nrpe.cfg file, so you could re-open the template and set mon_host appropriately.

recipe[my-nagios::server]

node.set['nagios']['server'] = true

recipe[my-nagios::client]

mon_host = search(:node, "nagios_server:true")
nrpecfg = resources("template[#{node['nagios']['nrpe']['conf_dir']}/nrpe.cfg]")
nrpecfg.variables(:mon_host => mon_host)

Cheers,
Joshua