Introducing the httpd cookbook

adamleff · August 25, 2014, 6:37pm

I think it's SUPER important to continue to recognize that Chef is a
toolkit, and these cookbooks that ChefInc and community members provided as
options. There is nothing requiring you to use them. I don't see this as
any different than choosing an actual web server - if you don't like Apache
or the way Apache does things, you don't HAVE to use Apache unless your
business/needs require you to do so.

Just like any language or software, as usage deepens and widens, patterns
emerge and evolve, and it's very possible that you will not agree with
them, much like this example today. Thankfully, you can either build upon
what is being provided, or forge your own path in a way that you and your
company feel comfortable.

Many folks, me included, find the wrapper cookbook option very freeing, and
I love that more and more people are embracing LWRPs or even HWRPs. It
allows me to make use of others' hard work while molding it into the way my
company expects me to do things, and I can still do so without needing to
be a ruby specialist. That doesn't make it the "right" solution, or even
that the old model of attribute-driven cookbooks is "obsolete" - it's just
the "right solution for me."

~Adam

On Mon, Aug 25, 2014 at 2:20 PM, Bráulio Bhavamitra brauliobo@gmail.com
wrote:

Hello Sean!

Again, there is a shift of paradigm going on. All the attribute overide
graph is now obselete with this wrapper cookbook logic. I'm worried that
the old model is being deprecated.

About the "dynamic_httpd" example, it really make evident the need from
users to configure things the old way. And it bogus that you need two
cookbooks for one simple thing!

cheers,
bráulio

On Mon, Aug 25, 2014 at 3:13 PM, Sean OMeara someara@opscode.com wrote:
Everybody just keep calm and carry on loving Chef =)

Bráulio!

I totally understand where you're coming from. It's as if a million
blog posts, tutorials, and Vagrant files suddenly cried out in horror.
You can't add a resource directly to a run_list, and that's annoying.
Meau Culpa.

I love that this conversation is happening. Let's riff on this a bit.

First, lets address resources vs cookbooks, then we can talk about
dynamically driven cookbooks via node attributes.

Here's the thing. Chef is supposed to be easy. People think it's hard.
Its usually people's experience consuming community cookbooks that
drives this perception. Simple cookbooks grow in scope and complexity
over time, as platform support grows, new platform versions come out,
and new versions of the technology under automation are released.

We're trying to get back to here: http://bit.ly/1mIOlyG

I don't think this is controversial. The trend towards publishing
cookbooks that ship new resources is a direct result of this. Creating
a resource has a lot of advantages over cookbooks.... First, it forces
the author to write down the interface (via parameters), defines a
scope, creates a closure around variables, and defines a domain of
predictability.

Regarding the lack or recipes in this cookbook:

The reason I chose not to ship any recipes is because that's the
simplest possible thing to do. (from the my perspective.) Breaking the
normal consumption pattern of "run list add" or include_recipe, drives
home the primitive-ness of this cookbook. This is a Chef module that
provides a new autonomous test-and-repair-bot, not a piece of
opinionated policy.

You know what you should totally do? Repair that consumption pattern.

Be the person who writes the popular cookbook that consumes this.
Publish it on the Supermarket.
Have your cookbook be the one appears in all the blog posts and
tutorials. Drive it with attributes. Call it "dynamic_httpd", or
something more clever.

Shipping the primitives gives you the freedom to do that.

-s

On Mon, Aug 25, 2014 at 1:22 PM, Jesse Nelson spheromak@gmail.com
wrote:
Using a wrapper cookbook you wouldn't need to set attributes if you
didn't
want to. For example If in my org I know the absolute authority for
configuring httpd is my cookbook then I have no need for attributes and
can
declare exactly what I want in the recipe. In your attribute model you
force
someone like me to use attributes when he doesn't want too. With just an
LWRP model you get the most flexibility with the least amount of
assumptions
on implementation.

Turns out high levels of flexibility with low levels of baked in
assumptions
is what most people want in a shared cookbook. If you want something
that
works out of the box then wait for someone to come along and publish a
more
apache2 like cookbook that used this httpd library cook.

WRT Simple I contest that this is the more simple thing:

https_service "mything" do
listen 80
run_user "awesemo"
end

VS
https_service node[:some_namespace][:http_service_name] do
   listen  node[:some_namespace][:http_port]
   run_user  node[:some_namespace][:http_user]
end
Which one of those is easier to operate ? If it's that you just want
to set
attributes in an environment or role. Then I would give you the benefit
and
say yea maybe the library cook isn't as easy as the non-library
'attribute-driven' cookbook. In this case I would also argue that the
management and operation of that model will far outweigh the time of
building a simple 'top level' cookbook to build your thing that is
declarative and easy to reason about.

Thanks Sean for all the work on this http cookbook I look forward to
abusing
it soon.

On Mon, Aug 25, 2014 at 9:52 AM, Bráulio Bhavamitra <
brauliobo@gmail.com>
wrote:
On Mon, Aug 25, 2014 at 1:15 PM, Sean OMeara someara@opscode.com
wrote:

You can absolutely use node attributes to drive your recipe... same
way you would use core Chef resource.

Your example would look more like this...

node['httpd']['services'].each do |service_name, val1, val2|
httpd_service service_name do
listen_ports val1
run_user val2
end
end

Correct, except that I would still use the send method.

What I'm suggesting is that a code like this must be on the httpd
cookbook, so that people can use it very fast for simple cases,
without a
wrapper cookbook.
-s

On Mon, Aug 25, 2014 at 11:22 AM, Bráulio Bhavamitra
brauliobo@gmail.com wrote:
But why left node values forgotten? LWRP can be used with node
values,
for
example:

node[:httpd][:services].each do |service|
httpd_service do
service.each{ |key, value| send key, value }
end
end

Also, the provider could have a way to specify how node attributes
should
fill LWRP

cheers,
bráulio

On Sat, Aug 23, 2014 at 1:04 AM, Lamont Granquist <
lamont@opscode.com>
wrote:
Yep, agreed with all of that. There's the detail of why shareable
recipes
have failed, though, and have been replaced by LWRPs. The more
involved
members of the community have all come to that realization, but its
come
at the
cost of years of fighting with it. I don't think the failures have
been
enumerated
clearly so that others understand it.

On Fri Aug 22 18:54:45 2014, Adam Jacob wrote:
I look at a little differently. I think, for most people, the
promise
of shareable recipes that encode policy has failed. If you make
them
flexible enough to matter (apaxhe2, say) you have a very
complicated
beast, when you probably only needed 20 lines of it.

I think it may well be that the having resources be the prime
unit of
re-use, rather than recipes, may well be the right abstraction in
the
end.

Adam

On Aug 22, 2014 6:31 PM, "Lamont Granquist" <lamont@opscode.com
mailto:lamont@opscode.com> wrote:
On Fri Aug 22 18:12:27 2014, Bráulio Bhavamitra wrote:

    I still cannot get this model of "all done by a wrapper
    cookbook using
    LWRP". It is just simpler to read node values...


When you go down that road you eventually wind up needing two
of
a
thing on a server and not just one instance. Then you start
putting arrays of hashes in your attributes. Since you're now
looping over an array and firing off a lot of resources you'll
want to internally implement that problem as LWRPs anyway.
Then
eventually you'll wind up wanting to merge arrays in your
attributes and you'll wind up on this page eventually:
https://coderanger.net/arrays-__and-chef/
<https://coderanger.net/arrays-and-chef/>

Its much easier to just expose the LWRPs.  Then the user can
use
node attributes, drive them with databags, or just statically
code
them in the LWRP.
--
"Lute pela sua ideologia. Seja um com sua ideologia. Viva pela sua
ideologia. Morra por sua ideologia" P.R. Sarkar

EITA - Educação, Informação e Tecnologias para Autogestão
Blog - Bráulio Bhavamitra
http://eita.org.br

"Paramapurusha é meu pai e Parama Prakriti é minha mãe. O universo é
meu lar
e todos nós somos cidadãos deste cosmo. Este universo é a
imaginação da
Mente Macrocósmica, e todas as entidades estão sendo criadas,
preservadas e
destruídas nas fases de extroversão e introversão do fluxo
imaginativo
cósmico. No âmbito pessoal, quando uma pessoa imagina algo em sua
mente,
naquele momento, essa pessoa é a única proprietária daquilo que ela
imagina,
e ninguém mais. Quando um ser humano criado mentalmente caminha por
um
milharal também imaginado, a pessoa imaginada não é a propriedade
desse
milharal, pois ele pertence ao indivíduo que o está imaginando. Este
universo foi criado na imaginação de Brahma, a Entidade Suprema, por
isso a
propriedade deste universo é de Brahma, e não dos microcosmos que
também
foram criados pela imaginação de Brahma. Nenhuma propriedade deste
mundo,
mutável ou imutável, pertence a um indivíduo em particular; tudo é o
patrimônio comum de todos."
Restante do texto em
A problemática de hoje em dia - Bráulio Bhavamitra
--
"Lute pela sua ideologia. Seja um com sua ideologia. Viva pela sua
ideologia. Morra por sua ideologia" P.R. Sarkar

EITA - Educação, Informação e Tecnologias para Autogestão
Blog - Bráulio Bhavamitra
http://eita.org.br

"Paramapurusha é meu pai e Parama Prakriti é minha mãe. O universo é
meu
lar e todos nós somos cidadãos deste cosmo. Este universo é a
imaginação da
Mente Macrocósmica, e todas as entidades estão sendo criadas,
preservadas e
destruídas nas fases de extroversão e introversão do fluxo imaginativo
cósmico. No âmbito pessoal, quando uma pessoa imagina algo em sua
mente,
naquele momento, essa pessoa é a única proprietária daquilo que ela
imagina,
e ninguém mais. Quando um ser humano criado mentalmente caminha por um
milharal também imaginado, a pessoa imaginada não é a propriedade desse
milharal, pois ele pertence ao indivíduo que o está imaginando. Este
universo foi criado na imaginação de Brahma, a Entidade Suprema, por
isso a
propriedade deste universo é de Brahma, e não dos microcosmos que
também
foram criados pela imaginação de Brahma. Nenhuma propriedade deste
mundo,
mutável ou imutável, pertence a um indivíduo em particular; tudo é o
patrimônio comum de todos."
Restante do texto em
A problemática de hoje em dia - Bráulio Bhavamitra
--
"Lute pela sua ideologia. Seja um com sua ideologia. Viva pela sua
ideologia. Morra por sua ideologia" P.R. Sarkar

EITA - Educação, Informação e Tecnologias para Autogestão
Blog - Bráulio Bhavamitra
http://eita.org.br

"Paramapurusha é meu pai e Parama Prakriti é minha mãe. O universo é meu
lar e todos nós somos cidadãos deste cosmo. Este universo é a imaginação da
Mente Macrocósmica, e todas as entidades estão sendo criadas, preservadas e
destruídas nas fases de extroversão e introversão do fluxo imaginativo
cósmico. No âmbito pessoal, quando uma pessoa imagina algo em sua mente,
naquele momento, essa pessoa é a única proprietária daquilo que ela
imagina, e ninguém mais. Quando um ser humano criado mentalmente caminha
por um milharal também imaginado, a pessoa imaginada não é a propriedade
desse milharal, pois ele pertence ao indivíduo que o está imaginando. Este
universo foi criado na imaginação de Brahma, a Entidade Suprema, por isso
a propriedade deste universo é de Brahma, e não dos microcosmos que também
foram criados pela imaginação de Brahma. Nenhuma propriedade deste mundo,
mutável ou imutável, pertence a um indivíduo em particular; tudo é o
patrimônio comum de todos."
Restante do texto em
A problemática de hoje em dia - Bráulio Bhavamitra

Jesse_Nelson · August 25, 2014, 6:41pm

Braulio,

This doesn't preclude you from using attributes. This adds a better
primitive that cookbooks that want to present an attribute-driven model can
use. So this cookbook is bottom layer with cleaner separation from the
'traditional' cookbook. Where apache2 cookbook has definitions you could
build the same apache2 cookbook that uses the resources from the http
cookbook instead. Nothing Is lost here, but everything is gained. Because
other users can now go make a different apache2 cookbook with their own
base assumptions. Or an organization can re-use the httpd primitives to
their own liking.

Re mixing code and conf:

Since we are actually configuring things and not just writing code, but
code that configures. Abstraction and Information hiding of the most
important information IMO is really bad. An operator or developer that
wants to know how a service got into the state of listening on port 1337
would have to chase that down first in recipe attributes. then in roles
then in environments, and grok attribute precedence to understand how it
is getting expressed. Declaring that state in the Authoritative location
the 'top level' recipe is IMO the most maintainable/operable way to build
infra with chef. Since I am stating clearly that this Fact about the
service/server is supposed to be this way, and this recipe is the
authority. IMO it's Composability over Inheritance (from the recipe
standpoint not the H/LWRP's).

On Mon, Aug 25, 2014 at 11:20 AM, Bráulio Bhavamitra brauliobo@gmail.com
wrote:

Hello Sean!

Again, there is a shift of paradigm going on. All the attribute overide
graph is now obselete with this wrapper cookbook logic. I'm worried that
the old model is being deprecated.

About the "dynamic_httpd" example, it really make evident the need from
users to configure things the old way. And it bogus that you need two
cookbooks for one simple thing!

cheers,
bráulio

On Mon, Aug 25, 2014 at 3:13 PM, Sean OMeara someara@opscode.com wrote:
Everybody just keep calm and carry on loving Chef =)

Bráulio!

I totally understand where you're coming from. It's as if a million
blog posts, tutorials, and Vagrant files suddenly cried out in horror.
You can't add a resource directly to a run_list, and that's annoying.
Meau Culpa.

I love that this conversation is happening. Let's riff on this a bit.

First, lets address resources vs cookbooks, then we can talk about
dynamically driven cookbooks via node attributes.

Here's the thing. Chef is supposed to be easy. People think it's hard.
Its usually people's experience consuming community cookbooks that
drives this perception. Simple cookbooks grow in scope and complexity
over time, as platform support grows, new platform versions come out,
and new versions of the technology under automation are released.

We're trying to get back to here: http://bit.ly/1mIOlyG

I don't think this is controversial. The trend towards publishing
cookbooks that ship new resources is a direct result of this. Creating
a resource has a lot of advantages over cookbooks.... First, it forces
the author to write down the interface (via parameters), defines a
scope, creates a closure around variables, and defines a domain of
predictability.

Regarding the lack or recipes in this cookbook:

The reason I chose not to ship any recipes is because that's the
simplest possible thing to do. (from the my perspective.) Breaking the
normal consumption pattern of "run list add" or include_recipe, drives
home the primitive-ness of this cookbook. This is a Chef module that
provides a new autonomous test-and-repair-bot, not a piece of
opinionated policy.

You know what you should totally do? Repair that consumption pattern.

Be the person who writes the popular cookbook that consumes this.
Publish it on the Supermarket.
Have your cookbook be the one appears in all the blog posts and
tutorials. Drive it with attributes. Call it "dynamic_httpd", or
something more clever.

Shipping the primitives gives you the freedom to do that.

-s

On Mon, Aug 25, 2014 at 1:22 PM, Jesse Nelson spheromak@gmail.com
wrote:
Using a wrapper cookbook you wouldn't need to set attributes if you
didn't
want to. For example If in my org I know the absolute authority for
configuring httpd is my cookbook then I have no need for attributes and
can
declare exactly what I want in the recipe. In your attribute model you
force
someone like me to use attributes when he doesn't want too. With just an
LWRP model you get the most flexibility with the least amount of
assumptions
on implementation.

Turns out high levels of flexibility with low levels of baked in
assumptions
is what most people want in a shared cookbook. If you want something
that
works out of the box then wait for someone to come along and publish a
more
apache2 like cookbook that used this httpd library cook.

WRT Simple I contest that this is the more simple thing:

https_service "mything" do
listen 80
run_user "awesemo"
end

VS
https_service node[:some_namespace][:http_service_name] do
   listen  node[:some_namespace][:http_port]
   run_user  node[:some_namespace][:http_user]
end
Which one of those is easier to operate ? If it's that you just want
to set
attributes in an environment or role. Then I would give you the benefit
and
say yea maybe the library cook isn't as easy as the non-library
'attribute-driven' cookbook. In this case I would also argue that the
management and operation of that model will far outweigh the time of
building a simple 'top level' cookbook to build your thing that is
declarative and easy to reason about.

Thanks Sean for all the work on this http cookbook I look forward to
abusing
it soon.

On Mon, Aug 25, 2014 at 9:52 AM, Bráulio Bhavamitra <
brauliobo@gmail.com>
wrote:
On Mon, Aug 25, 2014 at 1:15 PM, Sean OMeara someara@opscode.com
wrote:

You can absolutely use node attributes to drive your recipe... same
way you would use core Chef resource.

Your example would look more like this...

node['httpd']['services'].each do |service_name, val1, val2|
httpd_service service_name do
listen_ports val1
run_user val2
end
end

Correct, except that I would still use the send method.

What I'm suggesting is that a code like this must be on the httpd
cookbook, so that people can use it very fast for simple cases,
without a
wrapper cookbook.
-s

On Mon, Aug 25, 2014 at 11:22 AM, Bráulio Bhavamitra
brauliobo@gmail.com wrote:
But why left node values forgotten? LWRP can be used with node
values,
for
example:

node[:httpd][:services].each do |service|
httpd_service do
service.each{ |key, value| send key, value }
end
end

Also, the provider could have a way to specify how node attributes
should
fill LWRP

cheers,
bráulio

On Sat, Aug 23, 2014 at 1:04 AM, Lamont Granquist <
lamont@opscode.com>
wrote:
Yep, agreed with all of that. There's the detail of why shareable
recipes
have failed, though, and have been replaced by LWRPs. The more
involved
members of the community have all come to that realization, but its
come
at the
cost of years of fighting with it. I don't think the failures have
been
enumerated
clearly so that others understand it.

On Fri Aug 22 18:54:45 2014, Adam Jacob wrote:
I look at a little differently. I think, for most people, the
promise
of shareable recipes that encode policy has failed. If you make
them
flexible enough to matter (apaxhe2, say) you have a very
complicated
beast, when you probably only needed 20 lines of it.

I think it may well be that the having resources be the prime
unit of
re-use, rather than recipes, may well be the right abstraction in
the
end.

Adam

On Aug 22, 2014 6:31 PM, "Lamont Granquist" <lamont@opscode.com
mailto:lamont@opscode.com> wrote:
On Fri Aug 22 18:12:27 2014, Bráulio Bhavamitra wrote:

    I still cannot get this model of "all done by a wrapper
    cookbook using
    LWRP". It is just simpler to read node values...


When you go down that road you eventually wind up needing two
of
a
thing on a server and not just one instance. Then you start
putting arrays of hashes in your attributes. Since you're now
looping over an array and firing off a lot of resources you'll
want to internally implement that problem as LWRPs anyway.
Then
eventually you'll wind up wanting to merge arrays in your
attributes and you'll wind up on this page eventually:
https://coderanger.net/arrays-__and-chef/
<https://coderanger.net/arrays-and-chef/>

Its much easier to just expose the LWRPs.  Then the user can
use
node attributes, drive them with databags, or just statically
code
them in the LWRP.
--
"Lute pela sua ideologia. Seja um com sua ideologia. Viva pela sua
ideologia. Morra por sua ideologia" P.R. Sarkar

EITA - Educação, Informação e Tecnologias para Autogestão
Blog - Bráulio Bhavamitra
http://eita.org.br

"Paramapurusha é meu pai e Parama Prakriti é minha mãe. O universo é
meu lar
e todos nós somos cidadãos deste cosmo. Este universo é a
imaginação da
Mente Macrocósmica, e todas as entidades estão sendo criadas,
preservadas e
destruídas nas fases de extroversão e introversão do fluxo
imaginativo
cósmico. No âmbito pessoal, quando uma pessoa imagina algo em sua
mente,
naquele momento, essa pessoa é a única proprietária daquilo que ela
imagina,
e ninguém mais. Quando um ser humano criado mentalmente caminha por
um
milharal também imaginado, a pessoa imaginada não é a propriedade
desse
milharal, pois ele pertence ao indivíduo que o está imaginando. Este
universo foi criado na imaginação de Brahma, a Entidade Suprema, por
isso a
propriedade deste universo é de Brahma, e não dos microcosmos que
também
foram criados pela imaginação de Brahma. Nenhuma propriedade deste
mundo,
mutável ou imutável, pertence a um indivíduo em particular; tudo é o
patrimônio comum de todos."
Restante do texto em
A problemática de hoje em dia - Bráulio Bhavamitra
--
"Lute pela sua ideologia. Seja um com sua ideologia. Viva pela sua
ideologia. Morra por sua ideologia" P.R. Sarkar

EITA - Educação, Informação e Tecnologias para Autogestão
Blog - Bráulio Bhavamitra
http://eita.org.br

"Paramapurusha é meu pai e Parama Prakriti é minha mãe. O universo é
meu
lar e todos nós somos cidadãos deste cosmo. Este universo é a
imaginação da
Mente Macrocósmica, e todas as entidades estão sendo criadas,
preservadas e
destruídas nas fases de extroversão e introversão do fluxo imaginativo
cósmico. No âmbito pessoal, quando uma pessoa imagina algo em sua
mente,
naquele momento, essa pessoa é a única proprietária daquilo que ela
imagina,
e ninguém mais. Quando um ser humano criado mentalmente caminha por um
milharal também imaginado, a pessoa imaginada não é a propriedade desse
milharal, pois ele pertence ao indivíduo que o está imaginando. Este
universo foi criado na imaginação de Brahma, a Entidade Suprema, por
isso a
propriedade deste universo é de Brahma, e não dos microcosmos que
também
foram criados pela imaginação de Brahma. Nenhuma propriedade deste
mundo,
mutável ou imutável, pertence a um indivíduo em particular; tudo é o
patrimônio comum de todos."
Restante do texto em
A problemática de hoje em dia - Bráulio Bhavamitra
--
"Lute pela sua ideologia. Seja um com sua ideologia. Viva pela sua
ideologia. Morra por sua ideologia" P.R. Sarkar

EITA - Educação, Informação e Tecnologias para Autogestão
Blog - Bráulio Bhavamitra
http://eita.org.br

"Paramapurusha é meu pai e Parama Prakriti é minha mãe. O universo é meu
lar e todos nós somos cidadãos deste cosmo. Este universo é a imaginação da
Mente Macrocósmica, e todas as entidades estão sendo criadas, preservadas e
destruídas nas fases de extroversão e introversão do fluxo imaginativo
cósmico. No âmbito pessoal, quando uma pessoa imagina algo em sua mente,
naquele momento, essa pessoa é a única proprietária daquilo que ela
imagina, e ninguém mais. Quando um ser humano criado mentalmente caminha
por um milharal também imaginado, a pessoa imaginada não é a propriedade
desse milharal, pois ele pertence ao indivíduo que o está imaginando. Este
universo foi criado na imaginação de Brahma, a Entidade Suprema, por isso
a propriedade deste universo é de Brahma, e não dos microcosmos que também
foram criados pela imaginação de Brahma. Nenhuma propriedade deste mundo,
mutável ou imutável, pertence a um indivíduo em particular; tudo é o
patrimônio comum de todos."
Restante do texto em
A problemática de hoje em dia - Bráulio Bhavamitra

Phil_Dibowitz · September 6, 2014, 7:38am

On 08/22/2014 06:54 PM, Adam Jacob wrote:

I look at a little differently. I think, for most people, the promise of
shareable recipes that encode policy has failed. If you make them flexible
enough to matter (apaxhe2, say) you have a very complicated beast, when you
probably only needed 20 lines of it.

I think it may well be that the having resources be the prime unit of re-use,
rather than recipes, may well be the right abstraction in the end.

So I think that Adam's right in that sharable recipes have failed - but I
don't actually think the concept is broken.

I believe one of the core patterns common in the community yields cookbooks
that end up either not being flexible enough to be general, or being so
cumbersome as to be unusable. But it doesn't have to be this way.

I'm actually planning to have a session on this at summit, but here's the gist.

When we set out to build the base of cookbooks for Facebook, I had confidence
the overall approach would be useful to other organizations, but I didn't know
how re-usable the cookbooks themselves would be - and I didn't have any clue
how they'd interact with the concept of community cookbooks.

But after spending over a year traveling the world and talking about Chef to
Chef users - and after finally releasing two cookbooks (more to come), I
actually now believe what we did was make a new way to write community
cookbooks. Our goal in our "core cookbooks," as we call them, was to build
generic simple data-driven APIs that would be flexible enough for tons of
different teams across the company to use to do anything with a given
service/config/thing, without us having to know all the combinations ahead of
time. And that's pretty much, I now believe, what you want in a community
cookbook.

At the moment community cookbooks tend to build APIs that look like this

default['foo']['config_file'] = '/etc/foo'
default['foo']['logdir'] = '/var/log/foo'
default['foo']['loglevel'] = 'DEBUG'
default['foo']['primary_vip']['url'] = 'www.sample.com'
default['foo']['primary_vip']['root'] = '/var/foo/sample.com'
default['foo']['primary_vip']['allow_index'] = false
default['foo']['secondary_vip']['url'] = 'www.bobsyouruncle.com'
default['foo']['secondary_vip']['root'] = '/var/foo/bobsyouruncle.com'
default['foo']['secondary_vip']['allow_index'] = false

This seems like a simple API for this software called 'foo' (which I assume is
vaguely apache-esque in it's config concept, but uses an INI syntax).

Now, each of these is templatized in an otherwise-untemplatized config. This
looks like mostly text, and not code:

[global]
logdir = <%= node['foo']['logdir'] %>
loglevel = <%= node['foo']['loglevel'] %>

[<%= node['foo']['primary_vip']['url'] %>]
root = <%= node['foo']['root'] %>
...

And so on. Now this is a simple cookbook. It only has a few options. But
there's a problem. If Foo 2.0 adds a bunch of options, your cookbook
inherently doesn't support them until you change it. And if someone wants to
set something you didn't care about, now you have to add that into the
template, the attributes, the docs, etc.

So if someone wants to add alias support to each VIP now you gotta add:

node['foo']['primary_vip']['alias'] = 'awesome.sample.com'
node['foo']['secondary_vip']['alias'] = ''

And then modify the template and bla bla bla.

What if we did things differently. We give our attributes slightly more
structure - just enough so the template can care only about the format of the
config and not the config itself:

default['foo']['config_file'] = '/etc/foo/foo.conf'
default['foo']['config']['global'] => {
'logdir' => '/var/log/foo',
'loglevel' => 'DEBUG',
}
default['foo']['config']['www.sample.com'] => {
'root' => '/var/foo/sample.com',
}

And then we change our template to be way more flexible:

<% node['foo']['config'].to_hash.each do |section, config| %>
[<%= section %>]
<% config.each do |key, val| %>
<%= key %> = <%= val %>
<% end %>

<% end %>

That tiny template will write out the 'global' section and every VIP section.
It's complete, shorter than the other one, and supports every current and
future option - assuming a consistent format. And if you want have 99999 VIPs
in your config, just keep adding them, there's no max that the cookbook cares
about anymore.

If someone wants to use the allow_index feature, even though you didn't
populate it by default, they just add
node['foo']['config']['www.sample.com']['allow_index'] to a later cookbook, or
to their role. No problem.

That template is far more simple and flexible than the other version. And it's
easy to do things like support arrays on the left hand side of a sectional config:

...
<% config.each do |key, val| %>
<% val = val.join(', ') if val.is_a?(Array)
<%= key %> = <%= val %>
<% end %>
...

Which, given a user can just pass in 'foo, bar', is probably unnecessary, but
you could do it in a generic way like that.

To which you may ask "but how do I document all the options that you can put
under node['foo']['config'][VIP] ?" And the answer is.. Foo already has
documentation and your users already know how to configure Foo. And if they
don't, your cookbook isn't there to teach them. Your cookbook is to help them
automate that configuration.

You no longer have to decide between supporting everything and having a
grokable config.

Occasionally you'll run into something whose config is so asinine that coming
up with a generalized data structure to represent it doesn't seem feasible -
but the vast vast majority of the time there's an obvious and straight forward
mapping.

We do this for tons of stuff - and at the moment we've open-sourced cron and
sysctl.

The other thing that you end up doing when you go down this road is taking
applications that by default split up their config into file-sized chunks so
that RPMs or DEBs can drop off random files, and making one file that holds
all of the Chef-controlled parts. So let's say this is apache instead of
making one file per VirtualHost definition, you'd make a apache_chef.conf
which had all of your VirtualHosts. Now when someone removes one from their
Chef config, it's also removed from the servers - your entire config is
idempotent. Plus, you don't have to fire off tons of resources for every piece
of your config.

--
Phil Dibowitz phil@ipom.com
Open Source software and tech docs Insanity Palace of Metallica
http://www.phildev.net/ http://www.ipom.com/

"Be who you are and say what you feel, because those who mind don't matter
and those who matter don't mind."

Dr. Seuss

brauliobo · September 6, 2014, 1:17pm

Thanks Phil for excelent discussion and examples of how to use node
attributes.

When LWRP are coded, how do you think both ways to configure can the
connected?

cheers,
bráulio

On Sat, Sep 6, 2014 at 4:38 AM, Phil Dibowitz phil@ipom.com wrote:

On 08/22/2014 06:54 PM, Adam Jacob wrote:

I look at a little differently. I think, for most people, the promise of
shareable recipes that encode policy has failed. If you make them
flexible
enough to matter (apaxhe2, say) you have a very complicated beast, when
you
probably only needed 20 lines of it.

I think it may well be that the having resources be the prime unit of
re-use,
rather than recipes, may well be the right abstraction in the end.

So I think that Adam's right in that sharable recipes have failed - but I
don't actually think the concept is broken.

I believe one of the core patterns common in the community yields cookbooks
that end up either not being flexible enough to be general, or being so
cumbersome as to be unusable. But it doesn't have to be this way.

I'm actually planning to have a session on this at summit, but here's the
gist.

When we set out to build the base of cookbooks for Facebook, I had
confidence
the overall approach would be useful to other organizations, but I didn't
know
how re-usable the cookbooks themselves would be - and I didn't have any
clue
how they'd interact with the concept of community cookbooks.

But after spending over a year traveling the world and talking about Chef
to
Chef users - and after finally releasing two cookbooks (more to come), I
actually now believe what we did was make a new way to write community
cookbooks. Our goal in our "core cookbooks," as we call them, was to build
generic simple data-driven APIs that would be flexible enough for tons of
different teams across the company to use to do anything with a given
service/config/thing, without us having to know all the combinations ahead
of
time. And that's pretty much, I now believe, what you want in a community
cookbook.

At the moment community cookbooks tend to build APIs that look like this

default['foo']['config_file'] = '/etc/foo'
default['foo']['logdir'] = '/var/log/foo'
default['foo']['loglevel'] = 'DEBUG'
default['foo']['primary_vip']['url'] = 'www.sample.com'
default['foo']['primary_vip']['root'] = '/var/foo/sample.com'
default['foo']['primary_vip']['allow_index'] = false
default['foo']['secondary_vip']['url'] = 'www.bobsyouruncle.com'
default['foo']['secondary_vip']['root'] = '/var/foo/bobsyouruncle.com'
default['foo']['secondary_vip']['allow_index'] = false

This seems like a simple API for this software called 'foo' (which I
assume is
vaguely apache-esque in it's config concept, but uses an INI syntax).

Now, each of these is templatized in an otherwise-untemplatized config.
This
looks like mostly text, and not code:

[global]
logdir = <%= node['foo']['logdir'] %>
loglevel = <%= node['foo']['loglevel'] %>

[<%= node['foo']['primary_vip']['url'] %>]
root = <%= node['foo']['root'] %>
...

And so on. Now this is a simple cookbook. It only has a few options. But
there's a problem. If Foo 2.0 adds a bunch of options, your cookbook
inherently doesn't support them until you change it. And if someone wants
to
set something you didn't care about, now you have to add that into the
template, the attributes, the docs, etc.

So if someone wants to add alias support to each VIP now you gotta add:

node['foo']['primary_vip']['alias'] = 'awesome.sample.com'
node['foo']['secondary_vip']['alias'] = ''

And then modify the template and bla bla bla.

What if we did things differently. We give our attributes slightly more
structure - just enough so the template can care only about the format of
the
config and not the config itself:

default['foo']['config_file'] = '/etc/foo/foo.conf'
default['foo']['config']['global'] => {
'logdir' => '/var/log/foo',
'loglevel' => 'DEBUG',
}
default['foo']['config']['www.sample.com'] => {
'root' => '/var/foo/sample.com',
}

And then we change our template to be way more flexible:

<% node['foo']['config'].to_hash.each do |section, config| %>
[<%= section %>]
<% config.each do |key, val| %>
<%= key %> = <%= val %>
<% end %>

<% end %>

That tiny template will write out the 'global' section and every VIP
section.
It's complete, shorter than the other one, and supports every current and
future option - assuming a consistent format. And if you want have 99999
VIPs
in your config, just keep adding them, there's no max that the cookbook
cares
about anymore.

If someone wants to use the allow_index feature, even though you didn't
populate it by default, they just add
node['foo']['config']['www.sample.com']['allow_index'] to a later
cookbook, or
to their role. No problem.

That template is far more simple and flexible than the other version. And
it's
easy to do things like support arrays on the left hand side of a sectional
config:

...
<% config.each do |key, val| %>
<% val = val.join(', ') if val.is_a?(Array)
<%= key %> = <%= val %>
<% end %>
...

Which, given a user can just pass in 'foo, bar', is probably unnecessary,
but
you could do it in a generic way like that.

To which you may ask "but how do I document all the options that you can
put
under node['foo']['config'][VIP] ?" And the answer is.. Foo already has
documentation and your users already know how to configure Foo. And if they
don't, your cookbook isn't there to teach them. Your cookbook is to help
them
automate that configuration.

You no longer have to decide between supporting everything and having a
grokable config.

Occasionally you'll run into something whose config is so asinine that
coming
up with a generalized data structure to represent it doesn't seem feasible

but the vast vast majority of the time there's an obvious and straight
forward
mapping.

We do this for tons of stuff - and at the moment we've open-sourced cron
and
sysctl.

The other thing that you end up doing when you go down this road is taking
applications that by default split up their config into file-sized chunks
so
that RPMs or DEBs can drop off random files, and making one file that holds
all of the Chef-controlled parts. So let's say this is apache instead of
making one file per VirtualHost definition, you'd make a apache_chef.conf
which had all of your VirtualHosts. Now when someone removes one from their
Chef config, it's also removed from the servers - your entire config is
idempotent. Plus, you don't have to fire off tons of resources for every
piece
of your config.

--
Phil Dibowitz phil@ipom.com
Open Source software and tech docs Insanity Palace of Metallica
http://www.phildev.net/ http://www.ipom.com/

"Be who you are and say what you feel, because those who mind don't matter
and those who matter don't mind."

Dr. Seuss

--
"Lute pela sua ideologia. Seja um com sua ideologia. Viva pela sua
ideologia. Morra por sua ideologia" P.R. Sarkar

EITA - Educação, Informação e Tecnologias para Autogestão

"Paramapurusha é meu pai e Parama Prakriti é minha mãe. O universo é meu
lar e todos nós somos cidadãos deste cosmo. Este universo é a imaginação da
Mente Macrocósmica, e todas as entidades estão sendo criadas, preservadas e
destruídas nas fases de extroversão e introversão do fluxo imaginativo
cósmico. No âmbito pessoal, quando uma pessoa imagina algo em sua mente,
naquele momento, essa pessoa é a única proprietária daquilo que ela
imagina, e ninguém mais. Quando um ser humano criado mentalmente caminha
por um milharal também imaginado, a pessoa imaginada não é a propriedade
desse milharal, pois ele pertence ao indivíduo que o está imaginando. Este
universo foi criado na imaginação de Brahma, a Entidade Suprema, por isso
a propriedade deste universo é de Brahma, e não dos microcosmos que também
foram criados pela imaginação de Brahma. Nenhuma propriedade deste mundo,
mutável ou imutável, pertence a um indivíduo em particular; tudo é o
patrimônio comum de todos."
Restante do texto em

Phil_Dibowitz · September 7, 2014, 1:37am

On 09/06/2014 06:17 AM, Bráulio Bhavamitra wrote:

Thanks Phil for excelent discussion and examples of how to use node
attributes.

When LWRP are coded, how do you think both ways to configure can the
connected?

Well I think that LWRPs are often the wrong way to provide an API. Not because
LWRPs are themselves bad - they're awesome - but for the same reason we don't
use most of the standard built-in providers. We don't use 'cron' for example,
we built a template and provide a data-api as I described in the last email.
You can see this here:

https://github.com/facebook/chef-utils/tree/master/cookbooks/fb_cron

But there are times when an LWRP is what you need. The most common thing we
use LWRPs for is when the data we're pulling from the node object cannot be
put in a single file, and represents some dynamic set of files - and we won't
know what that set of files is until after everyone has munged the node
object. We avoid this wherever possible, because it makes system-level
idempotency more difficult, but sometimes you don't have a choice. So the
example here might be if the example software I was using required every vip
to be in a different file. Now again - often times it's how they set it up,
but it's not required. But lets just say it was absolutely required.

In this modified example the software requires /etc/foo/foo.conf and then each
site goes in a file like /etc/foo/vips/www.sample.com.conf. And lets assume,
just to make it more interesting that the main configuration doesn't take that
"[global]" header but the other ones do so they know the site name.

In this case I'd tweak the attributes slightly and put all the VIPs under
their own section:

default['foo']['config_file'] = '/etc/foo/foo.conf'
default['foo']['config']['global'] => {
'logdir' => '/var/log/foo',
'loglevel' => 'DEBUG',
}
default['foo']['config']['vips']['www.sample.com'] => {
'root' => '/var/foo/sample.com',
}

Then I'd the recipe to do:

template '/etc/foo/foo.conf' do
source 'foo.conf.erb'
owner 'root'
group 'root'
mode '0644'
end
foo_gen_site_files 'doit'

And that would call a 'gen_site_files' LWRP. Notice I'm not passing anything
to it and notice it's not meant for anyone else to call. Now inside the
provider I'd have:

action :run do
node['foo']['config']['vips'].keys do |url|
template "/etc/foo/vips/#{url}.conf" do
source 'vip.conf.erb'
user 'root'
group 'root'
mode '0644'
variables({'site' => url})
end
end

# Be idempotent - clean up crap we no longer own
# We have to do this because Foo is too dumb to allow
# all sites in a single file. I hate Foo.
Dir.glob('/etc/foo/vips/*.conf').each do |cfile|
  site = File.basename(cfile, '.conf')
  next if node['foo']['config']['vips'][site]
  file cfile do
    action :delete
  end
end

end

And actually since all this is doing is using built-in resources I'd actually
use notifying_action so I didn't have to deal with notifications myself... but
that's orthogonal to this point.

The vip.conf.erb template would now look like:

<% config = node['foo']['config']['vips'][@site].to_hash %>
[<%= @site %>]
<% config.each do |key, val| %>
<%= key %> = <%= val %>
<% end %>

And the foo.conf.erb template would now look like:

<% config = node['foo']'config']['global'].to_hash %>
<% config.each do |key, val| %>
<%= key %> = <%= val %>
<% end %>

There's a a few things to note here: variables to templates are evil and
should be avoided at all costs... rather use the node object from within your
template. In this case, I passed in static data that told the template where
to look in the node object and that's it.

The other use-case we have for LWRPs is in the relatively-rare case where we
have cookbooks that allow you to both enable or disable something. We do
this with autofs, so our fb_autofs cookbook's default recipe is included
everywhere and looks like this:

fb_autofs 'enable it' do
only_if { node['fb']['fb_autofs']['enable'] }
action :enable
end
fb_autofs 'disable it' do
not_if { node['fb']['fb_autofs']['enable'] }
action :disable
end

Since 'lazy' doesn't work on actions, we need two invocations of the LWRP
using only_if and not_if which are always lazily-evaluated.

The final use-case at FB for LWRPs is basically just a modification of the
first one in this email. When we want to deal with things like hardware or
mounts or things that require a complicated set of running command, looking at
output, running more commands. But even in these cases it works like the first
example above - the LWRP is just scheduled by it's own cookbook and it's
purely configured by tweaking the node object and it looks at that at
runtime inside the LWRP to decide what to do. This is basically just
abstracting what would otherwise be a long ruby_block or some such with
bunches of Mixlib::ShellOut calls.

--
Phil Dibowitz phil@ipom.com
Open Source software and tech docs Insanity Palace of Metallica
http://www.phildev.net/ http://www.ipom.com/

"Be who you are and say what you feel, because those who mind don't matter
and those who matter don't mind."

Dr. Seuss

Sean_OMeara1 · September 7, 2014, 3:15am

We're in violent agreement here.

The trick to re-usability is to separate what the consumer needs
control over (usually a configuration file) from what they don't care
about (directory permissions, package installations, etc).

The http_service provider(s) handle getting the service up and
listening, then completely offloads the meaningful configuration
details to the user. The resource does write a small configuration
file, which tweaks a very minimal amount of things (listen ports,
etc), but its main purpose in life is to include files from the conf.d
directory.

In the examples I found on the FB Github, the Phil has created an
"open ended" API that lets users encode a data structure in node
attributes. Then they're all consumed by a template resource. This is
a configuration-file-centric view of the world.

https://github.com/facebook/chef-utils/blob/master/cookbooks/fb_cron/recipes/default.rb#L60-L65

The other resource in the recipes are "hard wired" by the cookbook,
since they're not being driven by node attributes. The minimal amount
of cross-platform-ness represented in that example is already starting
to sprawl. There's a giant case statement at the top, and various
control flow logic around platform_family.

Most of the community cookbooks out there start out looking simple and
clean... when they're written for a single version of a single
platform. Then, over time, they descend into the pit of despair that
is case statements for package names, service names, init systems, and
filesystem paths. Driving all those details "open ended" will lead to

A Chef resource (implemented with the LWRP DSL or otherwise), creates
a "closure" around its subject.

This lets you everything the example recipe other than the template
resource, and shove it behind a wall of implementation detail. This
lets an "httpd_service" become cross platform while hiding all the
crazy platform details.

The user can then can drop off configuration via httpd_config, which
is just a thin wrapper around the template resource that calculates a
file path based on the instance name. The same attribute tricks can be
used around the contents of the template.

-s

On Sat, Sep 6, 2014 at 9:37 PM, Phil Dibowitz phil@ipom.com wrote:

On 09/06/2014 06:17 AM, Bráulio Bhavamitra wrote:

Thanks Phil for excelent discussion and examples of how to use node
attributes.

When LWRP are coded, how do you think both ways to configure can the
connected?

Well I think that LWRPs are often the wrong way to provide an API. Not because
LWRPs are themselves bad - they're awesome - but for the same reason we don't
use most of the standard built-in providers. We don't use 'cron' for example,
we built a template and provide a data-api as I described in the last email.
You can see this here:

https://github.com/facebook/chef-utils/tree/master/cookbooks/fb_cron

But there are times when an LWRP is what you need. The most common thing we
use LWRPs for is when the data we're pulling from the node object cannot be
put in a single file, and represents some dynamic set of files - and we won't
know what that set of files is until after everyone has munged the node
object. We avoid this wherever possible, because it makes system-level
idempotency more difficult, but sometimes you don't have a choice. So the
example here might be if the example software I was using required every vip
to be in a different file. Now again - often times it's how they set it up,
but it's not required. But lets just say it was absolutely required.

In this modified example the software requires /etc/foo/foo.conf and then each
site goes in a file like /etc/foo/vips/www.sample.com.conf. And lets assume,
just to make it more interesting that the main configuration doesn't take that
"[global]" header but the other ones do so they know the site name.

In this case I'd tweak the attributes slightly and put all the VIPs under
their own section:

default['foo']['config_file'] = '/etc/foo/foo.conf'
default['foo']['config']['global'] => {
'logdir' => '/var/log/foo',
'loglevel' => 'DEBUG',
}
default['foo']['config']['vips']['www.sample.com'] => {
'root' => '/var/foo/sample.com',
}

Then I'd the recipe to do:

template '/etc/foo/foo.conf' do
source 'foo.conf.erb'
owner 'root'
group 'root'
mode '0644'
end
foo_gen_site_files 'doit'

And that would call a 'gen_site_files' LWRP. Notice I'm not passing anything
to it and notice it's not meant for anyone else to call. Now inside the
provider I'd have:

action :run do
node['foo']['config']['vips'].keys do |url|
template "/etc/foo/vips/#{url}.conf" do
source 'vip.conf.erb'
user 'root'
group 'root'
mode '0644'
variables({'site' => url})
end
end
# Be idempotent - clean up crap we no longer own
# We have to do this because Foo is too dumb to allow
# all sites in a single file. I hate Foo.
Dir.glob('/etc/foo/vips/*.conf').each do |cfile|
  site = File.basename(cfile, '.conf')
  next if node['foo']['config']['vips'][site]
  file cfile do
    action :delete
  end
end
end

And actually since all this is doing is using built-in resources I'd actually
use notifying_action so I didn't have to deal with notifications myself... but
that's orthogonal to this point.

The vip.conf.erb template would now look like:

<% config = node['foo']['config']['vips'][@site].to_hash %>
[<%= @site %>]
<% config.each do |key, val| %>
<%= key %> = <%= val %>
<% end %>

And the foo.conf.erb template would now look like:

<% config = node['foo']'config']['global'].to_hash %>
<% config.each do |key, val| %>
<%= key %> = <%= val %>
<% end %>

There's a a few things to note here: variables to templates are evil and
should be avoided at all costs... rather use the node object from within your
template. In this case, I passed in static data that told the template where
to look in the node object and that's it.

The other use-case we have for LWRPs is in the relatively-rare case where we
have cookbooks that allow you to both enable or disable something. We do
this with autofs, so our fb_autofs cookbook's default recipe is included
everywhere and looks like this:

fb_autofs 'enable it' do
only_if { node['fb']['fb_autofs']['enable'] }
action :enable
end
fb_autofs 'disable it' do
not_if { node['fb']['fb_autofs']['enable'] }
action :disable
end

Since 'lazy' doesn't work on actions, we need two invocations of the LWRP
using only_if and not_if which are always lazily-evaluated.

The final use-case at FB for LWRPs is basically just a modification of the
first one in this email. When we want to deal with things like hardware or
mounts or things that require a complicated set of running command, looking at
output, running more commands. But even in these cases it works like the first
example above - the LWRP is just scheduled by it's own cookbook and it's
purely configured by tweaking the node object and it looks at that at
runtime inside the LWRP to decide what to do. This is basically just
abstracting what would otherwise be a long ruby_block or some such with
bunches of Mixlib::ShellOut calls.

--
Phil Dibowitz phil@ipom.com
Open Source software and tech docs Insanity Palace of Metallica
http://www.phildev.net/ http://www.ipom.com/

"Be who you are and say what you feel, because those who mind don't matter
and those who matter don't mind."

Dr. Seuss

Sean_OMeara1 · September 7, 2014, 3:17am

.... "lead madness" =)
-s

On Sat, Sep 6, 2014 at 11:15 PM, Sean OMeara someara@opscode.com wrote:

We're in violent agreement here.

The trick to re-usability is to separate what the consumer needs
control over (usually a configuration file) from what they don't care
about (directory permissions, package installations, etc).

The http_service provider(s) handle getting the service up and
listening, then completely offloads the meaningful configuration
details to the user. The resource does write a small configuration
file, which tweaks a very minimal amount of things (listen ports,
etc), but its main purpose in life is to include files from the conf.d
directory.

In the examples I found on the FB Github, the Phil has created an
"open ended" API that lets users encode a data structure in node
attributes. Then they're all consumed by a template resource. This is
a configuration-file-centric view of the world.

https://github.com/facebook/chef-utils/blob/master/cookbooks/fb_cron/recipes/default.rb#L60-L65

The other resource in the recipes are "hard wired" by the cookbook,
since they're not being driven by node attributes. The minimal amount
of cross-platform-ness represented in that example is already starting
to sprawl. There's a giant case statement at the top, and various
control flow logic around platform_family.

Most of the community cookbooks out there start out looking simple and
clean... when they're written for a single version of a single
platform. Then, over time, they descend into the pit of despair that
is case statements for package names, service names, init systems, and
filesystem paths. Driving all those details "open ended" will lead to

A Chef resource (implemented with the LWRP DSL or otherwise), creates
a "closure" around its subject.

This lets you everything the example recipe other than the template
resource, and shove it behind a wall of implementation detail. This
lets an "httpd_service" become cross platform while hiding all the
crazy platform details.

The user can then can drop off configuration via httpd_config, which
is just a thin wrapper around the template resource that calculates a
file path based on the instance name. The same attribute tricks can be
used around the contents of the template.

-s

On Sat, Sep 6, 2014 at 9:37 PM, Phil Dibowitz phil@ipom.com wrote:
On 09/06/2014 06:17 AM, Bráulio Bhavamitra wrote:

Thanks Phil for excelent discussion and examples of how to use node
attributes.

When LWRP are coded, how do you think both ways to configure can the
connected?

Well I think that LWRPs are often the wrong way to provide an API. Not because
LWRPs are themselves bad - they're awesome - but for the same reason we don't
use most of the standard built-in providers. We don't use 'cron' for example,
we built a template and provide a data-api as I described in the last email.
You can see this here:

https://github.com/facebook/chef-utils/tree/master/cookbooks/fb_cron

But there are times when an LWRP is what you need. The most common thing we
use LWRPs for is when the data we're pulling from the node object cannot be
put in a single file, and represents some dynamic set of files - and we won't
know what that set of files is until after everyone has munged the node
object. We avoid this wherever possible, because it makes system-level
idempotency more difficult, but sometimes you don't have a choice. So the
example here might be if the example software I was using required every vip
to be in a different file. Now again - often times it's how they set it up,
but it's not required. But lets just say it was absolutely required.

In this modified example the software requires /etc/foo/foo.conf and then each
site goes in a file like /etc/foo/vips/www.sample.com.conf. And lets assume,
just to make it more interesting that the main configuration doesn't take that
"[global]" header but the other ones do so they know the site name.

In this case I'd tweak the attributes slightly and put all the VIPs under
their own section:

default['foo']['config_file'] = '/etc/foo/foo.conf'
default['foo']['config']['global'] => {
'logdir' => '/var/log/foo',
'loglevel' => 'DEBUG',
}
default['foo']['config']['vips']['www.sample.com'] => {
'root' => '/var/foo/sample.com',
}

Then I'd the recipe to do:

template '/etc/foo/foo.conf' do
source 'foo.conf.erb'
owner 'root'
group 'root'
mode '0644'
end
foo_gen_site_files 'doit'

And that would call a 'gen_site_files' LWRP. Notice I'm not passing anything
to it and notice it's not meant for anyone else to call. Now inside the
provider I'd have:

action :run do
node['foo']['config']['vips'].keys do |url|
template "/etc/foo/vips/#{url}.conf" do
source 'vip.conf.erb'
user 'root'
group 'root'
mode '0644'
variables({'site' => url})
end
end
# Be idempotent - clean up crap we no longer own
# We have to do this because Foo is too dumb to allow
# all sites in a single file. I hate Foo.
Dir.glob('/etc/foo/vips/*.conf').each do |cfile|
  site = File.basename(cfile, '.conf')
  next if node['foo']['config']['vips'][site]
  file cfile do
    action :delete
  end
end
end

And actually since all this is doing is using built-in resources I'd actually
use notifying_action so I didn't have to deal with notifications myself... but
that's orthogonal to this point.

The vip.conf.erb template would now look like:

<% config = node['foo']['config']['vips'][@site].to_hash %>
[<%= @site %>]
<% config.each do |key, val| %>
<%= key %> = <%= val %>
<% end %>

And the foo.conf.erb template would now look like:

<% config = node['foo']'config']['global'].to_hash %>
<% config.each do |key, val| %>
<%= key %> = <%= val %>
<% end %>

There's a a few things to note here: variables to templates are evil and
should be avoided at all costs... rather use the node object from within your
template. In this case, I passed in static data that told the template where
to look in the node object and that's it.

The other use-case we have for LWRPs is in the relatively-rare case where we
have cookbooks that allow you to both enable or disable something. We do
this with autofs, so our fb_autofs cookbook's default recipe is included
everywhere and looks like this:

fb_autofs 'enable it' do
only_if { node['fb']['fb_autofs']['enable'] }
action :enable
end
fb_autofs 'disable it' do
not_if { node['fb']['fb_autofs']['enable'] }
action :disable
end

Since 'lazy' doesn't work on actions, we need two invocations of the LWRP
using only_if and not_if which are always lazily-evaluated.

The final use-case at FB for LWRPs is basically just a modification of the
first one in this email. When we want to deal with things like hardware or
mounts or things that require a complicated set of running command, looking at
output, running more commands. But even in these cases it works like the first
example above - the LWRP is just scheduled by it's own cookbook and it's
purely configured by tweaking the node object and it looks at that at
runtime inside the LWRP to decide what to do. This is basically just
abstracting what would otherwise be a long ruby_block or some such with
bunches of Mixlib::ShellOut calls.

--
Phil Dibowitz phil@ipom.com
Open Source software and tech docs Insanity Palace of Metallica
http://www.phildev.net/ http://www.ipom.com/

"Be who you are and say what you feel, because those who mind don't matter
and those who matter don't mind."

Dr. Seuss

brauliobo · September 7, 2014, 4:02am

I really like the design of fb_cron! It is simple and very powerful in
terms of what you can achieve with a simple hash (in JSON).

Also, I really agree that LWRP should be used when more complicated logic
is necessary, and so ruby code can really help.

Thanks for sharing some of the usecase of Facebook, a very important use
case of Chef.

But after this discussion, I also "agreed" to configure everything first by
LWRP and then build a node attributes interface around it.

Sean, it is important to note that the cross-platform issue will rise and
will be complicated to handle in both models (configuration by node
attributes and by LWRP).

It is important to note that I'm not critizing the httpd cookbook, but the
trend to do everything by LWRP and ignore node attributes.

cheers,
bráulio

On Sat, Sep 6, 2014 at 10:37 PM, Phil Dibowitz phil@ipom.com wrote:

On 09/06/2014 06:17 AM, Bráulio Bhavamitra wrote:

Thanks Phil for excelent discussion and examples of how to use node
attributes.

When LWRP are coded, how do you think both ways to configure can the
connected?

Well I think that LWRPs are often the wrong way to provide an API. Not
because
LWRPs are themselves bad - they're awesome - but for the same reason we
don't
use most of the standard built-in providers. We don't use 'cron' for
example,
we built a template and provide a data-api as I described in the last
email.
You can see this here:

https://github.com/facebook/chef-utils/tree/master/cookbooks/fb_cron

But there are times when an LWRP is what you need. The most common thing we
use LWRPs for is when the data we're pulling from the node object cannot be
put in a single file, and represents some dynamic set of files - and we
won't
know what that set of files is until after everyone has munged the node
object. We avoid this wherever possible, because it makes system-level
idempotency more difficult, but sometimes you don't have a choice. So the
example here might be if the example software I was using required every
vip
to be in a different file. Now again - often times it's how they set it up,
but it's not required. But lets just say it was absolutely required.

In this modified example the software requires /etc/foo/foo.conf and then
each
site goes in a file like /etc/foo/vips/www.sample.com.conf. And lets
assume,
just to make it more interesting that the main configuration doesn't take
that
"[global]" header but the other ones do so they know the site name.

In this case I'd tweak the attributes slightly and put all the VIPs under
their own section:

default['foo']['config_file'] = '/etc/foo/foo.conf'
default['foo']['config']['global'] => {
'logdir' => '/var/log/foo',
'loglevel' => 'DEBUG',
}
default['foo']['config']['vips']['www.sample.com'] => {
'root' => '/var/foo/sample.com',
}

Then I'd the recipe to do:

template '/etc/foo/foo.conf' do
source 'foo.conf.erb'
owner 'root'
group 'root'
mode '0644'
end
foo_gen_site_files 'doit'

And that would call a 'gen_site_files' LWRP. Notice I'm not passing
anything
to it and notice it's not meant for anyone else to call. Now inside the
provider I'd have:

action :run do
node['foo']['config']['vips'].keys do |url|
template "/etc/foo/vips/#{url}.conf" do
source 'vip.conf.erb'
user 'root'
group 'root'
mode '0644'
variables({'site' => url})
end
end
# Be idempotent - clean up crap we no longer own
# We have to do this because Foo is too dumb to allow
# all sites in a single file. I hate Foo.
Dir.glob('/etc/foo/vips/*.conf').each do |cfile|
  site = File.basename(cfile, '.conf')
  next if node['foo']['config']['vips'][site]
  file cfile do
    action :delete
  end
end
end

And actually since all this is doing is using built-in resources I'd
actually
use notifying_action so I didn't have to deal with notifications myself...
but
that's orthogonal to this point.

The vip.conf.erb template would now look like:

<% config = node['foo']['config']['vips'][@site].to_hash %>
[<%= @site %>]
<% config.each do |key, val| %>
<%= key %> = <%= val %>
<% end %>

And the foo.conf.erb template would now look like:

<% config = node['foo']'config']['global'].to_hash %>
<% config.each do |key, val| %>
<%= key %> = <%= val %>
<% end %>

There's a a few things to note here: variables to templates are evil and
should be avoided at all costs... rather use the node object from within
your
template. In this case, I passed in static data that told the template
where
to look in the node object and that's it.

The other use-case we have for LWRPs is in the relatively-rare case where
we
have cookbooks that allow you to both enable or disable something. We do
this with autofs, so our fb_autofs cookbook's default recipe is included
everywhere and looks like this:

fb_autofs 'enable it' do
only_if { node['fb']['fb_autofs']['enable'] }
action :enable
end
fb_autofs 'disable it' do
not_if { node['fb']['fb_autofs']['enable'] }
action :disable
end

Since 'lazy' doesn't work on actions, we need two invocations of the LWRP
using only_if and not_if which are always lazily-evaluated.

The final use-case at FB for LWRPs is basically just a modification of the
first one in this email. When we want to deal with things like hardware or
mounts or things that require a complicated set of running command,
looking at
output, running more commands. But even in these cases it works like the
first
example above - the LWRP is just scheduled by it's own cookbook and it's
purely configured by tweaking the node object and it looks at that at
runtime inside the LWRP to decide what to do. This is basically just
abstracting what would otherwise be a long ruby_block or some such with
bunches of Mixlib::ShellOut calls.

--
Phil Dibowitz phil@ipom.com
Open Source software and tech docs Insanity Palace of Metallica
http://www.phildev.net/ http://www.ipom.com/

"Be who you are and say what you feel, because those who mind don't matter
and those who matter don't mind."

Dr. Seuss

--
"Lute pela sua ideologia. Seja um com sua ideologia. Viva pela sua
ideologia. Morra por sua ideologia" P.R. Sarkar

EITA - Educação, Informação e Tecnologias para Autogestão

"Paramapurusha é meu pai e Parama Prakriti é minha mãe. O universo é meu
lar e todos nós somos cidadãos deste cosmo. Este universo é a imaginação da
Mente Macrocósmica, e todas as entidades estão sendo criadas, preservadas e
destruídas nas fases de extroversão e introversão do fluxo imaginativo
cósmico. No âmbito pessoal, quando uma pessoa imagina algo em sua mente,
naquele momento, essa pessoa é a única proprietária daquilo que ela
imagina, e ninguém mais. Quando um ser humano criado mentalmente caminha
por um milharal também imaginado, a pessoa imaginada não é a propriedade
desse milharal, pois ele pertence ao indivíduo que o está imaginando. Este
universo foi criado na imaginação de Brahma, a Entidade Suprema, por isso
a propriedade deste universo é de Brahma, e não dos microcosmos que também
foram criados pela imaginação de Brahma. Nenhuma propriedade deste mundo,
mutável ou imutável, pertence a um indivíduo em particular; tudo é o
patrimônio comum de todos."
Restante do texto em

Ranjib · September 7, 2014, 7:23am

i dont like the attribute driven lwrps. they couple your code with
attribute. historically the customization via chef's attribute system has
painful. They are extremely powerful to create flatter hierarchies, but
recipes, wrapper recipes etc is a pain. This is partially due to chef's
elaborate attribute precedence level, and also having a single attribute
system to represent ohai's related data.
note. attributes are public read only data. that means any attribute you
define is available for reads to the entire fleet. This also means you are
storing more data in chef server. this also means all the nodes that
searches for this node will fetch more data (unless you use partial search)
which will take more time and cpu to decompress at the client side. And
finally remember each attribute applied to N nodes will increase this N
times.
Most of the lwrps that I have find easy to use, extend and maintain are
less heavy on attributes (user_account, apt_repository, sudo, ruby_build
are some of those) ...
I still prefer to use attributes, to customize the recipe driven logic,
that are not offered via lwrp, but include_recipe.
cheers
ranjib

On Sat, Sep 6, 2014 at 9:02 PM, Bráulio Bhavamitra brauliobo@gmail.com
wrote:

I really like the design of fb_cron! It is simple and very powerful in
terms of what you can achieve with a simple hash (in JSON).

Also, I really agree that LWRP should be used when more complicated logic
is necessary, and so ruby code can really help.

Thanks for sharing some of the usecase of Facebook, a very important use
case of Chef.

But after this discussion, I also "agreed" to configure everything first
by LWRP and then build a node attributes interface around it.

Sean, it is important to note that the cross-platform issue will rise and
will be complicated to handle in both models (configuration by node
attributes and by LWRP).

It is important to note that I'm not critizing the httpd cookbook, but the
trend to do everything by LWRP and ignore node attributes.

cheers,
bráulio

On Sat, Sep 6, 2014 at 10:37 PM, Phil Dibowitz phil@ipom.com wrote:
On 09/06/2014 06:17 AM, Bráulio Bhavamitra wrote:

Thanks Phil for excelent discussion and examples of how to use node
attributes.

When LWRP are coded, how do you think both ways to configure can the
connected?

Well I think that LWRPs are often the wrong way to provide an API. Not
because
LWRPs are themselves bad - they're awesome - but for the same reason we
don't
use most of the standard built-in providers. We don't use 'cron' for
example,
we built a template and provide a data-api as I described in the last
email.
You can see this here:

https://github.com/facebook/chef-utils/tree/master/cookbooks/fb_cron

But there are times when an LWRP is what you need. The most common thing
we
use LWRPs for is when the data we're pulling from the node object cannot
be
put in a single file, and represents some dynamic set of files - and we
won't
know what that set of files is until after everyone has munged the node
object. We avoid this wherever possible, because it makes system-level
idempotency more difficult, but sometimes you don't have a choice. So the
example here might be if the example software I was using required
every vip
to be in a different file. Now again - often times it's how they set it
up,
but it's not required. But lets just say it was absolutely required.

In this modified example the software requires /etc/foo/foo.conf and then
each
site goes in a file like /etc/foo/vips/www.sample.com.conf. And lets
assume,
just to make it more interesting that the main configuration doesn't take
that
"[global]" header but the other ones do so they know the site name.

In this case I'd tweak the attributes slightly and put all the VIPs under
their own section:

default['foo']['config_file'] = '/etc/foo/foo.conf'
default['foo']['config']['global'] => {
'logdir' => '/var/log/foo',
'loglevel' => 'DEBUG',
}
default['foo']['config']['vips']['www.sample.com'] => {
'root' => '/var/foo/sample.com',
}

Then I'd the recipe to do:

template '/etc/foo/foo.conf' do
source 'foo.conf.erb'
owner 'root'
group 'root'
mode '0644'
end
foo_gen_site_files 'doit'

And that would call a 'gen_site_files' LWRP. Notice I'm not passing
anything
to it and notice it's not meant for anyone else to call. Now inside the
provider I'd have:

action :run do
node['foo']['config']['vips'].keys do |url|
template "/etc/foo/vips/#{url}.conf" do
source 'vip.conf.erb'
user 'root'
group 'root'
mode '0644'
variables({'site' => url})
end
end
# Be idempotent - clean up crap we no longer own
# We have to do this because Foo is too dumb to allow
# all sites in a single file. I hate Foo.
Dir.glob('/etc/foo/vips/*.conf').each do |cfile|
  site = File.basename(cfile, '.conf')
  next if node['foo']['config']['vips'][site]
  file cfile do
    action :delete
  end
end
end

And actually since all this is doing is using built-in resources I'd
actually
use notifying_action so I didn't have to deal with notifications
myself... but
that's orthogonal to this point.

The vip.conf.erb template would now look like:

<% config = node['foo']['config']['vips'][@site].to_hash %>
[<%= @site %>]
<% config.each do |key, val| %>
<%= key %> = <%= val %>
<% end %>

And the foo.conf.erb template would now look like:

<% config = node['foo']'config']['global'].to_hash %>
<% config.each do |key, val| %>
<%= key %> = <%= val %>
<% end %>

There's a a few things to note here: variables to templates are evil and
should be avoided at all costs... rather use the node object from within
your
template. In this case, I passed in static data that told the template
where
to look in the node object and that's it.

The other use-case we have for LWRPs is in the relatively-rare case where
we
have cookbooks that allow you to both enable or disable something. We do
this with autofs, so our fb_autofs cookbook's default recipe is included
everywhere and looks like this:

fb_autofs 'enable it' do
only_if { node['fb']['fb_autofs']['enable'] }
action :enable
end
fb_autofs 'disable it' do
not_if { node['fb']['fb_autofs']['enable'] }
action :disable
end

Since 'lazy' doesn't work on actions, we need two invocations of the LWRP
using only_if and not_if which are always lazily-evaluated.

The final use-case at FB for LWRPs is basically just a modification of the
first one in this email. When we want to deal with things like hardware or
mounts or things that require a complicated set of running command,
looking at
output, running more commands. But even in these cases it works like the
first
example above - the LWRP is just scheduled by it's own cookbook and it's
purely configured by tweaking the node object and it looks at that at
runtime inside the LWRP to decide what to do. This is basically just
abstracting what would otherwise be a long ruby_block or some such with
bunches of Mixlib::ShellOut calls.

--
Phil Dibowitz phil@ipom.com
Open Source software and tech docs Insanity Palace of Metallica
http://www.phildev.net/ http://www.ipom.com/

"Be who you are and say what you feel, because those who mind don't matter
and those who matter don't mind."

Dr. Seuss
--
"Lute pela sua ideologia. Seja um com sua ideologia. Viva pela sua
ideologia. Morra por sua ideologia" P.R. Sarkar

EITA - Educação, Informação e Tecnologias para Autogestão
Blog - Bráulio Bhavamitra
http://eita.org.br

"Paramapurusha é meu pai e Parama Prakriti é minha mãe. O universo é meu
lar e todos nós somos cidadãos deste cosmo. Este universo é a imaginação da
Mente Macrocósmica, e todas as entidades estão sendo criadas, preservadas e
destruídas nas fases de extroversão e introversão do fluxo imaginativo
cósmico. No âmbito pessoal, quando uma pessoa imagina algo em sua mente,
naquele momento, essa pessoa é a única proprietária daquilo que ela
imagina, e ninguém mais. Quando um ser humano criado mentalmente caminha
por um milharal também imaginado, a pessoa imaginada não é a propriedade
desse milharal, pois ele pertence ao indivíduo que o está imaginando. Este
universo foi criado na imaginação de Brahma, a Entidade Suprema, por isso
a propriedade deste universo é de Brahma, e não dos microcosmos que também
foram criados pela imaginação de Brahma. Nenhuma propriedade deste mundo,
mutável ou imutável, pertence a um indivíduo em particular; tudo é o
patrimônio comum de todos."
Restante do texto em
A problemática de hoje em dia - Bráulio Bhavamitra

brauliobo · January 21, 2015, 12:06am

I've managed to bridge node attributes and LWRP in a generic manner, see

On Thu, Sep 18, 2014 at 1:42 PM, Phil Dibowitz phil@ipom.com wrote:

On 09/07/2014 12:23 AM, Ranjib Dey wrote:

i dont like the attribute driven lwrps. they couple your code with
attribute.
historically the customization via chef's attribute system has painful.
They
are extremely powerful to create flatter hierarchies, but recipes,
wrapper
recipes etc is a pain. This is partially due to chef's elaborate
attribute
precedence level, and also having a single attribute system to represent
ohai's related data.
note. attributes are public read only data. that means any attribute you
define is available for reads to the entire fleet.

That's not true at all. If I have fb_cron that defines some skeleton
attributes and then fb_httpd that adds some cronjobs, only the things
running
fb_httpd see those attributes... which makes sense, since they need those
crons.

This also means you are
storing more data in chef server.

If you chose to run it that way. You can choose what attributes get sent
back
to the Chef server - in Chef 11 via the client.rb, or in Chef 10 via
whitelist_node_attrs. For Facebook, that's no attributes, but for most
people,
you could just namespace all this data such that it doesn't get sent back
to
the Chef server.

--
Phil Dibowitz phil@ipom.com
Open Source software and tech docs Insanity Palace of Metallica
http://www.phildev.net/ http://www.ipom.com/

"Be who you are and say what you feel, because those who mind don't matter
and those who matter don't mind."

Dr. Seuss

--
"Lute pela sua ideologia. Seja um com sua ideologia. Viva pela sua
ideologia. Morra por sua ideologia" P.R. Sarkar

EITA - Educação, Informação e Tecnologias para Autogestão

"Paramapurusha é meu pai e Parama Prakriti é minha mãe. O universo é meu
lar e todos nós somos cidadãos deste cosmo. Este universo é a imaginação da
Mente Macrocósmica, e todas as entidades estão sendo criadas, preservadas e
destruídas nas fases de extroversão e introversão do fluxo imaginativo
cósmico. No âmbito pessoal, quando uma pessoa imagina algo em sua mente,
naquele momento, essa pessoa é a única proprietária daquilo que ela
imagina, e ninguém mais. Quando um ser humano criado mentalmente caminha
por um milharal também imaginado, a pessoa imaginada não é a propriedade
desse milharal, pois ele pertence ao indivíduo que o está imaginando. Este
universo foi criado na imaginação de Brahma, a Entidade Suprema, por isso
a propriedade deste universo é de Brahma, e não dos microcosmos que também
foram criados pela imaginação de Brahma. Nenhuma propriedade deste mundo,
mutável ou imutável, pertence a um indivíduo em particular; tudo é o
patrimônio comum de todos."
Restante do texto em

Phil_Dibowitz · September 18, 2014, 4:42pm

On 09/07/2014 12:23 AM, Ranjib Dey wrote:

i dont like the attribute driven lwrps. they couple your code with attribute.
historically the customization via chef's attribute system has painful. They
are extremely powerful to create flatter hierarchies, but recipes, wrapper
recipes etc is a pain. This is partially due to chef's elaborate attribute
precedence level, and also having a single attribute system to represent
ohai's related data.
note. attributes are public read only data. that means any attribute you
define is available for reads to the entire fleet.

That's not true at all. If I have fb_cron that defines some skeleton
attributes and then fb_httpd that adds some cronjobs, only the things running
fb_httpd see those attributes... which makes sense, since they need those crons.

This also means you are
storing more data in chef server.

If you chose to run it that way. You can choose what attributes get sent back
to the Chef server - in Chef 11 via the client.rb, or in Chef 10 via
whitelist_node_attrs. For Facebook, that's no attributes, but for most people,
you could just namespace all this data such that it doesn't get sent back to
the Chef server.

--
Phil Dibowitz phil@ipom.com
Open Source software and tech docs Insanity Palace of Metallica
http://www.phildev.net/ http://www.ipom.com/

"Be who you are and say what you feel, because those who mind don't matter
and those who matter don't mind."

Dr. Seuss

Topic		Replies	Views
RE: Re: cookbook conflict: httpd/php/mysql Chef Infra (archive)	1	329	December 27, 2014
Cookbook Releases - Unsent draft edition Chef Infra (archive)	1	313	July 9, 2013
Opscode Cookbook Releases 6/10 Chef Infra (archive)	0	346	June 10, 2013
Cookbook conflict: httpd/php/mysql Chef Infra (archive)	1	339	December 18, 2014
COOK Releases, Winter 2012 Edition Chef Infra (archive)	0	340	December 21, 2012

Occasionally you'll run into something whose config is so asinine that coming up with a generalized data structure to represent it doesn't seem feasible

Related topics

Occasionally you'll run into something whose config is so asinine that
coming
up with a generalized data structure to represent it doesn't seem feasible