Puppet+Hiera vs. Chef, or handling the occasional special snowflake server

Hello folks,
I’m a user of both Puppet and Chef, I’m more familiar with and prefer the
latter overall, but am required to use the former at work. Recently I was
toying with the idea of moving to Chef, although it’s probably not a
realistic option, but it raised a few interesting questions.
While Puppet and Chef do roughly the same thing, they do it in slightly
different ways, and Puppet has something in particular that Chef doesn’t
have, which is Hiera http://docs.puppetlabs.com/hiera/1/.

If you’re not familiar with Hiera, it’s a simple way to apply settings to
any number of your servers (from a single node to an entire infrastructure
of thousands), you define the hierarchy of settings yourself – in a sense,
it’s kind of like being able to customize Chef’s attribute precedence and
having a single environment attached to all your nodes. The link above
explains it a lot better than I can.

Anyway, I was wondering how one might deal with losing Hiera when moving
from Puppet to Chef. It seems that you could achieve similar results with a
mix of cookbooks, roles and environments, like everything else, but at the
cost of ending up with a lot more of them than you’d normally have.

I guess a different way I might ask this is:
Imagine I have a group of servers, all configured the same way (same run
list, same environment), and I need to change something on just one of
them. A common scenario in my environment is having to attach YourKit
https://www.yourkit.com/ to a JVM process.
With Puppet/Hiera, I’d just add a two-line YAML file with the same name as
that one server’s FQDN, and when I run Puppet, that YAML file would be read
at compile time and the config change applied without affecting the other
nodes.
With Chef, I’d have to modify the node’s run list or take its environment
file, make a copy of it, modify the copy and assign that copied environment
to the node instead of the normal env file, lest I affect the other servers
in the same group.

Hiera is also capable of aggregating data from multiple sources into a
single hash or array to be applied to a node during a Puppet agent run.

Those are things that I never missed for as long as I used Chef, but now
that I know they’re out there, I’m wondering how I would handle things
without Hiera.

So in short, I was wondering if anybody has thoughts on the matter.
Obviously it helps if you have some experience with both tools. I’m far
from being a Chef expert so it’s possible I’m missing something. Any input
is appreciated. Thanks!
Fabien

But in Chef you have an object which represents each node, so why not just
edit that if you want to alter individual nodes?

Perhaps I’m not understanding

On 6 October 2015 at 13:18, Fabien Delpierre fabien.delpierre@gmail.com
wrote:

Hello folks,
I’m a user of both Puppet and Chef, I’m more familiar with and prefer the
latter overall, but am required to use the former at work. Recently I was
toying with the idea of moving to Chef, although it’s probably not a
realistic option, but it raised a few interesting questions.
While Puppet and Chef do roughly the same thing, they do it in slightly
different ways, and Puppet has something in particular that Chef doesn’t
have, which is Hiera http://docs.puppetlabs.com/hiera/1/.

If you’re not familiar with Hiera, it’s a simple way to apply settings to
any number of your servers (from a single node to an entire infrastructure
of thousands), you define the hierarchy of settings yourself – in a sense,
it’s kind of like being able to customize Chef’s attribute precedence and
having a single environment attached to all your nodes. The link above
explains it a lot better than I can.

Anyway, I was wondering how one might deal with losing Hiera when moving
from Puppet to Chef. It seems that you could achieve similar results with a
mix of cookbooks, roles and environments, like everything else, but at the
cost of ending up with a lot more of them than you’d normally have.

I guess a different way I might ask this is:
Imagine I have a group of servers, all configured the same way (same run
list, same environment), and I need to change something on just one of
them. A common scenario in my environment is having to attach YourKit
https://www.yourkit.com/ to a JVM process.
With Puppet/Hiera, I’d just add a two-line YAML file with the same name as
that one server’s FQDN, and when I run Puppet, that YAML file would be read
at compile time and the config change applied without affecting the other
nodes.
With Chef, I’d have to modify the node’s run list or take its environment
file, make a copy of it, modify the copy and assign that copied environment
to the node instead of the normal env file, lest I affect the other servers
in the same group.

Hiera is also capable of aggregating data from multiple sources into a
single hash or array to be applied to a node during a Puppet agent run.

Those are things that I never missed for as long as I used Chef, but now
that I know they’re out there, I’m wondering how I would handle things
without Hiera.

So in short, I was wondering if anybody has thoughts on the matter.
Obviously it helps if you have some experience with both tools. I’m far
from being a Chef expert so it’s possible I’m missing something. Any input
is appreciated. Thanks!
Fabien

you can use tags (or normal attribute precedence level, which is how
tags are implemented) to assign attributes directly to nodes for one-off
use cases.

On 10/06/2015 01:22 PM, Yoshi Spendiff wrote:

But in Chef you have an object which represents each node, so why not
just edit that if you want to alter individual nodes?

Perhaps I’m not understanding

On 6 October 2015 at 13:18, Fabien Delpierre
<fabien.delpierre@gmail.com mailto:fabien.delpierre@gmail.com> wrote:

Hello folks,
I'm a user of both Puppet and Chef, I'm more familiar with and
prefer the latter overall, but am required to use the former at
work. Recently I was toying with the idea of moving to Chef,
although it's probably not a realistic option, but it raised a few
interesting questions.
While Puppet and Chef do roughly the same thing, they do it in
slightly different ways, and Puppet has something in particular
that Chef doesn't have, which is Hiera
<http://docs.puppetlabs.com/hiera/1/>.

If you're not familiar with Hiera, it's a simple way to apply
settings to any number of your servers (from a single node to an
entire infrastructure of thousands), you define the hierarchy of
settings yourself -- in a sense, it's kind of like being able to
customize Chef's attribute precedence and having a single
environment attached to all your nodes. The link above explains it
a lot better than I can.

Anyway, I was wondering how one might deal with losing Hiera when
moving from Puppet to Chef. It seems that you could achieve
similar results with a mix of cookbooks, roles and environments,
like everything else, but at the cost of ending up with a lot more
of them than you'd normally have.

I guess a different way I might ask this is:
Imagine I have a group of servers, all configured the same way
(same run list, same environment), and I need to change something
on just one of them. A common scenario in my environment is having
to attach YourKit <https://www.yourkit.com/> to a JVM process.
With Puppet/Hiera, I'd just add a two-line YAML file with the same
name as that one server's FQDN, and when I run Puppet, that YAML
file would be read at compile time and the config change applied
without affecting the other nodes.
With Chef, I'd have to modify the node's run list or take its
environment file, make a copy of it, modify the copy and assign
that copied environment to the node instead of the normal env
file, lest I affect the other servers in the same group.

Hiera is also capable of aggregating data from multiple sources
into a single hash or array to be applied to a node during a
Puppet agent run.

Those are things that I never missed for as long as I used Chef,
but now that I know they're out there, I'm wondering how I would
handle things without Hiera.

So in short, I was wondering if anybody has thoughts on the
matter. Obviously it helps if you have some experience with both
tools. I'm far from being a Chef expert so it's possible I'm
missing something. Any input is appreciated. Thanks!
Fabien

Lamont,
Your mention of tags is interesting. I’d actually never heard of them and
it seems like there’s little documentation on the topic.
Could you share some examples?

On Tue, Oct 6, 2015 at 6:15 PM, Lamont Granquist lamont@chef.io wrote:

you can use tags (or normal attribute precedence level, which is how tags
are implemented) to assign attributes directly to nodes for one-off use
cases.

On 10/06/2015 01:22 PM, Yoshi Spendiff wrote:

But in Chef you have an object which represents each node, so why not just
edit that if you want to alter individual nodes?

Perhaps I’m not understanding

On 6 October 2015 at 13:18, Fabien Delpierre fabien.delpierre@gmail.com
wrote:

Hello folks,
I’m a user of both Puppet and Chef, I’m more familiar with and prefer the
latter overall, but am required to use the former at work. Recently I was
toying with the idea of moving to Chef, although it’s probably not a
realistic option, but it raised a few interesting questions.
While Puppet and Chef do roughly the same thing, they do it in slightly
different ways, and Puppet has something in particular that Chef doesn’t
have, which is Hiera http://docs.puppetlabs.com/hiera/1/.

If you’re not familiar with Hiera, it’s a simple way to apply settings to
any number of your servers (from a single node to an entire infrastructure
of thousands), you define the hierarchy of settings yourself – in a sense,
it’s kind of like being able to customize Chef’s attribute precedence and
having a single environment attached to all your nodes. The link above
explains it a lot better than I can.

Anyway, I was wondering how one might deal with losing Hiera when moving
from Puppet to Chef. It seems that you could achieve similar results with a
mix of cookbooks, roles and environments, like everything else, but at the
cost of ending up with a lot more of them than you’d normally have.

I guess a different way I might ask this is:
Imagine I have a group of servers, all configured the same way (same run
list, same environment), and I need to change something on just one of
them. A common scenario in my environment is having to attach YourKit
https://www.yourkit.com/ to a JVM process.
With Puppet/Hiera, I’d just add a two-line YAML file with the same name
as that one server’s FQDN, and when I run Puppet, that YAML file would be
read at compile time and the config change applied without affecting the
other nodes.
With Chef, I’d have to modify the node’s run list or take its environment
file, make a copy of it, modify the copy and assign that copied environment
to the node instead of the normal env file, lest I affect the other servers
in the same group.

Hiera is also capable of aggregating data from multiple sources into a
single hash or array to be applied to a node during a Puppet agent run.

Those are things that I never missed for as long as I used Chef, but now
that I know they’re out there, I’m wondering how I would handle things
without Hiera.

So in short, I was wondering if anybody has thoughts on the matter.
Obviously it helps if you have some experience with both tools. I’m far
from being a Chef expert so it’s possible I’m missing something. Any input
is appreciated. Thanks!
Fabien

On Oct 6, 2015, at 1:18 PM, Fabien Delpierre fabien.delpierre@gmail.com wrote:

Hello folks,
I’m a user of both Puppet and Chef, I’m more familiar with and prefer the latter overall, but am required to use the former at work. Recently I was toying with the idea of moving to Chef, although it’s probably not a realistic option, but it raised a few interesting questions.
While Puppet and Chef do roughly the same thing, they do it in slightly different ways, and Puppet has something in particular that Chef doesn’t have, which is Hiera.

If you’re not familiar with Hiera, it’s a simple way to apply settings to any number of your servers (from a single node to an entire infrastructure of thousands), you define the hierarchy of settings yourself – in a sense, it’s kind of like being able to customize Chef’s attribute precedence and having a single environment attached to all your nodes. The link above explains it a lot better than I can.

Anyway, I was wondering how one might deal with losing Hiera when moving from Puppet to Chef. It seems that you could achieve similar results with a mix of cookbooks, roles and environments, like everything else, but at the cost of ending up with a lot more of them than you’d normally have.

I guess a different way I might ask this is:
Imagine I have a group of servers, all configured the same way (same run list, same environment), and I need to change something on just one of them. A common scenario in my environment is having to attach YourKit to a JVM process.
With Puppet/Hiera, I’d just add a two-line YAML file with the same name as that one server’s FQDN, and when I run Puppet, that YAML file would be read at compile time and the config change applied without affecting the other nodes.
With Chef, I’d have to modify the node’s run list or take its environment file, make a copy of it, modify the copy and assign that copied environment to the node instead of the normal env file, lest I affect the other servers in the same group.

Hiera is also capable of aggregating data from multiple sources into a single hash or array to be applied to a node during a Puppet agent run.

Those are things that I never missed for as long as I used Chef, but now that I know they’re out there, I’m wondering how I would handle things without Hiera.

So in short, I was wondering if anybody has thoughts on the matter. Obviously it helps if you have some experience with both tools. I’m far from being a Chef expert so it’s possible I’m missing something. Any input is appreciated. Thanks!
Fabien

Generally you would try to use an existing grouping like a chef environment, role, or policy (if you are using any of those). For single node edits you would do it directly via knife node edit. For things that are between the scope of a role and environment, https://github.com/poise/poise-appenv can be a solution.

–Noah

Does this help?

https://docs.chef.io/dsl_recipe.html#tag-tagged-untag

  • Julian

On Tue, Oct 6, 2015 at 10:13 PM, Fabien Delpierre
fabien.delpierre@gmail.com wrote:

Lamont,
Your mention of tags is interesting. I’d actually never heard of them and it
seems like there’s little documentation on the topic.
Could you share some examples?

On Tue, Oct 6, 2015 at 6:15 PM, Lamont Granquist lamont@chef.io wrote:

you can use tags (or normal attribute precedence level, which is how tags
are implemented) to assign attributes directly to nodes for one-off use
cases.

On 10/06/2015 01:22 PM, Yoshi Spendiff wrote:

But in Chef you have an object which represents each node, so why not just
edit that if you want to alter individual nodes?

Perhaps I’m not understanding

On 6 October 2015 at 13:18, Fabien Delpierre fabien.delpierre@gmail.com
wrote:

Hello folks,
I’m a user of both Puppet and Chef, I’m more familiar with and prefer the
latter overall, but am required to use the former at work. Recently I was
toying with the idea of moving to Chef, although it’s probably not a
realistic option, but it raised a few interesting questions.
While Puppet and Chef do roughly the same thing, they do it in slightly
different ways, and Puppet has something in particular that Chef doesn’t
have, which is Hiera.

If you’re not familiar with Hiera, it’s a simple way to apply settings to
any number of your servers (from a single node to an entire infrastructure
of thousands), you define the hierarchy of settings yourself – in a sense,
it’s kind of like being able to customize Chef’s attribute precedence and
having a single environment attached to all your nodes. The link above
explains it a lot better than I can.

Anyway, I was wondering how one might deal with losing Hiera when moving
from Puppet to Chef. It seems that you could achieve similar results with a
mix of cookbooks, roles and environments, like everything else, but at the
cost of ending up with a lot more of them than you’d normally have.

I guess a different way I might ask this is:
Imagine I have a group of servers, all configured the same way (same run
list, same environment), and I need to change something on just one of them.
A common scenario in my environment is having to attach YourKit to a JVM
process.
With Puppet/Hiera, I’d just add a two-line YAML file with the same name
as that one server’s FQDN, and when I run Puppet, that YAML file would be
read at compile time and the config change applied without affecting the
other nodes.
With Chef, I’d have to modify the node’s run list or take its environment
file, make a copy of it, modify the copy and assign that copied environment
to the node instead of the normal env file, lest I affect the other servers
in the same group.

Hiera is also capable of aggregating data from multiple sources into a
single hash or array to be applied to a node during a Puppet agent run.

Those are things that I never missed for as long as I used Chef, but now
that I know they’re out there, I’m wondering how I would handle things
without Hiera.

So in short, I was wondering if anybody has thoughts on the matter.
Obviously it helps if you have some experience with both tools. I’m far from
being a Chef expert so it’s possible I’m missing something. Any input is
appreciated. Thanks!
Fabien


[ Julian C. Dunn jdunn@aquezada.com * Sorry, I’m ]
[ WWW: http://www.aquezada.com/staff/julian * only Web 1.0 ]
[ gopher://sdf.org/1/users/keymaker/ * compliant! ]
[ PGP: 91B3 7A9D 683C 7C16 715F 442C 6065 D533 FDC2 05B9 ]

“And if you have n nodes that share the same run list and environment, you
can only alter one of those n nodes by altering its run list or environment
– otherwise, the change would apply to all n nodes, no?”

No. Nodes can receive attributes by having a role with attributes in a run
list, in which case attributes are used from the role and/or by specifying
an environment when Chef is run, in which case attributes are used from the
environment. You can also set them in a recipe, which applies the
attributes to the node object itself. Finally you can just set attributes
directly against a node object by editing the node object directly in Chef
server, using something like knife node edit.

As an example I have a recipe which installs a Newrelic agent if a certain
attribute is set to true. In the production Chef environment I set the
attribute so that all production nodes have this. If I want to use newrelic
for a machine outside of production I can set that attribute on the
individual node object instead, which could be in any environment.

In your case you would put the same data from the YAML you need in a
specific attribute for the node you are altering (in JSON), and it would
change that node only.

On 6 October 2015 at 20:27, Julian C. Dunn jdunn@aquezada.com wrote:

Does this help?

https://docs.chef.io/dsl_recipe.html#tag-tagged-untag

  • Julian

On Tue, Oct 6, 2015 at 10:13 PM, Fabien Delpierre
fabien.delpierre@gmail.com wrote:

Lamont,
Your mention of tags is interesting. I’d actually never heard of them
and it
seems like there’s little documentation on the topic.
Could you share some examples?

On Tue, Oct 6, 2015 at 6:15 PM, Lamont Granquist lamont@chef.io wrote:

you can use tags (or normal attribute precedence level, which is how
tags

are implemented) to assign attributes directly to nodes for one-off use
cases.

On 10/06/2015 01:22 PM, Yoshi Spendiff wrote:

But in Chef you have an object which represents each node, so why not
just

edit that if you want to alter individual nodes?

Perhaps I’m not understanding

On 6 October 2015 at 13:18, Fabien Delpierre <
fabien.delpierre@gmail.com>

wrote:

Hello folks,
I’m a user of both Puppet and Chef, I’m more familiar with and prefer
the

latter overall, but am required to use the former at work. Recently I
was

toying with the idea of moving to Chef, although it’s probably not a
realistic option, but it raised a few interesting questions.
While Puppet and Chef do roughly the same thing, they do it in slightly
different ways, and Puppet has something in particular that Chef
doesn’t

have, which is Hiera.

If you’re not familiar with Hiera, it’s a simple way to apply settings
to

any number of your servers (from a single node to an entire
infrastructure

of thousands), you define the hierarchy of settings yourself – in a
sense,

it’s kind of like being able to customize Chef’s attribute precedence
and

having a single environment attached to all your nodes. The link above
explains it a lot better than I can.

Anyway, I was wondering how one might deal with losing Hiera when
moving

from Puppet to Chef. It seems that you could achieve similar results
with a

mix of cookbooks, roles and environments, like everything else, but at
the

cost of ending up with a lot more of them than you’d normally have.

I guess a different way I might ask this is:
Imagine I have a group of servers, all configured the same way (same
run

list, same environment), and I need to change something on just one of
them.

A common scenario in my environment is having to attach YourKit to a
JVM

process.
With Puppet/Hiera, I’d just add a two-line YAML file with the same name
as that one server’s FQDN, and when I run Puppet, that YAML file would
be

read at compile time and the config change applied without affecting
the

other nodes.
With Chef, I’d have to modify the node’s run list or take its
environment

file, make a copy of it, modify the copy and assign that copied
environment

to the node instead of the normal env file, lest I affect the other
servers

in the same group.

Hiera is also capable of aggregating data from multiple sources into a
single hash or array to be applied to a node during a Puppet agent run.

Those are things that I never missed for as long as I used Chef, but
now

that I know they’re out there, I’m wondering how I would handle things
without Hiera.

So in short, I was wondering if anybody has thoughts on the matter.
Obviously it helps if you have some experience with both tools. I’m
far from

being a Chef expert so it’s possible I’m missing something. Any input
is

appreciated. Thanks!
Fabien


[ Julian C. Dunn jdunn@aquezada.com * Sorry, I’m ]
[ WWW: http://www.aquezada.com/staff/julian * only Web 1.0 ]
[ gopher://sdf.org/1/users/keymaker/ * compliant! ]
[ PGP: 91B3 7A9D 683C 7C16 715F 442C 6065 D533 FDC2 05B9 ]

Generally you would try to use an existing grouping like a chef
environment, role, or policy (if you are using any of those). For single
node edits you would do it directly via knife node edit. For things that
are between the scope of a role and environment,
https://github.com/poise/poise-appenv can be a solution.

Good to know about knife node edit. Wouldn’t any change be overridden at
the next chef-client run, though?
I guess I ought to look into policies, too. I’ve never used them but I’m
not finding much documentation about them. Are we talking about the new-ish
Policyfile feature? That I can find documentation about.

Does this help?

https://docs.chef.io/dsl_recipe.html#tag-tagged-untag

It’ll get me started at least, thanks!

No. Nodes can receive attributes by having a role with attributes in a run

list, in which case attributes are used from the role and/or by specifying
an environment when Chef is run, in which case attributes are used from the
environment. You can also set them in a recipe, which applies the
attributes to the node object itself.

I was including that when I mentioned “editing the run list” – of course
that works but if I have multiple nodes that use the same set of
roles/recipes and I edit one of those roles or recipes, all of them will be
affected, which I may not want.

knife node edit does seem like it might help, but I’m concerned that the
change wouldn’t persist past a chef-client run.

As an example I have a recipe which installs a Newrelic agent if a certain

attribute is set to true. In the production Chef environment I set the
attribute so that all production nodes have this. If I want to use newrelic
for a machine outside of production I can set that attribute on the
individual node object instead, which could be in any environment.

So that makes sense if you have a recipe that’s designed to look for that
attribute, but that still seems like it wouldn’t accomplish what one might
do with Puppet’s Hiera.

To use a concrete example, again with attaching YourKit to a JVM process –
like I mentioned, Hiera can aggregate multiple hashes or arrays spread
between multiple sources into a single attribute.
For example, in one YAML file I could have:
foo::java_opts: "-Xms2g -Xmx2g"
And in another:
foo::java_opts: "-agentpath:/opt/yjp/bin/linux-x86-64/libyjpagent.so"
Then I could invoke this foo::java_opts in a recipe or a template (e.g. for
an init script) and Hiera would combine them into one so it produces an
init script might read: java -Xms2g -Xmx2g -agentpath:
/opt/yjp/bin/linux-x86-64/libyjpagent.so
When I’m done with YourKit, I can just delete the YAML file that contains
those bits and I never have to touch the nodes or the recipe since it only
looks for what’s in that java_opts attribute and spits it back out.

While so far it doesn’t seem like this would be possible in Chef, it does
seem like tags would help, if accounted for within a recipe, e.g.:
node.default[‘foo’][‘bar’] = false

if tagged?(‘bar’)
ruby_block ‘enable_bar’ do
block do
node[‘foo’][‘bar’] = true
end
end
end

if node[‘foo’][‘bar’] == true
(do unspeakable things)
end

On Tue, Oct 6, 2015 at 10:23 PM, Noah Kantrowitz noah@coderanger.net
wrote:

On Oct 6, 2015, at 1:18 PM, Fabien Delpierre fabien.delpierre@gmail.com
wrote:

Hello folks,
I’m a user of both Puppet and Chef, I’m more familiar with and prefer
the latter overall, but am required to use the former at work. Recently I
was toying with the idea of moving to Chef, although it’s probably not a
realistic option, but it raised a few interesting questions.
While Puppet and Chef do roughly the same thing, they do it in slightly
different ways, and Puppet has something in particular that Chef doesn’t
have, which is Hiera.

If you’re not familiar with Hiera, it’s a simple way to apply settings
to any number of your servers (from a single node to an entire
infrastructure of thousands), you define the hierarchy of settings yourself
– in a sense, it’s kind of like being able to customize Chef’s attribute
precedence and having a single environment attached to all your nodes. The
link above explains it a lot better than I can.

Anyway, I was wondering how one might deal with losing Hiera when moving
from Puppet to Chef. It seems that you could achieve similar results with a
mix of cookbooks, roles and environments, like everything else, but at the
cost of ending up with a lot more of them than you’d normally have.

I guess a different way I might ask this is:
Imagine I have a group of servers, all configured the same way (same run
list, same environment), and I need to change something on just one of
them. A common scenario in my environment is having to attach YourKit to a
JVM process.
With Puppet/Hiera, I’d just add a two-line YAML file with the same name
as that one server’s FQDN, and when I run Puppet, that YAML file would be
read at compile time and the config change applied without affecting the
other nodes.
With Chef, I’d have to modify the node’s run list or take its
environment file, make a copy of it, modify the copy and assign that copied
environment to the node instead of the normal env file, lest I affect the
other servers in the same group.

Hiera is also capable of aggregating data from multiple sources into a
single hash or array to be applied to a node during a Puppet agent run.

Those are things that I never missed for as long as I used Chef, but now
that I know they’re out there, I’m wondering how I would handle things
without Hiera.

So in short, I was wondering if anybody has thoughts on the matter.
Obviously it helps if you have some experience with both tools. I’m far
from being a Chef expert so it’s possible I’m missing something. Any input
is appreciated. Thanks!
Fabien

Generally you would try to use an existing grouping like a chef
environment, role, or policy (if you are using any of those). For single
node edits you would do it directly via knife node edit. For things that
are between the scope of a role and environment,
https://github.com/poise/poise-appenv can be a solution.

–Noah

On Wednesday, October 7, 2015 at 6:36 AM, Fabien Delpierre wrote:

Generally you would try to use an existing grouping like a chef environment, role, or policy (if you are using any of those). For single node edits you would do it directly via knife node edit. For things that are between the scope of a role and environment, https://github.com/poise/poise-appenv can be a solution.

Good to know about knife node edit. Wouldn’t any change be overridden at the next chef-client run, though?
I guess I ought to look into policies, too. I’ve never used them but I’m not finding much documentation about them. Are we talking about the new-ish Policyfile feature? That I can find documentation about.

Attributes at the “normal” level belong to the node and are NOT wiped for each run. All of the other levels belong to an external thing (role, environment, policyfile) so they are wiped and recreated.

And yeah, policies means policyfiles. We’re still hammering out the exact official terminology.

I was including that when I mentioned “editing the run list” – of course that works but if I have multiple nodes that use the same set of roles/recipes and I edit one of those roles or recipes, all of them will be affected, which I may not want.

knife node edit does seem like it might help, but I’m concerned that the change wouldn’t persist past a chef-client run.
knife node edit by default will only show you things that will be persisted across chef runs: run_list, environment, “normal” attributes.

As an example I have a recipe which installs a Newrelic agent if a certain attribute is set to true. In the production Chef environment I set the attribute so that all production nodes have this. If I want to use newrelic for a machine outside of production I can set that attribute on the individual node object instead, which could be in any environment.

So that makes sense if you have a recipe that’s designed to look for that attribute, but that still seems like it wouldn’t accomplish what one might do with Puppet’s Hiera.

To use a concrete example, again with attaching YourKit to a JVM process – like I mentioned, Hiera can aggregate multiple hashes or arrays spread between multiple sources into a single attribute.
This is exactly what the chef attribute precedence system does. My understanding is Hiera was written in response to chef’s attributes system, though as a second system it has the benefit of learning from our limitations so it might be more flexible in some ways. In particular Chef has only one attribute level where you can set node-specific values, but these get merged with cookbook/role/environment attributes. Arrays are tricky since sometimes you need them to merge and sometimes override, I would recommend sticking to hashes and then join things yourself.

For example, in one YAML file I could have:
foo::java_opts: "-Xms2g -Xmx2g"
And in another:
foo::java_opts: "-agentpath:/opt/yjp/bin/linux-x86-64/libyjpagent.so"
Then I could invoke this foo::java_opts in a recipe or a template (e.g. for an init script) and Hiera would combine them into one so it produces an init script might read: java -Xms2g -Xmx2g -agentpath:/opt/yjp/bin/linux-x86-64/libyjpagent.so
When I’m done with YourKit, I can just delete the YAML file that contains those bits and I never have to touch the nodes or the recipe since it only looks for what’s in that java_opts attribute and spits it back out.

While so far it doesn’t seem like this would be possible in Chef, it does seem like tags would help, if accounted for within a recipe, e.g.:
node.default[‘foo’][‘bar’] = false

if tagged?(‘bar’)
ruby_block ‘enable_bar’ do
block do
node[‘foo’][‘bar’] = true
end
end
end

if node[‘foo’][‘bar’] == true
(do unspeakable things)
end

My recommendation would be to make the snowflake case part of your cookbook for the non-snowflake case. That way it’s always easy to find out what your system will look like for both cases, and your snowflake configs will be tracked in your git history just like the non-snowflake.


Daniel DeLeo

Attributes on the node object are only overwritten in the Chef run if there
is something in the recipe code which specifically alters that attribute.
The node object is both the desired state and current state of the object
(although there’s future work to split that planned I believe).

So for example if you directly edit a node and set the following attributes

{
“normal”: {
“myattributes”: {
“foo”: true,
“bar”: true
}
}
}

Then they become available to recipe code under node[‘myattributes’][‘foo’]
and node[‘myattributes’][‘bar’]. If the recipe code specificies something
like this:
node.set[‘myattributes’][‘bar’] = false then at the end of the Chef run the
chef-client will upload any updated node state to the Chef server, which
would change the attributes to this:

{
“normal”: {
“myattributes”: {
“foo”: true,
“bar”: false
}
}
}

But notice that node[‘myattributes’][‘foo’] is still the same as what you
set because it wasn’t altered in a recipe. So you can set attributes on the
node object and use them in recipes without their values being changed by
Chef runs.

In terms of the merging of multiple attributes from different sources the
end result of attributes available to a Chef run is a merger of all of the
attributes of all precedence levels of all sources. I’m not aware of any
specific simple string merger like you said but it’s something that can
easily be done in recipe code (i.e. in the recipe code have a section that
looks for a node additional config and appends it if it exists, otherwise
don’t)

On 7 October 2015 at 06:36, Fabien Delpierre fabien.delpierre@gmail.com
wrote:

Generally you would try to use an existing grouping like a chef

environment, role, or policy (if you are using any of those). For single
node edits you would do it directly via knife node edit. For things that
are between the scope of a role and environment,
https://github.com/poise/poise-appenv can be a solution.

Good to know about knife node edit. Wouldn’t any change be overridden at
the next chef-client run, though?
I guess I ought to look into policies, too. I’ve never used them but I’m
not finding much documentation about them. Are we talking about the new-ish
Policyfile feature? That I can find documentation about.

Does this help?

https://docs.chef.io/dsl_recipe.html#tag-tagged-untag

It’ll get me started at least, thanks!

No. Nodes can receive attributes by having a role with attributes in a run

list, in which case attributes are used from the role and/or by specifying
an environment when Chef is run, in which case attributes are used from the
environment. You can also set them in a recipe, which applies the
attributes to the node object itself.

I was including that when I mentioned “editing the run list” – of course
that works but if I have multiple nodes that use the same set of
roles/recipes and I edit one of those roles or recipes, all of them will be
affected, which I may not want.

knife node edit does seem like it might help, but I’m concerned that the
change wouldn’t persist past a chef-client run.

As an example I have a recipe which installs a Newrelic agent if a certain

attribute is set to true. In the production Chef environment I set the
attribute so that all production nodes have this. If I want to use newrelic
for a machine outside of production I can set that attribute on the
individual node object instead, which could be in any environment.

So that makes sense if you have a recipe that’s designed to look for that
attribute, but that still seems like it wouldn’t accomplish what one might
do with Puppet’s Hiera.

To use a concrete example, again with attaching YourKit to a JVM process
– like I mentioned, Hiera can aggregate multiple hashes or arrays spread
between multiple sources into a single attribute.
For example, in one YAML file I could have:
foo::java_opts: "-Xms2g -Xmx2g"
And in another:
foo::java_opts: "-agentpath:/opt/yjp/bin/linux-x86-64/libyjpagent.so"
Then I could invoke this foo::java_opts in a recipe or a template (e.g.
for an init script) and Hiera would combine them into one so it produces an
init script might read: java -Xms2g -Xmx2g -agentpath:
/opt/yjp/bin/linux-x86-64/libyjpagent.so
When I’m done with YourKit, I can just delete the YAML file that contains
those bits and I never have to touch the nodes or the recipe since it only
looks for what’s in that java_opts attribute and spits it back out.

While so far it doesn’t seem like this would be possible in Chef, it does
seem like tags would help, if accounted for within a recipe, e.g.:
node.default[‘foo’][‘bar’] = false

if tagged?(‘bar’)
ruby_block ‘enable_bar’ do
block do
node[‘foo’][‘bar’] = true
end
end
end

if node[‘foo’][‘bar’] == true
(do unspeakable things)
end

On Tue, Oct 6, 2015 at 10:23 PM, Noah Kantrowitz noah@coderanger.net
wrote:

On Oct 6, 2015, at 1:18 PM, Fabien Delpierre fabien.delpierre@gmail.com
wrote:

Hello folks,
I’m a user of both Puppet and Chef, I’m more familiar with and prefer
the latter overall, but am required to use the former at work. Recently I
was toying with the idea of moving to Chef, although it’s probably not a
realistic option, but it raised a few interesting questions.
While Puppet and Chef do roughly the same thing, they do it in slightly
different ways, and Puppet has something in particular that Chef doesn’t
have, which is Hiera.

If you’re not familiar with Hiera, it’s a simple way to apply settings
to any number of your servers (from a single node to an entire
infrastructure of thousands), you define the hierarchy of settings yourself
– in a sense, it’s kind of like being able to customize Chef’s attribute
precedence and having a single environment attached to all your nodes. The
link above explains it a lot better than I can.

Anyway, I was wondering how one might deal with losing Hiera when
moving from Puppet to Chef. It seems that you could achieve similar results
with a mix of cookbooks, roles and environments, like everything else, but
at the cost of ending up with a lot more of them than you’d normally have.

I guess a different way I might ask this is:
Imagine I have a group of servers, all configured the same way (same
run list, same environment), and I need to change something on just one of
them. A common scenario in my environment is having to attach YourKit to a
JVM process.
With Puppet/Hiera, I’d just add a two-line YAML file with the same name
as that one server’s FQDN, and when I run Puppet, that YAML file would be
read at compile time and the config change applied without affecting the
other nodes.
With Chef, I’d have to modify the node’s run list or take its
environment file, make a copy of it, modify the copy and assign that copied
environment to the node instead of the normal env file, lest I affect the
other servers in the same group.

Hiera is also capable of aggregating data from multiple sources into a
single hash or array to be applied to a node during a Puppet agent run.

Those are things that I never missed for as long as I used Chef, but
now that I know they’re out there, I’m wondering how I would handle things
without Hiera.

So in short, I was wondering if anybody has thoughts on the matter.
Obviously it helps if you have some experience with both tools. I’m far
from being a Chef expert so it’s possible I’m missing something. Any input
is appreciated. Thanks!
Fabien

Generally you would try to use an existing grouping like a chef
environment, role, or policy (if you are using any of those). For single
node edits you would do it directly via knife node edit. For things that
are between the scope of a role and environment,
https://github.com/poise/poise-appenv can be a solution.

–Noah

Is there a way to get the time that a resource executes within the trace
output when running a recipe, for example for the code below, i would like
to see the
time it executed for each of the lines below? Ideally it would either be
before or after the Hostname first field.

Connecting to p134n55.pbm.ihost.com
p134n55.pbm.ihost.com Starting Chef Client, version 12.4.1
p134n55.pbm.ihost.com resolving cookbooks for run list: [“aix-base-setup”,
“ebs”]
p134n55.pbm.ihost.com Synchronizing Cookbooks:
p134n55.pbm.ihost.com - aix
p134n55.pbm.ihost.com - aix-base-setup
p134n55.pbm.ihost.com - ebs
p134n55.pbm.ihost.com Compiling Cookbooks…
p134n55.pbm.ihost.com Converging 2 resources

Jubal Kohlmeier jubal@us.ibm.com
503-747-1333 503-502-7733 cell
IT Specialist Oracle Applications Benchmark

try --force-logger

On Wed, Oct 7, 2015 at 10:06 AM, Jubal Kohlmeier jubal@us.ibm.com wrote:

Is there a way to get the time that a resource executes within the trace
output when running a recipe, for example for the code below, i would like
to see the
time it executed for each of the lines below? Ideally it would either be
before or after the Hostname first field.

Connecting to p134n55.pbm.ihost.com
p134n55.pbm.ihost.com Starting Chef Client, version 12.4.1
p134n55.pbm.ihost.com resolving cookbooks for run list:
[“aix-base-setup”, “ebs”]
p134n55.pbm.ihost.com Synchronizing Cookbooks:
p134n55.pbm.ihost.com - aix
p134n55.pbm.ihost.com - aix-base-setup
p134n55.pbm.ihost.com - ebs
p134n55.pbm.ihost.com Compiling Cookbooks…
p134n55.pbm.ihost.com Converging 2 resources

Jubal Kohlmeier jubal@us.ibm.com
503-747-1333 503-502-7733 cell
IT Specialist Oracle Applications Benchmark

Attributes at the “normal” level belong to the node and are NOT wiped for
each run. All of the other levels belong to an external thing (role,
environment, policyfile) so they are wiped and recreated.

Attributes on the node object are only overwritten in the Chef run if there

is something in the recipe code which specifically alters that attribute.
The node object is both the desired state and current state of the object
(although there’s future work to split that planned I believe).

Alright, so I never bothered to look up normal attributes, but that makes
sense then. Is it correct that, by default, knife node edit will set
normal-level attributes, and if you want to set something at a different
level as a one-off, you can use the -a flag?

Hiera can aggregate multiple hashes or arrays spread between multiple

sources into a single attribute.

This is exactly what the chef attribute precedence system does. My
understanding is Hiera was written in response to chef’s attributes system,
though as a second system it has the benefit of learning from our
limitations so it might be more flexible in some ways. In particular Chef
has only one attribute level where you can set node-specific values, but
these get merged with cookbook/role/environment attributes. Arrays are
tricky since sometimes you need them to merge and sometimes override, I
would recommend sticking to hashes and then join things yourself.

I guess I was confused. Admittedly I haven’t had to worry about merging
hashes in Chef attributes, but I thought that attributes were overwritten
by sources higher in the food chain, period. I now see that’s not the case
with hashes: https://docs.chef.io/attributes.html#about-deep-merge
Well, good to know!

So I guess I’m still trying to figure out if key Hiera functionality is
easily achieved in Chef or if you just have to use a totally different
approach. The examples I used kind of steered the conversation in the wrong
direction (although valuable things were still learned!).
Concretely, here’s how I might have Hiera set up:

:backends:

  • yaml
    :yaml:
    :datadir: /etc/puppet/hiera
    :hierarchy:
  • %{::region}/%{::fqdn}
  • %{::region}/%{env}
  • %{::region}/common
  • global

That means I can store a whole bunch of attributes in /etc/puppet/hiera/
global.yaml that will apply to all nodes.
Attributes stored in /etc/puppet/hiera/us-east/common.yaml can add to or
override attributes stored in global.yaml.
Attributes stored in /etc/puppet/hiera/us-west/common.yaml will not
interact/-fere with the ones in us-east.
Attributes stored in /etc/puppet/hiera/us-west/%{env}.yaml will be applied
to all nodes in us-west whose names match %{env} (e.g. if %{env} = foo then
all nodes called foo01, foo02, foo03, etc. will be matches.
Attributes stored in /etc/puppet/hiera/us-west/%{::fqdn}.yaml will only be
applied to the particular node named by its FQDN (that special snowflake I
was talking about) and will override attributes (or merge, if they are
arrays or hashes) that are stored lower in the hierarchy.

I suppose that one way this can be achieved with Chef is with a set of
roles, so I’d have common, us-east, us-west and roles, but it gets
dicey with my %{env} thing above where settings can be applied to all nodes
with similar names (I used ‘web’ in my example but it could be ‘foo’,
‘bar’, ‘asdf’…). I don’t know how I can do that with Chef as easily as
with Hiera. Can I?

On Wed, Oct 7, 2015 at 11:39 AM, Daniel DeLeo dan@kallistec.com wrote:

On Wednesday, October 7, 2015 at 6:36 AM, Fabien Delpierre wrote:

Generally you would try to use an existing grouping like a chef
environment, role, or policy (if you are using any of those). For single
node edits you would do it directly via knife node edit. For things that
are between the scope of a role and environment,
https://github.com/poise/poise-appenv can be a solution.

Good to know about knife node edit. Wouldn’t any change be overridden at
the next chef-client run, though?
I guess I ought to look into policies, too. I’ve never used them but I’m
not finding much documentation about them. Are we talking about the new-ish
Policyfile feature? That I can find documentation about.

Attributes at the “normal” level belong to the node and are NOT wiped for
each run. All of the other levels belong to an external thing (role,
environment, policyfile) so they are wiped and recreated.

And yeah, policies means policyfiles. We’re still hammering out the exact
official terminology.

I was including that when I mentioned “editing the run list” – of
course that works but if I have multiple nodes that use the same set of
roles/recipes and I edit one of those roles or recipes, all of them will be
affected, which I may not want.

knife node edit does seem like it might help, but I’m concerned that the
change wouldn’t persist past a chef-client run.
knife node edit by default will only show you things that will be
persisted across chef runs: run_list, environment, “normal” attributes.

As an example I have a recipe which installs a Newrelic agent if a
certain attribute is set to true. In the production Chef environment I set
the attribute so that all production nodes have this. If I want to use
newrelic for a machine outside of production I can set that attribute on
the individual node object instead, which could be in any environment.

So that makes sense if you have a recipe that’s designed to look for
that attribute, but that still seems like it wouldn’t accomplish what one
might do with Puppet’s Hiera.

To use a concrete example, again with attaching YourKit to a JVM process
– like I mentioned, Hiera can aggregate multiple hashes or arrays spread
between multiple sources into a single attribute.
This is exactly what the chef attribute precedence system does. My
understanding is Hiera was written in response to chef’s attributes system,
though as a second system it has the benefit of learning from our
limitations so it might be more flexible in some ways. In particular Chef
has only one attribute level where you can set node-specific values, but
these get merged with cookbook/role/environment attributes. Arrays are
tricky since sometimes you need them to merge and sometimes override, I
would recommend sticking to hashes and then join things yourself.

For example, in one YAML file I could have:
foo::java_opts: "-Xms2g -Xmx2g"
And in another:
foo::java_opts: "-agentpath:/opt/yjp/bin/linux-x86-64/libyjpagent.so"
Then I could invoke this foo::java_opts in a recipe or a template (e.g.
for an init script) and Hiera would combine them into one so it produces an
init script might read: java -Xms2g -Xmx2g
-agentpath:/opt/yjp/bin/linux-x86-64/libyjpagent.so
When I’m done with YourKit, I can just delete the YAML file that
contains those bits and I never have to touch the nodes or the recipe since
it only looks for what’s in that java_opts attribute and spits it back out.

While so far it doesn’t seem like this would be possible in Chef, it
does seem like tags would help, if accounted for within a recipe, e.g.:
node.default[‘foo’][‘bar’] = false

if tagged?(‘bar’)
ruby_block ‘enable_bar’ do
block do
node[‘foo’][‘bar’] = true
end
end
end

if node[‘foo’][‘bar’] == true
(do unspeakable things)
end

My recommendation would be to make the snowflake case part of your
cookbook for the non-snowflake case. That way it’s always easy to find out
what your system will look like for both cases, and your snowflake configs
will be tracked in your git history just like the non-snowflake.


Daniel DeLeo

On Wednesday, October 7, 2015 at 12:58 PM, Fabien Delpierre wrote:

Attributes at the “normal” level belong to the node and are NOT wiped for each run. All of the other levels belong to an external thing (role, environment, policyfile) so they are wiped and recreated.

Attributes on the node object are only overwritten in the Chef run if there is something in the recipe code which specifically alters that attribute. The node object is both the desired state and current state of the object (although there’s future work to split that planned I believe).

Alright, so I never bothered to look up normal attributes, but that makes sense then. Is it correct that, by default, knife node edit will set normal-level attributes, and if you want to set something at a different level as a one-off, you can use the -a flag?

Hiera can aggregate multiple hashes or arrays spread between multiple sources into a single attribute.
This is exactly what the chef attribute precedence system does. My understanding is Hiera was written in response to chef’s attributes system, though as a second system it has the benefit of learning from our limitations so it might be more flexible in some ways. In particular Chef has only one attribute level where you can set node-specific values, but these get merged with cookbook/role/environment attributes. Arrays are tricky since sometimes you need them to merge and sometimes override, I would recommend sticking to hashes and then join things yourself.

I guess I was confused. Admittedly I haven’t had to worry about merging hashes in Chef attributes, but I thought that attributes were overwritten by sources higher in the food chain, period. I now see that’s not the case with hashes: https://docs.chef.io/attributes.html#about-deep-merge
Well, good to know!

So I guess I’m still trying to figure out if key Hiera functionality is easily achieved in Chef or if you just have to use a totally different approach. The examples I used kind of steered the conversation in the wrong direction (although valuable things were still learned!).
Concretely, here’s how I might have Hiera set up:

:backends:

  • yaml
    :yaml:
    :datadir: /etc/puppet/hiera
    :hierarchy:
  • %{::region}/%{::fqdn}
  • %{::region}/%{env}
  • %{::region}/common
  • global

That means I can store a whole bunch of attributes in /etc/puppet/hiera/global.yaml that will apply to all nodes.
Attributes stored in /etc/puppet/hiera/us-east/common.yaml can add to or override attributes stored in global.yaml.
Attributes stored in /etc/puppet/hiera/us-west/common.yaml will not interact/-fere with the ones in us-east.
Attributes stored in /etc/puppet/hiera/us-west/%{env}.yaml will be applied to all nodes in us-west whose names match %{env} (e.g. if %{env} = foo then all nodes called foo01, foo02, foo03, etc. will be matches.
Attributes stored in /etc/puppet/hiera/us-west/%{::fqdn}.yaml will only be applied to the particular node named by its FQDN (that special snowflake I was talking about) and will override attributes (or merge, if they are arrays or hashes) that are stored lower in the hierarchy.

I suppose that one way this can be achieved with Chef is with a set of roles, so I’d have common, us-east, us-west and roles, but it gets dicey with my %{env} thing above where settings can be applied to all nodes with similar names (I used ‘web’ in my example but it could be ‘foo’, ‘bar’, ‘asdf’…). I don’t know how I can do that with Chef as easily as with Hiera. Can I?
Roles can also override other roles, so you can just set default attributes in roles and put all of the ones you need the run lists of the appropriate nodes. That said, you might just be better off making a cookbook that has only attributes files, and put a ruby statement like return unless node.something == “some value” at the top of each one. Then you can control which attributes get applied on whatever axis you like.


Daniel DeLeo

On 10/07/2015 08:39 AM, Daniel DeLeo wrote:

Attributes at the “normal” level belong to the node and are NOT wiped
for each run. All of the other levels belong to an external thing
(role, environment, policyfile) so they are wiped and recreated.

Note that tags are implemented as normal attributes so they are easy
ways to use normal attributes and have a nice understandable API, where
you expect them to persist and generally expect one-offs to occur and
have a nice ‘knife tag’ command line API to use.

In general, normal attributes cause all kinds of issues when people use
them to set attributes from cookbooks. Since normal attributes do
persist and don’t get cleared once they’re set in a cookbook they’re set
on the node. That can be confusing when you delete the line from the
cookbook which set the normal attribute and then expect the setting to
be gone in the next run but its not. For that reason we generally
discourage their use. It can be very confusing to have a chef-client
run behaving some way and you look through the cookbooks and roles and
it simply “cant” be doing that, but then you ‘knife node show’ and
discover a normal attribute that someone set on the node.

It is particularly bad that normal attributes came first and ‘node.set’
is a kind of obvious API to use to set attributes, but which sets a
’normal’ attribute. And the word ‘normal’ sounds like the kind of thing
that’s not bad to use – but generally it is.

However, for one-offs and tags they’re fine. You can also ‘tag’ nodes
with a role that sets the attributes (or a role-cookbook or
role-with-cookbook-that-acts-like-role-cookbook) and that way you can
easily see that kind of ‘tag’ on the run_list. Might create clutter on
the run_list, but that’s generally going to be a more flexible approach
than tags and you don’t risk any normal-attributes confusion (again
though the run_list itself is a normal attribute, so you’re using them
under the covers, but in a way that you’ve been trained to expect their
behavior).