Where to keep your <node>.json files?


#1

Ohai Chefs,

where do you keep your .json files?

My approach is to use knife node upload from file <node>.json whenever I
need to change something on the node. Ideally, I would want to make sure
that whatever I upload from-file must be already commited to version
control for traceability reasons (I’ve seen some approaches to ensure this,
but don’t quite remember them…).

But where do you keep these files? I was considering to put them under
chef-repo/nodes as the chef-repo git repo seems to be the right place for
me. But since there is no such directory in the default chef-repo structure
I was wondering if you guys know a better place for it?

Thanks,
Torben


#2

On Mon, Apr 23, 2012 at 07:36, Torben Knerr ukio@gmx.de wrote:

Ohai Chefs,

But where do you keep these files? I was considering to put them under
chef-repo/nodes as the chef-repo git repo seems to be the right place for
me. But since there is no such directory in the default chef-repo structure
I was wondering if you guys know a better place for it?

Well, I am using this directory for this purpose.


Motiejus Jakštys


#3

I don’t keep these. We only store them for backups when deleting
nodes, in the event that some data required for recovery is persisted
to node data.

I wouldn’t recommend the overhead of ensuring nodes are synchronized
from server to disk and back, but perhaps a tool could be built for it
(jenkins?)

Embrace chaos, destroy nodes regularly – build for failure.

–AJ

2012/4/23 Motiejus Jakštys desired.mta@gmail.com:

On Mon, Apr 23, 2012 at 07:36, Torben Knerr ukio@gmx.de wrote:

Ohai Chefs,

But where do you keep these files? I was considering to put them under
chef-repo/nodes as the chef-repo git repo seems to be the right place for
me. But since there is no such directory in the default chef-repo structure
I was wondering if you guys know a better place for it?

Well, I am using this directory for this purpose.


Motiejus Jakštys


#4

@AJ: how do you keep track of modifications to nodes, e.g. who added a new
role or cookbook? Does Chef Server keep track of that?

I was thinking that infrastructure-as-code wise one would keep the nodes in
source control as well, as I would do for databags (and I see good reasons
for doing that). Is this just a matter of taste or did I miss something?

Cheers,
Torben
Am 23.04.2012 10:59 schrieb “AJ Christensen” aj@junglist.gen.nz:

I don’t keep these. We only store them for backups when deleting
nodes, in the event that some data required for recovery is persisted
to node data.

I wouldn’t recommend the overhead of ensuring nodes are synchronized
from server to disk and back, but perhaps a tool could be built for it
(jenkins?)

Embrace chaos, destroy nodes regularly – build for failure.

–AJ

2012/4/23 Motiejus Jakštys desired.mta@gmail.com:

On Mon, Apr 23, 2012 at 07:36, Torben Knerr ukio@gmx.de wrote:

Ohai Chefs,

But where do you keep these files? I was considering to put them under
chef-repo/nodes as the chef-repo git repo seems to be the right place
for

me. But since there is no such directory in the default chef-repo
structure

I was wondering if you guys know a better place for it?

Well, I am using this directory for this purpose.


Motiejus Jakštys


#5

Hi, Torben

I think Spiceweasel could be a good idea to track all those changes:
http://wiki.opscode.com/display/chef/Spiceweasel

Still it doesn’t sync with your real infrastructure, but it is easy to
keep updated and is a good picture of your infrastructure-as-code.

Cheers

2012/4/23 Torben Knerr ukio@gmx.de:

@AJ: how do you keep track of modifications to nodes, e.g. who added a new
role or cookbook? Does Chef Server keep track of that?

I was thinking that infrastructure-as-code wise one would keep the nodes in
source control as well, as I would do for databags (and I see good reasons
for doing that). Is this just a matter of taste or did I miss something?

Cheers,
Torben

Am 23.04.2012 10:59 schrieb “AJ Christensen” aj@junglist.gen.nz:

I don’t keep these. We only store them for backups when deleting
nodes, in the event that some data required for recovery is persisted
to node data.

I wouldn’t recommend the overhead of ensuring nodes are synchronized
from server to disk and back, but perhaps a tool could be built for it
(jenkins?)

Embrace chaos, destroy nodes regularly – build for failure.

–AJ

2012/4/23 Motiejus Jakštys desired.mta@gmail.com:

On Mon, Apr 23, 2012 at 07:36, Torben Knerr ukio@gmx.de wrote:

Ohai Chefs,

But where do you keep these files? I was considering to put them under
chef-repo/nodes as the chef-repo git repo seems to be the right place
for
me. But since there is no such directory in the default chef-repo
structure
I was wondering if you guys know a better place for it?

Well, I am using this directory for this purpose.


Motiejus Jakštys


Juanje


#6

In the sphere of applications, we often have a codebase and a database.
Only the code is stored in source control. None of the data is stored in
source control, but we safeguard the data by taking frequent backups.

One approach to using Chef, an approach paralleling applications, is as
follows. Cookbooks are always code. Nodes and data-bags are always data.
Some roles and environments qualify as code and some qualify as data; this
varies from role to role and from environment to environment. Keep
cookbooks and the code-ish roles and environments in source control; write
code-ish roles and environments in Ruby. Keep node and data-bag data, along
with the data-ish roles and environments, out of source control and left to
the database; write data-ish roles and environments in JSON. To safeguard
the data, take frequent backups of the database.

What are people’s thoughts on this way of using Chef?

On Mon, Apr 23, 2012 at 12:31 PM, Juanje Ojeda Croissier <
juanje.ojeda@gmail.com> wrote:

Hi, Torben

I think Spiceweasel could be a good idea to track all those changes:
http://wiki.opscode.com/display/chef/Spiceweasel

Still it doesn’t sync with your real infrastructure, but it is easy to
keep updated and is a good picture of your infrastructure-as-code.

Cheers

2012/4/23 Torben Knerr ukio@gmx.de:

@AJ: how do you keep track of modifications to nodes, e.g. who added a
new
role or cookbook? Does Chef Server keep track of that?

I was thinking that infrastructure-as-code wise one would keep the nodes
in
source control as well, as I would do for databags (and I see good
reasons
for doing that). Is this just a matter of taste or did I miss something?

Cheers,
Torben

Am 23.04.2012 10:59 schrieb “AJ Christensen” aj@junglist.gen.nz:

I don’t keep these. We only store them for backups when deleting
nodes, in the event that some data required for recovery is persisted
to node data.

I wouldn’t recommend the overhead of ensuring nodes are synchronized
from server to disk and back, but perhaps a tool could be built for it
(jenkins?)

Embrace chaos, destroy nodes regularly – build for failure.

–AJ

2012/4/23 Motiejus Jakštys desired.mta@gmail.com:

On Mon, Apr 23, 2012 at 07:36, Torben Knerr ukio@gmx.de wrote:

Ohai Chefs,

But where do you keep these files? I was considering to put them
under

chef-repo/nodes as the chef-repo git repo seems to be the right place
for
me. But since there is no such directory in the default chef-repo
structure
I was wondering if you guys know a better place for it?

Well, I am using this directory for this purpose.


Motiejus Jakštys


Juanje

http://about.me/juanje


#7

Thanks Juanje, spiceweasel looks good, will have a look at it!
-Torben
Am 23.04.2012 18:32 schrieb “Juanje Ojeda Croissier” <juanje.ojeda@gmail.com

:

Hi, Torben

I think Spiceweasel could be a good idea to track all those changes:
http://wiki.opscode.com/display/chef/Spiceweasel

Still it doesn’t sync with your real infrastructure, but it is easy to
keep updated and is a good picture of your infrastructure-as-code.

Cheers

2012/4/23 Torben Knerr ukio@gmx.de:

@AJ: how do you keep track of modifications to nodes, e.g. who added a
new
role or cookbook? Does Chef Server keep track of that?

I was thinking that infrastructure-as-code wise one would keep the nodes
in
source control as well, as I would do for databags (and I see good
reasons
for doing that). Is this just a matter of taste or did I miss something?

Cheers,
Torben

Am 23.04.2012 10:59 schrieb “AJ Christensen” aj@junglist.gen.nz:

I don’t keep these. We only store them for backups when deleting
nodes, in the event that some data required for recovery is persisted
to node data.

I wouldn’t recommend the overhead of ensuring nodes are synchronized
from server to disk and back, but perhaps a tool could be built for it
(jenkins?)

Embrace chaos, destroy nodes regularly – build for failure.

–AJ

2012/4/23 Motiejus Jakštys desired.mta@gmail.com:

On Mon, Apr 23, 2012 at 07:36, Torben Knerr ukio@gmx.de wrote:

Ohai Chefs,

But where do you keep these files? I was considering to put them
under

chef-repo/nodes as the chef-repo git repo seems to be the right place
for
me. But since there is no such directory in the default chef-repo
structure
I was wondering if you guys know a better place for it?

Well, I am using this directory for this purpose.


Motiejus Jakštys


Juanje

http://about.me/juanje


#8

Am 23.04.2012 18:48 schrieb “Jay Feldblum” y_feldblum@yahoo.com:

In the sphere of applications, we often have a codebase and a database.
Only the code is stored in source control. None of the data is stored in
source control, but we safeguard the data by taking frequent backups.

One approach to using Chef, an approach paralleling applications, is as
follows. Cookbooks are always code. Nodes and data-bags are always data.
Some roles and environments qualify as code and some qualify as data; this
varies from role to role and from environment to environment. Keep
cookbooks and the code-ish roles and environments in source control; write
code-ish roles and environments in Ruby. Keep node and data-bag data, along
with the data-ish roles and environments, out of source control and left to
the database; write data-ish roles and environments in JSON. To safeguard
the data, take frequent backups of the database.

Can you give an example of code-ish vs. data-ish
roles/environments/databags?

For me all of them qualify as “code” because they are represented as text
and have the benefit that they can be easily version controlled, diffed and
merged.

What are people’s thoughts on this way of using Chef?

On Mon, Apr 23, 2012 at 12:31 PM, Juanje Ojeda Croissier <
juanje.ojeda@gmail.com> wrote:

Hi, Torben

I think Spiceweasel could be a good idea to track all those changes:
http://wiki.opscode.com/display/chef/Spiceweasel

Still it doesn’t sync with your real infrastructure, but it is easy to
keep updated and is a good picture of your infrastructure-as-code.

Cheers

2012/4/23 Torben Knerr ukio@gmx.de:

@AJ: how do you keep track of modifications to nodes, e.g. who added a
new

role or cookbook? Does Chef Server keep track of that?

I was thinking that infrastructure-as-code wise one would keep the
nodes in

source control as well, as I would do for databags (and I see good
reasons

for doing that). Is this just a matter of taste or did I miss
something?

Cheers,
Torben

Am 23.04.2012 10:59 schrieb “AJ Christensen” aj@junglist.gen.nz:

I don’t keep these. We only store them for backups when deleting
nodes, in the event that some data required for recovery is persisted
to node data.

I wouldn’t recommend the overhead of ensuring nodes are synchronized
from server to disk and back, but perhaps a tool could be built for it
(jenkins?)

Embrace chaos, destroy nodes regularly – build for failure.

–AJ

2012/4/23 Motiejus Jakštys desired.mta@gmail.com:

On Mon, Apr 23, 2012 at 07:36, Torben Knerr ukio@gmx.de wrote:

Ohai Chefs,

But where do you keep these files? I was considering to put them
under

chef-repo/nodes as the chef-repo git repo seems to be the right
place

for
me. But since there is no such directory in the default chef-repo
structure
I was wondering if you guys know a better place for it?

Well, I am using this directory for this purpose.


Motiejus Jakštys


Juanje

http://about.me/juanje


#9

E.g., roles named “mysql-pair-1”, “mysql-pair-2”, “mysql-pair-3”,
“mysql-pair-4”, etc., that are all essentially identical to each other but
have the effect of allowing you to set up multiple mysql pairs. You don’t
need every single pair in source control, if they are all essentially the
same. You might have a “mysql-pair” role which is included in each of the
above, and that might be in source control.

On Mon, Apr 23, 2012 at 1:12 PM, Torben Knerr ukio@gmx.de wrote:

Am 23.04.2012 18:48 schrieb “Jay Feldblum” y_feldblum@yahoo.com:

In the sphere of applications, we often have a codebase and a database.
Only the code is stored in source control. None of the data is stored in
source control, but we safeguard the data by taking frequent backups.

One approach to using Chef, an approach paralleling applications, is as
follows. Cookbooks are always code. Nodes and data-bags are always data.
Some roles and environments qualify as code and some qualify as data; this
varies from role to role and from environment to environment. Keep
cookbooks and the code-ish roles and environments in source control; write
code-ish roles and environments in Ruby. Keep node and data-bag data, along
with the data-ish roles and environments, out of source control and left to
the database; write data-ish roles and environments in JSON. To safeguard
the data, take frequent backups of the database.

Can you give an example of code-ish vs. data-ish
roles/environments/databags?

For me all of them qualify as “code” because they are represented as text
and have the benefit that they can be easily version controlled, diffed and
merged.

What are people’s thoughts on this way of using Chef?

On Mon, Apr 23, 2012 at 12:31 PM, Juanje Ojeda Croissier <
juanje.ojeda@gmail.com> wrote:

Hi, Torben

I think Spiceweasel could be a good idea to track all those changes:
http://wiki.opscode.com/display/chef/Spiceweasel

Still it doesn’t sync with your real infrastructure, but it is easy to
keep updated and is a good picture of your infrastructure-as-code.

Cheers

2012/4/23 Torben Knerr ukio@gmx.de:

@AJ: how do you keep track of modifications to nodes, e.g. who added
a new

role or cookbook? Does Chef Server keep track of that?

I was thinking that infrastructure-as-code wise one would keep the
nodes in

source control as well, as I would do for databags (and I see good
reasons

for doing that). Is this just a matter of taste or did I miss
something?

Cheers,
Torben

Am 23.04.2012 10:59 schrieb “AJ Christensen” aj@junglist.gen.nz:

I don’t keep these. We only store them for backups when deleting
nodes, in the event that some data required for recovery is persisted
to node data.

I wouldn’t recommend the overhead of ensuring nodes are synchronized
from server to disk and back, but perhaps a tool could be built for
it

(jenkins?)

Embrace chaos, destroy nodes regularly – build for failure.

–AJ

2012/4/23 Motiejus Jakštys desired.mta@gmail.com:

On Mon, Apr 23, 2012 at 07:36, Torben Knerr ukio@gmx.de wrote:

Ohai Chefs,

But where do you keep these files? I was considering to put them
under

chef-repo/nodes as the chef-repo git repo seems to be the right
place

for
me. But since there is no such directory in the default chef-repo
structure
I was wondering if you guys know a better place for it?

Well, I am using this directory for this purpose.


Motiejus Jakštys


Juanje

http://about.me/juanje


#10

2012/4/23 Torben Knerr ukio@gmx.de:

@AJ: how do you keep track of modifications to nodes, e.g. who added a new
role or cookbook? Does Chef Server keep track of that?

I was thinking that infrastructure-as-code wise one would keep the nodes in
source control as well, as I would do for databags (and I see good reasons
for doing that). Is this just a matter of taste or did I miss something?

The best-practice-goal is for nodes to be more of a commodity and not
have anything unique about them beyond their run_list. Data should be
stored elsewhere, which saves you the trouble of 1) backing up nodes
and 2) sifting through nodes for their unique data.

The Chef Server doesn’t keep an audit trail, but Opscode expects to
ship some features in Chef 11 that would be useful here.

Bryan


#11

Am 23.04.2012 19:49 schrieb “Bryan McLellan” btm@loftninjas.org:

2012/4/23 Torben Knerr ukio@gmx.de:

@AJ: how do you keep track of modifications to nodes, e.g. who added a
new

role or cookbook? Does Chef Server keep track of that?

I was thinking that infrastructure-as-code wise one would keep the
nodes in

source control as well, as I would do for databags (and I see good
reasons

for doing that). Is this just a matter of taste or did I miss something?

The best-practice-goal is for nodes to be more of a commodity and not
have anything unique about them beyond their run_list. Data should be
stored elsewhere, which saves you the trouble of 1) backing up nodes
and 2) sifting through nodes for their unique data.

Never thought about backups this way, but it’s totally neat that you need
to backup only the data stores, not any app servers or the like anymore…
:slight_smile:

But what about attribute overrides on node level? Is that something that we
should rather avoid and better do any cookbook configuration on a role
level instead?

The Chef Server doesn’t keep an audit trail, but Opscode expects to
ship some features in Chef 11 that would be useful here.

Interested to hear more if you can tell already.

Bryan


#12

On Tue, Apr 24, 2012 at 3:41 PM, Torben Knerr ukio@gmx.de wrote:

But what about attribute overrides on node level? Is that something that we
should rather avoid and better do any cookbook configuration on a role level
instead?

We have been able to avoid node specific attributes in our
infrastructure. Attributes are all in roles or environments. Sometimes
we store the configuration for a particular application in a databag.
Then our recipes may read the databag (often based on the current
environment) and publish relevant attributes from the databag into the
node - but usually we only do this when we need to use these
attributes in searches from other nodes. So far … this approach
seems to be working ok.


Cheers,

Peter Donald


#13

Hi,

And someone just pointed out to me a cookbook that may help you …
though I have yet to try it.

http://community.opscode.com/cookbooks/bag_config

On Tue, Apr 24, 2012 at 3:49 PM, Peter Donald peter@realityforge.org wrote:

On Tue, Apr 24, 2012 at 3:41 PM, Torben Knerr ukio@gmx.de wrote:

But what about attribute overrides on node level? Is that something that we
should rather avoid and better do any cookbook configuration on a role level
instead?

We have been able to avoid node specific attributes in our
infrastructure. Attributes are all in roles or environments. Sometimes
we store the configuration for a particular application in a databag.
Then our recipes may read the databag (often based on the current
environment) and publish relevant attributes from the databag into the
node - but usually we only do this when we need to use these
attributes in searches from other nodes. So far … this approach
seems to be working ok.


Cheers,

Peter Donald


Cheers,

Peter Donald


#14

That cookbook is built by Heavy Water – I guess it largely reflects
my views on transitive node data.

–AJ

On 24 April 2012 20:13, Peter Donald peter@realityforge.org wrote:

Hi,

And someone just pointed out to me a cookbook that may help you …
though I have yet to try it.

http://community.opscode.com/cookbooks/bag_config

On Tue, Apr 24, 2012 at 3:49 PM, Peter Donald peter@realityforge.org wrote:

On Tue, Apr 24, 2012 at 3:41 PM, Torben Knerr ukio@gmx.de wrote:

But what about attribute overrides on node level? Is that something that we
should rather avoid and better do any cookbook configuration on a role level
instead?

We have been able to avoid node specific attributes in our
infrastructure. Attributes are all in roles or environments. Sometimes
we store the configuration for a particular application in a databag.
Then our recipes may read the databag (often based on the current
environment) and publish relevant attributes from the databag into the
node - but usually we only do this when we need to use these
attributes in searches from other nodes. So far … this approach
seems to be working ok.


Cheers,

Peter Donald


Cheers,

Peter Donald


#15

On Tue, Apr 24, 2012 at 1:41 AM, Torben Knerr ukio@gmx.de wrote:

But what about attribute overrides on node level? Is that something that we
should rather avoid and better do any cookbook configuration on a role level
instead?

Generally avoid them. Ask yourself, is this something specific about
this computer, or about what it is doing? Usually the override is
configuring what the server is doing, like setting a configuration
value for your web server. Shouldn’t all your web servers be
configured the same way?

Sometimes your computer will be unique, like it has more or less ram
than another and you have to tune -Xmx for a Java VM, or whatever. Let
the automation do the work and upfactor this to the cookbook to choose
this value based on the environment, like the total memory installed
(via ohai) other roles assigned to the node.

The Chef Server doesn’t keep an audit trail, but Opscode expects to
ship some features in Chef 11 that would be useful here.

Interested to hear more if you can tell already.

We’ve got code written that audits actions on the API side, but it is
in the newer server code that is erlang based which we haven’t
released yet. We’re getting started on the planning portion of that
release over the next couple of months, so it’s too early to make any
claims about what specific features will look like.

Bryan


#16

Peter, AJ,

thanks for sharing your thoughts and the pointer to the bag_config
cookbook, I will have a look at it!

Torben

On Tue, Apr 24, 2012 at 10:27 AM, AJ Christensen aj@junglist.gen.nz wrote:

That cookbook is built by Heavy Water – I guess it largely reflects
my views on transitive node data.

–AJ

On 24 April 2012 20:13, Peter Donald peter@realityforge.org wrote:

Hi,

And someone just pointed out to me a cookbook that may help you …
though I have yet to try it.

http://community.opscode.com/cookbooks/bag_config

On Tue, Apr 24, 2012 at 3:49 PM, Peter Donald peter@realityforge.org
wrote:

On Tue, Apr 24, 2012 at 3:41 PM, Torben Knerr ukio@gmx.de wrote:

But what about attribute overrides on node level? Is that something
that we

should rather avoid and better do any cookbook configuration on a role
level

instead?

We have been able to avoid node specific attributes in our
infrastructure. Attributes are all in roles or environments. Sometimes
we store the configuration for a particular application in a databag.
Then our recipes may read the databag (often based on the current
environment) and publish relevant attributes from the databag into the
node - but usually we only do this when we need to use these
attributes in searches from other nodes. So far … this approach
seems to be working ok.


Cheers,

Peter Donald


Cheers,

Peter Donald


#17

On Tue, Apr 24, 2012 at 6:48 PM, Bryan McLellan btm@loftninjas.org wrote:

On Tue, Apr 24, 2012 at 1:41 AM, Torben Knerr ukio@gmx.de wrote:

But what about attribute overrides on node level? Is that something that
we
should rather avoid and better do any cookbook configuration on a role
level
instead?

Generally avoid them. Ask yourself, is this something specific about
this computer, or about what it is doing? Usually the override is
configuring what the server is doing, like setting a configuration
value for your web server. Shouldn’t all your web servers be
configured the same way?

Sometimes your computer will be unique, like it has more or less ram
than another and you have to tune -Xmx for a Java VM, or whatever. Let
the automation do the work and upfactor this to the cookbook to choose
this value based on the environment, like the total memory installed
(via ohai) other roles assigned to the node.

The Chef Server doesn’t keep an audit trail, but Opscode expects to
ship some features in Chef 11 that would be useful here.

Interested to hear more if you can tell already.

We’ve got code written that audits actions on the API side, but it is
in the newer server code that is erlang based which we haven’t
released yet. We’re getting started on the planning portion of that
release over the next couple of months, so it’s too early to make any
claims about what specific features will look like.

Bryan

Sounds very reasonable and makes totally sense to me. Lesson learned.

Thanks! :slight_smile: