MySQL, chef-solo and node.save


#1

Hey all,

Fletcher Nichol, Timberlake (jk) and I were bantering back and forth
on the twitters originally about the mysql cookbook blowing up on
failed initial runs on chef-solo (all related to the preseeded
passwords). Fletcher evidently has a PR coming about explicitly
failing out on chef-solo unless you specify all the passwords in your
node json. That’s a bit of a different discussion. What I wanted to
noodle out is the possibility of a chef-solo friendly node.save. There
were few options that came up:

  • Write back to the node json used during the call
  • Write somewhere else to disk

The first one is fraught with edge cases and violating expectations
when mutating that file. What if it’s under version control? What if
it’s immutable in some way? The list goes on. I think we can sort of
agree it’s a bad idea. So the second one came up. My thought is that
node.save when called under chef-solo should dump the node attributes
to the cache path as something like node-data.json (or possibly
something a bit more identifiable if you want to utilize the node data
between chef-solo runs on other nodes.

I realize that chef-solo will never replace server. I don’t think
Opscode would have objections to community-originated PRs for adding
some kind of functionality like this? It would, however, allow us to
unify how we create recipes that are friendly to both solo and server.
It also has, as Ranjib Dey pointed out on twitter, immense benefit to
the various testing initiatives out there.

I’d also like to bring up a use case for reference that we have at enStratus:

Much like OPC (as I understand it), we use chef-solo as our
"installer". enStratus is fairly modular and while the default mode of
installation is to collocate everything on a single machine, obviously
that doesn’t work. We typically support a single, two and four node
configuration of our stack for customer installs. Right now we use a
bit of a wrapper script and some manual error-prone data editing to
perform the two and four node installs. What would be nice is if we
could choose to persist node data between runs - primarily because of
the mysql cookbook thing - but also because we need to track some very
simple things like the mysql server IP but some passwords and
encryption keys. I having each node’s data persisted to
cache/node-state/node{1-4}.json and loaded would be a godsend for
making these things more idempotent.

Anyway, just looking for feedback/thoughts.

Thanks
John E. Vincent
@lusis


#2

I’m surprised that issue is still around. I hit that years ago and
just ended up not using solo since.

However, I’d vote for writing to a ‘new’ json file and that the
original node.json and the new one are merged during the run with the
new json overriding node.json. That way the original is not messed
with and you have the final end result available as a json file for
future runs.

Alex

On Wed, Oct 3, 2012 at 12:51 PM, John Vincent
lusis.org+chef-dev@gmail.com wrote:

Hey all,

Fletcher Nichol, Timberlake (jk) and I were bantering back and forth
on the twitters originally about the mysql cookbook blowing up on
failed initial runs on chef-solo (all related to the preseeded
passwords). Fletcher evidently has a PR coming about explicitly
failing out on chef-solo unless you specify all the passwords in your
node json. That’s a bit of a different discussion. What I wanted to
noodle out is the possibility of a chef-solo friendly node.save. There
were few options that came up:

  • Write back to the node json used during the call
  • Write somewhere else to disk

The first one is fraught with edge cases and violating expectations
when mutating that file. What if it’s under version control? What if
it’s immutable in some way? The list goes on. I think we can sort of
agree it’s a bad idea. So the second one came up. My thought is that
node.save when called under chef-solo should dump the node attributes
to the cache path as something like node-data.json (or possibly
something a bit more identifiable if you want to utilize the node data
between chef-solo runs on other nodes.

I realize that chef-solo will never replace server. I don’t think
Opscode would have objections to community-originated PRs for adding
some kind of functionality like this? It would, however, allow us to
unify how we create recipes that are friendly to both solo and server.
It also has, as Ranjib Dey pointed out on twitter, immense benefit to
the various testing initiatives out there.

I’d also like to bring up a use case for reference that we have at enStratus:

Much like OPC (as I understand it), we use chef-solo as our
"installer". enStratus is fairly modular and while the default mode of
installation is to collocate everything on a single machine, obviously
that doesn’t work. We typically support a single, two and four node
configuration of our stack for customer installs. Right now we use a
bit of a wrapper script and some manual error-prone data editing to
perform the two and four node installs. What would be nice is if we
could choose to persist node data between runs - primarily because of
the mysql cookbook thing - but also because we need to track some very
simple things like the mysql server IP but some passwords and
encryption keys. I having each node’s data persisted to
cache/node-state/node{1-4}.json and loaded would be a godsend for
making these things more idempotent.

Anyway, just looking for feedback/thoughts.

Thanks
John E. Vincent
@lusis


#3

John, Alex,

Chef isn’t really an interactive or back-and-forth conversation between
recipes and the node data. The Chef model is much more linear than that.

As you know, the general Chef model is that you specify the whole abstract
configuration (run-lists, node-data) up front, and then recipes just expand
it out by declaring resources based on the abstract configuration, and then
chef finishes the job by converging the server to match the declared
resources.

If you need to generate a password during one run and save it on the server
for use during the next run, you can use a simple trick as follows:

You can have a well-known password-storage file wherever you like (under
/var/chef or wherever else you like). During the recipe run (“compilation
phase”) you can test for the existence of the file. If it does not exist,
you generate a new password into a local variable and write it to the file
(still during the “compilation phase”); that’s what happens during the
first run. If it does exist, you read the password from the file into a
local variable; that’s what happens during all subsequent runs. After that
step, you can always copy the password into the node data, but of course
now there’s no longer any need to save the new node data, because it will
always be recomputed exactly with the same deterministic input and with the
same deterministic algorithm on every subsequent chef run.

Cheers,
Jay

On Wed, Oct 3, 2012 at 4:19 PM, Alex Soto apsoto@gmail.com wrote:

I’m surprised that issue is still around. I hit that years ago and
just ended up not using solo since.

However, I’d vote for writing to a ‘new’ json file and that the
original node.json and the new one are merged during the run with the
new json overriding node.json. That way the original is not messed
with and you have the final end result available as a json file for
future runs.

Alex

On Wed, Oct 3, 2012 at 12:51 PM, John Vincent
lusis.org+chef-dev@gmail.com wrote:

Hey all,

Fletcher Nichol, Timberlake (jk) and I were bantering back and forth
on the twitters originally about the mysql cookbook blowing up on
failed initial runs on chef-solo (all related to the preseeded
passwords). Fletcher evidently has a PR coming about explicitly
failing out on chef-solo unless you specify all the passwords in your
node json. That’s a bit of a different discussion. What I wanted to
noodle out is the possibility of a chef-solo friendly node.save. There
were few options that came up:

  • Write back to the node json used during the call
  • Write somewhere else to disk

The first one is fraught with edge cases and violating expectations
when mutating that file. What if it’s under version control? What if
it’s immutable in some way? The list goes on. I think we can sort of
agree it’s a bad idea. So the second one came up. My thought is that
node.save when called under chef-solo should dump the node attributes
to the cache path as something like node-data.json (or possibly
something a bit more identifiable if you want to utilize the node data
between chef-solo runs on other nodes.

I realize that chef-solo will never replace server. I don’t think
Opscode would have objections to community-originated PRs for adding
some kind of functionality like this? It would, however, allow us to
unify how we create recipes that are friendly to both solo and server.
It also has, as Ranjib Dey pointed out on twitter, immense benefit to
the various testing initiatives out there.

I’d also like to bring up a use case for reference that we have at
enStratus:

Much like OPC (as I understand it), we use chef-solo as our
"installer". enStratus is fairly modular and while the default mode of
installation is to collocate everything on a single machine, obviously
that doesn’t work. We typically support a single, two and four node
configuration of our stack for customer installs. Right now we use a
bit of a wrapper script and some manual error-prone data editing to
perform the two and four node installs. What would be nice is if we
could choose to persist node data between runs - primarily because of
the mysql cookbook thing - but also because we need to track some very
simple things like the mysql server IP but some passwords and
encryption keys. I having each node’s data persisted to
cache/node-state/node{1-4}.json and loaded would be a godsend for
making these things more idempotent.

Anyway, just looking for feedback/thoughts.

Thanks
John E. Vincent
@lusis


#4

On Wed, Oct 3, 2012 at 2:50 PM, Jay Feldblum y_feldblum@yahoo.com wrote:

John, Alex,

Chef isn’t really an interactive or back-and-forth conversation between
recipes and the node data. The Chef model is much more linear than that.

I wouldn’t disagree with that per se.

As you know, the general Chef model is that you specify the whole abstract
configuration (run-lists, node-data) up front, and then recipes just expand
it out by declaring resources based on the abstract configuration, and then
chef finishes the job by converging the server to match the declared
resources.

Still with you.

If you need to generate a password during one run and save it on the server
for use during the next run, you can use a simple trick as follows:

You can have a well-known password-storage file wherever you like (under
/var/chef or wherever else you like). During the recipe run (“compilation
phase”) you can test for the existence of the file. If it does not exist,
you generate a new password into a local variable and write it to the file
(still during the “compilation phase”); that’s what happens during the first
run. If it does exist, you read the password from the file into a local
variable; that’s what happens during all subsequent runs. After that step,
you can always copy the password into the node data, but of course now
there’s no longer any need to save the new node data, because it will always
be recomputed exactly with the same deterministic input and with the same
deterministic algorithm on every subsequent chef run.

Here’s where I’d disagree with you. If a node existed in totality on
its own, that would make AWESOME sense. Let’s take a scenario though.

I have a DB server. I have an app server. Under the current Chef model
(and the design of the official Opscode cookbooks) is that the node
NEEDS the resource, declares the resource. If I need a db created, a
migration run or a user created the consumer of that resource (in this
case the app server) does that work under its Chef cookbooks. Mind you
I’d argue that the creator of that resource should be the DB server.
However that’s just not possible in the current Chef world. I could
use Noah to do the orchestration but even I’d admit it’s kludgy. What
we essentially are faced with his how to declare a resource on another
node from the node that needs it.

So in our case, we have the declaration of databases and app users in
the recipe for each application. That makes sense in a logical
grouping. The DB Server role is responsible for standing up MySQL and
creating the base state. The application role is responsible for
creating the databases it needs. But to do that, it needs to know the
root password. The root password that lives on the db server - under
the node data for that node.

Sure we can refactor a bit and move the application specific DB
information into a role that runs on the db server but that doesn’t
fit how some folks think - that the database doesn’t belong to the db
server but to the application using it. I’ve done it both ways and I
honestly don’t know which way is entirely correct but it’s much easier
to minimize the need for chef runs to be coordinated between nodes.

Did that make any sense at all cause I’m having to reread what I wrote myself :wink:

Cheers,
Jay

On Wed, Oct 3, 2012 at 4:19 PM, Alex Soto apsoto@gmail.com wrote:

I’m surprised that issue is still around. I hit that years ago and
just ended up not using solo since.

However, I’d vote for writing to a ‘new’ json file and that the
original node.json and the new one are merged during the run with the
new json overriding node.json. That way the original is not messed
with and you have the final end result available as a json file for
future runs.

Alex

On Wed, Oct 3, 2012 at 12:51 PM, John Vincent
lusis.org+chef-dev@gmail.com wrote:

Hey all,

Fletcher Nichol, Timberlake (jk) and I were bantering back and forth
on the twitters originally about the mysql cookbook blowing up on
failed initial runs on chef-solo (all related to the preseeded
passwords). Fletcher evidently has a PR coming about explicitly
failing out on chef-solo unless you specify all the passwords in your
node json. That’s a bit of a different discussion. What I wanted to
noodle out is the possibility of a chef-solo friendly node.save. There
were few options that came up:

  • Write back to the node json used during the call
  • Write somewhere else to disk

The first one is fraught with edge cases and violating expectations
when mutating that file. What if it’s under version control? What if
it’s immutable in some way? The list goes on. I think we can sort of
agree it’s a bad idea. So the second one came up. My thought is that
node.save when called under chef-solo should dump the node attributes
to the cache path as something like node-data.json (or possibly
something a bit more identifiable if you want to utilize the node data
between chef-solo runs on other nodes.

I realize that chef-solo will never replace server. I don’t think
Opscode would have objections to community-originated PRs for adding
some kind of functionality like this? It would, however, allow us to
unify how we create recipes that are friendly to both solo and server.
It also has, as Ranjib Dey pointed out on twitter, immense benefit to
the various testing initiatives out there.

I’d also like to bring up a use case for reference that we have at
enStratus:

Much like OPC (as I understand it), we use chef-solo as our
"installer". enStratus is fairly modular and while the default mode of
installation is to collocate everything on a single machine, obviously
that doesn’t work. We typically support a single, two and four node
configuration of our stack for customer installs. Right now we use a
bit of a wrapper script and some manual error-prone data editing to
perform the two and four node installs. What would be nice is if we
could choose to persist node data between runs - primarily because of
the mysql cookbook thing - but also because we need to track some very
simple things like the mysql server IP but some passwords and
encryption keys. I having each node’s data persisted to
cache/node-state/node{1-4}.json and loaded would be a godsend for
making these things more idempotent.

Anyway, just looking for feedback/thoughts.

Thanks
John E. Vincent
@lusis


#5

John, Alex,

The technique is useful in the chef-solo world, where writing the node data
is impossible.

In the chef-client world, the technique is still useful, but you can adjust
it a little bit as well. Instead of backing the password variable with a
file, you can back it with a normal-level node attribute. In this system,
in the first chef-client run, the recipe generates the password and copies
it to a normal-level node attribute, while in the subsequent chef-client
runs, the recipe reads it from that same normal-level node attribute. Or
you can still back it with a file, but also copy it to a default-level or
override-level node attribute during each chef-client run.

Alternatively, you can simply pre-generate the password yourself and add it
to the node data in chef-solo or chef-client. Then all the recipe has to do
is look at the well-known node attribute, and the whole business in the
recipe of backing the password variable with a file becomes unnecessary.
You can also move the pre-generated password from the node data to a
data-bag item if you wish, since fetching a data-bag item by name is
supported in both chef-solo and chef-client.

Cheers,
Jay

On Wed, Oct 3, 2012 at 10:07 PM, John Vincent
lusis.org+chef-dev@gmail.comwrote:

On Wed, Oct 3, 2012 at 2:50 PM, Jay Feldblum y_feldblum@yahoo.com wrote:

John, Alex,

Chef isn’t really an interactive or back-and-forth conversation between
recipes and the node data. The Chef model is much more linear than that.

I wouldn’t disagree with that per se.

As you know, the general Chef model is that you specify the whole
abstract
configuration (run-lists, node-data) up front, and then recipes just
expand
it out by declaring resources based on the abstract configuration, and
then
chef finishes the job by converging the server to match the declared
resources.

Still with you.

If you need to generate a password during one run and save it on the
server
for use during the next run, you can use a simple trick as follows:

You can have a well-known password-storage file wherever you like (under
/var/chef or wherever else you like). During the recipe run (“compilation
phase”) you can test for the existence of the file. If it does not exist,
you generate a new password into a local variable and write it to the
file
(still during the “compilation phase”); that’s what happens during the
first
run. If it does exist, you read the password from the file into a local
variable; that’s what happens during all subsequent runs. After that
step,
you can always copy the password into the node data, but of course now
there’s no longer any need to save the new node data, because it will
always
be recomputed exactly with the same deterministic input and with the same
deterministic algorithm on every subsequent chef run.

Here’s where I’d disagree with you. If a node existed in totality on
its own, that would make AWESOME sense. Let’s take a scenario though.

I have a DB server. I have an app server. Under the current Chef model
(and the design of the official Opscode cookbooks) is that the node
NEEDS the resource, declares the resource. If I need a db created, a
migration run or a user created the consumer of that resource (in this
case the app server) does that work under its Chef cookbooks. Mind you
I’d argue that the creator of that resource should be the DB server.
However that’s just not possible in the current Chef world. I could
use Noah to do the orchestration but even I’d admit it’s kludgy. What
we essentially are faced with his how to declare a resource on another
node from the node that needs it.

So in our case, we have the declaration of databases and app users in
the recipe for each application. That makes sense in a logical
grouping. The DB Server role is responsible for standing up MySQL and
creating the base state. The application role is responsible for
creating the databases it needs. But to do that, it needs to know the
root password. The root password that lives on the db server - under
the node data for that node.

Sure we can refactor a bit and move the application specific DB
information into a role that runs on the db server but that doesn’t
fit how some folks think - that the database doesn’t belong to the db
server but to the application using it. I’ve done it both ways and I
honestly don’t know which way is entirely correct but it’s much easier
to minimize the need for chef runs to be coordinated between nodes.

Did that make any sense at all cause I’m having to reread what I wrote
myself :wink:

Cheers,
Jay

On Wed, Oct 3, 2012 at 4:19 PM, Alex Soto apsoto@gmail.com wrote:

I’m surprised that issue is still around. I hit that years ago and
just ended up not using solo since.

However, I’d vote for writing to a ‘new’ json file and that the
original node.json and the new one are merged during the run with the
new json overriding node.json. That way the original is not messed
with and you have the final end result available as a json file for
future runs.

Alex

On Wed, Oct 3, 2012 at 12:51 PM, John Vincent
lusis.org+chef-dev@gmail.com wrote:

Hey all,

Fletcher Nichol, Timberlake (jk) and I were bantering back and forth
on the twitters originally about the mysql cookbook blowing up on
failed initial runs on chef-solo (all related to the preseeded
passwords). Fletcher evidently has a PR coming about explicitly
failing out on chef-solo unless you specify all the passwords in your
node json. That’s a bit of a different discussion. What I wanted to
noodle out is the possibility of a chef-solo friendly node.save. There
were few options that came up:

  • Write back to the node json used during the call
  • Write somewhere else to disk

The first one is fraught with edge cases and violating expectations
when mutating that file. What if it’s under version control? What if
it’s immutable in some way? The list goes on. I think we can sort of
agree it’s a bad idea. So the second one came up. My thought is that
node.save when called under chef-solo should dump the node attributes
to the cache path as something like node-data.json (or possibly
something a bit more identifiable if you want to utilize the node data
between chef-solo runs on other nodes.

I realize that chef-solo will never replace server. I don’t think
Opscode would have objections to community-originated PRs for adding
some kind of functionality like this? It would, however, allow us to
unify how we create recipes that are friendly to both solo and server.
It also has, as Ranjib Dey pointed out on twitter, immense benefit to
the various testing initiatives out there.

I’d also like to bring up a use case for reference that we have at
enStratus:

Much like OPC (as I understand it), we use chef-solo as our
"installer". enStratus is fairly modular and while the default mode of
installation is to collocate everything on a single machine, obviously
that doesn’t work. We typically support a single, two and four node
configuration of our stack for customer installs. Right now we use a
bit of a wrapper script and some manual error-prone data editing to
perform the two and four node installs. What would be nice is if we
could choose to persist node data between runs - primarily because of
the mysql cookbook thing - but also because we need to track some very
simple things like the mysql server IP but some passwords and
encryption keys. I having each node’s data persisted to
cache/node-state/node{1-4}.json and loaded would be a godsend for
making these things more idempotent.

Anyway, just looking for feedback/thoughts.

Thanks
John E. Vincent
@lusis


#6

On Wed, Oct 3, 2012 at 7:34 PM, Jay Feldblum y_feldblum@yahoo.com wrote:

John, Alex,

The technique is useful in the chef-solo world, where writing the node data
is impossible.

In the chef-client world, the technique is still useful, but you can adjust
it a little bit as well. Instead of backing the password variable with a
file, you can back it with a normal-level node attribute. In this system, in
the first chef-client run, the recipe generates the password and copies it
to a normal-level node attribute, while in the subsequent chef-client runs,
the recipe reads it from that same normal-level node attribute. Or you can
still back it with a file, but also copy it to a default-level or
override-level node attribute during each chef-client run.

Alternatively, you can simply pre-generate the password yourself and add it
to the node data in chef-solo or chef-client. Then all the recipe has to do
is look at the well-known node attribute, and the whole business in the
recipe of backing the password variable with a file becomes unnecessary. You
can also move the pre-generated password from the node data to a data-bag
item if you wish, since fetching a data-bag item by name is supported in
both chef-solo and chef-client.

Something that I probably didn’t make clear is that we have a few
goals/objectives. I’m pasting this from a private email I sent someone
since it’s relevant here:

We're trying to accomplish a lot sure but we're also trying to minimize the "moving parts" and disconnect. I'll try and explain:
  • We use chef for the production hosted version of our app but a
    different set of cookbooks for now
  • We use chef-solo for our POC installs with another entirely different repo
  • We want to leverage the official opscode cookbooks as much as
    possible (limits our development effort).

We eventually want to unify the cookbooks that we use to install our
app - across the hosted platform, the POCs and the on-premise
installs. I don’t think that’s unreasonable. We can’t require that the
customer set up a Chef server. I FULLY believe that chef-solo and
puppet-apply are pretty much the best way to install software instead
of middling about with complicated unweildy shell scripts and trying
to handle OS differences, escaping command-lines and shit like that.
It also allows us to “cross-train” sort of. When you grok the solo
installer, you grok how production works.

So in my case I’d like to minimize the “customization” we need to
perform to official cookbooks as much as possible. If we customize it
in any way, we try and limit it to stuff that makes sense to push
upstream.

There’s a fair argument to be had here that chef-solo will never be
enough but we’ve gotten this far without any major headaches other
than the specific thing with the mysql password. In fact it’s cut our
POC and customer install times down to almost nothing. It gives us
MORE time to focus on integrating the product and not fighting
installation headaches.

Cheers,
Jay

On Wed, Oct 3, 2012 at 10:07 PM, John Vincent lusis.org+chef-dev@gmail.com
wrote:

On Wed, Oct 3, 2012 at 2:50 PM, Jay Feldblum y_feldblum@yahoo.com wrote:

John, Alex,

Chef isn’t really an interactive or back-and-forth conversation between
recipes and the node data. The Chef model is much more linear than that.

I wouldn’t disagree with that per se.

As you know, the general Chef model is that you specify the whole
abstract
configuration (run-lists, node-data) up front, and then recipes just
expand
it out by declaring resources based on the abstract configuration, and
then
chef finishes the job by converging the server to match the declared
resources.

Still with you.

If you need to generate a password during one run and save it on the
server
for use during the next run, you can use a simple trick as follows:

You can have a well-known password-storage file wherever you like (under
/var/chef or wherever else you like). During the recipe run
(“compilation
phase”) you can test for the existence of the file. If it does not
exist,
you generate a new password into a local variable and write it to the
file
(still during the “compilation phase”); that’s what happens during the
first
run. If it does exist, you read the password from the file into a local
variable; that’s what happens during all subsequent runs. After that
step,
you can always copy the password into the node data, but of course now
there’s no longer any need to save the new node data, because it will
always
be recomputed exactly with the same deterministic input and with the
same
deterministic algorithm on every subsequent chef run.

Here’s where I’d disagree with you. If a node existed in totality on
its own, that would make AWESOME sense. Let’s take a scenario though.

I have a DB server. I have an app server. Under the current Chef model
(and the design of the official Opscode cookbooks) is that the node
NEEDS the resource, declares the resource. If I need a db created, a
migration run or a user created the consumer of that resource (in this
case the app server) does that work under its Chef cookbooks. Mind you
I’d argue that the creator of that resource should be the DB server.
However that’s just not possible in the current Chef world. I could
use Noah to do the orchestration but even I’d admit it’s kludgy. What
we essentially are faced with his how to declare a resource on another
node from the node that needs it.

So in our case, we have the declaration of databases and app users in
the recipe for each application. That makes sense in a logical
grouping. The DB Server role is responsible for standing up MySQL and
creating the base state. The application role is responsible for
creating the databases it needs. But to do that, it needs to know the
root password. The root password that lives on the db server - under
the node data for that node.

Sure we can refactor a bit and move the application specific DB
information into a role that runs on the db server but that doesn’t
fit how some folks think - that the database doesn’t belong to the db
server but to the application using it. I’ve done it both ways and I
honestly don’t know which way is entirely correct but it’s much easier
to minimize the need for chef runs to be coordinated between nodes.

Did that make any sense at all cause I’m having to reread what I wrote
myself :wink:

Cheers,
Jay

On Wed, Oct 3, 2012 at 4:19 PM, Alex Soto apsoto@gmail.com wrote:

I’m surprised that issue is still around. I hit that years ago and
just ended up not using solo since.

However, I’d vote for writing to a ‘new’ json file and that the
original node.json and the new one are merged during the run with the
new json overriding node.json. That way the original is not messed
with and you have the final end result available as a json file for
future runs.

Alex

On Wed, Oct 3, 2012 at 12:51 PM, John Vincent
lusis.org+chef-dev@gmail.com wrote:

Hey all,

Fletcher Nichol, Timberlake (jk) and I were bantering back and forth
on the twitters originally about the mysql cookbook blowing up on
failed initial runs on chef-solo (all related to the preseeded
passwords). Fletcher evidently has a PR coming about explicitly
failing out on chef-solo unless you specify all the passwords in your
node json. That’s a bit of a different discussion. What I wanted to
noodle out is the possibility of a chef-solo friendly node.save.
There
were few options that came up:

  • Write back to the node json used during the call
  • Write somewhere else to disk

The first one is fraught with edge cases and violating expectations
when mutating that file. What if it’s under version control? What if
it’s immutable in some way? The list goes on. I think we can sort of
agree it’s a bad idea. So the second one came up. My thought is that
node.save when called under chef-solo should dump the node attributes
to the cache path as something like node-data.json (or possibly
something a bit more identifiable if you want to utilize the node
data
between chef-solo runs on other nodes.

I realize that chef-solo will never replace server. I don’t think
Opscode would have objections to community-originated PRs for adding
some kind of functionality like this? It would, however, allow us to
unify how we create recipes that are friendly to both solo and
server.
It also has, as Ranjib Dey pointed out on twitter, immense benefit to
the various testing initiatives out there.

I’d also like to bring up a use case for reference that we have at
enStratus:

Much like OPC (as I understand it), we use chef-solo as our
"installer". enStratus is fairly modular and while the default mode
of
installation is to collocate everything on a single machine,
obviously
that doesn’t work. We typically support a single, two and four node
configuration of our stack for customer installs. Right now we use a
bit of a wrapper script and some manual error-prone data editing to
perform the two and four node installs. What would be nice is if we
could choose to persist node data between runs - primarily because of
the mysql cookbook thing - but also because we need to track some
very
simple things like the mysql server IP but some passwords and
encryption keys. I having each node’s data persisted to
cache/node-state/node{1-4}.json and loaded would be a godsend for
making these things more idempotent.

Anyway, just looking for feedback/thoughts.

Thanks
John E. Vincent
@lusis


#7

Hi,

On Thu, Oct 4, 2012 at 12:07 PM, John Vincent
lusis.org+chef-dev@gmail.com wrote:

I have a DB server. I have an app server. Under the current Chef model
(and the design of the official Opscode cookbooks) is that the node
NEEDS the resource, declares the resource. If I need a db created, a
migration run or a user created the consumer of that resource (in this
case the app server) does that work under its Chef cookbooks. Mind you
I’d argue that the creator of that resource should be the DB server.
However that’s just not possible in the current Chef world. I could
use Noah to do the orchestration but even I’d admit it’s kludgy. What
we essentially are faced with his how to declare a resource on another
node from the node that needs it.

This is how we do it for some of our applications. Let’s say we have a
node that runs a service that provides resources to other
applications. Say we have a node that runs a message broker that hosts
topics and queues for other applications. We have N application nodes
that as part of their converge will publish override attributes that
declare their requirement for a topic or a queue. The next time the
broker node converges it uses search to pick up all the resources
requested by application nodes and then uses that data to drive the
configuration of the set of queues/topics that the broker hosts. The
releases is often a multi converge process where we first converge the
application nodes (so they publish requirements and disable the app
during an upgrade), converge the resource provider nodes (creating the
resources and possibly doing migrations), then converge the
application nodes (bringing them online using the new resources).

of course we rely fairly heavily on chef server to do this…


Cheers,

Peter Donald


#8

Good to know I’m not the only one then. FWIW, I was just using these
as examples. The real question is still if anyone has any
objects/ideas about allowing node.save from solo. It would definitely
make recipes more reusable I think.

On Wed, Oct 3, 2012 at 9:16 PM, Peter Donald peter@realityforge.org wrote:

Hi,

On Thu, Oct 4, 2012 at 12:07 PM, John Vincent
lusis.org+chef-dev@gmail.com wrote:

I have a DB server. I have an app server. Under the current Chef model
(and the design of the official Opscode cookbooks) is that the node
NEEDS the resource, declares the resource. If I need a db created, a
migration run or a user created the consumer of that resource (in this
case the app server) does that work under its Chef cookbooks. Mind you
I’d argue that the creator of that resource should be the DB server.
However that’s just not possible in the current Chef world. I could
use Noah to do the orchestration but even I’d admit it’s kludgy. What
we essentially are faced with his how to declare a resource on another
node from the node that needs it.

This is how we do it for some of our applications. Let’s say we have a
node that runs a service that provides resources to other
applications. Say we have a node that runs a message broker that hosts
topics and queues for other applications. We have N application nodes
that as part of their converge will publish override attributes that
declare their requirement for a topic or a queue. The next time the
broker node converges it uses search to pick up all the resources
requested by application nodes and then uses that data to drive the
configuration of the set of queues/topics that the broker hosts. The
releases is often a multi converge process where we first converge the
application nodes (so they publish requirements and disable the app
during an upgrade), converge the resource provider nodes (creating the
resources and possibly doing migrations), then converge the
application nodes (bringing them online using the new resources).

of course we rely fairly heavily on chef server to do this…


Cheers,

Peter Donald


#9

John,

I would not necessarily use node.save from chef-client either. Relying on
that makes too many assumptions that everything goes right on every
chef-client run. But of course this is a distributed system designed to be
tolerant of failures and to handle failures well, so making too many
assumptions that the chef-client run always succeeds can clash with chef’s
own assumptions about how your recipes behave.

Another approach might be to rely on the automatic node.save done at the
end of every chef-client run automatically, and during the recipe simply
adding data to the node attributes to be saved later without actually
performing the save. If it’s vital that some bit of data be remembered
until the next time the node is saved, you can use the file-backed variable
technique from earlier. This technique can also be thought of as an analog
of a write-ahead log.

Cheers,
Jay

On Thu, Oct 4, 2012 at 12:43 AM, John Vincent
lusis.org+chef-dev@gmail.comwrote:

Good to know I’m not the only one then. FWIW, I was just using these
as examples. The real question is still if anyone has any
objects/ideas about allowing node.save from solo. It would definitely
make recipes more reusable I think.

On Wed, Oct 3, 2012 at 9:16 PM, Peter Donald peter@realityforge.org
wrote:

Hi,

On Thu, Oct 4, 2012 at 12:07 PM, John Vincent
lusis.org+chef-dev@gmail.com wrote:

I have a DB server. I have an app server. Under the current Chef model
(and the design of the official Opscode cookbooks) is that the node
NEEDS the resource, declares the resource. If I need a db created, a
migration run or a user created the consumer of that resource (in this
case the app server) does that work under its Chef cookbooks. Mind you
I’d argue that the creator of that resource should be the DB server.
However that’s just not possible in the current Chef world. I could
use Noah to do the orchestration but even I’d admit it’s kludgy. What
we essentially are faced with his how to declare a resource on another
node from the node that needs it.

This is how we do it for some of our applications. Let’s say we have a
node that runs a service that provides resources to other
applications. Say we have a node that runs a message broker that hosts
topics and queues for other applications. We have N application nodes
that as part of their converge will publish override attributes that
declare their requirement for a topic or a queue. The next time the
broker node converges it uses search to pick up all the resources
requested by application nodes and then uses that data to drive the
configuration of the set of queues/topics that the broker hosts. The
releases is often a multi converge process where we first converge the
application nodes (so they publish requirements and disable the app
during an upgrade), converge the resource provider nodes (creating the
resources and possibly doing migrations), then converge the
application nodes (bringing them online using the new resources).

of course we rely fairly heavily on chef server to do this…


Cheers,

Peter Donald


#10

On Wed, Oct 3, 2012 at 10:34 PM, Jay Feldblum y_feldblum@yahoo.com wrote:

John,

I would not necessarily use node.save from chef-client either. Relying on
that makes too many assumptions that everything goes right on every
chef-client run. But of course this is a distributed system designed to be
tolerant of failures and to handle failures well, so making too many
assumptions that the chef-client run always succeeds can clash with chef’s
own assumptions about how your recipes behave.

Yeah I’m not a big fan of calling node.save myself. The problem is
that you HAVE to do it in several places. The mysql cookbook is a
really good example. This wasn’t an issue until the node.save was
moved to only happening at the end of a successful first run. But in
the case of the mysql cookbook, the problem is that if the initial run
of mysql::server fails, it’s impossible (or almost) to recover from.
The install was done with a preseeded randomly generated value for the
passwords on debian clones. There’s no way to rerun that cookbook and
have it pick up where it left off if it fails at some point during the
run of the mysql::server recipe unless you save the node attributes
right after you generate that random password.

Mind you, I’m a big proponent of (and performer of) testing FULL
bootstrap after every change. I design where I can for a bootstrap to
bring me fully into service as a given role. The only way to do that
is to know that the bootstrap will never fail when I need it and I can
go right into service when I need it.

Another approach might be to rely on the automatic node.save done at the end
of every chef-client run automatically, and during the recipe simply adding
data to the node attributes to be saved later without actually performing
the save. If it’s vital that some bit of data be remembered until the next
time the node is saved, you can use the file-backed variable technique from
earlier. This technique can also be thought of as an analog of a write-ahead
log.

Cheers,
Jay

On Thu, Oct 4, 2012 at 12:43 AM, John Vincent lusis.org+chef-dev@gmail.com
wrote:

Good to know I’m not the only one then. FWIW, I was just using these
as examples. The real question is still if anyone has any
objects/ideas about allowing node.save from solo. It would definitely
make recipes more reusable I think.

On Wed, Oct 3, 2012 at 9:16 PM, Peter Donald peter@realityforge.org
wrote:

Hi,

On Thu, Oct 4, 2012 at 12:07 PM, John Vincent
lusis.org+chef-dev@gmail.com wrote:

I have a DB server. I have an app server. Under the current Chef model
(and the design of the official Opscode cookbooks) is that the node
NEEDS the resource, declares the resource. If I need a db created, a
migration run or a user created the consumer of that resource (in this
case the app server) does that work under its Chef cookbooks. Mind you
I’d argue that the creator of that resource should be the DB server.
However that’s just not possible in the current Chef world. I could
use Noah to do the orchestration but even I’d admit it’s kludgy. What
we essentially are faced with his how to declare a resource on another
node from the node that needs it.

This is how we do it for some of our applications. Let’s say we have a
node that runs a service that provides resources to other
applications. Say we have a node that runs a message broker that hosts
topics and queues for other applications. We have N application nodes
that as part of their converge will publish override attributes that
declare their requirement for a topic or a queue. The next time the
broker node converges it uses search to pick up all the resources
requested by application nodes and then uses that data to drive the
configuration of the set of queues/topics that the broker hosts. The
releases is often a multi converge process where we first converge the
application nodes (so they publish requirements and disable the app
during an upgrade), converge the resource provider nodes (creating the
resources and possibly doing migrations), then converge the
application nodes (bringing them online using the new resources).

of course we rely fairly heavily on chef server to do this…


Cheers,

Peter Donald


#11

On Thu, Oct 4, 2012 at 12:43 PM, John Vincent
lusis.org+chef-dev@gmail.com wrote:

Yeah I’m not a big fan of calling node.save myself. The problem is
that you HAVE to do it in several places. The mysql cookbook is a
really good example. This wasn’t an issue until the node.save was
moved to only happening at the end of a successful first run. But in
the case of the mysql cookbook, the problem is that if the initial run
of mysql::server fails, it’s impossible (or almost) to recover from.
The install was done with a preseeded randomly generated value for the
passwords on debian clones. There’s no way to rerun that cookbook and
have it pick up where it left off if it fails at some point during the
run of the mysql::server recipe unless you save the node attributes
right after you generate that random password.

As an aside… I’ve never been comfortable with recipes storing
autogenerated passwords as node attributes. In this particular
example, I’d like Chef to bring up the database, then create database
users with credentials I’ve supplied in encrypted data bags. I
neither need nor want superuser credentials to be stored in the clear
elsewhere on the network.

Besides generated passwords, what else might a recipe want to record
for later reference? I’m struggling to find good examples. In my own
chef-repos, the only real example is Venda’s ssl_certificate
resource[1], which stores generated CSRs in node attributes. Data bag
items would have made more sense, but doing this on the open-source
Chef server would have involved all nodes being admin clients. Making
that resource work with chef-solo would involve something to ship CSRs
from nodes to a central location, at which point it wouldn’t be a big
deal to have the resource write out to disk itself.

We might also consider Dan Carley’s “Unautomating With Puppet”[2]
post, which makes a case for keeping some local state to help the CM
tool detect when it is being asked to make a dangerous change.
Implementing this in Chef would involve saving state apart from node
attributes.

Given all that, I wonder whether we’re defining the problem correctly.
Is the real issue with node.save’s absence in chef-solo (and its
consequences in chef-client), or is the "initial database password"
problem representative of a use case that wants for a different
solution altogether?

Zac

1: https://github.com/VendaTech/chef-cookbook-ssl
2: http://dan.carley.co/blog/2012/10/01/unautomating-with-puppet/


#12

On 10/4/12 2:14 AM, Zac Stevens wrote:

Given all that, I wonder whether we’re defining the problem correctly.
Is the real issue with node.save’s absence in chef-solo (and its
consequences in chef-client), or is the "initial database password"
problem representative of a use case that wants for a different
solution altogether?

This is my thought as well. I’ve done a fair amount with Chef Server and
Solo in conjunction with small projects and toolchains. I think
chef-solo beats the hell out of a pile of shell scripts for managing an
orchestrated environment install, but I think it’s not the entire answer
to the question.

No one can fangirl flail harder than I do over Chef and how much I love
it. But Solo is not an orchestration tool and all the solutions I’ve
been part of that use it in a toolchain have to do some hacky stuff to
make it work. One of the things I’m seeing more of out there are
orchestration tools that are tool aware, like Go and Maestro. But
everything relies on an agent existing on your target servers, which
doesn’t address the initial bootstrapping where you might not have this.

For the specific issue of the password, addressing the ephemeral nature
of proof of concept environments, I don’t see anything wrong with
pre-defining a plain-text password for things and either running follow
up configuration management to update the password later or just not.
It’s a POC and should only be around to prove something out. If you
really need to keep it around for a while, set up something to change
the password after the install.

If it were me, and my customer’s focus were already cloud-based, I’d
have a private Jenkins, or Go or I hear Red Hat has a hosted
orchestration service these days. I’ve also used Rundeck for this
missing piece. I would use this tool to fill in the missing pieces
where chef-solo wasn’t meant to go. You could even have your last step
be to create a cron job that deletes the orchestration agent 30 minutes
after the end of a successful build if you want to cut ties with it.

Your orchestration should be as well thought out as your application and
OS configuration strategy. If Solo is being used to configure servers,
how are things being provisioned in the first place? Knife scripts? AWS
shell scripts? I have a lot of thoughts on this because I’ve just spent
the last couple of weeks working out how to set up CI to: create
internal Openstack VMs and external RDS Databases, check out specific
branches of code, build a project, run Chef on the VMs based on
information passed in to the orchestration tool, and have several JBoss
JVMs start up successfully. We’re doing it with Jenkins and a target
environment attribute with databags keyed to it.

When all you’re using is chef-solo, everything starts to look like a
chef-nail. But sometimes you need a bigger toolkit and adding a second
tool to your repertoire will be less hacky, more beautiful and probably
not take any more time to figure out than shoe-horning all your configs
into chef-solo.

Ok, now I’m late for work. (officelife)

Sascha Bates | sascha.bates@gmail.com | 612 850 0444 |
sascha.bates@skype | sascha_bates@yahoo |