How to manage databases with chef?


#1

What are some best practices for coordinating cookbooks on several different servers? In my example, I have an application (owncloud) that also needs to create a database on another (MySQL) server. I do not want to allow remote root access to that MySQL server.

So the process should be something along the lines of:

  •      On owncloud server: chef-client install owncloud binaries and somehow communicates with chef-client on the database server
    
  •      On db server: create owncloud database and a user to access this database and communicates the user name and password back.
    
  •      On owncloud server: continue owncloud configuration
    

How is this type of scenario best handled in chef?

Thanks!


#2

Personally, I would not set up the database using the CHEF.

I would set up her hand, and would have used the logs and password to
access it on the client owncloud.

This is simple workaround and easy case.

Data is essentially static.

In my opinion in the simplest case - one database:
Better fulfill its setting machine with MYSQL manually. And the using
hostname, username, password with chef-client to configure data client
(owncloud) application.

Previously, I set up a base through CHEF.
But later it turned out that for personal use(or simple cases) is an
overhead.

Excusme for bad english.

2014-08-19 15:35 GMT+04:00 subscription@kkeane.com:

What are some best practices for coordinating cookbooks on several
different servers? In my example, I have an application (owncloud) that
also needs to create a database on another (MySQL) server. I do not want to
allow remote root access to that MySQL server.

So the process should be something along the lines of:

  •      On owncloud server: chef-client install owncloud binaries and
    

somehow communicates with chef-client on the database server

  •      On db server: create owncloud database and a user to access
    

this database and communicates the user name and password back.

  •      On owncloud server: continue owncloud configuration
    

How is this type of scenario best handled in chef?

Thanks!


Faithfully yours,

CVision Lab System Administrator
Vladimir Skubriev


#3

Coordination as such isn’t required - all you need to make sure of is that
MySQL is installed on the DB server and has the user setup before
initializing owncloud. There are several ways to do this but here is one
possible way:

  1. Run the mysql::server recipe [1] on the DB server (if mysql is not
    already installed) and have it create your user
  2. Have the owncloud recipe include a “mysql_database” resource block that
    creates the owncloud DB using the user creds
  3. Let the owncloud recipe do the rest

There is prior work here so I’d give this repo [2] a look but note that it
assumes that the database is on localhost. It should be easy enough to fork
or create your own cookbook. As for protecting secrets one could use
encrypted databags/chef-vault/citadel but that is easily it’s own
discussion [3].

[1] https://github.com/opscode-cookbooks/mysql
[2] https://github.com/onddo/owncloud-cookbook
[3] https://coderanger.net/chef-secrets/

Cheers,
Seth Thomas

On Tue, Aug 19, 2014 at 4:35 AM, subscription@kkeane.com wrote:

What are some best practices for coordinating cookbooks on several
different servers? In my example, I have an application (owncloud) that
also needs to create a database on another (MySQL) server. I do not want to
allow remote root access to that MySQL server.

So the process should be something along the lines of:

  •      On owncloud server: chef-client install owncloud binaries and
    

somehow communicates with chef-client on the database server

  •      On db server: create owncloud database and a user to access
    

this database and communicates the user name and password back.

  •      On owncloud server: continue owncloud configuration
    

How is this type of scenario best handled in chef?

Thanks!


#4

Thank you for the response! First off, normally I would agree with Vladimir’s suggestion – creating a database manually is indeed the easiest way to handle this specific situation. Another easy solution would be to simply keep the database on the same server.

But I was looking for a more generic solution, because I have run into similar types of coordination issue on several other occasions before (and usually resolved them manually)

As for your suggestion, Seth, I am not quite sure I follow how this should work. Step 1 is easy enough. Step 2 is the problem: the owncloud recipe would not be running on the database server, but on the owncloud server which, in my scenario, doesn’t even have permission to create databases. Yes, of course I know that it is possible to create a database remotely with the right permissions, but for a number of reasons, I don’t want to do it that way.

The main reason is that this is really just an example for the more generic scenario. The other reason is that I prefer to keep root access to MySQL limited to the database server.

So the more generic problem is:

Chef recipe runs on server 1, but the final state, and thus the success or failure of the recipe, also depends on something being done on server 2. I’m sure that this is a fairly common scenario; I’m just wondering how others handle it.

From: Seth Thomas [mailto:sthomas@opscode.com]
Sent: Tuesday, August 19, 2014 9:32 AM
To: chef@lists.opscode.com
Subject: [chef] Re: How to manage databases with chef?

Coordination as such isn’t required - all you need to make sure of is that MySQL is installed on the DB server and has the user setup before initializing owncloud. There are several ways to do this but here is one possible way:

  1. Run the mysql::server recipe [1] on the DB server (if mysql is not already installed) and have it create your user

  2. Have the owncloud recipe include a “mysql_database” resource block that creates the owncloud DB using the user creds

  3. Let the owncloud recipe do the rest

There is prior work here so I’d give this repo [2] a look but note that it assumes that the database is on localhost. It should be easy enough to fork or create your own cookbook. As for protecting secrets one could use encrypted databags/chef-vault/citadel but that is easily it’s own discussion [3].

[1] https://github.com/opscode-cookbooks/mysql

[2] https://github.com/onddo/owncloud-cookbook

[3] https://coderanger.net/chef-secrets/

Cheers,

Seth Thomas

On Tue, Aug 19, 2014 at 4:35 AM, <subscription@kkeane.com mailto:subscription@kkeane.com > wrote:

What are some best practices for coordinating cookbooks on several different servers? In my example, I have an application (owncloud) that also needs to create a database on another (MySQL) server. I do not want to allow remote root access to that MySQL server.

So the process should be something along the lines of:

  •      On owncloud server: chef-client install owncloud binaries and somehow communicates with chef-client on the database server
    
  •      On db server: create owncloud database and a user to access this database and communicates the user name and password back.
    
  •      On owncloud server: continue owncloud configuration
    

How is this type of scenario best handled in chef?

Thanks!


#5

On Wed, Aug 20, 2014 at 5:22 AM, Kevin Keane Subscription
subscription@kkeane.com wrote:

Thank you for the response! First off, normally I would agree with
Vladimir’s suggestion – creating a database manually is indeed the easiest
way to handle this specific situation. Another easy solution would be to
simply keep the database on the same server.

But I was looking for a more generic solution, because I have run into
similar types of coordination issue on several other occasions before (and
usually resolved them manually)

As for your suggestion, Seth, I am not quite sure I follow how this should
work. Step 1 is easy enough. Step 2 is the problem: the owncloud recipe
would not be running on the database server, but on the owncloud server
which, in my scenario, doesn’t even have permission to create databases.
Yes, of course I know that it is possible to create a database remotely with
the right permissions, but for a number of reasons, I don’t want to do it
that way.

The main reason is that this is really just an example for the more generic
scenario. The other reason is that I prefer to keep root access to MySQL
limited to the database server.

So the more generic problem is:

Chef recipe runs on server 1, but the final state, and thus the success or
failure of the recipe, also depends on something being done on server 2. I’m
sure that this is a fairly common scenario; I’m just wondering how others
handle it.

In my view, this kind of problem is best solved by designing it as a
system of autonomous actors a la Mark Burgess promise theory.

In other words, perform the database operations solely on the database
server, and don’t introduce cross-node dependencies or "orchestration"
unnecessarily. Design the application that is connecting to the
database in such a way that it is resilient to errors like not knowing
what its username and password are, inabillity to connect to the
database yet, etc. As soon as it gets that information and has all its
required operational parameters, it automatically recovers and begins
receiving live requests.

  • Julian


[ Julian C. Dunn jdunn@aquezada.com * Sorry, I’m ]
[ WWW: http://www.aquezada.com/staff/julian * only Web 1.0 ]
[ gopher://sdf.org/1/users/keymaker/ * compliant! ]
[ PGP: 91B3 7A9D 683C 7C16 715F 442C 6065 D533 FDC2 05B9 ]


#6

Exactly what Juliann said.

I wouldn’t have the database operations be done remotely, a recipe would
run on the database server to setup the database and a recipe would run on
the owncloud web frontend boxes. The owncloud web recipe would just need to
know a hostname/IP to point at and would simply work once the database was
up/created. No coordination required, it would just work once both recipes
had converged properly.

On Wed, Aug 20, 2014 at 7:38 AM, Julian C. Dunn jdunn@aquezada.com wrote:

On Wed, Aug 20, 2014 at 5:22 AM, Kevin Keane Subscription
subscription@kkeane.com wrote:

Thank you for the response! First off, normally I would agree with
Vladimir’s suggestion – creating a database manually is indeed the
easiest
way to handle this specific situation. Another easy solution would be to
simply keep the database on the same server.

But I was looking for a more generic solution, because I have run into
similar types of coordination issue on several other occasions before
(and
usually resolved them manually)

As for your suggestion, Seth, I am not quite sure I follow how this
should
work. Step 1 is easy enough. Step 2 is the problem: the owncloud recipe
would not be running on the database server, but on the owncloud server
which, in my scenario, doesn’t even have permission to create databases.
Yes, of course I know that it is possible to create a database remotely
with
the right permissions, but for a number of reasons, I don’t want to do it
that way.

The main reason is that this is really just an example for the more
generic
scenario. The other reason is that I prefer to keep root access to MySQL
limited to the database server.

So the more generic problem is:

Chef recipe runs on server 1, but the final state, and thus the success
or
failure of the recipe, also depends on something being done on server 2.
I’m
sure that this is a fairly common scenario; I’m just wondering how others
handle it.

In my view, this kind of problem is best solved by designing it as a
system of autonomous actors a la Mark Burgess promise theory.

In other words, perform the database operations solely on the database
server, and don’t introduce cross-node dependencies or "orchestration"
unnecessarily. Design the application that is connecting to the
database in such a way that it is resilient to errors like not knowing
what its username and password are, inabillity to connect to the
database yet, etc. As soon as it gets that information and has all its
required operational parameters, it automatically recovers and begins
receiving live requests.

  • Julian


[ Julian C. Dunn jdunn@aquezada.com * Sorry, I’m ]
[ WWW: http://www.aquezada.com/staff/julian * only Web 1.0 ]
[ gopher://sdf.org/1/users/keymaker/ * compliant! ]
[ PGP: 91B3 7A9D 683C 7C16 715F 442C 6065 D533 FDC2 05B9 ]


#7

I call this “divide and conquer”.

Some time ago, I realized that to keep everything in one place is not
appropriate and it is difficult.
On an example of setting redmine I was convinced completely.
I think owncloud is something similar that a redmine, which consists of a
database and frontend.

Just so easy to manage!

But in my case using recipes to setup database server or data bases, when a
database server is one for more than one applications is highly difficult
and very dangerous.

2014-08-21 0:05 GMT+04:00 Seth Thomas sthomas@opscode.com:

Exactly what Juliann said.

I wouldn’t have the database operations be done remotely, a recipe would
run on the database server to setup the database and a recipe would run on
the owncloud web frontend boxes. The owncloud web recipe would just need to
know a hostname/IP to point at and would simply work once the database was
up/created. No coordination required, it would just work once both recipes
had converged properly.

On Wed, Aug 20, 2014 at 7:38 AM, Julian C. Dunn jdunn@aquezada.com
wrote:

On Wed, Aug 20, 2014 at 5:22 AM, Kevin Keane Subscription
subscription@kkeane.com wrote:

Thank you for the response! First off, normally I would agree with
Vladimir’s suggestion – creating a database manually is indeed the
easiest
way to handle this specific situation. Another easy solution would be to
simply keep the database on the same server.

But I was looking for a more generic solution, because I have run into
similar types of coordination issue on several other occasions before
(and
usually resolved them manually)

As for your suggestion, Seth, I am not quite sure I follow how this
should
work. Step 1 is easy enough. Step 2 is the problem: the owncloud recipe
would not be running on the database server, but on the owncloud server
which, in my scenario, doesn’t even have permission to create databases.
Yes, of course I know that it is possible to create a database remotely
with
the right permissions, but for a number of reasons, I don’t want to do
it
that way.

The main reason is that this is really just an example for the more
generic
scenario. The other reason is that I prefer to keep root access to MySQL
limited to the database server.

So the more generic problem is:

Chef recipe runs on server 1, but the final state, and thus the success
or
failure of the recipe, also depends on something being done on server

  1. I’m

sure that this is a fairly common scenario; I’m just wondering how
others
handle it.

In my view, this kind of problem is best solved by designing it as a
system of autonomous actors a la Mark Burgess promise theory.

In other words, perform the database operations solely on the database
server, and don’t introduce cross-node dependencies or "orchestration"
unnecessarily. Design the application that is connecting to the
database in such a way that it is resilient to errors like not knowing
what its username and password are, inabillity to connect to the
database yet, etc. As soon as it gets that information and has all its
required operational parameters, it automatically recovers and begins
receiving live requests.

  • Julian


[ Julian C. Dunn jdunn@aquezada.com * Sorry, I’m ]
[ WWW: http://www.aquezada.com/staff/julian * only Web 1.0 ]
[ gopher://sdf.org/1/users/keymaker/ * compliant! ]
[ PGP: 91B3 7A9D 683C 7C16 715F 442C 6065 D533 FDC2 05B9 ]


Faithfully yours,

CVision Lab System Administrator
Vladimir Skubriev