Chef11 HA

Mark_Pimentel · February 11, 2013, 3:04pm

Now that Chef11 has a postgres backend, is there anyone out there doing
replication between chef servers for HA and also consolidated data. We
will have a Chef Server in each datacenter and wanted to be able to access
the data of all nodes no matter which server we are connected to.

Any suggestions or best practices that we should know about before we
attempt this?

Also, how are the cookbooks stored so that we could replicate those also?

–
Thanks,

Mark

Seth_Falcon_01 · February 11, 2013, 6:17pm

Hi Mark,

On Feb 11, 2013, at 7:04 AM, Mark Pimentel wrote:

Also, how are the cookbooks stored so that we could replicate those also?

The Chef 11 Server uses the bookshelf component to store cookbook content. The default data directory is: /var/opt/chef-server/bookshelf/data. This can be configured via chef-server.rb config as:

bookshlef['data_dir'] = PATH

Currently all cookbook content files are stored as flat files in the data directory.

seth

Jesse_Campbell · February 11, 2013, 6:47pm

In chef 10, if the checksum of the uploaded cookbook content was already in
the couch database, then the file would not get uploaded to the disk.
Is this the same in chef 11? I'd like to use a replicated database without
needing to synchronize the chef bookshelf store.

On Mon, Feb 11, 2013 at 1:17 PM, Seth Falcon seth@opscode.com wrote:

Hi Mark,

On Feb 11, 2013, at 7:04 AM, Mark Pimentel wrote:

Also, how are the cookbooks stored so that we could replicate those also?

The Chef 11 Server uses the bookshelf component to store cookbook content.
The default data directory is: /var/opt/chef-server/bookshelf/data. This
can be configured via chef-server.rb config as:

bookshlef['data_dir'] = PATH

Currently all cookbook content files are stored as flat files in the data
directory.

seth

Mark_Pimentel · February 11, 2013, 7:03pm

But even if the checksum did change, wouldn't you need to sync the
cookbooks to another server also? As the actual file would not be present?

On Mon, Feb 11, 2013 at 1:47 PM, Jesse Campbell hikeit@gmail.com wrote:

In chef 10, if the checksum of the uploaded cookbook content was already
in the couch database, then the file would not get uploaded to the disk.
Is this the same in chef 11? I'd like to use a replicated database without
needing to synchronize the chef bookshelf store.

On Mon, Feb 11, 2013 at 1:17 PM, Seth Falcon seth@opscode.com wrote:

Hi Mark,

On Feb 11, 2013, at 7:04 AM, Mark Pimentel wrote:

Also, how are the cookbooks stored so that we could replicate those
also?

The Chef 11 Server uses the bookshelf component to store cookbook
content. The default data directory is:
/var/opt/chef-server/bookshelf/data. This can be configured via
chef-server.rb config as:

bookshlef['data_dir'] = PATH

Currently all cookbook content files are stored as flat files in the data
directory.

seth

--
Thanks,

Mark

Adam_Jacob · February 11, 2013, 7:05pm

Not going to be an option – the files themselves are stored outside of the database (which makes sense, if you think about the total havoc that would be released if you stored them in the database, even as blobs).

Private Chef has HA built in via DRBD, and its what we recommend to everyone. (HA active/passive, with as many front-end servers as you need.)

Best,
Adam

From: Jesse Campbell <hikeit@gmail.com mailto:hikeit@gmail.com>
Reply-To: "chef@lists.opscode.com mailto:chef@lists.opscode.com" <chef@lists.opscode.com mailto:chef@lists.opscode.com>
Date: Monday, February 11, 2013 10:47 AM
To: chef <chef@lists.opscode.com mailto:chef@lists.opscode.com>
Subject: [chef] Re: Re: Chef11 HA

In chef 10, if the checksum of the uploaded cookbook content was already in the couch database, then the file would not get uploaded to the disk.
Is this the same in chef 11? I'd like to use a replicated database without needing to synchronize the chef bookshelf store.

On Mon, Feb 11, 2013 at 1:17 PM, Seth Falcon <seth@opscode.com mailto:seth@opscode.com> wrote:
Hi Mark,

On Feb 11, 2013, at 7:04 AM, Mark Pimentel wrote:

Also, how are the cookbooks stored so that we could replicate those also?

The Chef 11 Server uses the bookshelf component to store cookbook content. The default data directory is: /var/opt/chef-server/bookshelf/data. This can be configured via chef-server.rb config as:

bookshlef['data_dir'] = PATH

Currently all cookbook content files are stored as flat files in the data directory.

seth

Jesse_Campbell · February 11, 2013, 7:13pm

My intention was to upload cookbooks to all servers that are replicating
that database, hoping that it would check the filesystem for the existence
of the checksum files, rather than just looking at the database.

-jesse

On Mon, Feb 11, 2013 at 2:05 PM, Adam Jacob adam@opscode.com wrote:

Not going to be an option – the files themselves are stored outside of
the database (which makes sense, if you think about the total havoc that
would be released if you stored them in the database, even as blobs).

Private Chef has HA built in via DRBD, and its what we recommend to
everyone. (HA active/passive, with as many front-end servers as you need.)

Best,
Adam

From: Jesse Campbell hikeit@gmail.com
Reply-To: "chef@lists.opscode.com" chef@lists.opscode.com
Date: Monday, February 11, 2013 10:47 AM
To: chef chef@lists.opscode.com
Subject: [chef] Re: Re: Chef11 HA

In chef 10, if the checksum of the uploaded cookbook content was
already in the couch database, then the file would not get uploaded to the
disk.
Is this the same in chef 11? I'd like to use a replicated database without
needing to synchronize the chef bookshelf store.

On Mon, Feb 11, 2013 at 1:17 PM, Seth Falcon seth@opscode.com wrote:

Hi Mark,

On Feb 11, 2013, at 7:04 AM, Mark Pimentel wrote:

Also, how are the cookbooks stored so that we could replicate those
also?

The Chef 11 Server uses the bookshelf component to store cookbook
content. The default data directory is:
/var/opt/chef-server/bookshelf/data. This can be configured via
chef-server.rb config as:

bookshlef['data_dir'] = PATH

Currently all cookbook content files are stored as flat files in the data
directory.

seth

Vaidas_Jablonskis · February 11, 2013, 9:29pm

This might be slightly unrelated to this conversion, but what I wonder what
is stored in postgres database? What happens when this database gets
corrupted or data is lost, for instance?

Thanks,
Vaidas

On 11 February 2013 19:13, Jesse Campbell hikeit@gmail.com wrote:

My intention was to upload cookbooks to all servers that are replicating
that database, hoping that it would check the filesystem for the existence
of the checksum files, rather than just looking at the database.

-jesse

On Mon, Feb 11, 2013 at 2:05 PM, Adam Jacob adam@opscode.com wrote:

Not going to be an option – the files themselves are stored outside of
the database (which makes sense, if you think about the total havoc that
would be released if you stored them in the database, even as blobs).

Private Chef has HA built in via DRBD, and its what we recommend to
everyone. (HA active/passive, with as many front-end servers as you need.)

Best,
Adam

From: Jesse Campbell hikeit@gmail.com
Reply-To: "chef@lists.opscode.com" chef@lists.opscode.com
Date: Monday, February 11, 2013 10:47 AM
To: chef chef@lists.opscode.com
Subject: [chef] Re: Re: Chef11 HA

In chef 10, if the checksum of the uploaded cookbook content was
already in the couch database, then the file would not get uploaded to the
disk.
Is this the same in chef 11? I'd like to use a replicated database
without needing to synchronize the chef bookshelf store.

On Mon, Feb 11, 2013 at 1:17 PM, Seth Falcon seth@opscode.com wrote:

Hi Mark,

On Feb 11, 2013, at 7:04 AM, Mark Pimentel wrote:

Also, how are the cookbooks stored so that we could replicate those
also?

The Chef 11 Server uses the bookshelf component to store cookbook
content. The default data directory is:
/var/opt/chef-server/bookshelf/data. This can be configured via
chef-server.rb config as:

bookshlef['data_dir'] = PATH

Currently all cookbook content files are stored as flat files in the
data directory.

seth

--
Vaidas Jablonskis

Seth_Falcon_01 · February 11, 2013, 10:08pm

On Feb 11, 2013, at 11:13 AM, Jesse Campbell wrote:

My intention was to upload cookbooks to all servers that are replicating that database, hoping that it would check the filesystem for the existence of the checksum files, rather than just looking at the database.

The mapping of cookbook version object to cookbook content by checksum is stored in the db. Only the db is consulting during cookbook upload. So if the db is replicated and you upload cookbooks, you will end up missing cookbook content.

In Chef 11, checksums are deleted when no longer referenced by a cookbook version. So one workaround for now would be to delete all cookbook versions and then upload.

Seth_Falcon_01 · February 11, 2013, 10:35pm

On Feb 11, 2013, at 1:29 PM, Vaidas Jablonskis wrote:

This might be slightly unrelated to this conversion, but what I wonder what is stored in postgres database?

All of the Chef object data is stored in the db. You can explore the schema a bit like this:

root@chef-server-berkshelf:~# su - opscode-pgsql
$ bash
opscode-pgsql@chef-server-berkshelf:~$ which psql
/opt/chef-server/embedded/bin/psql
opscode-pgsql@chef-server-berkshelf:~$ psql opscode_chef
psql (9.2.1)
Type "help" for help.

And find the script used to initialize the schema here:

github.com

chef-boneyard/chef_db/blob/master/priv/pgsql_schema.sql

-- Copyright 2011-2012 Opscode, Inc. All Rights Reserved.
--
-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License.  You may obtain a copy of the License at
--
--   http://www.apache.org/licenses/LICENSE-2.0
--
-- Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  See the License for the specific language governing permissions and limitations under the License.

--
-- PostgreSQL database dump
--

SET statement_timeout = 0;
SET client_encoding = 'UTF8';
SET standard_conforming_strings = on;
SET check_function_bodies = false;
SET client_min_messages = warning;

--
-- Name: plpgsql; Type: EXTENSION; Schema: -; Owner: -

This file has been truncated. show original

What happens when this database gets corrupted or data is lost, for instance?

Bad things happen. If the db data is lost or corrupted, so is your Chef Server.

seth

Baruch_Shpirer · February 14, 2013, 4:45am

Is there any draft to the HA procedure/setup?

Also, if I configure postgresql for master-master replication
and use drbd for the bookshelf folder
does it mean i got 2 identical servers in async mode?
Will clients be using same validation public key in both sites?

Baruch

-----Original Message-----
From: Seth Falcon [mailto:seth@opscode.com]
Sent: Monday, February 11/02/2013 17:35
To: chef@lists.opscode.com
Subject: [chef] Re: Chef11 HA

On Feb 11, 2013, at 1:29 PM, Vaidas Jablonskis wrote:

This might be slightly unrelated to this conversion, but what I wonder what is stored in postgres database?

All of the Chef object data is stored in the db. You can explore the schema a bit like this:

root@chef-server-berkshelf:~# su - opscode-pgsql
$ bash
opscode-pgsql@chef-server-berkshelf:~$ which psql /opt/chef-server/embedded/bin/psql
opscode-pgsql@chef-server-berkshelf:~$ psql opscode_chef psql (9.2.1) Type "help" for help.

And find the script used to initialize the schema here:

github.com

chef-boneyard/chef_db/blob/master/priv/pgsql_schema.sql

-- Copyright 2011-2012 Opscode, Inc. All Rights Reserved.
--
-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License.  You may obtain a copy of the License at
--
--   http://www.apache.org/licenses/LICENSE-2.0
--
-- Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  See the License for the specific language governing permissions and limitations under the License.

--
-- PostgreSQL database dump
--

SET statement_timeout = 0;
SET client_encoding = 'UTF8';
SET standard_conforming_strings = on;
SET check_function_bodies = false;
SET client_min_messages = warning;

--
-- Name: plpgsql; Type: EXTENSION; Schema: -; Owner: -

This file has been truncated. show original

What happens when this database gets corrupted or data is lost, for instance?

Bad things happen. If the db data is lost or corrupted, so is your Chef Server.

seth

Adam_Jacob · February 14, 2013, 5:01pm

Using DRBD for this is a good idea. If you share /var/opt/chef-server via
DRBD, you can use the normal mechanisms for starting/stopping the cluster,
and be certain you will have identical data.

Private Chef supports this configuration out of the box, fwiw, but it's
equally possible with Open Source Chef.

Best,
Adam

On 2/13/13 8:45 PM, "Baruch Shpirer" baruchs@quickplay.com wrote:

Is there any draft to the HA procedure/setup?

Also, if I configure postgresql for master-master replication
and use drbd for the bookshelf folder
does it mean i got 2 identical servers in async mode?
Will clients be using same validation public key in both sites?

Baruch

-----Original Message-----
From: Seth Falcon [mailto:seth@opscode.com]
Sent: Monday, February 11/02/2013 17:35
To: chef@lists.opscode.com
Subject: [chef] Re: Chef11 HA

On Feb 11, 2013, at 1:29 PM, Vaidas Jablonskis wrote:

This might be slightly unrelated to this conversion, but what I wonder
what is stored in postgres database?

All of the Chef object data is stored in the db. You can explore the
schema a bit like this:

root@chef-server-berkshelf:~# su - opscode-pgsql
$ bash
opscode-pgsql@chef-server-berkshelf:~$ which psql
/opt/chef-server/embedded/bin/psql
opscode-pgsql@chef-server-berkshelf:~$ psql opscode_chef psql (9.2.1)
Type "help" for help.

opscode_chef=# \d
List of relations
Schema | Name | Type | Owner
--------+-------------------------------+----------+---------------
public | checksums | table | opscode-pgsql
public | clients | table | opscode-pgsql
public | cookbook_version_checksums | table | opscode-pgsql
public | cookbook_version_dependencies | view | opscode-pgsql
public | cookbook_versions | table | opscode-pgsql
public | cookbook_versions_by_rank | view | opscode-pgsql
public | cookbooks | table | opscode-pgsql
public | cookbooks_id_seq | sequence | opscode-pgsql
public | data_bag_items | table | opscode-pgsql
public | data_bags | table | opscode-pgsql
public | environments | table | opscode-pgsql
public | joined_cookbook_version | view | opscode-pgsql
public | nodes | table | opscode-pgsql
public | osc_users | table | opscode-pgsql
public | roles | table | opscode-pgsql
public | sandboxed_checksums | table | opscode-pgsql
public | schema_info | table | opscode-pgsql
(17 rows)

And find the script used to initialize the schema here:
chef_db/priv/pgsql_schema.sql at master · chef-boneyard/chef_db · GitHub

What happens when this database gets corrupted or data is lost, for
instance?

Bad things happen. If the db data is lost or corrupted, so is your Chef
Server.

seth

Mark_Pimentel · February 14, 2013, 5:56pm

Say this scenario is configured across sites, with each chef server serving
different data centers. Would the keys be the same for both servers?

This would be used in a scenario where we have a main deployment chef
whereby we would control all objects with the complementary servers
replicating cookbook data as well as user and node information. The
other servers would simply replicate back their node information.

On Thu, Feb 14, 2013 at 12:01 PM, Adam Jacob adam@opscode.com wrote:

Using DRBD for this is a good idea. If you share /var/opt/chef-server via
DRBD, you can use the normal mechanisms for starting/stopping the cluster,
and be certain you will have identical data.

Private Chef supports this configuration out of the box, fwiw, but it's
equally possible with Open Source Chef.

Best,
Adam

On 2/13/13 8:45 PM, "Baruch Shpirer" baruchs@quickplay.com wrote:

Is there any draft to the HA procedure/setup?

Also, if I configure postgresql for master-master replication
and use drbd for the bookshelf folder
does it mean i got 2 identical servers in async mode?
Will clients be using same validation public key in both sites?

Baruch

-----Original Message-----
From: Seth Falcon [mailto:seth@opscode.com]
Sent: Monday, February 11/02/2013 17:35
To: chef@lists.opscode.com
Subject: [chef] Re: Chef11 HA

On Feb 11, 2013, at 1:29 PM, Vaidas Jablonskis wrote:

This might be slightly unrelated to this conversion, but what I wonder
what is stored in postgres database?

All of the Chef object data is stored in the db. You can explore the
schema a bit like this:

root@chef-server-berkshelf:~# su - opscode-pgsql
$ bash
opscode-pgsql@chef-server-berkshelf:~$ which psql
/opt/chef-server/embedded/bin/psql
opscode-pgsql@chef-server-berkshelf:~$ psql opscode_chef psql (9.2.1)
Type "help" for help.

opscode_chef=# \d
List of relations
Schema | Name | Type | Owner
--------+-------------------------------+----------+---------------
public | checksums | table | opscode-pgsql
public | clients | table | opscode-pgsql
public | cookbook_version_checksums | table | opscode-pgsql
public | cookbook_version_dependencies | view | opscode-pgsql
public | cookbook_versions | table | opscode-pgsql
public | cookbook_versions_by_rank | view | opscode-pgsql
public | cookbooks | table | opscode-pgsql
public | cookbooks_id_seq | sequence | opscode-pgsql
public | data_bag_items | table | opscode-pgsql
public | data_bags | table | opscode-pgsql
public | environments | table | opscode-pgsql
public | joined_cookbook_version | view | opscode-pgsql
public | nodes | table | opscode-pgsql
public | osc_users | table | opscode-pgsql
public | roles | table | opscode-pgsql
public | sandboxed_checksums | table | opscode-pgsql
public | schema_info | table | opscode-pgsql
(17 rows)

And find the script used to initialize the schema here:
chef_db/priv/pgsql_schema.sql at master · chef-boneyard/chef_db · GitHub

What happens when this database gets corrupted or data is lost, for
instance?

Bad things happen. If the db data is lost or corrupted, so is your Chef
Server.

seth

--
Thanks,

Mark

Adam_Jacob · February 15, 2013, 1:04am

We tend to recommend against this, as you are usually leaking both data and control across failure domains.

Think about it this way: when it fails, do you really want to add the increased latency? How about data replication when you are split brained? How do you fail back to being in multiple datacenters? Is one primary, the other passive?

The alternative is to treat each as an isolated failure domain, make them HA, and solve the consistency problem at the delivery of data level. It works much, much better.

Best,
Adam

From: Mark Pimentel <markpimentel22@gmail.com mailto:markpimentel22@gmail.com>
Reply-To: "chef@lists.opscode.com mailto:chef@lists.opscode.com" <chef@lists.opscode.com mailto:chef@lists.opscode.com>
Date: Thursday, February 14, 2013 9:56 AM
To: "chef@lists.opscode.com mailto:chef@lists.opscode.com" <chef@lists.opscode.com mailto:chef@lists.opscode.com>
Subject: [chef] Re: Re: RE: Re: Chef11 HA

Say this scenario is configured across sites, with each chef server serving different data centers. Would the keys be the same for both servers?

This would be used in a scenario where we have a main deployment chef whereby we would control all objects with the complementary servers replicating cookbook data as well as user and node information. The other servers would simply replicate back their node information.

On Thu, Feb 14, 2013 at 12:01 PM, Adam Jacob <adam@opscode.com mailto:adam@opscode.com> wrote:
Using DRBD for this is a good idea. If you share /var/opt/chef-server via
DRBD, you can use the normal mechanisms for starting/stopping the cluster,
and be certain you will have identical data.

Private Chef supports this configuration out of the box, fwiw, but it's
equally possible with Open Source Chef.

Best,
Adam

On 2/13/13 8:45 PM, "Baruch Shpirer" <baruchs@quickplay.com mailto:baruchs@quickplay.com> wrote:

Is there any draft to the HA procedure/setup?

Also, if I configure postgresql for master-master replication
and use drbd for the bookshelf folder
does it mean i got 2 identical servers in async mode?
Will clients be using same validation public key in both sites?

Baruch

-----Original Message-----
From: Seth Falcon [mailto:seth@opscode.com mailto:seth@opscode.com]
Sent: Monday, February 11/02/2013 17:35
To: <chef@lists.opscode.com mailto:chef@lists.opscode.com>
Subject: [chef] Re: Chef11 HA

On Feb 11, 2013, at 1:29 PM, Vaidas Jablonskis wrote:

This might be slightly unrelated to this conversion, but what I wonder
what is stored in postgres database?

All of the Chef object data is stored in the db. You can explore the
schema a bit like this:

root@chef-server-berkshelf:~# su - opscode-pgsql
$ bash
opscode-pgsql@chef-server-berkshelf:~$ which psql
/opt/chef-server/embedded/bin/psql
opscode-pgsql@chef-server-berkshelf:~$ psql opscode_chef psql (9.2.1)
Type "help" for help.

opscode_chef=# \d
List of relations
Schema | Name | Type | Owner
--------+-------------------------------+----------+---------------
public | checksums | table | opscode-pgsql
public | clients | table | opscode-pgsql
public | cookbook_version_checksums | table | opscode-pgsql
public | cookbook_version_dependencies | view | opscode-pgsql
public | cookbook_versions | table | opscode-pgsql
public | cookbook_versions_by_rank | view | opscode-pgsql
public | cookbooks | table | opscode-pgsql
public | cookbooks_id_seq | sequence | opscode-pgsql
public | data_bag_items | table | opscode-pgsql
public | data_bags | table | opscode-pgsql
public | environments | table | opscode-pgsql
public | joined_cookbook_version | view | opscode-pgsql
public | nodes | table | opscode-pgsql
public | osc_users | table | opscode-pgsql
public | roles | table | opscode-pgsql
public | sandboxed_checksums | table | opscode-pgsql
public | schema_info | table | opscode-pgsql
(17 rows)

And find the script used to initialize the schema here:
chef_db/priv/pgsql_schema.sql at master · chef-boneyard/chef_db · GitHub

What happens when this database gets corrupted or data is lost, for
instance?

Bad things happen. If the db data is lost or corrupted, so is your Chef
Server.

seth

--
Thanks,

Mark

Baruch_Shpirer · February 15, 2013, 1:10am

Can you define “to treat each as an isolated failure domain, make them HA”

From: Adam Jacob [mailto:adam@opscode.com]
Sent: Thursday, February 14/02/2013 20:05
To: chef@lists.opscode.com
Subject: [chef] Re: Re: Re: RE: Re: Chef11 HA

We tend to recommend against this, as you are usually leaking both data and control across failure domains.

Think about it this way: when it fails, do you really want to add the increased latency? How about data replication when you are split brained? How do you fail back to being in multiple datacenters? Is one primary, the other passive?

The alternative is to treat each as an isolated failure domain, make them HA, and solve the consistency problem at the delivery of data level. It works much, much better.

Best,
Adam

From: Mark Pimentel <markpimentel22@gmail.com mailto:markpimentel22@gmail.com>
Reply-To: "chef@lists.opscode.com mailto:chef@lists.opscode.com" <chef@lists.opscode.com mailto:chef@lists.opscode.com>
Date: Thursday, February 14, 2013 9:56 AM
To: "chef@lists.opscode.com mailto:chef@lists.opscode.com" <chef@lists.opscode.com mailto:chef@lists.opscode.com>
Subject: [chef] Re: Re: RE: Re: Chef11 HA

Say this scenario is configured across sites, with each chef server serving different data centers. Would the keys be the same for both servers?

This would be used in a scenario where we have a main deployment chef whereby we would control all objects with the complementary servers replicating cookbook data as well as user and node information. The other servers would simply replicate back their node information.

On Thu, Feb 14, 2013 at 12:01 PM, Adam Jacob <adam@opscode.com mailto:adam@opscode.com> wrote:
Using DRBD for this is a good idea. If you share /var/opt/chef-server via
DRBD, you can use the normal mechanisms for starting/stopping the cluster,
and be certain you will have identical data.

Private Chef supports this configuration out of the box, fwiw, but it's
equally possible with Open Source Chef.

Best,
Adam

On 2/13/13 8:45 PM, "Baruch Shpirer" <baruchs@quickplay.com mailto:baruchs@quickplay.com> wrote:

Is there any draft to the HA procedure/setup?

Also, if I configure postgresql for master-master replication
and use drbd for the bookshelf folder
does it mean i got 2 identical servers in async mode?
Will clients be using same validation public key in both sites?

Baruch

-----Original Message-----
From: Seth Falcon [mailto:seth@opscode.com mailto:seth@opscode.com]
Sent: Monday, February 11/02/2013 17:35
To: <chef@lists.opscode.com mailto:chef@lists.opscode.com>
Subject: [chef] Re: Chef11 HA

On Feb 11, 2013, at 1:29 PM, Vaidas Jablonskis wrote:

This might be slightly unrelated to this conversion, but what I wonder
what is stored in postgres database?

All of the Chef object data is stored in the db. You can explore the
schema a bit like this:

root@chef-server-berkshelf:~# su - opscode-pgsql
$ bash
opscode-pgsql@chef-server-berkshelf:~$ which psql
/opt/chef-server/embedded/bin/psql
opscode-pgsql@chef-server-berkshelf:~$ psql opscode_chef psql (9.2.1)
Type "help" for help.

opscode_chef=# \d
List of relations
Schema | Name | Type | Owner
--------+-------------------------------+----------+---------------
public | checksums | table | opscode-pgsql
public | clients | table | opscode-pgsql
public | cookbook_version_checksums | table | opscode-pgsql
public | cookbook_version_dependencies | view | opscode-pgsql
public | cookbook_versions | table | opscode-pgsql
public | cookbook_versions_by_rank | view | opscode-pgsql
public | cookbooks | table | opscode-pgsql
public | cookbooks_id_seq | sequence | opscode-pgsql
public | data_bag_items | table | opscode-pgsql
public | data_bags | table | opscode-pgsql
public | environments | table | opscode-pgsql
public | joined_cookbook_version | view | opscode-pgsql
public | nodes | table | opscode-pgsql
public | osc_users | table | opscode-pgsql
public | roles | table | opscode-pgsql
public | sandboxed_checksums | table | opscode-pgsql
public | schema_info | table | opscode-pgsql
(17 rows)

And find the script used to initialize the schema here:
chef_db/priv/pgsql_schema.sql at master · chef-boneyard/chef_db · GitHub

What happens when this database gets corrupted or data is lost, for
instance?

Bad things happen. If the db data is lost or corrupted, so is your Chef
Server.

seth

--
Thanks,

Mark

Adam_Jacob · February 15, 2013, 1:12am

Yes – have an HA pair (or at least HA Backends, with multiple API front-ends) in each failure domain. Make each failure domain highly available, and make the system partition tolerant by enforcing that no writes ever need to cross the boundary.

Adam

From: Baruch Shpirer <baruchs@quickplay.com mailto:baruchs@quickplay.com>
Reply-To: "chef@lists.opscode.com mailto:chef@lists.opscode.com" <chef@lists.opscode.com mailto:chef@lists.opscode.com>
Date: Thursday, February 14, 2013 5:10 PM
To: "chef@lists.opscode.com mailto:chef@lists.opscode.com" <chef@lists.opscode.com mailto:chef@lists.opscode.com>
Subject: [chef] RE: Re: Re: Re: RE: Re: Chef11 HA

Can you define “to treat each as an isolated failure domain, make them HA”

From: Adam Jacob [mailto:adam@opscode.com]
Sent: Thursday, February 14/02/2013 20:05
To: chef@lists.opscode.com mailto:chef@lists.opscode.com
Subject: [chef] Re: Re: Re: RE: Re: Chef11 HA

We tend to recommend against this, as you are usually leaking both data and control across failure domains.

Think about it this way: when it fails, do you really want to add the increased latency? How about data replication when you are split brained? How do you fail back to being in multiple datacenters? Is one primary, the other passive?

The alternative is to treat each as an isolated failure domain, make them HA, and solve the consistency problem at the delivery of data level. It works much, much better.

Best,
Adam

From: Mark Pimentel <markpimentel22@gmail.com mailto:markpimentel22@gmail.com>
Reply-To: "chef@lists.opscode.com mailto:chef@lists.opscode.com" <chef@lists.opscode.com mailto:chef@lists.opscode.com>
Date: Thursday, February 14, 2013 9:56 AM
To: "chef@lists.opscode.com mailto:chef@lists.opscode.com" <chef@lists.opscode.com mailto:chef@lists.opscode.com>
Subject: [chef] Re: Re: RE: Re: Chef11 HA

Say this scenario is configured across sites, with each chef server serving different data centers. Would the keys be the same for both servers?

This would be used in a scenario where we have a main deployment chef whereby we would control all objects with the complementary servers replicating cookbook data as well as user and node information. The other servers would simply replicate back their node information.

On Thu, Feb 14, 2013 at 12:01 PM, Adam Jacob <adam@opscode.com mailto:adam@opscode.com> wrote:
Using DRBD for this is a good idea. If you share /var/opt/chef-server via
DRBD, you can use the normal mechanisms for starting/stopping the cluster,
and be certain you will have identical data.

Private Chef supports this configuration out of the box, fwiw, but it's
equally possible with Open Source Chef.

Best,
Adam

On 2/13/13 8:45 PM, "Baruch Shpirer" <baruchs@quickplay.com mailto:baruchs@quickplay.com> wrote:

Is there any draft to the HA procedure/setup?

Also, if I configure postgresql for master-master replication
and use drbd for the bookshelf folder
does it mean i got 2 identical servers in async mode?
Will clients be using same validation public key in both sites?

Baruch

-----Original Message-----
From: Seth Falcon [mailto:seth@opscode.com mailto:seth@opscode.com]
Sent: Monday, February 11/02/2013 17:35
To: <chef@lists.opscode.com mailto:chef@lists.opscode.com>
Subject: [chef] Re: Chef11 HA

On Feb 11, 2013, at 1:29 PM, Vaidas Jablonskis wrote:

This might be slightly unrelated to this conversion, but what I wonder
what is stored in postgres database?

All of the Chef object data is stored in the db. You can explore the
schema a bit like this:

root@chef-server-berkshelf:~# su - opscode-pgsql
$ bash
opscode-pgsql@chef-server-berkshelf:~$ which psql
/opt/chef-server/embedded/bin/psql
opscode-pgsql@chef-server-berkshelf:~$ psql opscode_chef psql (9.2.1)
Type "help" for help.

opscode_chef=# \d
List of relations
Schema | Name | Type | Owner
--------+-------------------------------+----------+---------------
public | checksums | table | opscode-pgsql
public | clients | table | opscode-pgsql
public | cookbook_version_checksums | table | opscode-pgsql
public | cookbook_version_dependencies | view | opscode-pgsql
public | cookbook_versions | table | opscode-pgsql
public | cookbook_versions_by_rank | view | opscode-pgsql
public | cookbooks | table | opscode-pgsql
public | cookbooks_id_seq | sequence | opscode-pgsql
public | data_bag_items | table | opscode-pgsql
public | data_bags | table | opscode-pgsql
public | environments | table | opscode-pgsql
public | joined_cookbook_version | view | opscode-pgsql
public | nodes | table | opscode-pgsql
public | osc_users | table | opscode-pgsql
public | roles | table | opscode-pgsql
public | sandboxed_checksums | table | opscode-pgsql
public | schema_info | table | opscode-pgsql
(17 rows)

And find the script used to initialize the schema here:
chef_db/priv/pgsql_schema.sql at master · chef-boneyard/chef_db · GitHub

What happens when this database gets corrupted or data is lost, for
instance?

Bad things happen. If the db data is lost or corrupted, so is your Chef
Server.

seth

--
Thanks,

Mark

Baruch_Shpirer · February 15, 2013, 2:15pm

How would you go about creating the HA pair?
Some docs/drafts/pointers?

From: Adam Jacob [mailto:adam@opscode.com]
Sent: Thursday, February 14/02/2013 20:13
To: chef@lists.opscode.com
Subject: [chef] Re: RE: Re: Re: Re: RE: Re: Chef11 HA

Yes – have an HA pair (or at least HA Backends, with multiple API front-ends) in each failure domain. Make each failure domain highly available, and make the system partition tolerant by enforcing that no writes ever need to cross the boundary.

Adam

From: Baruch Shpirer <baruchs@quickplay.com mailto:baruchs@quickplay.com>
Reply-To: "chef@lists.opscode.com mailto:chef@lists.opscode.com" <chef@lists.opscode.com mailto:chef@lists.opscode.com>
Date: Thursday, February 14, 2013 5:10 PM
To: "chef@lists.opscode.com mailto:chef@lists.opscode.com" <chef@lists.opscode.com mailto:chef@lists.opscode.com>
Subject: [chef] RE: Re: Re: Re: RE: Re: Chef11 HA

Can you define “to treat each as an isolated failure domain, make them HA”

From: Adam Jacob [mailto:adam@opscode.com]
Sent: Thursday, February 14/02/2013 20:05
To: chef@lists.opscode.com mailto:chef@lists.opscode.com
Subject: [chef] Re: Re: Re: RE: Re: Chef11 HA

We tend to recommend against this, as you are usually leaking both data and control across failure domains.

Think about it this way: when it fails, do you really want to add the increased latency? How about data replication when you are split brained? How do you fail back to being in multiple datacenters? Is one primary, the other passive?

The alternative is to treat each as an isolated failure domain, make them HA, and solve the consistency problem at the delivery of data level. It works much, much better.

Best,
Adam

From: Mark Pimentel <markpimentel22@gmail.com mailto:markpimentel22@gmail.com>
Reply-To: "chef@lists.opscode.com mailto:chef@lists.opscode.com" <chef@lists.opscode.com mailto:chef@lists.opscode.com>
Date: Thursday, February 14, 2013 9:56 AM
To: "chef@lists.opscode.com mailto:chef@lists.opscode.com" <chef@lists.opscode.com mailto:chef@lists.opscode.com>
Subject: [chef] Re: Re: RE: Re: Chef11 HA

Say this scenario is configured across sites, with each chef server serving different data centers. Would the keys be the same for both servers?

This would be used in a scenario where we have a main deployment chef whereby we would control all objects with the complementary servers replicating cookbook data as well as user and node information. The other servers would simply replicate back their node information.

On Thu, Feb 14, 2013 at 12:01 PM, Adam Jacob <adam@opscode.com mailto:adam@opscode.com> wrote:
Using DRBD for this is a good idea. If you share /var/opt/chef-server via
DRBD, you can use the normal mechanisms for starting/stopping the cluster,
and be certain you will have identical data.

Private Chef supports this configuration out of the box, fwiw, but it's
equally possible with Open Source Chef.

Best,
Adam

On 2/13/13 8:45 PM, "Baruch Shpirer" <baruchs@quickplay.com mailto:baruchs@quickplay.com> wrote:

Is there any draft to the HA procedure/setup?

Also, if I configure postgresql for master-master replication
and use drbd for the bookshelf folder
does it mean i got 2 identical servers in async mode?
Will clients be using same validation public key in both sites?

Baruch

-----Original Message-----
From: Seth Falcon [mailto:seth@opscode.com mailto:seth@opscode.com]
Sent: Monday, February 11/02/2013 17:35
To: <chef@lists.opscode.com mailto:chef@lists.opscode.com>
Subject: [chef] Re: Chef11 HA

On Feb 11, 2013, at 1:29 PM, Vaidas Jablonskis wrote:

This might be slightly unrelated to this conversion, but what I wonder
what is stored in postgres database?

All of the Chef object data is stored in the db. You can explore the
schema a bit like this:

root@chef-server-berkshelf:~# su - opscode-pgsql
$ bash
opscode-pgsql@chef-server-berkshelf:~$ which psql
/opt/chef-server/embedded/bin/psql
opscode-pgsql@chef-server-berkshelf:~$ psql opscode_chef psql (9.2.1)
Type "help" for help.

opscode_chef=# \d
List of relations
Schema | Name | Type | Owner
--------+-------------------------------+----------+---------------
public | checksums | table | opscode-pgsql
public | clients | table | opscode-pgsql
public | cookbook_version_checksums | table | opscode-pgsql
public | cookbook_version_dependencies | view | opscode-pgsql
public | cookbook_versions | table | opscode-pgsql
public | cookbook_versions_by_rank | view | opscode-pgsql
public | cookbooks | table | opscode-pgsql
public | cookbooks_id_seq | sequence | opscode-pgsql
public | data_bag_items | table | opscode-pgsql
public | data_bags | table | opscode-pgsql
public | environments | table | opscode-pgsql
public | joined_cookbook_version | view | opscode-pgsql
public | nodes | table | opscode-pgsql
public | osc_users | table | opscode-pgsql
public | roles | table | opscode-pgsql
public | sandboxed_checksums | table | opscode-pgsql
public | schema_info | table | opscode-pgsql
(17 rows)

And find the script used to initialize the schema here:
chef_db/priv/pgsql_schema.sql at master · chef-boneyard/chef_db · GitHub

What happens when this database gets corrupted or data is lost, for
instance?

Bad things happen. If the db data is lost or corrupted, so is your Chef
Server.

seth

--
Thanks,

Mark

Jesse_Campbell · February 15, 2013, 5:01pm

In the past, the official answer had been that the private chef paid
offering comes out of the box with HA.
Have you worked with your company's money people to see if they'll shell
out for it? I'd love to say my company would, but it has been an uphill
battle... giving back to the community through patches is great, but giving
back to ops code by paying to keep them in the black would be nice.

For HA, you'll need to take a look inside the current installer. There are
multiple back end store components (solr, bookshelf, postgres) which all
need replication or clustering, then there are middle tier services like
the chef expander (or whatever it is called now) and the message queue
(used to be rabbitmq), and the chef server api, all of which need to be
deployed in multiple places hitting those replicated backends (the MQ might
want to be treated like a back end component).

Then you'll want load balancing between the server api endpoints, then
you'll want to have the webui and knife and chef client pointing at the
load balancer.
If you heavily use the webui, deploy that in multiple places too, and load
balance it.

For multiple datacenters, you'll want to get some kind of reliable
replication for the backend components (solr, bookshelf, postgres), and
have separate copies of the front and middle tiers in each DC pointing to
the replicated back end.

It isn't an easy problem to solve, which is why opscode is hoping you'll
pay for it

-Jesse
On Feb 15, 2013 9:16 AM, "Baruch Shpirer" baruchs@quickplay.com wrote:

How would you go about creating the HA pair?****

Some docs/drafts/pointers?****

From: Adam Jacob [mailto:adam@opscode.com]
Sent: Thursday, February 14/02/2013 20:13
To: chef@lists.opscode.com
Subject: [chef] Re: RE: Re: Re: Re: RE: Re: Chef11 HA****

Yes – have an HA pair (or at least HA Backends, with multiple API
front-ends) in each failure domain. Make each failure domain highly
available, and make the system partition tolerant by enforcing that no
writes ever need to cross the boundary.****

Adam****

*From: *Baruch Shpirer baruchs@quickplay.com
*Reply-To: *"chef@lists.opscode.com" chef@lists.opscode.com
*Date: *Thursday, February 14, 2013 5:10 PM
To: "chef@lists.opscode.com" chef@lists.opscode.com
Subject: [chef] RE: Re: Re: Re: RE: Re: Chef11 HA

Can you define “to treat each as an isolated failure domain, make them HA”

From: Adam Jacob [mailto:adam@opscode.com adam@opscode.com]
Sent: Thursday, February 14/02/2013 20:05
To: chef@lists.opscode.com
Subject: [chef] Re: Re: Re: RE: Re: Chef11 HA****

We tend to recommend against this, as you are usually leaking both data
and control across failure domains.****

Think about it this way: when it fails, do you really want to add the
increased latency? How about data replication when you are split brained?
How do you fail back to being in multiple datacenters? Is one primary, the
other passive?****

The alternative is to treat each as an isolated failure domain, make them
HA, and solve the consistency problem at the delivery of data level. It
works much, much better.****

Best,****

Adam****

*From: *Mark Pimentel markpimentel22@gmail.com
*Reply-To: *"chef@lists.opscode.com" chef@lists.opscode.com
*Date: *Thursday, February 14, 2013 9:56 AM
To: "chef@lists.opscode.com" chef@lists.opscode.com
Subject: [chef] Re: Re: RE: Re: Chef11 HA

Say this scenario is configured across sites, with each chef server
serving different data centers. Would the keys be the same for both
servers? ****

This would be used in a scenario where we have a main deployment chef
whereby we would control all objects with the complementary servers
replicating cookbook data as well as user and node information. The
other servers would simply replicate back their node information.****

On Thu, Feb 14, 2013 at 12:01 PM, Adam Jacob adam@opscode.com wrote:****

Using DRBD for this is a good idea. If you share /var/opt/chef-server via
DRBD, you can use the normal mechanisms for starting/stopping the cluster,
and be certain you will have identical data.

Private Chef supports this configuration out of the box, fwiw, but it's
equally possible with Open Source Chef.

Best,
Adam****

On 2/13/13 8:45 PM, "Baruch Shpirer" baruchs@quickplay.com wrote:

Is there any draft to the HA procedure/setup?

Also, if I configure postgresql for master-master replication
and use drbd for the bookshelf folder
does it mean i got 2 identical servers in async mode?
Will clients be using same validation public key in both sites?

Baruch

-----Original Message-----
From: Seth Falcon [mailto:seth@opscode.com]
Sent: Monday, February 11/02/2013 17:35
To: chef@lists.opscode.com
Subject: [chef] Re: Chef11 HA

On Feb 11, 2013, at 1:29 PM, Vaidas Jablonskis wrote:

This might be slightly unrelated to this conversion, but what I wonder
what is stored in postgres database?

All of the Chef object data is stored in the db. You can explore the
schema a bit like this:

root@chef-server-berkshelf:~# su - opscode-pgsql
$ bash
opscode-pgsql@chef-server-berkshelf:~$ which psql
/opt/chef-server/embedded/bin/psql
opscode-pgsql@chef-server-berkshelf:~$ psql opscode_chef psql (9.2.1)
Type "help" for help.

opscode_chef=# \d
List of relations
Schema | Name | Type | Owner
--------+-------------------------------+----------+---------------
public | checksums | table | opscode-pgsql
public | clients | table | opscode-pgsql
public | cookbook_version_checksums | table | opscode-pgsql
public | cookbook_version_dependencies | view | opscode-pgsql
public | cookbook_versions | table | opscode-pgsql
public | cookbook_versions_by_rank | view | opscode-pgsql
public | cookbooks | table | opscode-pgsql
public | cookbooks_id_seq | sequence | opscode-pgsql
public | data_bag_items | table | opscode-pgsql
public | data_bags | table | opscode-pgsql
public | environments | table | opscode-pgsql
public | joined_cookbook_version | view | opscode-pgsql
public | nodes | table | opscode-pgsql
public | osc_users | table | opscode-pgsql
public | roles | table | opscode-pgsql
public | sandboxed_checksums | table | opscode-pgsql
public | schema_info | table | opscode-pgsql
(17 rows)

And find the script used to initialize the schema here:
chef_db/priv/pgsql_schema.sql at master · chef-boneyard/chef_db · GitHub

What happens when this database gets corrupted or data is lost, for
instance?

Bad things happen. If the db data is lost or corrupted, so is your Chef
Server.

seth

--
Thanks,

Mark ****

Adam_Jacob · February 15, 2013, 5:46pm

The Private Chef documentation has some background on what we do architecturally. The configuration is different (we support HA topologies out of the box in Private Chef, and you need to assemble it yourself with Open Source Chef,) but the architecture is one we're used to great success with many customers.

http://private-chef-docs.opscode.com/installation/ha.html

Best,
Adam

From: Baruch Shpirer <baruchs@quickplay.com mailto:baruchs@quickplay.com>
Reply-To: "chef@lists.opscode.com mailto:chef@lists.opscode.com" <chef@lists.opscode.com mailto:chef@lists.opscode.com>
Date: Friday, February 15, 2013 6:15 AM
To: "chef@lists.opscode.com mailto:chef@lists.opscode.com" <chef@lists.opscode.com mailto:chef@lists.opscode.com>
Subject: [chef] RE: Re: RE: Re: Re: Re: RE: Re: Chef11 HA

How would you go about creating the HA pair?
Some docs/drafts/pointers?

From: Adam Jacob [mailto:adam@opscode.com]
Sent: Thursday, February 14/02/2013 20:13
To: chef@lists.opscode.com mailto:chef@lists.opscode.com
Subject: [chef] Re: RE: Re: Re: Re: RE: Re: Chef11 HA

Yes – have an HA pair (or at least HA Backends, with multiple API front-ends) in each failure domain. Make each failure domain highly available, and make the system partition tolerant by enforcing that no writes ever need to cross the boundary.

Adam

From: Baruch Shpirer <baruchs@quickplay.com mailto:baruchs@quickplay.com>
Reply-To: "chef@lists.opscode.com mailto:chef@lists.opscode.com" <chef@lists.opscode.com mailto:chef@lists.opscode.com>
Date: Thursday, February 14, 2013 5:10 PM
To: "chef@lists.opscode.com mailto:chef@lists.opscode.com" <chef@lists.opscode.com mailto:chef@lists.opscode.com>
Subject: [chef] RE: Re: Re: Re: RE: Re: Chef11 HA

Can you define “to treat each as an isolated failure domain, make them HA”

From: Adam Jacob [mailto:adam@opscode.com]
Sent: Thursday, February 14/02/2013 20:05
To: chef@lists.opscode.com mailto:chef@lists.opscode.com
Subject: [chef] Re: Re: Re: RE: Re: Chef11 HA

We tend to recommend against this, as you are usually leaking both data and control across failure domains.

Think about it this way: when it fails, do you really want to add the increased latency? How about data replication when you are split brained? How do you fail back to being in multiple datacenters? Is one primary, the other passive?

The alternative is to treat each as an isolated failure domain, make them HA, and solve the consistency problem at the delivery of data level. It works much, much better.

Best,
Adam

From: Mark Pimentel <markpimentel22@gmail.com mailto:markpimentel22@gmail.com>
Reply-To: "chef@lists.opscode.com mailto:chef@lists.opscode.com" <chef@lists.opscode.com mailto:chef@lists.opscode.com>
Date: Thursday, February 14, 2013 9:56 AM
To: "chef@lists.opscode.com mailto:chef@lists.opscode.com" <chef@lists.opscode.com mailto:chef@lists.opscode.com>
Subject: [chef] Re: Re: RE: Re: Chef11 HA

Say this scenario is configured across sites, with each chef server serving different data centers. Would the keys be the same for both servers?

This would be used in a scenario where we have a main deployment chef whereby we would control all objects with the complementary servers replicating cookbook data as well as user and node information. The other servers would simply replicate back their node information.

On Thu, Feb 14, 2013 at 12:01 PM, Adam Jacob <adam@opscode.com mailto:adam@opscode.com> wrote:
Using DRBD for this is a good idea. If you share /var/opt/chef-server via
DRBD, you can use the normal mechanisms for starting/stopping the cluster,
and be certain you will have identical data.

Private Chef supports this configuration out of the box, fwiw, but it's
equally possible with Open Source Chef.

Best,
Adam

On 2/13/13 8:45 PM, "Baruch Shpirer" <baruchs@quickplay.com mailto:baruchs@quickplay.com> wrote:

Is there any draft to the HA procedure/setup?

Also, if I configure postgresql for master-master replication
and use drbd for the bookshelf folder
does it mean i got 2 identical servers in async mode?
Will clients be using same validation public key in both sites?

Baruch

-----Original Message-----
From: Seth Falcon [mailto:seth@opscode.com mailto:seth@opscode.com]
Sent: Monday, February 11/02/2013 17:35
To: <chef@lists.opscode.com mailto:chef@lists.opscode.com>
Subject: [chef] Re: Chef11 HA

On Feb 11, 2013, at 1:29 PM, Vaidas Jablonskis wrote:

This might be slightly unrelated to this conversion, but what I wonder
what is stored in postgres database?

All of the Chef object data is stored in the db. You can explore the
schema a bit like this:

root@chef-server-berkshelf:~# su - opscode-pgsql
$ bash
opscode-pgsql@chef-server-berkshelf:~$ which psql
/opt/chef-server/embedded/bin/psql
opscode-pgsql@chef-server-berkshelf:~$ psql opscode_chef psql (9.2.1)
Type "help" for help.

opscode_chef=# \d
List of relations
Schema | Name | Type | Owner
--------+-------------------------------+----------+---------------
public | checksums | table | opscode-pgsql
public | clients | table | opscode-pgsql
public | cookbook_version_checksums | table | opscode-pgsql
public | cookbook_version_dependencies | view | opscode-pgsql
public | cookbook_versions | table | opscode-pgsql
public | cookbook_versions_by_rank | view | opscode-pgsql
public | cookbooks | table | opscode-pgsql
public | cookbooks_id_seq | sequence | opscode-pgsql
public | data_bag_items | table | opscode-pgsql
public | data_bags | table | opscode-pgsql
public | environments | table | opscode-pgsql
public | joined_cookbook_version | view | opscode-pgsql
public | nodes | table | opscode-pgsql
public | osc_users | table | opscode-pgsql
public | roles | table | opscode-pgsql
public | sandboxed_checksums | table | opscode-pgsql
public | schema_info | table | opscode-pgsql
(17 rows)

And find the script used to initialize the schema here:
chef_db/priv/pgsql_schema.sql at master · chef-boneyard/chef_db · GitHub

What happens when this database gets corrupted or data is lost, for
instance?

Bad things happen. If the db data is lost or corrupted, so is your Chef
Server.

seth

--
Thanks,

Mark

Mark_Pimentel · February 15, 2013, 6:27pm

Much appreciated and many thanks. While we still have somewhat of an
uphill battle to acquire private chef ourselves, we do what we can by
preaching its benefits in the org. We are continually trying to prove its
worth and hope that someday we could be in a position to acquire the
private chef offering. With that said, we do contribute where we can as I
have also authored a cookbook and will be attending ChefConf along with one
of the workshops.

I must also say how helpful you guys have been to the community at large.

The support is excellent.

Thank you very much.

On Fri, Feb 15, 2013 at 12:46 PM, Adam Jacob adam@opscode.com wrote:

The Private Chef documentation has some background on what we do
architecturally. The configuration is different (we support HA topologies
out of the box in Private Chef, and you need to assemble it yourself with
Open Source Chef,) but the architecture is one we're used to great success
with many customers.

http://private-chef-docs.opscode.com/installation/ha.html

Best,
Adam

From: Baruch Shpirer baruchs@quickplay.com
Reply-To: "chef@lists.opscode.com" chef@lists.opscode.com
Date: Friday, February 15, 2013 6:15 AM
To: "chef@lists.opscode.com" chef@lists.opscode.com
Subject: [chef] RE: Re: RE: Re: Re: Re: RE: Re: Chef11 HA

How would you go about creating the HA pair?****

Some docs/drafts/pointers?****

From: Adam Jacob [mailto:adam@opscode.com adam@opscode.com]
Sent: Thursday, February 14/02/2013 20:13
To: chef@lists.opscode.com
Subject: [chef] Re: RE: Re: Re: Re: RE: Re: Chef11 HA****

Yes – have an HA pair (or at least HA Backends, with multiple API
front-ends) in each failure domain. Make each failure domain highly
available, and make the system partition tolerant by enforcing that no
writes ever need to cross the boundary.****

Adam****

*From: *Baruch Shpirer baruchs@quickplay.com
*Reply-To: *"chef@lists.opscode.com" chef@lists.opscode.com
*Date: *Thursday, February 14, 2013 5:10 PM
To: "chef@lists.opscode.com" chef@lists.opscode.com
Subject: [chef] RE: Re: Re: Re: RE: Re: Chef11 HA

Can you define “to treat each as an isolated failure domain, make them HA”

From: Adam Jacob [mailto:adam@opscode.com adam@opscode.com]
Sent: Thursday, February 14/02/2013 20:05
To: chef@lists.opscode.com
Subject: [chef] Re: Re: Re: RE: Re: Chef11 HA****

We tend to recommend against this, as you are usually leaking both data
and control across failure domains.****

Think about it this way: when it fails, do you really want to add the
increased latency? How about data replication when you are split brained?
How do you fail back to being in multiple datacenters? Is one primary, the
other passive?****

The alternative is to treat each as an isolated failure domain, make them
HA, and solve the consistency problem at the delivery of data level. It
works much, much better.****

Best,****

Adam****

*From: *Mark Pimentel markpimentel22@gmail.com
*Reply-To: *"chef@lists.opscode.com" chef@lists.opscode.com
*Date: *Thursday, February 14, 2013 9:56 AM
To: "chef@lists.opscode.com" chef@lists.opscode.com
Subject: [chef] Re: Re: RE: Re: Chef11 HA

Say this scenario is configured across sites, with each chef server
serving different data centers. Would the keys be the same for both
servers? ****

This would be used in a scenario where we have a main deployment chef
whereby we would control all objects with the complementary servers
replicating cookbook data as well as user and node information. The
other servers would simply replicate back their node information.****

On Thu, Feb 14, 2013 at 12:01 PM, Adam Jacob adam@opscode.com wrote:****

Using DRBD for this is a good idea. If you share /var/opt/chef-server via
DRBD, you can use the normal mechanisms for starting/stopping the cluster,
and be certain you will have identical data.

Private Chef supports this configuration out of the box, fwiw, but it's
equally possible with Open Source Chef.

Best,
Adam****

On 2/13/13 8:45 PM, "Baruch Shpirer" baruchs@quickplay.com wrote:

Is there any draft to the HA procedure/setup?

Also, if I configure postgresql for master-master replication
and use drbd for the bookshelf folder
does it mean i got 2 identical servers in async mode?
Will clients be using same validation public key in both sites?

Baruch

-----Original Message-----
From: Seth Falcon [mailto:seth@opscode.com]
Sent: Monday, February 11/02/2013 17:35
To: chef@lists.opscode.com
Subject: [chef] Re: Chef11 HA

On Feb 11, 2013, at 1:29 PM, Vaidas Jablonskis wrote:

This might be slightly unrelated to this conversion, but what I wonder
what is stored in postgres database?

All of the Chef object data is stored in the db. You can explore the
schema a bit like this:

root@chef-server-berkshelf:~# su - opscode-pgsql
$ bash
opscode-pgsql@chef-server-berkshelf:~$ which psql
/opt/chef-server/embedded/bin/psql
opscode-pgsql@chef-server-berkshelf:~$ psql opscode_chef psql (9.2.1)
Type "help" for help.

opscode_chef=# \d
List of relations
Schema | Name | Type | Owner
--------+-------------------------------+----------+---------------
public | checksums | table | opscode-pgsql
public | clients | table | opscode-pgsql
public | cookbook_version_checksums | table | opscode-pgsql
public | cookbook_version_dependencies | view | opscode-pgsql
public | cookbook_versions | table | opscode-pgsql
public | cookbook_versions_by_rank | view | opscode-pgsql
public | cookbooks | table | opscode-pgsql
public | cookbooks_id_seq | sequence | opscode-pgsql
public | data_bag_items | table | opscode-pgsql
public | data_bags | table | opscode-pgsql
public | environments | table | opscode-pgsql
public | joined_cookbook_version | view | opscode-pgsql
public | nodes | table | opscode-pgsql
public | osc_users | table | opscode-pgsql
public | roles | table | opscode-pgsql
public | sandboxed_checksums | table | opscode-pgsql
public | schema_info | table | opscode-pgsql
(17 rows)

And find the script used to initialize the schema here:
chef_db/priv/pgsql_schema.sql at master · chef-boneyard/chef_db · GitHub

What happens when this database gets corrupted or data is lost, for
instance?

Bad things happen. If the db data is lost or corrupted, so is your Chef
Server.

seth

--
Thanks,

Mark ****

--
Thanks,

Mark

Adam_Jacob · February 15, 2013, 8:32pm

In case I wasn't clear, though, nothing is stopping you from adopting that architecture, other than having to string together DRBD, an HA stack (pacemaker/keepalived), and the right HA resource files. It's not trivial, but it's totally possible, and nothing in the Chef 11 packages stops you – the knobs are all there.

Adam

From: Mark Pimentel <markpimentel22@gmail.com mailto:markpimentel22@gmail.com>
Reply-To: "chef@lists.opscode.com mailto:chef@lists.opscode.com" <chef@lists.opscode.com mailto:chef@lists.opscode.com>
Date: Friday, February 15, 2013 10:27 AM
To: "chef@lists.opscode.com mailto:chef@lists.opscode.com" <chef@lists.opscode.com mailto:chef@lists.opscode.com>
Subject: [chef] Re: Re: RE: Re: RE: Re: Re: Re: RE: Re: Chef11 HA

Much appreciated and many thanks. While we still have somewhat of an uphill battle to acquire private chef ourselves, we do what we can by preaching its benefits in the org. We are continually trying to prove its worth and hope that someday we could be in a position to acquire the private chef offering. With that said, we do contribute where we can as I have also authored a cookbook and will be attending ChefConf along with one of the workshops.

I must also say how helpful you guys have been to the community at large.

The support is excellent.

Thank you very much.

On Fri, Feb 15, 2013 at 12:46 PM, Adam Jacob <adam@opscode.com mailto:adam@opscode.com> wrote:
The Private Chef documentation has some background on what we do architecturally. The configuration is different (we support HA topologies out of the box in Private Chef, and you need to assemble it yourself with Open Source Chef,) but the architecture is one we're used to great success with many customers.

http://private-chef-docs.opscode.com/installation/ha.html

Best,
Adam

From: Baruch Shpirer <baruchs@quickplay.com mailto:baruchs@quickplay.com>
Reply-To: "chef@lists.opscode.com mailto:chef@lists.opscode.com" <chef@lists.opscode.com mailto:chef@lists.opscode.com>
Date: Friday, February 15, 2013 6:15 AM
To: "chef@lists.opscode.com mailto:chef@lists.opscode.com" <chef@lists.opscode.com mailto:chef@lists.opscode.com>
Subject: [chef] RE: Re: RE: Re: Re: Re: RE: Re: Chef11 HA

How would you go about creating the HA pair?
Some docs/drafts/pointers?

From: Adam Jacob [mailto:adam@opscode.com]
Sent: Thursday, February 14/02/2013 20:13
To: chef@lists.opscode.com mailto:chef@lists.opscode.com
Subject: [chef] Re: RE: Re: Re: Re: RE: Re: Chef11 HA

Yes – have an HA pair (or at least HA Backends, with multiple API front-ends) in each failure domain. Make each failure domain highly available, and make the system partition tolerant by enforcing that no writes ever need to cross the boundary.

Adam

From: Baruch Shpirer <baruchs@quickplay.com mailto:baruchs@quickplay.com>
Reply-To: "chef@lists.opscode.com mailto:chef@lists.opscode.com" <chef@lists.opscode.com mailto:chef@lists.opscode.com>
Date: Thursday, February 14, 2013 5:10 PM
To: "chef@lists.opscode.com mailto:chef@lists.opscode.com" <chef@lists.opscode.com mailto:chef@lists.opscode.com>
Subject: [chef] RE: Re: Re: Re: RE: Re: Chef11 HA

Can you define “to treat each as an isolated failure domain, make them HA”

From: Adam Jacob [mailto:adam@opscode.com]
Sent: Thursday, February 14/02/2013 20:05
To: chef@lists.opscode.com mailto:chef@lists.opscode.com
Subject: [chef] Re: Re: Re: RE: Re: Chef11 HA

We tend to recommend against this, as you are usually leaking both data and control across failure domains.

Think about it this way: when it fails, do you really want to add the increased latency? How about data replication when you are split brained? How do you fail back to being in multiple datacenters? Is one primary, the other passive?

The alternative is to treat each as an isolated failure domain, make them HA, and solve the consistency problem at the delivery of data level. It works much, much better.

Best,
Adam

From: Mark Pimentel <markpimentel22@gmail.com mailto:markpimentel22@gmail.com>
Reply-To: "chef@lists.opscode.com mailto:chef@lists.opscode.com" <chef@lists.opscode.com mailto:chef@lists.opscode.com>
Date: Thursday, February 14, 2013 9:56 AM
To: "chef@lists.opscode.com mailto:chef@lists.opscode.com" <chef@lists.opscode.com mailto:chef@lists.opscode.com>
Subject: [chef] Re: Re: RE: Re: Chef11 HA

Say this scenario is configured across sites, with each chef server serving different data centers. Would the keys be the same for both servers?

This would be used in a scenario where we have a main deployment chef whereby we would control all objects with the complementary servers replicating cookbook data as well as user and node information. The other servers would simply replicate back their node information.

On Thu, Feb 14, 2013 at 12:01 PM, Adam Jacob <adam@opscode.com mailto:adam@opscode.com> wrote:
Using DRBD for this is a good idea. If you share /var/opt/chef-server via
DRBD, you can use the normal mechanisms for starting/stopping the cluster,
and be certain you will have identical data.

Private Chef supports this configuration out of the box, fwiw, but it's
equally possible with Open Source Chef.

Best,
Adam

On 2/13/13 8:45 PM, "Baruch Shpirer" <baruchs@quickplay.com mailto:baruchs@quickplay.com> wrote:

Is there any draft to the HA procedure/setup?

Also, if I configure postgresql for master-master replication
and use drbd for the bookshelf folder
does it mean i got 2 identical servers in async mode?
Will clients be using same validation public key in both sites?

Baruch

-----Original Message-----
From: Seth Falcon [mailto:seth@opscode.com mailto:seth@opscode.com]
Sent: Monday, February 11/02/2013 17:35
To: <chef@lists.opscode.com mailto:chef@lists.opscode.com>
Subject: [chef] Re: Chef11 HA

On Feb 11, 2013, at 1:29 PM, Vaidas Jablonskis wrote:

This might be slightly unrelated to this conversion, but what I wonder
what is stored in postgres database?

All of the Chef object data is stored in the db. You can explore the
schema a bit like this:

root@chef-server-berkshelf:~# su - opscode-pgsql
$ bash
opscode-pgsql@chef-server-berkshelf:~$ which psql
/opt/chef-server/embedded/bin/psql
opscode-pgsql@chef-server-berkshelf:~$ psql opscode_chef psql (9.2.1)
Type "help" for help.

opscode_chef=# \d
List of relations
Schema | Name | Type | Owner
--------+-------------------------------+----------+---------------
public | checksums | table | opscode-pgsql
public | clients | table | opscode-pgsql
public | cookbook_version_checksums | table | opscode-pgsql
public | cookbook_version_dependencies | view | opscode-pgsql
public | cookbook_versions | table | opscode-pgsql
public | cookbook_versions_by_rank | view | opscode-pgsql
public | cookbooks | table | opscode-pgsql
public | cookbooks_id_seq | sequence | opscode-pgsql
public | data_bag_items | table | opscode-pgsql
public | data_bags | table | opscode-pgsql
public | environments | table | opscode-pgsql
public | joined_cookbook_version | view | opscode-pgsql
public | nodes | table | opscode-pgsql
public | osc_users | table | opscode-pgsql
public | roles | table | opscode-pgsql
public | sandboxed_checksums | table | opscode-pgsql
public | schema_info | table | opscode-pgsql
(17 rows)

And find the script used to initialize the schema here:
chef_db/priv/pgsql_schema.sql at master · chef-boneyard/chef_db · GitHub

What happens when this database gets corrupted or data is lost, for
instance?

Bad things happen. If the db data is lost or corrupted, so is your Chef
Server.

seth

--
Thanks,

Mark

--
Thanks,

Mark

Topic		Replies	Views
Chef server 11 and cookbooks storage Chef Infra (archive)	5	446	February 1, 2013
Chef-Server Files Chef Infra (archive)	1	241	October 5, 2011
Where is the storage of cooknooks in Chef Server 10? Chef Infra (archive)	2	268	March 11, 2013
Are chef-repo used for store server data in a one place controlled by CVS? Chef Infra (archive)	2	318	October 29, 2013
Mysql Master Slave Replication.community cookbooks? Chef Infra (archive)	0	465	December 4, 2016

Chef11 HA

Related Topics