Backup chef server


#1

How can I backup my Chef server so that when I perform a restore I get
an exact replica of the old server?

I’m trying this on an Ubuntu Oneiric instance on EC2, running chef
0.10.4-1 from the opscode apt repository.

My current approach is this: On backup, I shut down all chef-server
services (chef-client, chef-server-webui, chef-server, chef-expander,
chef-solr, and couchdb)
in that order. I then create a backup of the directories /etc/chef,
/etc/couchdb, /var/lib/chef, /var/lib/couchdb, /var/cache/chef,
/var/log/chef, and /var/log/couchdb, before I start the services in
the reverse order. (BTW: If I don’t back up the /var/cache/chef
directory, the server becomes completely inaccessible, complaining
about unauthorized access when I try to access it.)

On restore, I shut down the services, clean out the directories above,
and restore the backuped up files before I start the services again.

This procedure almost works, the status page in the webui has an
updated ‘Last check-in’ column, etc., except I get the following
errors when running chef-client:

INFO: HTTP Request Returned 500 Internal Server Error: Connection
failed - user: chef
ERROR: Server returned error for
http://localhost:4000/nodes/chef-server.telespor.org, retrying 1/5 in
3s

INFO: HTTP Request Returned 500 Internal Server Error: Connection
failed - user: chef
ERROR: Server returned error for
http://localhost:4000/nodes/chef-server.telespor.org, retrying 5/5 in
56s
INFO: HTTP Request Returned 500 Internal Server Error: Connection
failed - user: chef
ERROR: Running exception handlers
FATAL: Saving node information to /var/cache/chef/failed-run-data.json
ERROR: Exception handlers complete
ERROR: Net::HTTPFatalError: 500 "Internal Server Error"
FATAL: Stacktrace dumped to /var/cache/chef/chef-stacktrace.out
ERROR: Sleeping for 1800 seconds before trying again

/var/cache/chef/chef-stacktrace.out:
Net::HTTPFatalError: 500 “Internal Server Error”
/usr/lib/ruby/1.8/net/http.rb:2105:in error!' /usr/lib/ruby/vendor_ruby/chef/rest.rb:237:inapi_request’
/usr/lib/ruby/vendor_ruby/chef/rest.rb:288:in retriable_rest_request' /usr/lib/ruby/vendor_ruby/chef/rest.rb:218:inapi_request’
/usr/lib/ruby/vendor_ruby/chef/rest.rb:130:in put_rest' /usr/lib/ruby/vendor_ruby/chef/node.rb:626:insave’
/usr/lib/ruby/vendor_ruby/chef/client.rb:203:in save_updated_node' /usr/lib/ruby/vendor_ruby/chef/client.rb:161:inrun’
/usr/lib/ruby/vendor_ruby/chef/application/client.rb:239:in run_application' /usr/lib/ruby/vendor_ruby/chef/application/client.rb:229:inloop’
/usr/lib/ruby/vendor_ruby/chef/application/client.rb:229:in run_application' /usr/lib/ruby/vendor_ruby/chef/application.rb:67:inrun’
/usr/bin/chef-client:25

and the following in /var/log/chef/server.log:

merb : chef-server (api) : worker (port 4000) ~ Connection failed -
user: chef - (Bunny::ProtocolError)
/usr/lib/ruby/1.8/bunny/client08.rb:196:in open_connection' /usr/lib/ruby/1.8/bunny/client08.rb:397:instart’
/usr/lib/ruby/1.8/bunny/client08.rb:389:in loop' /usr/lib/ruby/1.8/bunny/client08.rb:389:instart’
/usr/lib/ruby/vendor_ruby/chef/index_queue/amqp_client.rb:45:in amqp_client' /usr/lib/ruby/vendor_ruby/chef/index_queue/amqp_client.rb:72:inqueue_for_object’
/usr/lib/ruby/vendor_ruby/chef/index_queue/indexable.rb:95:in publish_object' /usr/lib/ruby/vendor_ruby/chef/index_queue/indexable.rb:74:inadd_to_index’
/usr/lib/ruby/vendor_ruby/chef/couchdb.rb:118:in store' /usr/lib/ruby/vendor_ruby/chef/node.rb:618:incdb_save’
/usr/share/chef-server-api/app/controllers/nodes.rb:69:in update' /usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:insend’
/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in
_call_action' /usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:289:in_dispatch’
/usr/lib/ruby/1.8/merb-core/controller/merb_controller.rb:252:in _dispatch' /usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:102:indispatch_action’
/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:74:in handle' /usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:36:inhandle’
/usr/lib/ruby/1.8/merb-core/rack/application.rb:17:in call' /usr/lib/ruby/1.8/merb-core/rack/middleware/static.rb:28:incall’
/usr/lib/ruby/1.8/rack/content_length.rb:13:in call' /usr/lib/ruby/1.8/thin/connection.rb:76:inpre_process’
/usr/lib/ruby/1.8/thin/connection.rb:74:in catch' /usr/lib/ruby/1.8/thin/connection.rb:74:inpre_process’
/usr/lib/ruby/1.8/thin/connection.rb:57:in process' /usr/lib/ruby/1.8/thin/connection.rb:42:inreceive_data’
/usr/lib/ruby/1.8/eventmachine.rb:257:in run_machine' /usr/lib/ruby/1.8/eventmachine.rb:257:inrun’
/usr/lib/ruby/1.8/thin/backends/base.rb:57:in start' /usr/lib/ruby/1.8/thin/server.rb:156:instart’
/usr/lib/ruby/1.8/merb-core/rack/adapter/thin.rb:30:in start_server' /usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:298:instart_at_port’
/usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:128:in start' /usr/lib/ruby/1.8/merb-core/server.rb:174:inbootup’
/usr/lib/ruby/1.8/merb-core/server.rb:159:in daemonize' /usr/lib/ruby/1.8/merb-core/server.rb:143:infork’
/usr/lib/ruby/1.8/merb-core/server.rb:143:in daemonize' /usr/lib/ruby/1.8/merb-core/server.rb:35:instart’
/usr/lib/ruby/1.8/merb-core.rb:170:in `start’
/usr/sbin/chef-server:86

I would really like to resolve this error, as I don’t want to base my
entire infrastructure on a piece of software I don’t trust the backups
of.

I also realize that I could just provision a completely new
chef-server, push my chef repository (including environments, roles,
etc.), but this would require manually to replace every client’s
/etc/validation.pem, and mapping nodes to roles (this last step may be
unnecessary with the chef_server_backup.rb script). I really want to
be able to provision a new chef-server without having to touch the
clients.

If there is a way to accomplish this, please update the ‘Backing Up
Chef Server’ page on the wiki, as the methods mentioned there really
doesn’t work very well.

Magne Rasmussen


#2

On Tuesday, October 25, 2011 at 2:50 AM, Magne Rasmussen wrote:

How can I backup my Chef server so that when I perform a restore I get
an exact replica of the old server?

I’m trying this on an Ubuntu Oneiric instance on EC2, running chef
0.10.4-1 from the opscode apt repository.

My current approach is this: On backup, I shut down all chef-server
services (chef-client, chef-server-webui, chef-server, chef-expander,
chef-solr, and couchdb)
in that order. I then create a backup of the directories /etc/chef,
/etc/couchdb, /var/lib/chef, /var/lib/couchdb, /var/cache/chef,
/var/log/chef, and /var/log/couchdb, before I start the services in
the reverse order. (BTW: If I don’t back up the /var/cache/chef
directory, the server becomes completely inaccessible, complaining
about unauthorized access when I try to access it.)

On restore, I shut down the services, clean out the directories above,
and restore the backuped up files before I start the services again.

This procedure almost works, the status page in the webui has an
updated ‘Last check-in’ column, etc., except I get the following
errors when running chef-client:

INFO: HTTP Request Returned 500 Internal Server Error: Connection
failed - user: chef
ERROR: Server returned error for
http://localhost:4000/nodes/chef-server.telespor.org, retrying 1/5 in
3s

INFO: HTTP Request Returned 500 Internal Server Error: Connection
failed - user: chef
ERROR: Server returned error for
http://localhost:4000/nodes/chef-server.telespor.org, retrying 5/5 in
56s
INFO: HTTP Request Returned 500 Internal Server Error: Connection
failed - user: chef
ERROR: Running exception handlers
FATAL: Saving node information to /var/cache/chef/failed-run-data.json
ERROR: Exception handlers complete
ERROR: Net::HTTPFatalError: 500 "Internal Server Error"
FATAL: Stacktrace dumped to /var/cache/chef/chef-stacktrace.out
ERROR: Sleeping for 1800 seconds before trying again

/var/cache/chef/chef-stacktrace.out:
Net::HTTPFatalError: 500 “Internal Server Error”
/usr/lib/ruby/1.8/net/http.rb:2105:in error!' /usr/lib/ruby/vendor_ruby/chef/rest.rb:237:inapi_request’
/usr/lib/ruby/vendor_ruby/chef/rest.rb:288:in retriable_rest_request' /usr/lib/ruby/vendor_ruby/chef/rest.rb:218:inapi_request’
/usr/lib/ruby/vendor_ruby/chef/rest.rb:130:in put_rest' /usr/lib/ruby/vendor_ruby/chef/node.rb:626:insave’
/usr/lib/ruby/vendor_ruby/chef/client.rb:203:in save_updated_node' /usr/lib/ruby/vendor_ruby/chef/client.rb:161:inrun’
/usr/lib/ruby/vendor_ruby/chef/application/client.rb:239:in run_application' /usr/lib/ruby/vendor_ruby/chef/application/client.rb:229:inloop’
/usr/lib/ruby/vendor_ruby/chef/application/client.rb:229:in run_application' /usr/lib/ruby/vendor_ruby/chef/application.rb:67:inrun’
/usr/bin/chef-client:25

and the following in /var/log/chef/server.log:

merb : chef-server (api) : worker (port 4000) ~ Connection failed -
user: chef - (Bunny::ProtocolError)
/usr/lib/ruby/1.8/bunny/client08.rb:196:in open_connection' /usr/lib/ruby/1.8/bunny/client08.rb:397:instart’
/usr/lib/ruby/1.8/bunny/client08.rb:389:in loop' /usr/lib/ruby/1.8/bunny/client08.rb:389:instart’
/usr/lib/ruby/vendor_ruby/chef/index_queue/amqp_client.rb:45:in amqp_client' /usr/lib/ruby/vendor_ruby/chef/index_queue/amqp_client.rb:72:inqueue_for_object’
/usr/lib/ruby/vendor_ruby/chef/index_queue/indexable.rb:95:in publish_object' /usr/lib/ruby/vendor_ruby/chef/index_queue/indexable.rb:74:inadd_to_index’
/usr/lib/ruby/vendor_ruby/chef/couchdb.rb:118:in store' /usr/lib/ruby/vendor_ruby/chef/node.rb:618:incdb_save’
/usr/share/chef-server-api/app/controllers/nodes.rb:69:in update' /usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:insend’
/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in
_call_action' /usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:289:in_dispatch’
/usr/lib/ruby/1.8/merb-core/controller/merb_controller.rb:252:in _dispatch' /usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:102:indispatch_action’
/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:74:in handle' /usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:36:inhandle’
/usr/lib/ruby/1.8/merb-core/rack/application.rb:17:in call' /usr/lib/ruby/1.8/merb-core/rack/middleware/static.rb:28:incall’
/usr/lib/ruby/1.8/rack/content_length.rb:13:in call' /usr/lib/ruby/1.8/thin/connection.rb:76:inpre_process’
/usr/lib/ruby/1.8/thin/connection.rb:74:in catch' /usr/lib/ruby/1.8/thin/connection.rb:74:inpre_process’
/usr/lib/ruby/1.8/thin/connection.rb:57:in process' /usr/lib/ruby/1.8/thin/connection.rb:42:inreceive_data’
/usr/lib/ruby/1.8/eventmachine.rb:257:in run_machine' /usr/lib/ruby/1.8/eventmachine.rb:257:inrun’
/usr/lib/ruby/1.8/thin/backends/base.rb:57:in start' /usr/lib/ruby/1.8/thin/server.rb:156:instart’
/usr/lib/ruby/1.8/merb-core/rack/adapter/thin.rb:30:in start_server' /usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:298:instart_at_port’
/usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:128:in start' /usr/lib/ruby/1.8/merb-core/server.rb:174:inbootup’
/usr/lib/ruby/1.8/merb-core/server.rb:159:in daemonize' /usr/lib/ruby/1.8/merb-core/server.rb:143:infork’
/usr/lib/ruby/1.8/merb-core/server.rb:143:in daemonize' /usr/lib/ruby/1.8/merb-core/server.rb:35:instart’
/usr/lib/ruby/1.8/merb-core.rb:170:in `start’
/usr/sbin/chef-server:86

I would really like to resolve this error, as I don’t want to base my
entire infrastructure on a piece of software I don’t trust the backups
of.

I also realize that I could just provision a completely new
chef-server, push my chef repository (including environments, roles,
etc.), but this would require manually to replace every client’s
/etc/validation.pem, and mapping nodes to roles (this last step may be
unnecessary with the chef_server_backup.rb script). I really want to
be able to provision a new chef-server without having to touch the
clients.

If there is a way to accomplish this, please update the ‘Backing Up
Chef Server’ page on the wiki, as the methods mentioned there really
doesn’t work very well.

Magne Rasmussen
You need to either back up rabbitmq’s database or configure the chef user for rabbit. Where exactly that is may be dependent on the system packages for rabbit, but start looking at /var/lib/rabbitmq/mnesia/


Dan DeLeo