Can't re-create node after restoring CouchDB


#1

Hi!

I had a corrupted chef.couchdb and had to restore last night’s backup.
After re-uploading today’s changes, everything seems to work fine.

Only a node that I added during the day can’t be recreated. It’s not in
the node list (it didn’t yet exist last night, after all), it’s not in
CouchDB (as far as I can see), but bootstrapping the client gives me

HTTP Request Returned 409 Conflict: Client already exists
HTTP Request Returned 403 Forbidden: You are not allowed to take

this action.

knife node create MYNODENAME causes the following server stack trace:

—8<------8<------8<------8<------8<------8<------8<------8<—

chef-server (api) : worker (port 4000) ~ undefined method update_from!' for nil:NilClass - (NoMethodError)/usr/share/chef-server-api/app/controllers/nodes.rb:68:inupdate’/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in
send'/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in_call_action’/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:289:in
_dispatch'/usr/lib/ruby/1.8/merb-core/controller/merb_controller.rb:252:in_dispatch’/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:102:in
dispatch_action'/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:74:inhandle’/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:36:in
handle'/usr/lib/ruby/1.8/merb-core/rack/application.rb:17:incall’
/usr/lib/ruby/1.8/merb-core/rack/middleware/static.rb:28:in
call'/usr/lib/ruby/1.8/rack/content_length.rb:13:incall’
/usr/lib/ruby/1.8/thin/connection.rb:76:in pre_process' /usr/lib/ruby/1.8/thin/connection.rb:74:incatch’
/usr/lib/ruby/1.8/thin/connection.rb:74:in pre_process' /usr/lib/ruby/1.8/thin/connection.rb:57:inprocess’
/usr/lib/ruby/1.8/thin/connection.rb:42:in receive_data' /usr/lib/ruby/vendor_ruby/eventmachine.rb:256:inrun_machine’
/usr/lib/ruby/vendor_ruby/eventmachine.rb:256:in run' /usr/lib/ruby/1.8/thin/backends/base.rb:57:instart’
/usr/lib/ruby/1.8/thin/server.rb:156:in start' /usr/lib/ruby/1.8/merb-core/rack/adapter/thin.rb:30:instart_server’
/usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:298:in start_at_port' /usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:95:inspawn_worker’
/usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:139:in start' /usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:138:inupto’
/usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:138:in start' /usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:137:incatch’
/usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:137:in start' /usr/lib/ruby/1.8/merb-core/server.rb:174:inbootup’
/usr/lib/ruby/1.8/merb-core/server.rb:159:in daemonize' /usr/lib/ruby/1.8/merb-core/server.rb:143:infork’
/usr/lib/ruby/1.8/merb-core/server.rb:143:in daemonize' /usr/lib/ruby/1.8/merb-core/server.rb:35:instart’
/usr/lib/ruby/1.8/merb-core.rb:170:in `start’
/usr/sbin/chef-server:86

—8<------8<------8<------8<------8<------8<------8<------8<—

Is there something outside CouchDB that prevents the node from being
created?

Best regards,
Jochen


#2

delete the client from chef server using knife (knife client delete ), and delete the old client.pem from the node itself. then run
chef-client on the node, and it will reregister itself.

On Tue, Mar 26, 2013 at 12:45 PM, Jochen Lillich jochen@freistil.it wrote:

Hi!

I had a corrupted chef.couchdb and had to restore last night’s backup.
After re-uploading today’s changes, everything seems to work fine.

Only a node that I added during the day can’t be recreated. It’s not in
the node list (it didn’t yet exist last night, after all), it’s not in
CouchDB (as far as I can see), but bootstrapping the client gives me

HTTP Request Returned 409 Conflict: Client already exists
HTTP Request Returned 403 Forbidden: You are not allowed to take

this action.

knife node create MYNODENAME causes the following server stack trace:

—8<------8<------8<------8<------8<------8<------8<------8<—

chef-server (api) : worker (port 4000) ~ undefined method `update_from!'
for nil:NilClass -
(NoMethodError)/usr/share/chef-server-api/app/controllers/nodes.rb:68:in

update'/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:insend’/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in

_call_action'/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:289:in_dispatch’/usr/lib/ruby/1.8/merb-core/controller/merb_controller.rb:252:in
_dispatch'/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:102:indispatch_action’/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:74:in
handle'/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:36:inhandle’/usr/lib/ruby/1.8/merb-core/rack/application.rb:17:in call' /usr/lib/ruby/1.8/merb-core/rack/middleware/static.rb:28:incall’/usr/lib/ruby/1.8/rack/content_length.rb:13:in call' /usr/lib/ruby/1.8/thin/connection.rb:76:inpre_process’
/usr/lib/ruby/1.8/thin/connection.rb:74:in catch' /usr/lib/ruby/1.8/thin/connection.rb:74:inpre_process’
/usr/lib/ruby/1.8/thin/connection.rb:57:in process' /usr/lib/ruby/1.8/thin/connection.rb:42:inreceive_data’
/usr/lib/ruby/vendor_ruby/eventmachine.rb:256:in run_machine' /usr/lib/ruby/vendor_ruby/eventmachine.rb:256:inrun’
/usr/lib/ruby/1.8/thin/backends/base.rb:57:in start' /usr/lib/ruby/1.8/thin/server.rb:156:instart’
/usr/lib/ruby/1.8/merb-core/rack/adapter/thin.rb:30:in start_server' /usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:298:instart_at_port’
/usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:95:in spawn_worker' /usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:139:instart’
/usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:138:in upto' /usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:138:instart’
/usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:137:in catch' /usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:137:instart’
/usr/lib/ruby/1.8/merb-core/server.rb:174:in bootup' /usr/lib/ruby/1.8/merb-core/server.rb:159:indaemonize’
/usr/lib/ruby/1.8/merb-core/server.rb:143:in fork' /usr/lib/ruby/1.8/merb-core/server.rb:143:indaemonize’
/usr/lib/ruby/1.8/merb-core/server.rb:35:in start' /usr/lib/ruby/1.8/merb-core.rb:170:instart’
/usr/sbin/chef-server:86

—8<------8<------8<------8<------8<------8<------8<------8<—

Is there something outside CouchDB that prevents the node from being
created?

Best regards,
Jochen


#3

Ranjib Dey wrote:

delete the client from chef server using knife (knife client delete
), and delete the old client.pem from the node itself. then
run chef-client on the node, and it will reregister itself.

I already tried that, but there is no client with that name:

$ knife client delete MYNODE
Do you really want to delete MYNODE? (Y/N) y
ERROR: JSON::ParserError: 743: unexpected token at 'null'

Neither knife client list nor the CouchDB web UI display the client name.

That’s why I asked if there could be something outside CouchDB (maybe the server filesystem) that prevents re-creating the client. Some kind of remains from my node creation today.

Thanks,
Jochen


#4

You might have not taken a consistent backup?

Clients aren’t stored on the file-system – only cookbook manifests
(IIRC), and only in chef10, it’s different (Bookshelf) in chef11.

Jump into your database and delete the matching client documents.

–AJ

On 27 March 2013 09:01, Jochen Lillich jochen@freistil.it wrote:

Ranjib Dey wrote:

delete the client from chef server using knife (knife client delete
), and delete the old client.pem from the node itself. then
run chef-client on the node, and it will reregister itself.

I already tried that, but there is no client with that name:

$ knife client delete MYNODE
Do you really want to delete MYNODE? (Y/N) y
ERROR: JSON::ParserError: 743: unexpected token at 'null'

Neither knife client list nor the CouchDB web UI display the client name.

That’s why I asked if there could be something outside CouchDB (maybe the server filesystem) that prevents re-creating the client. Some kind of remains from my node creation today.

Thanks,
Jochen


#5

fujin, the public keys of the clients are not stored? whats does knife
client list enumerates then?

On Tue, Mar 26, 2013 at 1:03 PM, AJ Christensen aj@junglist.gen.nz wrote:

You might have not taken a consistent backup?

Clients aren’t stored on the file-system – only cookbook manifests
(IIRC), and only in chef10, it’s different (Bookshelf) in chef11.

Jump into your database and delete the matching client documents.

–AJ

On 27 March 2013 09:01, Jochen Lillich jochen@freistil.it wrote:

Ranjib Dey wrote:

delete the client from chef server using knife (knife client delete
), and delete the old client.pem from the node itself. then
run chef-client on the node, and it will reregister itself.

I already tried that, but there is no client with that name:

$ knife client delete MYNODE
Do you really want to delete MYNODE? (Y/N) y
ERROR: JSON::ParserError: 743: unexpected token at 'null'

Neither knife client list nor the CouchDB web UI display the client
name.

That’s why I asked if there could be something outside CouchDB (maybe
the server filesystem) that prevents re-creating the client. Some kind of
remains from my node creation today.

Thanks,
Jochen


#6

AJ Christensen wrote:

Jump into your database and delete the matching client documents.

As I already wrote: Neither knife client list nor the CouchDB web UI display the client name.

I can create a node with another name without problems, BTW.


#7

Could it be something in the Solr index? A “knife search node” for the name doesn’t have a result for the client, though.


#8

no… knife node/client set of commands interacts with couch only. Try
debugging it from irb/pry.
irb>
require ‘chef’
Chef::Config.from_file(’<knife.rb>’)
n = Chef::Node.load(‘name’)
etc.
also tail the chef server api log

On Tue, Mar 26, 2013 at 1:38 PM, Jochen Lillich jochen@freistil.it wrote:

Could it be something in the Solr index? A “knife search node” for the
name doesn’t have a result for the client, though.


#9

Right, clients’ public keys not stored on disk, AFAIK, (?) – stored in Couch.

As I mentioned, you probably did not take a consistent backup, or
there was some problem with restoring a document. Have you inspected
all documents using the _clients design document? I would anticipate
finding a dodgy document, removing/editing it, and being able to
proceed.

–AJ

On 27 March 2013 09:05, Ranjib Dey dey.ranjib@gmail.com wrote:

fujin, the public keys of the clients are not stored? whats does knife
client list enumerates then?

On Tue, Mar 26, 2013 at 1:03 PM, AJ Christensen aj@junglist.gen.nz wrote:

You might have not taken a consistent backup?

Clients aren’t stored on the file-system – only cookbook manifests
(IIRC), and only in chef10, it’s different (Bookshelf) in chef11.

Jump into your database and delete the matching client documents.

–AJ

On 27 March 2013 09:01, Jochen Lillich jochen@freistil.it wrote:

Ranjib Dey wrote:

delete the client from chef server using knife (knife client delete
), and delete the old client.pem from the node itself. then
run chef-client on the node, and it will reregister itself.

I already tried that, but there is no client with that name:

$ knife client delete MYNODE
Do you really want to delete MYNODE? (Y/N) y
ERROR: JSON::ParserError: 743: unexpected token at 'null'

Neither knife client list nor the CouchDB web UI display the client
name.

That’s why I asked if there could be something outside CouchDB (maybe
the server filesystem) that prevents re-creating the client. Some kind of
remains from my node creation today.

Thanks,
Jochen


#10

AJ Christensen wrote:

As I mentioned, you probably did not take a consistent backup

I seem to have solved it just this minute. You’re right – it was pilot error.

I had not done a consistent restore. According to http://lists.opscode.com/sympa/arc/chef/2012-05/msg00258.html, there actually are files that somehow reference the database.

I did another restore, this time not only /var/lib/couchdb but also /var/lib/chef and /var/lib/rabbitmq. Now, I’ve been able to reregister the node.

Thanks for your help!
Jochen