Can't re-create node after restoring CouchDB

Hi!

I had a corrupted chef.couchdb and had to restore last night’s backup.
After re-uploading today’s changes, everything seems to work fine.

Only a node that I added during the day can’t be recreated. It’s not in
the node list (it didn’t yet exist last night, after all), it’s not in
CouchDB (as far as I can see), but bootstrapping the client gives me

HTTP Request Returned 409 Conflict: Client already exists
HTTP Request Returned 403 Forbidden: You are not allowed to take

this action.

knife node create MYNODENAME causes the following server stack trace:

—8<------8<------8<------8<------8<------8<------8<------8<—

chef-server (api) : worker (port 4000) ~ undefined method update_from!' for nil:NilClass - (NoMethodError)/usr/share/chef-server-api/app/controllers/nodes.rb:68:inupdate’/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in
send'/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in_call_action’/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:289:in
_dispatch'/usr/lib/ruby/1.8/merb-core/controller/merb_controller.rb:252:in_dispatch’/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:102:in
dispatch_action'/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:74:inhandle’/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:36:in
handle'/usr/lib/ruby/1.8/merb-core/rack/application.rb:17:incall’
/usr/lib/ruby/1.8/merb-core/rack/middleware/static.rb:28:in
call'/usr/lib/ruby/1.8/rack/content_length.rb:13:incall’
/usr/lib/ruby/1.8/thin/connection.rb:76:in pre_process' /usr/lib/ruby/1.8/thin/connection.rb:74:incatch’
/usr/lib/ruby/1.8/thin/connection.rb:74:in pre_process' /usr/lib/ruby/1.8/thin/connection.rb:57:inprocess’
/usr/lib/ruby/1.8/thin/connection.rb:42:in receive_data' /usr/lib/ruby/vendor_ruby/eventmachine.rb:256:inrun_machine’
/usr/lib/ruby/vendor_ruby/eventmachine.rb:256:in run' /usr/lib/ruby/1.8/thin/backends/base.rb:57:instart’
/usr/lib/ruby/1.8/thin/server.rb:156:in start' /usr/lib/ruby/1.8/merb-core/rack/adapter/thin.rb:30:instart_server’
/usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:298:in start_at_port' /usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:95:inspawn_worker’
/usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:139:in start' /usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:138:inupto’
/usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:138:in start' /usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:137:incatch’
/usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:137:in start' /usr/lib/ruby/1.8/merb-core/server.rb:174:inbootup’
/usr/lib/ruby/1.8/merb-core/server.rb:159:in daemonize' /usr/lib/ruby/1.8/merb-core/server.rb:143:infork’
/usr/lib/ruby/1.8/merb-core/server.rb:143:in daemonize' /usr/lib/ruby/1.8/merb-core/server.rb:35:instart’
/usr/lib/ruby/1.8/merb-core.rb:170:in `start’
/usr/sbin/chef-server:86

—8<------8<------8<------8<------8<------8<------8<------8<—

Is there something outside CouchDB that prevents the node from being
created?

Best regards,
Jochen

delete the client from chef server using knife (knife client delete ), and delete the old client.pem from the node itself. then run
chef-client on the node, and it will reregister itself.

On Tue, Mar 26, 2013 at 12:45 PM, Jochen Lillich jochen@freistil.it wrote:

Hi!

I had a corrupted chef.couchdb and had to restore last night's backup.
After re-uploading today's changes, everything seems to work fine.

Only a node that I added during the day can't be recreated. It's not in
the node list (it didn't yet exist last night, after all), it's not in
CouchDB (as far as I can see), but bootstrapping the client gives me

HTTP Request Returned 409 Conflict: Client already exists
HTTP Request Returned 403 Forbidden: You are not allowed to take

this action.

knife node create MYNODENAME causes the following server stack trace:

---8<------8<------8<------8<------8<------8<------8<------8<---

chef-server (api) : worker (port 4000) ~ undefined method `update_from!'
for nil:NilClass -
(NoMethodError)/usr/share/chef-server-api/app/controllers/nodes.rb:68:in

update'/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in send'/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in

_call_action'/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:289:in _dispatch'/usr/lib/ruby/1.8/merb-core/controller/merb_controller.rb:252:in
_dispatch'/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:102:in dispatch_action'/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:74:in
handle'/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:36:in handle'/usr/lib/ruby/1.8/merb-core/rack/application.rb:17:in call' /usr/lib/ruby/1.8/merb-core/rack/middleware/static.rb:28:in call'/usr/lib/ruby/1.8/rack/content_length.rb:13:in call' /usr/lib/ruby/1.8/thin/connection.rb:76:in pre_process'
/usr/lib/ruby/1.8/thin/connection.rb:74:in catch' /usr/lib/ruby/1.8/thin/connection.rb:74:in pre_process'
/usr/lib/ruby/1.8/thin/connection.rb:57:in process' /usr/lib/ruby/1.8/thin/connection.rb:42:in receive_data'
/usr/lib/ruby/vendor_ruby/eventmachine.rb:256:in run_machine' /usr/lib/ruby/vendor_ruby/eventmachine.rb:256:in run'
/usr/lib/ruby/1.8/thin/backends/base.rb:57:in start' /usr/lib/ruby/1.8/thin/server.rb:156:in start'
/usr/lib/ruby/1.8/merb-core/rack/adapter/thin.rb:30:in start_server' /usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:298:in start_at_port'
/usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:95:in spawn_worker' /usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:139:in start'
/usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:138:in upto' /usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:138:in start'
/usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:137:in catch' /usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:137:in start'
/usr/lib/ruby/1.8/merb-core/server.rb:174:in bootup' /usr/lib/ruby/1.8/merb-core/server.rb:159:in daemonize'
/usr/lib/ruby/1.8/merb-core/server.rb:143:in fork' /usr/lib/ruby/1.8/merb-core/server.rb:143:in daemonize'
/usr/lib/ruby/1.8/merb-core/server.rb:35:in start' /usr/lib/ruby/1.8/merb-core.rb:170:in start'
/usr/sbin/chef-server:86

---8<------8<------8<------8<------8<------8<------8<------8<---

Is there something outside CouchDB that prevents the node from being
created?

Best regards,
Jochen

Ranjib Dey wrote:

delete the client from chef server using knife (knife client delete
), and delete the old client.pem from the node itself. then
run chef-client on the node, and it will reregister itself.

I already tried that, but there is no client with that name:

$ knife client delete MYNODE
Do you really want to delete MYNODE? (Y/N) y
ERROR: JSON::ParserError: 743: unexpected token at 'null'

Neither knife client list nor the CouchDB web UI display the client name.

That's why I asked if there could be something outside CouchDB (maybe the server filesystem) that prevents re-creating the client. Some kind of remains from my node creation today.

Thanks,
Jochen

You might have not taken a consistent backup?

Clients aren't stored on the file-system -- only cookbook manifests
(IIRC), and only in chef10, it's different (Bookshelf) in chef11.

Jump into your database and delete the matching client documents.

--AJ

On 27 March 2013 09:01, Jochen Lillich jochen@freistil.it wrote:

Ranjib Dey wrote:

delete the client from chef server using knife (knife client delete
), and delete the old client.pem from the node itself. then
run chef-client on the node, and it will reregister itself.

I already tried that, but there is no client with that name:

$ knife client delete MYNODE
Do you really want to delete MYNODE? (Y/N) y
ERROR: JSON::ParserError: 743: unexpected token at 'null'

Neither knife client list nor the CouchDB web UI display the client name.

That's why I asked if there could be something outside CouchDB (maybe the server filesystem) that prevents re-creating the client. Some kind of remains from my node creation today.

Thanks,
Jochen

fujin, the public keys of the clients are not stored? whats does knife
client list enumerates then?

On Tue, Mar 26, 2013 at 1:03 PM, AJ Christensen aj@junglist.gen.nz wrote:

You might have not taken a consistent backup?

Clients aren't stored on the file-system -- only cookbook manifests
(IIRC), and only in chef10, it's different (Bookshelf) in chef11.

Jump into your database and delete the matching client documents.

--AJ

On 27 March 2013 09:01, Jochen Lillich jochen@freistil.it wrote:

Ranjib Dey wrote:

delete the client from chef server using knife (knife client delete
), and delete the old client.pem from the node itself. then
run chef-client on the node, and it will reregister itself.

I already tried that, but there is no client with that name:

$ knife client delete MYNODE
Do you really want to delete MYNODE? (Y/N) y
ERROR: JSON::ParserError: 743: unexpected token at 'null'

Neither knife client list nor the CouchDB web UI display the client
name.

That's why I asked if there could be something outside CouchDB (maybe
the server filesystem) that prevents re-creating the client. Some kind of
remains from my node creation today.

Thanks,
Jochen

AJ Christensen wrote:

Jump into your database and delete the matching client documents.

As I already wrote: Neither knife client list nor the CouchDB web UI display the client name.

I can create a node with another name without problems, BTW.

Could it be something in the Solr index? A “knife search node” for the name doesn’t have a result for the client, though.

no.. knife node/client set of commands interacts with couch only. Try
debugging it from irb/pry.
irb>
require 'chef'
Chef::Config.from_file('<knife.rb>')
n = Chef::Node.load('name')
etc.
also tail the chef server api log

On Tue, Mar 26, 2013 at 1:38 PM, Jochen Lillich jochen@freistil.it wrote:

Could it be something in the Solr index? A "knife search node" for the
name doesn't have a result for the client, though.

Right, clients' public keys not stored on disk, AFAIK, (?) -- stored in Couch.

As I mentioned, you probably did not take a consistent backup, or
there was some problem with restoring a document. Have you inspected
all documents using the _clients design document? I would anticipate
finding a dodgy document, removing/editing it, and being able to
proceed.

--AJ

On 27 March 2013 09:05, Ranjib Dey dey.ranjib@gmail.com wrote:

fujin, the public keys of the clients are not stored? whats does knife
client list enumerates then?

On Tue, Mar 26, 2013 at 1:03 PM, AJ Christensen aj@junglist.gen.nz wrote:

You might have not taken a consistent backup?

Clients aren't stored on the file-system -- only cookbook manifests
(IIRC), and only in chef10, it's different (Bookshelf) in chef11.

Jump into your database and delete the matching client documents.

--AJ

On 27 March 2013 09:01, Jochen Lillich jochen@freistil.it wrote:

Ranjib Dey wrote:

delete the client from chef server using knife (knife client delete
), and delete the old client.pem from the node itself. then
run chef-client on the node, and it will reregister itself.

I already tried that, but there is no client with that name:

$ knife client delete MYNODE
Do you really want to delete MYNODE? (Y/N) y
ERROR: JSON::ParserError: 743: unexpected token at 'null'

Neither knife client list nor the CouchDB web UI display the client
name.

That's why I asked if there could be something outside CouchDB (maybe
the server filesystem) that prevents re-creating the client. Some kind of
remains from my node creation today.

Thanks,
Jochen

AJ Christensen wrote:

As I mentioned, you probably did not take a consistent backup

I seem to have solved it just this minute. You're right -- it was pilot error.

I had not done a consistent restore. According to chef - [chef] Re: undefined method `couchdb_rev=' for nil:NilClass, there actually are files that somehow reference the database.

I did another restore, this time not only /var/lib/couchdb but also /var/lib/chef and /var/lib/rabbitmq. Now, I've been able to reregister the node.

Thanks for your help!
Jochen