# Can't re-create node after restoring CouchDB

**URL:** <https://discourse.chef.io/t/cant-re-create-node-after-restoring-couchdb/3798>\
**Category:** Chef Infra (archive)\
**Created:** [March 26, 2013, 7:45pm UTC](https://discourse.chef.io/t/cant-re-create-node-after-restoring-couchdb/3798 "2013-03-26T19:45:44Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jochen\_Lillich1](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/jochen_lillich1/32/572_2.png) [@Jochen\_Lillich1](https://discourse.chef.io/u/Jochen_Lillich1)\
**Post date:** [March 26, 2013, 7:45pm UTC](https://discourse.chef.io/t/cant-re-create-node-after-restoring-couchdb/3798/1 "2013-03-26T19:45:44Z")

</div>

Hi!

I had a corrupted chef.couchdb and had to restore last night’s backup.  
After re-uploading today’s changes, everything seems to work fine.

Only a node that I added during the day can’t be recreated. It’s not in  
the node list (it didn’t yet exist last night, after all), it’s not in  
CouchDB (as far as I can see), but bootstrapping the client gives me

```
HTTP Request Returned 409 Conflict: Client already exists
HTTP Request Returned 403 Forbidden: You are not allowed to take

```

this action.

`knife node create MYNODENAME` causes the following server stack trace:

—8\<------8\<------8\<------8\<------8\<------8\<------8\<------8\<—

chef-server (api) : worker (port 4000) ~ undefined method `update_from!' for nil:NilClass - (NoMethodError)/usr/share/chef-server-api/app/controllers/nodes.rb:68:in`update’/usr/lib/ruby/1.8/merb-core/controller/abstract\_controller.rb:315:in  
`send'/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in`\_call\_action’/usr/lib/ruby/1.8/merb-core/controller/abstract\_controller.rb:289:in  
`_dispatch'/usr/lib/ruby/1.8/merb-core/controller/merb_controller.rb:252:in`\_dispatch’/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:102:in  
`dispatch_action'/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:74:in`handle’/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:36:in  
`handle'/usr/lib/ruby/1.8/merb-core/rack/application.rb:17:in`call’  
/usr/lib/ruby/1.8/merb-core/rack/middleware/static.rb:28:in  
`call'/usr/lib/ruby/1.8/rack/content_length.rb:13:in`call’  
/usr/lib/ruby/1.8/thin/connection.rb:76:in `pre_process' /usr/lib/ruby/1.8/thin/connection.rb:74:in`catch’  
/usr/lib/ruby/1.8/thin/connection.rb:74:in `pre_process' /usr/lib/ruby/1.8/thin/connection.rb:57:in`process’  
/usr/lib/ruby/1.8/thin/connection.rb:42:in `receive_data' /usr/lib/ruby/vendor_ruby/eventmachine.rb:256:in`run\_machine’  
/usr/lib/ruby/vendor\_ruby/eventmachine.rb:256:in `run' /usr/lib/ruby/1.8/thin/backends/base.rb:57:in`start’  
/usr/lib/ruby/1.8/thin/server.rb:156:in `start' /usr/lib/ruby/1.8/merb-core/rack/adapter/thin.rb:30:in`start\_server’  
/usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:298:in `start_at_port' /usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:95:in`spawn\_worker’  
/usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:139:in `start' /usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:138:in`upto’  
/usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:138:in `start' /usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:137:in`catch’  
/usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:137:in `start' /usr/lib/ruby/1.8/merb-core/server.rb:174:in`bootup’  
/usr/lib/ruby/1.8/merb-core/server.rb:159:in `daemonize' /usr/lib/ruby/1.8/merb-core/server.rb:143:in`fork’  
/usr/lib/ruby/1.8/merb-core/server.rb:143:in `daemonize' /usr/lib/ruby/1.8/merb-core/server.rb:35:in`start’  
/usr/lib/ruby/1.8/merb-core.rb:170:in `start’  
/usr/sbin/chef-server:86

—8\<------8\<------8\<------8\<------8\<------8\<------8\<------8\<—

Is there something outside CouchDB that prevents the node from being  
created?

Best regards,  
Jochen

---

<div class="post-metadata">

**Author:** ![Ranjib](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/ranjib/32/1040_2.png) [@Ranjib](https://discourse.chef.io/u/Ranjib)\
**Post date:** [March 26, 2013, 7:48pm UTC](https://discourse.chef.io/t/cant-re-create-node-after-restoring-couchdb/3798/2 "2013-03-26T19:48:11Z")

</div>

delete the client from chef server using knife (knife client delete ), and delete the old client.pem from the node itself. then run  
chef-client on the node, and it will reregister itself.

On Tue, Mar 26, 2013 at 12:45 PM, Jochen Lillich [jochen@freistil.it](mailto:jochen@freistil.it) wrote:

> Hi!
> 
> I had a corrupted chef.couchdb and had to restore last night's backup.  
> After re-uploading today's changes, everything seems to work fine.
> 
> Only a node that I added during the day can't be recreated. It's not in  
> the node list (it didn't yet exist last night, after all), it's not in  
> CouchDB (as far as I can see), but bootstrapping the client gives me
> 
> ```
> HTTP Request Returned 409 Conflict: Client already exists
> HTTP Request Returned 403 Forbidden: You are not allowed to take
> 
> ```
> 
> this action.
> 
> `knife node create MYNODENAME` causes the following server stack trace:
> 
> ---8\<------8\<------8\<------8\<------8\<------8\<------8\<------8\<---
> 
> chef-server (api) : worker (port 4000) ~ undefined method `update\_from!'  
> for nil:NilClass -  
> (NoMethodError)/usr/share/chef-server-api/app/controllers/nodes.rb:68:in
> 
> `update'/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in `send'/usr/lib/ruby/1.8/merb-core/controller/abstract\_controller.rb:315:in
> 
> `_call_action'/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:289:in `\_dispatch'/usr/lib/ruby/1.8/merb-core/controller/merb\_controller.rb:252:in  
> `_dispatch'/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:102:in `dispatch\_action'/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:74:in  
> `handle'/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:36:in `handle'/usr/lib/ruby/1.8/merb-core/rack/application.rb:17:in `call' /usr/lib/ruby/1.8/merb-core/rack/middleware/static.rb:28:in `call'/usr/lib/ruby/1.8/rack/content\_length.rb:13:in `call' /usr/lib/ruby/1.8/thin/connection.rb:76:in `pre\_process'  
> /usr/lib/ruby/1.8/thin/connection.rb:74:in `catch' /usr/lib/ruby/1.8/thin/connection.rb:74:in `pre\_process'  
> /usr/lib/ruby/1.8/thin/connection.rb:57:in `process' /usr/lib/ruby/1.8/thin/connection.rb:42:in `receive\_data'  
> /usr/lib/ruby/vendor\_ruby/eventmachine.rb:256:in `run_machine' /usr/lib/ruby/vendor_ruby/eventmachine.rb:256:in `run'  
> /usr/lib/ruby/1.8/thin/backends/base.rb:57:in `start' /usr/lib/ruby/1.8/thin/server.rb:156:in `start'  
> /usr/lib/ruby/1.8/merb-core/rack/adapter/thin.rb:30:in `start_server' /usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:298:in `start\_at\_port'  
> /usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:95:in `spawn_worker' /usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:139:in `start'  
> /usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:138:in `upto' /usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:138:in `start'  
> /usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:137:in `catch' /usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:137:in `start'  
> /usr/lib/ruby/1.8/merb-core/server.rb:174:in `bootup' /usr/lib/ruby/1.8/merb-core/server.rb:159:in `daemonize'  
> /usr/lib/ruby/1.8/merb-core/server.rb:143:in `fork' /usr/lib/ruby/1.8/merb-core/server.rb:143:in `daemonize'  
> /usr/lib/ruby/1.8/merb-core/server.rb:35:in `start' /usr/lib/ruby/1.8/merb-core.rb:170:in `start'  
> /usr/sbin/chef-server:86
> 
> ---8\<------8\<------8\<------8\<------8\<------8\<------8\<------8\<---
> 
> Is there something outside CouchDB that prevents the node from being  
> created?
> 
> Best regards,  
> Jochen

---

<div class="post-metadata">

**Author:** ![Jochen\_Lillich1](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/jochen_lillich1/32/572_2.png) [@Jochen\_Lillich1](https://discourse.chef.io/u/Jochen_Lillich1)\
**Post date:** [March 26, 2013, 8:01pm UTC](https://discourse.chef.io/t/cant-re-create-node-after-restoring-couchdb/3798/3 "2013-03-26T20:01:20Z")

</div>

Ranjib Dey wrote:

> delete the client from chef server using knife (knife client delete  
> ), and delete the old client.pem from the node itself. then  
> run chef-client on the node, and it will reregister itself.

I already tried that, but there is no client with that name:

```
$ knife client delete MYNODE
Do you really want to delete MYNODE? (Y/N) y
ERROR: JSON::ParserError: 743: unexpected token at 'null'

```

Neither `knife client list` nor the CouchDB web UI display the client name.

That's why I asked if there could be something outside CouchDB (maybe the server filesystem) that prevents re-creating the client. Some kind of remains from my node creation today.

Thanks,  
Jochen

---

<div class="post-metadata">

**Author:** ![AJ\_Christensen](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/aj_christensen/32/525_2.png) [@AJ\_Christensen](https://discourse.chef.io/u/AJ_Christensen)\
**Post date:** [March 26, 2013, 8:03pm UTC](https://discourse.chef.io/t/cant-re-create-node-after-restoring-couchdb/3798/4 "2013-03-26T20:03:43Z")

</div>

You might have not taken a consistent backup?

Clients aren't stored on the file-system -- only cookbook manifests  
(IIRC), and only in chef10, it's different (Bookshelf) in chef11.

Jump into your database and delete the matching client documents.

--AJ

On 27 March 2013 09:01, Jochen Lillich [jochen@freistil.it](mailto:jochen@freistil.it) wrote:

> Ranjib Dey wrote:
> 
> > delete the client from chef server using knife (knife client delete  
> > ), and delete the old client.pem from the node itself. then  
> > run chef-client on the node, and it will reregister itself.
> 
> I already tried that, but there is no client with that name:
> 
> ```
> $ knife client delete MYNODE
> Do you really want to delete MYNODE? (Y/N) y
> ERROR: JSON::ParserError: 743: unexpected token at 'null'
> 
> ```
> 
> Neither `knife client list` nor the CouchDB web UI display the client name.
> 
> That's why I asked if there could be something outside CouchDB (maybe the server filesystem) that prevents re-creating the client. Some kind of remains from my node creation today.
> 
> Thanks,  
> Jochen

---

<div class="post-metadata">

**Author:** ![Ranjib](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/ranjib/32/1040_2.png) [@Ranjib](https://discourse.chef.io/u/Ranjib)\
**Post date:** [March 26, 2013, 8:05pm UTC](https://discourse.chef.io/t/cant-re-create-node-after-restoring-couchdb/3798/5 "2013-03-26T20:05:12Z")

</div>

fujin, the public keys of the clients are not stored? whats does knife  
client list enumerates then?

On Tue, Mar 26, 2013 at 1:03 PM, AJ Christensen [aj@junglist.gen.nz](mailto:aj@junglist.gen.nz) wrote:

> You might have not taken a consistent backup?
> 
> Clients aren't stored on the file-system -- only cookbook manifests  
> (IIRC), and only in chef10, it's different (Bookshelf) in chef11.
> 
> Jump into your database and delete the matching client documents.
> 
> --AJ
> 
> On 27 March 2013 09:01, Jochen Lillich [jochen@freistil.it](mailto:jochen@freistil.it) wrote:
> 
> > Ranjib Dey wrote:
> > 
> > > delete the client from chef server using knife (knife client delete  
> > > ), and delete the old client.pem from the node itself. then  
> > > run chef-client on the node, and it will reregister itself.
> > 
> > I already tried that, but there is no client with that name:
> > 
> > ```
> > $ knife client delete MYNODE
> > Do you really want to delete MYNODE? (Y/N) y
> > ERROR: JSON::ParserError: 743: unexpected token at 'null'
> > 
> > ```
> > 
> > Neither `knife client list` nor the CouchDB web UI display the client  
> > name.
> > 
> > That's why I asked if there could be something outside CouchDB (maybe  
> > the server filesystem) that prevents re-creating the client. Some kind of  
> > remains from my node creation today.
> > 
> > Thanks,  
> > Jochen

---

<div class="post-metadata">

**Author:** ![Jochen\_Lillich1](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/jochen_lillich1/32/572_2.png) [@Jochen\_Lillich1](https://discourse.chef.io/u/Jochen_Lillich1)\
**Post date:** [March 26, 2013, 8:06pm UTC](https://discourse.chef.io/t/cant-re-create-node-after-restoring-couchdb/3798/6 "2013-03-26T20:06:26Z")

</div>

AJ Christensen wrote:

> Jump into your database and delete the matching client documents.

As I already wrote: Neither `knife client list` nor the CouchDB web UI display the client name.

I can create a node with another name without problems, BTW.

---

<div class="post-metadata">

**Author:** ![Jochen\_Lillich1](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/jochen_lillich1/32/572_2.png) [@Jochen\_Lillich1](https://discourse.chef.io/u/Jochen_Lillich1)\
**Post date:** [March 26, 2013, 8:38pm UTC](https://discourse.chef.io/t/cant-re-create-node-after-restoring-couchdb/3798/7 "2013-03-26T20:38:40Z")

</div>

Could it be something in the Solr index? A “knife search node” for the name doesn’t have a result for the client, though.

---

<div class="post-metadata">

**Author:** ![Ranjib](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/ranjib/32/1040_2.png) [@Ranjib](https://discourse.chef.io/u/Ranjib)\
**Post date:** [March 26, 2013, 8:41pm UTC](https://discourse.chef.io/t/cant-re-create-node-after-restoring-couchdb/3798/8 "2013-03-26T20:41:19Z")

</div>

no.. knife node/client set of commands interacts with couch only. Try  
debugging it from irb/pry.  
irb\>  
require 'chef'  
Chef::Config.from\_file('\<knife.rb\>')  
n = Chef::Node.load('name')  
etc.  
also tail the chef server api log

On Tue, Mar 26, 2013 at 1:38 PM, Jochen Lillich [jochen@freistil.it](mailto:jochen@freistil.it) wrote:

> Could it be something in the Solr index? A "knife search node" for the  
> name doesn't have a result for the client, though.

---

<div class="post-metadata">

**Author:** ![AJ\_Christensen](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/aj_christensen/32/525_2.png) [@AJ\_Christensen](https://discourse.chef.io/u/AJ_Christensen)\
**Post date:** [March 26, 2013, 8:52pm UTC](https://discourse.chef.io/t/cant-re-create-node-after-restoring-couchdb/3798/9 "2013-03-26T20:52:51Z")

</div>

Right, clients' public keys not stored on disk, AFAIK, (?) -- stored in Couch.

As I mentioned, you probably did not take a consistent backup, or  
there was some problem with restoring a document. Have you inspected  
all documents using the \_clients design document? I would anticipate  
finding a dodgy document, removing/editing it, and being able to  
proceed.

--AJ

On 27 March 2013 09:05, Ranjib Dey [dey.ranjib@gmail.com](mailto:dey.ranjib@gmail.com) wrote:

> fujin, the public keys of the clients are not stored? whats does knife  
> client list enumerates then?
> 
> On Tue, Mar 26, 2013 at 1:03 PM, AJ Christensen [aj@junglist.gen.nz](mailto:aj@junglist.gen.nz) wrote:
> 
> > You might have not taken a consistent backup?
> > 
> > Clients aren't stored on the file-system -- only cookbook manifests  
> > (IIRC), and only in chef10, it's different (Bookshelf) in chef11.
> > 
> > Jump into your database and delete the matching client documents.
> > 
> > --AJ
> > 
> > On 27 March 2013 09:01, Jochen Lillich [jochen@freistil.it](mailto:jochen@freistil.it) wrote:
> > 
> > > Ranjib Dey wrote:
> > > 
> > > > delete the client from chef server using knife (knife client delete  
> > > > ), and delete the old client.pem from the node itself. then  
> > > > run chef-client on the node, and it will reregister itself.
> > > 
> > > I already tried that, but there is no client with that name:
> > > 
> > > ```
> > > $ knife client delete MYNODE
> > > Do you really want to delete MYNODE? (Y/N) y
> > > ERROR: JSON::ParserError: 743: unexpected token at 'null'
> > > 
> > > ```
> > > 
> > > Neither `knife client list` nor the CouchDB web UI display the client  
> > > name.
> > > 
> > > That's why I asked if there could be something outside CouchDB (maybe  
> > > the server filesystem) that prevents re-creating the client. Some kind of  
> > > remains from my node creation today.
> > > 
> > > Thanks,  
> > > Jochen

---

<div class="post-metadata">

**Author:** ![Jochen\_Lillich1](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/jochen_lillich1/32/572_2.png) [@Jochen\_Lillich1](https://discourse.chef.io/u/Jochen_Lillich1)\
**Post date:** [March 26, 2013, 9:03pm UTC](https://discourse.chef.io/t/cant-re-create-node-after-restoring-couchdb/3798/10 "2013-03-26T21:03:08Z")

</div>

AJ Christensen wrote:

> As I mentioned, you probably did not take a consistent backup

I seem to have solved it just this minute. You're right -- it was pilot error.

I had not done a consistent restore. According to [chef - [chef] Re: undefined method `couchdb\_rev=' for nil:NilClass](http://lists.opscode.com/sympa/arc/chef/2012-05/msg00258.html), there actually are files that somehow reference the database.

I did another restore, this time not only /var/lib/couchdb but also /var/lib/chef and /var/lib/rabbitmq. Now, I've been able to reregister the node.

Thanks for your help!  
Jochen