RE: Re: odd errors after restoring couchdb onto new chef server

Chef Infra (archive)
1

Kallen,

Do you have the steps you used to backup and then restore the couchdb to
a newly built server? If you could post them, I would be extremely
grateful..

Thanks

-----Original Message-----
From: kallen@groknaut.net [mailto:kallen@groknaut.net]
Sent: Tuesday, February 28, 2012 2:09 PM
To: chef@lists.opscode.com
Subject: [chef] Re: Re: Re: Re: Re: odd errors after restoring couchdb
onto new chef server

just to close out this thread:

to solve this on the newly built server, i purged all cookbooks, then
reuploaded all from my chef-repo in git. i didn't attempt to backup and
restore the checksum_path dir.

the new restored server seems to be working ok.

thanks for the help!

kallen

On Fri, 24 Feb 2012, Daniel DeLeo wrote:

On Friday, February 24, 2012 at 6:18 PM, kallen@groknaut.net wrote:

On Fri, 24 Feb 2012, kallen@groknaut.net
(mailto:kallen@groknaut.net) wrote:

On Sat, 25 Feb 2012, AJ Christensen wrote:

  1. wierd errors running a chef-client against new server,
    like:

did you copy the upload/copy the cookbooks to the new server?
are they in the file cache? Kind of looks like couchdb has
information about the files, but the files aren't on the
chef-server file system (hazard
guess)

i hadn't done an upload/copy of the cookbooks to the new server
because i thought the cookbooks were loaded in the couchdb-load.
when i run knife cookbook list, it looks like all my cookbooks are
already there.

regardless, i just reuploaded all cookbooks from my chef-repo to
the new server. i saw no upload errors. now, running chef-client
on a non-chef-server node, pointing chef_server_url directly at
port 4000 rather than hitting nginx proxy over https, is erroring
out on "EOFError - end of file reached", which is similar to error
shown in gist.

[Fri, 24 Feb 2012 23:13:08 +0000] DEBUG: Sending HTTP Request via
GET to
chef.venus.spergacula.com:4000/cookbooks/mrepo/0.0.1/files/c0cffe5
0c70191353d8bd9a8bd568ce5
(http://chef.venus.spergacula.com:4000/cookbooks/mrepo/0.0.1/files
/c0cffe50c70191353d8bd9a8bd568ce5)
[Fri, 24 Feb 2012 23:13:08 +0000] ERROR: Running exception
handlers [Fri, 24 Feb 2012 23:13:08 +0000] FATAL: Saving node
information to /var/cache/chef/failed-run-data.json
[Fri, 24 Feb 2012 23:13:08 +0000] ERROR: Exception handlers
complete [Fri, 24 Feb 2012 23:13:08 +0000] DEBUG: Re-raising
exception: EOFError - end of file reached
/usr/lib/ruby/1.8/net/protocol.rb:135:in sysread' /usr/lib/ruby/1.8/net/protocol.rb:135:in rbuf_fill'
/usr/lib/ruby/1.8/timeout.rb:67:in timeout' /usr/lib/ruby/1.8/timeout.rb:101:in timeout'

i'm looking at chef-server debug log and couchdb log. but nothing
is jumping out at me as relevant. but i may not recognize
relevancy..

last 3 lines of couchdb log fwiw:

[Fri, 24 Feb 2012 23:18:13 GMT] [info] [<0.17029.0>] 127.0.0.1 - -

'POST' /chef/_all_docs?include_docs=true 200 [Fri, 24 Feb 2012
23:18:14 GMT] [info] [<0.17030.0>] 127.0.0.1 - - 'GET'
/chef/_design/id_map/_view/name_to_id?key=[%22client%22,%22admin2.
venus.spergacula.com
(http://22admin2.venus.spergacula.com)%22]&include_docs=true 200
[Fri, 24 Feb 2012 23:18:14 GMT] [info] [<0.17031.0>] 127.0.0.1 - -

'GET'
/chef/_design/id_map/_view/name_to_id?key=[%22cookbook_version%22,
%22mrepo-0.0.1%22]&include_docs=true 200

thoughts?

trying various things, flailing. for the heck of it:

  • deleted possible problematic cookbook (mrepo) and reuploaded. same
    errors.
  • deleted client node which i'm testing on (admin2), reran, same
    errors.
  • reloaded couchdb dump again

and now a different cookbook download claims "EOFError - end of file
reached".

[Sat, 25 Feb 2012 02:14:22 +0000] DEBUG: Sending HTTP Request via
GET to
cheftain04.venus.spergacula.com:4000/cookbooks/begin/0.0.1/files/e44
7bc0fcbef6dafe72007fe70507767
(http://cheftain04.venus.spergacula.com:4000/cookbooks/begin/0.0.1/f
iles/e447bc0fcbef6dafe72007fe70507767)
[Sat, 25 Feb 2012 02:14:22 +0000] ERROR: Running exception handlers
[Sat, 25 Feb 2012 02:14:22 +0000] FATAL: Saving node information to
/var/cache/chef/failed-run-data.json
[Sat, 25 Feb 2012 02:14:22 +0000] ERROR: Exception handlers complete

[Sat, 25 Feb 2012 02:14:22 +0000] DEBUG: Re-raising exception:
EOFError - end of file reached
/usr/lib/ruby/1.8/net/protocol.rb:135:in sysread' /usr/lib/ruby/1.8/net/protocol.rb:135:in rbuf_fill'

dunno what to do here..
The files that comprise your cookbooks aren't stored in couch, they're
stored according to their MD5 under whatever you've configured for
checksum_path in your server.rb. When you backup/restore your chef
server, you need to restore these files also.

The error you're seeing occurs because the file transfer happens
outside of the normal request cycle, so server side errors aren't
converted to 500s[1].

To fix it, you can either bulk delete all cookbooks using the
--purge option, or copy the files.

HTH,

--
Dan DeLeo

  1. Merb is trying to hand off the work to another thread (in the case
    of threaded servers) or the event loop (in case of evented servers like
    thin) so that the transfer can happen while other requests are
    processed. I'm not sure if this even works any more. In any case, I
    believe a user has contributed a fix for this problem.
2

couchdb-load http://:5984/chef < filenameofdatabasedump

Assuming you've installed that tool from here:
http://code.google.com/p/couchdb-python/

On Wed, Mar 7, 2012 at 9:35 AM, Van Fossan,Randy vanfossr@oclc.org wrote:

Kallen,

Do you have the steps you used to backup and then restore the couchdb to
a newly built server? If you could post them, I would be extremely
grateful..

Thanks

-----Original Message-----
From: kallen@groknaut.net [mailto:kallen@groknaut.net]
Sent: Tuesday, February 28, 2012 2:09 PM
To: chef@lists.opscode.com
Subject: [chef] Re: Re: Re: Re: Re: odd errors after restoring couchdb
onto new chef server

just to close out this thread:

to solve this on the newly built server, i purged all cookbooks, then
reuploaded all from my chef-repo in git. i didn't attempt to backup and
restore the checksum_path dir.

the new restored server seems to be working ok.

thanks for the help!

kallen

On Fri, 24 Feb 2012, Daniel DeLeo wrote:

On Friday, February 24, 2012 at 6:18 PM, kallen@groknaut.net wrote:

On Fri, 24 Feb 2012, kallen@groknaut.net
(mailto:kallen@groknaut.net) wrote:

On Sat, 25 Feb 2012, AJ Christensen wrote:

  1. wierd errors running a chef-client against new server,
    like:

did you copy the upload/copy the cookbooks to the new server?
are they in the file cache? Kind of looks like couchdb has
information about the files, but the files aren't on the
chef-server file system (hazard
guess)

i hadn't done an upload/copy of the cookbooks to the new server
because i thought the cookbooks were loaded in the couchdb-load.
when i run knife cookbook list, it looks like all my cookbooks are
already there.

regardless, i just reuploaded all cookbooks from my chef-repo to
the new server. i saw no upload errors. now, running chef-client
on a non-chef-server node, pointing chef_server_url directly at
port 4000 rather than hitting nginx proxy over https, is erroring
out on "EOFError - end of file reached", which is similar to error
shown in gist.

[Fri, 24 Feb 2012 23:13:08 +0000] DEBUG: Sending HTTP Request via
GET to
chef.venus.spergacula.com:4000/cookbooks/mrepo/0.0.1/files/c0cffe5
0c70191353d8bd9a8bd568ce5
(http://chef.venus.spergacula.com:4000/cookbooks/mrepo/0.0.1/files
/c0cffe50c70191353d8bd9a8bd568ce5)
[Fri, 24 Feb 2012 23:13:08 +0000] ERROR: Running exception
handlers [Fri, 24 Feb 2012 23:13:08 +0000] FATAL: Saving node
information to /var/cache/chef/failed-run-data.json
[Fri, 24 Feb 2012 23:13:08 +0000] ERROR: Exception handlers
complete [Fri, 24 Feb 2012 23:13:08 +0000] DEBUG: Re-raising
exception: EOFError - end of file reached
/usr/lib/ruby/1.8/net/protocol.rb:135:in sysread' /usr/lib/ruby/1.8/net/protocol.rb:135:in rbuf_fill'
/usr/lib/ruby/1.8/timeout.rb:67:in timeout' /usr/lib/ruby/1.8/timeout.rb:101:in timeout'

i'm looking at chef-server debug log and couchdb log. but nothing
is jumping out at me as relevant. but i may not recognize
relevancy..

last 3 lines of couchdb log fwiw:

[Fri, 24 Feb 2012 23:18:13 GMT] [info] [<0.17029.0>] 127.0.0.1 - -

'POST' /chef/_all_docs?include_docs=true 200 [Fri, 24 Feb 2012
23:18:14 GMT] [info] [<0.17030.0>] 127.0.0.1 - - 'GET'
/chef/_design/id_map/_view/name_to_id?key=[%22client%22,%22admin2.
venus.spergacula.com
(http://22admin2.venus.spergacula.com)%22]&include_docs=true 200
[Fri, 24 Feb 2012 23:18:14 GMT] [info] [<0.17031.0>] 127.0.0.1 - -

'GET'
/chef/_design/id_map/_view/name_to_id?key=[%22cookbook_version%22,
%22mrepo-0.0.1%22]&include_docs=true 200

thoughts?

trying various things, flailing. for the heck of it:

  • deleted possible problematic cookbook (mrepo) and reuploaded. same
    errors.
  • deleted client node which i'm testing on (admin2), reran, same
    errors.
  • reloaded couchdb dump again

and now a different cookbook download claims "EOFError - end of file
reached".

[Sat, 25 Feb 2012 02:14:22 +0000] DEBUG: Sending HTTP Request via
GET to
cheftain04.venus.spergacula.com:4000/cookbooks/begin/0.0.1/files/e44
7bc0fcbef6dafe72007fe70507767
(http://cheftain04.venus.spergacula.com:4000/cookbooks/begin/0.0.1/f
iles/e447bc0fcbef6dafe72007fe70507767)
[Sat, 25 Feb 2012 02:14:22 +0000] ERROR: Running exception handlers
[Sat, 25 Feb 2012 02:14:22 +0000] FATAL: Saving node information to
/var/cache/chef/failed-run-data.json
[Sat, 25 Feb 2012 02:14:22 +0000] ERROR: Exception handlers complete

[Sat, 25 Feb 2012 02:14:22 +0000] DEBUG: Re-raising exception:
EOFError - end of file reached
/usr/lib/ruby/1.8/net/protocol.rb:135:in sysread' /usr/lib/ruby/1.8/net/protocol.rb:135:in rbuf_fill'

dunno what to do here..
The files that comprise your cookbooks aren't stored in couch, they're
stored according to their MD5 under whatever you've configured for
checksum_path in your server.rb. When you backup/restore your chef
server, you need to restore these files also.

The error you're seeing occurs because the file transfer happens
outside of the normal request cycle, so server side errors aren't
converted to 500s[1].

To fix it, you can either bulk delete all cookbooks using the
--purge option, or copy the files.

HTH,

--
Dan DeLeo

  1. Merb is trying to hand off the work to another thread (in the case
    of threaded servers) or the event loop (in case of evented servers like
    thin) so that the transfer can happen while other requests are
    processed. I'm not sure if this even works any more. In any case, I
    believe a user has contributed a fix for this problem.

--
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.

3

more verbosely, here's what i did. and i think this verbosity will
help you not get tripped up on some things, like i did while working
thru this:

curl -XDELETE http://127.0.0.1:5984/chef
curl -XPUT http://127.0.0.1:5984/chef

the above assumes you have a fresh chef-server install and that it
started. when it starts, it creates a new keypair for the chef-validator
client. i want to retain my original keypair (chef-validator pubkey
present in couchdb and the validation.pem private key) so that i don't
have to rekey my entire infrastructure.

couchdb-load --input=chef_couchdb.latest http://127.0.0.1:5984/chef --ignore-errors 1>couchdb-load.stdout 2>couchdb-load.stderr

why --ignore-errors? i dunno :\ it didn't load without that. but it
seeeeeems to be ok.

after this loads, i fire up a working knife client and make sure my
chef-validator pubkey is what i want.

and then i delete all cookbooks, and reload them all from my chef-repo
in git.

and i have the best of intentions to add my experience with restoring
a chef server from backup to the opscode wiki .. :>

On Wed, 07 Mar 2012, Chris wrote:

couchdb-load http://:5984/chef < filenameofdatabasedump

Assuming you've installed that tool from here:
http://code.google.com/p/couchdb-python/

On Wed, Mar 7, 2012 at 9:35 AM, Van Fossan,Randy vanfossr@oclc.org wrote:

Kallen,

Do you have the steps you used to backup and then restore the couchdb to
a newly built server? If you could post them, I would be extremely
grateful..

Thanks

-----Original Message-----
From: kallen@groknaut.net [mailto:kallen@groknaut.net]
Sent: Tuesday, February 28, 2012 2:09 PM
To: chef@lists.opscode.com
Subject: [chef] Re: Re: Re: Re: Re: odd errors after restoring couchdb
onto new chef server

just to close out this thread:

to solve this on the newly built server, i purged all cookbooks, then
reuploaded all from my chef-repo in git. i didn't attempt to backup and
restore the checksum_path dir.

the new restored server seems to be working ok.

thanks for the help!

kallen

On Fri, 24 Feb 2012, Daniel DeLeo wrote:

On Friday, February 24, 2012 at 6:18 PM, kallen@groknaut.net wrote:

On Fri, 24 Feb 2012, kallen@groknaut.net
(mailto:kallen@groknaut.net) wrote:

On Sat, 25 Feb 2012, AJ Christensen wrote:

  1. wierd errors running a chef-client against new server,
    like:

did you copy the upload/copy the cookbooks to the new server?
are they in the file cache? Kind of looks like couchdb has
information about the files, but the files aren't on the
chef-server file system (hazard
guess)

i hadn't done an upload/copy of the cookbooks to the new server
because i thought the cookbooks were loaded in the couchdb-load.
when i run knife cookbook list, it looks like all my cookbooks are
already there.

regardless, i just reuploaded all cookbooks from my chef-repo to
the new server. i saw no upload errors. now, running chef-client
on a non-chef-server node, pointing chef_server_url directly at
port 4000 rather than hitting nginx proxy over https, is erroring
out on "EOFError - end of file reached", which is similar to error
shown in gist.

[Fri, 24 Feb 2012 23:13:08 +0000] DEBUG: Sending HTTP Request via
GET to
chef.venus.spergacula.com:4000/cookbooks/mrepo/0.0.1/files/c0cffe5
0c70191353d8bd9a8bd568ce5
(http://chef.venus.spergacula.com:4000/cookbooks/mrepo/0.0.1/files
/c0cffe50c70191353d8bd9a8bd568ce5)
[Fri, 24 Feb 2012 23:13:08 +0000] ERROR: Running exception
handlers [Fri, 24 Feb 2012 23:13:08 +0000] FATAL: Saving node
information to /var/cache/chef/failed-run-data.json
[Fri, 24 Feb 2012 23:13:08 +0000] ERROR: Exception handlers
complete [Fri, 24 Feb 2012 23:13:08 +0000] DEBUG: Re-raising
exception: EOFError - end of file reached
/usr/lib/ruby/1.8/net/protocol.rb:135:in sysread' /usr/lib/ruby/1.8/net/protocol.rb:135:in rbuf_fill'
/usr/lib/ruby/1.8/timeout.rb:67:in timeout' /usr/lib/ruby/1.8/timeout.rb:101:in timeout'

i'm looking at chef-server debug log and couchdb log. but nothing
is jumping out at me as relevant. but i may not recognize
relevancy..

last 3 lines of couchdb log fwiw:

[Fri, 24 Feb 2012 23:18:13 GMT] [info] [<0.17029.0>] 127.0.0.1 - -

'POST' /chef/_all_docs?include_docs=true 200 [Fri, 24 Feb 2012
23:18:14 GMT] [info] [<0.17030.0>] 127.0.0.1 - - 'GET'
/chef/_design/id_map/_view/name_to_id?key=[%22client%22,%22admin2.
venus.spergacula.com
(http://22admin2.venus.spergacula.com)%22]&include_docs=true 200
[Fri, 24 Feb 2012 23:18:14 GMT] [info] [<0.17031.0>] 127.0.0.1 - -

'GET'
/chef/_design/id_map/_view/name_to_id?key=[%22cookbook_version%22,
%22mrepo-0.0.1%22]&include_docs=true 200

thoughts?

trying various things, flailing. for the heck of it:

  • deleted possible problematic cookbook (mrepo) and reuploaded. same
    errors.
  • deleted client node which i'm testing on (admin2), reran, same
    errors.
  • reloaded couchdb dump again

and now a different cookbook download claims "EOFError - end of file
reached".

[Sat, 25 Feb 2012 02:14:22 +0000] DEBUG: Sending HTTP Request via
GET to
cheftain04.venus.spergacula.com:4000/cookbooks/begin/0.0.1/files/e44
7bc0fcbef6dafe72007fe70507767
(http://cheftain04.venus.spergacula.com:4000/cookbooks/begin/0.0.1/f
iles/e447bc0fcbef6dafe72007fe70507767)
[Sat, 25 Feb 2012 02:14:22 +0000] ERROR: Running exception handlers
[Sat, 25 Feb 2012 02:14:22 +0000] FATAL: Saving node information to
/var/cache/chef/failed-run-data.json
[Sat, 25 Feb 2012 02:14:22 +0000] ERROR: Exception handlers complete

[Sat, 25 Feb 2012 02:14:22 +0000] DEBUG: Re-raising exception:
EOFError - end of file reached
/usr/lib/ruby/1.8/net/protocol.rb:135:in sysread' /usr/lib/ruby/1.8/net/protocol.rb:135:in rbuf_fill'

dunno what to do here..
The files that comprise your cookbooks aren't stored in couch, they're
stored according to their MD5 under whatever you've configured for
checksum_path in your server.rb. When you backup/restore your chef
server, you need to restore these files also.

The error you're seeing occurs because the file transfer happens
outside of the normal request cycle, so server side errors aren't
converted to 500s[1].

To fix it, you can either bulk delete all cookbooks using the
--purge option, or copy the files.

HTH,

--
Dan DeLeo

  1. Merb is trying to hand off the work to another thread (in the case
    of threaded servers) or the event loop (in case of evented servers like
    thin) so that the transfer can happen while other requests are
    processed. I'm not sure if this even works any more. In any case, I
    believe a user has contributed a fix for this problem.

--
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.