Chef-server(-api) not starting on starting after fresh install

Chef Infra (archive)
1

Hi,

I’ve been trying to learn chef for two days now, and can’t get it to run.

I’m installing chef on Debain (Wheezy/sid) on bare iron, and from the
repositories opscodes provides. (I’m a package manager junkie, shoot me)

I’m using the world’s most generic x86_64 hardware.

chef-server will.not.start.

regardless from which repo i try to install from (i.e. wheezy-0.10,
squeeze-0.10, or sid-0.10) nor which version i try. (i.e. -0.10.8-1 or
-0.10.4-1)

everything fails identically.

Someone brought up that my hostname might be an issue, it’s
mamabear.ross154 (no trailing .com or .net or whatever, just the
.domain at the end. having a trailing “.” makes no difference), but
I’m not convinced this isn’t a red herring. (double negative! my bad)

The failure is the same when run from the /etc/init.d script or run
from the command line, or during the installation of the package:

from dpkg…
Setting up chef-server-api (0.10.4-1) …

Creating config file /etc/chef/server.rb with new version
Starting chef-server : ~ In 3405
failed!

from the command line (with log_level set to :debug in server.rb):

mamabear:/etc# sudo -u chef chef-server -p 4000 -e production -a thin
Loading init file from /usr/share/chef-server-api/config/init.rb
[Wed, 21 Dec 2011 17:11:01 -0800] DEBUG: Sending HTTP Request via GET
to localhost:5984/_all_dbs
[Wed, 21 Dec 2011 17:11:01 -0800] DEBUG: Sending HTTP Request via GET
to localhost:5984/chef/_design/id_map
[Wed, 21 Dec 2011 17:11:01 -0800] DEBUG: Sending HTTP Request via GET
to localhost:5984/chef/_design/nodes
[Wed, 21 Dec 2011 17:11:01 -0800] DEBUG: Sending HTTP Request via GET
to localhost:5984/chef/_design/roles
[Wed, 21 Dec 2011 17:11:01 -0800] DEBUG: Sending HTTP Request via GET
to localhost:5984/chef/_design/data_bags
[Wed, 21 Dec 2011 17:11:01 -0800] DEBUG: Sending HTTP Request via GET
to localhost:5984/chef/_design/clients
[Wed, 21 Dec 2011 17:11:01 -0800] DEBUG: Sending HTTP Request via GET
to localhost:5984/chef/_design/users
[Wed, 21 Dec 2011 17:11:01 -0800] DEBUG: Sending HTTP Request via GET
to localhost:5984/chef/_design/cookbooks
[Wed, 21 Dec 2011 17:11:01 -0800] DEBUG: Sending HTTP Request via GET
to localhost:5984/chef/_design/sandboxes
[Wed, 21 Dec 2011 17:11:01 -0800] DEBUG: Sending HTTP Request via GET
to localhost:5984/chef/_design/checksums
[Wed, 21 Dec 2011 17:11:01 -0800] DEBUG: Sending HTTP Request via GET
to localhost:5984/chef/_design/environments
[Wed, 21 Dec 2011 17:11:01 -0800] DEBUG: Sending HTTP Request via GET
to localhost:5984/chef/_design/id_map/_view/name_to_id
[Wed, 21 Dec 2011 17:11:01 -0800] INFO: HTTP Request Returned 500
Internal Server Error: compilation_error
[Wed, 21 Dec 2011 17:11:01 -0800] ERROR: Server returned error for
http://localhost:5984/chef/_design/id_map/_view/name_to_id?include_docs=true&key=[“client”,“chef-validator”], retrying 1/5 in
3s
[Wed, 21 Dec 2011 17:11:04 -0800] INFO: HTTP Request Returned 500
Internal Server Error: compilation_error
[Wed, 21 Dec 2011 17:11:04 -0800] ERROR: Server returned error for
http://localhost:5984/chef/_design/id_map/_view/name_to_id?include_docs=true&key=[“client”,“chef-validator”], retrying 2/5 in
8s
[Wed, 21 Dec 2011 17:11:12 -0800] INFO: HTTP Request Returned 500
Internal Server Error: compilation_error
[Wed, 21 Dec 2011 17:11:12 -0800] ERROR: Server returned error for
http://localhost:5984/chef/_design/id_map/_view/name_to_id?include_docs=true&key=[“client”,“chef-validator”], retrying 3/5 in
11s
[Wed, 21 Dec 2011 17:11:23 -0800] INFO: HTTP Request Returned 500
Internal Server Error: compilation_error
[Wed, 21 Dec 2011 17:11:23 -0800] ERROR: Server returned error for
http://localhost:5984/chef/_design/id_map/_view/name_to_id?include_docs=true&key=[“client”,“chef-validator”], retrying 4/5 in
23s
[Wed, 21 Dec 2011 17:11:46 -0800] INFO: HTTP Request Returned 500
Internal Server Error: compilation_error
[Wed, 21 Dec 2011 17:11:46 -0800] ERROR: Server returned error for
http://localhost:5984/chef/_design/id_map/_view/name_to_id?include_docs=true&key=[“client”,“chef-validator”], retrying 5/5 in
47s
^[[Wed, 21 Dec 2011 17:12:33 -0800] INFO: HTTP Request Returned 500
Internal Server Error: compilation_error
/usr/lib/ruby/1.8/net/http.rb:2105:in error!': 500 "Internal Server Error" (Net::HTTPFatalError) from /usr/lib/ruby/vendor_ruby/chef/rest.rb:237:inapi_request’
from /usr/lib/ruby/vendor_ruby/chef/rest.rb:288:in
retriable_rest_request' from /usr/lib/ruby/vendor_ruby/chef/rest.rb:218:inapi_request’
from /usr/lib/ruby/vendor_ruby/chef/rest.rb:114:in get_rest' from /usr/lib/ruby/vendor_ruby/chef/couchdb.rb:225:inget_view’
from /usr/lib/ruby/vendor_ruby/chef/couchdb.rb:210:in find_by_name' from /usr/lib/ruby/vendor_ruby/chef/couchdb.rb:135:inload’
from /usr/lib/ruby/vendor_ruby/chef/api_client.rb:201:in cdb_load' from /usr/lib/ruby/vendor_ruby/chef/certificate.rb:140:ingen_validation_key’
from /usr/share/chef-server-api/config/init.rb:84
from /usr/lib/ruby/1.8/merb-core/bootloader.rb:521:in load' from /usr/lib/ruby/1.8/merb-core/bootloader.rb:521:inload_initfile’
from /usr/lib/ruby/1.8/merb-core/bootloader.rb:388:in run' from /usr/lib/ruby/1.8/merb-core/bootloader.rb:99:inrun’
from /usr/lib/ruby/1.8/merb-core/server.rb:172:in bootup' from /usr/lib/ruby/1.8/merb-core/server.rb:42:instart’
from /usr/lib/ruby/1.8/merb-core.rb:170:in `start’
from /usr/sbin/chef-server:86

it was suggested (in http://tickets.opscode.com/browse/CHEF-2827 that
i consult the mailing list and) that I try a curl command on the URL
that fails. the output is just as opaque to me…

it says…

curl

http://localhost:5984/chef/_design/id_map/_view/name_to_id?include_docs=true&key=[“client”,“chef-validator”]
{“error”:“compilation_error”,“reason”:“Expression does not eval to a
function. ((new String(” function(doc) {\n
emit([ doc.chef_type, doc.name], doc._id);\n
}\n")))"}

So far, it appears two different people have been unable to reproduce
this problem (one was in an aws session, i believe, i don’t know about
the other), and I’d be as happy as a clam to deduce that this is an
environmental problem, but since I’m only just trying to learn chef at
all to begin with, I’m really short on further ideas. After googling
and wikiing and IRCing, I’m now mailing-listing

any help would be appreciated, tia.
-m

2

On Wednesday, December 21, 2011 at 5:08 PM, Infor Mation wrote:

Hi,

I've been trying to learn chef for two days now, and can't get it to run.

I'm installing chef on Debain (Wheezy/sid) on bare iron, and from the
repositories opscodes provides. (I'm a package manager junkie, shoot me)

I'm using the world's most generic x86_64 hardware.

chef-server will.not.start.

curl

http://localhost:5984/chef/_design/id_map/_view/name_to_id?include_docs=true\&key=["client","chef-validator"]
{"error":"compilation_error","reason":"Expression does not eval to a
function. ((new String(" function(doc) {\n
emit([ doc.chef_type, doc.name (http://doc.name)], doc._id);\n
}\n")))"}

CouchDB is for some reason choking on the javascript views that the chef-server creates. What version of CouchDB is this?

So far, it appears two different people have been unable to reproduce
this problem (one was in an aws session, i believe, i don't know about
the other), and I'd be as happy as a clam to deduce that this is an
environmental problem, but since I'm only just trying to learn chef at
all to begin with, I'm really short on further ideas. After googling
and wikiing and IRCing, I'm now mailing-listing

Off the top of my head, the only environmental bit that seems like it could bite is if you have crazy encoding/locale settings.

any help would be appreciated, tia.
-m

--
Dan DeLeo

3

Quoting Daniel DeLeo dan@kallistec.com:

On Wednesday, December 21, 2011 at 5:08 PM, Infor Mation wrote:

Hi,

I've been trying to learn chef for two days now, and can't get it to run.

I'm installing chef on Debain (Wheezy/sid) on bare iron, and from the
repositories opscodes provides. (I'm a package manager junkie, shoot me)

I'm using the world's most generic x86_64 hardware.

chef-server will.not.start.

curl

http://localhost:5984/chef/_design/id_map/_view/name_to_id?include_docs=true\&key=["client","chef-validator"]
{"error":"compilation_error","reason":"Expression does not eval to a
function. ((new String(" function(doc) {\n
emit([ doc.chef_type, doc.name (http://doc.name)], doc._id);\n
}\n")))"}

CouchDB is for some reason choking on the javascript views that the
chef-server creates. What version of CouchDB is this?

1.1.1

this got me to thinking.. so I downgraded to the only other version
according to apt-cache show, which was 0.11.0-2.3

i had different results, it seemed to crash in the same place upon
creation of the chef.couch database, but failed because of a
depreciated gem or call.. "Gem.activate", I believe.

this taught me that chef.couch was what i needed to blow away.

So I reupgraded couchdb to 1.1.1 and let chef-server run the
initialization of the database.

I got different results than I've ever gotten (because chef is smart
enough to initialize a db if one doesn't exist, that's cool) but, it
proceeded to get a compilation_error on
localhost:5984/chef/_design/id_map/_view/name_to_id again

...and again...

curl

http://localhost:5984/chef/_design/id_map/_view/name_to_id?include_docs=true\&key=["client","chef-validator"]
{"error":"compilation_error","reason":"Expression does not eval to a
function. ((new String(" function(doc) {\n
emit([ doc.chef_type, doc.name], doc._id);\n
}\n")))"}

Off the top of my head, the only environmental bit that seems like
it could bite is if you have crazy encoding/locale settings.

The output from locale seems pretty normal:

mamabear:~# locale
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

any help would be appreciated, tia.
-m

--
Dan DeLeo

Square one again. It was a good clue tho, thanks.
-m

4

On Thursday, December 22, 2011 at 1:13 AM, Infor Mation wrote:

Quoting Daniel DeLeo <dan@kallistec.com (mailto:dan@kallistec.com)>:

On Wednesday, December 21, 2011 at 5:08 PM, Infor Mation wrote:

Hi,

I've been trying to learn chef for two days now, and can't get it to run.

I'm installing chef on Debain (Wheezy/sid) on bare iron, and from the
repositories opscodes provides. (I'm a package manager junkie, shoot me)

I'm using the world's most generic x86_64 hardware.

chef-server will.not.start.

curl

http://localhost:5984/chef/_design/id_map/_view/name_to_id?include_docs=true\&amp;key=["client","chef-validator"]
{"error":"compilation_error","reason":"Expression does not eval to a
function. ((new String(" function(doc) {\n
emit([ doc.chef_type, doc.name (http://doc.name)], doc._id);\n
}\n")))"}

CouchDB is for some reason choking on the javascript views that the
chef-server creates. What version of CouchDB is this?

1.1.1

this got me to thinking.. so I downgraded to the only other version
according to apt-cache show, which was 0.11.0-2.3

i had different results, it seemed to crash in the same place upon
creation of the chef.couch database, but failed because of a
depreciated gem or call.. "Gem.activate", I believe.

I think you're on the right track, and there is probably a bug either on our end or CouchDB's end with 1.1.1.

As for the Gem.activate error, I believe it's a packaging bug. We've delayed upgrading Merb to keep everything compatible with the versions of libraries in the Ubuntu/Debian releases we were targeting, and now the world has moved on. Downgrading rubygems may be an effective workaround until we can figure it out.

The output from locale seems pretty normal:

Yeah, I was wondering if there were encoding shenanigans causing newlines to get mangled and make the js code invalid, but it's probably a CouchDB version thing.

Square one again. It was a good clue tho, thanks.
Actually, you were getting warmer.
-m

--
Dan DeLeo

5

On 12/22/2011 09:12 AM, Daniel DeLeo wrote:

Actually, you were getting warmer.

you were right!

Solved it by:

downgrading couchdb to 0.11.0-2.3 from 1.1.1-1

downgrading rubygems to 1.3.7-3 from 1.8.12-1

now it starts!

I've upgraded the ticket to reflect this, too. I can't change seem to
change the status of the ticket, so i guess my comments will have to
suffice.

THANK YOU very much!

-m

6

Quoting Info info@ross154.net:

Solved it by:

downgrading couchdb to 0.11.0-2.3 from 1.1.1-1

downgrading rubygems to 1.3.7-3 from 1.8.12-1

now it starts!

THANK YOU very much!

-m

ok, out of the fire, into the frying pan. now that chef-server starts,
chef-server-webui doesn't start.

It appears related to whatever's listening on port 4000 (the API, I'm
guessing), but the output from trying to run chef-server-webui is:

/usr/sbin/chef-server-webui -p 4040 -e production -a thin -L

/var/log/chef/server-webui.log -C /etc/chef/webui.rb -u chef -G chef
Loading init file from /usr/share/chef-server-webui/config/init.rb
[Thu, 22 Dec 2011 16:34:04 -0800] INFO: HTTP Request Returned 500
Internal Server Error: uninitialized constant
Mixlib::Authentication::MissingAuthenticationHeader
[Thu, 22 Dec 2011 16:34:04 -0800] ERROR: Server returned error for
http://localhost:4000/users/admin, retrying 1/5 in 3s
[Thu, 22 Dec 2011 16:34:07 -0800] INFO: HTTP Request Returned 500
Internal Server Error: uninitialized constant
Mixlib::Authentication::MissingAuthenticationHeader
[Thu, 22 Dec 2011 16:34:07 -0800] ERROR: Server returned error for
http://localhost:4000/users/admin, retrying 2/5 in 6s
[Thu, 22 Dec 2011 16:34:13 -0800] INFO: HTTP Request Returned 500
Internal Server Error: uninitialized constant
Mixlib::Authentication::MissingAuthenticationHeader
[Thu, 22 Dec 2011 16:34:13 -0800] ERROR: Server returned error for
http://localhost:4000/users/admin, retrying 3/5 in 10s
[Thu, 22 Dec 2011 16:34:23 -0800] INFO: HTTP Request Returned 500
Internal Server Error: uninitialized constant
Mixlib::Authentication::MissingAuthenticationHeader
[Thu, 22 Dec 2011 16:34:23 -0800] ERROR: Server returned error for
http://localhost:4000/users/admin, retrying 4/5 in 27s
[Thu, 22 Dec 2011 16:34:50 -0800] INFO: HTTP Request Returned 500
Internal Server Error: uninitialized constant
Mixlib::Authentication::MissingAuthenticationHeader
[Thu, 22 Dec 2011 16:34:50 -0800] ERROR: Server returned error for
http://localhost:4000/users/admin, retrying 5/5 in 59s
[Thu, 22 Dec 2011 16:35:49 -0800] INFO: HTTP Request Returned 500
Internal Server Error: uninitialized constant
Mixlib::Authentication::MissingAuthenticationHeader
/usr/lib/ruby/1.8/net/http.rb:2105:in error!': 500 "Internal Server Error" (Net::HTTPFatalError) from /usr/lib/ruby/vendor_ruby/chef/rest.rb:245:in api_request'
from /usr/lib/ruby/vendor_ruby/chef/rest.rb:296:in
retriable_rest_request' from /usr/lib/ruby/vendor_ruby/chef/rest.rb:226:in api_request'
from /usr/lib/ruby/vendor_ruby/chef/rest.rb:130:in put_rest' from /usr/lib/ruby/vendor_ruby/chef/webui_user.rb:181:in save'
from /usr/share/chef-server-webui/config/init.rb:59
from /usr/lib/ruby/1.8/merb-core/bootloader.rb:521:in load' from /usr/lib/ruby/1.8/merb-core/bootloader.rb:521:in load_initfile'
from /usr/lib/ruby/1.8/merb-core/bootloader.rb:388:in run' from /usr/lib/ruby/1.8/merb-core/bootloader.rb:99:in run'
from /usr/lib/ruby/1.8/merb-core/server.rb:172:in bootup' from /usr/lib/ruby/1.8/merb-core/server.rb:42:in start'
from /usr/lib/ruby/1.8/merb-core.rb:170:in `start'
from /usr/sbin/chef-server-webui:73

and for completenesses sake, here are all the packages i have in
relation to mixlib(?)

dpkg -l | grep mixlib

ii libmixlib-authentication-ruby 1.1.4-1
Ruby mixin client/server header signing authentication/verification
ii libmixlib-authentication-ruby1.8 1.1.2-1
Ruby mixin client/server header signing authentication/verification
ii libmixlib-cli-ruby 1.2.0-1 A
Ruby mixin for creating command line applications
ii libmixlib-cli-ruby1.8 1.2.0-1 A
Ruby 1.8 mixin for creating command line applications
ii libmixlib-config-ruby 1.1.2-2
Transitional package for ruby-mixlib-config
ii libmixlib-log-ruby 1.3.0-1 A
gem that provides a simple mixin for log functionality
ii libmixlib-log-ruby1.8 1.1.0-1 A
simple Ruby 1.8 mixin for creating a logger object
ii ruby-mixlib-config 1.1.2-2
Simple class based config mechanism for Ruby

oh, so, there's a happy ending of sorts,

upgrading libmixlib-authentication-ruby1.8 to 1.1.4-1 fixes it! Not
being explicitly called out as a required version (eg ">= 1.1.3" like
it's brother libmixlib-authentication-ruby) might be a small issue tho.

cheers,
-m