500 error on server when client has large passwd file?

Hi All,

this is my first crack at chef so I’m prepared to be wildy wrong, but
I think it cannot handle the size of my default password file.

the setup:

I’m using 0.10.8 opensource server running on Debian 6.0
(stable/squeeze) from opscode apt repository

workstaion is also Debian/Squeeze 0.10.8 from OpsCode repo

I have two nodes setup both installed with knife bootstrap from my
workstaion:

node1 is a bare install of ubuntu-server 11.04

node2 is our current standard Debian/Squeeze workstation install which
includes all kinds of packages and all of our user accouts (spoiler:
this node fails to run but reducing number of accounts makes it work):

wc -l /etc/passwd
2551 /etc/passwd

both have empty runlists.

node1 bootstrapped nicely

node2 got all the packages and configs to make chef-client go
registered client and node on the server but on first run:

[Thu, 23 Feb 2012 16:52:05 -0500] INFO: *** Chef 0.10.8 ***
[Thu, 23 Feb 2012 16:52:09 -0500] INFO: Run List is []
[Thu, 23 Feb 2012 16:52:09 -0500] INFO: Run List expands to []
[Thu, 23 Feb 2012 16:52:09 -0500] INFO: Starting Chef Run for node2.csail.mit.edu
[Thu, 23 Feb 2012 16:52:09 -0500] INFO: Running start handlers
[Thu, 23 Feb 2012 16:52:09 -0500] INFO: Start handlers complete.
[Thu, 23 Feb 2012 16:52:10 -0500] INFO: Loading cookbooks []
[Thu, 23 Feb 2012 16:46:55 -0500] WARN: Node node2.csail.mit.edu has an empty run list.
[Thu, 23 Feb 2012 16:46:59 -0500] INFO: HTTP Request Returned 500 Internal Server Error: 400 “Bad Request”
[Thu, 23 Feb 2012 16:46:59 -0500] ERROR: Server returned error for
http://chef.csail.mit.edu:4000/nodes/lore.csail.mit.edu, retrying 1/5
in 3s

Looking on the server I notices an insanly long line logs with all the
ohai info (most notably user account info) and these errors after:

merb : chef-server (api) : worker (port 4000) ~ 400 “Bad Request” - (Net::HTTPServerException)
/usr/lib/ruby/1.8/net/http.rb:2105:in error!' /usr/lib/ruby/vendor_ruby/chef/rest.rb:245:inapi_request’
/usr/lib/ruby/vendor_ruby/chef/rest.rb:296:in retriable_rest_request' /usr/lib/ruby/vendor_ruby/chef/rest.rb:226:inapi_request’
/usr/lib/ruby/vendor_ruby/chef/rest.rb:130:in put_rest' /usr/lib/ruby/vendor_ruby/chef/couchdb.rb:114:instore’
/usr/lib/ruby/vendor_ruby/chef/node.rb:618:in cdb_save' /usr/share/chef-server-api/app/controllers/nodes.rb:69:inupdate’
/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in send' /usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in_call_action’
/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:289:in _dispatch' /usr/lib/ruby/1.8/merb-core/controller/merb_controller.rb:252:in_dispatch’
/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:102:in dispatch_action' /usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:74:inhandle’
/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:36:in handle' /usr/lib/ruby/1.8/merb-core/rack/application.rb:17:incall’
/usr/lib/ruby/1.8/merb-core/rack/middleware/static.rb:28:in call' /usr/lib/ruby/1.8/rack/content_length.rb:13:incall’
/usr/lib/ruby/1.8/thin/connection.rb:76:in pre_process' /usr/lib/ruby/1.8/thin/connection.rb:74:incatch’
/usr/lib/ruby/1.8/thin/connection.rb:74:in pre_process' /usr/lib/ruby/1.8/thin/connection.rb:57:inprocess’
/usr/lib/ruby/1.8/thin/connection.rb:42:in receive_data' /usr/lib/ruby/1.8/eventmachine.rb:257:inrun_machine’
/usr/lib/ruby/1.8/eventmachine.rb:257:in run' /usr/lib/ruby/1.8/thin/backends/base.rb:57:instart’
/usr/lib/ruby/1.8/thin/server.rb:156:in start' /usr/lib/ruby/1.8/merb-core/rack/adapter/thin.rb:30:instart_server’
/usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:298:in start_at_port' /usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:128:instart’
/usr/lib/ruby/1.8/merb-core/server.rb:174:in bootup' /usr/lib/ruby/1.8/merb-core/server.rb:159:indaemonize’
/usr/lib/ruby/1.8/merb-core/server.rb:143:in fork' /usr/lib/ruby/1.8/merb-core/server.rb:143:indaemonize’
/usr/lib/ruby/1.8/merb-core/server.rb:35:in start' /usr/lib/ruby/1.8/merb-core.rb:170:instart’
/usr/sbin/chef-server:86

Restricting /etc/passwd on node2 to just system accounts and myself:

wc -l /etc/passwd
26 /etc/passwd

chef-client run succeeds

Not sure if you got a response but I think you're right. My guess is there is a max request size variable either in merb or in the chef server code that you're hitting, as I have never heard of couchdb having problems with large objects (disclaimer: I'm not a couchdb expert).

As is often the case with Ruby, adding one line of code or config somewhere will probably fix your problem but the trick is figuring out what the line should be and where it should go...

(mobile edition)

On Feb 23, 2012, at 2:11 PM, Jonathan Proulx jon@jonproulx.com wrote:

Hi All,

this is my first crack at chef so I'm prepared to be wildy wrong, but
I think it cannot handle the size of my default password file.

the setup:

I'm using 0.10.8 opensource server running on Debian 6.0
(stable/squeeze) from opscode apt repository

workstaion is also Debian/Squeeze 0.10.8 from OpsCode repo

I have two nodes setup both installed with knife bootstrap from my
workstaion:

node1 is a bare install of ubuntu-server 11.04

node2 is our current standard Debian/Squeeze workstation install which
includes all kinds of packages and all of our user accouts (spoiler:
this node fails to run but reducing number of accounts makes it work):

wc -l /etc/passwd
2551 /etc/passwd

both have empty runlists.

node1 bootstrapped nicely

node2 got all the packages and configs to make chef-client go
registered client and node on the server but on first run:

[Thu, 23 Feb 2012 16:52:05 -0500] INFO: *** Chef 0.10.8 ***
[Thu, 23 Feb 2012 16:52:09 -0500] INFO: Run List is
[Thu, 23 Feb 2012 16:52:09 -0500] INFO: Run List expands to
[Thu, 23 Feb 2012 16:52:09 -0500] INFO: Starting Chef Run for node2.csail.mit.edu
[Thu, 23 Feb 2012 16:52:09 -0500] INFO: Running start handlers
[Thu, 23 Feb 2012 16:52:09 -0500] INFO: Start handlers complete.
[Thu, 23 Feb 2012 16:52:10 -0500] INFO: Loading cookbooks
[Thu, 23 Feb 2012 16:46:55 -0500] WARN: Node node2.csail.mit.edu has an empty run list.
[Thu, 23 Feb 2012 16:46:59 -0500] INFO: HTTP Request Returned 500 Internal Server Error: 400 "Bad Request"
[Thu, 23 Feb 2012 16:46:59 -0500] ERROR: Server returned error for
http://chef.csail.mit.edu:4000/nodes/lore.csail.mit.edu, retrying 1/5
in 3s

Looking on the server I notices an insanly long line logs with all the
ohai info (most notably user account info) and these errors after:

merb : chef-server (api) : worker (port 4000) ~ 400 "Bad Request" - (Net::HTTPServerException)
/usr/lib/ruby/1.8/net/http.rb:2105:in error!' /usr/lib/ruby/vendor_ruby/chef/rest.rb:245:in api_request'
/usr/lib/ruby/vendor_ruby/chef/rest.rb:296:in retriable_rest_request' /usr/lib/ruby/vendor_ruby/chef/rest.rb:226:in api_request'
/usr/lib/ruby/vendor_ruby/chef/rest.rb:130:in put_rest' /usr/lib/ruby/vendor_ruby/chef/couchdb.rb:114:in store'
/usr/lib/ruby/vendor_ruby/chef/node.rb:618:in cdb_save' /usr/share/chef-server-api/app/controllers/nodes.rb:69:in update'
/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in send' /usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in _call_action'
/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:289:in _dispatch' /usr/lib/ruby/1.8/merb-core/controller/merb_controller.rb:252:in _dispatch'
/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:102:in dispatch_action' /usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:74:in handle'
/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:36:in handle' /usr/lib/ruby/1.8/merb-core/rack/application.rb:17:in call'
/usr/lib/ruby/1.8/merb-core/rack/middleware/static.rb:28:in call' /usr/lib/ruby/1.8/rack/content_length.rb:13:in call'
/usr/lib/ruby/1.8/thin/connection.rb:76:in pre_process' /usr/lib/ruby/1.8/thin/connection.rb:74:in catch'
/usr/lib/ruby/1.8/thin/connection.rb:74:in pre_process' /usr/lib/ruby/1.8/thin/connection.rb:57:in process'
/usr/lib/ruby/1.8/thin/connection.rb:42:in receive_data' /usr/lib/ruby/1.8/eventmachine.rb:257:in run_machine'
/usr/lib/ruby/1.8/eventmachine.rb:257:in run' /usr/lib/ruby/1.8/thin/backends/base.rb:57:in start'
/usr/lib/ruby/1.8/thin/server.rb:156:in start' /usr/lib/ruby/1.8/merb-core/rack/adapter/thin.rb:30:in start_server'
/usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:298:in start_at_port' /usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:128:in start'
/usr/lib/ruby/1.8/merb-core/server.rb:174:in bootup' /usr/lib/ruby/1.8/merb-core/server.rb:159:in daemonize'
/usr/lib/ruby/1.8/merb-core/server.rb:143:in fork' /usr/lib/ruby/1.8/merb-core/server.rb:143:in daemonize'
/usr/lib/ruby/1.8/merb-core/server.rb:35:in start' /usr/lib/ruby/1.8/merb-core.rb:170:in start'
/usr/sbin/chef-server:86

Restricting /etc/passwd on node2 to just system accounts and myself:

wc -l /etc/passwd
26 /etc/passwd

chef-client run succeeds