Chef Server Crashes on Cookbook Upload

Chef Infra (archive)
1

Hi Chefs,

I had recently provisioned a new chef-server using 10.14.4 and was in the
process of moving all of our cookbooks onto the new chef-server when I get this
error client side,

ERROR: Failed to upload : 500 Internal Server Error
{“error”:[“No such file or directory -
/tmp/sandbox20121003-5631-1ofw8am.lock”]}
ERROR: Failed to upload : 500 Internal Server Error
{“error”:[“No such file or directory -
/tmp/sandbox20121003-5631-1li831d.lock”]}
ERROR: Failed to upload : 500 Internal Server Error
{“error”:[“No such file or directory - /tmp/sandbox20121003-5631-le9ona.lock”]}
ERROR: Failed to upload : Server broke connection
ERROR: Failed to upload : Server broke connection
/opt/chef/embedded/lib/ruby/1.9.1/net/protocol.rb:199:in write': Broken pipe (Errno::EPIPE) from /opt/chef/embedded/lib/ruby/1.9.1/net/protocol.rb:199:inwrite0’
from /opt/chef/embedded/lib/ruby/1.9.1/net/protocol.rb:173:in block in write' from /opt/chef/embedded/lib/ruby/1.9.1/net/protocol.rb:190:inwriting’
from /opt/chef/embedded/lib/ruby/1.9.1/net/protocol.rb:172:in write' from /opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:1937:insend_request_with_body’
from /opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:1919:in exec' from /opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:1317:inblock in
transport_request’
from /opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:1316:in catch' from /opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:1316:intransport_request’
from /opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:1293:in request' from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/rest-client-1.6.7/lib/restclient/net_http_ext.rb:51:inrequest’
from
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/rest-client-1.6.7/lib/restclient/request.rb:176:in
block in transmit' from /opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:745:instart’
from
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/rest-client-1.6.7/lib/restclient/request.rb:172:in
transmit' from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/rest-client-1.6.7/lib/restclient/request.rb:64:inexecute’
from
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in
execute' from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/rest-client-1.6.7/lib/restclient/resource.rb:76:input’
from
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.14.4/lib/chef/cookbook_uploader.rb:142:in
block in uploader_function_for' from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.14.4/lib/chef/cookbook_uploader.rb:25:incall’
from
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.14.4/lib/chef/cookbook_uploader.rb:25:in
block (3 levels) in setup_worker_threads' from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.14.4/lib/chef/cookbook_uploader.rb:24:inloop’
from
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.14.4/lib/chef/cookbook_uploader.rb:24:in
`block (2 levels) in setup_worker_threads’

Chef-Server had died and needed to be restarted. Most of the cookbooks made it
but a few did not, re-running the upload process kills it again.

Looking at the server logs,

merb : chef-server (api) : worker (port 4000) ~ Started request handling:
2012-10-03 12:50:17 -0500
merb : chef-server (api) : worker (port 4000) ~ Routed to:
{“controller”=>“sandboxes”, “action”=>“upload_checksum”,
“sandbox_id”=>“e0f51964514641e2979656e67b1ff4d1”,
“checksum”=>“97070fbb7de37914603425fff17e9bde”}
merb : chef-server (api) : worker (port 4000) ~ Params:
{“controller”=>“sandboxes”, “action”=>“upload_checksum”,
“sandbox_id”=>“e0f51964514641e2979656e67b1ff4d1”,
“checksum”=>“97070fbb7de37914603425fff17e9bde”}
merb : chef-server (api) : worker (port 4000) ~ No such file or directory -
/tmp/sandbox20121003-5631-1ulbjbm.lock - (Errno::ENOENT)
/opt/chef/embedded/lib/ruby/1.9.1/tempfile.rb:346:in rmdir' /opt/chef/embedded/lib/ruby/1.9.1/tempfile.rb:346:inrmdir’
/opt/chef/embedded/lib/ruby/1.9.1/tempfile.rb:338:in ensure in locking' /opt/chef/embedded/lib/ruby/1.9.1/tempfile.rb:338:inlocking’
/opt/chef/embedded/lib/ruby/1.9.1/tempfile.rb:144:in block in initialize' /opt/chef/embedded/lib/ruby/1.9.1/tmpdir.rb:133:increate’
/opt/chef/embedded/lib/ruby/1.9.1/tempfile.rb:134:in initialize' /opt/chef/embedded/lib/ruby/1.9.1/tempfile.rb:316:innew’
/opt/chef/embedded/lib/ruby/1.9.1/tempfile.rb:316:in open' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-server-api-10.14.4/app/models/sandbox_file.rb:101:incommit_stringio_to’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-server-api-10.14.4/app/models/sandbox_file.rb:53:in
commit_to' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-server-api-10.14.4/app/controllers/sandboxes.rb:96:inupload_checksum’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/controller/abstract_controller.rb:321:in
_call_action' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/controller/abstract_controller.rb:289:inblock in _dispatch’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/controller/abstract_controller.rb:284:in
catch' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/controller/abstract_controller.rb:284:in_dispatch’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/controller/merb_controller.rb:285:in
_dispatch' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/controller/merb_controller.rb:262:in_call’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/controller/merb_controller.rb:252:in
call' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/dispatch/dispatcher.rb:91:indispatch_action’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/dispatch/dispatcher.rb:69:in
handle' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/dispatch/dispatcher.rb:29:inhandle’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/rack/application.rb:17:in
call' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/rack/middleware/static.rb:28:incall’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/rack-1.4.1/lib/rack/content_length.rb:14:in
call' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/connection.rb:81:inblock in pre_process’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/connection.rb:79:in
catch' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/connection.rb:79:inpre_process’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/connection.rb:54:in
process' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/connection.rb:39:inreceive_data’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/eventmachine-1.0.0/lib/eventmachine.rb:187:in
run_machine' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/eventmachine-1.0.0/lib/eventmachine.rb:187:inrun’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/backends/base.rb:63:in
start' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/server.rb:159:instart’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/rack/adapter/thin.rb:30:in
start_server' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/rack/adapter/abstract.rb:305:instart_at_port’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/rack/adapter/abstract.rb:138:in
start' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/server.rb:174:inbootup’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/server.rb:159:in
block in daemonize' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/server.rb:143:infork’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/server.rb:143:in
daemonize' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/server.rb:35:instart’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core.rb:165:in
start' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-server-api-10.14.4/bin/chef-server:84:in<top (required)>’
/usr/sbin/chef-server:23:in load' /usr/sbin/chef-server:23:in’

2

Looks like the Tempfile code that chef-server uses to temporarily store uploaded files is failing to create tempfiles, and this is causing a secondary error when it attempts to clean up after itself.

Is /tmp full on your chef-server box? Are you able to create files and directories there by hand?

--
Daniel DeLeo

On Wednesday, October 3, 2012 at 10:38 AM, bjbq4d@gmail.com wrote:

Hi Chefs,

I had recently provisioned a new chef-server using 10.14.4 and was in the
process of moving all of our cookbooks onto the new chef-server when I get this
error client side,

ERROR: Failed to upload : 500 Internal Server Error
{"error":["No such file or directory -
/tmp/sandbox20121003-5631-1ofw8am.lock"]}
ERROR: Failed to upload : 500 Internal Server Error
{"error":["No such file or directory -
/tmp/sandbox20121003-5631-1li831d.lock"]}
ERROR: Failed to upload : 500 Internal Server Error
{"error":["No such file or directory - /tmp/sandbox20121003-5631-le9ona.lock"]}
ERROR: Failed to upload : Server broke connection
ERROR: Failed to upload : Server broke connection
/opt/chef/embedded/lib/ruby/1.9.1/net/protocol.rb:199:in write': Broken pipe (Errno::EPIPE) from /opt/chef/embedded/lib/ruby/1.9.1/net/protocol.rb:199:in write0'
from /opt/chef/embedded/lib/ruby/1.9.1/net/protocol.rb:173:in block in write' from /opt/chef/embedded/lib/ruby/1.9.1/net/protocol.rb:190:in writing'
from /opt/chef/embedded/lib/ruby/1.9.1/net/protocol.rb:172:in write' from /opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:1937:in send_request_with_body'
from /opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:1919:in exec' from /opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:1317:in block in
transport_request'
from /opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:1316:in catch' from /opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:1316:in transport_request'
from /opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:1293:in request' from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/rest-client-1.6.7/lib/restclient/net_http_ext.rb:51:in request'
from
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/rest-client-1.6.7/lib/restclient/request.rb:176:in
block in transmit' from /opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:745:in start'
from
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/rest-client-1.6.7/lib/restclient/request.rb:172:in
transmit' from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/rest-client-1.6.7/lib/restclient/request.rb:64:in execute'
from
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in
execute' from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/rest-client-1.6.7/lib/restclient/resource.rb:76:in put'
from
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.14.4/lib/chef/cookbook_uploader.rb:142:in
block in uploader_function_for' from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.14.4/lib/chef/cookbook_uploader.rb:25:in call'
from
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.14.4/lib/chef/cookbook_uploader.rb:25:in
block (3 levels) in setup_worker_threads' from /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.14.4/lib/chef/cookbook_uploader.rb:24:in loop'
from
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.14.4/lib/chef/cookbook_uploader.rb:24:in
`block (2 levels) in setup_worker_threads'

Chef-Server had died and needed to be restarted. Most of the cookbooks made it
but a few did not, re-running the upload process kills it again.

Looking at the server logs,

merb : chef-server (api) : worker (port 4000) ~ Started request handling:
2012-10-03 12:50:17 -0500
merb : chef-server (api) : worker (port 4000) ~ Routed to:
{"controller"=>"sandboxes", "action"=>"upload_checksum",
"sandbox_id"=>"e0f51964514641e2979656e67b1ff4d1",
"checksum"=>"97070fbb7de37914603425fff17e9bde"}
merb : chef-server (api) : worker (port 4000) ~ Params:
{"controller"=>"sandboxes", "action"=>"upload_checksum",
"sandbox_id"=>"e0f51964514641e2979656e67b1ff4d1",
"checksum"=>"97070fbb7de37914603425fff17e9bde"}
merb : chef-server (api) : worker (port 4000) ~ No such file or directory -
/tmp/sandbox20121003-5631-1ulbjbm.lock - (Errno::ENOENT)
/opt/chef/embedded/lib/ruby/1.9.1/tempfile.rb:346:in rmdir' /opt/chef/embedded/lib/ruby/1.9.1/tempfile.rb:346:in rmdir'
/opt/chef/embedded/lib/ruby/1.9.1/tempfile.rb:338:in ensure in locking' /opt/chef/embedded/lib/ruby/1.9.1/tempfile.rb:338:in locking'
/opt/chef/embedded/lib/ruby/1.9.1/tempfile.rb:144:in block in initialize' /opt/chef/embedded/lib/ruby/1.9.1/tmpdir.rb:133:in create'
/opt/chef/embedded/lib/ruby/1.9.1/tempfile.rb:134:in initialize' /opt/chef/embedded/lib/ruby/1.9.1/tempfile.rb:316:in new'
/opt/chef/embedded/lib/ruby/1.9.1/tempfile.rb:316:in open' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-server-api-10.14.4/app/models/sandbox_file.rb:101:in commit_stringio_to'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-server-api-10.14.4/app/models/sandbox_file.rb:53:in
commit_to' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-server-api-10.14.4/app/controllers/sandboxes.rb:96:in upload_checksum'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/controller/abstract_controller.rb:321:in
_call_action' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/controller/abstract_controller.rb:289:in block in _dispatch'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/controller/abstract_controller.rb:284:in
catch' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/controller/abstract_controller.rb:284:in _dispatch'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/controller/merb_controller.rb:285:in
_dispatch' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/controller/merb_controller.rb:262:in _call'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/controller/merb_controller.rb:252:in
call' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/dispatch/dispatcher.rb:91:in dispatch_action'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/dispatch/dispatcher.rb:69:in
handle' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/dispatch/dispatcher.rb:29:in handle'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/rack/application.rb:17:in
call' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/rack/middleware/static.rb:28:in call'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/rack-1.4.1/lib/rack/content_length.rb:14:in
call' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/connection.rb:81:in block in pre_process'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/connection.rb:79:in
catch' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/connection.rb:79:in pre_process'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/connection.rb:54:in
process' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/connection.rb:39:in receive_data'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/eventmachine-1.0.0/lib/eventmachine.rb:187:in
run_machine' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/eventmachine-1.0.0/lib/eventmachine.rb:187:in run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/backends/base.rb:63:in
start' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/server.rb:159:in start'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/rack/adapter/thin.rb:30:in
start_server' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/rack/adapter/abstract.rb:305:in start_at_port'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/rack/adapter/abstract.rb:138:in
start' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/server.rb:174:in bootup'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/server.rb:159:in
block in daemonize' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/server.rb:143:in fork'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/server.rb:143:in
daemonize' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/server.rb:35:in start'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core.rb:165:in
start' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-server-api-10.14.4/bin/chef-server:84:in <top (required)>'
/usr/sbin/chef-server:23:in load' /usr/sbin/chef-server:23:in '

3

Ahh, that was it. My /tmp file system was nearly full. Removing some junk
fixed the issue. Thanks!

On Wed, Oct 3, 2012 at 1:19 PM, Daniel DeLeo dan@kallistec.com wrote:

Looks like the Tempfile code that chef-server uses to temporarily store
uploaded files is failing to create tempfiles, and this is causing a
secondary error when it attempts to clean up after itself.

Is /tmp full on your chef-server box? Are you able to create files and
directories there by hand?

--
Daniel DeLeo

On Wednesday, October 3, 2012 at 10:38 AM, bjbq4d@gmail.com wrote:

Hi Chefs,

I had recently provisioned a new chef-server using 10.14.4 and was in the
process of moving all of our cookbooks onto the new chef-server when I get
this
error client side,

ERROR: Failed to upload : 500 Internal Server Error
{"error":["No such file or directory -
/tmp/sandbox20121003-5631-1ofw8am.lock"]}
ERROR: Failed to upload : 500 Internal Server Error
{"error":["No such file or directory -
/tmp/sandbox20121003-5631-1li831d.lock"]}
ERROR: Failed to upload : 500 Internal Server Error
{"error":["No such file or directory -
/tmp/sandbox20121003-5631-le9ona.lock"]}
ERROR: Failed to upload : Server broke connection
ERROR: Failed to upload : Server broke connection
/opt/chef/embedded/lib/ruby/1.9.1/net/protocol.rb:199:in write': Broken pipe (Errno::EPIPE) from /opt/chef/embedded/lib/ruby/1.9.1/net/protocol.rb:199:in write0'
from /opt/chef/embedded/lib/ruby/1.9.1/net/protocol.rb:173:in block in write' from /opt/chef/embedded/lib/ruby/1.9.1/net/protocol.rb:190:in writing'
from /opt/chef/embedded/lib/ruby/1.9.1/net/protocol.rb:172:in write' from /opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:1937:in send_request_with_body'
from /opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:1919:in exec' from /opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:1317:in block in
transport_request'
from /opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:1316:in catch' from /opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:1316:in transport_request'
from /opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:1293:in `request'
from

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/rest-client-1.6.7/lib/restclient/net_http_ext.rb:51:in
`request'
from

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/rest-client-1.6.7/lib/restclient/request.rb:176:in
block in transmit' from /opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:745:in start'
from

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/rest-client-1.6.7/lib/restclient/request.rb:172:in
`transmit'
from

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/rest-client-1.6.7/lib/restclient/request.rb:64:in
`execute'
from

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in
`execute'
from

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/rest-client-1.6.7/lib/restclient/resource.rb:76:in
`put'
from

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.14.4/lib/chef/cookbook_uploader.rb:142:in
`block in uploader_function_for'
from

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.14.4/lib/chef/cookbook_uploader.rb:25:in
`call'
from

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.14.4/lib/chef/cookbook_uploader.rb:25:in
`block (3 levels) in setup_worker_threads'
from

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.14.4/lib/chef/cookbook_uploader.rb:24:in
`loop'
from

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.14.4/lib/chef/cookbook_uploader.rb:24:in
`block (2 levels) in setup_worker_threads'

Chef-Server had died and needed to be restarted. Most of the cookbooks
made it
but a few did not, re-running the upload process kills it again.

Looking at the server logs,

merb : chef-server (api) : worker (port 4000) ~ Started request handling:
2012-10-03 12:50:17 -0500
merb : chef-server (api) : worker (port 4000) ~ Routed to:
{"controller"=>"sandboxes", "action"=>"upload_checksum",
"sandbox_id"=>"e0f51964514641e2979656e67b1ff4d1",
"checksum"=>"97070fbb7de37914603425fff17e9bde"}
merb : chef-server (api) : worker (port 4000) ~ Params:
{"controller"=>"sandboxes", "action"=>"upload_checksum",
"sandbox_id"=>"e0f51964514641e2979656e67b1ff4d1",
"checksum"=>"97070fbb7de37914603425fff17e9bde"}
merb : chef-server (api) : worker (port 4000) ~ No such file or directory -
/tmp/sandbox20121003-5631-1ulbjbm.lock - (Errno::ENOENT)
/opt/chef/embedded/lib/ruby/1.9.1/tempfile.rb:346:in rmdir' /opt/chef/embedded/lib/ruby/1.9.1/tempfile.rb:346:in rmdir'
/opt/chef/embedded/lib/ruby/1.9.1/tempfile.rb:338:in ensure in locking' /opt/chef/embedded/lib/ruby/1.9.1/tempfile.rb:338:in locking'
/opt/chef/embedded/lib/ruby/1.9.1/tempfile.rb:144:in block in initialize' /opt/chef/embedded/lib/ruby/1.9.1/tmpdir.rb:133:in create'
/opt/chef/embedded/lib/ruby/1.9.1/tempfile.rb:134:in initialize' /opt/chef/embedded/lib/ruby/1.9.1/tempfile.rb:316:in new'
/opt/chef/embedded/lib/ruby/1.9.1/tempfile.rb:316:in `open'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-server-api-10.14.4/app/models/sandbox_file.rb:101:in
`commit_stringio_to'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-server-api-10.14.4/app/models/sandbox_file.rb:53:in
`commit_to'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-server-api-10.14.4/app/controllers/sandboxes.rb:96:in
`upload_checksum'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/controller/abstract_controller.rb:321:in
`_call_action'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/controller/abstract_controller.rb:289:in
`block in _dispatch'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/controller/abstract_controller.rb:284:in
`catch'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/controller/abstract_controller.rb:284:in
`_dispatch'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/controller/merb_controller.rb:285:in
`_dispatch'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/controller/merb_controller.rb:262:in
`_call'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/controller/merb_controller.rb:252:in
`call'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/dispatch/dispatcher.rb:91:in
`dispatch_action'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/dispatch/dispatcher.rb:69:in
`handle'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/dispatch/dispatcher.rb:29:in
`handle'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/rack/application.rb:17:in
`call'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/rack/middleware/static.rb:28:in
`call'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/rack-1.4.1/lib/rack/content_length.rb:14:in
`call'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/connection.rb:81:in
`block in pre_process'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/connection.rb:79:in
`catch'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/connection.rb:79:in
`pre_process'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/connection.rb:54:in
`process'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/connection.rb:39:in
`receive_data'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/eventmachine-1.0.0/lib/eventmachine.rb:187:in
`run_machine'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/eventmachine-1.0.0/lib/eventmachine.rb:187:in
`run'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/backends/base.rb:63:in
`start'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/server.rb:159:in
`start'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/rack/adapter/thin.rb:30:in
`start_server'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/rack/adapter/abstract.rb:305:in
`start_at_port'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/rack/adapter/abstract.rb:138:in
`start'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/server.rb:174:in
`bootup'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/server.rb:159:in
`block in daemonize'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/server.rb:143:in
`fork'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/server.rb:143:in
`daemonize'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core/server.rb:35:in
`start'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/merb-core-1.1.3/lib/merb-core.rb:165:in
`start'

/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-server-api-10.14.4/bin/chef-server:84:in
<top (required)>' /usr/sbin/chef-server:23:in load'
/usr/sbin/chef-server:23:in `'

--
-Bryan