Cookbook upload: HTTP 500: Too many links

Kallen · February 8, 2012, 6:06am

hiya. seeing this error on one of my chef servers. anyone run into this?
know what to do?

chef:~/ops/chef-repo$ knife cookbook upload -a -o ./cookbooks/ -V
Uploading activemq [1.0.0]
INFO: Saving activemq
INFO: Validating ruby files
INFO: Validating templates
INFO: Syntax OK
INFO: HTTP Request Returned 500 Internal Server Error: Too many links - /var/cache/chef/sandboxes/26f2b236b39b403ca272034f5fa95599
ERROR: Server returned error for https://chef.dr.spergtastic.com/sandboxes, retrying 1/5 in 4s
INFO: HTTP Request Returned 500 Internal Server Error: Too many links - /var/cache/chef/sandboxes/242d0dbbe1a84a98b11a6ab776fb5ddd
ERROR: Server returned error for https://chef.dr.spergtastic.com/sandboxes, retrying 2/5 in 7s
INFO: HTTP Request Returned 500 Internal Server Error: Too many links - /var/cache/chef/sandboxes/848dbce5df314dfcaaed178c1e50856f
ERROR: Server returned error for https://chef.dr.spergtastic.com/sandboxes, retrying 3/5 in 11s
INFO: HTTP Request Returned 500 Internal Server Error: Too many links - /var/cache/chef/sandboxes/5e975b05a06347c0bb949cbe1c6f549e

i try uploading just one cookbook, and the error is different:

chef:~/ops/chef-repo$ knife cookbook upload s3cmd -o ./cookbooks/
Uploading s3cmd [0.0.1]
ERROR: Server returned error for https://chef.dr.spergtastic.com/sandboxes, retrying 1/5 in 3s
ERROR: Server returned error for https://chef.dr.spergtastic.com/sandboxes, retrying 2/5 in 7s
ERROR: Server returned error for https://chef.dr.spergtastic.com/sandboxes, retrying 3/5 in 10s

i tried restarting couchdb and chef-server, still no joy.

chef-server is 0.10.8-1, ubuntu natty.
this is a secondary chef-server in my infrastructure serving a DR
datacenter. it loads the same sets of cookbooks, data bags, etc from
chef-repo in git.

fwiw, i realized i hadn't been compacting couchdb here. the chef couchdb
on the fs was 1.9G. i ran compact and it's now 45Mb. but i'm still seeing
the above error.

also fwiw, above compact was done like so, not via chef-server cookbook:

/usr/bin/curl -H "Content-Type: application/json" -X POST http://localhost:5984/chef/_compact
/usr/bin/curl -H "Content-Type: application/json" -X POST http://localhost:5984/chef/_view_cleanup
for view in registrations sandboxes id_map clients checksums cookbooks data_bags nodes roles users; do
/usr/bin/curl -H "Content-Type: application/json" -X POST http://localhost:5984/chef/_compact/$view
done

thanks
kallen

and one more bit of log in case it's helpful:

merb : chef-server (api) : worker (port 4000) ~ Started request handling: Wed Feb 08 05:47:27 +0000 2012
merb : chef-server (api) : worker (port 4000) ~ Params: {"action"=>"create", "checksums"=>{"83580e30b45b9a8e6b3359d47
d14208e"=>nil, "d217102ad8b64da1aa17218621820d49"=>nil, "099b134c847994fc237be5593604fad7"=>nil, "cd33cd00aad2e9f89a6
74baa9c49b24c"=>nil, "cd357558feae778ce719e03dedd4f074"=>nil, "bfccb2c807ce7d9fdb9c6a2cf6f49133"=>nil}, "controller"=

"sandboxes"}
merb : chef-server (api) : worker (port 4000) ~ Too many links - /var/cache/chef/sandboxes/26f2b236b39b403ca272034f5fa95599 - (Errno::EMLINK)
/usr/lib/ruby/1.8/fileutils.rb:243:in mkdir' /usr/lib/ruby/1.8/fileutils.rb:243:in fu_mkdir'
/usr/lib/ruby/1.8/fileutils.rb:217:in mkdir_p' /usr/lib/ruby/1.8/fileutils.rb:215:in reverse_each'
/usr/lib/ruby/1.8/fileutils.rb:215:in mkdir_p' /usr/lib/ruby/1.8/fileutils.rb:201:in each'
/usr/lib/ruby/1.8/fileutils.rb:201:in mkdir_p' /usr/share/chef-server-api/app/controllers/sandboxes.rb:76:in create'
/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in send' /usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in _call_action'
/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:289:in _dispatch' /usr/lib/ruby/1.8/merb-core/controller/merb_controller.rb:252:in _dispatch'
/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:102:in dispatch_action' /usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:74:in handle'
/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:36:in handle' /usr/lib/ruby/1.8/merb-core/rack/application.rb:17:in call'
/usr/lib/ruby/1.8/merb-core/rack/middleware/static.rb:28:in call' /usr/lib/ruby/1.8/rack/content_length.rb:13:in call'
/usr/lib/ruby/1.8/thin/connection.rb:76:in pre_process' /usr/lib/ruby/1.8/thin/connection.rb:74:in catch'
/usr/lib/ruby/1.8/thin/connection.rb:74:in pre_process' /usr/lib/ruby/1.8/thin/connection.rb:57:in process'
/usr/lib/ruby/1.8/thin/connection.rb:42:in receive_data' /usr/lib/ruby/1.8/eventmachine.rb:257:in run_machine'
/usr/lib/ruby/1.8/eventmachine.rb:257:in run' /usr/lib/ruby/1.8/thin/backends/base.rb:57:in start'
/usr/lib/ruby/1.8/thin/server.rb:156:in start' /usr/lib/ruby/1.8/merb-core/rack/adapter/thin.rb:30:in start_server'
/usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:298:in start_at_port' /usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:128:in start'
/usr/lib/ruby/1.8/merb-core/server.rb:174:in bootup' /usr/lib/ruby/1.8/merb-core/server.rb:159:in daemonize'
/usr/lib/ruby/1.8/merb-core/server.rb:143:in fork' /usr/lib/ruby/1.8/merb-core/server.rb:143:in daemonize'
/usr/lib/ruby/1.8/merb-core/server.rb:35:in start' /usr/lib/ruby/1.8/merb-core.rb:170:in start'
/usr/sbin/chef-server:86

AJ_Christensen · February 8, 2012, 6:08am

what file system are you using on the chef server? what is the
contents of the directory the error comes from? have you got some
crazy symlink stuff going on in a cookbook?

--AJ

On 8 February 2012 19:06, kallen@groknaut.net wrote:

hiya. seeing this error on one of my chef servers. anyone run into this?
know what to do?

chef:~/ops/chef-repo$ knife cookbook upload -a -o ./cookbooks/ -V
Uploading activemq [1.0.0]
INFO: Saving activemq
INFO: Validating ruby files
INFO: Validating templates
INFO: Syntax OK
INFO: HTTP Request Returned 500 Internal Server Error: Too many links - /var/cache/chef/sandboxes/26f2b236b39b403ca272034f5fa95599
ERROR: Server returned error for https://chef.dr.spergtastic.com/sandboxes, retrying 1/5 in 4s
INFO: HTTP Request Returned 500 Internal Server Error: Too many links - /var/cache/chef/sandboxes/242d0dbbe1a84a98b11a6ab776fb5ddd
ERROR: Server returned error for https://chef.dr.spergtastic.com/sandboxes, retrying 2/5 in 7s
INFO: HTTP Request Returned 500 Internal Server Error: Too many links - /var/cache/chef/sandboxes/848dbce5df314dfcaaed178c1e50856f
ERROR: Server returned error for https://chef.dr.spergtastic.com/sandboxes, retrying 3/5 in 11s
INFO: HTTP Request Returned 500 Internal Server Error: Too many links - /var/cache/chef/sandboxes/5e975b05a06347c0bb949cbe1c6f549e

i try uploading just one cookbook, and the error is different:

chef:~/ops/chef-repo$ knife cookbook upload s3cmd -o ./cookbooks/
Uploading s3cmd [0.0.1]
ERROR: Server returned error for https://chef.dr.spergtastic.com/sandboxes, retrying 1/5 in 3s
ERROR: Server returned error for https://chef.dr.spergtastic.com/sandboxes, retrying 2/5 in 7s
ERROR: Server returned error for https://chef.dr.spergtastic.com/sandboxes, retrying 3/5 in 10s

i tried restarting couchdb and chef-server, still no joy.

chef-server is 0.10.8-1, ubuntu natty.
this is a secondary chef-server in my infrastructure serving a DR
datacenter. it loads the same sets of cookbooks, data bags, etc from
chef-repo in git.

fwiw, i realized i hadn't been compacting couchdb here. the chef couchdb
on the fs was 1.9G. i ran compact and it's now 45Mb. but i'm still seeing
the above error.

also fwiw, above compact was done like so, not via chef-server cookbook:

/usr/bin/curl -H "Content-Type: application/json" -X POST http://localhost:5984/chef/_compact
/usr/bin/curl -H "Content-Type: application/json" -X POST http://localhost:5984/chef/_view_cleanup
for view in registrations sandboxes id_map clients checksums cookbooks data_bags nodes roles users; do
/usr/bin/curl -H "Content-Type: application/json" -X POST http://localhost:5984/chef/_compact/$view
done

thanks
kallen

and one more bit of log in case it's helpful:

merb : chef-server (api) : worker (port 4000) ~ Started request handling: Wed Feb 08 05:47:27 +0000 2012
merb : chef-server (api) : worker (port 4000) ~ Params: {"action"=>"create", "checksums"=>{"83580e30b45b9a8e6b3359d47
d14208e"=>nil, "d217102ad8b64da1aa17218621820d49"=>nil, "099b134c847994fc237be5593604fad7"=>nil, "cd33cd00aad2e9f89a6
74baa9c49b24c"=>nil, "cd357558feae778ce719e03dedd4f074"=>nil, "bfccb2c807ce7d9fdb9c6a2cf6f49133"=>nil}, "controller"=

"sandboxes"}
merb : chef-server (api) : worker (port 4000) ~ Too many links - /var/cache/chef/sandboxes/26f2b236b39b403ca272034f5fa95599 - (Errno::EMLINK)
/usr/lib/ruby/1.8/fileutils.rb:243:in mkdir' /usr/lib/ruby/1.8/fileutils.rb:243:in fu_mkdir'
/usr/lib/ruby/1.8/fileutils.rb:217:in mkdir_p' /usr/lib/ruby/1.8/fileutils.rb:215:in reverse_each'
/usr/lib/ruby/1.8/fileutils.rb:215:in mkdir_p' /usr/lib/ruby/1.8/fileutils.rb:201:in each'
/usr/lib/ruby/1.8/fileutils.rb:201:in mkdir_p' /usr/share/chef-server-api/app/controllers/sandboxes.rb:76:in create'
/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in send' /usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in _call_action'
/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:289:in _dispatch' /usr/lib/ruby/1.8/merb-core/controller/merb_controller.rb:252:in _dispatch'
/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:102:in dispatch_action' /usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:74:in handle'
/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:36:in handle' /usr/lib/ruby/1.8/merb-core/rack/application.rb:17:in call'
/usr/lib/ruby/1.8/merb-core/rack/middleware/static.rb:28:in call' /usr/lib/ruby/1.8/rack/content_length.rb:13:in call'
/usr/lib/ruby/1.8/thin/connection.rb:76:in pre_process' /usr/lib/ruby/1.8/thin/connection.rb:74:in catch'
/usr/lib/ruby/1.8/thin/connection.rb:74:in pre_process' /usr/lib/ruby/1.8/thin/connection.rb:57:in process'
/usr/lib/ruby/1.8/thin/connection.rb:42:in receive_data' /usr/lib/ruby/1.8/eventmachine.rb:257:in run_machine'
/usr/lib/ruby/1.8/eventmachine.rb:257:in run' /usr/lib/ruby/1.8/thin/backends/base.rb:57:in start'
/usr/lib/ruby/1.8/thin/server.rb:156:in start' /usr/lib/ruby/1.8/merb-core/rack/adapter/thin.rb:30:in start_server'
/usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:298:in start_at_port' /usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:128:in start'
/usr/lib/ruby/1.8/merb-core/server.rb:174:in bootup' /usr/lib/ruby/1.8/merb-core/server.rb:159:in daemonize'
/usr/lib/ruby/1.8/merb-core/server.rb:143:in fork' /usr/lib/ruby/1.8/merb-core/server.rb:143:in daemonize'
/usr/lib/ruby/1.8/merb-core/server.rb:35:in start' /usr/lib/ruby/1.8/merb-core.rb:170:in start'
/usr/sbin/chef-server:86

Kallen · February 8, 2012, 6:13am

On Wed, 08 Feb 2012, AJ Christensen wrote:

what file system are you using on the chef server? what is the
contents of the directory the error comes from? have you got some
crazy symlink stuff going on in a cookbook?

chef server filesystem is ext3.

no crazy symlink stuff is going on in any cookbooks (that i know of).
nothing written by me anyway.

contents of the directory the error comes from .. you mean cwd?

chef:~/ops/chef-repo$ ls -l
total 48
drwxr-xr-x 2 wopr wopr 4096 2012-01-13 00:32 certificates
-rw-r--r-- 1 wopr wopr 156 2012-01-13 00:32 chefignore
drwxr-xr-x 2 wopr wopr 4096 2012-01-13 00:32 config
drwxr-xr-x 49 wopr wopr 4096 2012-02-03 23:55 cookbooks
drwxr-xr-x 11 wopr wopr 4096 2012-02-07 08:20 data_bags
drwxr-xr-x 2 wopr wopr 4096 2012-01-13 00:32 environments
drwxr-xr-x 2 wopr wopr 4096 2012-02-07 08:20 nodes
-rw-r--r-- 1 wopr wopr 2484 2012-01-13 00:32 Rakefile
-rw-r--r-- 1 wopr wopr 3521 2012-01-13 00:32 README.md
drwxr-xr-x 2 wopr wopr 4096 2012-02-07 08:20 roles
drwxr-xr-x 23 wopr wopr 4096 2012-01-13 00:32 site-cookbooks
drwxr-xr-x 2 wopr wopr 4096 2012-01-13 00:32 tasks

chef:~/ops/chef-repo$ ls -1 cookbooks/ | wc -l
48

On 8 February 2012 19:06, kallen@groknaut.net wrote:

hiya. seeing this error on one of my chef servers. anyone run into this?
know what to do?

chef:~/ops/chef-repo$ knife cookbook upload -a -o ./cookbooks/ -V
Uploading activemq [1.0.0]
INFO: Saving activemq
INFO: Validating ruby files
INFO: Validating templates
INFO: Syntax OK
INFO: HTTP Request Returned 500 Internal Server Error: Too many links - /var/cache/chef/sandboxes/26f2b236b39b403ca272034f5fa95599
ERROR: Server returned error for https://chef.dr.spergtastic.com/sandboxes, retrying 1/5 in 4s
INFO: HTTP Request Returned 500 Internal Server Error: Too many links - /var/cache/chef/sandboxes/242d0dbbe1a84a98b11a6ab776fb5ddd
ERROR: Server returned error for https://chef.dr.spergtastic.com/sandboxes, retrying 2/5 in 7s
INFO: HTTP Request Returned 500 Internal Server Error: Too many links - /var/cache/chef/sandboxes/848dbce5df314dfcaaed178c1e50856f
ERROR: Server returned error for https://chef.dr.spergtastic.com/sandboxes, retrying 3/5 in 11s
INFO: HTTP Request Returned 500 Internal Server Error: Too many links - /var/cache/chef/sandboxes/5e975b05a06347c0bb949cbe1c6f549e

i try uploading just one cookbook, and the error is different:

chef:~/ops/chef-repo$ knife cookbook upload s3cmd -o ./cookbooks/
Uploading s3cmd [0.0.1]
ERROR: Server returned error for https://chef.dr.spergtastic.com/sandboxes, retrying 1/5 in 3s
ERROR: Server returned error for https://chef.dr.spergtastic.com/sandboxes, retrying 2/5 in 7s
ERROR: Server returned error for https://chef.dr.spergtastic.com/sandboxes, retrying 3/5 in 10s

i tried restarting couchdb and chef-server, still no joy.

chef-server is 0.10.8-1, ubuntu natty.
this is a secondary chef-server in my infrastructure serving a DR
datacenter. it loads the same sets of cookbooks, data bags, etc from
chef-repo in git.

fwiw, i realized i hadn't been compacting couchdb here. the chef couchdb
on the fs was 1.9G. i ran compact and it's now 45Mb. but i'm still seeing
the above error.

also fwiw, above compact was done like so, not via chef-server cookbook:

/usr/bin/curl -H "Content-Type: application/json" -X POST http://localhost:5984/chef/_compact
/usr/bin/curl -H "Content-Type: application/json" -X POST http://localhost:5984/chef/_view_cleanup
for view in registrations sandboxes id_map clients checksums cookbooks data_bags nodes roles users; do
/usr/bin/curl -H "Content-Type: application/json" -X POST http://localhost:5984/chef/_compact/$view
done

thanks
kallen

and one more bit of log in case it's helpful:

merb : chef-server (api) : worker (port 4000) ~ Started request handling: Wed Feb 08 05:47:27 +0000 2012
merb : chef-server (api) : worker (port 4000) ~ Params: {"action"=>"create", "checksums"=>{"83580e30b45b9a8e6b3359d47
d14208e"=>nil, "d217102ad8b64da1aa17218621820d49"=>nil, "099b134c847994fc237be5593604fad7"=>nil, "cd33cd00aad2e9f89a6
74baa9c49b24c"=>nil, "cd357558feae778ce719e03dedd4f074"=>nil, "bfccb2c807ce7d9fdb9c6a2cf6f49133"=>nil}, "controller"=

"sandboxes"}
merb : chef-server (api) : worker (port 4000) ~ Too many links - /var/cache/chef/sandboxes/26f2b236b39b403ca272034f5fa95599 - (Errno::EMLINK)
/usr/lib/ruby/1.8/fileutils.rb:243:in mkdir' /usr/lib/ruby/1.8/fileutils.rb:243:in fu_mkdir'
/usr/lib/ruby/1.8/fileutils.rb:217:in mkdir_p' /usr/lib/ruby/1.8/fileutils.rb:215:in reverse_each'
/usr/lib/ruby/1.8/fileutils.rb:215:in mkdir_p' /usr/lib/ruby/1.8/fileutils.rb:201:in each'
/usr/lib/ruby/1.8/fileutils.rb:201:in mkdir_p' /usr/share/chef-server-api/app/controllers/sandboxes.rb:76:in create'
/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in send' /usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in _call_action'
/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:289:in _dispatch' /usr/lib/ruby/1.8/merb-core/controller/merb_controller.rb:252:in _dispatch'
/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:102:in dispatch_action' /usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:74:in handle'
/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:36:in handle' /usr/lib/ruby/1.8/merb-core/rack/application.rb:17:in call'
/usr/lib/ruby/1.8/merb-core/rack/middleware/static.rb:28:in call' /usr/lib/ruby/1.8/rack/content_length.rb:13:in call'
/usr/lib/ruby/1.8/thin/connection.rb:76:in pre_process' /usr/lib/ruby/1.8/thin/connection.rb:74:in catch'
/usr/lib/ruby/1.8/thin/connection.rb:74:in pre_process' /usr/lib/ruby/1.8/thin/connection.rb:57:in process'
/usr/lib/ruby/1.8/thin/connection.rb:42:in receive_data' /usr/lib/ruby/1.8/eventmachine.rb:257:in run_machine'
/usr/lib/ruby/1.8/eventmachine.rb:257:in run' /usr/lib/ruby/1.8/thin/backends/base.rb:57:in start'
/usr/lib/ruby/1.8/thin/server.rb:156:in start' /usr/lib/ruby/1.8/merb-core/rack/adapter/thin.rb:30:in start_server'
/usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:298:in start_at_port' /usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:128:in start'
/usr/lib/ruby/1.8/merb-core/server.rb:174:in bootup' /usr/lib/ruby/1.8/merb-core/server.rb:159:in daemonize'
/usr/lib/ruby/1.8/merb-core/server.rb:143:in fork' /usr/lib/ruby/1.8/merb-core/server.rb:143:in daemonize'
/usr/lib/ruby/1.8/merb-core/server.rb:35:in start' /usr/lib/ruby/1.8/merb-core.rb:170:in start'
/usr/sbin/chef-server:86

AJ_Christensen · February 8, 2012, 6:25am

No, I meant the directories such as:

/var/cache/chef/sandboxes/26f2b236b39b403ca272034f5fa95599

Cheers,

--AJ

On 8 February 2012 19:13, kallen@groknaut.net wrote:

On Wed, 08 Feb 2012, AJ Christensen wrote:

what file system are you using on the chef server? what is the
contents of the directory the error comes from? have you got some
crazy symlink stuff going on in a cookbook?

chef server filesystem is ext3.

no crazy symlink stuff is going on in any cookbooks (that i know of).
nothing written by me anyway.

contents of the directory the error comes from .. you mean cwd?

chef:~/ops/chef-repo$ ls -l
total 48
drwxr-xr-x 2 wopr wopr 4096 2012-01-13 00:32 certificates
-rw-r--r-- 1 wopr wopr 156 2012-01-13 00:32 chefignore
drwxr-xr-x 2 wopr wopr 4096 2012-01-13 00:32 config
drwxr-xr-x 49 wopr wopr 4096 2012-02-03 23:55 cookbooks
drwxr-xr-x 11 wopr wopr 4096 2012-02-07 08:20 data_bags
drwxr-xr-x 2 wopr wopr 4096 2012-01-13 00:32 environments
drwxr-xr-x 2 wopr wopr 4096 2012-02-07 08:20 nodes
-rw-r--r-- 1 wopr wopr 2484 2012-01-13 00:32 Rakefile
-rw-r--r-- 1 wopr wopr 3521 2012-01-13 00:32 README.md
drwxr-xr-x 2 wopr wopr 4096 2012-02-07 08:20 roles
drwxr-xr-x 23 wopr wopr 4096 2012-01-13 00:32 site-cookbooks
drwxr-xr-x 2 wopr wopr 4096 2012-01-13 00:32 tasks

chef:~/ops/chef-repo$ ls -1 cookbooks/ | wc -l
48

On 8 February 2012 19:06, kallen@groknaut.net wrote:

hiya. seeing this error on one of my chef servers. anyone run into this?
know what to do?

chef:~/ops/chef-repo$ knife cookbook upload -a -o ./cookbooks/ -V
Uploading activemq [1.0.0]
INFO: Saving activemq
INFO: Validating ruby files
INFO: Validating templates
INFO: Syntax OK
INFO: HTTP Request Returned 500 Internal Server Error: Too many links - /var/cache/chef/sandboxes/26f2b236b39b403ca272034f5fa95599
ERROR: Server returned error for https://chef.dr.spergtastic.com/sandboxes, retrying 1/5 in 4s
INFO: HTTP Request Returned 500 Internal Server Error: Too many links - /var/cache/chef/sandboxes/242d0dbbe1a84a98b11a6ab776fb5ddd
ERROR: Server returned error for https://chef.dr.spergtastic.com/sandboxes, retrying 2/5 in 7s
INFO: HTTP Request Returned 500 Internal Server Error: Too many links - /var/cache/chef/sandboxes/848dbce5df314dfcaaed178c1e50856f
ERROR: Server returned error for https://chef.dr.spergtastic.com/sandboxes, retrying 3/5 in 11s
INFO: HTTP Request Returned 500 Internal Server Error: Too many links - /var/cache/chef/sandboxes/5e975b05a06347c0bb949cbe1c6f549e

i try uploading just one cookbook, and the error is different:

chef:~/ops/chef-repo$ knife cookbook upload s3cmd -o ./cookbooks/
Uploading s3cmd [0.0.1]
ERROR: Server returned error for https://chef.dr.spergtastic.com/sandboxes, retrying 1/5 in 3s
ERROR: Server returned error for https://chef.dr.spergtastic.com/sandboxes, retrying 2/5 in 7s
ERROR: Server returned error for https://chef.dr.spergtastic.com/sandboxes, retrying 3/5 in 10s

i tried restarting couchdb and chef-server, still no joy.

chef-server is 0.10.8-1, ubuntu natty.
this is a secondary chef-server in my infrastructure serving a DR
datacenter. it loads the same sets of cookbooks, data bags, etc from
chef-repo in git.

fwiw, i realized i hadn't been compacting couchdb here. the chef couchdb
on the fs was 1.9G. i ran compact and it's now 45Mb. but i'm still seeing
the above error.

also fwiw, above compact was done like so, not via chef-server cookbook:

/usr/bin/curl -H "Content-Type: application/json" -X POST http://localhost:5984/chef/_compact
/usr/bin/curl -H "Content-Type: application/json" -X POST http://localhost:5984/chef/_view_cleanup
for view in registrations sandboxes id_map clients checksums cookbooks data_bags nodes roles users; do
/usr/bin/curl -H "Content-Type: application/json" -X POST http://localhost:5984/chef/_compact/$view
done

thanks
kallen

and one more bit of log in case it's helpful:

merb : chef-server (api) : worker (port 4000) ~ Started request handling: Wed Feb 08 05:47:27 +0000 2012
merb : chef-server (api) : worker (port 4000) ~ Params: {"action"=>"create", "checksums"=>{"83580e30b45b9a8e6b3359d47
d14208e"=>nil, "d217102ad8b64da1aa17218621820d49"=>nil, "099b134c847994fc237be5593604fad7"=>nil, "cd33cd00aad2e9f89a6
74baa9c49b24c"=>nil, "cd357558feae778ce719e03dedd4f074"=>nil, "bfccb2c807ce7d9fdb9c6a2cf6f49133"=>nil}, "controller"=

"sandboxes"}
merb : chef-server (api) : worker (port 4000) ~ Too many links - /var/cache/chef/sandboxes/26f2b236b39b403ca272034f5fa95599 - (Errno::EMLINK)
/usr/lib/ruby/1.8/fileutils.rb:243:in mkdir' /usr/lib/ruby/1.8/fileutils.rb:243:in fu_mkdir'
/usr/lib/ruby/1.8/fileutils.rb:217:in mkdir_p' /usr/lib/ruby/1.8/fileutils.rb:215:in reverse_each'
/usr/lib/ruby/1.8/fileutils.rb:215:in mkdir_p' /usr/lib/ruby/1.8/fileutils.rb:201:in each'
/usr/lib/ruby/1.8/fileutils.rb:201:in mkdir_p' /usr/share/chef-server-api/app/controllers/sandboxes.rb:76:in create'
/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in send' /usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in _call_action'
/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:289:in _dispatch' /usr/lib/ruby/1.8/merb-core/controller/merb_controller.rb:252:in _dispatch'
/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:102:in dispatch_action' /usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:74:in handle'
/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:36:in handle' /usr/lib/ruby/1.8/merb-core/rack/application.rb:17:in call'
/usr/lib/ruby/1.8/merb-core/rack/middleware/static.rb:28:in call' /usr/lib/ruby/1.8/rack/content_length.rb:13:in call'
/usr/lib/ruby/1.8/thin/connection.rb:76:in pre_process' /usr/lib/ruby/1.8/thin/connection.rb:74:in catch'
/usr/lib/ruby/1.8/thin/connection.rb:74:in pre_process' /usr/lib/ruby/1.8/thin/connection.rb:57:in process'
/usr/lib/ruby/1.8/thin/connection.rb:42:in receive_data' /usr/lib/ruby/1.8/eventmachine.rb:257:in run_machine'
/usr/lib/ruby/1.8/eventmachine.rb:257:in run' /usr/lib/ruby/1.8/thin/backends/base.rb:57:in start'
/usr/lib/ruby/1.8/thin/server.rb:156:in start' /usr/lib/ruby/1.8/merb-core/rack/adapter/thin.rb:30:in start_server'
/usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:298:in start_at_port' /usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:128:in start'
/usr/lib/ruby/1.8/merb-core/server.rb:174:in bootup' /usr/lib/ruby/1.8/merb-core/server.rb:159:in daemonize'
/usr/lib/ruby/1.8/merb-core/server.rb:143:in fork' /usr/lib/ruby/1.8/merb-core/server.rb:143:in daemonize'
/usr/lib/ruby/1.8/merb-core/server.rb:35:in start' /usr/lib/ruby/1.8/merb-core.rb:170:in start'
/usr/sbin/chef-server:86

Kallen · February 8, 2012, 6:33am

On Wed, 08 Feb 2012, AJ Christensen wrote:

No, I meant the directories such as:

/var/cache/chef/sandboxes/26f2b236b39b403ca272034f5fa95599

ahh. well...

getting no such file or dir for the paths mentioned in the errors below.
but:

DR chef server having the problem:
find /var/cache/chef/sandboxes -maxdepth 1 -type d | wc -l
31999

prod chef server that's thankfully not having problem:
find /var/cache/chef/sandboxes -maxdepth 1 -type d | wc -l
1162

yeah that's a lotta dirs in that dir.

how'd that happen? i wonder what dirs therein i can prune, how to
prevent it from happening again. or if i'll need to take a heavy hammer
and wipe and recreate this DR chef server.

On 8 February 2012 19:13, kallen@groknaut.net wrote:

On Wed, 08 Feb 2012, AJ Christensen wrote:

what file system are you using on the chef server? what is the
contents of the directory the error comes from? have you got some
crazy symlink stuff going on in a cookbook?

chef server filesystem is ext3.

no crazy symlink stuff is going on in any cookbooks (that i know of).
nothing written by me anyway.

contents of the directory the error comes from .. you mean cwd?

chef:~/ops/chef-repo$ ls -l
total 48
drwxr-xr-x 2 wopr wopr 4096 2012-01-13 00:32 certificates
-rw-r--r-- 1 wopr wopr 156 2012-01-13 00:32 chefignore
drwxr-xr-x 2 wopr wopr 4096 2012-01-13 00:32 config
drwxr-xr-x 49 wopr wopr 4096 2012-02-03 23:55 cookbooks
drwxr-xr-x 11 wopr wopr 4096 2012-02-07 08:20 data_bags
drwxr-xr-x 2 wopr wopr 4096 2012-01-13 00:32 environments
drwxr-xr-x 2 wopr wopr 4096 2012-02-07 08:20 nodes
-rw-r--r-- 1 wopr wopr 2484 2012-01-13 00:32 Rakefile
-rw-r--r-- 1 wopr wopr 3521 2012-01-13 00:32 README.md
drwxr-xr-x 2 wopr wopr 4096 2012-02-07 08:20 roles
drwxr-xr-x 23 wopr wopr 4096 2012-01-13 00:32 site-cookbooks
drwxr-xr-x 2 wopr wopr 4096 2012-01-13 00:32 tasks

chef:~/ops/chef-repo$ ls -1 cookbooks/ | wc -l
48

On 8 February 2012 19:06, kallen@groknaut.net wrote:

hiya. seeing this error on one of my chef servers. anyone run into this?
know what to do?

chef:~/ops/chef-repo$ knife cookbook upload -a -o ./cookbooks/ -V
Uploading activemq [1.0.0]
INFO: Saving activemq
INFO: Validating ruby files
INFO: Validating templates
INFO: Syntax OK
INFO: HTTP Request Returned 500 Internal Server Error: Too many links - /var/cache/chef/sandboxes/26f2b236b39b403ca272034f5fa95599
ERROR: Server returned error for https://chef.dr.spergtastic.com/sandboxes, retrying 1/5 in 4s
INFO: HTTP Request Returned 500 Internal Server Error: Too many links - /var/cache/chef/sandboxes/242d0dbbe1a84a98b11a6ab776fb5ddd
ERROR: Server returned error for https://chef.dr.spergtastic.com/sandboxes, retrying 2/5 in 7s
INFO: HTTP Request Returned 500 Internal Server Error: Too many links - /var/cache/chef/sandboxes/848dbce5df314dfcaaed178c1e50856f
ERROR: Server returned error for https://chef.dr.spergtastic.com/sandboxes, retrying 3/5 in 11s
INFO: HTTP Request Returned 500 Internal Server Error: Too many links - /var/cache/chef/sandboxes/5e975b05a06347c0bb949cbe1c6f549e

i try uploading just one cookbook, and the error is different:

chef:~/ops/chef-repo$ knife cookbook upload s3cmd -o ./cookbooks/
Uploading s3cmd [0.0.1]
ERROR: Server returned error for https://chef.dr.spergtastic.com/sandboxes, retrying 1/5 in 3s
ERROR: Server returned error for https://chef.dr.spergtastic.com/sandboxes, retrying 2/5 in 7s
ERROR: Server returned error for https://chef.dr.spergtastic.com/sandboxes, retrying 3/5 in 10s

i tried restarting couchdb and chef-server, still no joy.

chef-server is 0.10.8-1, ubuntu natty.
this is a secondary chef-server in my infrastructure serving a DR
datacenter. it loads the same sets of cookbooks, data bags, etc from
chef-repo in git.

fwiw, i realized i hadn't been compacting couchdb here. the chef couchdb
on the fs was 1.9G. i ran compact and it's now 45Mb. but i'm still seeing
the above error.

also fwiw, above compact was done like so, not via chef-server cookbook:

/usr/bin/curl -H "Content-Type: application/json" -X POST http://localhost:5984/chef/_compact
/usr/bin/curl -H "Content-Type: application/json" -X POST http://localhost:5984/chef/_view_cleanup
for view in registrations sandboxes id_map clients checksums cookbooks data_bags nodes roles users; do
/usr/bin/curl -H "Content-Type: application/json" -X POST http://localhost:5984/chef/_compact/$view
done

thanks
kallen

and one more bit of log in case it's helpful:

merb : chef-server (api) : worker (port 4000) ~ Started request handling: Wed Feb 08 05:47:27 +0000 2012
merb : chef-server (api) : worker (port 4000) ~ Params: {"action"=>"create", "checksums"=>{"83580e30b45b9a8e6b3359d47
d14208e"=>nil, "d217102ad8b64da1aa17218621820d49"=>nil, "099b134c847994fc237be5593604fad7"=>nil, "cd33cd00aad2e9f89a6
74baa9c49b24c"=>nil, "cd357558feae778ce719e03dedd4f074"=>nil, "bfccb2c807ce7d9fdb9c6a2cf6f49133"=>nil}, "controller"=

"sandboxes"}
merb : chef-server (api) : worker (port 4000) ~ Too many links - /var/cache/chef/sandboxes/26f2b236b39b403ca272034f5fa95599 - (Errno::EMLINK)
/usr/lib/ruby/1.8/fileutils.rb:243:in mkdir' /usr/lib/ruby/1.8/fileutils.rb:243:in fu_mkdir'
/usr/lib/ruby/1.8/fileutils.rb:217:in mkdir_p' /usr/lib/ruby/1.8/fileutils.rb:215:in reverse_each'
/usr/lib/ruby/1.8/fileutils.rb:215:in mkdir_p' /usr/lib/ruby/1.8/fileutils.rb:201:in each'
/usr/lib/ruby/1.8/fileutils.rb:201:in mkdir_p' /usr/share/chef-server-api/app/controllers/sandboxes.rb:76:in create'
/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in send' /usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in _call_action'
/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:289:in _dispatch' /usr/lib/ruby/1.8/merb-core/controller/merb_controller.rb:252:in _dispatch'
/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:102:in dispatch_action' /usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:74:in handle'
/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:36:in handle' /usr/lib/ruby/1.8/merb-core/rack/application.rb:17:in call'
/usr/lib/ruby/1.8/merb-core/rack/middleware/static.rb:28:in call' /usr/lib/ruby/1.8/rack/content_length.rb:13:in call'
/usr/lib/ruby/1.8/thin/connection.rb:76:in pre_process' /usr/lib/ruby/1.8/thin/connection.rb:74:in catch'
/usr/lib/ruby/1.8/thin/connection.rb:74:in pre_process' /usr/lib/ruby/1.8/thin/connection.rb:57:in process'
/usr/lib/ruby/1.8/thin/connection.rb:42:in receive_data' /usr/lib/ruby/1.8/eventmachine.rb:257:in run_machine'
/usr/lib/ruby/1.8/eventmachine.rb:257:in run' /usr/lib/ruby/1.8/thin/backends/base.rb:57:in start'
/usr/lib/ruby/1.8/thin/server.rb:156:in start' /usr/lib/ruby/1.8/merb-core/rack/adapter/thin.rb:30:in start_server'
/usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:298:in start_at_port' /usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:128:in start'
/usr/lib/ruby/1.8/merb-core/server.rb:174:in bootup' /usr/lib/ruby/1.8/merb-core/server.rb:159:in daemonize'
/usr/lib/ruby/1.8/merb-core/server.rb:143:in fork' /usr/lib/ruby/1.8/merb-core/server.rb:143:in daemonize'
/usr/lib/ruby/1.8/merb-core/server.rb:35:in start' /usr/lib/ruby/1.8/merb-core.rb:170:in start'
/usr/sbin/chef-server:86

AJ_Christensen · February 8, 2012, 6:43am

You can probably smash the sandboxes directory and re-upload all of
the cookbooks, do backups or whatever.

What are all the files? some bad symlinking? you might actually have
to look at them, not just count them

--AJ

On 8 February 2012 19:33, kallen@groknaut.net wrote:

On Wed, 08 Feb 2012, AJ Christensen wrote:

No, I meant the directories such as:

/var/cache/chef/sandboxes/26f2b236b39b403ca272034f5fa95599

ahh. well...

getting no such file or dir for the paths mentioned in the errors below.
but:

DR chef server having the problem:
find /var/cache/chef/sandboxes -maxdepth 1 -type d | wc -l
31999

prod chef server that's thankfully not having problem:
find /var/cache/chef/sandboxes -maxdepth 1 -type d | wc -l
1162

yeah that's a lotta dirs in that dir.

how'd that happen? i wonder what dirs therein i can prune, how to
prevent it from happening again. or if i'll need to take a heavy hammer
and wipe and recreate this DR chef server.

On 8 February 2012 19:13, kallen@groknaut.net wrote:

On Wed, 08 Feb 2012, AJ Christensen wrote:

what file system are you using on the chef server? what is the
contents of the directory the error comes from? have you got some
crazy symlink stuff going on in a cookbook?

chef server filesystem is ext3.

no crazy symlink stuff is going on in any cookbooks (that i know of).
nothing written by me anyway.

contents of the directory the error comes from .. you mean cwd?

chef:~/ops/chef-repo$ ls -l
total 48
drwxr-xr-x 2 wopr wopr 4096 2012-01-13 00:32 certificates
-rw-r--r-- 1 wopr wopr 156 2012-01-13 00:32 chefignore
drwxr-xr-x 2 wopr wopr 4096 2012-01-13 00:32 config
drwxr-xr-x 49 wopr wopr 4096 2012-02-03 23:55 cookbooks
drwxr-xr-x 11 wopr wopr 4096 2012-02-07 08:20 data_bags
drwxr-xr-x 2 wopr wopr 4096 2012-01-13 00:32 environments
drwxr-xr-x 2 wopr wopr 4096 2012-02-07 08:20 nodes
-rw-r--r-- 1 wopr wopr 2484 2012-01-13 00:32 Rakefile
-rw-r--r-- 1 wopr wopr 3521 2012-01-13 00:32 README.md
drwxr-xr-x 2 wopr wopr 4096 2012-02-07 08:20 roles
drwxr-xr-x 23 wopr wopr 4096 2012-01-13 00:32 site-cookbooks
drwxr-xr-x 2 wopr wopr 4096 2012-01-13 00:32 tasks

chef:~/ops/chef-repo$ ls -1 cookbooks/ | wc -l
48

On 8 February 2012 19:06, kallen@groknaut.net wrote:

hiya. seeing this error on one of my chef servers. anyone run into this?
know what to do?

chef:~/ops/chef-repo$ knife cookbook upload -a -o ./cookbooks/ -V
Uploading activemq [1.0.0]
INFO: Saving activemq
INFO: Validating ruby files
INFO: Validating templates
INFO: Syntax OK
INFO: HTTP Request Returned 500 Internal Server Error: Too many links - /var/cache/chef/sandboxes/26f2b236b39b403ca272034f5fa95599
ERROR: Server returned error for https://chef.dr.spergtastic.com/sandboxes, retrying 1/5 in 4s
INFO: HTTP Request Returned 500 Internal Server Error: Too many links - /var/cache/chef/sandboxes/242d0dbbe1a84a98b11a6ab776fb5ddd
ERROR: Server returned error for https://chef.dr.spergtastic.com/sandboxes, retrying 2/5 in 7s
INFO: HTTP Request Returned 500 Internal Server Error: Too many links - /var/cache/chef/sandboxes/848dbce5df314dfcaaed178c1e50856f
ERROR: Server returned error for https://chef.dr.spergtastic.com/sandboxes, retrying 3/5 in 11s
INFO: HTTP Request Returned 500 Internal Server Error: Too many links - /var/cache/chef/sandboxes/5e975b05a06347c0bb949cbe1c6f549e

i try uploading just one cookbook, and the error is different:

chef:~/ops/chef-repo$ knife cookbook upload s3cmd -o ./cookbooks/
Uploading s3cmd [0.0.1]
ERROR: Server returned error for https://chef.dr.spergtastic.com/sandboxes, retrying 1/5 in 3s
ERROR: Server returned error for https://chef.dr.spergtastic.com/sandboxes, retrying 2/5 in 7s
ERROR: Server returned error for https://chef.dr.spergtastic.com/sandboxes, retrying 3/5 in 10s

i tried restarting couchdb and chef-server, still no joy.

chef-server is 0.10.8-1, ubuntu natty.
this is a secondary chef-server in my infrastructure serving a DR
datacenter. it loads the same sets of cookbooks, data bags, etc from
chef-repo in git.

fwiw, i realized i hadn't been compacting couchdb here. the chef couchdb
on the fs was 1.9G. i ran compact and it's now 45Mb. but i'm still seeing
the above error.

also fwiw, above compact was done like so, not via chef-server cookbook:

/usr/bin/curl -H "Content-Type: application/json" -X POST http://localhost:5984/chef/_compact
/usr/bin/curl -H "Content-Type: application/json" -X POST http://localhost:5984/chef/_view_cleanup
for view in registrations sandboxes id_map clients checksums cookbooks data_bags nodes roles users; do
/usr/bin/curl -H "Content-Type: application/json" -X POST http://localhost:5984/chef/_compact/$view
done

thanks
kallen

and one more bit of log in case it's helpful:

merb : chef-server (api) : worker (port 4000) ~ Started request handling: Wed Feb 08 05:47:27 +0000 2012
merb : chef-server (api) : worker (port 4000) ~ Params: {"action"=>"create", "checksums"=>{"83580e30b45b9a8e6b3359d47
d14208e"=>nil, "d217102ad8b64da1aa17218621820d49"=>nil, "099b134c847994fc237be5593604fad7"=>nil, "cd33cd00aad2e9f89a6
74baa9c49b24c"=>nil, "cd357558feae778ce719e03dedd4f074"=>nil, "bfccb2c807ce7d9fdb9c6a2cf6f49133"=>nil}, "controller"=

"sandboxes"}
merb : chef-server (api) : worker (port 4000) ~ Too many links - /var/cache/chef/sandboxes/26f2b236b39b403ca272034f5fa95599 - (Errno::EMLINK)
/usr/lib/ruby/1.8/fileutils.rb:243:in mkdir' /usr/lib/ruby/1.8/fileutils.rb:243:in fu_mkdir'
/usr/lib/ruby/1.8/fileutils.rb:217:in mkdir_p' /usr/lib/ruby/1.8/fileutils.rb:215:in reverse_each'
/usr/lib/ruby/1.8/fileutils.rb:215:in mkdir_p' /usr/lib/ruby/1.8/fileutils.rb:201:in each'
/usr/lib/ruby/1.8/fileutils.rb:201:in mkdir_p' /usr/share/chef-server-api/app/controllers/sandboxes.rb:76:in create'
/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in send' /usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:315:in _call_action'
/usr/lib/ruby/1.8/merb-core/controller/abstract_controller.rb:289:in _dispatch' /usr/lib/ruby/1.8/merb-core/controller/merb_controller.rb:252:in _dispatch'
/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:102:in dispatch_action' /usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:74:in handle'
/usr/lib/ruby/1.8/merb-core/dispatch/dispatcher.rb:36:in handle' /usr/lib/ruby/1.8/merb-core/rack/application.rb:17:in call'
/usr/lib/ruby/1.8/merb-core/rack/middleware/static.rb:28:in call' /usr/lib/ruby/1.8/rack/content_length.rb:13:in call'
/usr/lib/ruby/1.8/thin/connection.rb:76:in pre_process' /usr/lib/ruby/1.8/thin/connection.rb:74:in catch'
/usr/lib/ruby/1.8/thin/connection.rb:74:in pre_process' /usr/lib/ruby/1.8/thin/connection.rb:57:in process'
/usr/lib/ruby/1.8/thin/connection.rb:42:in receive_data' /usr/lib/ruby/1.8/eventmachine.rb:257:in run_machine'
/usr/lib/ruby/1.8/eventmachine.rb:257:in run' /usr/lib/ruby/1.8/thin/backends/base.rb:57:in start'
/usr/lib/ruby/1.8/thin/server.rb:156:in start' /usr/lib/ruby/1.8/merb-core/rack/adapter/thin.rb:30:in start_server'
/usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:298:in start_at_port' /usr/lib/ruby/1.8/merb-core/rack/adapter/abstract.rb:128:in start'
/usr/lib/ruby/1.8/merb-core/server.rb:174:in bootup' /usr/lib/ruby/1.8/merb-core/server.rb:159:in daemonize'
/usr/lib/ruby/1.8/merb-core/server.rb:143:in fork' /usr/lib/ruby/1.8/merb-core/server.rb:143:in daemonize'
/usr/lib/ruby/1.8/merb-core/server.rb:35:in start' /usr/lib/ruby/1.8/merb-core.rb:170:in start'
/usr/sbin/chef-server:86

Kallen · February 8, 2012, 7:17am

On Wed, 08 Feb 2012, AJ Christensen wrote:

You can probably smash the sandboxes directory and re-upload all of
the cookbooks, do backups or whatever.

What are all the files? some bad symlinking? you might actually have
to look at them, not just count them

i find nothing but directories under /var/cache/chef/sandboxes/; there
are no symlinks. with ext3 on this fs, i've hit the max number of
subdirectories in one directory: 32000.

i'll smash away and re-upload.

thanks for helping me find wtf was goin' on

k

On 8 February 2012 19:33, kallen@groknaut.net wrote:

On Wed, 08 Feb 2012, AJ Christensen wrote:

No, I meant the directories such as:

/var/cache/chef/sandboxes/26f2b236b39b403ca272034f5fa95599

ahh. well...

getting no such file or dir for the paths mentioned in the errors below.
but:

DR chef server having the problem:
find /var/cache/chef/sandboxes -maxdepth 1 -type d | wc -l
31999

prod chef server that's thankfully not having problem:
find /var/cache/chef/sandboxes -maxdepth 1 -type d | wc -l
1162

yeah that's a lotta dirs in that dir.

how'd that happen? i wonder what dirs therein i can prune, how to
prevent it from happening again. or if i'll need to take a heavy hammer
and wipe and recreate this DR chef server.

Topic		Replies	Views
500 error when upload cookbook? Chef Infra (archive)	4	4755	June 13, 2016
Error uploading Cookbooks Chef Infra (archive)	4	1842	March 7, 2013
SSL error while uploading cookbooks to hosted chef Chef Infra (archive)	0	750	March 5, 2016
Cookbook Upload via Chef Server API Chef Infra (archive)	2	580	March 30, 2017
Knife cookbook upload <cookbook_name> does not work Chef Infra (archive)	1	1108	July 21, 2016

Cookbook upload: HTTP 500: Too many links

Related topics