Cookbook upload internal server error

I just spun up a Chef Server community AMI on EC2. It’s running 11.0.10 and
I’ve setup a Chef workstation with 11.10.4 along with knife. When I run “knife
cookbook upload” (I tried both --all and a basic one like iptables), I get:

Uploading iptables [0.12.0]
ERROR: internal server error
Response: internal service error

On the server side, I run “sudo chef-server-ctl tail” and I see this:


2014-02-24_01:29:10.31425 =ERROR REPORT==== 24-Feb-2014::01:29:10 ===
2014-02-24_01:29:10.31426 Checking presence of checksum:
<<“1f3ac5801386aa2b22e4cbb9877fb431”>> for org
<<“00000000000000000000000000000000”>> from bucket “bookshelf” has taken longer
than 5000 ms
2014-02-24_01:29:10.31768
2014-02-24_01:29:10.31770 =ERROR REPORT==== 24-Feb-2014::01:29:10 ===
2014-02-24_01:29:10.31771 webmachine error:
path="/sandboxes/000000000000c0ddfbf55f671a4b55be"
2014-02-24_01:29:10.31772 {error,
2014-02-24_01:29:10.31773 {throw,
2014-02-24_01:29:10.31773 {checksum_check_error,13},
2014-02-24_01:29:10.31774
[{chef_wm_named_sandbox,validate_checksums_uploaded,2,
2014-02-24_01:29:10.31775
[{file,“src/chef_wm_named_sandbox.erl”},{line,144}]},
2014-02-24_01:29:10.31776 {chef_wm_named_sandbox,from_json,2,
2014-02-24_01:29:10.31776
[{file,“src/chef_wm_named_sandbox.erl”},{line,99}]},
2014-02-24_01:29:10.31777 {webmachine_resource,resource_call,3,
2014-02-24_01:29:10.31778
[{file,“src/webmachine_resource.erl”},{line,166}]},
2014-02-24_01:29:10.31778 {webmachine_resource,do,3,
2014-02-24_01:29:10.31779
[{file,“src/webmachine_resource.erl”},{line,125}]},
2014-02-24_01:29:10.31780 {webmachine_decision_core,resource_call,1,
2014-02-24_01:29:10.31781
[{file,“src/webmachine_decision_core.erl”},{line,48}]},
2014-02-24_01:29:10.31781 {webmachine_decision_core,accept_helper,0,
2014-02-24_01:29:10.31782
[{file,“src/webmachine_decision_core.erl”},{line,583}]},
2014-02-24_01:29:10.31783 {webmachine_decision_core,decision,1,
2014-02-24_01:29:10.31784
[{file,“src/webmachine_decision_core.erl”},{line,489}]},
2014-02-24_01:29:10.31784 {webmachine_decision_core,handle_request,2,
2014-02-24_01:29:10.31786
[{file,“src/webmachine_decision_core.erl”},{line,33}]}]}}

The ellipsis (…) is actually that first “ERROR REPORT” repeated several
times, I’m guessing for each piece of the cookbook. I also see this:

==> /var/log/chef-server/erchef/erchef.log.1 <==
2014-02-24T01:29:10Z erchef@127.0.0.1 ERR req_id=StLBKjUP42+NHX+GoRa6zQ==;
status=500; method=PUT; path=/sandboxes/000000000000c0ddfbf55f671a4b55be;
user=yoway; msg=[]; req_time=5064; rdbms_time=3; rdbms_count=3; s3_time=5039;
s3_count=1

That s3_time is just over the 5000ms time in the first error, so I’m wondering
if this has something to do with s3, although I’m not using s3 afaik.

Googling these error messages returns discussions about how chef-server
redirects api requests across its components using fqdn’s. Here are a couple
suggestions I found and tried:

1/ set hostname to public dns name, add public dns name to /etc/hosts,
reconfigure chef-server
2/ use ip address instead of fqdn in /etc/chef-server/chef-server.rb
3/ Set s3_url_ttl to 3600 in /etc/chef-server/chef-server.rb

None of these approaches worked for me. Any help is appreciated. Thank you.

We ran into exactly this issue on Friday and just had to chef-server-ctl restart and delete the cranky hashes in the bookshelf folder (not sure if the latter was required).

--Noah

On Feb 23, 2014, at 5:51 PM, yoway@sumologic.com wrote:

I just spun up a Chef Server community AMI on EC2. It's running 11.0.10 and
I've setup a Chef workstation with 11.10.4 along with knife. When I run "knife
cookbook upload" (I tried both --all and a basic one like iptables), I get:

Uploading iptables [0.12.0]
ERROR: internal server error
Response: internal service error

On the server side, I run "sudo chef-server-ctl tail" and I see this:

...
2014-02-24_01:29:10.31425 =ERROR REPORT==== 24-Feb-2014::01:29:10 ===
2014-02-24_01:29:10.31426 Checking presence of checksum:
<<"1f3ac5801386aa2b22e4cbb9877fb431">> for org
<<"00000000000000000000000000000000">> from bucket "bookshelf" has taken longer
than 5000 ms
2014-02-24_01:29:10.31768
2014-02-24_01:29:10.31770 =ERROR REPORT==== 24-Feb-2014::01:29:10 ===
2014-02-24_01:29:10.31771 webmachine error:
path="/sandboxes/000000000000c0ddfbf55f671a4b55be"
2014-02-24_01:29:10.31772 {error,
2014-02-24_01:29:10.31773 {throw,
2014-02-24_01:29:10.31773 {checksum_check_error,13},
2014-02-24_01:29:10.31774
[{chef_wm_named_sandbox,validate_checksums_uploaded,2,
2014-02-24_01:29:10.31775
[{file,"src/chef_wm_named_sandbox.erl"},{line,144}]},
2014-02-24_01:29:10.31776 {chef_wm_named_sandbox,from_json,2,
2014-02-24_01:29:10.31776
[{file,"src/chef_wm_named_sandbox.erl"},{line,99}]},
2014-02-24_01:29:10.31777 {webmachine_resource,resource_call,3,
2014-02-24_01:29:10.31778
[{file,"src/webmachine_resource.erl"},{line,166}]},
2014-02-24_01:29:10.31778 {webmachine_resource,do,3,
2014-02-24_01:29:10.31779
[{file,"src/webmachine_resource.erl"},{line,125}]},
2014-02-24_01:29:10.31780 {webmachine_decision_core,resource_call,1,
2014-02-24_01:29:10.31781
[{file,"src/webmachine_decision_core.erl"},{line,48}]},
2014-02-24_01:29:10.31781 {webmachine_decision_core,accept_helper,0,
2014-02-24_01:29:10.31782
[{file,"src/webmachine_decision_core.erl"},{line,583}]},
2014-02-24_01:29:10.31783 {webmachine_decision_core,decision,1,
2014-02-24_01:29:10.31784
[{file,"src/webmachine_decision_core.erl"},{line,489}]},
2014-02-24_01:29:10.31784 {webmachine_decision_core,handle_request,2,
2014-02-24_01:29:10.31786
[{file,"src/webmachine_decision_core.erl"},{line,33}]}]}}

The ellipsis (...) is actually that first "ERROR REPORT" repeated several
times, I'm guessing for each piece of the cookbook. I also see this:

==> /var/log/chef-server/erchef/erchef.log.1 <==
2014-02-24T01:29:10Z erchef@127.0.0.1 ERR req_id=StLBKjUP42+NHX+GoRa6zQ==;
status=500; method=PUT; path=/sandboxes/000000000000c0ddfbf55f671a4b55be;
user=yoway; msg=; req_time=5064; rdbms_time=3; rdbms_count=3; s3_time=5039;
s3_count=1

That s3_time is just over the 5000ms time in the first error, so I'm wondering
if this has something to do with s3, although I'm not using s3 afaik.

Googling these error messages returns discussions about how chef-server
redirects api requests across its components using fqdn's. Here are a couple
suggestions I found and tried:

1/ set hostname to public dns name, add public dns name to /etc/hosts,
reconfigure chef-server
2/ use ip address instead of fqdn in /etc/chef-server/chef-server.rb
3/ Set s3_url_ttl to 3600 in /etc/chef-server/chef-server.rb

None of these approaches worked for me. Any help is appreciated. Thank you.