FATAL: Net::HTTPServerException: 400 "Bad Request"

Chef Infra (archive)
1

Anyone know how to see the json data that is uploaded from the chef client to the chef server at the end of a client run? I’m having a problem with a few of my new clients and the /var/log/chef-server/erchef/erchef.log.1 file on my server reports a status 400 error and the reason is “error,invalid_json”.

Here is the result of the chef-client run:

[root@espreference-dev-01 cache]# chef-client
Starting Chef Client, version 11.4.4
resolving cookbooks for run list: []
Synchronizing Cookbooks:
Compiling Cookbooks…
[2013-06-06T15:00:07-07:00] WARN: Node espreference-dev-01.mydomain.com has an empty run list.
Converging 0 resources
[2013-06-06T15:00:07-07:00] ERROR: Running exception handlers
[2013-06-06T15:00:07-07:00] ERROR: Report handler Chef::Handler::ErrorReport raised #<ArgumentError: invalid byte sequence in UTF-8>
[2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/1.9.1/json/common.rb:278:in match' [2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/1.9.1/json/common.rb:278:ingenerate’
[2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/1.9.1/json/common.rb:278:in pretty_generate' [2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/json_compat.rb:105:into_json_pretty’
[2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/handler/error_report.rb:27:in report' [2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/handler.rb:226:inrun_report_unsafe’
[2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/handler.rb:214:in run_report_safely' [2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/handler.rb:118:inblock in run_exception_handlers’
[2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/handler.rb:117:in each' [2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/handler.rb:117:inrun_exception_handlers’
[2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/handler.rb:127:in block in <class:Handler>' [2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/client.rb:120:incall’
[2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/client.rb:120:in block in run_failed' [2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/client.rb:119:ineach’
[2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/client.rb:119:in run_failed' [2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/client.rb:485:inrescue in do_run’
[2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/client.rb:494:in do_run' [2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/client.rb:200:inrun’
[2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/application.rb:190:in run_chef_client' [2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/application/client.rb:297:inblock in run_application’
[2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/application/client.rb:290:in loop' [2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/application/client.rb:290:inrun_application’
[2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/application.rb:73:in run' [2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/bin/chef-client:26:in<top (required)>’
[2013-06-06T15:00:07-07:00] ERROR: /usr/bin/chef-client:23:in load' [2013-06-06T15:00:07-07:00] ERROR: /usr/bin/chef-client:23:in'
[2013-06-06T15:00:07-07:00] ERROR: Exception handlers complete
Chef Client failed. 0 resources updated
[2013-06-06T15:00:07-07:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2013-06-06T15:00:07-07:00] FATAL: Net::HTTPServerException: 400 “Bad Request”

Here are the related lines from /var/log/chef-server/erchef/erchef.log.1:

2013-06-06T22:23:24Z erchef@127.0.0.1mailto:erchef@127.0.0.1 INFO req_id=BFvDlbbQ5Do6+PJRwb1MIg==; status=200; method=GET; path=/nodes/espreference-dev-01.mydomain.com; user=espreference-dev-01.mydomain.com; msg=[]; req_time=6; rdbms_time=2; rdbms_count=2
2013-06-06T22:23:24Z erchef@127.0.0.1mailto:erchef@127.0.0.1 INFO req_id=zHqXSayC6813QKc5LpsUkQ==; status=200; method=GET; path=/environments/dev; user=espreference-dev-01.mydomain.com; msg=[]; req_time=5; rdbms_time=2; rdbms_count=2
2013-06-06T22:23:25Z erchef@127.0.0.1mailto:erchef@127.0.0.1 INFO req_id=qNAXFE5gpU8H2Gr/CRBcBg==; status=200; method=POST; path=/environments/dev/cookbook_versions; user=espreference-dev-01.mydomain.com; msg=[]; req_time=6; rdbms_time=3; rdbms_count=3
2013-06-06T22:23:25Z erchef@127.0.0.1mailto:erchef@127.0.0.1 INFO req_id=Z1qWeYDfteJMHN850ssIxA==; status=400; method=PUT; path=/nodes/espreference-dev-01.mydomain.com; user=espreference-dev-01.mydomain.com; msg={error,invalid_json}; req_time=4

Jesse Mauntel
Sr. DevOps Advocate | IT
24 Hour Fitness USA, Inc.
760.918.4554 voice
[cid:image001.png@01CE62CF.3FCEE8C0]

2

For anyone following the thread, one of the guys on my team (thanks Will Albenzi) figured this out. The gecos field for one of our users had an invalid UTF-8 character in it. Once we fixed that, the install completed without issues.

From: Jesse Mauntel [mailto:jmauntel@24hourfit.com]
Sent: Thursday, June 06, 2013 16:03
To: chef@lists.opscode.com
Subject: [chef] FATAL: Net::HTTPServerException: 400 “Bad Request”

Anyone know how to see the json data that is uploaded from the chef client to the chef server at the end of a client run? I’m having a problem with a few of my new clients and the /var/log/chef-server/erchef/erchef.log.1 file on my server reports a status 400 error and the reason is “error,invalid_json”.

Here is the result of the chef-client run:

[root@espreference-dev-01 cache]# chef-client
Starting Chef Client, version 11.4.4
resolving cookbooks for run list: []
Synchronizing Cookbooks:
Compiling Cookbooks…
[2013-06-06T15:00:07-07:00] WARN: Node espreference-dev-01.mydomain.com has an empty run list.
Converging 0 resources
[2013-06-06T15:00:07-07:00] ERROR: Running exception handlers
[2013-06-06T15:00:07-07:00] ERROR: Report handler Chef::Handler::ErrorReport raised #<ArgumentError: invalid byte sequence in UTF-8>
[2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/1.9.1/json/common.rb:278:in match' [2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/1.9.1/json/common.rb:278:ingenerate’
[2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/1.9.1/json/common.rb:278:in pretty_generate' [2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/json_compat.rb:105:into_json_pretty’
[2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/handler/error_report.rb:27:in report' [2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/handler.rb:226:inrun_report_unsafe’
[2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/handler.rb:214:in run_report_safely' [2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/handler.rb:118:inblock in run_exception_handlers’
[2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/handler.rb:117:in each' [2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/handler.rb:117:inrun_exception_handlers’
[2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/handler.rb:127:in block in <class:Handler>' [2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/client.rb:120:incall’
[2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/client.rb:120:in block in run_failed' [2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/client.rb:119:ineach’
[2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/client.rb:119:in run_failed' [2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/client.rb:485:inrescue in do_run’
[2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/client.rb:494:in do_run' [2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/client.rb:200:inrun’
[2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/application.rb:190:in run_chef_client' [2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/application/client.rb:297:inblock in run_application’
[2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/application/client.rb:290:in loop' [2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/application/client.rb:290:inrun_application’
[2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/application.rb:73:in run' [2013-06-06T15:00:07-07:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/bin/chef-client:26:in<top (required)>’
[2013-06-06T15:00:07-07:00] ERROR: /usr/bin/chef-client:23:in load' [2013-06-06T15:00:07-07:00] ERROR: /usr/bin/chef-client:23:in'
[2013-06-06T15:00:07-07:00] ERROR: Exception handlers complete
Chef Client failed. 0 resources updated
[2013-06-06T15:00:07-07:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2013-06-06T15:00:07-07:00] FATAL: Net::HTTPServerException: 400 “Bad Request”

Here are the related lines from /var/log/chef-server/erchef/erchef.log.1:

2013-06-06T22:23:24Z erchef@127.0.0.1mailto:erchef@127.0.0.1 INFO req_id=BFvDlbbQ5Do6+PJRwb1MIg==; status=200; method=GET; path=/nodes/espreference-dev-01.mydomain.com; user=espreference-dev-01.mydomain.com; msg=[]; req_time=6; rdbms_time=2; rdbms_count=2
2013-06-06T22:23:24Z erchef@127.0.0.1mailto:erchef@127.0.0.1 INFO req_id=zHqXSayC6813QKc5LpsUkQ==; status=200; method=GET; path=/environments/dev; user=espreference-dev-01.mydomain.com; msg=[]; req_time=5; rdbms_time=2; rdbms_count=2
2013-06-06T22:23:25Z erchef@127.0.0.1mailto:erchef@127.0.0.1 INFO req_id=qNAXFE5gpU8H2Gr/CRBcBg==; status=200; method=POST; path=/environments/dev/cookbook_versions; user=espreference-dev-01.mydomain.com; msg=[]; req_time=6; rdbms_time=3; rdbms_count=3
2013-06-06T22:23:25Z erchef@127.0.0.1mailto:erchef@127.0.0.1 INFO req_id=Z1qWeYDfteJMHN850ssIxA==; status=400; method=PUT; path=/nodes/espreference-dev-01.mydomain.com; user=espreference-dev-01.mydomain.com; msg={error,invalid_json}; req_time=4

Jesse Mauntel
Sr. DevOps Advocate | IT
24 Hour Fitness USA, Inc.
760.918.4554 voice
[cid:image001.png@01CE6CC4.5AADC310]

3

On Wednesday, June 19, 2013 at 8:09 AM, Jesse Mauntel wrote:

For anyone following the thread, one of the guys on my team (thanks Will Albenzi) figured this out. The gecos field for one of our users had an invalid UTF-8 character in it. Once we fixed that, the install completed without issues.

Chef Client failed. 0 resources updated

[2013-06-06T15:00:07-07:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2013-06-06T15:00:07-07:00] FATAL: Net::HTTPServerException: 400 "Bad Request"

Can you share the contents of /var/chef/cache/chef-stacktrace.out ?

--
Daniel DeLeo

4

Here you go.

Generated at 2013-06-18 14:19:53 -0700
Net::HTTPServerException: 400 “Bad Request”
/opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:2632:in error!' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/rest.rb:199:inblock in raw_http_request’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/rest.rb:280:in retriable_rest_request' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/rest.rb:164:inraw_http_request’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/rest.rb:158:in api_request' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/rest.rb:125:input’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/node.rb:489:in save' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/client.rb:236:insave_updated_node’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/client.rb:471:in do_run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/client.rb:200:inrun’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/application.rb:190:in run_chef_client' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/application/client.rb:297:inblock in run_application’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/application/client.rb:290:in loop' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/application/client.rb:290:inrun_application’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/lib/chef/application.rb:73:in run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.4.4/bin/chef-client:26:in<top (required)>’
/usr/bin/chef-client:23:in load' /usr/bin/chef-client:23:in’

From: Daniel DeLeo [mailto:ddeleo@kallistec.com] On Behalf Of Daniel DeLeo
Sent: Wednesday, June 19, 2013 09:14
To: chef@lists.opscode.com
Subject: [chef] Re: RE: FATAL: Net::HTTPServerException: 400 “Bad Request”

On Wednesday, June 19, 2013 at 8:09 AM, Jesse Mauntel wrote:

For anyone following the thread, one of the guys on my team (thanks Will Albenzi) figured this out. The gecos field for one of our users had an invalid UTF-8 character in it. Once we fixed that, the install completed without issues.

Chef Client failed. 0 resources updated

[2013-06-06T15:00:07-07:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out

[2013-06-06T15:00:07-07:00] FATAL: Net::HTTPServerException: 400 "Bad Request"
Can you share the contents of /var/chef/cache/chef-stacktrace.out ?

–
Daniel DeLeo