400 bad request

Chef Infra (archive)
1

Hey guys,

I've probably bootstrapped around 50 nodes by now and for some reason I've
been starting to get a 400 Bad Request error on all of the nodes i try to
bootstrap. I've also tried to run chef-client on the nodes, but still
getting the same error:

qavapi01.qaorch.com Recipe: cron::chef

qavapi01.qaorch.com * cookbook_file[/root/chef.sh] action create (up to
date)
qavapi01.qaorch.com * cron_d[chef_script] action create (up to date)
qavapi01.qaorch.com Recipe:
qavapi01.qaorch.com * template[/etc/cron.d/chef_script] action create
(up to date)
qavapi01.qaorch.com [2015-08-06T23:30:37-04:00] ERROR: Running exception
handlers
qavapi01.qaorch.com [2015-08-06T23:30:37-04:00] ERROR: Exception handlers
complete
qavapi01.qaorch.com [2015-08-06T23:30:37-04:00] FATAL: Stacktrace dumped
to /var/chef/cache/chef-stacktrace.out
qavapi01.qaorch.com Chef Client failed. 27 resources updated

*qavapi01.qaorch.com http://qavapi01.qaorch.com
[2015-08-06T23:30:37-04:00] ERROR: 400 "Bad Request"*qavapi01.qaorch.com
[2015-08-06T23:30:37-04:00] FATAL: Chef::Exceptions::ChildConvergeError:
Chef run process exited unsuccessfully (exit code 1)

What would be the reason for this?

Here is the stack output from the node:

Generated at 2015-08-06 23:34:11 -0400

Net::HTTPServerException: 400 "Bad Request"/opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:2633:in
error!' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2/lib/chef/http.rb:140:in request'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2/lib/chef/http.rb:113:in
put' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2/lib/chef/node.rb:497:in save'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2/lib/chef/client.rb:262:in
save_updated_node' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2/lib/chef/client.rb:502:in do_run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2/lib/chef/client.rb:199:in
block in run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2/lib/chef/client.rb:193:in fork'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2/lib/chef/client.rb:193:in
run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2/lib/chef/application.rb:208:in run_chef_client'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2/lib/chef/application/client.rb:312:in
block in run_application' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2/lib/chef/application/client.rb:304:in loop'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2/lib/chef/application/client.rb:304:in
run_application' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2/lib/chef/application.rb:66:in run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2/bin/chef-client:26:in
<top (required)>' /usr/bin/chef-client:23:in load'
/usr/bin/chef-client:23:in `'

--
Nikhil Shah • System Administrator
nshah@theorchard.com • p. (+1) 212 308-5648

THE ORCHARD Distribution Done Right
23 E 4th St Fl 3, New York, NY 10003

Follow us: The Daily Rind http://www.dailyrindblog.com/ • Facebook
http://www.facebook.com/theorchard • Twitter
http://www.twitter.com/orchtweets • YouTube
http://www.youtube.com/user/theorchard • LinkedIn
http://www.linkedin.com/company/the-orchard

Privileged And Confidential Communication. This electronic transmission,
and any documents attached hereto, (a) are protected by the Electronic
Communications Privacy Act (18 USC §§ 2510-2521), (b) may contain
confidential and/or legally privileged information, and (c) are for the
sole use of the intended recipient named above. If you have received this
electronic message in error, please notify the sender and delete the
electronic message. Any disclosure, copying, distribution, or use of the
contents of the information received in error is strictly prohibited.

2

On Thursday, August 6, 2015 at 8:36 PM, Nikhil Shah wrote:

Hey guys,

I've probably bootstrapped around 50 nodes by now and for some reason I've been starting to get a 400 Bad Request error on all of the nodes i try to bootstrap. I've also tried to run chef-client on the nodes, but still getting the same error:

qavapi01.qaorch.com (http://qavapi01.qaorch.com) Recipe: cron::chef
qavapi01.qaorch.com (http://qavapi01.qaorch.com) * cookbook_file[/root/chef.sh (http://chef.sh)] action create (up to date)
qavapi01.qaorch.com (http://qavapi01.qaorch.com) * cron_d[chef_script] action create (up to date)
qavapi01.qaorch.com (http://qavapi01.qaorch.com) Recipe:
qavapi01.qaorch.com (http://qavapi01.qaorch.com) * template[/etc/cron.d/chef_script] action create (up to date)
qavapi01.qaorch.com (http://qavapi01.qaorch.com) [2015-08-06T23:30:37-04:00] ERROR: Running exception handlers
qavapi01.qaorch.com (http://qavapi01.qaorch.com) [2015-08-06T23:30:37-04:00] ERROR: Exception handlers complete
qavapi01.qaorch.com (http://qavapi01.qaorch.com) [2015-08-06T23:30:37-04:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
qavapi01.qaorch.com (http://qavapi01.qaorch.com) Chef Client failed. 27 resources updated
qavapi01.qaorch.com (http://qavapi01.qaorch.com) [2015-08-06T23:30:37-04:00] ERROR: 400 "Bad Request"
qavapi01.qaorch.com (http://qavapi01.qaorch.com) [2015-08-06T23:30:37-04:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

What would be the reason for this?

Here is the stack output from the node:

Generated at 2015-08-06 23:34:11 -0400
Net::HTTPServerException: 400 "Bad Request"
/opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:2633:in error!' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2/lib/chef/http.rb:140:in request'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2/lib/chef/http.rb:113:in put' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2/lib/chef/node.rb:497:in save'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2/lib/chef/client.rb:262:in save_updated_node' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2/lib/chef/client.rb:502:in do_run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2/lib/chef/client.rb:199:in block in run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2/lib/chef/client.rb:193:in fork'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2/lib/chef/client.rb:193:in run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2/lib/chef/application.rb:208:in run_chef_client'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2/lib/chef/application/client.rb:312:in block in run_application' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2/lib/chef/application/client.rb:304:in loop'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2/lib/chef/application/client.rb:304:in run_application' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2/lib/chef/application.rb:66:in run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.8.2/bin/chef-client:26:in <top (required)>' /usr/bin/chef-client:23:in load'
/usr/bin/chef-client:23:in `'

This means that your chef-client run is generally working okay, but when Chef attempts to save the node data back to the server, the submitted data is somehow invalid. There are several possibilities but the most common is that there are invalid characters in the node data (JSON data must be valid UTF-8). Several such bugs have been fixed in recent versions of Chef, you should try installing a newer version and see if that resolves the issue. If you’re totally unable to upgrade, you can try pretty printing the node data and looking for any non-ascii characters; you might be able to delete that data via a cookbook or prevent ohai from loading it with config.

--
Nikhil Shah • System Administrator
nshah@theorchard.com (mailto:nshah@theorchard.com) • p. (+1) 212 308-5648

--
Daniel DeLeo

3

This means that your chef-client run is generally working okay, but when
Chef attempts to save the node data back to the server, the submitted data
is somehow invalid. There are several possibilities but the most common is
that there are invalid characters in the node data (JSON data must be valid
UTF-8). Several such bugs have been fixed in recent versions of Chef, you
should try installing a newer version and see if that resolves the issue.
If you’re totally unable to upgrade, you can try pretty printing the node
data and looking for any non-ascii characters; you might be able to delete
that data via a cookbook or prevent ohai from loading it with config.

Hi,

Can you please share more information about pretty printing the node-data?

I am having the same error but on a node that was cloned from an existing
chef-managed node, from which I removed the content under /et/chef/.
Whenever I try to bootstrap it or run chef-client, I get the following
errors:

mpereira@client-machine:~/chef-repo

[master]$ sudo chef-client
Starting Chef Client, version 12.4.1
/opt/chef/embedded/apps/chef/lib/chef/mixin/path_sanity.rb:25: warning:
Insecure world writable dir /nis3/home in PATH, mode 040777
Creating a new client identity for client-machine using the validator key.

================================================================================
Chef encountered an error attempting to create the client "client-machine"

================================================================================
Invalid Request Data:

The data in your request was invalid (HTTP 400).
Server Response:

Since Server API v1, all keys must be updated via the keys endpoint.

Running handlers:
[2015-08-07T15:25:57-04:00] ERROR: Running exception handlers
Running handlers complete
[2015-08-07T15:25:57-04:00] ERROR: Exception handlers complete
Chef Client failed. 0 resources updated in 1.552398 seconds
[2015-08-07T15:25:57-04:00] FATAL: Stacktrace dumped to
/var/chef/cache/chef-stacktrace.out
[2015-08-07T15:25:57-04:00] ERROR: 400 "Bad Request"
[2015-08-07T15:25:57-04:00] FATAL: Chef::Exceptions::ChildConvergeError:
Chef run process exited unsuccessfully (exit code 1)
mpereira@client-machine:~/chef-repo
[master]$ sudo cat /var/chef/cache/chef-stacktrace.out
Generated at 2015-08-07 15:25:57 -0400
Net::HTTPServerException: 400 "Bad Request"
/opt/chef/embedded/lib/ruby/2.1.0/net/http/response.rb:119:in error!' /opt/chef/embedded/apps/chef/lib/chef/http.rb:145:in request'
/opt/chef/embedded/apps/chef/lib/chef/http.rb:118:in put' /opt/chef/embedded/apps/chef/lib/chef/api_client/registration.rb:99:in update'
/opt/chef/embedded/apps/chef/lib/chef/api_client/registration.rb:89:in
rescue in create_or_update' /opt/chef/embedded/apps/chef/lib/chef/api_client/registration.rb:84:in create_or_update'
/opt/chef/embedded/apps/chef/lib/chef/api_client/registration.rb:57:in
run' /opt/chef/embedded/apps/chef/lib/chef/client.rb:608:in register'
/opt/chef/embedded/apps/chef/lib/chef/client.rb:253:in run' /opt/chef/embedded/apps/chef/lib/chef/application.rb:270:in block in
fork_chef_client'
/opt/chef/embedded/apps/chef/lib/chef/application.rb:258:in fork' /opt/chef/embedded/apps/chef/lib/chef/application.rb:258:in fork_chef_client'
/opt/chef/embedded/apps/chef/lib/chef/application.rb:224:in block in run_chef_client' /opt/chef/embedded/apps/chef/lib/chef/local_mode.rb:39:in with_server_connectivity'
/opt/chef/embedded/apps/chef/lib/chef/application.rb:212:in
run_chef_client' /opt/chef/embedded/apps/chef/lib/chef/application/client.rb:402:in block
in interval_run_chef_client'
/opt/chef/embedded/apps/chef/lib/chef/application/client.rb:392:in loop' /opt/chef/embedded/apps/chef/lib/chef/application/client.rb:392:in interval_run_chef_client'
/opt/chef/embedded/apps/chef/lib/chef/application/client.rb:382:in
run_application' /opt/chef/embedded/apps/chef/lib/chef/application.rb:60:in run'
/opt/chef/embedded/apps/chef/bin/chef-client:26:in <top (required)>' /usr/bin/chef-client:55:in load'
/usr/bin/chef-client:55:in `'

The chef server version is 12.1.0-1 and the client-machine
has chef-12.4.1-1.el5

Thanks for your help

4

Please disregard my first question, I should have looked better in the
docs. I need to parse the output to extract the relevant information to
this problem.

5

On Monday, August 10, 2015 at 6:10 AM, Michael Pereira wrote:

Hi,

Can you please share more information about pretty printing the node-data?

I am having the same error but on a node that was cloned from an existing chef-managed node, from which I removed the content under /et/chef/. Whenever I try to bootstrap it or run chef-client, I get the following errors:
I wrote this in another email but didn’t send it to the list (doh!): Your error has the same/similar symptom as Nikhil, but the underlying cause is different.

mpereira@client-machine:~/chef-repo
[master]$ sudo chef-client
Starting Chef Client, version 12.4.1
/opt/chef/embedded/apps/chef/lib/chef/mixin/path_sanity.rb:25: warning: Insecure world writable dir /nis3/home in PATH, mode 040777
Creating a new client identity for client-machine using the validator key.

Chef encountered an error attempting to create the client "client-machine"

Invalid Request Data:

The data in your request was invalid (HTTP 400).
Server Response:

Since Server API v1, all keys must be updated via the keys endpoint.

Running handlers:
[2015-08-07T15:25:57-04:00] ERROR: Running exception handlers
Running handlers complete
[2015-08-07T15:25:57-04:00] ERROR: Exception handlers complete
Chef Client failed. 0 resources updated in 1.552398 seconds
[2015-08-07T15:25:57-04:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2015-08-07T15:25:57-04:00] ERROR: 400 "Bad Request"
[2015-08-07T15:25:57-04:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
mpereira@client-machine:~/chef-repo
[master]$ sudo cat /var/chef/cache/chef-stacktrace.out
Generated at 2015-08-07 15:25:57 -0400
Net::HTTPServerException: 400 "Bad Request"
/opt/chef/embedded/lib/ruby/2.1.0/net/http/response.rb:119:in error!' /opt/chef/embedded/apps/chef/lib/chef/http.rb:145:in request'
/opt/chef/embedded/apps/chef/lib/chef/http.rb:118:in put' /opt/chef/embedded/apps/chef/lib/chef/api_client/registration.rb:99:in update'
/opt/chef/embedded/apps/chef/lib/chef/api_client/registration.rb:89:in rescue in create_or_update' /opt/chef/embedded/apps/chef/lib/chef/api_client/registration.rb:84:in create_or_update'
/opt/chef/embedded/apps/chef/lib/chef/api_client/registration.rb:57:in run' /opt/chef/embedded/apps/chef/lib/chef/client.rb:608:in register'
/opt/chef/embedded/apps/chef/lib/chef/client.rb:253:in `run'

From this stack trace, it looks like you’re re-using a client/node name that already exists on the server. Some recent changes to the API have made this raise a different error than it used to, but it wouldn’t have worked before. The best way to deal with this is to delete the "client-machine” client and node on the server before you bootstrap a new one.

We will look at making a change to have this produce a clearer error (I think the error message was much better in the past).

The chef server version is 12.1.0-1 and the client-machine has chef-12.4.1-1.el5

Thanks for your help

--
Michael Pereira
(201) 855-9607

HTH,

--
Daniel DeLeo

6

From this stack trace, it looks like you’re re-using a client/node name
that already exists on the server. Some recent changes to the API have made
this raise a different error than it used to, but it wouldn’t have worked
before. The best way to deal with this is to delete the "client-machine”
client and node on the server before you bootstrap a new one.

We will look at making a change to have this produce a clearer error (I
think the error message was much better in the past).

You're right, good catch from the minimal stack trace! I managed to find
the correct error in the long debug output :

[2015-08-10T13:51:53-04:00] DEBUG: ---- HTTP Response Body ----

[2015-08-10T13:51:53-04:00] DEBUG: {"error":["Client already exists"]}
[2015-08-10T13:51:53-04:00] DEBUG: ---- End HTTP Response Body -----

Now the interesting part is that this node does not show up in the knife
node list command output. I removed the chef-client rpm and delete the
content under /etc/chef/ before trying to use the cloned machine with chef
but I'm not sure if it is not picking up the original machine private key
from somewhere else and sending it to the server.

Regards,
Michael

7

On Monday, August 10, 2015 at 11:20 AM, Michael Pereira wrote:

From this stack trace, it looks like you’re re-using a client/node name that already exists on the server. Some recent changes to the API have made this raise a different error than it used to, but it wouldn’t have worked before. The best way to deal with this is to delete the "client-machine” client and node on the server before you bootstrap a new one.

We will look at making a change to have this produce a clearer error (I think the error message was much better in the past).
You're right, good catch from the minimal stack trace! I managed to find the correct error in the long debug output :

[2015-08-10T13:51:53-04:00] DEBUG: ---- HTTP Response Body ----
[2015-08-10T13:51:53-04:00] DEBUG: {"error":["Client already exists"]}
[2015-08-10T13:51:53-04:00] DEBUG: ---- End HTTP Response Body -----

Now the interesting part is that this node does not show up in the knife node list command output. I removed the chef-client rpm and delete the content under /etc/chef/ before trying to use the cloned machine with chef but I'm not sure if it is not picking up the original machine private key from somewhere else and sending it to the server.
Each machine you manage generally has two corresponding objects on the server. One is the node, which as you know is the data about that machine’s state and so on. The other part is the client, which is more or less an “API key” object (kind of like a user account, but doesn’t have a password and can only be a part of one organization). When you remove an old node, you should delete both of these things (e.g., knife client delete client-machine).

Regards,
Michael

HTH,

--
Daniel DeLeo