Ohai Chefs!
After a recent DNS snafu, it seems that all 6 of my chef clients stopped
communicating with my open source chef server. In an effort to correct
this, I deleted the clients and nodes in the Chef server Web UI with the
intention of creating them again.
So I went into my chef-repo (kept in git) and the subdirectory I created
for the host I was attempting to recreate. And deleted the client.rb and
validation.pem. Then I ran:
knife configure client ./ and got two fresh files. Which were moved into
/etc/chef on that host. Knife (fortunately) was working just fine
throughout.
But when I went to run the chef-client command again I started seeing these
errors:
[root@logs:~/chef-repo/logs] #chef-client
[2013-12-18T23:31:13-05:00] INFO: Forking chef instance to converge…
Starting Chef Client, version 11.6.2
[2013-12-18T23:31:13-05:00] INFO: *** Chef 11.6.2 ***
Creating a new client identity for logs.jokefire.com using the validator
key.
[2013-12-18T23:31:14-05:00] INFO: Client key /etc/chef/client.pem is not
present - registering
[2013-12-18T23:31:19-05:00] INFO: HTTP Request Returned 500 Internal Server
Error: internal service error
[2013-12-18T23:31:19-05:00] ERROR: Server returned error for
https://chef.jokefire.com/clients, retrying 1/5 in 4s
[2013-12-18T23:31:29-05:00] INFO: HTTP Request Returned 500 Internal Server
Error: internal service error
[2013-12-18T23:31:29-05:00] ERROR: Server returned error for
https://chef.jokefire.com/clients, retrying 2/5 in 8s
[2013-12-18T23:31:42-05:00] ERROR: Server returned error for
https://chef.jokefire.com/clients, retrying 3/5 in 11s
[2013-12-18T23:31:59-05:00] INFO: HTTP Request Returned 500 Internal Server
Error: internal service error
[2013-12-18T23:31:59-05:00] ERROR: Server returned error for
https://chef.jokefire.com/clients, retrying 4/5 in 19s
[2013-12-18T23:32:21-05:00] ERROR: Server returned error for
https://chef.jokefire.com/clients, retrying 5/5 in 51s
[2013-12-18T23:33:17-05:00] INFO: HTTP Request Returned 500 Internal Server
Error: internal service error
[2013-12-18T23:33:17-05:00] WARN: Failed to register new client, 4 tries
remaining
[2013-12-18T23:33:17-05:00] WARN: Response: HTTP 500 - 500 “Internal Server
Error”
[2013-12-18T23:33:23-05:00] INFO: HTTP Request Returned 500 Internal Server
Error: internal service error
[2013-12-18T23:33:23-05:00] ERROR: Server returned error for
https://chef.jokefire.com/clients, retrying 1/5 in 3s
[2013-12-18T23:33:31-05:00] INFO: HTTP Request Returned 500 Internal Server
Error: internal service error
[2013-12-18T23:33:31-05:00] ERROR: Server returned error for
https://chef.jokefire.com/clients, retrying 2/5 in 7s
[2013-12-18T23:33:41-05:00] ERROR: Server returned error for
https://chef.jokefire.com/clients, retrying 3/5 in 11s
[2013-12-18T23:33:58-05:00] INFO: HTTP Request Returned 500 Internal Server
Error: internal service error
[2013-12-18T23:33:58-05:00] ERROR: Server returned error for
https://chef.jokefire.com/clients, retrying 4/5 in 18s
[2013-12-18T23:34:20-05:00] ERROR: Server returned error for
https://chef.jokefire.com/clients, retrying 5/5 in 55s
[2013-12-18T23:35:21-05:00] INFO: HTTP Request Returned 500 Internal Server
Error: internal service error
[2013-12-18T23:35:21-05:00] WARN: Failed to register new client, 3 tries
remaining
[2013-12-18T23:35:21-05:00] WARN: Response: HTTP 500 - 500 “Internal Server
Error”
[2013-12-18T23:35:26-05:00] INFO: HTTP Request Returned 500 Internal Server
Error: internal service error
[2013-12-18T23:35:26-05:00] ERROR: Server returned error for
https://chef.jokefire.com/clients, retrying 1/5 in 4s
[2013-12-18T23:35:35-05:00] INFO: HTTP Request Returned 500 Internal Server
Error: internal service error
[2013-12-18T23:35:35-05:00] ERROR: Server returned error for
https://chef.jokefire.com/clients, retrying 2/5 in 7s
[2013-12-18T23:35:43-05:00] ERROR: Server returned error for
https://chef.jokefire.com/clients, retrying 3/5 in 12s
I was just hoping for a little guidance on how to get this show back on the
road.
Thanks!
Tim
–
GPG me!!
gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
On Wednesday, December 18, 2013 at 8:36 PM, Tim Dunphy wrote:
Ohai Chefs!
After a recent DNS snafu, it seems that all 6 of my chef clients stopped communicating with my open source chef server. In an effort to correct this, I deleted the clients and nodes in the Chef server Web UI with the intention of creating them again.
So I went into my chef-repo (kept in git) and the subdirectory I created for the host I was attempting to recreate. And deleted the client.rb and validation.pem. Then I ran:
knife configure client ./ and got two fresh files. Which were moved into /etc/chef on that host. Knife (fortunately) was working just fine throughout.
But when I went to run the chef-client command again I started seeing these errors:
[root@logs:~/chef-repo/logs] #chef-client
[2013-12-18T23:31:13-05:00] INFO: Forking chef instance to converge...
Starting Chef Client, version 11.6.2
[2013-12-18T23:31:13-05:00] INFO: *** Chef 11.6.2 ***
Creating a new client identity for logs.jokefire.com (http://logs.jokefire.com) using the validator key.
[2013-12-18T23:31:14-05:00] INFO: Client key /etc/chef/client.pem is not present - registering
[2013-12-18T23:31:19-05:00] INFO: HTTP Request Returned 500 Internal Server Error: internal service error
[2013-12-18T23:31:19-05:00] ERROR: Server returned error for https://chef.jokefire.com/clients, retrying 1/5 in 4s
[2013-12-18T23:31:29-05:00] INFO: HTTP Request Returned 500 Internal Server Error: internal service error
[2013-12-18T23:31:29-05:00] ERROR: Server returned error for https://chef.jokefire.com/clients, retrying 2/5 in 8s
[2013-12-18T23:31:42-05:00] ERROR: Server returned error for https://chef.jokefire.com/clients, retrying 3/5 in 11s
[2013-12-18T23:31:59-05:00] INFO: HTTP Request Returned 500 Internal Server Error: internal service error
[2013-12-18T23:31:59-05:00] ERROR: Server returned error for https://chef.jokefire.com/clients, retrying 4/5 in 19s
[2013-12-18T23:32:21-05:00] ERROR: Server returned error for https://chef.jokefire.com/clients, retrying 5/5 in 51s
[2013-12-18T23:33:17-05:00] INFO: HTTP Request Returned 500 Internal Server Error: internal service error
[2013-12-18T23:33:17-05:00] WARN: Failed to register new client, 4 tries remaining
[2013-12-18T23:33:17-05:00] WARN: Response: HTTP 500 - 500 "Internal Server Error"
[2013-12-18T23:33:23-05:00] INFO: HTTP Request Returned 500 Internal Server Error: internal service error
[2013-12-18T23:33:23-05:00] ERROR: Server returned error for https://chef.jokefire.com/clients, retrying 1/5 in 3s
[2013-12-18T23:33:31-05:00] INFO: HTTP Request Returned 500 Internal Server Error: internal service error
[2013-12-18T23:33:31-05:00] ERROR: Server returned error for https://chef.jokefire.com/clients, retrying 2/5 in 7s
[2013-12-18T23:33:41-05:00] ERROR: Server returned error for https://chef.jokefire.com/clients, retrying 3/5 in 11s
[2013-12-18T23:33:58-05:00] INFO: HTTP Request Returned 500 Internal Server Error: internal service error
[2013-12-18T23:33:58-05:00] ERROR: Server returned error for https://chef.jokefire.com/clients, retrying 4/5 in 18s
[2013-12-18T23:34:20-05:00] ERROR: Server returned error for https://chef.jokefire.com/clients, retrying 5/5 in 55s
[2013-12-18T23:35:21-05:00] INFO: HTTP Request Returned 500 Internal Server Error: internal service error
[2013-12-18T23:35:21-05:00] WARN: Failed to register new client, 3 tries remaining
[2013-12-18T23:35:21-05:00] WARN: Response: HTTP 500 - 500 "Internal Server Error"
[2013-12-18T23:35:26-05:00] INFO: HTTP Request Returned 500 Internal Server Error: internal service error
[2013-12-18T23:35:26-05:00] ERROR: Server returned error for https://chef.jokefire.com/clients, retrying 1/5 in 4s
[2013-12-18T23:35:35-05:00] INFO: HTTP Request Returned 500 Internal Server Error: internal service error
[2013-12-18T23:35:35-05:00] ERROR: Server returned error for https://chef.jokefire.com/clients, retrying 2/5 in 7s
[2013-12-18T23:35:43-05:00] ERROR: Server returned error for https://chef.jokefire.com/clients, retrying 3/5 in 12s
I was just hoping for a little guidance on how to get this show back on the road.
Thanks!
Tim
You need to look at the server to see why it’s returning 500s. chef-server-ctl tail will show you all the logs.
--
Daniel DeLeo