Client getting 404 on chef server URI /en-US/nodes


#1

hi. this is wierd. today i brought a few ec2 clients out of a stopped
state and attempted to put them thru their chef-client run, as usual.
suddenly they are trying to access a URI on the chef server that doesn’t
exist: /en-US/* … thus they’re failing with 404.

all my other nodes use this same chef server, same chef-client version
(chef-full-0.10.10-1, CentOS 5.8), and are not exhibiting this behavior
currently.

here’s a bit of log from one of the failing clients. you can see the last time
chef ran back in August, it was accessing the correct URI. upon its first run
today, it tried the right URI, then suddenly, inexplicably tried prepending
that URI with “/en-US”.

10/26 23:38[root@test-mobile-prod-ue1 chef]# grep “via GET to chef.prod.gogiants.com” /var/log/chef/client.log | tail -6
[Wed, 08 Aug 2012 20:25:18 +0000] DEBUG: Sending HTTP Request via GET to chef.prod.gogiants.com:443/data/secrets/creds-prod
[Wed, 08 Aug 2012 20:25:18 +0000] DEBUG: Sending HTTP Request via GET to chef.prod.gogiants.com:443/data/yum/yum-tasks
[Fri, 26 Oct 2012 23:31:46 +0000] DEBUG: Sending HTTP Request via GET to chef.prod.gogiants.com:443/nodes/test-mobile.prod.gogiants.com
[Fri, 26 Oct 2012 23:31:46 +0000] DEBUG: Sending HTTP Request via GET to chef.prod.gogiants.com:443/en-US/nodes/test-mobile.prod.gogiants.com
[Fri, 26 Oct 2012 23:31:48 +0000] DEBUG: Sending HTTP Request via GET to chef.prod.gogiants.com:443/en-US/nodes
[Fri, 26 Oct 2012 23:35:12 +0000] DEBUG: Sending HTTP Request via GET to chef.prod.gogiants.com:443/en-US/clients

does anyone know what’s going on here? what to look for?

more complete log output is in a gist:

thanks!
kallen


#2

On Friday, October 26, 2012 at 5:01 PM, kallen@groknaut.net wrote:

hi. this is wierd. today i brought a few ec2 clients out of a stopped
state and attempted to put them thru their chef-client run, as usual.
suddenly they are trying to access a URI on the chef server that doesn’t
exist: /en-US/* … thus they’re failing with 404.

all my other nodes use this same chef server, same chef-client version
(chef-full-0.10.10-1, CentOS 5.8), and are not exhibiting this behavior
currently.

here’s a bit of log from one of the failing clients. you can see the last time
chef ran back in August, it was accessing the correct URI. upon its first run
today, it tried the right URI, then suddenly, inexplicably tried prepending
that URI with “/en-US”.

10/26 23:38[root@test-mobile-prod-ue1 chef]# grep “via GET to chef.prod.gogiants.com (http://chef.prod.gogiants.com)” /var/log/chef/client.log | tail -6
[Wed, 08 Aug 2012 20:25:18 +0000] DEBUG: Sending HTTP Request via GET to chef.prod.gogiants.com:443/data/secrets/creds-prod (http://chef.prod.gogiants.com:443/data/secrets/creds-prod)
[Wed, 08 Aug 2012 20:25:18 +0000] DEBUG: Sending HTTP Request via GET to chef.prod.gogiants.com:443/data/yum/yum-tasks (http://chef.prod.gogiants.com:443/data/yum/yum-tasks)
[Fri, 26 Oct 2012 23:31:46 +0000] DEBUG: Sending HTTP Request via GET to chef.prod.gogiants.com:443/nodes/test-mobile.prod.gogiants.com (http://chef.prod.gogiants.com:443/nodes/test-mobile.prod.gogiants.com)
[Fri, 26 Oct 2012 23:31:46 +0000] DEBUG: Sending HTTP Request via GET to chef.prod.gogiants.com:443/en-US/nodes/test-mobile.prod.gogiants.com (http://chef.prod.gogiants.com:443/en-US/nodes/test-mobile.prod.gogiants.com)
[Fri, 26 Oct 2012 23:31:48 +0000] DEBUG: Sending HTTP Request via GET to chef.prod.gogiants.com:443/en-US/nodes (http://chef.prod.gogiants.com:443/en-US/nodes)
[Fri, 26 Oct 2012 23:35:12 +0000] DEBUG: Sending HTTP Request via GET to chef.prod.gogiants.com:443/en-US/clients (http://chef.prod.gogiants.com:443/en-US/clients)

does anyone know what’s going on here? what to look for?

more complete log output is in a gist:
https://gist.github.com/3962319

thanks!
kallen

Chef is following a redirect, maybe check your load balancer config? Also there’s a bit more info in the debug output from http responses in newer versions of chef if you need more information and upgrading is an option.


Daniel DeLeo


#3

On Mon, 29 Oct 2012, Daniel DeLeo wrote:

On Friday, October 26, 2012 at 5:01 PM, kallen@groknaut.net wrote:

does anyone know what’s going on here? what to look for?

more complete log output is in a gist:
https://gist.github.com/3962319

Chef is following a redirect, maybe check your load balancer config? Also there’s a bit more info in the debug output from http responses in newer versions of chef if you need more information and upgrading is an option.

thanks for the response.

i found it. it wasn’t a redirect in my reverse proxy, but a
misconfig that had the clients trying to talk to the wrong server,
which happened to be listening on 443, providing that redirect.

yay