Chef-client issues: trying to connect to localhost

Chef Infra (archive)
1

The gist is here

Node:

Rackspace Cloud Server

Ubuntu 10.04 LTS

Hosted Chef Server

Package install 10.04

http://wiki.opscode.com/display/chef/Package+Installation+on+Debian+and+Ubuntu

Using nginx for ssl

http://warwickp.com/2010/10/hosting-chef-server-behind-nginx-proxy

Bootstrap works with no problems

$ knife bootstrap XXX.XXX.XXX.XXX -x root -P XXXXX -d ubuntu10.04-apt -l debug

New node shows up

$ knife node list
[
“local-workstation.local”,
“web-app”
]

$ knife node run_list add web-app ‘role[base]’
{
“run_list”: [
“role[base]”
]
}

Confirm run list

knife node show web-app -r
{
“run_list”: [
“role[base]”
]
}

And the fuckary begins

Doesn’t work

web-app $ chef-client -l debug

It shits out this error

retriable_rest_request: Connection refused - Connection refused connecting to localhost:4000 for /cookbooks/runit/0.14.1/files/ac87b692389b0d9d99e628a9f25fba19, giving up (Errno::ECONNREFUSED)

I have no clue why the chef-run is trying to connect to localhost:4000 instead of my chef server.

I examined the debug output from the chef run. And I found this…

X-Ops-UserId:web-app’
Header hash: {“X-Ops-Content-Hash”=>“2jmj7l5rSw0yVb/vlWAYkK/YBwk=”, “X-Ops-Authorization-1”=>“4wnsQ+…”, “X-Ops-Userid”=>“web-app”, “X-Ops-Sign”=>“version=1.0”, “X-Ops-Authorization-2”=>“nfg…”, “X-Ops-Authorization-3”=>“cfHD…”, “X-Ops-Authorization-4”=>“DxbX…”, “X-Ops-Authorization-5”=>“xie…”, “X-Ops-Authorization-6”=>“lGdE…”, “X-Ops-Timestamp”=>“2011-01-08T18:39:41Z”}
[Sat, 08 Jan 2011 18:39:41 +0000] DEBUG: Sending HTTP Request via GET to chef.my-chef-server.com:443/nodes/web-app/cookbooks
[Sat, 08 Jan 2011 18:39:42 +0000] DEBUG: Cookbooks to load: {“runit”=>#<Chef::CookbookVersion:0x7fe831bc8b78 @file_vendor=nil, @couchdb_rev=nil, @definition_filenames=[“runit_service.rb”], @manifest_records_by_path={“templates/gentoo/runit-start.sh.erb”=>{“name”=>“runit-start.sh.erb”, “url”=>“http://localhost:4000/cookbooks/runit/0.14.1/files/bd03533917da171167995131c075ea43”, “checksum”=>“bd03533917da171167995131c075ea43”, “path”=>“templates/gentoo/runit-start.sh.erb”, “specificity”=>“gentoo”}…

I’m clueless to why these cookbooks have “url”=>“http://localhost:4000/…” instead of my chef server url

/etc/chef/client.rb contains the correct chef server url

web-app $ chef-client -c /etc/chef/client.rb

Fails for the same localhost reasons

THIS WORKS

web-app $ chef-client -S http://chef.my-chef-server.com:4000

The logs show the cookbooks have “url”=>“http://chef.my-chef-server.com:4000” instead of localhost

Nginx on my chef server is an apt-get install

And my chef server web ui works

ANY HALP WUD BE GREATLY APPRECIATD

2

Most likely you haven't configured your proxy to set the correct
HTTP_X_FORWARDED_HOST header.

On Sat, Jan 8, 2011 at 4:29 PM, Kirk Richey kirkosis@gmail.com wrote:

The gist is here
chef-troubles.txt · GitHub

Node:

Rackspace Cloud Server

Ubuntu 10.04 LTS

Hosted Chef Server

Package install 10.04

http://wiki.opscode.com/display/chef/Package+Installation+on+Debian+and+Ubuntu

Using nginx for ssl

http://warwickp.com/2010/10/hosting-chef-server-behind-nginx-proxy

Bootstrap works with no problems

$ knife bootstrap XXX.XXX.XXX.XXX -x root -P XXXXX -d ubuntu10.04-apt -l debug

New node shows up

$ knife node list
[
"local-workstation.local",
"web-app"
]

$ knife node run_list add web-app 'role[base]'
{
"run_list": [
"role[base]"
]
}

Confirm run list

knife node show web-app -r
{
"run_list": [
"role[base]"
]
}

And the fuckary begins

Doesn't work

web-app $ chef-client -l debug

It shits out this error

retriable_rest_request: Connection refused - Connection refused connecting to localhost:4000 for /cookbooks/runit/0.14.1/files/ac87b692389b0d9d99e628a9f25fba19, giving up (Errno::ECONNREFUSED)

I have no clue why the chef-run is trying to connect to localhost:4000 instead of my chef server.

I examined the debug output from the chef run. And I found this...

X-Ops-UserId:web-app'
Header hash: {"X-Ops-Content-Hash"=>"2jmj7l5rSw0yVb/vlWAYkK/YBwk=", "X-Ops-Authorization-1"=>"4wnsQ+...", "X-Ops-Userid"=>"web-app", "X-Ops-Sign"=>"version=1.0", "X-Ops-Authorization-2"=>"nfg...", "X-Ops-Authorization-3"=>"cfHD...", "X-Ops-Authorization-4"=>"DxbX...", "X-Ops-Authorization-5"=>"xie...", "X-Ops-Authorization-6"=>"lGdE...", "X-Ops-Timestamp"=>"2011-01-08T18:39:41Z"}
[Sat, 08 Jan 2011 18:39:41 +0000] DEBUG: Sending HTTP Request via GET to chef.my-chef-server.com:443/nodes/web-app/cookbooks
[Sat, 08 Jan 2011 18:39:42 +0000] DEBUG: Cookbooks to load: {"runit"=>#<Chef::CookbookVersion:0x7fe831bc8b78 @file_vendor=nil, @couchdb_rev=nil, @definition_filenames=["runit_service.rb"], @manifest_records_by_path={"templates/gentoo/runit-start.sh.erb"=>{"name"=>"runit-start.sh.erb", "url"=>"http://localhost:4000/cookbooks/runit/0.14.1/files/bd03533917da171167995131c075ea43", "checksum"=>"bd03533917da171167995131c075ea43", "path"=>"templates/gentoo/runit-start.sh.erb", "specificity"=>"gentoo"}....

I'm clueless to why these cookbooks have "url"=>"http://localhost:4000/..." instead of my chef server url

/etc/chef/client.rb contains the correct chef server url

web-app $ chef-client -c /etc/chef/client.rb

Fails for the same localhost reasons

THIS WORKS

web-app $ chef-client -S http://chef.my-chef-server.com:4000

The logs show the cookbooks have "url"=>"http://chef.my-chef-server.com:4000" instead of localhost

Nginx on my chef server is an apt-get install

And my chef server web ui works

ANY HALP WUD BE GREATLY APPRECIATD

3

On Jan 8, 2011, at 8:03 PM, Daniel DeLeo wrote:

Most likely you haven't configured your proxy to set the correct
HTTP_X_FORWARDED_HOST header.

How would I do that?

Here is the gist to my nginx conf

/etc/nginx/nginx.conf

worker_processes 1;

error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;

events {
worker_connections 1024;

multi_accept on;

}

http {
include /etc/nginx/mime.types;

access_log /var/log/nginx/access.log;

sendfile on;
#tcp_nopush on;

#keepalive_timeout 0;
keepalive_timeout 65;
tcp_nodelay on;

gzip on;
gzip_disable "MSIE [1-6].(?!.*SV1)";

#local chef-server
upstream chef_api_local {
server localhost:4000;
}

#local chef webui
upstream chef_webui_local {
server localhost:4040;
}

include /etc/nginx/conf.d/.conf;
include /etc/nginx/sites-enabled/;
}

/etc/nginx/sites-enabled/chef.my-chef-server.com

server {

server_name chef.my-chef-server.com;
ssl on;
ssl_certificate /etc/ssl/certs/myssl.crt;
ssl_certificate_key /etc/ssl/private/myssl.key;
listen 443;

root /var/www;

location / {
  #API request incoming
  if ( $http_x_ops_timestamp != "") {
     proxy_pass http://chef_api_local;
     break;
  }
     # webui request incoming
      proxy_pass http://chef_webui_local;	

}

}

On Sat, Jan 8, 2011 at 4:29 PM, Kirk Richey kirkosis@gmail.com wrote:

The gist is here
chef-troubles.txt · GitHub

Node:

Rackspace Cloud Server

Ubuntu 10.04 LTS

Hosted Chef Server

Package install 10.04

http://wiki.opscode.com/display/chef/Package+Installation+on+Debian+and+Ubuntu

Using nginx for ssl

http://warwickp.com/2010/10/hosting-chef-server-behind-nginx-proxy

Bootstrap works with no problems

$ knife bootstrap XXX.XXX.XXX.XXX -x root -P XXXXX -d ubuntu10.04-apt -l debug

New node shows up

$ knife node list
[
"local-workstation.local",
"web-app"
]

$ knife node run_list add web-app 'role[base]'
{
"run_list": [
"role[base]"
]
}

Confirm run list

knife node show web-app -r
{
"run_list": [
"role[base]"
]
}

And the fuckary begins

Doesn't work

web-app $ chef-client -l debug

It shits out this error

retriable_rest_request: Connection refused - Connection refused connecting to localhost:4000 for /cookbooks/runit/0.14.1/files/ac87b692389b0d9d99e628a9f25fba19, giving up (Errno::ECONNREFUSED)

I have no clue why the chef-run is trying to connect to localhost:4000 instead of my chef server.

I examined the debug output from the chef run. And I found this...

X-Ops-UserId:web-app'
Header hash: {"X-Ops-Content-Hash"=>"2jmj7l5rSw0yVb/vlWAYkK/YBwk=", "X-Ops-Authorization-1"=>"4wnsQ+...", "X-Ops-Userid"=>"web-app", "X-Ops-Sign"=>"version=1.0", "X-Ops-Authorization-2"=>"nfg...", "X-Ops-Authorization-3"=>"cfHD...", "X-Ops-Authorization-4"=>"DxbX...", "X-Ops-Authorization-5"=>"xie...", "X-Ops-Authorization-6"=>"lGdE...", "X-Ops-Timestamp"=>"2011-01-08T18:39:41Z"}
[Sat, 08 Jan 2011 18:39:41 +0000] DEBUG: Sending HTTP Request via GET to chef.my-chef-server.com:443/nodes/web-app/cookbooks
[Sat, 08 Jan 2011 18:39:42 +0000] DEBUG: Cookbooks to load: {"runit"=>#<Chef::CookbookVersion:0x7fe831bc8b78 @file_vendor=nil, @couchdb_rev=nil, @definition_filenames=["runit_service.rb"], @manifest_records_by_path={"templates/gentoo/runit-start.sh.erb"=>{"name"=>"runit-start.sh.erb", "url"=>"http://localhost:4000/cookbooks/runit/0.14.1/files/bd03533917da171167995131c075ea43", "checksum"=>"bd03533917da171167995131c075ea43", "path"=>"templates/gentoo/runit-start.sh.erb", "specificity"=>"gentoo"}....

I'm clueless to why these cookbooks have "url"=>"http://localhost:4000/..." instead of my chef server url

/etc/chef/client.rb contains the correct chef server url

web-app $ chef-client -c /etc/chef/client.rb

Fails for the same localhost reasons

THIS WORKS

web-app $ chef-client -S http://chef.my-chef-server.com:4000

The logs show the cookbooks have "url"=>"http://chef.my-chef-server.com:4000" instead of localhost

Nginx on my chef server is an apt-get install

And my chef server web ui works

ANY HALP WUD BE GREATLY APPRECIATD

4

On Sat, Jan 8, 2011 at 5:22 PM, Kirk Richey kirkosis@gmail.com wrote:

On Jan 8, 2011, at 8:03 PM, Daniel DeLeo wrote:

Most likely you haven't configured your proxy to set the correct
HTTP_X_FORWARDED_HOST header.

How would I do that?

Actually, the either the Host header or X-Forwarded-Host header will
work, though the X-Forwarded-Host header will be preferred if both are
set. We have the following in our server { block:

proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto http;

But that is for an internal LB, so you probably want to set
X-Forwarded-Proto to https. The Chef server needs these headers to
know what the absolute URLs of the requests it responds to are so that
it can correctly generate an absolute URL--otherwise it can't know the
hostname or protocol to use, as you've experienced.

HTH,
Dan DeLeo

5

That worked like a charm.

Knuckle Bumps

On Jan 8, 2011, at 8:43 PM, Daniel DeLeo wrote:

On Sat, Jan 8, 2011 at 5:22 PM, Kirk Richey kirkosis@gmail.com wrote:

On Jan 8, 2011, at 8:03 PM, Daniel DeLeo wrote:

Most likely you haven't configured your proxy to set the correct
HTTP_X_FORWARDED_HOST header.

How would I do that?

Actually, the either the Host header or X-Forwarded-Host header will
work, though the X-Forwarded-Host header will be preferred if both are
set. We have the following in our server { block:

proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto http;

But that is for an internal LB, so you probably want to set
X-Forwarded-Proto to https. The Chef server needs these headers to
know what the absolute URLs of the requests it responds to are so that
it can correctly generate an absolute URL--otherwise it can't know the
hostname or protocol to use, as you've experienced.

HTH,
Dan DeLeo