Chef-client doesn't work "automatically"

I’ve been having some trouble with my chef-clients, I’ve my chef-server
working properly, and my chef-clients can work just fine (with 0.7). I can
login, and execute chef-client as root and it will work wonderfully.

But after that it just doesn’t work. In the log you see this every 30mins:

[Wed, 17 Jun 2009 09:15:09 +0200] INFO: Starting Chef Run
[Wed, 17 Jun 2009 09:15:09 +0200] ERROR: Net::HTTPFatalError
[Wed, 17 Jun 2009 09:15:09 +0200] FATAL: 502 “Bad Gateway”
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/1.8/net/http.rb:2097:in
error!' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:232:inrun_request’
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:230:in
run_request' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:95:inpost_rest’
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/client.rb:223:in
authenticate' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/client.rb:74:inrun’
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:163:in
run_application' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:161:inloop’
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:161:in
run_application' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application.rb:53:inrun’
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/bin/chef-client:26
/usr/bin/chef-client:19:in `load’
/usr/bin/chef-client:19
[Wed, 17 Jun 2009 09:15:09 +0200] FATAL: Sleeping for 1815 seconds before
trying again

Then I log in and execute it manually, and this is the log:

[Wed, 17 Jun 2009 09:38:36 +0200] INFO: Starting Chef Run
[Wed, 17 Jun 2009 09:38:42 +0200] INFO: Updating
template[/home/kom/.ssh/authorized_keys2] at /home/kom/.ssh/authorized_keys2

[Wed, 17 Jun 2009 09:38:53 +0200] INFO: Backing up
template[/etc/nginx/sites-available/comerc] to
/etc/nginx/sites-available/comerc.chef-20090617093853
[Wed, 17 Jun 2009 09:38:53 +0200] INFO: Chef Run complete in 17.132551
seconds

Any ideas why might this be happening? I see the chef-client is running as
root:

ps aux | grep chef-client

root 26512 0.0 0.0 112 28 ? Ss Jun09 0:00 runsv
chef-client
root 408 0.0 6.5 36616 24112 ? S Jun12 0:14
/opt/ruby-enterprise-1.8.6-20090610/bin/ruby /usr/bin/chef-client -i 1800 -s
20 -L /var/log/chef/client.log

And nothing else I can think about.

Been thinking of using a cron job, but I don’t really see the point when the
service is running properly everywhere.

Cheers!


Albert Llop
Tech Department
albert@kingsofmambo.com

On 17/06/2009, at 7:41 PM, Albert Llop wrote:

I've been having some trouble with my chef-clients, I've my chef-
server working properly, and my chef-clients can work just fine
(with 0.7). I can login, and execute chef-client as root and it will
work wonderfully.

But after that it just doesn't work. In the log you see this every
30mins:

[Wed, 17 Jun 2009 09:15:09 +0200] INFO: Starting Chef Run
[Wed, 17 Jun 2009 09:15:09 +0200] ERROR: Net::HTTPFatalError
[Wed, 17 Jun 2009 09:15:09 +0200] FATAL: 502 "Bad Gateway"
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/1.8/net/http.rb:2097:in
error!' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/rest.rb:232:in run_request'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/lib/chef/rest.rb:230:in run_request' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/rest.rb:95:in post_rest'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/lib/chef/client.rb:223:in authenticate' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/client.rb:74:in run'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/lib/chef/application/client.rb:163:in run_application' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/application/client.rb:161:in loop'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/lib/chef/application/client.rb:161:in run_application' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/application.rb:53:in run'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/bin/chef-client:26
/usr/bin/chef-client:19:in `load'
/usr/bin/chef-client:19
[Wed, 17 Jun 2009 09:15:09 +0200] FATAL: Sleeping for 1815 seconds
before trying again

Could you possibly set your log_level to debug on both the client and
the server, and drop another email back with the increased log
verbosity?

--
AJ Christensen, Software Engineer
Opscode, Inc.
E: aj@opscode.com

Hi again,
I set the log level to debug on one of the nodes, and restarted the client.
After the restart it all worked fine for 14hours (until midnight more or
less), and then stopped working.

The client log looks very similar with the debug option:

[Thu, 18 Jun 2009 00:33:28 +0200] INFO: Starting Chef Run
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Registering nouvel_kingsofmambo_net
for an openid
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Sending HTTP Request via GET to
/registrations/nouvel_kingsofmambo_net
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Authenticating
nouvel_kingsofmambo_net via openid
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Sending HTTP Request via POST to
/openid/consumer/start
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Sending HTTP Request via GET to
/openid/server?openid.assoc_handle=%7BHMAC-SHA1%7D%7B4a2f838d%7D%7Bhe1kgg%3D%3D%7D&openid.claimed_id=https%3A%2F%
2Fchef.kingsofmambo.net
%2Fopenid%2Fserver%2Fnode%2Fnouvel_kingsofmambo_net&openid.identity=https%3A%2F%
2Fchef.kingsofmambo.net
%2Fopenid%2Fserver%2Fnode%2Fnouvel_kingsofmambo_net&openid.mode=checkid_setup&openid.ns=http%3A%2F%
2Fspecs.openid.net%2Fauth%2F2.0&openid.realm=https%3A%2F%
2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer&openid.return_to=https%3A%2F%
2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer%2Fcomplete
[Thu, 18 Jun 2009 00:33:29 +0200] ERROR: Net::HTTPFatalError
[Thu, 18 Jun 2009 00:33:29 +0200] FATAL: 502 "Bad Gateway"
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/1.8/net/http.rb:2097:in
error!' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:232:in run_request'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:230:in
run_request' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:95:in post_rest'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/client.rb:223:in
authenticate' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/client.rb:74:in run'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:163:in
run_application' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:161:in loop'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:161:in
run_application' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application.rb:53:in run'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/bin/chef-client:26
/usr/bin/chef-client:19:in `load'
/usr/bin/chef-client:19
[Thu, 18 Jun 2009 00:33:29 +0200] FATAL: Sleeping for 1817 seconds before
trying again

On the server side:

~ Started request handling: Thu Jun 18 00:33:28 +0200 2009
~ Params: {"format"=>nil, "action"=>"show",
"id"=>"nouvel_kingsofmambo_net",
"controller"=>"chef_server_slice/openid_register"}
~ {:before_filters_time=>0.0, :action_time=>0.0, :dispatch_time=>0.004,
:after_filters_time=>0.0}
~

~ Started request handling: Thu Jun 18 00:33:28 +0200 2009
~ Params: {"action"=>"node_page", "id"=>"nouvel_kingsofmambo_net",
"controller"=>"chef_server_slice/openid_server"}
~ {:before_filters_time=>0.0, :action_time=>0.004, :dispatch_time=>0.004,
:after_filters_time=>0.0}
~

~ Started request handling: Thu Jun 18 00:33:28 +0200 2009
~ Params: {"submit"=>"Verify", "action"=>"start",
"controller"=>"chef_server_slice/openid_consumer", "openid_identifier"=>"
https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net"}
~ WARNING: making https request to
https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_netwithout
verifying server certificate; no CA path was specified.
~ Generated checkid_setup request to
https://chef.kingsofmambo.net/openid/server with assocication
{HMAC-SHA1}{4a2f838d}{he1kgg==}
~ Generated checkid_setup request to
https://chef.kingsofmambo.net/openid/server with assocication
{HMAC-SHA1}{4a2f838d}{he1kgg==}
~ Redirecting to:
https://chef.kingsofmambo.net/openid/server?openid.assoc_handle={HMAC-SHA1}{4a2f838d}{he1kgg%3D%3D}&openid.claimed_id=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode%2Fnouvel_kingsofmambo_net&openid.identity=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode%2Fnouvel_kingsofmambo_net&openid.mode=checkid_setup&openid.ns=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0&openid.realm=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer&openid.return_to=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer%2Fcomplete(302)
~ {:before_filters_time=>0.0, :action_time=>0.024001,
:dispatch_time=>0.024001, :after_filters_time=>0.0}
~

~ Started request handling: Thu Jun 18 00:33:29 +0200 2009
~ Params: {"openid.claimed_id"=>"
https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net",
"openid.mode"=>"checkid_setup", "openid.return_to"=>"
https://chef.kingsofmambo.net/openid/consumer/complete", "openid.ns"=>"
http://specs.openid.net/auth/2.0", "action"=>"index",
"controller"=>"chef_server_slice/openid_server", "openid.identity"=>"
https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net",
"openid.assoc_handle"=>"{HMAC-SHA1}{4a2f838d}{he1kgg==}", "openid.realm"=>"
https://chef.kingsofmambo.net/openid/consumer"}
~ {:before_filters_time=>0.0, :action_time=>0.004, :dispatch_time=>0.004,
:after_filters_time=>0.0}
~

Thanks beforehand!

--
Albert Llop
Tech Department
albert@kingsofmambo.com

2009/6/17 Arjuna Christensen aj@opscode.com

On 17/06/2009, at 7:41 PM, Albert Llop wrote:

I've been having some trouble with my chef-clients, I've my chef-server
working properly, and my chef-clients can work just fine (with 0.7). I can
login, and execute chef-client as root and it will work wonderfully.

But after that it just doesn't work. In the log you see this every 30mins:

[Wed, 17 Jun 2009 09:15:09 +0200] INFO: Starting Chef Run
[Wed, 17 Jun 2009 09:15:09 +0200] ERROR: Net::HTTPFatalError
[Wed, 17 Jun 2009 09:15:09 +0200] FATAL: 502 "Bad Gateway"
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/1.8/net/http.rb:2097:in
error!' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:232:in run_request'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:230:in
run_request' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:95:in post_rest'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/client.rb:223:in
authenticate' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/client.rb:74:in run'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:163:in
run_application' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:161:in loop'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:161:in
run_application' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application.rb:53:in run'

/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/bin/chef-client:26
/usr/bin/chef-client:19:in `load'
/usr/bin/chef-client:19
[Wed, 17 Jun 2009 09:15:09 +0200] FATAL: Sleeping for 1815 seconds before
trying again

Could you possibly set your log_level to debug on both the client and the
server, and drop another email back with the increased log verbosity?

--
AJ Christensen, Software Engineer
Opscode, Inc.
E: aj@opscode.com adam@opscode.com

Hey!

On 18/06/2009, at 7:53 PM, Albert Llop wrote:

Hi again,
I set the log level to debug on one of the nodes, and restarted the
client. After the restart it all worked fine for 14hours (until
midnight more or less), and then stopped working.

The client log looks very similar with the debug option:

[Thu, 18 Jun 2009 00:33:28 +0200] INFO: Starting Chef Run
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Registering
nouvel_kingsofmambo_net for an openid
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Sending HTTP Request via
GET to /registrations/nouvel_kingsofmambo_net
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Authenticating
nouvel_kingsofmambo_net via openid
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Sending HTTP Request via
POST to /openid/consumer/start
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Sending HTTP Request via
GET to /openid/server?openid.assoc_handle=%7BHMAC-SHA1%7D%7B4a2f838d
%7D%7Bhe1kgg%3D%3D%7D&openid.claimed_id=https%3A%2F
%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode
%2Fnouvel_kingsofmambo_net&openid.identity=https%3A%2F
%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode
%2Fnouvel_kingsofmambo_net&openid.mode=checkid_setup&openid.ns=http
%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0&openid.realm=https%3A%2F
%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer&openid.return_to=https
%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer%2Fcomplete
[Thu, 18 Jun 2009 00:33:29 +0200] ERROR: Net::HTTPFatalError
[Thu, 18 Jun 2009 00:33:29 +0200] FATAL: 502 "Bad Gateway"
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/1.8/net/http.rb:2097:in
error!' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/rest.rb:232:in run_request'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/lib/chef/rest.rb:230:in run_request' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/rest.rb:95:in post_rest'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/lib/chef/client.rb:223:in authenticate' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/client.rb:74:in run'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/lib/chef/application/client.rb:163:in run_application' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/application/client.rb:161:in loop'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/lib/chef/application/client.rb:161:in run_application' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/application.rb:53:in run'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/bin/chef-client:26
/usr/bin/chef-client:19:in `load'
/usr/bin/chef-client:19
[Thu, 18 Jun 2009 00:33:29 +0200] FATAL: Sleeping for 1817 seconds
before trying again

Doesn't seem to be anything indicative on the server side, so I'd
suggest a network or temporal issue such as DNS. :\

On the server side:

~ Started request handling: Thu Jun 18 00:33:28 +0200 2009
~ Params: {"format"=>nil, "action"=>"show",
"id"=>"nouvel_kingsofmambo_net", "controller"=>"chef_server_slice/
openid_register"}
~
{:before_filters_time

0.0
, :action_time=>0.0, :dispatch_time=>0.004, :after_filters_time=>0.0}
~

~ Started request handling: Thu Jun 18 00:33:28 +0200 2009
~ Params: {"action"=>"node_page", "id"=>"nouvel_kingsofmambo_net",
"controller"=>"chef_server_slice/openid_server"}
~
{:before_filters_time

0.0
, :action_time
=>0.004, :dispatch_time=>0.004, :after_filters_time=>0.0}
~

~ Started request handling: Thu Jun 18 00:33:28 +0200 2009
~ Params: {"submit"=>"Verify", "action"=>"start",
"controller"=>"chef_server_slice/openid_consumer",
"openid_identifier"=>"https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net
"}
~ WARNING: making https request to https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net
without verifying server certificate; no CA path was specified.
~ Generated checkid_setup request to https://chef.kingsofmambo.net/openid/server
with assocication {HMAC-SHA1}{4a2f838d}{he1kgg==}
~ Generated checkid_setup request to https://chef.kingsofmambo.net/openid/server
with assocication {HMAC-SHA1}{4a2f838d}{he1kgg==}
~ Redirecting to: https://chef.kingsofmambo.net/openid/server?openid.assoc_handle={HMAC-SHA1}{4a2f838d}{he1kgg%3D%3D}&openid.claimed_id=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode%2Fnouvel_kingsofmambo_net&openid.identity=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode%2Fnouvel_kingsofmambo_net&openid.mode=checkid_setup&openid.ns=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0&openid.realm=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer&openid.return_to=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer%2Fcomplete
(302)
~
{:before_filters_time

0.0
, :action_time
=>0.024001, :dispatch_time=>0.024001, :after_filters_time=>0.0}
~

~ Started request handling: Thu Jun 18 00:33:29 +0200 2009
~ Params: {"openid.claimed_id"=>"https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net
", "openid.mode"=>"checkid_setup", "openid.return_to"=>"https://chef.kingsofmambo.net/openid/consumer/complete
", "openid.ns"=>"http://specs.openid.net/auth/2.0",
"action"=>"index", "controller"=>"chef_server_slice/openid_server",
"openid.identity"=>"https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net
", "openid.assoc_handle"=>"{HMAC-SHA1}{4a2f838d}{he1kgg==}",
"openid.realm"=>"https://chef.kingsofmambo.net/openid/consumer"}
~
{:before_filters_time

0.0
, :action_time
=>0.004, :dispatch_time=>0.004, :after_filters_time=>0.0}
~

Thanks beforehand!

--
Albert Llop
Tech Department
albert@kingsofmambo.com

2009/6/17 Arjuna Christensen aj@opscode.com

On 17/06/2009, at 7:41 PM, Albert Llop wrote:

I've been having some trouble with my chef-clients, I've my chef-
server working properly, and my chef-clients can work just fine
(with 0.7). I can login, and execute chef-client as root and it
will work wonderfully.

But after that it just doesn't work. In the log you see this every
30mins:

[Wed, 17 Jun 2009 09:15:09 +0200] INFO: Starting Chef Run
[Wed, 17 Jun 2009 09:15:09 +0200] ERROR: Net::HTTPFatalError
[Wed, 17 Jun 2009 09:15:09 +0200] FATAL: 502 "Bad Gateway"
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/1.8/net/http.rb:
2097:in error!' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/rest.rb:232:in run_request'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/lib/chef/rest.rb:230:in run_request' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/rest.rb:95:in post_rest'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/lib/chef/client.rb:223:in authenticate' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/client.rb:74:in run'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/lib/chef/application/client.rb:163:in run_application' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/application/client.rb:161:in loop'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/lib/chef/application/client.rb:161:in run_application' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/application.rb:53:in run'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/bin/chef-client:26
/usr/bin/chef-client:19:in `load'
/usr/bin/chef-client:19
[Wed, 17 Jun 2009 09:15:09 +0200] FATAL: Sleeping for 1815 seconds
before trying again

Could you possibly set your log_level to debug on both the client
and the server, and drop another email back with the increased log
verbosity?

--
AJ Christensen, Software Engineer
Opscode, Inc.
E: aj@opscode.com

--
AJ Christensen, Software Engineer
Opscode, Inc.
E: aj@opscode.com

After some sleep I decided that the "502 "Bad Gateway"" error looks too much
like nginx, and I checked its erorr log (why didn't i do it before?

Whenever one of these nodes that try to contact the server and is in "zombie
mode" like i like to call them (because chef-client is working... but not
properly), nginx gives an error:

2009/06/19 11:11:38 [error] 27044#0: *40 upstream sent too big header while
reading response header from upstream
, client: 67.23.26.105, server:
chef.kingsofmambo.net, request: "GET
/openid/server?openid.assoc_handle=%7BHMAC-SHA1%7D%7B4a2f838d%7D%7Bhe1kgg%3D%3D%7D&openid.claimed_id=https%3A%2F%
2Fchef.kingsofmambo.net
%2Fopenid%2Fserver%2Fnode%2Fpotter_kingsofmambo_net&openid.identity=https%3A%2F%
2Fchef.kingsofmambo.net
%2Fopenid%2Fserver%2Fnode%2Fpotter_kingsofmambo_net&openid.mode=checkid_setup&openid.ns=http%3A%2F%
2Fspecs.openid.net%2Fauth%2F2.0&openid.realm=https%3A%2F%
2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer&openid.return_to=https%3A%2F%
2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer%2Fcomplete HTTP/1.1", upstream:
"unix:/tmp/passenger.27019/master/helper_server.sock:", host: "
chef.kingsofmambo.net"

The only information I can find googling is about problems with fastcgi,
that I don't think have anything to do with chef (and passenger in general).

Any suggestions?

thanks again

Albert Llop
Tech Department
albert@kingsofmambo.com

2009/6/18 Arjuna Christensen aj@opscode.com

Hey!
On 18/06/2009, at 7:53 PM, Albert Llop wrote:

Hi again,
I set the log level to debug on one of the nodes, and restarted the client.
After the restart it all worked fine for 14hours (until midnight more or
less), and then stopped working.

The client log looks very similar with the debug option:

[Thu, 18 Jun 2009 00:33:28 +0200] INFO: Starting Chef Run
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Registering
nouvel_kingsofmambo_net for an openid
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Sending HTTP Request via GET to
/registrations/nouvel_kingsofmambo_net
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Authenticating
nouvel_kingsofmambo_net via openid
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Sending HTTP Request via POST to
/openid/consumer/start
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Sending HTTP Request via GET to
/openid/server?openid.assoc_handle=%7BHMAC-SHA1%7D%7B4a2f838d%7D%7Bhe1kgg%3D%3D%7D&openid.claimed_id=https%3A%2F%
2Fchef.kingsofmambo.net
%2Fopenid%2Fserver%2Fnode%2Fnouvel_kingsofmambo_net&openid.identity=https%3A%2F%
2Fchef.kingsofmambo.net
%2Fopenid%2Fserver%2Fnode%2Fnouvel_kingsofmambo_net&openid.mode=checkid_setup&openid.ns=http%3A%2F%
2Fspecs.openid.net%2Fauth%2F2.0&openid.realm=https%3A%2F%
2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer&openid.return_to=https%3A%2F%
2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer%2Fcomplete
[Thu, 18 Jun 2009 00:33:29 +0200] ERROR: Net::HTTPFatalError
[Thu, 18 Jun 2009 00:33:29 +0200] FATAL: 502 "Bad Gateway"
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/1.8/net/http.rb:2097:in
error!' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:232:in run_request'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:230:in
run_request' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:95:in post_rest'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/client.rb:223:in
authenticate' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/client.rb:74:in run'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:163:in
run_application' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:161:in loop'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:161:in
run_application' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application.rb:53:in run'

/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/bin/chef-client:26
/usr/bin/chef-client:19:in `load'
/usr/bin/chef-client:19
[Thu, 18 Jun 2009 00:33:29 +0200] FATAL: Sleeping for 1817 seconds before
trying again

Doesn't seem to be anything indicative on the server side, so I'd suggest a
network or temporal issue such as DNS. :\

On the server side:

~ Started request handling: Thu Jun 18 00:33:28 +0200 2009
~ Params: {"format"=>nil, "action"=>"show",
"id"=>"nouvel_kingsofmambo_net",
"controller"=>"chef_server_slice/openid_register"}
~ {:before_filters_time=>0.0, :action_time=>0.0, :dispatch_time=>0.004,
:after_filters_time=>0.0}
~

~ Started request handling: Thu Jun 18 00:33:28 +0200 2009
~ Params: {"action"=>"node_page", "id"=>"nouvel_kingsofmambo_net",
"controller"=>"chef_server_slice/openid_server"}
~ {:before_filters_time=>0.0, :action_time=>0.004, :dispatch_time=>0.004,
:after_filters_time=>0.0}
~

~ Started request handling: Thu Jun 18 00:33:28 +0200 2009
~ Params: {"submit"=>"Verify", "action"=>"start",
"controller"=>"chef_server_slice/openid_consumer", "openid_identifier"=>"
https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net"}
~ WARNING: making https request to
https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_netwithout verifying server certificate; no CA path was specified.
~ Generated checkid_setup request to
https://chef.kingsofmambo.net/openid/server with assocication
{HMAC-SHA1}{4a2f838d}{he1kgg==}
~ Generated checkid_setup request to
https://chef.kingsofmambo.net/openid/server with assocication
{HMAC-SHA1}{4a2f838d}{he1kgg==}
~ Redirecting to:
https://chef.kingsofmambo.net/openid/server?openid.assoc_handle={HMAC-SHA1}{4a2f838d}{he1kgg%3D%3D}&openid.claimed_id=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode%2Fnouvel_kingsofmambo_net&openid.identity=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode%2Fnouvel_kingsofmambo_net&openid.mode=checkid_setup&openid.ns=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0&openid.realm=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer&openid.return_to=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer%2Fcomplete(302)
~ {:before_filters_time=>0.0, :action_time=>0.024001,
:dispatch_time=>0.024001, :after_filters_time=>0.0}
~

~ Started request handling: Thu Jun 18 00:33:29 +0200 2009
~ Params: {"openid.claimed_id"=>"
https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net",
"openid.mode"=>"checkid_setup", "openid.return_to"=>"
https://chef.kingsofmambo.net/openid/consumer/complete", "openid.ns"=>"
http://specs.openid.net/auth/2.0", "action"=>"index",
"controller"=>"chef_server_slice/openid_server", "openid.identity"=>"
https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net",
"openid.assoc_handle"=>"{HMAC-SHA1}{4a2f838d}{he1kgg==}", "openid.realm"=>"
https://chef.kingsofmambo.net/openid/consumer"}
~ {:before_filters_time=>0.0, :action_time=>0.004, :dispatch_time=>0.004,
:after_filters_time=>0.0}
~

Thanks beforehand!

--
Albert Llop
Tech Department
albert@kingsofmambo.com

2009/6/17 Arjuna Christensen aj@opscode.com

On 17/06/2009, at 7:41 PM, Albert Llop wrote:

I've been having some trouble with my chef-clients, I've my chef-server
working properly, and my chef-clients can work just fine (with 0.7). I can
login, and execute chef-client as root and it will work wonderfully.

But after that it just doesn't work. In the log you see this every 30mins:

[Wed, 17 Jun 2009 09:15:09 +0200] INFO: Starting Chef Run
[Wed, 17 Jun 2009 09:15:09 +0200] ERROR: Net::HTTPFatalError
[Wed, 17 Jun 2009 09:15:09 +0200] FATAL: 502 "Bad Gateway"
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/1.8/net/http.rb:2097:in
error!' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:232:in run_request'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:230:in
run_request' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:95:in post_rest'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/client.rb:223:in
authenticate' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/client.rb:74:in run'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:163:in
run_application' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:161:in loop'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:161:in
run_application' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application.rb:53:in run'

/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/bin/chef-client:26
/usr/bin/chef-client:19:in `load'
/usr/bin/chef-client:19
[Wed, 17 Jun 2009 09:15:09 +0200] FATAL: Sleeping for 1815 seconds before
trying again

Could you possibly set your log_level to debug on both the client and the
server, and drop another email back with the increased log verbosity?

--
AJ Christensen, Software Engineer
Opscode, Inc.
E: aj@opscode.com adam@opscode.com

--
AJ Christensen, Software Engineer
Opscode, Inc.
E: aj@opscode.com adam@opscode.com

On 19/06/2009, at 9:17 PM, Albert Llop wrote:

After some sleep I decided that the "502 "Bad Gateway"" error looks
too much like nginx, and I checked its erorr log (why didn't i do it
before?

Whenever one of these nodes that try to contact the server and is in
"zombie mode" like i like to call them (because chef-client is
working... but not properly), nginx gives an error:

2009/06/19 11:11:38 [error] 27044#0: *40 upstream sent too big
header while reading response header from upstream, client:
67.23.26.105, server: chef.kingsofmambo.net, request: "GET /openid/
server?openid.assoc_handle=%7BHMAC-SHA1%7D%7B4a2f838d%7D%7Bhe1kgg%3D
%3D%7D&openid.claimed_id=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid
%2Fserver%2Fnode%2Fpotter_kingsofmambo_net&openid.identity=https%3A
%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode
%2Fpotter_kingsofmambo_net&openid.mode=checkid_setup&openid.ns=http
%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0&openid.realm=https%3A%2F
%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer&openid.return_to=https
%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer%2Fcomplete HTTP/
1.1", upstream: "unix:/tmp/passenger.27019/master/
helper_server.sock:", host: "chef.kingsofmambo.net"

Very interesting. I wasn't aware you were running underneath another
container (although this should definitely work correctly.)

We're currently experiencing some issues with chef clients receiving
cookies bigger than 4kB from the Chef Server, which may be related to
this. I wish Nginx would tell us what the header was that was too big.

I'm opening a ticket, and I'll scrub down the openid stuff while
running under Nginx Passenger and see if I can get it to fold.

Thanks for the additional information!

2009/6/18 Arjuna Christensen aj@opscode.com
Hey!

On 18/06/2009, at 7:53 PM, Albert Llop wrote:

Hi again,
I set the log level to debug on one of the nodes, and restarted the
client. After the restart it all worked fine for 14hours (until
midnight more or less), and then stopped working.

The client log looks very similar with the debug option:

[Thu, 18 Jun 2009 00:33:28 +0200] INFO: Starting Chef Run
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Registering
nouvel_kingsofmambo_net for an openid
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Sending HTTP Request via
GET to /registrations/nouvel_kingsofmambo_net
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Authenticating
nouvel_kingsofmambo_net via openid
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Sending HTTP Request via
POST to /openid/consumer/start
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Sending HTTP Request via
GET to /openid/server?openid.assoc_handle=%7BHMAC-SHA1%7D%7B4a2f838d
%7D%7Bhe1kgg%3D%3D%7D&openid.claimed_id=https%3A%2F
%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode
%2Fnouvel_kingsofmambo_net&openid.identity=https%3A%2F
%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode
%2Fnouvel_kingsofmambo_net&openid.mode=checkid_setup&openid.ns=http
%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0&openid.realm=https%3A%2F
%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer&openid.return_to=https
%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer%2Fcomplete
[Thu, 18 Jun 2009 00:33:29 +0200] ERROR: Net::HTTPFatalError
[Thu, 18 Jun 2009 00:33:29 +0200] FATAL: 502 "Bad Gateway"
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/1.8/net/http.rb:
2097:in error!' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/rest.rb:232:in run_request'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/lib/chef/rest.rb:230:in run_request' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/rest.rb:95:in post_rest'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/lib/chef/client.rb:223:in authenticate' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/client.rb:74:in run'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/lib/chef/application/client.rb:163:in run_application' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/application/client.rb:161:in loop'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/lib/chef/application/client.rb:161:in run_application' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/application.rb:53:in run'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/bin/chef-client:26
/usr/bin/chef-client:19:in `load'
/usr/bin/chef-client:19
[Thu, 18 Jun 2009 00:33:29 +0200] FATAL: Sleeping for 1817 seconds
before trying again

Doesn't seem to be anything indicative on the server side, so I'd
suggest a network or temporal issue such as DNS. :\

On the server side:

~ Started request handling: Thu Jun 18 00:33:28 +0200 2009
~ Params: {"format"=>nil, "action"=>"show",
"id"=>"nouvel_kingsofmambo_net", "controller"=>"chef_server_slice/
openid_register"}
~
{:before_filters_time

0.0
, :action_time=>0.0, :dispatch_time=>0.004, :after_filters_time=>0.0}
~

~ Started request handling: Thu Jun 18 00:33:28 +0200 2009
~ Params: {"action"=>"node_page", "id"=>"nouvel_kingsofmambo_net",
"controller"=>"chef_server_slice/openid_server"}
~
{:before_filters_time

0.0
, :action_time
=>0.004, :dispatch_time=>0.004, :after_filters_time=>0.0}
~

~ Started request handling: Thu Jun 18 00:33:28 +0200 2009
~ Params: {"submit"=>"Verify", "action"=>"start",
"controller"=>"chef_server_slice/openid_consumer",
"openid_identifier"=>"https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net
"}
~ WARNING: making https request to https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net
without verifying server certificate; no CA path was specified.
~ Generated checkid_setup request to https://chef.kingsofmambo.net/openid/server
with assocication {HMAC-SHA1}{4a2f838d}{he1kgg==}
~ Generated checkid_setup request to https://chef.kingsofmambo.net/openid/server
with assocication {HMAC-SHA1}{4a2f838d}{he1kgg==}
~ Redirecting to: https://chef.kingsofmambo.net/openid/server?openid.assoc_handle={HMAC-SHA1}{4a2f838d}{he1kgg%3D%3D}&openid.claimed_id=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode%2Fnouvel_kingsofmambo_net&openid.identity=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode%2Fnouvel_kingsofmambo_net&openid.mode=checkid_setup&openid.ns=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0&openid.realm=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer&openid.return_to=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer%2Fcomplete
(302)
~
{:before_filters_time

0.0
, :action_time
=>0.024001, :dispatch_time=>0.024001, :after_filters_time=>0.0}
~

~ Started request handling: Thu Jun 18 00:33:29 +0200 2009
~ Params: {"openid.claimed_id"=>"https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net
", "openid.mode"=>"checkid_setup", "openid.return_to"=>"https://chef.kingsofmambo.net/openid/consumer/complete
", "openid.ns"=>"http://specs.openid.net/auth/2.0",
"action"=>"index", "controller"=>"chef_server_slice/openid_server",
"openid.identity"=>"https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net
", "openid.assoc_handle"=>"{HMAC-SHA1}{4a2f838d}{he1kgg==}",
"openid.realm"=>"https://chef.kingsofmambo.net/openid/consumer"}
~
{:before_filters_time

0.0
, :action_time
=>0.004, :dispatch_time=>0.004, :after_filters_time=>0.0}
~

Thanks beforehand!

--
Albert Llop
Tech Department
albert@kingsofmambo.com

2009/6/17 Arjuna Christensen aj@opscode.com

On 17/06/2009, at 7:41 PM, Albert Llop wrote:

I've been having some trouble with my chef-clients, I've my chef-
server working properly, and my chef-clients can work just fine
(with 0.7). I can login, and execute chef-client as root and it
will work wonderfully.

But after that it just doesn't work. In the log you see this every
30mins:

[Wed, 17 Jun 2009 09:15:09 +0200] INFO: Starting Chef Run
[Wed, 17 Jun 2009 09:15:09 +0200] ERROR: Net::HTTPFatalError
[Wed, 17 Jun 2009 09:15:09 +0200] FATAL: 502 "Bad Gateway"
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/1.8/net/http.rb:
2097:in error!' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/rest.rb:232:in run_request'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/lib/chef/rest.rb:230:in run_request' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/rest.rb:95:in post_rest'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/lib/chef/client.rb:223:in authenticate' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/client.rb:74:in run'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/lib/chef/application/client.rb:163:in run_application' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/application/client.rb:161:in loop'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/lib/chef/application/client.rb:161:in run_application' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/application.rb:53:in run'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/bin/chef-client:26
/usr/bin/chef-client:19:in `load'
/usr/bin/chef-client:19
[Wed, 17 Jun 2009 09:15:09 +0200] FATAL: Sleeping for 1815 seconds
before trying again

Could you possibly set your log_level to debug on both the client
and the server, and drop another email back with the increased log
verbosity?

--
AJ Christensen, Software Engineer
Opscode, Inc.
E: aj@opscode.com

--
AJ Christensen, Software Engineer
Opscode, Inc.
E: aj@opscode.com

--
AJ Christensen, Software Engineer
Opscode, Inc.
E: aj@opscode.com

Thanks a lot, know that bit of info about the big cookies might be useful!

And yeah, it's been working under nginx for some time, obtained all the info
I needed for that from this repo on github

:slight_smile:

Albert Llop
Tech Department
albert@kingsofmambo.com

2009/6/19 Arjuna Christensen aj@opscode.com

On 19/06/2009, at 9:17 PM, Albert Llop wrote:

After some sleep I decided that the "502 "Bad Gateway"" error looks too
much like nginx, and I checked its erorr log (why didn't i do it before?

Whenever one of these nodes that try to contact the server and is in
"zombie mode" like i like to call them (because chef-client is working...
but not properly), nginx gives an error:

2009/06/19 11:11:38 [error] 27044#0: *40 upstream sent too big header
while reading response header from upstream
, client: 67.23.26.105,
server: chef.kingsofmambo.net, request: "GET
/openid/server?openid.assoc_handle=%7BHMAC-SHA1%7D%7B4a2f838d%7D%7Bhe1kgg%3D%3D%7D&openid.claimed_id=https%3A%2F%
2Fchef.kingsofmambo.net
%2Fopenid%2Fserver%2Fnode%2Fpotter_kingsofmambo_net&openid.identity=https%3A%2F%
2Fchef.kingsofmambo.net
%2Fopenid%2Fserver%2Fnode%2Fpotter_kingsofmambo_net&openid.mode=checkid_setup&openid.ns=http%3A%2F%
2Fspecs.openid.net%2Fauth%2F2.0&openid.realm=https%3A%2F%
2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer&openid.return_to=https%3A%2F%
2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer%2Fcomplete HTTP/1.1",
upstream: "unix:/tmp/passenger.27019/master/helper_server.sock:", host: "
chef.kingsofmambo.net"

Very interesting. I wasn't aware you were running underneath another
container (although this should definitely work correctly.)

We're currently experiencing some issues with chef clients receiving
cookies bigger than 4kB from the Chef Server, which may be related to this.
I wish Nginx would tell us what the header was that was too big.

I'm opening a ticket, and I'll scrub down the openid stuff while running
under Nginx Passenger and see if I can get it to fold.

Thanks for the additional information!

2009/6/18 Arjuna Christensen aj@opscode.com

Hey!
On 18/06/2009, at 7:53 PM, Albert Llop wrote:

Hi again,
I set the log level to debug on one of the nodes, and restarted the
client. After the restart it all worked fine for 14hours (until midnight
more or less), and then stopped working.

The client log looks very similar with the debug option:

[Thu, 18 Jun 2009 00:33:28 +0200] INFO: Starting Chef Run
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Registering
nouvel_kingsofmambo_net for an openid
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Sending HTTP Request via GET to
/registrations/nouvel_kingsofmambo_net
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Authenticating
nouvel_kingsofmambo_net via openid
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Sending HTTP Request via POST to
/openid/consumer/start
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Sending HTTP Request via GET to
/openid/server?openid.assoc_handle=%7BHMAC-SHA1%7D%7B4a2f838d%7D%7Bhe1kgg%3D%3D%7D&openid.claimed_id=https%3A%2F%
2Fchef.kingsofmambo.net
%2Fopenid%2Fserver%2Fnode%2Fnouvel_kingsofmambo_net&openid.identity=https%3A%2F%
2Fchef.kingsofmambo.net
%2Fopenid%2Fserver%2Fnode%2Fnouvel_kingsofmambo_net&openid.mode=checkid_setup&openid.ns=http%3A%2F%
2Fspecs.openid.net%2Fauth%2F2.0&openid.realm=https%3A%2F%
2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer&openid.return_to=https%3A%2F%
2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer%2Fcomplete
[Thu, 18 Jun 2009 00:33:29 +0200] ERROR: Net::HTTPFatalError
[Thu, 18 Jun 2009 00:33:29 +0200] FATAL: 502 "Bad Gateway"
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/1.8/net/http.rb:2097:in
error!' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:232:in run_request'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:230:in
run_request' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:95:in post_rest'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/client.rb:223:in
authenticate' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/client.rb:74:in run'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:163:in
run_application' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:161:in loop'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:161:in
run_application' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application.rb:53:in run'

/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/bin/chef-client:26
/usr/bin/chef-client:19:in `load'
/usr/bin/chef-client:19
[Thu, 18 Jun 2009 00:33:29 +0200] FATAL: Sleeping for 1817 seconds before
trying again

Doesn't seem to be anything indicative on the server side, so I'd suggest
a network or temporal issue such as DNS. :\

On the server side:

~ Started request handling: Thu Jun 18 00:33:28 +0200 2009
~ Params: {"format"=>nil, "action"=>"show",
"id"=>"nouvel_kingsofmambo_net",
"controller"=>"chef_server_slice/openid_register"}
~ {:before_filters_time=>0.0, :action_time=>0.0, :dispatch_time=>0.004,
:after_filters_time=>0.0}
~

~ Started request handling: Thu Jun 18 00:33:28 +0200 2009
~ Params: {"action"=>"node_page", "id"=>"nouvel_kingsofmambo_net",
"controller"=>"chef_server_slice/openid_server"}
~ {:before_filters_time=>0.0, :action_time=>0.004, :dispatch_time=>0.004,
:after_filters_time=>0.0}
~

~ Started request handling: Thu Jun 18 00:33:28 +0200 2009
~ Params: {"submit"=>"Verify", "action"=>"start",
"controller"=>"chef_server_slice/openid_consumer", "openid_identifier"=>"
https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net
"}
~ WARNING: making https request to
https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_netwithout verifying server certificate; no CA path was specified.
~ Generated checkid_setup request to
https://chef.kingsofmambo.net/openid/server with assocication
{HMAC-SHA1}{4a2f838d}{he1kgg==}
~ Generated checkid_setup request to
https://chef.kingsofmambo.net/openid/server with assocication
{HMAC-SHA1}{4a2f838d}{he1kgg==}
~ Redirecting to:
https://chef.kingsofmambo.net/openid/server?openid.assoc_handle={HMAC-SHA1}{4a2f838d}{he1kgg%3D%3D}&openid.claimed_id=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode%2Fnouvel_kingsofmambo_net&openid.identity=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode%2Fnouvel_kingsofmambo_net&openid.mode=checkid_setup&openid.ns=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0&openid.realm=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer&openid.return_to=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer%2Fcomplete(302)
~ {:before_filters_time=>0.0, :action_time=>0.024001,
:dispatch_time=>0.024001, :after_filters_time=>0.0}
~

~ Started request handling: Thu Jun 18 00:33:29 +0200 2009
~ Params: {"openid.claimed_id"=>"
https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net",
"openid.mode"=>"checkid_setup", "openid.return_to"=>"
https://chef.kingsofmambo.net/openid/consumer/complete", "openid.ns"=>"
http://specs.openid.net/auth/2.0", "action"=>"index",
"controller"=>"chef_server_slice/openid_server", "openid.identity"=>"
https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net",
"openid.assoc_handle"=>"{HMAC-SHA1}{4a2f838d}{he1kgg==}", "openid.realm"=>"
https://chef.kingsofmambo.net/openid/consumer"}
~ {:before_filters_time=>0.0, :action_time=>0.004, :dispatch_time=>0.004,
:after_filters_time=>0.0}
~

Thanks beforehand!

--
Albert Llop
Tech Department
albert@kingsofmambo.com

2009/6/17 Arjuna Christensen aj@opscode.com

On 17/06/2009, at 7:41 PM, Albert Llop wrote:

I've been having some trouble with my chef-clients, I've my chef-server
working properly, and my chef-clients can work just fine (with 0.7). I can
login, and execute chef-client as root and it will work wonderfully.

But after that it just doesn't work. In the log you see this every
30mins:

[Wed, 17 Jun 2009 09:15:09 +0200] INFO: Starting Chef Run
[Wed, 17 Jun 2009 09:15:09 +0200] ERROR: Net::HTTPFatalError
[Wed, 17 Jun 2009 09:15:09 +0200] FATAL: 502 "Bad Gateway"
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/1.8/net/http.rb:2097:in
error!' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:232:in run_request'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:230:in
run_request' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:95:in post_rest'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/client.rb:223:in
authenticate' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/client.rb:74:in run'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:163:in
run_application' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:161:in loop'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:161:in
run_application' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application.rb:53:in run'

/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/bin/chef-client:26
/usr/bin/chef-client:19:in `load'
/usr/bin/chef-client:19
[Wed, 17 Jun 2009 09:15:09 +0200] FATAL: Sleeping for 1815 seconds before
trying again

Could you possibly set your log_level to debug on both the client and the
server, and drop another email back with the increased log verbosity?

--
AJ Christensen, Software Engineer
Opscode, Inc.
E: aj@opscode.com adam@opscode.com

--
AJ Christensen, Software Engineer
Opscode, Inc.
E: aj@opscode.com adam@opscode.com

--
AJ Christensen, Software Engineer
Opscode, Inc.
E: aj@opscode.com adam@opscode.com

On 19/06/2009, at 10:05 PM, Albert Llop mrsimo@gmail.com wrote:

Thanks a lot, know that bit of info about the big cookies might be
useful!

And yeah, it's been working under nginx for some time, obtained all
the info I needed for that from this repo on github GitHub - fujin/cookbooks: Opscode Cookbooks for Chef

Oddly enough, that's one of my branches on my fork - I'm fujin!

Glad it's (kind of) working for you, was really just a hack of Josh's
apache stuff.

:slight_smile:

Albert Llop
Tech Department
albert@kingsofmambo.com

2009/6/19 Arjuna Christensen aj@opscode.com

On 19/06/2009, at 9:17 PM, Albert Llop wrote:

After some sleep I decided that the "502 "Bad Gateway"" error looks
too much like nginx, and I checked its erorr log (why didn't i do
it before?

Whenever one of these nodes that try to contact the server and is
in "zombie mode" like i like to call them (because chef-client is
working... but not properly), nginx gives an error:

2009/06/19 11:11:38 [error] 27044#0: *40 upstream sent too big
header while reading response header from upstream, client:
67.23.26.105, server: chef.kingsofmambo.net, request: "GET /openid/
server?openid.assoc_handle=%7BHMAC-SHA1%7D%7B4a2f838d%7D%7Bhe1kgg%3D
%3D%7D&openid.claimed_id=https%3A%2F%2Fchef.kingsofmambo.net
%2Fopenid%2Fserver%2Fnode
%2Fpotter_kingsofmambo_net&openid.identity=https%3A%2F
%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode
%2Fpotter_kingsofmambo_net&openid.mode=checkid_setup&openid.ns=http
%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0&openid.realm=https%3A%2F
%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer&openid.return_to=https
%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer%2Fcomplete HTTP/
1.1", upstream: "unix:/tmp/passenger.27019/master/
helper_server.sock:", host: "chef.kingsofmambo.net"

Very interesting. I wasn't aware you were running underneath another
container (although this should definitely work correctly.)

We're currently experiencing some issues with chef clients receiving
cookies bigger than 4kB from the Chef Server, which may be related
to this. I wish Nginx would tell us what the header was that was too
big.

I'm opening a ticket, and I'll scrub down the openid stuff while
running under Nginx Passenger and see if I can get it to fold.

Thanks for the additional information!

2009/6/18 Arjuna Christensen aj@opscode.com
Hey!

On 18/06/2009, at 7:53 PM, Albert Llop wrote:

Hi again,
I set the log level to debug on one of the nodes, and restarted
the client. After the restart it all worked fine for 14hours
(until midnight more or less), and then stopped working.

The client log looks very similar with the debug option:

[Thu, 18 Jun 2009 00:33:28 +0200] INFO: Starting Chef Run
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Registering
nouvel_kingsofmambo_net for an openid
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Sending HTTP Request via
GET to /registrations/nouvel_kingsofmambo_net
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Authenticating
nouvel_kingsofmambo_net via openid
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Sending HTTP Request via
POST to /openid/consumer/start
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Sending HTTP Request via
GET to /openid/server?openid.assoc_handle=%7BHMAC-SHA1%7D
%7B4a2f838d%7D%7Bhe1kgg%3D%3D%7D&openid.claimed_id=https%3A%2F
%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode
%2Fnouvel_kingsofmambo_net&openid.identity=https%3A%2F
%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode
%2Fnouvel_kingsofmambo_net&openid.mode=checkid_setup&openid.ns=http
%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0&openid.realm=https%3A%2F
%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer&openid.return_to=https
%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer%2Fcomplete
[Thu, 18 Jun 2009 00:33:29 +0200] ERROR: Net::HTTPFatalError
[Thu, 18 Jun 2009 00:33:29 +0200] FATAL: 502 "Bad Gateway"
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/1.8/net/http.rb:
2097:in error!' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/rest.rb:232:in run_request'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/lib/chef/rest.rb:230:in run_request' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/rest.rb:95:in post_rest'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/lib/chef/client.rb:223:in authenticate' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/client.rb:74:in run'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/lib/chef/application/client.rb:163:in run_application' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/application/client.rb:161:in loop'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/lib/chef/application/client.rb:161:in run_application' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/application.rb:53:in run'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/bin/chef-client:26
/usr/bin/chef-client:19:in `load'
/usr/bin/chef-client:19
[Thu, 18 Jun 2009 00:33:29 +0200] FATAL: Sleeping for 1817 seconds
before trying again

Doesn't seem to be anything indicative on the server side, so I'd
suggest a network or temporal issue such as DNS. :\

On the server side:

~ Started request handling: Thu Jun 18 00:33:28 +0200 2009
~ Params: {"format"=>nil, "action"=>"show",
"id"=>"nouvel_kingsofmambo_net", "controller"=>"chef_server_slice/
openid_register"}
~
{:b
efore_filters_time=

0.0,
:action_time=>0.0, :dispatch_time=>0.004, :after_filters_time=>0.0}
~

~ Started request handling: Thu Jun 18 00:33:28 +0200 2009
~ Params: {"action"=>"node_page",
"id"=>"nouvel_kingsofmambo_net", "controller"=>"chef_server_slice/
openid_server"}
~
{:b
efore_filters_time=

0.0,
:a
ction_time=>0.004, :dispatch_time=>0.004, :after_filters_time=>0.0}
~

~ Started request handling: Thu Jun 18 00:33:28 +0200 2009
~ Params: {"submit"=>"Verify", "action"=>"start",
"controller"=>"chef_server_slice/openid_consumer",
"openid_identifier"=>"https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net
"}
~ WARNING: making https request to https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net
without verifying server certificate; no CA path was specified.
~ Generated checkid_setup request to https://chef.kingsofmambo.net/openid/server
with assocication {HMAC-SHA1}{4a2f838d}{he1kgg==}
~ Generated checkid_setup request to https://chef.kingsofmambo.net/openid/server
with assocication {HMAC-SHA1}{4a2f838d}{he1kgg==}
~ Redirecting to: https://chef.kingsofmambo.net/openid/server?openid.assoc_handle={HMAC-SHA1}{4a2f838d}{he1kgg%3D%3D}&openid.claimed_id=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode%2Fnouvel_kingsofmambo_net&openid.identity=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode%2Fnouvel_kingsofmambo_net&openid.mode=checkid_setup&openid.ns=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0&openid.realm=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer&openid.return_to=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer%2Fcomplete
(302)
~
{:b
efore_filters_time=

0.0,
:a
ction_time=
0.024001, :dispatch_time=>0.024001, :after_filters_time=>0.0}
~

~ Started request handling: Thu Jun 18 00:33:29 +0200 2009
~ Params: {"openid.claimed_id"=>"https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net
", "openid.mode"=>"checkid_setup", "openid.return_to"=>"https://chef.kingsofmambo.net/openid/consumer/complete
", "openid.ns"=>"http://specs.openid.net/auth/2.0",
"action"=>"index", "controller"=>"chef_server_slice/
openid_server", "openid.identity"=>"https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net
", "openid.assoc_handle"=>"{HMAC-SHA1}{4a2f838d}{he1kgg==}",
"openid.realm"=>"https://chef.kingsofmambo.net/openid/consumer"}
~
{:b
efore_filters_time=

0.0,
:a
ction_time=>0.004, :dispatch_time=>0.004, :after_filters_time=>0.0}
~

Thanks beforehand!

--
Albert Llop
Tech Department
albert@kingsofmambo.com

2009/6/17 Arjuna Christensen aj@opscode.com

On 17/06/2009, at 7:41 PM, Albert Llop wrote:

I've been having some trouble with my chef-clients, I've my chef-
server working properly, and my chef-clients can work just fine
(with 0.7). I can login, and execute chef-client as root and it
will work wonderfully.

But after that it just doesn't work. In the log you see this
every 30mins:

[Wed, 17 Jun 2009 09:15:09 +0200] INFO: Starting Chef Run
[Wed, 17 Jun 2009 09:15:09 +0200] ERROR: Net::HTTPFatalError
[Wed, 17 Jun 2009 09:15:09 +0200] FATAL: 502 "Bad Gateway"
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/1.8/net/http.rb:
2097:in error!' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/rest.rb:232:in run_request'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/lib/chef/rest.rb:230:in run_request' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/rest.rb:95:in post_rest'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/lib/chef/client.rb:223:in authenticate' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/client.rb:74:in run'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/lib/chef/application/client.rb:163:in run_application' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/application/client.rb:161:in loop'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/lib/chef/application/client.rb:161:in run_application' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/application.rb:53:in run'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/bin/chef-client:26
/usr/bin/chef-client:19:in `load'
/usr/bin/chef-client:19
[Wed, 17 Jun 2009 09:15:09 +0200] FATAL: Sleeping for 1815
seconds before trying again

Could you possibly set your log_level to debug on both the client
and the server, and drop another email back with the increased log
verbosity?

--
AJ Christensen, Software Engineer
Opscode, Inc.
E: aj@opscode.com

--
AJ Christensen, Software Engineer
Opscode, Inc.
E: aj@opscode.com

--
AJ Christensen, Software Engineer
Opscode, Inc.
E: aj@opscode.com

Hi there again,

all this has been solved updating everything (server and clients) to the
0.7.4. Thanks a lot for the fixes!

Albert Llop
Tech Department
albert@kingsofmambo.com

2009/6/19 Arjuna Christensen aj@opscode.com

On 19/06/2009, at 10:05 PM, Albert Llop mrsimo@gmail.com wrote:

Thanks a lot, know that bit of info about the big cookies might be useful!

And yeah, it's been working under nginx for some time, obtained all the
info I needed for that from this repo on github
http://github.com/fujin/cookbooks/tree/nginx-chef-server
GitHub - fujin/cookbooks: Opscode Cookbooks for Chef

Oddly enough, that's one of my branches on my fork - I'm fujin!

Glad it's (kind of) working for you, was really just a hack of Josh's
apache stuff.

:slight_smile:

Albert Llop
Tech Department
albert@kingsofmambo.comalbert@kingsofmambo.com

2009/6/19 Arjuna Christensen < aj@opscode.comaj@opscode.com>

On 19/06/2009, at 9:17 PM, Albert Llop wrote:

After some sleep I decided that the "502 "Bad Gateway"" error looks too
much like nginx, and I checked its erorr log (why didn't i do it before?

Whenever one of these nodes that try to contact the server and is in
"zombie mode" like i like to call them (because chef-client is working...
but not properly), nginx gives an error:

2009/06/19 11:11:38 [error] 27044#0: *40 upstream sent too big header
while reading response header from upstream
, client: 67.23.26.105,
server: http://chef.kingsofmambo.netchef.kingsofmambo.net, request:
"GET
/openid/server?openid.assoc_handle=%7BHMAC-SHA1%7D%7B4a2f838d%7D%7Bhe1kgg%3D%3D%7D&openid.claimed_id=https%3A%2F%http://2Fchef.kingsofmambo.net
2Fchef.kingsofmambo.net
%2Fopenid%2Fserver%2Fnode%2Fpotter_kingsofmambo_net&openid.identity=https%3A%2F%http://2Fchef.kingsofmambo.net
2Fchef.kingsofmambo.net
%2Fopenid%2Fserver%2Fnode%2Fpotter_kingsofmambo_net&openid.mode=checkid_setup&openid.ns=http%3A%2F%http://2Fspecs.openid.net
2Fspecs.openid.net%2Fauth%2F2.0&openid.realm=https%3A%2F%http://2Fchef.kingsofmambo.net
2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer&openid.return_to=https%3A%2F%http://2Fchef.kingsofmambo.net
2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer%2Fcomplete HTTP/1.1",
upstream: "unix:/tmp/passenger.27019/master/helper_server.sock:", host: "http://chef.kingsofmambo.net
chef.kingsofmambo.net"

Very interesting. I wasn't aware you were running underneath another
container (although this should definitely work correctly.)

We're currently experiencing some issues with chef clients receiving
cookies bigger than 4kB from the Chef Server, which may be related to this.
I wish Nginx would tell us what the header was that was too big.

I'm opening a ticket, and I'll scrub down the openid stuff while running
under Nginx Passenger and see if I can get it to fold.

Thanks for the additional information!

2009/6/18 Arjuna Christensen < aj@opscode.comaj@opscode.com>

Hey!
On 18/06/2009, at 7:53 PM, Albert Llop wrote:

Hi again,
I set the log level to debug on one of the nodes, and restarted the
client. After the restart it all worked fine for 14hours (until midnight
more or less), and then stopped working.

The client log looks very similar with the debug option:

[Thu, 18 Jun 2009 00:33:28 +0200] INFO: Starting Chef Run
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Registering
nouvel_kingsofmambo_net for an openid
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Sending HTTP Request via GET to
/registrations/nouvel_kingsofmambo_net
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Authenticating
nouvel_kingsofmambo_net via openid
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Sending HTTP Request via POST to
/openid/consumer/start
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Sending HTTP Request via GET to
/openid/server?openid.assoc_handle=%7BHMAC-SHA1%7D%7B4a2f838d%7D%7Bhe1kgg%3D%3D%7D&openid.claimed_id=https%3A%2F%http://2Fchef.kingsofmambo.net
2Fchef.kingsofmambo.net
%2Fopenid%2Fserver%2Fnode%2Fnouvel_kingsofmambo_net&openid.identity=https%3A%2F%http://2Fchef.kingsofmambo.net
2Fchef.kingsofmambo.net
%2Fopenid%2Fserver%2Fnode%2Fnouvel_kingsofmambo_net&openid.mode=checkid_setup&openid.ns=http%3A%2F%http://2Fspecs.openid.net
2Fspecs.openid.net%2Fauth%2F2.0&openid.realm=https%3A%2F%http://2Fchef.kingsofmambo.net
2Fchef.kingsofmambo.net
%2Fopenid%2Fconsumer&openid.return_to=https%3A%2F%http://2Fchef.kingsofmambo.net
2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer%2Fcomplete
[Thu, 18 Jun 2009 00:33:29 +0200] ERROR: Net::HTTPFatalError
[Thu, 18 Jun 2009 00:33:29 +0200] FATAL: 502 "Bad Gateway"
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/1.8/net/http.rb:2097:in
error!' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:232:in run_request'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:230:in
run_request' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:95:in post_rest'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/client.rb:223:in
authenticate' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/client.rb:74:in run'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:163:in
run_application' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:161:in loop'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:161:in
run_application' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application.rb:53:in run'

/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/bin/chef-client:26
/usr/bin/chef-client:19:in `load'
/usr/bin/chef-client:19
[Thu, 18 Jun 2009 00:33:29 +0200] FATAL: Sleeping for 1817 seconds before
trying again

Doesn't seem to be anything indicative on the server side, so I'd suggest
a network or temporal issue such as DNS. :\

On the server side:

~ Started request handling: Thu Jun 18 00:33:28 +0200 2009
~ Params: {"format"=>nil, "action"=>"show",
"id"=>"nouvel_kingsofmambo_net",
"controller"=>"chef_server_slice/openid_register"}
~ {:before_filters_time=>0.0, :action_time=>0.0, :dispatch_time=>0.004,
:after_filters_time=>0.0}
~

~ Started request handling: Thu Jun 18 00:33:28 +0200 2009
~ Params: {"action"=>"node_page", "id"=>"nouvel_kingsofmambo_net",
"controller"=>"chef_server_slice/openid_server"}
~ {:before_filters_time=>0.0, :action_time=>0.004,
:dispatch_time=>0.004, :after_filters_time=>0.0}
~

~ Started request handling: Thu Jun 18 00:33:28 +0200 2009
~ Params: {"submit"=>"Verify", "action"=>"start",
"controller"=>"chef_server_slice/openid_consumer", "openid_identifier"=>"https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net
https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net
"}
~ WARNING: making https request to
https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net
https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_netwithout verifying server certificate; no CA path was specified.
~ Generated checkid_setup request to
https://chef.kingsofmambo.net/openid/server
https://chef.kingsofmambo.net/openid/server with assocication
{HMAC-SHA1}{4a2f838d}{he1kgg==}
~ Generated checkid_setup request to
https://chef.kingsofmambo.net/openid/server
https://chef.kingsofmambo.net/openid/server with assocication
{HMAC-SHA1}{4a2f838d}{he1kgg==}
~ Redirecting to:
https://chef.kingsofmambo.net/openid/server?openid.assoc_handle={HMAC-SHA1}{4a2f838d}{he1kgg%3D%3D}&openid.claimed_id=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode%2Fnouvel_kingsofmambo_net&openid.identity=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode%2Fnouvel_kingsofmambo_net&openid.mode=checkid_setup&openid.ns=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0&openid.realm=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer&openid.return_to=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer%2Fcomplete
https://chef.kingsofmambo.net/openid/server?openid.assoc_handle={HMAC-SHA1}{4a2f838d}{he1kgg%3D%3D}&openid.claimed_id=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode%2Fnouvel_kingsofmambo_net&openid.identity=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode%2Fnouvel_kingsofmambo_net&openid.mode=checkid_setup&openid.ns=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0&openid.realm=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer&openid.return_to=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer%2Fcomplete(302)
~ {:before_filters_time=>0.0, :action_time=>0.024001,
:dispatch_time=>0.024001, :after_filters_time=>0.0}
~

~ Started request handling: Thu Jun 18 00:33:29 +0200 2009
~ Params: {"openid.claimed_id"=>"https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net
https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net",
"openid.mode"=>"checkid_setup", "openid.return_to"=>"https://chef.kingsofmambo.net/openid/consumer/complete
https://chef.kingsofmambo.net/openid/consumer/complete", "openid.ns"=>"http://specs.openid.net/auth/2.0
http://specs.openid.net/auth/2.0", "action"=>"index",
"controller"=>"chef_server_slice/openid_server", "openid.identity"=>"https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net
https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net",
"openid.assoc_handle"=>"{HMAC-SHA1}{4a2f838d}{he1kgg==}", "openid.realm"=>"https://chef.kingsofmambo.net/openid/consumer
https://chef.kingsofmambo.net/openid/consumer"}
~ {:before_filters_time=>0.0, :action_time=>0.004,
:dispatch_time=>0.004, :after_filters_time=>0.0}
~

Thanks beforehand!

--
Albert Llop
Tech Department
albert@kingsofmambo.comalbert@kingsofmambo.com

2009/6/17 Arjuna Christensen < aj@opscode.comaj@opscode.com>

On 17/06/2009, at 7:41 PM, Albert Llop wrote:

I've been having some trouble with my chef-clients, I've my chef-server
working properly, and my chef-clients can work just fine (with 0.7). I can
login, and execute chef-client as root and it will work wonderfully.

But after that it just doesn't work. In the log you see this every
30mins:

[Wed, 17 Jun 2009 09:15:09 +0200] INFO: Starting Chef Run
[Wed, 17 Jun 2009 09:15:09 +0200] ERROR: Net::HTTPFatalError
[Wed, 17 Jun 2009 09:15:09 +0200] FATAL: 502 "Bad Gateway"
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/1.8/net/http.rb:2097:in
error!' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:232:in run_request'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:230:in
run_request' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:95:in post_rest'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/client.rb:223:in
authenticate' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/client.rb:74:in run'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:163:in
run_application' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:161:in loop'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:161:in
run_application' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application.rb:53:in run'

/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/bin/chef-client:26
/usr/bin/chef-client:19:in `load'
/usr/bin/chef-client:19
[Wed, 17 Jun 2009 09:15:09 +0200] FATAL: Sleeping for 1815 seconds
before trying again

Could you possibly set your log_level to debug on both the client and
the server, and drop another email back with the increased log verbosity?

--
AJ Christensen, Software Engineer
Opscode, Inc.
E: adam@opscode.comaj@opscode.com

--
AJ Christensen, Software Engineer
Opscode, Inc.
E: adam@opscode.comaj@opscode.com

--
AJ Christensen, Software Engineer
Opscode, Inc.
E: adam@opscode.comaj@opscode.com

On 30/06/2009, at 12:11 AM, Albert Llop wrote:

Hi there again,

all this has been solved updating everything (server and clients) to
the 0.7.4. Thanks a lot for the fixes!

Not a problem. Sorry for not alerting you when I fixed it, slipped my
mind. I'm glad you've got everything working!

--
Albert Llop
Tech Department
albert@kingsofmambo.com

2009/6/19 Arjuna Christensen aj@opscode.com

On 19/06/2009, at 10:05 PM, Albert Llop mrsimo@gmail.com wrote:

Thanks a lot, know that bit of info about the big cookies might be
useful!

And yeah, it's been working under nginx for some time, obtained all
the info I needed for that from this repo on github GitHub - fujin/cookbooks: Opscode Cookbooks for Chef

Oddly enough, that's one of my branches on my fork - I'm fujin!

Glad it's (kind of) working for you, was really just a hack of
Josh's apache stuff.

:slight_smile:

Albert Llop
Tech Department
albert@kingsofmambo.com

2009/6/19 Arjuna Christensen aj@opscode.com

On 19/06/2009, at 9:17 PM, Albert Llop wrote:

After some sleep I decided that the "502 "Bad Gateway"" error
looks too much like nginx, and I checked its erorr log (why didn't
i do it before?

Whenever one of these nodes that try to contact the server and is
in "zombie mode" like i like to call them (because chef-client is
working... but not properly), nginx gives an error:

2009/06/19 11:11:38 [error] 27044#0: *40 upstream sent too big
header while reading response header from upstream, client:
67.23.26.105, server: chef.kingsofmambo.net, request: "GET /openid/
server?openid.assoc_handle=%7BHMAC-SHA1%7D%7B4a2f838d%7D%7Bhe1kgg
%3D%3D%7D&openid.claimed_id=https%3A%2F%2Fchef.kingsofmambo.net
%2Fopenid%2Fserver%2Fnode
%2Fpotter_kingsofmambo_net&openid.identity=https%3A%2F
%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode
%2Fpotter_kingsofmambo_net&openid.mode=checkid_setup&openid.ns=http
%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0&openid.realm=https%3A%2F
%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer&openid.return_to=https
%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer%2Fcomplete HTTP/
1.1", upstream: "unix:/tmp/passenger.27019/master/
helper_server.sock:", host: "chef.kingsofmambo.net"

Very interesting. I wasn't aware you were running underneath
another container (although this should definitely work correctly.)

We're currently experiencing some issues with chef clients
receiving cookies bigger than 4kB from the Chef Server, which may
be related to this. I wish Nginx would tell us what the header was
that was too big.

I'm opening a ticket, and I'll scrub down the openid stuff while
running under Nginx Passenger and see if I can get it to fold.

Thanks for the additional information!

2009/6/18 Arjuna Christensen aj@opscode.com
Hey!

On 18/06/2009, at 7:53 PM, Albert Llop wrote:

Hi again,
I set the log level to debug on one of the nodes, and restarted
the client. After the restart it all worked fine for 14hours
(until midnight more or less), and then stopped working.

The client log looks very similar with the debug option:

[Thu, 18 Jun 2009 00:33:28 +0200] INFO: Starting Chef Run
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Registering
nouvel_kingsofmambo_net for an openid
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Sending HTTP Request via
GET to /registrations/nouvel_kingsofmambo_net
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Authenticating
nouvel_kingsofmambo_net via openid
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Sending HTTP Request via
POST to /openid/consumer/start
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Sending HTTP Request via
GET to /openid/server?openid.assoc_handle=%7BHMAC-SHA1%7D
%7B4a2f838d%7D%7Bhe1kgg%3D%3D%7D&openid.claimed_id=https%3A%2F
%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode
%2Fnouvel_kingsofmambo_net&openid.identity=https%3A%2F
%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode
%2Fnouvel_kingsofmambo_net&openid.mode=checkid_setup&openid.ns=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0&openid.realm=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer&openid.return_to=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer%2Fcomplete
[Thu, 18 Jun 2009 00:33:29 +0200] ERROR: Net::HTTPFatalError
[Thu, 18 Jun 2009 00:33:29 +0200] FATAL: 502 "Bad Gateway"
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/1.8/net/http.rb:
2097:in error!' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/rest.rb:232:in run_request'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/lib/chef/rest.rb:230:in run_request' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/rest.rb:95:in post_rest'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/lib/chef/client.rb:223:in authenticate' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/client.rb:74:in run'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/lib/chef/application/client.rb:163:in run_application' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/application/client.rb:161:in loop'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/lib/chef/application/client.rb:161:in run_application' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/application.rb:53:in run'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/bin/chef-client:26
/usr/bin/chef-client:19:in `load'
/usr/bin/chef-client:19
[Thu, 18 Jun 2009 00:33:29 +0200] FATAL: Sleeping for 1817
seconds before trying again

Doesn't seem to be anything indicative on the server side, so I'd
suggest a network or temporal issue such as DNS. :\

On the server side:

~ Started request handling: Thu Jun 18 00:33:28 +0200 2009
~ Params: {"format"=>nil, "action"=>"show",
"id"=>"nouvel_kingsofmambo_net", "controller"=>"chef_server_slice/
openid_register"}
~
{:before_filters_time

0.0
, :action_time
=>0.0, :dispatch_time=>0.004, :after_filters_time=>0.0}
~

~ Started request handling: Thu Jun 18 00:33:28 +0200 2009
~ Params: {"action"=>"node_page",
"id"=>"nouvel_kingsofmambo_net", "controller"=>"chef_server_slice/
openid_server"}
~
{:before_filters_time

0.0
, :action_time
=>0.004, :dispatch_time=>0.004, :after_filters_time=>0.0}
~

~ Started request handling: Thu Jun 18 00:33:28 +0200 2009
~ Params: {"submit"=>"Verify", "action"=>"start",
"controller"=>"chef_server_slice/openid_consumer",
"openid_identifier"=>"https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net
"}
~ WARNING: making https request to https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net
without verifying server certificate; no CA path was specified.
~ Generated checkid_setup request to https://chef.kingsofmambo.net/openid/server
with assocication {HMAC-SHA1}{4a2f838d}{he1kgg==}
~ Generated checkid_setup request to https://chef.kingsofmambo.net/openid/server
with assocication {HMAC-SHA1}{4a2f838d}{he1kgg==}
~ Redirecting to: https://chef.kingsofmambo.net/openid/server?openid.assoc_handle={HMAC-SHA1}{4a2f838d}{he1kgg%3D%3D}&openid.claimed_id=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode%2Fnouvel_kingsofmambo_net&openid.identity=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode%2Fnouvel_kingsofmambo_net&openid.mode=checkid_setup&openid.ns=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0&openid.realm=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer&openid.return_to=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer%2Fcomplete
(302)
~
{:before_filters_time

0.0
, :action_time
=>0.024001, :dispatch_time=>0.024001, :after_filters_time=>0.0}
~

~ Started request handling: Thu Jun 18 00:33:29 +0200 2009
~ Params: {"openid.claimed_id"=>"https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net
", "openid.mode"=>"checkid_setup", "openid.return_to"=>"https://chef.kingsofmambo.net/openid/consumer/complete
", "openid.ns"=>"http://specs.openid.net/auth/2.0",
"action"=>"index", "controller"=>"chef_server_slice/
openid_server", "openid.identity"=>"https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net
", "openid.assoc_handle"=>"{HMAC-SHA1}{4a2f838d}{he1kgg==}",
"openid.realm"=>"https://chef.kingsofmambo.net/openid/consumer"}
~
{:before_filters_time

0.0
, :action_time
=>0.004, :dispatch_time=>0.004, :after_filters_time=>0.0}
~

Thanks beforehand!

--
Albert Llop
Tech Department
albert@kingsofmambo.com

2009/6/17 Arjuna Christensen aj@opscode.com

On 17/06/2009, at 7:41 PM, Albert Llop wrote:

I've been having some trouble with my chef-clients, I've my chef-
server working properly, and my chef-clients can work just fine
(with 0.7). I can login, and execute chef-client as root and it
will work wonderfully.

But after that it just doesn't work. In the log you see this
every 30mins:

[Wed, 17 Jun 2009 09:15:09 +0200] INFO: Starting Chef Run
[Wed, 17 Jun 2009 09:15:09 +0200] ERROR: Net::HTTPFatalError
[Wed, 17 Jun 2009 09:15:09 +0200] FATAL: 502 "Bad Gateway"
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/1.8/net/http.rb:
2097:in error!' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/rest.rb:232:in run_request'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/lib/chef/rest.rb:230:in run_request' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/rest.rb:95:in post_rest'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/lib/chef/client.rb:223:in authenticate' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/client.rb:74:in run'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/lib/chef/application/client.rb:163:in run_application' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/application/client.rb:161:in loop'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/lib/chef/application/client.rb:161:in run_application' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/ chef-0.7.0/lib/chef/application.rb:53:in run'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/
chef-0.7.0/bin/chef-client:26
/usr/bin/chef-client:19:in `load'
/usr/bin/chef-client:19
[Wed, 17 Jun 2009 09:15:09 +0200] FATAL: Sleeping for 1815
seconds before trying again

Could you possibly set your log_level to debug on both the client
and the server, and drop another email back with the increased
log verbosity?

--
AJ Christensen, Software Engineer
Opscode, Inc.
E: aj@opscode.com

--
AJ Christensen, Software Engineer
Opscode, Inc.
E: aj@opscode.com

--
AJ Christensen, Software Engineer
Opscode, Inc.
E: aj@opscode.com

--
AJ Christensen, Software Engineer
Opscode, Inc.
E: aj@opscode.com

I saw it on the release notes, it's ok =) I just didn't have time to update
everything.

:smiley: thanks for the fix!!

Albert Llop
Tech Department
albert@kingsofmambo.com

2009/6/30 Arjuna Christensen aj@opscode.com

On 30/06/2009, at 12:11 AM, Albert Llop wrote:

Hi there again,

all this has been solved updating everything (server and clients) to the
0.7.4. Thanks a lot for the fixes!

Not a problem. Sorry for not alerting you when I fixed it, slipped my mind.
I'm glad you've got everything working!

--
Albert Llop
Tech Department
albert@kingsofmambo.com

2009/6/19 Arjuna Christensen aj@opscode.com

On 19/06/2009, at 10:05 PM, Albert Llop mrsimo@gmail.com wrote:

Thanks a lot, know that bit of info about the big cookies might be useful!

And yeah, it's been working under nginx for some time, obtained all the
info I needed for that from this repo on github
http://github.com/fujin/cookbooks/tree/nginx-chef-server
GitHub - fujin/cookbooks: Opscode Cookbooks for Chef

Oddly enough, that's one of my branches on my fork - I'm fujin!

Glad it's (kind of) working for you, was really just a hack of Josh's
apache stuff.

:slight_smile:

Albert Llop
Tech Department
albert@kingsofmambo.comalbert@kingsofmambo.com

2009/6/19 Arjuna Christensen < aj@opscode.comaj@opscode.com>

On 19/06/2009, at 9:17 PM, Albert Llop wrote:

After some sleep I decided that the "502 "Bad Gateway"" error looks too
much like nginx, and I checked its erorr log (why didn't i do it before?

Whenever one of these nodes that try to contact the server and is in
"zombie mode" like i like to call them (because chef-client is working...
but not properly), nginx gives an error:

2009/06/19 11:11:38 [error] 27044#0: *40 upstream sent too big header
while reading response header from upstream
, client: 67.23.26.105,
server: http://chef.kingsofmambo.netchef.kingsofmambo.net, request:
"GET
/openid/server?openid.assoc_handle=%7BHMAC-SHA1%7D%7B4a2f838d%7D%7Bhe1kgg%3D%3D%7D&openid.claimed_id=https%3A%2F%http://2Fchef.kingsofmambo.net
2Fchef.kingsofmambo.net
%2Fopenid%2Fserver%2Fnode%2Fpotter_kingsofmambo_net&openid.identity=https%3A%2F%http://2Fchef.kingsofmambo.net
2Fchef.kingsofmambo.net
%2Fopenid%2Fserver%2Fnode%2Fpotter_kingsofmambo_net&openid.mode=checkid_setup&openid.ns=http%3A%2F%http://2Fspecs.openid.net
2Fspecs.openid.net%2Fauth%2F2.0&openid.realm=https%3A%2F%http://2Fchef.kingsofmambo.net
2Fchef.kingsofmambo.net
%2Fopenid%2Fconsumer&openid.return_to=https%3A%2F%http://2Fchef.kingsofmambo.net
2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer%2Fcomplete HTTP/1.1",
upstream: "unix:/tmp/passenger.27019/master/helper_server.sock:", host: "http://chef.kingsofmambo.net
chef.kingsofmambo.net"

Very interesting. I wasn't aware you were running underneath another
container (although this should definitely work correctly.)

We're currently experiencing some issues with chef clients receiving
cookies bigger than 4kB from the Chef Server, which may be related to this.
I wish Nginx would tell us what the header was that was too big.

I'm opening a ticket, and I'll scrub down the openid stuff while running
under Nginx Passenger and see if I can get it to fold.

Thanks for the additional information!

2009/6/18 Arjuna Christensen < aj@opscode.comaj@opscode.com>

Hey!
On 18/06/2009, at 7:53 PM, Albert Llop wrote:

Hi again,
I set the log level to debug on one of the nodes, and restarted the
client. After the restart it all worked fine for 14hours (until midnight
more or less), and then stopped working.

The client log looks very similar with the debug option:

[Thu, 18 Jun 2009 00:33:28 +0200] INFO: Starting Chef Run
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Registering
nouvel_kingsofmambo_net for an openid
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Sending HTTP Request via GET to
/registrations/nouvel_kingsofmambo_net
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Authenticating
nouvel_kingsofmambo_net via openid
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Sending HTTP Request via POST
to /openid/consumer/start
[Thu, 18 Jun 2009 00:33:28 +0200] DEBUG: Sending HTTP Request via GET to
/openid/server?openid.assoc_handle=%7BHMAC-SHA1%7D%7B4a2f838d%7D%7Bhe1kgg%3D%3D%7D&openid.claimed_id=https%3A%2F%http://2Fchef.kingsofmambo.net
2Fchef.kingsofmambo.net
%2Fopenid%2Fserver%2Fnode%2Fnouvel_kingsofmambo_net&openid.identity=https%3A%2F%http://2Fchef.kingsofmambo.net
2Fchef.kingsofmambo.net
%2Fopenid%2Fserver%2Fnode%2Fnouvel_kingsofmambo_net&openid.mode=checkid_setup&openid.ns=http%3A%2F%http://2Fspecs.openid.net
2Fspecs.openid.net%2Fauth%2F2.0&openid.realm=https%3A%2F%http://2Fchef.kingsofmambo.net
2Fchef.kingsofmambo.net
%2Fopenid%2Fconsumer&openid.return_to=https%3A%2F%http://2Fchef.kingsofmambo.net
2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer%2Fcomplete
[Thu, 18 Jun 2009 00:33:29 +0200] ERROR: Net::HTTPFatalError
[Thu, 18 Jun 2009 00:33:29 +0200] FATAL: 502 "Bad Gateway"
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/1.8/net/http.rb:2097:in
error!' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:232:in run_request'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:230:in
run_request' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:95:in post_rest'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/client.rb:223:in
authenticate' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/client.rb:74:in run'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:163:in
run_application' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:161:in loop'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:161:in
run_application' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application.rb:53:in run'

/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/bin/chef-client:26
/usr/bin/chef-client:19:in `load'
/usr/bin/chef-client:19
[Thu, 18 Jun 2009 00:33:29 +0200] FATAL: Sleeping for 1817 seconds
before trying again

Doesn't seem to be anything indicative on the server side, so I'd
suggest a network or temporal issue such as DNS. :\

On the server side:

~ Started request handling: Thu Jun 18 00:33:28 +0200 2009
~ Params: {"format"=>nil, "action"=>"show",
"id"=>"nouvel_kingsofmambo_net",
"controller"=>"chef_server_slice/openid_register"}
~ {:before_filters_time=>0.0, :action_time=>0.0, :dispatch_time=>0.004,
:after_filters_time=>0.0}
~

~ Started request handling: Thu Jun 18 00:33:28 +0200 2009
~ Params: {"action"=>"node_page", "id"=>"nouvel_kingsofmambo_net",
"controller"=>"chef_server_slice/openid_server"}
~ {:before_filters_time=>0.0, :action_time=>0.004,
:dispatch_time=>0.004, :after_filters_time=>0.0}
~

~ Started request handling: Thu Jun 18 00:33:28 +0200 2009
~ Params: {"submit"=>"Verify", "action"=>"start",
"controller"=>"chef_server_slice/openid_consumer", "openid_identifier"=>"https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net
https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net
"}
~ WARNING: making https request to
https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net
https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_netwithout verifying server certificate; no CA path was specified.
~ Generated checkid_setup request to
https://chef.kingsofmambo.net/openid/server
https://chef.kingsofmambo.net/openid/server with assocication
{HMAC-SHA1}{4a2f838d}{he1kgg==}
~ Generated checkid_setup request to
https://chef.kingsofmambo.net/openid/server
https://chef.kingsofmambo.net/openid/server with assocication
{HMAC-SHA1}{4a2f838d}{he1kgg==}
~ Redirecting to:
https://chef.kingsofmambo.net/openid/server?openid.assoc_handle={HMAC-SHA1}{4a2f838d}{he1kgg%3D%3D}&openid.claimed_id=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode%2Fnouvel_kingsofmambo_net&openid.identity=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode%2Fnouvel_kingsofmambo_net&openid.mode=checkid_setup&openid.ns=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0&openid.realm=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer&openid.return_to=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer%2Fcomplete
https://chef.kingsofmambo.net/openid/server?openid.assoc_handle={HMAC-SHA1}{4a2f838d}{he1kgg%3D%3D}&openid.claimed_id=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode%2Fnouvel_kingsofmambo_net&openid.identity=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fserver%2Fnode%2Fnouvel_kingsofmambo_net&openid.mode=checkid_setup&openid.ns=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0&openid.realm=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer&openid.return_to=https%3A%2F%2Fchef.kingsofmambo.net%2Fopenid%2Fconsumer%2Fcomplete(302)
~ {:before_filters_time=>0.0, :action_time=>0.024001,
:dispatch_time=>0.024001, :after_filters_time=>0.0}
~

~ Started request handling: Thu Jun 18 00:33:29 +0200 2009
~ Params: {"openid.claimed_id"=>"https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net
https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net",
"openid.mode"=>"checkid_setup", "openid.return_to"=>"https://chef.kingsofmambo.net/openid/consumer/complete
https://chef.kingsofmambo.net/openid/consumer/complete", "openid.ns"=>"http://specs.openid.net/auth/2.0
http://specs.openid.net/auth/2.0", "action"=>"index",
"controller"=>"chef_server_slice/openid_server", "openid.identity"=>"https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net
https://chef.kingsofmambo.net/openid/server/node/nouvel_kingsofmambo_net",
"openid.assoc_handle"=>"{HMAC-SHA1}{4a2f838d}{he1kgg==}", "openid.realm"=>"https://chef.kingsofmambo.net/openid/consumer
https://chef.kingsofmambo.net/openid/consumer"}
~ {:before_filters_time=>0.0, :action_time=>0.004,
:dispatch_time=>0.004, :after_filters_time=>0.0}
~

Thanks beforehand!

--
Albert Llop
Tech Department
albert@kingsofmambo.comalbert@kingsofmambo.com

2009/6/17 Arjuna Christensen < aj@opscode.comaj@opscode.com>

On 17/06/2009, at 7:41 PM, Albert Llop wrote:

I've been having some trouble with my chef-clients, I've my chef-server
working properly, and my chef-clients can work just fine (with 0.7). I can
login, and execute chef-client as root and it will work wonderfully.

But after that it just doesn't work. In the log you see this every
30mins:

[Wed, 17 Jun 2009 09:15:09 +0200] INFO: Starting Chef Run
[Wed, 17 Jun 2009 09:15:09 +0200] ERROR: Net::HTTPFatalError
[Wed, 17 Jun 2009 09:15:09 +0200] FATAL: 502 "Bad Gateway"
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/1.8/net/http.rb:2097:in
error!' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:232:in run_request'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:230:in
run_request' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/rest.rb:95:in post_rest'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/client.rb:223:in
authenticate' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/client.rb:74:in run'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:163:in
run_application' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:161:in loop'
/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application/client.rb:161:in
run_application' /opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/lib/chef/application.rb:53:in run'

/opt/ruby-enterprise-1.8.6-20090610/lib/ruby/gems/1.8/gems/chef-0.7.0/bin/chef-client:26
/usr/bin/chef-client:19:in `load'
/usr/bin/chef-client:19
[Wed, 17 Jun 2009 09:15:09 +0200] FATAL: Sleeping for 1815 seconds
before trying again

Could you possibly set your log_level to debug on both the client and
the server, and drop another email back with the increased log verbosity?

--
AJ Christensen, Software Engineer
Opscode, Inc.
E: adam@opscode.comaj@opscode.com

--
AJ Christensen, Software Engineer
Opscode, Inc.
E: adam@opscode.comaj@opscode.com

--
AJ Christensen, Software Engineer
Opscode, Inc.
E: adam@opscode.comaj@opscode.com

--
AJ Christensen, Software Engineer
Opscode, Inc.
E: aj@opscode.com adam@opscode.com