Chef server cannot work for timeout

Hello, all

I had a ubuntu server running chef-server_11.0.12-1.ubuntu.12.04_amd64.deb
Recently I had restart the chef server, and then I cannot bootstrap VM and
ran chef client on it.

Here are some related errors, what could be the root cause?
And how can I fix it?

=============================================================================

[2015-07-29T08:10:28-07:00] INFO: Running start handlers
[2015-07-29T08:10:28-07:00] INFO: Start handlers complete.
[2015-07-29T08:15:28-07:00] ERROR: Timeout connecting to test.com:443 for
/environments/_default/cookbook_versions, retry 1/5
[2015-07-29T08:20:34-07:00] ERROR: Timeout connecting to test.com:443 for
/environments/_default/cookbook_versions, retry 2/5
[2015-07-29T08:25:39-07:00] ERROR: Timeout connecting to test.com:443 for
/environments/_default/cookbook_versions, retry 3/5

================================================================================
Error Resolving Cookbooks for Run List:

Authentication Error:

Failed to authenticate to the chef server (http 401).
The request failed because your clock has drifted by more than 15 minutes.
( I tried to sync time, but it doesn’t work)
Syncing your clock to an NTP Time source should resolve the issue.

[2015-07-29T08:25:44-07:00] ERROR: Running exception handlers
[2015-07-29T08:25:44-07:00] FATAL: Saving node information to
/var/chef/cache/failed-run-data.json
[2015-07-29T08:25:44-07:00] ERROR: Exception handlers complete
[2015-07-29T08:25:44-07:00] FATAL: Stacktrace dumped to
/var/chef/cache/chef-stacktrace.out
[2015-07-29T08:25:44-07:00] FATAL: Net::HTTPServerException: 401
"Unauthorized"

And in chef server I can such errors:

2015/07/29 07:34:37 [error] 2664#0: *51 upstream timed out (110: Connection
timed out) while reading response header from upstream, client:
38.102.156.135, server: chef3.cloud.intalio.com, request: “POST
/environments/_default/cookbook_versions HTTP/1.1”, upstream: “
http://127.0.0.1:8000/environments/_default/cookbook_versions”, host: "
chef3.cloud.intalio.com:443"
2015/07/29 07:39:41 [error] 2665#0: *61 upstream timed out (110: Connection
timed out) while reading response header from upstream, client:
38.102.156.135, server: chef3.cloud.intalio.com, request: “POST
/environments/_default/cookbook_versions HTTP/1.1”, upstream: “
http://127.0.0.1:8000/environments/_default/cookbook_versions”, host: "
chef3.cloud.intalio.com:443"
2015/07/29 07:44:46 [error] 2665#0: *63 upstream timed out (110: Connection
timed out) while reading response header from upstream, client:
38.102.156.135, server: chef3.cloud.intalio.com, request: “POST
/environments/_default/cookbook_versions HTTP/1.1”, upstream: “
http://127.0.0.1:8000/environments/_default/cookbook_versions”, host: "
chef3.cloud.intalio.com:443"
2015/07/29 08:15:29 [error] 2664#0: *805 upstream timed out (110:
Connection timed out) while reading response header from upstream, client:
38.102.156.135, server: chef3.cloud.intalio.com, request: “POST
/environments/_default/cookbook_versions HTTP/1.1”, upstream: “
http://127.0.0.1:8000/environments/_default/cookbook_versions”, host: "
chef3.cloud.intalio.com:443"
2015/07/29 08:20:34 [error] 2664#0: *807 upstream timed out (110:
Connection timed out) while reading response header from upstream, client:
38.102.156.135, server: chef3.cloud.intalio.com, request: “POST
/environments/_default/cookbook_versions HTTP/1.1”, upstream: “
http://127.0.0.1:8000/environments/_default/cookbook_versions”, host: "
chef3.cloud.intalio.com:443"
2015/07/29 08:25:39 [error] 2664#0: *809 upstream timed out (110:
Connection timed out) while reading response header from upstream, client:
38.102.156.135, server: chef3.cloud.intalio.com, request: “POST
/environments/_default/cookbook_versions HTTP/1.1”, upstream: “
http://127.0.0.1:8000/environments/_default/cookbook_versions”, host:
"chef3.cloud.intalio.com:443

And in erchef code, I can see such:

2015-07-29_17:04:04.97022 ** Found 0 name clashes in code paths
2015-07-29_17:04:05.20423
2015-07-29_17:04:05.20425 =INFO REPORT==== 29-Jul-2015::10:04:05 ===
2015-07-29_17:04:05.20425 Starting sqerl supervisor
2015-07-29_17:04:07.13848
2015-07-29_17:04:07.13850 =INFO REPORT==== 29-Jul-2015::10:04:07 ===
2015-07-29_17:04:07.13850 starting stats_hero_sup
2015-07-29_17:04:07.14937
2015-07-29_17:04:07.14938 =INFO REPORT==== 29-Jul-2015::10:04:07 ===
2015-07-29_17:04:07.14939 starting stats_hero_sender_sup with 1 senders
2015-07-29_17:04:07.16031
2015-07-29_17:04:07.16032 =INFO REPORT==== 29-Jul-2015::10:04:07 ===
2015-07-29_17:04:07.16033 stats_hero_sender starting pid: <0.216.0> port:
42498
2015-07-29_17:04:07.17695
2015-07-29_17:04:07.17696 =INFO REPORT==== 29-Jul-2015::10:04:07 ===
2015-07-29_17:04:07.17697 bulk_fetch_batch_size set to 5
2015-07-29_17:04:07.35016
2015-07-29_17:04:07.35018 =INFO REPORT==== 29-Jul-2015::10:04:07 ===
2015-07-29_17:04:07.35018 Chef Keyring starting
2015-07-29_17:04:07.42652
2015-07-29_17:04:07.42655 =INFO REPORT==== 29-Jul-2015::10:04:07 ===
2015-07-29_17:04:07.42656 Loaded key default of type RSAPrivateKey from
file /etc/chef-server/chef-webui.pem
2015-07-29_17:04:07.44778
2015-07-29_17:04:07.44779 =INFO REPORT==== 29-Jul-2015::10:04:07 ===
2015-07-29_17:04:07.44780 starting chef_index_sup
2015-07-29_17:04:07.44816
2015-07-29_17:04:07.44817 =INFO REPORT==== 29-Jul-2015::10:04:07 ===
2015-07-29_17:04:07.44818 Connecting to Rabbit at 127.0.0.1:5672/chef
(exchange: <<>>)
2015-07-29_17:04:07.48567
2015-07-29_17:04:07.48569 =ERROR REPORT==== 29-Jul-2015::10:04:07 ===
2015-07-29_17:04:07.48569 Could not start the network driver: econnrefused
2015-07-29_17:04:07.48779
2015-07-29_17:04:07.48780 =ERROR REPORT==== 29-Jul-2015::10:04:07 ===
2015-07-29_17:04:07.48780 ** Generic server <0.252.0> terminating
2015-07-29_17:04:07.48780 ** Last message in was connect
2015-07-29_17:04:07.48780 ** When Server state == {state,<0.250.0>,
2015-07-29_17:04:07.48781
{amqp_params,<<“chef”>>,
2015-07-29_17:04:07.48781
<<“050f4cfb8b32d9a5f8d5aed9c1d7f5e5f7aa6fab125b5b204ed418f30b73b76e29e88c8fdbd1dacf6000b106e5a9de5edf10”>>,
2015-07-29_17:04:07.48781
<<"/chef">>,“127.0.0.1”,5672,0,0,0,none,[]},
2015-07-29_17:04:07.48782
undefined,undefined,undefined,undefined,undefined,
2015-07-29_17:04:07.48782
undefined,<0.251.0>,false,undefined,
2015-07-29_17:04:07.48782 {{0,nil},
2015-07-29_17:04:07.48782
{dict,0,16,16,8,80,48,
2015-07-29_17:04:07.48783
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
2015-07-29_17:04:07.48783 [],[]},
2015-07-29_17:04:07.48783
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
2015-07-29_17:04:07.48783 [],[]}}}},
2015-07-29_17:04:07.48784
undefined,#Fun<amqp_connection_sup.0.67288451>}
2015-07-29_17:04:07.48784 ** Reason for termination ==
2015-07-29_17:04:07.48784 ** econnrefused
2015-07-29_17:04:07.50116
2015-07-29_17:04:07.50117 =ERROR REPORT==== 29-Jul-2015::10:04:07 ===
2015-07-29_17:04:07.50118 “Could not connect, scheduling reconnect.“
2015-07-29_17:04:07.50118 error: {{error,
2015-07-29_17:04:07.50118 {badmatch,
2015-07-29_17:04:07.50119 {error,
2015-07-29_17:04:07.50119 {auth_failure_likely,
2015-07-29_17:04:07.50119 {econnrefused,
2015-07-29_17:04:07.50119 {gen_server,call,
2015-07-29_17:04:07.50120
[<0.252.0>,connect,infinity]}}}}},
2015-07-29_17:04:07.50120 [{bunny_util,connect,1,
2015-07-29_17:04:07.50120
[{file,“src/bunny_util.erl”},{line,191}]},
2015-07-29_17:04:07.50120 {gen_bunny_mon,do_connect,3,
2015-07-29_17:04:07.50121
[{file,“src/gen_bunny_mon.erl”},{line,192}]},
2015-07-29_17:04:07.50121 {gen_bunny_mon,handle_call,3,
2015-07-29_17:04:07.50121
[{file,“src/gen_bunny_mon.erl”},{line,87}]},
2015-07-29_17:04:07.50122 {gen_server,handle_msg,5,
2015-07-29_17:04:07.50122
[{file,“gen_server.erl”},{line,588}]},
2015-07-29_17:04:07.50122 {proc_lib,init_p_do_apply,3,
2015-07-29_17:04:07.50122
[{file,“proc_lib.erl”},{line,227}]}]},
2015-07-29_17:04:07.50123 {connection_info,
2015-07-29_17:04:07.50123 {network,“127.0.0.1”,5672,
2015-07-29_17:04:07.50123 {<<“chef”>>,
2015-07-29_17:04:07.50123
<<“050f4cfb8b32d9a5f8d5aed9c1d7f5e5f7aa6fab125b5b204ed418f30b73b76e29e88c8fdbd1dacf6000b106e5a9de5edf10”>>},
2015-07-29_17:04:07.50124 <<”/chef”>>}}}
2015-07-29_17:04:08.49101
2015-07-29_17:04:08.49103 =ERROR REPORT==== 29-Jul-2015::10:04:08 ===
2015-07-29_17:04:08.49104 Could not start the network driver: econnrefused
2015-07-29_17:04:08.49202
2015-07-29_17:04:08.49203 =ERROR REPORT==== 29-Jul-2015::10:04:08 ===
2015-07-29_17:04:08.49204 ** Generic server <0.259.0> terminating
2015-07-29_17:04:08.49204 ** Last message in was connect
2015-07-29_17:04:08.49204 ** When Server state == {state,<0.257.0>,
2015-07-29_17:04:08.49204
{amqp_params,<<“chef”>>,
2015-07-29_17:04:08.49205
<<“050f4cfb8b32d9a5f8d5aed9c1d7f5e5f7aa6fab125b5b204ed418f30b73b76e29e88c8fdbd1dacf6000b106e5a9de5edf10”>>,
2015-07-29_17:04:08.49205
<<"/chef">>,“127.0.0.1”,5672,0,0,0,none,[]},
2015-07-29_17:04:08.49205
undefined,undefined,undefined,undefined,undefined,
2015-07-29_17:04:08.49206
undefined,<0.258.0>,false,undefined,
2015-07-29_17:04:08.49206 {{0,nil},
2015-07-29_17:04:08.49206
{dict,0,16,16,8,80,48,
2015-07-29_17:04:08.49206
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
2015-07-29_17:04:08.49207 [],[]},
2015-07-29_17:04:08.49207
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
2015-07-29_17:04:08.49207 [],[]}}}},
2015-07-29_17:04:08.49207
undefined,#Fun<amqp_connection_sup.0.67288451>}
2015-07-29_17:04:08.49208 ** Reason for termination ==
2015-07-29_17:04:08.49208 ** econnrefused
2015-07-29_17:04:08.49499
2015-07-29_17:04:08.49500 =ERROR REPORT==== 29-Jul-2015::10:04:08 ===
2015-07-29_17:04:08.49500 “Could not connect, scheduling reconnect.“
2015-07-29_17:04:08.49501 error: {{error,
2015-07-29_17:04:08.49501 {badmatch,
2015-07-29_17:04:08.49501 {error,
2015-07-29_17:04:08.49501 {auth_failure_likely,
2015-07-29_17:04:08.49502 {econnrefused,
2015-07-29_17:04:08.49502 {gen_server,call,
2015-07-29_17:04:08.49502
[<0.259.0>,connect,infinity]}}}}},
2015-07-29_17:04:08.49502 [{bunny_util,connect,1,
2015-07-29_17:04:08.49503
[{file,“src/bunny_util.erl”},{line,191}]},
2015-07-29_17:04:08.49503 {gen_bunny_mon,do_connect,3,
2015-07-29_17:04:08.49503
[{file,“src/gen_bunny_mon.erl”},{line,192}]},
2015-07-29_17:04:08.49504 {gen_bunny_mon,handle_info,2,
2015-07-29_17:04:08.49504
[{file,“src/gen_bunny_mon.erl”},{line,134}]},
2015-07-29_17:04:08.49504 {gen_server,handle_msg,5,
2015-07-29_17:04:08.49504
[{file,“gen_server.erl”},{line,607}]},
2015-07-29_17:04:08.49505 {proc_lib,init_p_do_apply,3,
2015-07-29_17:04:08.49505
[{file,“proc_lib.erl”},{line,227}]}]},
2015-07-29_17:04:08.49505 {connection_info,
2015-07-29_17:04:08.49505 {network,“127.0.0.1”,5672,
2015-07-29_17:04:08.49506 {<<“chef”>>,
2015-07-29_17:04:08.49506
<<“050f4cfb8b32d9a5f8d5aed9c1d7f5e5f7aa6fab125b5b204ed418f30b73b76e29e88c8fdbd1dacf6000b106e5a9de5edf10”>>},
2015-07-29_17:04:08.49507 <<”/chef”>>}}}
2015-07-29_17:04:09.49193
2015-07-29_17:04:09.49195 =ERROR REPORT==== 29-Jul-2015::10:04:09 ===
2015-07-29_17:04:09.49195 Could not start the network driver: econnrefused
2015-07-29_17:04:09.49284
2015-07-29_17:04:09.49285 =ERROR REPORT==== 29-Jul-2015::10:04:09 ===
2015-07-29_17:04:09.49285 ** Generic server <0.262.0> terminating
2015-07-29_17:04:09.49285 ** Last message in was connect
2015-07-29_17:04:09.49286 ** When Server state == {state,<0.260.0>,
2015-07-29_17:04:09.49286
{amqp_params,<<“chef”>>,
2015-07-29_17:04:09.49286
<<“050f4cfb8b32d9a5f8d5aed9c1d7f5e5f7aa6fab125b5b204ed418f30b73b76e29e88c8fdbd1dacf6000b106e5a9de5edf10”>>,
2015-07-29_17:04:09.49287
<<"/chef">>,“127.0.0.1”,5672,0,0,0,none,[]},
2015-07-29_17:04:09.49288
undefined,undefined,undefined,undefined,undefined,
2015-07-29_17:04:09.49288
undefined,<0.261.0>,false,undefined,
2015-07-29_17:04:09.49288 {{0,nil},
2015-07-29_17:04:09.49288
{dict,0,16,16,8,80,48,
2015-07-29_17:04:09.49289
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
2015-07-29_17:04:09.49289 [],[]},
2015-07-29_17:04:09.49289
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
2015-07-29_17:04:09.49289 [],[]}}}},
2015-07-29_17:04:09.49290
undefined,#Fun<amqp_connection_sup.0.67288451>}
2015-07-29_17:04:09.49290 ** Reason for termination ==
2015-07-29_17:04:09.49290 ** econnrefused
2015-07-29_17:04:09.49575
2015-07-29_17:04:09.49576 =ERROR REPORT==== 29-Jul-2015::10:04:09 ===
2015-07-29_17:04:09.49576 “Could not connect, scheduling reconnect.“
2015-07-29_17:04:09.49576 error: {{error,
2015-07-29_17:04:09.49577 {badmatch,
2015-07-29_17:04:09.49577 {error,
2015-07-29_17:04:09.49577 {auth_failure_likely,
2015-07-29_17:04:09.49578 {econnrefused,
2015-07-29_17:04:09.49578 {gen_server,call,
2015-07-29_17:04:09.49578
[<0.262.0>,connect,infinity]}}}}},
2015-07-29_17:04:09.49578 [{bunny_util,connect,1,
2015-07-29_17:04:09.49579
[{file,“src/bunny_util.erl”},{line,191}]},
2015-07-29_17:04:09.49579 {gen_bunny_mon,do_connect,3,
2015-07-29_17:04:09.49579
[{file,“src/gen_bunny_mon.erl”},{line,192}]},
2015-07-29_17:04:09.49579 {gen_bunny_mon,handle_info,2,
2015-07-29_17:04:09.49580
[{file,“src/gen_bunny_mon.erl”},{line,134}]},
2015-07-29_17:04:09.49580 {gen_server,handle_msg,5,
2015-07-29_17:04:09.49580
[{file,“gen_server.erl”},{line,607}]},
2015-07-29_17:04:09.49580 {proc_lib,init_p_do_apply,3,
2015-07-29_17:04:09.49581
[{file,“proc_lib.erl”},{line,227}]}]},
2015-07-29_17:04:09.49581 {connection_info,
2015-07-29_17:04:09.49581 {network,“127.0.0.1”,5672,
2015-07-29_17:04:09.49581 {<<“chef”>>,
2015-07-29_17:04:09.49582
<<“050f4cfb8b32d9a5f8d5aed9c1d7f5e5f7aa6fab125b5b204ed418f30b73b76e29e88c8fdbd1dacf6000b106e5a9de5edf10”>>},
2015-07-29_17:04:09.49583 <<”/chef”>>}}}
2015-07-29_17:04:10.49287
2015-07-29_17:04:10.49289 =ERROR REPORT==== 29-Jul-2015::10:04:10 ===
2015-07-29_17:04:10.49289 Could not start the network driver: econnrefused
2015-07-29_17:04:10.49389
2015-07-29_17:04:10.49391 =ERROR REPORT==== 29-Jul-2015::10:04:10 ===
2015-07-29_17:04:10.49391 ** Generic server <0.265.0> terminating
2015-07-29_17:04:10.49391 ** Last message in was connect
2015-07-29_17:04:10.49392 ** When Server state == {state,<0.263.0>,
2015-07-29_17:04:10.49392
{amqp_params,<<“chef”>>,
2015-07-29_17:04:10.49392
<<“050f4cfb8b32d9a5f8d5aed9c1d7f5e5f7aa6fab125b5b204ed418f30b73b76e29e88c8fdbd1dacf6000b106e5a9de5edf10”>>,
2015-07-29_17:04:10.49393
<<"/chef">>,“127.0.0.1”,5672,0,0,0,none,[]},
2015-07-29_17:04:10.49393
undefined,undefined,undefined,undefined,undefined,
2015-07-29_17:04:10.49393
undefined,<0.264.0>,false,undefined,
2015-07-29_17:04:10.49393 {{0,nil},
2015-07-29_17:04:10.49394
{dict,0,16,16,8,80,48,
2015-07-29_17:04:10.49394
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
2015-07-29_17:04:10.49395 [],[]},
2015-07-29_17:04:10.49396
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
2015-07-29_17:04:10.49396 [],[]}}}},
2015-07-29_17:04:10.49396
undefined,#Fun<amqp_connection_sup.0.67288451>}
2015-07-29_17:04:10.49396 ** Reason for termination ==
2015-07-29_17:04:10.49397 ** econnrefused
2015-07-29_17:04:10.49964
2015-07-29_17:04:10.49965 =ERROR REPORT==== 29-Jul-2015::10:04:10 ===
2015-07-29_17:04:10.49966 “Could not connect, scheduling reconnect.“
2015-07-29_17:04:10.49966 error: {{error,
2015-07-29_17:04:10.49966 {badmatch,
2015-07-29_17:04:10.49966 {error,
2015-07-29_17:04:10.49967 {auth_failure_likely,
2015-07-29_17:04:10.49967 {econnrefused,
2015-07-29_17:04:10.49967 {gen_server,call,
2015-07-29_17:04:10.49968
[<0.265.0>,connect,infinity]}}}}},
2015-07-29_17:04:10.49968 [{bunny_util,connect,1,
2015-07-29_17:04:10.49968
[{file,“src/bunny_util.erl”},{line,191}]},
2015-07-29_17:04:10.49968 {gen_bunny_mon,do_connect,3,
2015-07-29_17:04:10.49969
[{file,“src/gen_bunny_mon.erl”},{line,192}]},
2015-07-29_17:04:10.49969 {gen_bunny_mon,handle_info,2,
2015-07-29_17:04:10.49969
[{file,“src/gen_bunny_mon.erl”},{line,134}]},
2015-07-29_17:04:10.49969 {gen_server,handle_msg,5,
2015-07-29_17:04:10.49970
[{file,“gen_server.erl”},{line,607}]},
2015-07-29_17:04:10.49970 {proc_lib,init_p_do_apply,3,
2015-07-29_17:04:10.49970
[{file,“proc_lib.erl”},{line,227}]}]},
2015-07-29_17:04:10.49970 {connection_info,
2015-07-29_17:04:10.49971 {network,“127.0.0.1”,5672,
2015-07-29_17:04:10.49971 {<<“chef”>>,
2015-07-29_17:04:10.49971
<<“050f4cfb8b32d9a5f8d5aed9c1d7f5e5f7aa6fab125b5b204ed418f30b73b76e29e88c8fdbd1dacf6000b106e5a9de5edf10”>>},
2015-07-29_17:04:10.49972 <<”/chef”>>}}}
2015-07-29_17:04:11.49372
2015-07-29_17:04:11.49373 =ERROR REPORT==== 29-Jul-2015::10:04:11 ===
2015-07-29_17:04:11.49374 Could not start the network driver: econnrefused
2015-07-29_17:04:11.49463
2015-07-29_17:04:11.49464 =ERROR REPORT==== 29-Jul-2015::10:04:11 ===
2015-07-29_17:04:11.49464 ** Generic server <0.268.0> terminating
2015-07-29_17:04:11.49464 ** Last message in was connect
2015-07-29_17:04:11.49465 ** When Server state == {state,<0.266.0>,
2015-07-29_17:04:11.49465
{amqp_params,<<“chef”>>,
2015-07-29_17:04:11.49465
<<“050f4cfb8b32d9a5f8d5aed9c1d7f5e5f7aa6fab125b5b204ed418f30b73b76e29e88c8fdbd1dacf6000b106e5a9de5edf10”>>,
2015-07-29_17:04:11.49466
<<"/chef">>,“127.0.0.1”,5672,0,0,0,none,[]},
2015-07-29_17:04:11.49466
undefined,undefined,undefined,undefined,undefined,
2015-07-29_17:04:11.49466
undefined,<0.267.0>,false,undefined,
2015-07-29_17:04:11.49466 {{0,nil},
2015-07-29_17:04:11.49467
{dict,0,16,16,8,80,48,
2015-07-29_17:04:11.49467
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
2015-07-29_17:04:11.49467 [],[]},
2015-07-29_17:04:11.49467
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
2015-07-29_17:04:11.49468 [],[]}}}},
2015-07-29_17:04:11.49468
undefined,#Fun<amqp_connection_sup.0.67288451>}
2015-07-29_17:04:11.49468 ** Reason for termination ==
2015-07-29_17:04:11.49468 ** econnrefused
2015-07-29_17:04:11.49752
2015-07-29_17:04:11.49753 =ERROR REPORT==== 29-Jul-2015::10:04:11 ===
2015-07-29_17:04:11.49754 “Could not connect, scheduling reconnect.“
2015-07-29_17:04:11.49754 error: {{error,
2015-07-29_17:04:11.49754 {badmatch,
2015-07-29_17:04:11.49755 {error,
2015-07-29_17:04:11.49755 {auth_failure_likely,
2015-07-29_17:04:11.49755 {econnrefused,
2015-07-29_17:04:11.49755 {gen_server,call,
2015-07-29_17:04:11.49756
[<0.268.0>,connect,infinity]}}}}},
2015-07-29_17:04:11.49756 [{bunny_util,connect,1,
2015-07-29_17:04:11.49756
[{file,“src/bunny_util.erl”},{line,191}]},
2015-07-29_17:04:11.49756 {gen_bunny_mon,do_connect,3,
2015-07-29_17:04:11.49757
[{file,“src/gen_bunny_mon.erl”},{line,192}]},
2015-07-29_17:04:11.49757 {gen_bunny_mon,handle_info,2,
2015-07-29_17:04:11.49757
[{file,“src/gen_bunny_mon.erl”},{line,134}]},
2015-07-29_17:04:11.49758 {gen_server,handle_msg,5,
2015-07-29_17:04:11.49758
[{file,“gen_server.erl”},{line,607}]},
2015-07-29_17:04:11.49758 {proc_lib,init_p_do_apply,3,
2015-07-29_17:04:11.49758
[{file,“proc_lib.erl”},{line,227}]}]},
2015-07-29_17:04:11.49759 {connection_info,
2015-07-29_17:04:11.49759 {network,“127.0.0.1”,5672,
2015-07-29_17:04:11.49759 {<<“chef”>>,
2015-07-29_17:04:11.49759
<<“050f4cfb8b32d9a5f8d5aed9c1d7f5e5f7aa6fab125b5b204ed418f30b73b76e29e88c8fdbd1dacf6000b106e5a9de5edf10”>>},
2015-07-29_17:04:11.49760 <<”/chef”>>}}}
2015-07-29_17:04:12.52524
2015-07-29_17:04:12.52525 =ERROR REPORT==== 29-Jul-2015::10:04:12 ===
2015-07-29_17:04:12.52526 Could not start the network driver: econnrefused
2015-07-29_17:04:12.52625
2015-07-29_17:04:12.52626 =ERROR REPORT==== 29-Jul-2015::10:04:12 ===
2015-07-29_17:04:12.52627 ** Generic server <0.271.0> terminating
2015-07-29_17:04:12.52627 ** Last message in was connect
2015-07-29_17:04:12.52627 ** When Server state == {state,<0.269.0>,
2015-07-29_17:04:12.52628
{amqp_params,<<“chef”>>,
2015-07-29_17:04:12.52628
<<“050f4cfb8b32d9a5f8d5aed9c1d7f5e5f7aa6fab125b5b204ed418f30b73b76e29e88c8fdbd1dacf6000b106e5a9de5edf10”>>,
2015-07-29_17:04:12.52628
<<"/chef">>,“127.0.0.1”,5672,0,0,0,none,[]},
2015-07-29_17:04:12.52628
undefined,undefined,undefined,undefined,undefined,
2015-07-29_17:04:12.52629
undefined,<0.270.0>,false,undefined,
2015-07-29_17:04:12.52629 {{0,nil},
2015-07-29_17:04:12.52629
{dict,0,16,16,8,80,48,
2015-07-29_17:04:12.52629
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
2015-07-29_17:04:12.52630 [],[]},
2015-07-29_17:04:12.52630
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
2015-07-29_17:04:12.52630 [],[]}}}},
2015-07-29_17:04:12.52630
undefined,#Fun<amqp_connection_sup.0.67288451>}
2015-07-29_17:04:12.52631 ** Reason for termination ==
2015-07-29_17:04:12.52631 ** econnrefused
2015-07-29_17:04:12.52914
2015-07-29_17:04:12.52915 =ERROR REPORT==== 29-Jul-2015::10:04:12 ===
2015-07-29_17:04:12.52915 “Could not connect, scheduling reconnect.“
2015-07-29_17:04:12.52916 error: {{error,
2015-07-29_17:04:12.52916 {badmatch,
2015-07-29_17:04:12.52916 {error,
2015-07-29_17:04:12.52916 {auth_failure_likely,
2015-07-29_17:04:12.52917 {econnrefused,
2015-07-29_17:04:12.52917 {gen_server,call,
2015-07-29_17:04:12.52917
[<0.271.0>,connect,infinity]}}}}},
2015-07-29_17:04:12.52918 [{bunny_util,connect,1,
2015-07-29_17:04:12.52918
[{file,“src/bunny_util.erl”},{line,191}]},
2015-07-29_17:04:12.52919 {gen_bunny_mon,do_connect,3,
2015-07-29_17:04:12.52919
[{file,“src/gen_bunny_mon.erl”},{line,192}]},
2015-07-29_17:04:12.52919 {gen_bunny_mon,handle_info,2,
2015-07-29_17:04:12.52919
[{file,“src/gen_bunny_mon.erl”},{line,134}]},
2015-07-29_17:04:12.52920 {gen_server,handle_msg,5,
2015-07-29_17:04:12.52920
[{file,“gen_server.erl”},{line,607}]},
2015-07-29_17:04:12.52920 {proc_lib,init_p_do_apply,3,
2015-07-29_17:04:12.52920
[{file,“proc_lib.erl”},{line,227}]}]},
2015-07-29_17:04:12.52921 {connection_info,
2015-07-29_17:04:12.52921 {network,“127.0.0.1”,5672,
2015-07-29_17:04:12.52921 {<<“chef”>>,
2015-07-29_17:04:12.52921
<<“050f4cfb8b32d9a5f8d5aed9c1d7f5e5f7aa6fab125b5b204ed418f30b73b76e29e88c8fdbd1dacf6000b106e5a9de5edf10”>>},
2015-07-29_17:04:12.52922 <<”/chef”>>}}}
2015-07-29_17:04:13.49614
2015-07-29_17:04:13.49615 =ERROR REPORT==== 29-Jul-2015::10:04:13 ===
2015-07-29_17:04:13.49616 Could not start the network driver: econnrefused
2015-07-29_17:04:13.49707
2015-07-29_17:04:13.49708 =ERROR REPORT==== 29-Jul-2015::10:04:13 ===
2015-07-29_17:04:13.49709 ** Generic server <0.275.0> terminating
2015-07-29_17:04:13.49709 ** Last message in was connect
2015-07-29_17:04:13.49709 ** When Server state == {state,<0.273.0>,
2015-07-29_17:04:13.49710
{amqp_params,<<“chef”>>,
2015-07-29_17:04:13.49710
<<“050f4cfb8b32d9a5f8d5aed9c1d7f5e5f7aa6fab125b5b204ed418f30b73b76e29e88c8fdbd1dacf6000b106e5a9de5edf10”>>,
2015-07-29_17:04:13.49710
<<"/chef">>,“127.0.0.1”,5672,0,0,0,none,[]},
2015-07-29_17:04:13.49710
undefined,undefined,undefined,undefined,undefined,
2015-07-29_17:04:13.49711
undefined,<0.274.0>,false,undefined,
2015-07-29_17:04:13.49711 {{0,nil},
2015-07-29_17:04:13.49711
{dict,0,16,16,8,80,48,
2015-07-29_17:04:13.49712
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
2015-07-29_17:04:13.49712 [],[]},
2015-07-29_17:04:13.49712
{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
2015-07-29_17:04:13.49712 [],[]}}}},
2015-07-29_17:04:13.49713
undefined,#Fun<amqp_connection_sup.0.67288451>}
2015-07-29_17:04:13.49713 ** Reason for termination ==
2015-07-29_17:04:13.49713 ** econnrefused
2015-07-29_17:04:13.49997
2015-07-29_17:04:13.49998 =ERROR REPORT==== 29-Jul-2015::10:04:13 ===
2015-07-29_17:04:13.49998 “Could not connect, scheduling reconnect.“
2015-07-29_17:04:13.49999 error: {{error,
2015-07-29_17:04:13.49999 {badmatch,
2015-07-29_17:04:13.49999 {error,
2015-07-29_17:04:13.49999 {auth_failure_likely,
2015-07-29_17:04:13.50000 {econnrefused,
2015-07-29_17:04:13.50000 {gen_server,call,
2015-07-29_17:04:13.50000
[<0.275.0>,connect,infinity]}}}}},
2015-07-29_17:04:13.50000 [{bunny_util,connect,1,
2015-07-29_17:04:13.50001
[{file,“src/bunny_util.erl”},{line,191}]},
2015-07-29_17:04:13.50001 {gen_bunny_mon,do_connect,3,
2015-07-29_17:04:13.50001
[{file,“src/gen_bunny_mon.erl”},{line,192}]},
2015-07-29_17:04:13.50001 {gen_bunny_mon,handle_info,2,
2015-07-29_17:04:13.50002
[{file,“src/gen_bunny_mon.erl”},{line,134}]},
2015-07-29_17:04:13.50002 {gen_server,handle_msg,5,
2015-07-29_17:04:13.50003
[{file,“gen_server.erl”},{line,607}]},
2015-07-29_17:04:13.50003 {proc_lib,init_p_do_apply,3,
2015-07-29_17:04:13.50004
[{file,“proc_lib.erl”},{line,227}]}]},
2015-07-29_17:04:13.50004 {connection_info,
2015-07-29_17:04:13.50004 {network,“127.0.0.1”,5672,
2015-07-29_17:04:13.50004 {<<“chef”>>,
2015-07-29_17:04:13.50005
<<“050f4cfb8b32d9a5f8d5aed9c1d7f5e5f7aa6fab125b5b204ed418f30b73b76e29e88c8fdbd1dacf6000b106e5a9de5edf10”>>},
2015-07-29_17:04:13.50005 <<”/chef”>>}}}
2015-07-29_17:04:14.63158
2015-07-29_17:04:14.63159 =INFO REPORT==== 29-Jul-2015::10:04:14 ===

On Wednesday, July 29, 2015 at 10:32 AM, Stanley Shen wrote:

Hello, all

I had a ubuntu server running chef-server_11.0.12-1.ubuntu.12.04_amd64.deb
Recently I had restart the chef server, and then I cannot bootstrap VM and ran chef client on it.

Here are some related errors, what could be the root cause?
And how can I fix it?

=============================================================================

[2015-07-29T08:10:28-07:00] INFO: Running start handlers
[2015-07-29T08:10:28-07:00] INFO: Start handlers complete.
[2015-07-29T08:15:28-07:00] ERROR: Timeout connecting to test.com:443 (http://test.com:443) for /environments/_default/cookbook_versions, retry 1/5
[2015-07-29T08:20:34-07:00] ERROR: Timeout connecting to test.com:443 (http://test.com:443) for /environments/_default/cookbook_versions, retry 2/5
[2015-07-29T08:25:39-07:00] ERROR: Timeout connecting to test.com:443 (http://test.com:443) for /environments/_default/cookbook_versions, retry 3/5

================================================================================
Error Resolving Cookbooks for Run List:

Authentication Error:

Failed to authenticate to the chef server (http 401).
The request failed because your clock has drifted by more than 15 minutes. ( I tried to sync time, but it doesn’t work)
Syncing your clock to an NTP Time source should resolve the issue.

Since you recently rebooted your Chef Server, it’s possible that the clock on the Chef Server is the issue, rather than the node running chef-client.

[2015-07-29T08:25:44-07:00] ERROR: Running exception handlers
[2015-07-29T08:25:44-07:00] FATAL: Saving node information to /var/chef/cache/failed-run-data.json
[2015-07-29T08:25:44-07:00] ERROR: Exception handlers complete
[2015-07-29T08:25:44-07:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2015-07-29T08:25:44-07:00] FATAL: Net::HTTPServerException: 401 “Unauthorized”

And in chef server I can such errors:

2015/07/29 07:34:37 [error] 2664#0: *51 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 38.102.156.135, server: chef3.cloud.intalio.com (http://chef3.cloud.intalio.com), request: “POST /environments/_default/cookbook_versions HTTP/1.1”, upstream: “http://127.0.0.1:8000/environments/_default/cookbook_versions”, host: "chef3.cloud.intalio.com:443 (http://chef3.cloud.intalio.com:443)"
2015/07/29 07:39:41 [error] 2665#0: *61 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 38.102.156.135, server: chef3.cloud.intalio.com (http://chef3.cloud.intalio.com), request: “POST /environments/_default/cookbook_versions HTTP/1.1”, upstream: “http://127.0.0.1:8000/environments/_default/cookbook_versions”, host: "chef3.cloud.intalio.com:443 (http://chef3.cloud.intalio.com:443)"
2015/07/29 07:44:46 [error] 2665#0: *63 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 38.102.156.135, server: chef3.cloud.intalio.com (http://chef3.cloud.intalio.com), request: “POST /environments/_default/cookbook_versions HTTP/1.1”, upstream: “http://127.0.0.1:8000/environments/_default/cookbook_versions”, host: "chef3.cloud.intalio.com:443 (http://chef3.cloud.intalio.com:443)"
2015/07/29 08:15:29 [error] 2664#0: *805 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 38.102.156.135, server: chef3.cloud.intalio.com (http://chef3.cloud.intalio.com), request: “POST /environments/_default/cookbook_versions HTTP/1.1”, upstream: “http://127.0.0.1:8000/environments/_default/cookbook_versions”, host: "chef3.cloud.intalio.com:443 (http://chef3.cloud.intalio.com:443)"
2015/07/29 08:20:34 [error] 2664#0: *807 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 38.102.156.135, server: chef3.cloud.intalio.com (http://chef3.cloud.intalio.com), request: “POST /environments/_default/cookbook_versions HTTP/1.1”, upstream: “http://127.0.0.1:8000/environments/_default/cookbook_versions”, host: "chef3.cloud.intalio.com:443 (http://chef3.cloud.intalio.com:443)"
2015/07/29 08:25:39 [error] 2664#0: *809 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 38.102.156.135, server: chef3.cloud.intalio.com (http://chef3.cloud.intalio.com), request: “POST /environments/_default/cookbook_versions HTTP/1.1”, upstream: “http://127.0.0.1:8000/environments/_default/cookbook_versions”, host: "chef3.cloud.intalio.com:443 (http://chef3.cloud.intalio.com:443)”

And in erchef code, I can see such:

2015-07-29_17:04:04.97022 ** Found 0 name clashes in code paths
2015-07-29_17:04:05.20423
2015-07-29_17:04:05.20425 =INFO REPORT==== 29-Jul-2015::10:04:05 ===
2015-07-29_17:04:05.20425 Starting sqerl supervisor
2015-07-29_17:04:07.13848
2015-07-29_17:04:07.13850 =INFO REPORT==== 29-Jul-2015::10:04:07 ===
2015-07-29_17:04:07.13850 starting stats_hero_sup
2015-07-29_17:04:07.14937
2015-07-29_17:04:07.14938 =INFO REPORT==== 29-Jul-2015::10:04:07 ===
2015-07-29_17:04:07.14939 starting stats_hero_sender_sup with 1 senders
2015-07-29_17:04:07.16031
2015-07-29_17:04:07.16032 =INFO REPORT==== 29-Jul-2015::10:04:07 ===
2015-07-29_17:04:07.16033 stats_hero_sender starting pid: <0.216.0> port: 42498
2015-07-29_17:04:07.17695
2015-07-29_17:04:07.17696 =INFO REPORT==== 29-Jul-2015::10:04:07 ===
2015-07-29_17:04:07.17697 bulk_fetch_batch_size set to 5
2015-07-29_17:04:07.35016
2015-07-29_17:04:07.35018 =INFO REPORT==== 29-Jul-2015::10:04:07 ===
2015-07-29_17:04:07.35018 Chef Keyring starting
2015-07-29_17:04:07.42652
2015-07-29_17:04:07.42655 =INFO REPORT==== 29-Jul-2015::10:04:07 ===
2015-07-29_17:04:07.42656 Loaded key default of type RSAPrivateKey from file /etc/chef-server/chef-webui.pem
2015-07-29_17:04:07.44778
2015-07-29_17:04:07.44779 =INFO REPORT==== 29-Jul-2015::10:04:07 ===
2015-07-29_17:04:07.44780 starting chef_index_sup
2015-07-29_17:04:07.44816
2015-07-29_17:04:07.44817 =INFO REPORT==== 29-Jul-2015::10:04:07 ===
2015-07-29_17:04:07.44818 Connecting to Rabbit at 127.0.0.1:5672/chef (http://127.0.0.1:5672/chef) (exchange: <<>>)
2015-07-29_17:04:07.48567
2015-07-29_17:04:07.48569 =ERROR REPORT==== 29-Jul-2015::10:04:07 ===
2015-07-29_17:04:07.48569 Could not start the network driver: econnrefused
2015-07-29_17:04:07.48779
2015-07-29_17:04:07.48780 =ERROR REPORT==== 29-Jul-2015::10:04:07 ===
2015-07-29_17:04:07.48780 ** Generic server <0.252.0> terminating
2015-07-29_17:04:07.48780 ** Last message in was connect
2015-07-29_17:04:07.48780 ** When Server state == {state,<0.250.0>,
2015-07-29_17:04:07.48781 {amqp_params,<<“chef”>>,
2015-07-29_17:04:07.48781 <<“050f4cfb8b32d9a5f8d5aed9c1d7f5e5f7aa6fab125b5b204ed418f30b73b76e29e88c8fdbd1dacf6000b106e5a9de5edf10”>>,
2015-07-29_17:04:07.48781 <<"/chef">>,“127.0.0.1”,5672,0,0,0,none,[]},
2015-07-29_17:04:07.48782 undefined,undefined,undefined,undefined,undefined,
2015-07-29_17:04:07.48782 undefined,<0.251.0>,false,undefined,
2015-07-29_17:04:07.48782 {{0,nil},
2015-07-29_17:04:07.48782 {dict,0,16,16,8,80,48,
2015-07-29_17:04:07.48783 {[],[],[],[],[],[],[],[],[],[],[],[],[],[],
2015-07-29_17:04:07.48783 [],[]},
2015-07-29_17:04:07.48783 {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
2015-07-29_17:04:07.48783 [],[]}}}},
2015-07-29_17:04:07.48784 undefined,#Fun<amqp_connection_sup.0.67288451>}
2015-07-29_17:04:07.48784 ** Reason for termination ==
2015-07-29_17:04:07.48784 ** econnrefused

This means rabbitmq is down (or just not up yet).

HTH,


Daniel DeLeo