Intermittent Errno::ETIMEDOUT: error connecting to chef server

Hi,

I have seen this problem intermittently when creating AWS instances. This
causes the configuration automation very unreliable. Could somebody shed some
clue about this?

thx,

Xianfeng

[root@ip-10-159-66-147 cache]# more chef-stacktrace.out
Generated at Tue Dec 04 08:19:32 +0000 2012
Errno::ETIMEDOUT: remote_file[omakase-release.rpm] (mspot_omakase::default line
38) had an error: Errno::ETIMEDOUT: Error c
onnecting to
http://omakase.mspot.com/yum/5/x86_64/omakase-release-5-2.noarch.rpm -
Connection timed out - connect(2)
/usr/lib/ruby/1.8/net/http.rb:560:in initialize' /usr/lib/ruby/1.8/net/http.rb:560:inopen’
/usr/lib/ruby/1.8/net/http.rb:560:in connect' /usr/lib/ruby/1.8/timeout.rb:53:intimeout’
/usr/lib/ruby/1.8/timeout.rb:101:in timeout' /usr/lib/ruby/1.8/net/http.rb:560:inconnect’
/usr/lib/ruby/1.8/net/http.rb:553:in do_start' /usr/lib/ruby/1.8/net/http.rb:542:instart’
/usr/lib/ruby/1.8/net/http.rb:1035:in request' /usr/lib/ruby/gems/1.8/gems/chef-0.10.6/bin/../lib/chef/rest/rest_request.rb:84:incall’
/usr/lib/ruby/gems/1.8/gems/chef-0.10.6/bin/…/lib/chef/rest/rest_request.rb:99:in
hide_net_http_bug' /usr/lib/ruby/gems/1.8/gems/chef-0.10.6/bin/../lib/chef/rest/rest_request.rb:83:incall’
/usr/lib/ruby/gems/1.8/gems/chef-0.10.6/bin/…/lib/chef/rest.rb:261:in
streaming_request' /usr/lib/ruby/gems/1.8/gems/chef-0.10.6/bin/../lib/chef/rest.rb:296:inretriable_rest_request’
/usr/lib/ruby/gems/1.8/gems/chef-0.10.6/bin/…/lib/chef/rest.rb:259:in
streaming_request' /usr/lib/ruby/gems/1.8/gems/chef-0.10.6/bin/../lib/chef/rest.rb:139:infetch’
/usr/lib/ruby/gems/1.8/gems/chef-0.10.6/bin/…/lib/chef/provider/remote_file.rb:42:in
action_create' /usr/lib/ruby/gems/1.8/gems/chef-0.10.6/bin/../lib/chef/resource.rb:440:insend’
/usr/lib/ruby/gems/1.8/gems/chef-0.10.6/bin/…/lib/chef/resource.rb:440:in
run_action' /usr/lib/ruby/gems/1.8/gems/chef-0.10.6/bin/../lib/chef/runner.rb:45:inrun_action’
/usr/lib/ruby/gems/1.8/gems/chef-0.10.6/bin/…/lib/chef/runner.rb:81:in
converge' /usr/lib/ruby/gems/1.8/gems/chef-0.10.6/bin/../lib/chef/runner.rb:81:ineach’
/usr/lib/ruby/gems/1.8/gems/chef-0.10.6/bin/…/lib/chef/runner.rb:81:in
converge' /usr/lib/ruby/gems/1.8/gems/chef-0.10.6/bin/../lib/chef/resource_collection.rb:94:inexecute_each_resource’
/usr/lib/ruby/gems/1.8/gems/chef-0.10.6/bin/…/lib/chef/resource_collection/stepable_iterator.rb:116:in
call' /usr/lib/ruby/gems/1.8/gems/chef-0.10.6/bin/../lib/chef/resource_collection/stepable_iterator.rb:116:incall_iterator_bloc
k’
/usr/lib/ruby/gems/1.8/gems/chef-0.10.6/bin/…/lib/chef/resource_collection/stepable_iterator.rb:85:in
step' /usr/lib/ruby/gems/1.8/gems/chef-0.10.6/bin/../lib/chef/resource_collection/stepable_iterator.rb:104:initerate’
/usr/lib/ruby/gems/1.8/gems/chef-0.10.6/bin/…/lib/chef/resource_collection/stepable_iterator.rb:55:in
each_with_index' /usr/lib/ruby/gems/1.8/gems/chef-0.10.6/bin/../lib/chef/resource_collection.rb:92:inexecute_each_resource’
/usr/lib/ruby/gems/1.8/gems/chef-0.10.6/bin/…/lib/chef/runner.rb:76:in
converge' /usr/lib/ruby/gems/1.8/gems/chef-0.10.6/bin/../lib/chef/client.rb:312:inconverge’
/usr/lib/ruby/gems/1.8/gems/chef-0.10.6/bin/…/lib/chef/client.rb:160:in run' /usr/lib/ruby/gems/1.8/gems/chef-0.10.6/bin/../lib/chef/application/client.rb:239:inrun_application’
/usr/lib/ruby/gems/1.8/gems/chef-0.10.6/bin/…/lib/chef/application/client.rb:229:in
loop' /usr/lib/ruby/gems/1.8/gems/chef-0.10.6/bin/../lib/chef/application/client.rb:229:inrun_application’
/usr/lib/ruby/gems/1.8/gems/chef-0.10.6/bin/…/lib/chef/application.rb:67:in
run' /usr/lib/ruby/gems/1.8/gems/chef-0.10.6/bin/chef-client:26 /usr/bin/chef-client:19:inload’
/usr/bin/chef-client:19
[root@ip-10-159-66-147 cache]#

On Tuesday, December 4, 2012 at 11:26 AM, xyuan@mspot.com wrote:

Hi,

I have seen this problem intermittently when creating AWS instances. This
causes the configuration automation very unreliable. Could somebody shed some
clue about this?

thx,

Xianfeng

[root@ip-10-159-66-147 cache]# more chef-stacktrace.out
Generated at Tue Dec 04 08:19:32 +0000 2012
Errno::ETIMEDOUT: remote_file[omakase-release.rpm] (mspot_omakase::default line
38) had an error: Errno::ETIMEDOUT: Error c
onnecting to
http://omakase.mspot.com/yum/5/x86_64/omakase-release-5-2.noarch.rpm -

/usr/lib/ruby/gems/1.8/gems/chef-0.10.6/bin/../lib/chef/provider/remote_file.rb:42:in
`action_create'

You have a remote_file resource that is fetching an RPM from an unreliable server. If you control that server, you should investigate it and fix it. If you do not, you could store the RPM elsewhere, either in the cookbook (though if you store your cookbooks in git, this will be a huge PITA as git accumulates different versions of the RPM over time), or S3 (makes sense because you're already using AWS anyway) or some other file storage service.

--
Daniel DeLeo