# Chef Server timeouts
**URL:** https://discourse.chef.io/t/chef-server-timeouts/4239
**Category:** Chef Infra (archive)
**Created:** [July 24, 2013, 3:34pm UTC](https://discourse.chef.io/t/chef-server-timeouts/4239 "2013-07-24T15:34:21Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@Cassiano\_Leal](https://discourse.chef.io/u/Cassiano_Leal)
#### Post date: [July 24, 2013, 3:34pm UTC](https://discourse.chef.io/t/chef-server-timeouts/4239/1 "2013-07-24T15:34:21Z")
Hi,
I’m having an issue with my OSS Chef Server. Clients time out while trying to connect to it.
I found this on /var/log/chef-server/nginx/error.log:
2013/07/24 12:29:44 [error] 859#0: \*30 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.0.1.160, server: ip-10-0-1-10.sa-east-1.compute.internal, request: “POST /environments/production/cookbook\_versions HTTP/1.1”, upstream: “[http://127.0.0.1:8000/environments/production/cookbook\_versions](http://127.0.0.1:8000/environments/production/cookbook_versions)”, host: "10.0.1.10:443”
My guess is that the problem is in the “upstream” service that runs on port 8000, but I’m not sure which service that is. Where do I look?
## Cheers,
Cassiano Leal
[http://cassianoleal.com](http://cassianoleal.com)
[http://twitter.com/cassianoleal](http://twitter.com/cassianoleal)
---
### Author:  [@Chris](https://discourse.chef.io/u/Chris)
#### Post date: [July 24, 2013, 3:41pm UTC](https://discourse.chef.io/t/chef-server-timeouts/4239/2 "2013-07-24T15:41:13Z")
I've run into this too. I just migrated to OSC 11 on Monday and less than
24 hrs later had this problem. It only happened once though. I don't have
any more info on it either.
On Wed, Jul 24, 2013 at 8:34 AM, Cassiano Leal [cassianoleal@gmail.com](mailto:cassianoleal@gmail.com)wrote:
> Hi,
>
> I’m having an issue with my OSS Chef Server. Clients time out while trying
> to connect to it.
>
> I found this on /var/log/chef-server/nginx/error.log:
>
> 2013/07/24 12:29:44 [error] 859#0: \*30 upstream timed out (110: Connection
> timed out) while reading response header from upstream, client: 10.0.1.160,
> server: ip-10-0-1-10.sa-east-1.compute.internal, request: "POST
> /environments/production/cookbook\_versions HTTP/1.1", upstream: "
> [http://127.0.0.1:8000/environments/production/cookbook\_versions](http://127.0.0.1:8000/environments/production/cookbook_versions)", host:
> "10.0.1.10:443”
>
> My guess is that the problem is in the “upstream” service that runs on
> port 8000, but I’m not sure which service that is. Where do I look?
>
> ## Cheers,
>
> Cassiano Leal
> [http://cassianoleal.com](http://cassianoleal.com)
> [http://twitter.com/cassianoleal](http://twitter.com/cassianoleal)
--
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
---
### Author:  [@Chris](https://discourse.chef.io/u/Chris)
#### Post date: [July 24, 2013, 3:42pm UTC](https://discourse.chef.io/t/chef-server-timeouts/4239/3 "2013-07-24T15:42:00Z")
BTW, port 8000 should be the erChef process
On Wed, Jul 24, 2013 at 8:41 AM, Chris [grocerylist@gmail.com](mailto:grocerylist@gmail.com) wrote:
> I've run into this too. I just migrated to OSC 11 on Monday and less than
> 24 hrs later had this problem. It only happened once though. I don't have
> any more info on it either.
>
> On Wed, Jul 24, 2013 at 8:34 AM, Cassiano Leal [cassianoleal@gmail.com](mailto:cassianoleal@gmail.com)wrote:
>
> > Hi,
> >
> > I’m having an issue with my OSS Chef Server. Clients time out while
> > trying to connect to it.
> >
> > I found this on /var/log/chef-server/nginx/error.log:
> >
> > 2013/07/24 12:29:44 [error] 859#0: \*30 upstream timed out (110:
> > Connection timed out) while reading response header from upstream, client:
> > 10.0.1.160, server: ip-10-0-1-10.sa-east-1.compute.internal, request: "POST
> > /environments/production/cookbook\_versions HTTP/1.1", upstream: "
> > [http://127.0.0.1:8000/environments/production/cookbook\_versions](http://127.0.0.1:8000/environments/production/cookbook_versions)", host:
> > "10.0.1.10:443”
> >
> > My guess is that the problem is in the “upstream” service that runs on
> > port 8000, but I’m not sure which service that is. Where do I look?
> >
> > ## Cheers,
> >
> > Cassiano Leal
> > [http://cassianoleal.com](http://cassianoleal.com)
> > [http://twitter.com/cassianoleal](http://twitter.com/cassianoleal)
>
> --
> Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
> permitted by applicable law.
--
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
---
### Author:  [@Cassiano\_Leal](https://discourse.chef.io/u/Cassiano_Leal)
#### Post date: [July 24, 2013, 5:16pm UTC](https://discourse.chef.io/t/chef-server-timeouts/4239/4 "2013-07-24T17:16:41Z")
The only things logged for erchef are:
2013-07-24T17:08:09Z [erchef@127.0.0.1](mailto:erchef@127.0.0.1) INFO req\_id=x+AfrOJ1AUtORncdS9mIfg==; status=200; method=GET; path=/nodes/apps; user=apps; msg=[]; req\_time=504; rdbms\_time=422; rdbms\_count=2
2013-07-24T17:08:09Z [erchef@127.0.0.1](mailto:erchef@127.0.0.1) INFO req\_id=4QrUsBd/rJ77XlBB2hWogw==; status=200; method=GET; path=/roles/api; user=apps; msg=[]; req\_time=479; rdbms\_time=373; rdbms\_count=2
2013-07-24T17:08:10Z [erchef@127.0.0.1](mailto:erchef@127.0.0.1) INFO req\_id=55zYGCExTCxAYJAaXdGMGA==; status=200; method=GET; path=/roles/manager; user=apps; msg=[]; req\_time=488; rdbms\_time=356; rdbms\_count=2
2013-07-24T17:08:10Z [erchef@127.0.0.1](mailto:erchef@127.0.0.1) INFO req\_id=a8NzqLLMByrRSPSdbvazQw==; status=200; method=GET; path=/roles/web; user=apps; msg=[]; req\_time=476; rdbms\_time=349; rdbms\_count=2
2013-07-24T17:08:11Z [erchef@127.0.0.1](mailto:erchef@127.0.0.1) INFO req\_id=Ec3C1KgepqDadQu/WJpTPA==; status=200; method=GET; path=/environments/production; user=apps; msg=[]; req\_time=469; rdbms\_time=342; rdbms\_count=2
I’m not sure how this is helpful…
–
Cassiano Leal
[http://cassianoleal.com](http://cassianoleal.com)
[http://twitter.com/cassianoleal](http://twitter.com/cassianoleal)
On July 24, 2013 at 12:42:19, Chris ([grocerylist@gmail.com](mailto:grocerylist@gmail.com)) wrote:
BTW, port 8000 should be the erChef process
On Wed, Jul 24, 2013 at 8:41 AM, Chris [grocerylist@gmail.com](mailto:grocerylist@gmail.com) wrote:
I’ve run into this too. I just migrated to OSC 11 on Monday and less than 24 hrs later had this problem. It only happened once though. I don’t have any more info on it either.
On Wed, Jul 24, 2013 at 8:34 AM, Cassiano Leal [cassianoleal@gmail.com](mailto:cassianoleal@gmail.com) wrote:
Hi,
I’m having an issue with my OSS Chef Server. Clients time out while trying to connect to it.
I found this on /var/log/chef-server/nginx/error.log:
2013/07/24 12:29:44 [error] 859#0: \*30 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.0.1.160, server: ip-10-0-1-10.sa-east-1.compute.internal, request: “POST /environments/production/cookbook\_versions HTTP/1.1”, upstream: “[http://127.0.0.1:8000/environments/production/cookbook\_versions](http://127.0.0.1:8000/environments/production/cookbook_versions)”, host: "10.0.1.10:443”
My guess is that the problem is in the “upstream” service that runs on port 8000, but I’m not sure which service that is. Where do I look?
## Cheers,
Cassiano Leal
[http://cassianoleal.com](http://cassianoleal.com)
[http://twitter.com/cassianoleal](http://twitter.com/cassianoleal)
–
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
–
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
---
### Author:  [@Cassiano\_Leal](https://discourse.chef.io/u/Cassiano_Leal)
#### Post date: [July 24, 2013, 5:58pm UTC](https://discourse.chef.io/t/chef-server-timeouts/4239/5 "2013-07-24T17:58:11Z")
I ran chef-server-ctl test and got these failures:
[http://pastie.org/private/6jqnpte37herj8jyomhmq](http://pastie.org/private/6jqnpte37herj8jyomhmq)
Any idea what’s happening? Some indexing gone wrong maybe?
–
Cassiano Leal
[http://cassianoleal.com](http://cassianoleal.com)
[http://twitter.com/cassianoleal](http://twitter.com/cassianoleal)
On July 24, 2013 at 14:16:41, Cassiano Leal ([cassianoleal@gmail.com](mailto:cassianoleal@gmail.com)) wrote:
The only things logged for erchef are:
2013-07-24T17:08:09Z [erchef@127.0.0.1](mailto:erchef@127.0.0.1) INFO req\_id=x+AfrOJ1AUtORncdS9mIfg==; status=200; method=GET; path=/nodes/apps; user=apps; msg=[]; req\_time=504; rdbms\_time=422; rdbms\_count=2
2013-07-24T17:08:09Z [erchef@127.0.0.1](mailto:erchef@127.0.0.1) INFO req\_id=4QrUsBd/rJ77XlBB2hWogw==; status=200; method=GET; path=/roles/api; user=apps; msg=[]; req\_time=479; rdbms\_time=373; rdbms\_count=2
2013-07-24T17:08:10Z [erchef@127.0.0.1](mailto:erchef@127.0.0.1) INFO req\_id=55zYGCExTCxAYJAaXdGMGA==; status=200; method=GET; path=/roles/manager; user=apps; msg=[]; req\_time=488; rdbms\_time=356; rdbms\_count=2
2013-07-24T17:08:10Z [erchef@127.0.0.1](mailto:erchef@127.0.0.1) INFO req\_id=a8NzqLLMByrRSPSdbvazQw==; status=200; method=GET; path=/roles/web; user=apps; msg=[]; req\_time=476; rdbms\_time=349; rdbms\_count=2
2013-07-24T17:08:11Z [erchef@127.0.0.1](mailto:erchef@127.0.0.1) INFO req\_id=Ec3C1KgepqDadQu/WJpTPA==; status=200; method=GET; path=/environments/production; user=apps; msg=[]; req\_time=469; rdbms\_time=342; rdbms\_count=2
I’m not sure how this is helpful…
–
Cassiano Leal
[http://cassianoleal.com](http://cassianoleal.com)
[http://twitter.com/cassianoleal](http://twitter.com/cassianoleal)
On July 24, 2013 at 12:42:19, Chris ([grocerylist@gmail.com](mailto:grocerylist@gmail.com)) wrote:
BTW, port 8000 should be the erChef process
On Wed, Jul 24, 2013 at 8:41 AM, Chris [grocerylist@gmail.com](mailto:grocerylist@gmail.com) wrote:
I’ve run into this too. I just migrated to OSC 11 on Monday and less than 24 hrs later had this problem. It only happened once though. I don’t have any more info on it either.
On Wed, Jul 24, 2013 at 8:34 AM, Cassiano Leal [cassianoleal@gmail.com](mailto:cassianoleal@gmail.com) wrote:
Hi,
I’m having an issue with my OSS Chef Server. Clients time out while trying to connect to it.
I found this on /var/log/chef-server/nginx/error.log:
2013/07/24 12:29:44 [error] 859#0: \*30 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.0.1.160, server: ip-10-0-1-10.sa-east-1.compute.internal, request: “POST /environments/production/cookbook\_versions HTTP/1.1”, upstream: “[http://127.0.0.1:8000/environments/production/cookbook\_versions](http://127.0.0.1:8000/environments/production/cookbook_versions)”, host: "10.0.1.10:443”
My guess is that the problem is in the “upstream” service that runs on port 8000, but I’m not sure which service that is. Where do I look?
## Cheers,
Cassiano Leal
[http://cassianoleal.com](http://cassianoleal.com)
[http://twitter.com/cassianoleal](http://twitter.com/cassianoleal)
–
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
–
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
---
### Author:  [@sdelano](https://discourse.chef.io/u/sdelano)
#### Post date: [July 24, 2013, 6:46pm UTC](https://discourse.chef.io/t/chef-server-timeouts/4239/6 "2013-07-24T18:46:18Z")
Hi there,
Can you find the corresponding POST request in /var/log/chef-server/nginx/access.log? The post will be to “/environments/production/cookbook\_versions” and might be able to shed some more light on where the timeout occurred (e.g. connection timeout vs. read timeout).
There is an outstanding bug in the depsolver erlang library that has been causing CPU hangs on the chef server: [http://tickets.opscode.com/browse/CHEF-3921](http://tickets.opscode.com/browse/CHEF-3921)
Cheers!
Stephen
From: Cassiano Leal \<[cassianoleal@gmail.com](mailto:cassianoleal@gmail.com)[mailto:cassianoleal@gmail.com](mailto:cassianoleal@gmail.com)\>
Reply-To: "chef@lists.opscode.com[mailto:chef@lists.opscode.com](mailto:chef@lists.opscode.com)" \<[chef@lists.opscode.com](mailto:chef@lists.opscode.com)[mailto:chef@lists.opscode.com](mailto:chef@lists.opscode.com)\>
Date: Wednesday, July 24, 2013 10:58 AM
To: "chef@lists.opscode.com[mailto:chef@lists.opscode.com](mailto:chef@lists.opscode.com)" \<[chef@lists.opscode.com](mailto:chef@lists.opscode.com)[mailto:chef@lists.opscode.com](mailto:chef@lists.opscode.com)\>
Subject: [chef] Re: Re: Chef Server timeouts
I ran chef-server-ctl test and got these failures:
[http://pastie.org/private/6jqnpte37herj8jyomhmq](http://pastie.org/private/6jqnpte37herj8jyomhmq)
Any idea what’s happening? Some indexing gone wrong maybe?
–
Cassiano Leal
[http://cassianoleal.com](http://cassianoleal.com)
[http://twitter.com/cassianoleal](http://twitter.com/cassianoleal)
On July 24, 2013 at 14:16:41, Cassiano Leal ([cassianoleal@gmail.com](mailto:cassianoleal@gmail.com)[mailto:cassianoleal@gmail.com](mailto:cassianoleal@gmail.com)) wrote:
The only things logged for erchef are:
2013-07-24T17:08:09Z [erchef@127.0.0.1](mailto:erchef@127.0.0.1)[mailto:erchef@127.0.0.1](mailto:erchef@127.0.0.1) INFO req\_id=x+AfrOJ1AUtORncdS9mIfg==; status=200; method=GET; path=/nodes/apps; user=apps; msg=[]; req\_time=504; rdbms\_time=422; rdbms\_count=2
2013-07-24T17:08:09Z [erchef@127.0.0.1](mailto:erchef@127.0.0.1)[mailto:erchef@127.0.0.1](mailto:erchef@127.0.0.1) INFO req\_id=4QrUsBd/rJ77XlBB2hWogw==; status=200; method=GET; path=/roles/api; user=apps; msg=[]; req\_time=479; rdbms\_time=373; rdbms\_count=2
2013-07-24T17:08:10Z [erchef@127.0.0.1](mailto:erchef@127.0.0.1)[mailto:erchef@127.0.0.1](mailto:erchef@127.0.0.1) INFO req\_id=55zYGCExTCxAYJAaXdGMGA==; status=200; method=GET; path=/roles/manager; user=apps; msg=[]; req\_time=488; rdbms\_time=356; rdbms\_count=2
2013-07-24T17:08:10Z [erchef@127.0.0.1](mailto:erchef@127.0.0.1)[mailto:erchef@127.0.0.1](mailto:erchef@127.0.0.1) INFO req\_id=a8NzqLLMByrRSPSdbvazQw==; status=200; method=GET; path=/roles/web; user=apps; msg=[]; req\_time=476; rdbms\_time=349; rdbms\_count=2
2013-07-24T17:08:11Z [erchef@127.0.0.1](mailto:erchef@127.0.0.1)[mailto:erchef@127.0.0.1](mailto:erchef@127.0.0.1) INFO req\_id=Ec3C1KgepqDadQu/WJpTPA==; status=200; method=GET; path=/environments/production; user=apps; msg=[]; req\_time=469; rdbms\_time=342; rdbms\_count=2
I’m not sure how this is helpful…
–
Cassiano Leal
[http://cassianoleal.com](http://cassianoleal.com)
[http://twitter.com/cassianoleal](http://twitter.com/cassianoleal)
On July 24, 2013 at 12:42:19, Chris ([grocerylist@gmail.com](mailto:grocerylist@gmail.com)[mailto:grocerylist@gmail.com](mailto:grocerylist@gmail.com)) wrote:
BTW, port 8000 should be the erChef process
On Wed, Jul 24, 2013 at 8:41 AM, Chris \<[grocerylist@gmail.com](mailto:grocerylist@gmail.com)[mailto:grocerylist@gmail.com](mailto:grocerylist@gmail.com)\> wrote:
I’ve run into this too. I just migrated to OSC 11 on Monday and less than 24 hrs later had this problem. It only happened once though. I don’t have any more info on it either.
On Wed, Jul 24, 2013 at 8:34 AM, Cassiano Leal \<[cassianoleal@gmail.com](mailto:cassianoleal@gmail.com)[mailto:cassianoleal@gmail.com](mailto:cassianoleal@gmail.com)\> wrote:
Hi,
I’m having an issue with my OSS Chef Server. Clients time out while trying to connect to it.
I found this on /var/log/chef-server/nginx/error.log:
2013/07/24 12:29:44 [error] 859#0: \*30 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.0.1.160, server: ip-10-0-1-10.sa-east-1.compute.internal, request: “POST /environments/production/cookbook\_versions HTTP/1.1”, upstream: “[http://127.0.0.1:8000/environments/production/cookbook\_versions](http://127.0.0.1:8000/environments/production/cookbook_versions)”, host: “10.0.1.10:443”
My guess is that the problem is in the “upstream” service that runs on port 8000, but I’m not sure which service that is. Where do I look?
Cheers,
–
Cassiano Leal
[http://cassianoleal.com](http://cassianoleal.com)
[http://twitter.com/cassianoleal](http://twitter.com/cassianoleal)
–
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
–
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
---
### Author:  [@Cassiano\_Leal](https://discourse.chef.io/u/Cassiano_Leal)
#### Post date: [July 24, 2013, 7:17pm UTC](https://discourse.chef.io/t/chef-server-timeouts/4239/7 "2013-07-24T19:17:44Z")
I got new ones:
==\> error.log \<==
2013/07/24 16:15:56 [error] 859#0: \*1771 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.0.1.137, server: ip-10-0-1-10.sa-east-1.compute.internal, request: “POST /environments/production/cookbook\_versions HTTP/1.1”, upstream: “[http://127.0.0.1:8000/environments/production/cookbook\_versions](http://127.0.0.1:8000/environments/production/cookbook_versions)”, host: “10.0.1.10:443”
==\> access.log \<==
10.0.1.137 - - [24/Jul/2013:16:15:56 -0300] “POST /environments/production/cookbook\_versions HTTP/1.1” 504 “300.113” 182 “-” “Chef Client/11.4.4 (ruby-1.9.3-p286; ohai-6.16.0; x86\_64-linux; +http://opscode.com)” “127.0.0.1:8000” “504” “300.065” “11.4.4” “algorithm=sha1;version=1.0;” “apps” “2013-07-24T19:11:11Z” "FkL+xFBHbcjWA94iv8c+Izoud/w=“ 1059
Another thing I noticed is that beam.smp is consuming 100% of a CPU core, and it’s been that way for hours.
–
Cassiano Leal
[http://cassianoleal.com](http://cassianoleal.com)
[http://twitter.com/cassianoleal](http://twitter.com/cassianoleal)
On July 24, 2013 at 15:46:40, Stephen Delano ([stephen@opscode.com](mailto:stephen@opscode.com)) wrote:
Hi there,
Can you find the corresponding POST request in /var/log/chef-server/nginx/access.log? The post will be to “/environments/production/cookbook\_versions” and might be able to shed some more light on where the timeout occurred (e.g. connection timeout vs. read timeout).
There is an outstanding bug in the depsolver erlang library that has been causing CPU hangs on the chef server: [http://tickets.opscode.com/browse/CHEF-3921](http://tickets.opscode.com/browse/CHEF-3921)
Cheers!
Stephen
From: Cassiano Leal [cassianoleal@gmail.com](mailto:cassianoleal@gmail.com)
Reply-To: "chef@lists.opscode.com" [chef@lists.opscode.com](mailto:chef@lists.opscode.com)
Date: Wednesday, July 24, 2013 10:58 AM
To: "chef@lists.opscode.com" [chef@lists.opscode.com](mailto:chef@lists.opscode.com)
Subject: [chef] Re: Re: Chef Server timeouts
I ran chef-server-ctl test and got these failures:
[http://pastie.org/private/6jqnpte37herj8jyomhmq](http://pastie.org/private/6jqnpte37herj8jyomhmq)
Any idea what’s happening? Some indexing gone wrong maybe?
–
Cassiano Leal
[http://cassianoleal.com](http://cassianoleal.com)
[http://twitter.com/cassianoleal](http://twitter.com/cassianoleal)
On July 24, 2013 at 14:16:41, Cassiano Leal ([cassianoleal@gmail.com](mailto:cassianoleal@gmail.com)) wrote:
The only things logged for erchef are:
2013-07-24T17:08:09Z [erchef@127.0.0.1](mailto:erchef@127.0.0.1) INFO req\_id=x+AfrOJ1AUtORncdS9mIfg==; status=200; method=GET; path=/nodes/apps; user=apps; msg=[]; req\_time=504; rdbms\_time=422; rdbms\_count=2
2013-07-24T17:08:09Z [erchef@127.0.0.1](mailto:erchef@127.0.0.1) INFO req\_id=4QrUsBd/rJ77XlBB2hWogw==; status=200; method=GET; path=/roles/api; user=apps; msg=[]; req\_time=479; rdbms\_time=373; rdbms\_count=2
2013-07-24T17:08:10Z [erchef@127.0.0.1](mailto:erchef@127.0.0.1) INFO req\_id=55zYGCExTCxAYJAaXdGMGA==; status=200; method=GET; path=/roles/manager; user=apps; msg=[]; req\_time=488; rdbms\_time=356; rdbms\_count=2
2013-07-24T17:08:10Z [erchef@127.0.0.1](mailto:erchef@127.0.0.1) INFO req\_id=a8NzqLLMByrRSPSdbvazQw==; status=200; method=GET; path=/roles/web; user=apps; msg=[]; req\_time=476; rdbms\_time=349; rdbms\_count=2
2013-07-24T17:08:11Z [erchef@127.0.0.1](mailto:erchef@127.0.0.1) INFO req\_id=Ec3C1KgepqDadQu/WJpTPA==; status=200; method=GET; path=/environments/production; user=apps; msg=[]; req\_time=469; rdbms\_time=342; rdbms\_count=2
I’m not sure how this is helpful…
–
Cassiano Leal
[http://cassianoleal.com](http://cassianoleal.com)
[http://twitter.com/cassianoleal](http://twitter.com/cassianoleal)
On July 24, 2013 at 12:42:19, Chris ([grocerylist@gmail.com](mailto:grocerylist@gmail.com)) wrote:
BTW, port 8000 should be the erChef process
On Wed, Jul 24, 2013 at 8:41 AM, Chris [grocerylist@gmail.com](mailto:grocerylist@gmail.com) wrote:
I’ve run into this too. I just migrated to OSC 11 on Monday and less than 24 hrs later had this problem. It only happened once though. I don’t have any more info on it either.
On Wed, Jul 24, 2013 at 8:34 AM, Cassiano Leal [cassianoleal@gmail.com](mailto:cassianoleal@gmail.com) wrote:
Hi,
I’m having an issue with my OSS Chef Server. Clients time out while trying to connect to it.
I found this on /var/log/chef-server/nginx/error.log:
2013/07/24 12:29:44 [error] 859#0: \*30 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.0.1.160, server: ip-10-0-1-10.sa-east-1.compute.internal, request: “POST /environments/production/cookbook\_versions HTTP/1.1”, upstream: “[http://127.0.0.1:8000/environments/production/cookbook\_versions](http://127.0.0.1:8000/environments/production/cookbook_versions)”, host: "10.0.1.10:443”
My guess is that the problem is in the “upstream” service that runs on port 8000, but I’m not sure which service that is. Where do I look?
## Cheers,
Cassiano Leal
[http://cassianoleal.com](http://cassianoleal.com)
[http://twitter.com/cassianoleal](http://twitter.com/cassianoleal)
–
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
–
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
---
### Author:  [@Cassiano\_Leal](https://discourse.chef.io/u/Cassiano_Leal)
#### Post date: [July 24, 2013, 9:18pm UTC](https://discourse.chef.io/t/chef-server-timeouts/4239/8 "2013-07-24T21:18:08Z")
Ok, so I “solved” this issue by deleting all cookbooks from the server, then re-uploading them while carefully checking versions pinned down for the environment at the same time.
Some of them had lower versions than the current one pinned down and something must have gone awry because of that, but there’s no way I could pinpoint the problem by looking at the logs.
I really wish there was more information spit out. One good example would be that when a node doesn’t find a cookbook _version_ it wouldn’t log that it couldn’t find _the cookbook_ on the server. Printing out which version it wants along with the name would go a long way here. 🙂
–
Cassiano Leal
[http://cassianoleal.com](http://cassianoleal.com)
[http://twitter.com/cassianoleal](http://twitter.com/cassianoleal)
On July 24, 2013 at 16:17:46, Cassiano Leal ([cassianoleal@gmail.com](mailto:cassianoleal@gmail.com)) wrote:
I got new ones:
==\> error.log \<==
2013/07/24 16:15:56 [error] 859#0: \*1771 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.0.1.137, server: ip-10-0-1-10.sa-east-1.compute.internal, request: “POST /environments/production/cookbook\_versions HTTP/1.1”, upstream: “[http://127.0.0.1:8000/environments/production/cookbook\_versions](http://127.0.0.1:8000/environments/production/cookbook_versions)”, host: “10.0.1.10:443”
==\> access.log \<==
10.0.1.137 - - [24/Jul/2013:16:15:56 -0300] “POST /environments/production/cookbook\_versions HTTP/1.1” 504 “300.113” 182 “-” “Chef Client/11.4.4 (ruby-1.9.3-p286; ohai-6.16.0; x86\_64-linux; +http://opscode.com)” “127.0.0.1:8000” “504” “300.065” “11.4.4” “algorithm=sha1;version=1.0;” “apps” “2013-07-24T19:11:11Z” "FkL+xFBHbcjWA94iv8c+Izoud/w=“ 1059
Another thing I noticed is that beam.smp is consuming 100% of a CPU core, and it’s been that way for hours.
–
Cassiano Leal
[http://cassianoleal.com](http://cassianoleal.com)
[http://twitter.com/cassianoleal](http://twitter.com/cassianoleal)
On July 24, 2013 at 15:46:40, Stephen Delano ([stephen@opscode.com](mailto:stephen@opscode.com)) wrote:
Hi there,
Can you find the corresponding POST request in /var/log/chef-server/nginx/access.log? The post will be to “/environments/production/cookbook\_versions” and might be able to shed some more light on where the timeout occurred (e.g. connection timeout vs. read timeout).
There is an outstanding bug in the depsolver erlang library that has been causing CPU hangs on the chef server: [http://tickets.opscode.com/browse/CHEF-3921](http://tickets.opscode.com/browse/CHEF-3921)
Cheers!
Stephen
From: Cassiano Leal [cassianoleal@gmail.com](mailto:cassianoleal@gmail.com)
Reply-To: "chef@lists.opscode.com" [chef@lists.opscode.com](mailto:chef@lists.opscode.com)
Date: Wednesday, July 24, 2013 10:58 AM
To: "chef@lists.opscode.com" [chef@lists.opscode.com](mailto:chef@lists.opscode.com)
Subject: [chef] Re: Re: Chef Server timeouts
I ran chef-server-ctl test and got these failures:
[http://pastie.org/private/6jqnpte37herj8jyomhmq](http://pastie.org/private/6jqnpte37herj8jyomhmq)
Any idea what’s happening? Some indexing gone wrong maybe?
–
Cassiano Leal
[http://cassianoleal.com](http://cassianoleal.com)
[http://twitter.com/cassianoleal](http://twitter.com/cassianoleal)
On July 24, 2013 at 14:16:41, Cassiano Leal ([cassianoleal@gmail.com](mailto:cassianoleal@gmail.com)) wrote:
The only things logged for erchef are:
2013-07-24T17:08:09Z [erchef@127.0.0.1](mailto:erchef@127.0.0.1) INFO req\_id=x+AfrOJ1AUtORncdS9mIfg==; status=200; method=GET; path=/nodes/apps; user=apps; msg=[]; req\_time=504; rdbms\_time=422; rdbms\_count=2
2013-07-24T17:08:09Z [erchef@127.0.0.1](mailto:erchef@127.0.0.1) INFO req\_id=4QrUsBd/rJ77XlBB2hWogw==; status=200; method=GET; path=/roles/api; user=apps; msg=[]; req\_time=479; rdbms\_time=373; rdbms\_count=2
2013-07-24T17:08:10Z [erchef@127.0.0.1](mailto:erchef@127.0.0.1) INFO req\_id=55zYGCExTCxAYJAaXdGMGA==; status=200; method=GET; path=/roles/manager; user=apps; msg=[]; req\_time=488; rdbms\_time=356; rdbms\_count=2
2013-07-24T17:08:10Z [erchef@127.0.0.1](mailto:erchef@127.0.0.1) INFO req\_id=a8NzqLLMByrRSPSdbvazQw==; status=200; method=GET; path=/roles/web; user=apps; msg=[]; req\_time=476; rdbms\_time=349; rdbms\_count=2
2013-07-24T17:08:11Z [erchef@127.0.0.1](mailto:erchef@127.0.0.1) INFO req\_id=Ec3C1KgepqDadQu/WJpTPA==; status=200; method=GET; path=/environments/production; user=apps; msg=[]; req\_time=469; rdbms\_time=342; rdbms\_count=2
I’m not sure how this is helpful…
–
Cassiano Leal
[http://cassianoleal.com](http://cassianoleal.com)
[http://twitter.com/cassianoleal](http://twitter.com/cassianoleal)
On July 24, 2013 at 12:42:19, Chris ([grocerylist@gmail.com](mailto:grocerylist@gmail.com)) wrote:
BTW, port 8000 should be the erChef process
On Wed, Jul 24, 2013 at 8:41 AM, Chris [grocerylist@gmail.com](mailto:grocerylist@gmail.com) wrote:
I’ve run into this too. I just migrated to OSC 11 on Monday and less than 24 hrs later had this problem. It only happened once though. I don’t have any more info on it either.
On Wed, Jul 24, 2013 at 8:34 AM, Cassiano Leal [cassianoleal@gmail.com](mailto:cassianoleal@gmail.com) wrote:
Hi,
I’m having an issue with my OSS Chef Server. Clients time out while trying to connect to it.
I found this on /var/log/chef-server/nginx/error.log:
2013/07/24 12:29:44 [error] 859#0: \*30 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.0.1.160, server: ip-10-0-1-10.sa-east-1.compute.internal, request: “POST /environments/production/cookbook\_versions HTTP/1.1”, upstream: “[http://127.0.0.1:8000/environments/production/cookbook\_versions](http://127.0.0.1:8000/environments/production/cookbook_versions)”, host: "10.0.1.10:443”
My guess is that the problem is in the “upstream” service that runs on port 8000, but I’m not sure which service that is. Where do I look?
## Cheers,
Cassiano Leal
[http://cassianoleal.com](http://cassianoleal.com)
[http://twitter.com/cassianoleal](http://twitter.com/cassianoleal)
–
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
–
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.