Chef Server Slowing Down

Chef Infra (archive)
1

I’ve noticed that my chef server, which is running version 11.0.12-1, slows
down over time. It takes about a week to become totally unresponsive. A
reboot fixes the issue every time. Affects cookbook uploading and client
communication. Hasn’t just happened once, but repeatedly.

Is this a known issue? Would rather not upgrade right now unless someone
can point me to a specific bug. Running on Amazon Linux 2014-09, obviously
on EC2.

Doug

2

On Dec 29, 2014, at 11:44 AM, Douglas Garstang doug.garstang@gmail.com wrote:

I've noticed that my chef server, which is running version 11.0.12-1, slows down over time. It takes about a week to become totally unresponsive. A reboot fixes the issue every time. Affects cookbook uploading and client communication. Hasn't just happened once, but repeatedly.

Is this a known issue? Would rather not upgrade right now unless someone can point me to a specific bug. Running on Amazon Linux 2014-09, obviously on EC2.

Likely runaway depsolvers, try not running a thoroughly outdated version. 11.1.0 had the fix for that issue in particular.

--Noah

3

To elaborate on Noah's point, 11.0 introduced an Erlang based depsolver
for cookbooks. The thinking was that this depsolver, while less robust,
would be faster and allow for easier installs of the server, since it
didn't rely on gecode. It was thought that the edge cases that gecode
handled that the Erlang depsolver didn't would only affect a few people.
That was a bad assumption. It turns out that many cookbooks have
dependencies that are hard to solve for and the Erlang depsolver wasn't
up to the task, where as gecode was. With the 11.1 release the server
switched back to using gecode. Noah is right that upgrade to at least
11.1 should help you out. If you don't want to jump straight to 12,
which is a more involved upgrade process, you can upgrade to 11.1+ by
installing the package and then using the chef-server-ctl upgrade.

http://docs.chef.io/open_source/upgrade_server_open_source.html

Upgrading to 11.1+ will also help when you transition to 12, since the
upgrade process to get to 12 is designed for 11.1+ and is known to not
currently work with older versions of the server.

The release notes for the 11.1 release are found here, for reference:
http://docs.chef.io/release/osc_11-1/release_notes.html

Also note I've assumed you're running the open source version of the
server and that we're not talking about the enterprise version (with 12,
these versions are one and the same).

  • Mark Mzyk

Noah Kantrowitz mailto:noah@coderanger.net
December 29, 2014 at 2:54 PM

Likely runaway depsolvers, try not running a thoroughly outdated
version. 11.1.0 had the fix for that issue in particular.

--Noah

Douglas Garstang mailto:doug.garstang@gmail.com
December 29, 2014 at 2:44 PM
I've noticed that my chef server, which is running version 11.0.12-1,
slows down over time. It takes about a week to become totally
unresponsive. A reboot fixes the issue every time. Affects cookbook
uploading and client communication. Hasn't just happened once, but
repeatedly.

Is this a known issue? Would rather not upgrade right now unless
someone can point me to a specific bug. Running on Amazon Linux
2014-09, obviously on EC2.

Doug

4

So now chef server still relies on erlang?

On Monday, December 29, 2014, Mark Mzyk mmzyk@chef.io wrote:

To elaborate on Noah's point, 11.0 introduced an Erlang based depsolver
for cookbooks. The thinking was that this depsolver, while less robust,
would be faster and allow for easier installs of the server, since it
didn't rely on gecode. It was thought that the edge cases that gecode
handled that the Erlang depsolver didn't would only affect a few people.
That was a bad assumption. It turns out that many cookbooks have
dependencies that are hard to solve for and the Erlang depsolver wasn't up
to the task, where as gecode was. With the 11.1 release the server switched
back to using gecode. Noah is right that upgrade to at least 11.1 should
help you out. If you don't want to jump straight to 12, which is a more
involved upgrade process, you can upgrade to 11.1+ by installing the
package and then using the chef-server-ctl upgrade.

http://docs.chef.io/open_source/upgrade_server_open_source.html

Upgrading to 11.1+ will also help when you transition to 12, since the
upgrade process to get to 12 is designed for 11.1+ and is known to not
currently work with older versions of the server.

The release notes for the 11.1 release are found here, for reference:
http://docs.chef.io/release/osc_11-1/release_notes.html

Also note I've assumed you're running the open source version of the
server and that we're not talking about the enterprise version (with 12,
these versions are one and the same).

  • Mark Mzyk

</mail/u/0/s/?view=att&th=14a97e706d3e5eb9&attid=0.1.1&disp=emb&zw&atsh=1>
Noah Kantrowitz
December 29, 2014 at 2:54 PM
Likely runaway depsolvers, try not running a thoroughly outdated version.
11.1.0 had the fix for that issue in particular.

--Noah

</mail/u/0/s/?view=att&th=14a97e706d3e5eb9&attid=0.1.2&disp=emb&zw&atsh=1>
Douglas Garstang
December 29, 2014 at 2:44 PM
I've noticed that my chef server, which is running version 11.0.12-1,
slows down over time. It takes about a week to become totally unresponsive.
A reboot fixes the issue every time. Affects cookbook uploading and client
communication. Hasn't just happened once, but repeatedly.

Is this a known issue? Would rather not upgrade right now unless someone
can point me to a specific bug. Running on Amazon Linux 2014-09, obviously
on EC2.

Doug

--
"Lute pela sua ideologia. Seja um com sua ideologia. Viva pela sua
ideologia. Morra por sua ideologia" P.R. Sarkar

EITA - Educação, Informação e Tecnologias para Autogestão

"Paramapurusha é meu pai e Parama Prakriti é minha mãe. O universo é meu
lar e todos nós somos cidadãos deste cosmo. Este universo é a imaginação da
Mente Macrocósmica, e todas as entidades estão sendo criadas, preservadas e
destruídas nas fases de extroversão e introversão do fluxo imaginativo
cósmico. No âmbito pessoal, quando uma pessoa imagina algo em sua mente,
naquele momento, essa pessoa é a única proprietária daquilo que ela
imagina, e ninguém mais. Quando um ser humano criado mentalmente caminha
por um milharal também imaginado, a pessoa imaginada não é a propriedade
desse milharal, pois ele pertence ao indivíduo que o está imaginando. Este
universo foi criado na imaginação de Brahma, a Entidade Suprema, por isso
a propriedade deste universo é de Brahma, e não dos microcosmos que também
foram criados pela imaginação de Brahma. Nenhuma propriedade deste mundo,
mutável ou imutável, pertence a um indivíduo em particular; tudo é o
patrimônio comum de todos."
Restante do texto em

5

Yes, the Chef Server is still primarily Erlang and the core components
are entirely in Erlang, as has been the case since Chef 11. However,
some services the server uses are not written in Erlang. In this case
the service under discussion is the depsolving service that is used to
resolve cookbook dependencies. The server makes use of gecode, which is
a depsolver written in C. For a while an all Erlang based depsolver was
used, but for the reasons previously given the decision was made to
switch back to gecode, the C based depsolver.

Other non-Erlang based components include Solr, which is Java and is
used for search, and the oc-id service, which is written in Ruby.

  • Mark Mzyk

Bráulio Bhavamitra mailto:brauliobo@gmail.com
December 30, 2014 at 7:22 AM
So now chef server still relies on erlang?

On Monday, December 29, 2014, Mark Mzyk <mmzyk@chef.io
mailto:mmzyk@chef.io> wrote:

To elaborate on Noah's point, 11.0 introduced an Erlang based
depsolver for cookbooks. The thinking was that this depsolver, while
less robust, would be faster and allow for easier installs of the
server, since it didn't rely on gecode. It was thought that the edge
cases that gecode handled that the Erlang depsolver didn't would only
affect a few people. That was a bad assumption. It turns out that many
cookbooks have dependencies that are hard to solve for and the Erlang
depsolver wasn't up to the task, where as gecode was. With the 11.1
release the server switched back to using gecode. Noah is right that
upgrade to at least 11.1 should help you out. If you don't want to
jump straight to 12, which is a more involved upgrade process, you can
upgrade to 11.1+ by installing the package and then using the
chef-server-ctl upgrade.

http://docs.chef.io/open_source/upgrade_server_open_source.html

Upgrading to 11.1+ will also help when you transition to 12, since
the upgrade process to get to 12 is designed for 11.1+ and is known to
not currently work with older versions of the server.

The release notes for the 11.1 release are found here, for
reference: http://docs.chef.io/release/osc_11-1/release_notes.html

Also note I've assumed you're running the open source version of the
server and that we're not talking about the enterprise version (with
12, these versions are one and the same).

  • Mark Mzyk

</mail/u/0/s/?view=att&th=14a97e706d3e5eb9&attid=0.1.1&disp=emb&zw&atsh=1>

Noah Kantrowitz
December 29, 2014 at 2:54 PM
Likely runaway depsolvers, try not running a thoroughly outdated
version. 11.1.0 had the fix for that issue in particular.

--Noah

</mail/u/0/s/?view=att&th=14a97e706d3e5eb9&attid=0.1.2&disp=emb&zw&atsh=1>

Douglas Garstang
December 29, 2014 at 2:44 PM
I've noticed that my chef server, which is running version
11.0.12-1, slows down over time. It takes about a week to become
totally unresponsive. A reboot fixes the issue every time. Affects
cookbook uploading and client communication. Hasn't just happened
once, but repeatedly.

Is this a known issue? Would rather not upgrade right now unless
someone can point me to a specific bug. Running on Amazon Linux
2014-09, obviously on EC2.

Doug

--
"Lute pela sua ideologia. Seja um com sua ideologia. Viva pela sua
ideologia. Morra por sua ideologia" P.R. Sarkar

EITA - Educação, Informação e Tecnologias para Autogestão
http://cirandas.net/brauliobo
http://eita.org.br http://eita.org.br/

"Paramapurusha é meu pai e Parama Prakriti é minha mãe. O universo
é meu lar e todos nós somos cidadãos deste cosmo. Este universo é
a imaginação da Mente Macrocósmica, e todas as entidades estão
sendo criadas, preservadas e destruídas nas fases de extroversão
e introversão do fluxo imaginativo cósmico. No âmbito pessoal,
quando uma pessoa imagina algo em sua mente, naquele momento, essa
pessoa é a única proprietária daquilo que ela imagina, e ninguém mais.
Quando um ser humano criado mentalmente caminha por um milharal
também imaginado, a pessoa imaginada não é a propriedade desse
milharal, pois ele pertence ao indivíduo que o está imaginando. Este
universo foi criado na imaginação de Brahma, a Entidade Suprema, por
isso a propriedade deste universo é de Brahma, e não dos microcosmos
que também foram criados pela imaginação de Brahma. Nenhuma
propriedade deste mundo, mutável ou imutável, pertence a um indivíduo
em particular; tudo é o patrimônio comum de todos."
Restante do texto em
A problemática de hoje em dia - Bráulio Bhavamitra

Mark Mzyk mailto:mmzyk@chef.io
December 29, 2014 at 4:14 PM
To elaborate on Noah's point, 11.0 introduced an Erlang based
depsolver for cookbooks. The thinking was that this depsolver, while
less robust, would be faster and allow for easier installs of the
server, since it didn't rely on gecode. It was thought that the edge
cases that gecode handled that the Erlang depsolver didn't would only
affect a few people. That was a bad assumption. It turns out that many
cookbooks have dependencies that are hard to solve for and the Erlang
depsolver wasn't up to the task, where as gecode was. With the 11.1
release the server switched back to using gecode. Noah is right that
upgrade to at least 11.1 should help you out. If you don't want to
jump straight to 12, which is a more involved upgrade process, you can
upgrade to 11.1+ by installing the package and then using the
chef-server-ctl upgrade.

http://docs.chef.io/open_source/upgrade_server_open_source.html

Upgrading to 11.1+ will also help when you transition to 12, since the
upgrade process to get to 12 is designed for 11.1+ and is known to not
currently work with older versions of the server.

The release notes for the 11.1 release are found here, for reference:
http://docs.chef.io/release/osc_11-1/release_notes.html

Also note I've assumed you're running the open source version of the
server and that we're not talking about the enterprise version (with
12, these versions are one and the same).

  • Mark Mzyk

Noah Kantrowitz mailto:noah@coderanger.net
December 29, 2014 at 2:54 PM

Likely runaway depsolvers, try not running a thoroughly outdated
version. 11.1.0 had the fix for that issue in particular.

--Noah

Douglas Garstang mailto:doug.garstang@gmail.com
December 29, 2014 at 2:44 PM
I've noticed that my chef server, which is running version 11.0.12-1,
slows down over time. It takes about a week to become totally
unresponsive. A reboot fixes the issue every time. Affects cookbook
uploading and client communication. Hasn't just happened once, but
repeatedly.

Is this a known issue? Would rather not upgrade right now unless
someone can point me to a specific bug. Running on Amazon Linux
2014-09, obviously on EC2.

Doug