100% CPU usage of Chef-server beam.smp


#1

Hi. Today I encountered with broken bootstrap process on my Chef at AWS. The
simple bootstrap command stuck and then I got these errors.

http://pastebin.com/qUrJnz6h

The command was:

knife ec2 server create -N oss22.host.com -x root -E prod -i
~/.ssh/key-08-2013.pem -Z us-east-1b -g default -f m1.small -S key-08-2013 -r
"role[base_server],recipe[nexus],recipe[etc_hosts]"

The main point is in the server the process beam.smp began to consume 100% CPU.

It’s reproducible issue. Some info from this affected erlang process, I hope
will be helpful:

http://pastebin.com/N7g16195

The chef server version=11.0.2;erchef=1.2.6, chef-client 11.4.4.


#2

Usually when I see this issue it’s because Chef cannot resolve all cookbook dependencies.

Unfortunately, I haven’t found an easy and practical way to check that apart from actually looking down the cookbook dependency tree until I find the culprit.

To get the chef server back to normal CPU usage, restart it. If you try to run the chef client without solving the issue, the CPU will once again be consumed by beam.smp.

Good luck!

On August 28, 2013 at 11:54:32, nowarry@gmail.com (nowarry@gmail.com) wrote:

Hi. Today I encountered with broken bootstrap process on my Chef at AWS. The
simple bootstrap command stuck and then I got these errors.

http://pastebin.com/qUrJnz6h

The command was:

knife ec2 server create -N oss22.host.com -x root -E prod -i
~/.ssh/key-08-2013.pem -Z us-east-1b -g default -f m1.small -S key-08-2013 -r
"role[base_server],recipe[nexus],recipe[etc_hosts]"

The main point is in the server the process beam.smp began to consume 100% CPU.

It’s reproducible issue. Some info from this affected erlang process, I hope
will be helpful:

http://pastebin.com/N7g16195

The chef server version=11.0.2;erchef=1.2.6, chef-client 11.4.4.

Cassiano Leal
http://cassianoleal.com
http://twitter.com/cassianoleal


#3

Had this happen to me a few weeks ago. Ended up going nuclear, removing all of my cookbooks and then re-uploading them. In any event, it was the push I needed to start using berkshelf.

From the reading I did, it seems that if you pin an object to a specific version of a cookbook an then that cookbook goes missing or is removed in favor of a newer one, chaos ensues. I’m sure it had to do with the dramatic jump in the version on the osx community cookbook, but I didn’t have the time to burn tracking it down.

From: Cassiano Leal <cassianoleal@gmail.commailto:cassianoleal@gmail.com>
Reply-To: "chef@lists.opscode.commailto:chef@lists.opscode.com" <chef@lists.opscode.commailto:chef@lists.opscode.com>
Date: Wednesday, August 28, 2013 10:42 AM
To: "chef@lists.opscode.commailto:chef@lists.opscode.com" <chef@lists.opscode.commailto:chef@lists.opscode.com>, "nowarry@gmail.commailto:nowarry@gmail.com" <nowarry@gmail.commailto:nowarry@gmail.com>
Subject: [chef] Re: 100% CPU usage of Chef-server beam.smp

Usually when I see this issue it’s because Chef cannot resolve all cookbook dependencies.

Unfortunately, I haven’t found an easy and practical way to check that apart from actually looking down the cookbook dependency tree until I find the culprit.

To get the chef server back to normal CPU usage, restart it. If you try to run the chef client without solving the issue, the CPU will once again be consumed by beam.smp.

Good luck!

On August 28, 2013 at 11:54:32, nowarry@gmail.commailto:nowarry@gmail.com (nowarry@gmail.commailto:nowarry@gmail.com) wrote:

Hi. Today I encountered with broken bootstrap process on my Chef at AWS. The
simple bootstrap command stuck and then I got these errors.

http://pastebin.com/qUrJnz6h

The command was:

knife ec2 server create -N oss22.host.com -x root -E prod -i
~/.ssh/key-08-2013.pem -Z us-east-1b -g default -f m1.small -S key-08-2013 -r
"role[base_server],recipe[nexus],recipe[etc_hosts]"

The main point is in the server the process beam.smp began to consume 100% CPU.

It’s reproducible issue. Some info from this affected erlang process, I hope
will be helpful:

http://pastebin.com/N7g16195

The chef server version=11.0.2;erchef=1.2.6, chef-client 11.4.4.

Cassiano Leal
http://cassianoleal.com
http://twitter.com/cassianoleal


#4

Thanks guys. I deleted all cookbooks as suggested by Aaron and re-uploaded
again. The problem is gone.

On Wed, Aug 28, 2013 at 6:56 PM, Aaron Binford abinford@cleversafe.comwrote:

Had this happen to me a few weeks ago. Ended up going nuclear, removing
all of my cookbooks and then re-uploading them. In any event, it was the
push I needed to start using berkshelf.

From the reading I did, it seems that if you pin an object to a specific
version of a cookbook an then that cookbook goes missing or is removed in
favor of a newer one, chaos ensues. I’m sure it had to do with the dramatic
jump in the version on the osx community cookbook, but I didn’t have the
time to burn tracking it down.

From: Cassiano Leal cassianoleal@gmail.com
Reply-To: "chef@lists.opscode.com" chef@lists.opscode.com
Date: Wednesday, August 28, 2013 10:42 AM
To: "chef@lists.opscode.com" chef@lists.opscode.com, "nowarry@gmail.com"
nowarry@gmail.com
Subject: [chef] Re: 100% CPU usage of Chef-server beam.smp

Usually when I see this issue it’s because Chef cannot resolve all
cookbook dependencies.

Unfortunately, I haven’t found an easy and practical way to check that
apart from actually looking down the cookbook dependency tree until I find
the culprit.

To get the chef server back to normal CPU usage, restart it. If you try to
run the chef client without solving the issue, the CPU will once again be
consumed by beam.smp.

Good luck!

On August 28, 2013 at 11:54:32, nowarry@gmail.com (nowarry@gmail.com)
wrote:

Hi. Today I encountered with broken bootstrap process on my Chef at AWS.
The
simple bootstrap command stuck and then I got these errors.

http://pastebin.com/qUrJnz6h

The command was:

knife ec2 server create -N oss22.host.com -x root -E prod -i
~/.ssh/key-08-2013.pem -Z us-east-1b -g default -f m1.small -S key-08-2013
-r
"role[base_server],recipe[nexus],recipe[etc_hosts]"

The main point is in the server the process beam.smp began to consume 100%
CPU.

It’s reproducible issue. Some info from this affected erlang process, I
hope
will be helpful:

http://pastebin.com/N7g16195

The chef server version=11.0.2;erchef=1.2.6, chef-client 11.4.4.


Cassiano Leal
http://cassianoleal.com
http://twitter.com/cassianoleal