Issue on EC2

Chef Infra (archive)
1

Hi,

I have difficulties to make chef-server works on an EC2 instance. I always get an issue which seems to cause processes to restart in an infinite loop. Processes always fall down and a lot of CPU is consumed by always restarting them. You can find an extract of logs at http://pastebin.com/BAzP11t9

Using the server works but as processes fall down I get some “bad gateway” errors from nginx and a lot of CPU is consumed for wasted

thanks

Cyril SCETBON

2
  1. What is the instance size you are using ?
  2. What is the version of Chef Server that you are using?
  3. Are you using RDS for postgres, or hosting it on the same server?

Thanks
Gourav

On Mon, Jan 13, 2014 at 9:34 PM, Cyril Scetbon cyril.scetbon@free.frwrote:

Hi,

I have difficulties to make chef-server works on an EC2 instance. I always
get an issue which seems to cause processes to restart in an infinite loop.
Processes always fall down and a lot of CPU is consumed by always
restarting them. You can find an extract of logs at
chef errors - Pastebin.com

Using the server works but as processes fall down I get some "bad gateway"
errors from nginx and a lot of CPU is consumed for wasted

thanks

Cyril SCETBON

--
Founder,
Initcron | www.initcron.com
Blog: www.initcron.org
Skype: initcron | Phone: +91 96 2328 2328

3

On 13 Jan 2014, at 17:25, Gourav Shah gs@initcron.org wrote:

  1. What is the instance size you are using ?
    I tried from small to xlarge
  2. What is the version of Chef Server that you are using?
    last version 11.0.10
  3. Are you using RDS for postgres, or hosting it on the same server?
    postgresql on the same server (no issue coming from postgresql except on small where I had to reduce shmmall memory parameter)

Thanks
Gourav

On Mon, Jan 13, 2014 at 9:34 PM, Cyril Scetbon cyril.scetbon@free.fr wrote:
Hi,

I have difficulties to make chef-server works on an EC2 instance. I always get an issue which seems to cause processes to restart in an infinite loop. Processes always fall down and a lot of CPU is consumed by always restarting them. You can find an extract of logs at chef errors - Pastebin.com

Using the server works but as processes fall down I get some "bad gateway" errors from nginx and a lot of CPU is consumed for wasted

thanks

Cyril SCETBON

--
Founder,
Initcron | www.initcron.com
Blog: www.initcron.org
Skype: initcron | Phone: +91 96 2328 2328

4

Hi there,

cyril.scetbon@free.fr writes:

I have difficulties to make chef-server works on an EC2 instance. I
always get an issue which seems to cause processes to restart in an
infinite loop. Processes always fall down and a lot of CPU is consumed
by always restarting them. You can find an extract of logs at
chef errors - Pastebin.com

The logs suggest that the rabbitmq service is unable to start because
there is already an Erlang vm with the same name:

==> /var/log/chef-server/rabbitmq/current <==
2014-01-13_15:59:18.08077 0 plugins activated:
2014-01-13_15:59:18.08081
2014-01-13_15:59:18.12693 node with name "rabbit" already running on "localhost"
2014-01-13_15:59:18.13304 diagnostics:
2014-01-13_15:59:18.13335 - nodes and their ports on localhost: [{rabbit,47989},
2014-01-13_15:59:18.13337                                        {bookshelf,57685},
2014-01-13_15:59:18.13338                                        {erchef,44117},
2014-01-13_15:59:18.13338                                        {rabbitmqprelaunch12288,54518}]
2014-01-13_15:59:18.13343 - current node:
'rabbitmqprelaunch12288@chef-server'

So investigating that might be useful. Are you running any other Erlang
projects on your node?

When you run chef-server-ctl stop, look for Erlang vms via

ps -ef|grep [b]eam

Are there any running?

  • seth

--
Seth Falcon | Development Lead | CHEF | http://www.getchef.com/ | @sfalcon

5

Thanks, that was exactly the root cause !

Regards

Cyril SCETBON

On 13 Jan 2014, at 19:20, Seth Falcon seth@getchef.com wrote:

Hi there,

cyril.scetbon@free.fr writes:

I have difficulties to make chef-server works on an EC2 instance. I
always get an issue which seems to cause processes to restart in an
infinite loop. Processes always fall down and a lot of CPU is consumed
by always restarting them. You can find an extract of logs at
chef errors - Pastebin.com

The logs suggest that the rabbitmq service is unable to start because
there is already an Erlang vm with the same name:

==> /var/log/chef-server/rabbitmq/current <==
2014-01-13_15:59:18.08077 0 plugins activated:
2014-01-13_15:59:18.08081
2014-01-13_15:59:18.12693 node with name "rabbit" already running on "localhost"
2014-01-13_15:59:18.13304 diagnostics:
2014-01-13_15:59:18.13335 - nodes and their ports on localhost: [{rabbit,47989},
2014-01-13_15:59:18.13337 {bookshelf,57685},
2014-01-13_15:59:18.13338 {erchef,44117},
2014-01-13_15:59:18.13338 {rabbitmqprelaunch12288,54518}]
2014-01-13_15:59:18.13343 - current node:
'rabbitmqprelaunch12288@chef-server'

So investigating that might be useful. Are you running any other Erlang
projects on your node?

When you run chef-server-ctl stop, look for Erlang vms via

ps -ef|grep [b]eam

Are there any running?

  • seth

--
Seth Falcon | Development Lead | CHEF | http://www.getchef.com/ | @sfalcon