Error AWS AutoScale and rabbitmq

Chef Infra (archive)
1

Hope someone can help with this!
Converge fails when using AWS AutoScaling, but succeeds when doing a
normal EC2 server create.
The source of the error is in the rabbitmq cookbook. Somehow the hash
that is expected for kernel (I believe this is Erlang kernel?) becomes
an array.

Starting with "knife ec2 server create" the instance comes up fine.
Adding the same run_list to auto scale config, the converge fails. I
tried adding this to the node

override[:rabbitmq][:kernel] = {}

But that had no effect.
I did find someone else having a very similar issue installing with
Vagrant, but there was no response.

Any ideas?

NoMethodError

undefined method `each_pair' for :Array

Cookbook Trace:

/var/chef/cache/cookbooks/rabbitmq/libraries/default.rb:35:in

format_kernel_parameters' /var/chef/cache/cookbooks/rabbitmq/recipes/default.rb:174:in from_file'
/var/chef/cache/cookbooks/rabbitmq/recipes/default.rb:168:in
from_file' /var/chef/cache/cookbooks/bti_api/recipes/default.rb:10:in from_file'

Relevant File Content:

/var/chef/cache/cookbooks/rabbitmq/libraries/default.rb:

28: # This parameter is special and needs commas instead of
periods.
29: rendered << "{inet_dist_use_interface,
{#{kernel[:inet_dist_use_interface].gsub(/./, ',')}}}" if
kernel[:inet_dist_use_interface]
30: kernel.delete(:inet_dist_use_interface)
31:
32: # Otherwise, we can just render it nicely as Erlang
wants. This
33: # theoretically opens the door for arbitrary kernel_app
parameters to be
34: # declared.
35>> kernel.select { |k, v| !v.nil? }.each_pair do |param, val|
36: rendered << "{#{param}, #{val}}"
37: end
38:
39: rendered.each { |r| r.prepend(' ') }.join(",\n")
40: end
41: end
42: end
43:

Running handlers:
[2014-08-07T22:21:09+00:00] ERROR: Running exception handlers
Running handlers complete
[2014-08-07T22:21:09+00:00] ERROR: Exception handlers complete
[2014-08-07T22:21:09+00:00] FATAL: Stacktrace dumped to
/var/chef/cache/chef-stacktrace.out
Chef Client failed. 0 resources updated in 11.03884 seconds
[2014-08-07T22:21:09+00:00] INFO: Sending resource update report
(run-id: 90aeb225-8af0-4f21-a901-98258173bc03)
[2014-08-07T22:21:10+00:00] ERROR: undefined method `each_pair' for
:Array
[2014-08-07T22:21:10+00:00] FATAL:
Chef::Exceptions::ChildConvergeError: Chef run process exited
unsuccessfully (exit code 1)

--
Liam Kirsher
PGP: http://liam.numenet.com/pgp/

2

Sounds like that’s being overridden that in node attributes somehow. The default attributes[0] for that cookbook set a few attributes under kernel:

Erlang kernel application options

See http://www.erlang.org/doc/man/kernel_app.html

default[‘rabbitmq’][‘kernel’][‘inet_dist_listen_min’] = nil

default[‘rabbitmq’][‘kernel’][‘inet_dist_listen_max’] = nil

Tell Erlang what IP to bind to

default[‘rabbitmq’][‘kernel’][‘inet_dist_use_interface’] = nil

I would assume you’re setting that interface at least as the cookbook doesn’t attempt to determine that automatically as far as I can tell. Can you open chef-shell once it fails once being bootstrapped by AWS and see what node.rabbitmq.kernel actually is?

Dan

[0] https://github.com/kennonkwok/rabbitmq/blob/master/attributes/default.rb#L26-32

On Friday, August 8, 2014 at 10:38 AM, Liam Kirsher wrote:

Hope someone can help with this!
Converge fails when using AWS AutoScaling, but succeeds when doing a normal EC2 server create.
The source of the error is in the rabbitmq cookbook. Somehow the hash that is expected for kernel (I believe this is Erlang kernel?) becomes an array.

Starting with “knife ec2 server create” the instance comes up fine.
Adding the same run_list to auto scale config, the converge fails. I tried adding this to the node

override[:rabbitmq][:kernel] = {}

But that had no effect.
I did find someone else having a very similar issue installing with Vagrant, but there was no response.
https://github.com/kennonkwok/rabbitmq/issues/124

Any ideas?

NoMethodError

undefined method `each_pair’ for []:Array

Cookbook Trace:

/var/chef/cache/cookbooks/rabbitmq/libraries/default.rb:35:in `format_kernel_parameters'
/var/chef/cache/cookbooks/rabbitmq/recipes/default.rb:174:in `from_file'
/var/chef/cache/cookbooks/rabbitmq/recipes/default.rb:168:in `from_file'
/var/chef/cache/cookbooks/bti_api/recipes/default.rb:10:in `from_file'

Relevant File Content:

/var/chef/cache/cookbooks/rabbitmq/libraries/default.rb:

28: # This parameter is special and needs commas instead of periods.
29: rendered << “{inet_dist_use_interface, {#{kernel[:inet_dist_use_interface].gsub(/./, ‘,’)}}}” if kernel[:inet_dist_use_interface]
30: kernel.delete(:inet_dist_use_interface)
31:
32: # Otherwise, we can just render it nicely as Erlang wants. This
33: # theoretically opens the door for arbitrary kernel_app parameters to be
34: # declared.
35>> kernel.select { |k, v| !v.nil? }.each_pair do |param, val|
36: rendered << “{#{param}, #{val}}“
37: end
38:
39: rendered.each { |r| r.prepend(’ ') }.join(”,\n”)
40: end
41: end
42: end
43:

Running handlers:
[2014-08-07T22:21:09+00:00] ERROR: Running exception handlers
Running handlers complete
[2014-08-07T22:21:09+00:00] ERROR: Exception handlers complete
[2014-08-07T22:21:09+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
Chef Client failed. 0 resources updated in 11.03884 seconds
[2014-08-07T22:21:09+00:00] INFO: Sending resource update report (run-id: 90aeb225-8af0-4f21-a901-98258173bc03)
[2014-08-07T22:21:10+00:00] ERROR: undefined method `each_pair’ for []:Array
[2014-08-07T22:21:10+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

– Liam Kirsher PGP: http://liam.numenet.com/pgp/

3

On 08/08/2014 10:49 AM, Daniel Condomitti wrote:

Sounds like that’s being overridden that in node attributes somehow.
Yeah. I’ve searched for references to rabbitmq and kernel, and found
nothing. Not setting any of these in my recipe.
The default attributes[0] for that cookbook set a few attributes under
kernel:

Erlang kernel application options

See http://www.erlang.org/doc/man/kernel_app.html

default[‘rabbitmq’][‘kernel’][‘inet_dist_listen_min’] = nil

default[‘rabbitmq’][‘kernel’][‘inet_dist_listen_max’] = nil

Tell Erlang what IP to bind to

default[‘rabbitmq’][‘kernel’][‘inet_dist_use_interface’] = nil

I would assume you’re setting that interface at least as the cookbook
doesn’t attempt to determine that automatically as far as I can tell.
I think the kernel parameters must be optional.
Can you open chef-shell once it fails once being bootstrapped by AWS
and see what node.rabbitmq.kernel actually is?
I’m attempting to do this, but not familiar with chef-shell so that may
be why it’s not working. Can you offer a little more info on how to
start chef-shell and what commands to use in it?

chef-shell -c client.rb -j first-boot.json
…
chef > node.rabbitmq.kernel
NoMethodError: Undefined method or attribute rabbitmq' onnode’

Dan

[0] https://github.com/kennonkwok/rabbitmq/blob/master/attributes/default.rb#L26-32

On Friday, August 8, 2014 at 10:38 AM, Liam Kirsher wrote:

Hope someone can help with this!
Converge fails when using AWS AutoScaling, but succeeds when doing a
normal EC2 server create.
The source of the error is in the rabbitmq cookbook. Somehow the
hash that is expected for kernel (I believe this is Erlang kernel?)
becomes an array.

Starting with “knife ec2 server create” the instance comes up fine.
Adding the same run_list to auto scale config, the converge fails. I
tried adding this to the node

override[:rabbitmq][:kernel] = {}

But that had no effect.
I did find someone else having a very similar issue installing with
Vagrant, but there was no response.
https://github.com/kennonkwok/rabbitmq/issues/124

Any ideas?

NoMethodError

undefined method `each_pair’ for []:Array

Cookbook Trace:

/var/chef/cache/cookbooks/rabbitmq/libraries/default.rb:35:in

format_kernel_parameters' /var/chef/cache/cookbooks/rabbitmq/recipes/default.rb:174:infrom_file’
/var/chef/cache/cookbooks/rabbitmq/recipes/default.rb:168:in
from_file' /var/chef/cache/cookbooks/bti_api/recipes/default.rb:10:infrom_file’

Relevant File Content:

/var/chef/cache/cookbooks/rabbitmq/libraries/default.rb:

28: # This parameter is special and needs commas instead
of periods.
29: rendered << “{inet_dist_use_interface,
{#{kernel[:inet_dist_use_interface].gsub(/./, ‘,’)}}}” if
kernel[:inet_dist_use_interface]
30: kernel.delete(:inet_dist_use_interface)
31:
32: # Otherwise, we can just render it nicely as Erlang
wants. This
33: # theoretically opens the door for arbitrary
kernel_app parameters to be
34: # declared.
35>> kernel.select { |k, v| !v.nil? }.each_pair do |param, val|
36: rendered << “{#{param}, #{val}}“
37: end
38:
39: rendered.each { |r| r.prepend(’ ') }.join(”,\n”)
40: end
41: end
42: end
43:

Running handlers:
[2014-08-07T22:21:09+00:00] ERROR: Running exception handlers
Running handlers complete
[2014-08-07T22:21:09+00:00] ERROR: Exception handlers complete
[2014-08-07T22:21:09+00:00] FATAL: Stacktrace dumped to
/var/chef/cache/chef-stacktrace.out
Chef Client failed. 0 resources updated in 11.03884 seconds
[2014-08-07T22:21:09+00:00] INFO: Sending resource update report
(run-id: 90aeb225-8af0-4f21-a901-98258173bc03)
[2014-08-07T22:21:10+00:00] ERROR: undefined method `each_pair’ for
[]:Array
[2014-08-07T22:21:10+00:00] FATAL:
Chef::Exceptions::ChildConvergeError: Chef run process exited
unsuccessfully (exit code 1)

–
Liam Kirsher
PGP: http://liam.numenet.com/pgp/

–
Liam Kirsher
PGP: http://liam.numenet.com/pgp/

4

Fixed it. It turned out I had gotten two versions of ruby on there, and
that was messing things up. Exactly how I’m not sure, but it’s
working now.

My boot script now has –
apt-get update
apt-get upgrade
apt-get install -y ruby1.9.1-dev ruby1.9.1 rubygems s3cmd
gem1.9.1 install ohai chef --no-rdoc --no-ri --verbose
mkdir -p /etc/chef
cd /etc/chef

On 08/08/2014 03:11 PM, Liam Kirsher wrote:

On 08/08/2014 10:49 AM, Daniel Condomitti wrote:

Sounds like that’s being overridden that in node attributes somehow.
Yeah. I’ve searched for references to rabbitmq and kernel, and found
nothing. Not setting any of these in my recipe.
The default attributes[0] for that cookbook set a few attributes
under kernel:

Erlang kernel application options

See http://www.erlang.org/doc/man/kernel_app.html

default[‘rabbitmq’][‘kernel’][‘inet_dist_listen_min’] = nil

default[‘rabbitmq’][‘kernel’][‘inet_dist_listen_max’] = nil

Tell Erlang what IP to bind to

default[‘rabbitmq’][‘kernel’][‘inet_dist_use_interface’] = nil

I would assume you’re setting that interface at least as the cookbook
doesn’t attempt to determine that automatically as far as I can tell.
I think the kernel parameters must be optional.
Can you open chef-shell once it fails once being bootstrapped by AWS
and see what node.rabbitmq.kernel actually is?
I’m attempting to do this, but not familiar with chef-shell so that
may be why it’s not working. Can you offer a little more info on how
to start chef-shell and what commands to use in it?

chef-shell -c client.rb -j first-boot.json
…
chef > node.rabbitmq.kernel
NoMethodError: Undefined method or attribute rabbitmq' onnode’

Dan

[0] https://github.com/kennonkwok/rabbitmq/blob/master/attributes/default.rb#L26-32

On Friday, August 8, 2014 at 10:38 AM, Liam Kirsher wrote:

Hope someone can help with this!
Converge fails when using AWS AutoScaling, but succeeds when doing a
normal EC2 server create.
The source of the error is in the rabbitmq cookbook. Somehow the
hash that is expected for kernel (I believe this is Erlang kernel?)
becomes an array.

Starting with “knife ec2 server create” the instance comes up fine.
Adding the same run_list to auto scale config, the converge fails.
I tried adding this to the node

override[:rabbitmq][:kernel] = {}

But that had no effect.
I did find someone else having a very similar issue installing with
Vagrant, but there was no response.
https://github.com/kennonkwok/rabbitmq/issues/124

Any ideas?

NoMethodError

undefined method `each_pair’ for []:Array

Cookbook Trace:

/var/chef/cache/cookbooks/rabbitmq/libraries/default.rb:35:in

format_kernel_parameters' /var/chef/cache/cookbooks/rabbitmq/recipes/default.rb:174:infrom_file’
/var/chef/cache/cookbooks/rabbitmq/recipes/default.rb:168:in
from_file' /var/chef/cache/cookbooks/bti_api/recipes/default.rb:10:infrom_file’

Relevant File Content:

/var/chef/cache/cookbooks/rabbitmq/libraries/default.rb:

28: # This parameter is special and needs commas instead
of periods.
29: rendered << “{inet_dist_use_interface,
{#{kernel[:inet_dist_use_interface].gsub(/./, ‘,’)}}}” if
kernel[:inet_dist_use_interface]
30: kernel.delete(:inet_dist_use_interface)
31:
32: # Otherwise, we can just render it nicely as Erlang
wants. This
33: # theoretically opens the door for arbitrary
kernel_app parameters to be
34: # declared.
35>> kernel.select { |k, v| !v.nil? }.each_pair do |param,
val|
36: rendered << “{#{param}, #{val}}“
37: end
38:
39: rendered.each { |r| r.prepend(’ ') }.join(”,\n”)
40: end
41: end
42: end
43:

Running handlers:
[2014-08-07T22:21:09+00:00] ERROR: Running exception handlers
Running handlers complete
[2014-08-07T22:21:09+00:00] ERROR: Exception handlers complete
[2014-08-07T22:21:09+00:00] FATAL: Stacktrace dumped to
/var/chef/cache/chef-stacktrace.out
Chef Client failed. 0 resources updated in 11.03884 seconds
[2014-08-07T22:21:09+00:00] INFO: Sending resource update report
(run-id: 90aeb225-8af0-4f21-a901-98258173bc03)
[2014-08-07T22:21:10+00:00] ERROR: undefined method `each_pair’ for
[]:Array
[2014-08-07T22:21:10+00:00] FATAL:
Chef::Exceptions::ChildConvergeError: Chef run process exited
unsuccessfully (exit code 1)

–
Liam Kirsher
PGP: http://liam.numenet.com/pgp/

–
Liam Kirsher
PGP: http://liam.numenet.com/pgp/

–
Liam Kirsher
PGP: http://liam.numenet.com/pgp/