Ohai 6.18 and 6.20 failing on ec2 hi1.xlarge

Chef Infra (archive)
1

hai.

I don’t know if the instance type has anything to do with this, but ohai is
failing on the only ones we use of this ec2 instance type. Ohai and chef had
been running fine on these for many weeks, but today started failing, and
I don’t know why. Nothing changed that I know of.

In general, we run chef 11.6.0 (ohai 6.18.0), but I grabbed 11.8.2 / 6.20.0 to
see if the behavior would change, and it didn’t.

On the problem host, accessing user-data and meta-data work just fine using
"curl http://169.254.169.254/latest/user-data", etc.

But “ohai -l debug ec2” throws:
#<RuntimeError: Encountered error retrieving EC2 metadata (returned 404 response)

[2013-12-12T20:36:14+00:00] DEBUG: Loading plugin ec2
[2013-12-12T20:36:14+00:00] DEBUG: has_ec2_mac? == true
[2013-12-12T20:36:14+00:00] DEBUG: can_metadata_connect? == true
[2013-12-12T20:36:14+00:00] DEBUG: looks_like_ec2? == true
[2013-12-12T20:36:14+00:00] DEBUG: EC2 metadata version: 2012-01-12
[2013-12-12T20:36:14+00:00] DEBUG: Plugin ec2 threw exception #<RuntimeError: Encountered error retrieving EC2 metadata (returned 404 response)> /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/ec2_metadata.rb:104:in metadata_get' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/ec2_metadata.rb:120:inblock in fetch_metadata’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/ec2_metadata.rb:113:in each' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/ec2_metadata.rb:113:infetch_metadata’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/ec2_metadata.rb:130:in block in fetch_metadata' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/ec2_metadata.rb:113:ineach’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/ec2_metadata.rb:113:in fetch_metadata' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/plugins/ec2.rb:52:infrom_file’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/from_file.rb:29:in instance_eval' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/from_file.rb:29:infrom_file’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:216:in block in require_plugin' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:211:ineach’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:211:in require_plugin' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/plugins/cloud.rb:19:infrom_file’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/from_file.rb:29:in instance_eval' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/from_file.rb:29:infrom_file’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:216:in block in require_plugin' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:211:ineach’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:211:in require_plugin' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:140:inblock (2 levels) in all_plugins’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:133:in each' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:133:inblock in all_plugins’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:131:in each' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:131:inall_plugins’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/application.rb:97:in run_application' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/application.rb:75:inrun’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/bin/ohai:51:in <top (required)>' /opt/chef/bin/ohai:23:inload’
/opt/chef/bin/ohai:23:in `’

Is this known? Any help or work arounds?

thanks!
kallen

2

Are you running with IAM roles with an '=' in the name?

On Thu, Dec 12, 2013 at 3:40 PM, kallen@groknaut.net wrote:

hai.

I don't know if the instance type has anything to do with this, but ohai is
failing on the only ones we use of this ec2 instance type. Ohai and chef
had
been running fine on these for many weeks, but today started failing, and
I don't know why. Nothing changed that I know of.

In general, we run chef 11.6.0 (ohai 6.18.0), but I grabbed 11.8.2 /
6.20.0 to
see if the behavior would change, and it didn't.

On the problem host, accessing user-data and meta-data work just fine using
"curl http://169.254.169.254/latest/user-data", etc.

But "ohai -l debug ec2" throws:
#<RuntimeError: Encountered error retrieving EC2 metadata (returned 404
response)

[2013-12-12T20:36:14+00:00] DEBUG: Loading plugin ec2
[2013-12-12T20:36:14+00:00] DEBUG: has_ec2_mac? == true
[2013-12-12T20:36:14+00:00] DEBUG: can_metadata_connect? == true
[2013-12-12T20:36:14+00:00] DEBUG: looks_like_ec2? == true
[2013-12-12T20:36:14+00:00] DEBUG: EC2 metadata version: 2012-01-12
[2013-12-12T20:36:14+00:00] DEBUG: Plugin ec2 threw exception
#<RuntimeError: Encountered error retrieving EC2 metadata (returned 404
response)>
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/ec2_metadata.rb:104:in
metadata_get' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/ec2_metadata.rb:120:in block in fetch_metadata'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/ec2_metadata.rb:113:in
each' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/ec2_metadata.rb:113:in fetch_metadata'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/ec2_metadata.rb:130:in
block in fetch_metadata' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/ec2_metadata.rb:113:in each'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/ec2_metadata.rb:113:in
fetch_metadata' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/plugins/ec2.rb:52:in from_file'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/from_file.rb:29:in
instance_eval' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/from_file.rb:29:in from_file'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:216:in
block in require_plugin' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:211:in each'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:211:in
require_plugin' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/plugins/cloud.rb:19:in from_file'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/from_file.rb:29:in
instance_eval' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/from_file.rb:29:in from_file'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:216:in
block in require_plugin' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:211:in each'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:211:in
require_plugin' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:140:in block (2 levels) in all_plugins'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:133:in
each' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:133:in block in all_plugins'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:131:in
each' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:131:in all_plugins'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/application.rb:97:in
run_application' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/application.rb:75:in run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/bin/ohai:51:in
<top (required)>' /opt/chef/bin/ohai:23:in load'
/opt/chef/bin/ohai:23:in `'

Is this known? Any help or work arounds?

thanks!
kallen

3

On Thu, 12 Dec 2013, Andrew Gross wrote:

Are you running with IAM roles with an '=' in the name?

No, we didn't launch this instance with an IAM instance profile.

On Thu, Dec 12, 2013 at 3:40 PM, kallen@groknaut.net wrote:

hai.

I don't know if the instance type has anything to do with this, but ohai is
failing on the only ones we use of this ec2 instance type. Ohai and chef
had
been running fine on these for many weeks, but today started failing, and
I don't know why. Nothing changed that I know of.

In general, we run chef 11.6.0 (ohai 6.18.0), but I grabbed 11.8.2 /
6.20.0 to
see if the behavior would change, and it didn't.

On the problem host, accessing user-data and meta-data work just fine using
"curl http://169.254.169.254/latest/user-data", etc.

But "ohai -l debug ec2" throws:
#<RuntimeError: Encountered error retrieving EC2 metadata (returned 404
response)

[2013-12-12T20:36:14+00:00] DEBUG: Loading plugin ec2
[2013-12-12T20:36:14+00:00] DEBUG: has_ec2_mac? == true
[2013-12-12T20:36:14+00:00] DEBUG: can_metadata_connect? == true
[2013-12-12T20:36:14+00:00] DEBUG: looks_like_ec2? == true
[2013-12-12T20:36:14+00:00] DEBUG: EC2 metadata version: 2012-01-12
[2013-12-12T20:36:14+00:00] DEBUG: Plugin ec2 threw exception
#<RuntimeError: Encountered error retrieving EC2 metadata (returned 404
response)>
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/ec2_metadata.rb:104:in
metadata_get' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/ec2_metadata.rb:120:in block in fetch_metadata'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/ec2_metadata.rb:113:in
each' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/ec2_metadata.rb:113:in fetch_metadata'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/ec2_metadata.rb:130:in
block in fetch_metadata' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/ec2_metadata.rb:113:in each'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/ec2_metadata.rb:113:in
fetch_metadata' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/plugins/ec2.rb:52:in from_file'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/from_file.rb:29:in
instance_eval' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/from_file.rb:29:in from_file'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:216:in
block in require_plugin' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:211:in each'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:211:in
require_plugin' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/plugins/cloud.rb:19:in from_file'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/from_file.rb:29:in
instance_eval' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/from_file.rb:29:in from_file'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:216:in
block in require_plugin' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:211:in each'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:211:in
require_plugin' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:140:in block (2 levels) in all_plugins'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:133:in
each' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:133:in block in all_plugins'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:131:in
each' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:131:in all_plugins'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/application.rb:97:in
run_application' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/application.rb:75:in run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/bin/ohai:51:in
<top (required)>' /opt/chef/bin/ohai:23:in load'
/opt/chef/bin/ohai:23:in `'

Is this known? Any help or work arounds?

thanks!
kallen

4

On Thu, 12 Dec 2013, kallen@groknaut.net wrote:

On Thu, 12 Dec 2013, Andrew Gross wrote:

Are you running with IAM roles with an '=' in the name?

No, we didn't launch this instance with an IAM instance profile.

I added some debugging to the ohai code, ran it, and I can see which URI it's
barfing on on once instance vs another:

http://169.254.169.254/2012-01-12/meta-data/metrics/vhostmd

That returns 200 on all of our instances except the 2 having problems right now.
And the 2 having problems only just started having this problem today. I think
I'll go talk to AWS support now.

thanks!

On Thu, Dec 12, 2013 at 3:40 PM, kallen@groknaut.net wrote:

hai.

I don't know if the instance type has anything to do with this, but ohai is
failing on the only ones we use of this ec2 instance type. Ohai and chef
had
been running fine on these for many weeks, but today started failing, and
I don't know why. Nothing changed that I know of.

In general, we run chef 11.6.0 (ohai 6.18.0), but I grabbed 11.8.2 /
6.20.0 to
see if the behavior would change, and it didn't.

On the problem host, accessing user-data and meta-data work just fine using
"curl http://169.254.169.254/latest/user-data", etc.

But "ohai -l debug ec2" throws:
#<RuntimeError: Encountered error retrieving EC2 metadata (returned 404
response)

[2013-12-12T20:36:14+00:00] DEBUG: Loading plugin ec2
[2013-12-12T20:36:14+00:00] DEBUG: has_ec2_mac? == true
[2013-12-12T20:36:14+00:00] DEBUG: can_metadata_connect? == true
[2013-12-12T20:36:14+00:00] DEBUG: looks_like_ec2? == true
[2013-12-12T20:36:14+00:00] DEBUG: EC2 metadata version: 2012-01-12
[2013-12-12T20:36:14+00:00] DEBUG: Plugin ec2 threw exception
#<RuntimeError: Encountered error retrieving EC2 metadata (returned 404
response)>
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/ec2_metadata.rb:104:in
metadata_get' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/ec2_metadata.rb:120:in block in fetch_metadata'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/ec2_metadata.rb:113:in
each' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/ec2_metadata.rb:113:in fetch_metadata'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/ec2_metadata.rb:130:in
block in fetch_metadata' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/ec2_metadata.rb:113:in each'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/ec2_metadata.rb:113:in
fetch_metadata' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/plugins/ec2.rb:52:in from_file'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/from_file.rb:29:in
instance_eval' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/from_file.rb:29:in from_file'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:216:in
block in require_plugin' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:211:in each'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:211:in
require_plugin' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/plugins/cloud.rb:19:in from_file'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/from_file.rb:29:in
instance_eval' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/from_file.rb:29:in from_file'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:216:in
block in require_plugin' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:211:in each'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:211:in
require_plugin' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:140:in block (2 levels) in all_plugins'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:133:in
each' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:133:in block in all_plugins'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:131:in
each' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:131:in all_plugins'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/application.rb:97:in
run_application' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/application.rb:75:in run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/bin/ohai:51:in
<top (required)>' /opt/chef/bin/ohai:23:in load'
/opt/chef/bin/ohai:23:in `'

Is this known? Any help or work arounds?

thanks!
kallen

5

On Thu, 12 Dec 2013, kallen@groknaut.net wrote:

On Thu, 12 Dec 2013, kallen@groknaut.net wrote:

On Thu, 12 Dec 2013, Andrew Gross wrote:

Are you running with IAM roles with an '=' in the name?

No, we didn't launch this instance with an IAM instance profile.

I added some debugging to the ohai code, ran it, and I can see which URI it's
barfing on on once instance vs another:

http://169.254.169.254/2012-01-12/meta-data/metrics/vhostmd

That returns 200 on all of our instances except the 2 having problems right now.
And the 2 having problems only just started having this problem today. I think
I'll go talk to AWS support now.

After talking with AWS support, that URL now returns 200, and ohai and chef
works again (because node['ec2']['userdata'] no longer returns nil).

I'm wondering, if just one URI returns 404 to ohai, but all others don't, why is
the ohai result an empty Hash? By querying ec2 metadata by hand, I could see
all of what my recipes cared about. Is ohai all or nothing?

kallen

On Thu, Dec 12, 2013 at 3:40 PM, kallen@groknaut.net wrote:

hai.

I don't know if the instance type has anything to do with this, but ohai is
failing on the only ones we use of this ec2 instance type. Ohai and chef
had
been running fine on these for many weeks, but today started failing, and
I don't know why. Nothing changed that I know of.

In general, we run chef 11.6.0 (ohai 6.18.0), but I grabbed 11.8.2 /
6.20.0 to
see if the behavior would change, and it didn't.

On the problem host, accessing user-data and meta-data work just fine using
"curl http://169.254.169.254/latest/user-data", etc.

But "ohai -l debug ec2" throws:
#<RuntimeError: Encountered error retrieving EC2 metadata (returned 404
response)

[2013-12-12T20:36:14+00:00] DEBUG: Loading plugin ec2
[2013-12-12T20:36:14+00:00] DEBUG: has_ec2_mac? == true
[2013-12-12T20:36:14+00:00] DEBUG: can_metadata_connect? == true
[2013-12-12T20:36:14+00:00] DEBUG: looks_like_ec2? == true
[2013-12-12T20:36:14+00:00] DEBUG: EC2 metadata version: 2012-01-12
[2013-12-12T20:36:14+00:00] DEBUG: Plugin ec2 threw exception
#<RuntimeError: Encountered error retrieving EC2 metadata (returned 404
response)>
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/ec2_metadata.rb:104:in
metadata_get' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/ec2_metadata.rb:120:in block in fetch_metadata'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/ec2_metadata.rb:113:in
each' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/ec2_metadata.rb:113:in fetch_metadata'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/ec2_metadata.rb:130:in
block in fetch_metadata' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/ec2_metadata.rb:113:in each'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/ec2_metadata.rb:113:in
fetch_metadata' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/plugins/ec2.rb:52:in from_file'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/from_file.rb:29:in
instance_eval' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/from_file.rb:29:in from_file'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:216:in
block in require_plugin' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:211:in each'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:211:in
require_plugin' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/plugins/cloud.rb:19:in from_file'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/from_file.rb:29:in
instance_eval' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/mixin/from_file.rb:29:in from_file'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:216:in
block in require_plugin' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:211:in each'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:211:in
require_plugin' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:140:in block (2 levels) in all_plugins'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:133:in
each' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:133:in block in all_plugins'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:131:in
each' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/system.rb:131:in all_plugins'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/application.rb:97:in
run_application' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/lib/ohai/application.rb:75:in run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/ohai-6.18.0/bin/ohai:51:in
<top (required)>' /opt/chef/bin/ohai:23:in load'
/opt/chef/bin/ohai:23:in `'

Is this known? Any help or work arounds?

thanks!
kallen

6

On Friday, December 13, 2013 at 11:38 AM, kallen@groknaut.net wrote:

After talking with AWS support, that URL now returns 200, and ohai and chef
works again (because node['ec2']['userdata'] no longer returns nil).

I'm wondering, if just one URI returns 404 to ohai, but all others don't, why is
the ohai result an empty Hash? By querying ec2 metadata by hand, I could see
all of what my recipes cared about. Is ohai all or nothing?

kallen
Most ohai plugins are written such that a single exception will fail the whole plugin, and ohai is currently designed to let plugins fail. Upcoming changes in ohai 7.0 will support specifying a set of required attributes, so if a plugin that provides those attributes fails, ohai will expose the error instead of swallowing it. This feature may not make it into 7.0.0, but the internal changes are in place for it already, so if the feature isn’t finished for 7.0.0 we’ll add it during the 7.x series.

--
Daniel DeLeo