Ohai timeouts inside VPC

Douglas_Garstang · January 30, 2015, 5:47pm

I'm running chef and using ohai plugins inside EC2 instances running in a
VPC. Aside from creating the hints file, it seems that any ohai reloads are
horribly slow.

It looks like someone has created a cookbook as a workaround for a issue
with similar symptons. Is this a known issue?

Doug

ssd · January 30, 2015, 7:20pm

Hi Douglas,

Unfortunately, reliably detecting VPC nodes is difficult. If you are
only seeing slowness without the ohai hint in place, then my guess is
that it is a result of the fact that we detect being on ec2 by making
a network call to the metadata API:

github.com

chef/ohai/blob/main/lib/ohai/plugins/ec2.rb#L45


      
          
          # look for amazon string in dmi vendor bios data within the sys tree.
          # this works even if the system lacks dmidecode use by the Dmi plugin
          # this gets us detection of new Xen-less HVM instances that are within a VPC
          # @return [Boolean] do we have Amazon DMI data?
          def has_ec2_amazon_dmi?
            # detect a version of '4.2.amazon'
            if file_val_if_exists("/sys/class/dmi/id/bios_vendor").to_s.include?("Amazon")
              logger.trace("Plugin EC2: has_ec2_amazon_dmi? == true")
              true
            else
              logger.trace("Plugin EC2: has_ec2_amazon_dmi? == false")
              false
            end
          end
          
          # look for amazon string in dmi bios version data within the sys tree.
          # this works even if the system lacks dmidecode use by the Dmi plugin
          # this gets us detection of HVM instances that are within a VPC
          # @return [Boolean] do we have Amazon DMI data?
          def has_ec2_xen_dmi?

If you are seeing the reload slowness even with the hint enabled, then
I would recommend running chef-client with debug logging turned on.
The timestamps in the log can often help find which plugins are slow
to load.

It looks like someone has created a cookbook as a workaround for a issue with similar symptons. Is this a known issue?

The difficulty and slowness of detecting EC2 nodes is known and was
one of the primary motivators for Ohai's hint system. Currently,
using an ohai hint when you know you are on EC2 and disabling the
plugin when you know you are not on EC2 is the best method to avoid
the speed penalty (and unreliability) of the ec2 detection methods we
have.

The ec2 hint can be put on disk by the knife-ec2 plugin at the time of
provisioning or by the ohai cookbook's ohai_hint LWRP on the first
chef-client run. (The ohai cookbook is also capable of disabling
plugins).

Hope this helps.

Cheers,

Steven

On Fri, Jan 30, 2015 at 5:47 PM, Douglas Garstang
doug.garstang@gmail.com wrote:

I'm running chef and using ohai plugins inside EC2 instances running in a
VPC. Aside from creating the hints file, it seems that any ohai reloads are
horribly slow.

It looks like someone has created a cookbook as a workaround for a issue
with similar symptons. Is this a known issue?

GitHub - marcelom/ohai_ec2_hack: Fixed Ohai EC2 bug (OHAI-310) detecting EC2 instance inside VPC.

Doug

ssd · January 30, 2015, 7:24pm

Hi,

A quick follow up. I just saw your other thread about writing a VPC
plugin. It looks like my knowledge about it being difficult to detect
VPC might be out of date? If so, improvements to the ec2 plugin
itself are also welcome.

Cheers,

Steven

On Fri, Jan 30, 2015 at 7:20 PM, Steven Danna steve@chef.io wrote:

Hi Douglas,

Unfortunately, reliably detecting VPC nodes is difficult. If you are
only seeing slowness without the ohai hint in place, then my guess is
that it is a result of the fact that we detect being on ec2 by making
a network call to the metadata API:

https://github.com/chef/ohai/blob/master/lib/ohai/plugins/ec2.rb#L45

If you are seeing the reload slowness even with the hint enabled, then
I would recommend running chef-client with debug logging turned on.
The timestamps in the log can often help find which plugins are slow
to load.

It looks like someone has created a cookbook as a workaround for a issue with similar symptons. Is this a known issue?

The difficulty and slowness of detecting EC2 nodes is known and was
one of the primary motivators for Ohai's hint system. Currently,
using an ohai hint when you know you are on EC2 and disabling the
plugin when you know you are not on EC2 is the best method to avoid
the speed penalty (and unreliability) of the ec2 detection methods we
have.

The ec2 hint can be put on disk by the knife-ec2 plugin at the time of
provisioning or by the ohai cookbook's ohai_hint LWRP on the first
chef-client run. (The ohai cookbook is also capable of disabling
plugins).

Hope this helps.

Cheers,

Steven

On Fri, Jan 30, 2015 at 5:47 PM, Douglas Garstang
doug.garstang@gmail.com wrote:

I'm running chef and using ohai plugins inside EC2 instances running in a
VPC. Aside from creating the hints file, it seems that any ohai reloads are
horribly slow.

It looks like someone has created a cookbook as a workaround for a issue
with similar symptons. Is this a known issue?

GitHub - marcelom/ohai_ec2_hack: Fixed Ohai EC2 bug (OHAI-310) detecting EC2 instance inside VPC.

Doug

Douglas_Garstang · January 30, 2015, 7:28pm

Steven,

I actually wrote a new plugin. Can't use it tho. Looks like the data
returned by custom plugins is not available in attributes/default.rb (data
in core ohai seems to be though). Since I imagine this is the most likely
place you would want to check (I'm using it to check if it's a VPC and set
attributes used by an upstream cookbook), then, well, it's kinda pointless.
Actually, this would make most ohai plugins pointless.

Doug.

On Fri, Jan 30, 2015 at 11:24 AM, Steven Danna steve@chef.io wrote:

Hi,

A quick follow up. I just saw your other thread about writing a VPC
plugin. It looks like my knowledge about it being difficult to detect
VPC might be out of date? If so, improvements to the ec2 plugin
itself are also welcome.

Cheers,

Steven

On Fri, Jan 30, 2015 at 7:20 PM, Steven Danna steve@chef.io wrote:

Hi Douglas,

Unfortunately, reliably detecting VPC nodes is difficult. If you are
only seeing slowness without the ohai hint in place, then my guess is
that it is a result of the fact that we detect being on ec2 by making
a network call to the metadata API:

https://github.com/chef/ohai/blob/master/lib/ohai/plugins/ec2.rb#L45

If you are seeing the reload slowness even with the hint enabled, then
I would recommend running chef-client with debug logging turned on.
The timestamps in the log can often help find which plugins are slow
to load.

It looks like someone has created a cookbook as a workaround for a
issue with similar symptons. Is this a known issue?

The difficulty and slowness of detecting EC2 nodes is known and was
one of the primary motivators for Ohai's hint system. Currently,
using an ohai hint when you know you are on EC2 and disabling the
plugin when you know you are not on EC2 is the best method to avoid
the speed penalty (and unreliability) of the ec2 detection methods we
have.

The ec2 hint can be put on disk by the knife-ec2 plugin at the time of
provisioning or by the ohai cookbook's ohai_hint LWRP on the first
chef-client run. (The ohai cookbook is also capable of disabling
plugins).

Hope this helps.

Cheers,

Steven

On Fri, Jan 30, 2015 at 5:47 PM, Douglas Garstang
doug.garstang@gmail.com wrote:

I'm running chef and using ohai plugins inside EC2 instances running in
a
VPC. Aside from creating the hints file, it seems that any ohai reloads
are
horribly slow.

It looks like someone has created a cookbook as a workaround for a issue
with similar symptons. Is this a known issue?

GitHub - marcelom/ohai_ec2_hack: Fixed Ohai EC2 bug (OHAI-310) detecting EC2 instance inside VPC.

Doug

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

Douglas_Garstang · January 30, 2015, 7:30pm

This is what I've ended up with in attributes/default.rb. Not very pretty
but seems to work.

vpc_id = nil
node['ec2']['network_interfaces_macs'].each_pair do |iface, data|
vpc_id = data['vpc_id'] if data.key?('vpc_id')
end

case vpc_id
when nil
default['openldap']['server'] = node['slice-ldap']['ldap_server_url']
else
default['openldap']['server'] =
node['slice-ldap']['ldap_server_url_vpc']
end

Is there a better way?

Doug.

On Fri, Jan 30, 2015 at 11:28 AM, Douglas Garstang doug.garstang@gmail.com
wrote:

Steven,

I actually wrote a new plugin. Can't use it tho. Looks like the data
returned by custom plugins is not available in attributes/default.rb (data
in core ohai seems to be though). Since I imagine this is the most likely
place you would want to check (I'm using it to check if it's a VPC and set
attributes used by an upstream cookbook), then, well, it's kinda pointless.
Actually, this would make most ohai plugins pointless.

Doug.

On Fri, Jan 30, 2015 at 11:24 AM, Steven Danna steve@chef.io wrote:

Hi,

A quick follow up. I just saw your other thread about writing a VPC
plugin. It looks like my knowledge about it being difficult to detect
VPC might be out of date? If so, improvements to the ec2 plugin
itself are also welcome.

Cheers,

Steven

On Fri, Jan 30, 2015 at 7:20 PM, Steven Danna steve@chef.io wrote:

Hi Douglas,

Unfortunately, reliably detecting VPC nodes is difficult. If you are
only seeing slowness without the ohai hint in place, then my guess is
that it is a result of the fact that we detect being on ec2 by making
a network call to the metadata API:

https://github.com/chef/ohai/blob/master/lib/ohai/plugins/ec2.rb#L45

If you are seeing the reload slowness even with the hint enabled, then
I would recommend running chef-client with debug logging turned on.
The timestamps in the log can often help find which plugins are slow
to load.

It looks like someone has created a cookbook as a workaround for a
issue with similar symptons. Is this a known issue?

The difficulty and slowness of detecting EC2 nodes is known and was
one of the primary motivators for Ohai's hint system. Currently,
using an ohai hint when you know you are on EC2 and disabling the
plugin when you know you are not on EC2 is the best method to avoid
the speed penalty (and unreliability) of the ec2 detection methods we
have.

The ec2 hint can be put on disk by the knife-ec2 plugin at the time of
provisioning or by the ohai cookbook's ohai_hint LWRP on the first
chef-client run. (The ohai cookbook is also capable of disabling
plugins).

Hope this helps.

Cheers,

Steven

On Fri, Jan 30, 2015 at 5:47 PM, Douglas Garstang
doug.garstang@gmail.com wrote:

I'm running chef and using ohai plugins inside EC2 instances running
in a
VPC. Aside from creating the hints file, it seems that any ohai
reloads are
horribly slow.

It looks like someone has created a cookbook as a workaround for a
issue
with similar symptons. Is this a known issue?

GitHub - marcelom/ohai_ec2_hack: Fixed Ohai EC2 bug (OHAI-310) detecting EC2 instance inside VPC.

Doug

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

Douglas_Garstang · January 30, 2015, 7:51pm

Actually, I had to use this, even uglier. Without the has_key checks, this
will fail on non ec2 including vagrant.

vpc_id = nil
if node.has_key?('ec2')
if node[:ec2].has_key?('network_interfaces_macs')
node['ec2']['network_interfaces_macs'].each_pair do |iface, data|
vpc_id = data['vpc_id'] if data.key?('vpc_id')
end
end
end
case vpc_id
when nil
default['openldap']['server'] = node['slice-ldap']['ldap_server_url']
else
default['openldap']['server'] =
node['slice-ldap']['ldap_server_url_vpc']
end

Seems ridiculous that we can't use custom ohai plugins in this situation.

Doug.

On Fri, Jan 30, 2015 at 11:30 AM, Douglas Garstang doug.garstang@gmail.com
wrote:

This is what I've ended up with in attributes/default.rb. Not very pretty
but seems to work.

vpc_id = nil
node['ec2']['network_interfaces_macs'].each_pair do |iface, data|
vpc_id = data['vpc_id'] if data.key?('vpc_id')
end

case vpc_id
when nil
default['openldap']['server'] = node['slice-ldap']['ldap_server_url']
else
default['openldap']['server'] =
node['slice-ldap']['ldap_server_url_vpc']
end

Is there a better way?

Doug.

On Fri, Jan 30, 2015 at 11:28 AM, Douglas Garstang <
doug.garstang@gmail.com> wrote:

Steven,

I actually wrote a new plugin. Can't use it tho. Looks like the data
returned by custom plugins is not available in attributes/default.rb (data
in core ohai seems to be though). Since I imagine this is the most likely
place you would want to check (I'm using it to check if it's a VPC and set
attributes used by an upstream cookbook), then, well, it's kinda pointless.
Actually, this would make most ohai plugins pointless.

Doug.

On Fri, Jan 30, 2015 at 11:24 AM, Steven Danna steve@chef.io wrote:

Hi,

A quick follow up. I just saw your other thread about writing a VPC
plugin. It looks like my knowledge about it being difficult to detect
VPC might be out of date? If so, improvements to the ec2 plugin
itself are also welcome.

Cheers,

Steven

On Fri, Jan 30, 2015 at 7:20 PM, Steven Danna steve@chef.io wrote:

Hi Douglas,

Unfortunately, reliably detecting VPC nodes is difficult. If you are
only seeing slowness without the ohai hint in place, then my guess is
that it is a result of the fact that we detect being on ec2 by making
a network call to the metadata API:

https://github.com/chef/ohai/blob/master/lib/ohai/plugins/ec2.rb#L45

If you are seeing the reload slowness even with the hint enabled, then
I would recommend running chef-client with debug logging turned on.
The timestamps in the log can often help find which plugins are slow
to load.

It looks like someone has created a cookbook as a workaround for a
issue with similar symptons. Is this a known issue?

The difficulty and slowness of detecting EC2 nodes is known and was
one of the primary motivators for Ohai's hint system. Currently,
using an ohai hint when you know you are on EC2 and disabling the
plugin when you know you are not on EC2 is the best method to avoid
the speed penalty (and unreliability) of the ec2 detection methods we
have.

The ec2 hint can be put on disk by the knife-ec2 plugin at the time of
provisioning or by the ohai cookbook's ohai_hint LWRP on the first
chef-client run. (The ohai cookbook is also capable of disabling
plugins).

Hope this helps.

Cheers,

Steven

On Fri, Jan 30, 2015 at 5:47 PM, Douglas Garstang
doug.garstang@gmail.com wrote:

I'm running chef and using ohai plugins inside EC2 instances running
in a
VPC. Aside from creating the hints file, it seems that any ohai
reloads are
horribly slow.

It looks like someone has created a cookbook as a workaround for a
issue
with similar symptons. Is this a known issue?

GitHub - marcelom/ohai_ec2_hack: Fixed Ohai EC2 bug (OHAI-310) detecting EC2 instance inside VPC.

Doug

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

--
Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627

Topic		Replies	Views
Still necessary to hint Ohai on EC2 VPC? Chef Infra (archive)	4	424	August 12, 2014
Ohai on a AWS VPC instance Chef Infra (archive)	2	859	January 21, 2014
Detecting AWS VPC w/OHAI Chef Infra (archive)	2	805	May 1, 2014
Ohai Custom plugin for VPC Chef Infra (archive)	3	358	January 30, 2015
IPMI / Ohai Chef Infra (archive)	11	687	September 24, 2013

Ohai timeouts inside VPC

Related topics