Ohai timeouts inside VPC


#1

I’m running chef and using ohai plugins inside EC2 instances running in a
VPC. Aside from creating the hints file, it seems that any ohai reloads are
horribly slow.

It looks like someone has created a cookbook as a workaround for a issue
with similar symptons. Is this a known issue?

Doug


#2

Hi Douglas,

Unfortunately, reliably detecting VPC nodes is difficult. If you are
only seeing slowness without the ohai hint in place, then my guess is
that it is a result of the fact that we detect being on ec2 by making
a network call to the metadata API:

If you are seeing the reload slowness even with the hint enabled, then
I would recommend running chef-client with debug logging turned on.
The timestamps in the log can often help find which plugins are slow
to load.

It looks like someone has created a cookbook as a workaround for a issue with similar symptons. Is this a known issue?

The difficulty and slowness of detecting EC2 nodes is known and was
one of the primary motivators for Ohai’s hint system. Currently,
using an ohai hint when you know you are on EC2 and disabling the
plugin when you know you are not on EC2 is the best method to avoid
the speed penalty (and unreliability) of the ec2 detection methods we
have.

The ec2 hint can be put on disk by the knife-ec2 plugin at the time of
provisioning or by the ohai cookbook’s ohai_hint LWRP on the first
chef-client run. (The ohai cookbook is also capable of disabling
plugins).

Hope this helps.

Cheers,

Steven

On Fri, Jan 30, 2015 at 5:47 PM, Douglas Garstang
doug.garstang@gmail.com wrote:

I’m running chef and using ohai plugins inside EC2 instances running in a
VPC. Aside from creating the hints file, it seems that any ohai reloads are
horribly slow.

It looks like someone has created a cookbook as a workaround for a issue
with similar symptons. Is this a known issue?

https://github.com/marcelom/ohai_ec2_hack

Doug


#3

Hi,

A quick follow up. I just saw your other thread about writing a VPC
plugin. It looks like my knowledge about it being difficult to detect
VPC might be out of date? If so, improvements to the ec2 plugin
itself are also welcome.

Cheers,

Steven

On Fri, Jan 30, 2015 at 7:20 PM, Steven Danna steve@chef.io wrote:

Hi Douglas,

Unfortunately, reliably detecting VPC nodes is difficult. If you are
only seeing slowness without the ohai hint in place, then my guess is
that it is a result of the fact that we detect being on ec2 by making
a network call to the metadata API:

https://github.com/chef/ohai/blob/master/lib/ohai/plugins/ec2.rb#L45

If you are seeing the reload slowness even with the hint enabled, then
I would recommend running chef-client with debug logging turned on.
The timestamps in the log can often help find which plugins are slow
to load.

It looks like someone has created a cookbook as a workaround for a issue with similar symptons. Is this a known issue?

The difficulty and slowness of detecting EC2 nodes is known and was
one of the primary motivators for Ohai’s hint system. Currently,
using an ohai hint when you know you are on EC2 and disabling the
plugin when you know you are not on EC2 is the best method to avoid
the speed penalty (and unreliability) of the ec2 detection methods we
have.

The ec2 hint can be put on disk by the knife-ec2 plugin at the time of
provisioning or by the ohai cookbook’s ohai_hint LWRP on the first
chef-client run. (The ohai cookbook is also capable of disabling
plugins).

Hope this helps.

Cheers,

Steven

On Fri, Jan 30, 2015 at 5:47 PM, Douglas Garstang
doug.garstang@gmail.com wrote:

I’m running chef and using ohai plugins inside EC2 instances running in a
VPC. Aside from creating the hints file, it seems that any ohai reloads are
horribly slow.

It looks like someone has created a cookbook as a workaround for a issue
with similar symptons. Is this a known issue?

https://github.com/marcelom/ohai_ec2_hack

Doug


#4

Steven,

I actually wrote a new plugin. Can’t use it tho. Looks like the data
returned by custom plugins is not available in attributes/default.rb (data
in core ohai seems to be though). Since I imagine this is the most likely
place you would want to check (I’m using it to check if it’s a VPC and set
attributes used by an upstream cookbook), then, well, it’s kinda pointless.
Actually, this would make most ohai plugins pointless. :frowning:

Doug.

On Fri, Jan 30, 2015 at 11:24 AM, Steven Danna steve@chef.io wrote:

Hi,

A quick follow up. I just saw your other thread about writing a VPC
plugin. It looks like my knowledge about it being difficult to detect
VPC might be out of date? If so, improvements to the ec2 plugin
itself are also welcome.

Cheers,

Steven

On Fri, Jan 30, 2015 at 7:20 PM, Steven Danna steve@chef.io wrote:

Hi Douglas,

Unfortunately, reliably detecting VPC nodes is difficult. If you are
only seeing slowness without the ohai hint in place, then my guess is
that it is a result of the fact that we detect being on ec2 by making
a network call to the metadata API:

https://github.com/chef/ohai/blob/master/lib/ohai/plugins/ec2.rb#L45

If you are seeing the reload slowness even with the hint enabled, then
I would recommend running chef-client with debug logging turned on.
The timestamps in the log can often help find which plugins are slow
to load.

It looks like someone has created a cookbook as a workaround for a
issue with similar symptons. Is this a known issue?

The difficulty and slowness of detecting EC2 nodes is known and was
one of the primary motivators for Ohai’s hint system. Currently,
using an ohai hint when you know you are on EC2 and disabling the
plugin when you know you are not on EC2 is the best method to avoid
the speed penalty (and unreliability) of the ec2 detection methods we
have.

The ec2 hint can be put on disk by the knife-ec2 plugin at the time of
provisioning or by the ohai cookbook’s ohai_hint LWRP on the first
chef-client run. (The ohai cookbook is also capable of disabling
plugins).

Hope this helps.

Cheers,

Steven

On Fri, Jan 30, 2015 at 5:47 PM, Douglas Garstang
doug.garstang@gmail.com wrote:

I’m running chef and using ohai plugins inside EC2 instances running in
a

VPC. Aside from creating the hints file, it seems that any ohai reloads
are

horribly slow.

It looks like someone has created a cookbook as a workaround for a issue
with similar symptons. Is this a known issue?

https://github.com/marcelom/ohai_ec2_hack

Doug


Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627


#5

This is what I’ve ended up with in attributes/default.rb. Not very pretty
but seems to work.

vpc_id = nil
node[‘ec2’][‘network_interfaces_macs’].each_pair do |iface, data|
vpc_id = data[‘vpc_id’] if data.key?(‘vpc_id’)
end

case vpc_id
when nil
default[‘openldap’][‘server’] = node[‘slice-ldap’][‘ldap_server_url’]
else
default[‘openldap’][‘server’] =
node[‘slice-ldap’][‘ldap_server_url_vpc’]
end

Is there a better way?

Doug.

On Fri, Jan 30, 2015 at 11:28 AM, Douglas Garstang doug.garstang@gmail.com
wrote:

Steven,

I actually wrote a new plugin. Can’t use it tho. Looks like the data
returned by custom plugins is not available in attributes/default.rb (data
in core ohai seems to be though). Since I imagine this is the most likely
place you would want to check (I’m using it to check if it’s a VPC and set
attributes used by an upstream cookbook), then, well, it’s kinda pointless.
Actually, this would make most ohai plugins pointless. :frowning:

Doug.

On Fri, Jan 30, 2015 at 11:24 AM, Steven Danna steve@chef.io wrote:

Hi,

A quick follow up. I just saw your other thread about writing a VPC
plugin. It looks like my knowledge about it being difficult to detect
VPC might be out of date? If so, improvements to the ec2 plugin
itself are also welcome.

Cheers,

Steven

On Fri, Jan 30, 2015 at 7:20 PM, Steven Danna steve@chef.io wrote:

Hi Douglas,

Unfortunately, reliably detecting VPC nodes is difficult. If you are
only seeing slowness without the ohai hint in place, then my guess is
that it is a result of the fact that we detect being on ec2 by making
a network call to the metadata API:

https://github.com/chef/ohai/blob/master/lib/ohai/plugins/ec2.rb#L45

If you are seeing the reload slowness even with the hint enabled, then
I would recommend running chef-client with debug logging turned on.
The timestamps in the log can often help find which plugins are slow
to load.

It looks like someone has created a cookbook as a workaround for a
issue with similar symptons. Is this a known issue?

The difficulty and slowness of detecting EC2 nodes is known and was
one of the primary motivators for Ohai’s hint system. Currently,
using an ohai hint when you know you are on EC2 and disabling the
plugin when you know you are not on EC2 is the best method to avoid
the speed penalty (and unreliability) of the ec2 detection methods we
have.

The ec2 hint can be put on disk by the knife-ec2 plugin at the time of
provisioning or by the ohai cookbook’s ohai_hint LWRP on the first
chef-client run. (The ohai cookbook is also capable of disabling
plugins).

Hope this helps.

Cheers,

Steven

On Fri, Jan 30, 2015 at 5:47 PM, Douglas Garstang
doug.garstang@gmail.com wrote:

I’m running chef and using ohai plugins inside EC2 instances running
in a

VPC. Aside from creating the hints file, it seems that any ohai
reloads are

horribly slow.

It looks like someone has created a cookbook as a workaround for a
issue

with similar symptons. Is this a known issue?

https://github.com/marcelom/ohai_ec2_hack

Doug


Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627


Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627


#6

Actually, I had to use this, even uglier. Without the has_key checks, this
will fail on non ec2 including vagrant.

vpc_id = nil
if node.has_key?(‘ec2’)
if node[:ec2].has_key?(‘network_interfaces_macs’)
node[‘ec2’][‘network_interfaces_macs’].each_pair do |iface, data|
vpc_id = data[‘vpc_id’] if data.key?(‘vpc_id’)
end
end
end
case vpc_id
when nil
default[‘openldap’][‘server’] = node[‘slice-ldap’][‘ldap_server_url’]
else
default[‘openldap’][‘server’] =
node[‘slice-ldap’][‘ldap_server_url_vpc’]
end

Seems ridiculous that we can’t use custom ohai plugins in this situation.

Doug.

On Fri, Jan 30, 2015 at 11:30 AM, Douglas Garstang doug.garstang@gmail.com
wrote:

This is what I’ve ended up with in attributes/default.rb. Not very pretty
but seems to work.

vpc_id = nil
node[‘ec2’][‘network_interfaces_macs’].each_pair do |iface, data|
vpc_id = data[‘vpc_id’] if data.key?(‘vpc_id’)
end

case vpc_id
when nil
default[‘openldap’][‘server’] = node[‘slice-ldap’][‘ldap_server_url’]
else
default[‘openldap’][‘server’] =
node[‘slice-ldap’][‘ldap_server_url_vpc’]
end

Is there a better way?

Doug.

On Fri, Jan 30, 2015 at 11:28 AM, Douglas Garstang <
doug.garstang@gmail.com> wrote:

Steven,

I actually wrote a new plugin. Can’t use it tho. Looks like the data
returned by custom plugins is not available in attributes/default.rb (data
in core ohai seems to be though). Since I imagine this is the most likely
place you would want to check (I’m using it to check if it’s a VPC and set
attributes used by an upstream cookbook), then, well, it’s kinda pointless.
Actually, this would make most ohai plugins pointless. :frowning:

Doug.

On Fri, Jan 30, 2015 at 11:24 AM, Steven Danna steve@chef.io wrote:

Hi,

A quick follow up. I just saw your other thread about writing a VPC
plugin. It looks like my knowledge about it being difficult to detect
VPC might be out of date? If so, improvements to the ec2 plugin
itself are also welcome.

Cheers,

Steven

On Fri, Jan 30, 2015 at 7:20 PM, Steven Danna steve@chef.io wrote:

Hi Douglas,

Unfortunately, reliably detecting VPC nodes is difficult. If you are
only seeing slowness without the ohai hint in place, then my guess is
that it is a result of the fact that we detect being on ec2 by making
a network call to the metadata API:

https://github.com/chef/ohai/blob/master/lib/ohai/plugins/ec2.rb#L45

If you are seeing the reload slowness even with the hint enabled, then
I would recommend running chef-client with debug logging turned on.
The timestamps in the log can often help find which plugins are slow
to load.

It looks like someone has created a cookbook as a workaround for a
issue with similar symptons. Is this a known issue?

The difficulty and slowness of detecting EC2 nodes is known and was
one of the primary motivators for Ohai’s hint system. Currently,
using an ohai hint when you know you are on EC2 and disabling the
plugin when you know you are not on EC2 is the best method to avoid
the speed penalty (and unreliability) of the ec2 detection methods we
have.

The ec2 hint can be put on disk by the knife-ec2 plugin at the time of
provisioning or by the ohai cookbook’s ohai_hint LWRP on the first
chef-client run. (The ohai cookbook is also capable of disabling
plugins).

Hope this helps.

Cheers,

Steven

On Fri, Jan 30, 2015 at 5:47 PM, Douglas Garstang
doug.garstang@gmail.com wrote:

I’m running chef and using ohai plugins inside EC2 instances running
in a

VPC. Aside from creating the hints file, it seems that any ohai
reloads are

horribly slow.

It looks like someone has created a cookbook as a workaround for a
issue

with similar symptons. Is this a known issue?

https://github.com/marcelom/ohai_ec2_hack

Doug


Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627


Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627


Regards,

Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627