AWS cookbook and the run list

Chef Infra (archive)
1

I want to use the AWS cookbook to attach EBS volumes. I wasn’t sure what
the best way to ensure that it only happens when the node is actually an
EC2 instance, and not a local Vagrant VM. Separate roles? Separate
environments and use env_run_lists? Is there something like if
node[‘is_ec2’]?

Thanks!
Greg

2

We make pretty frequent usage of

if node[:virtualization][:system] == "vbox"

On Wed, Jan 14, 2015 at 1:47 PM, Greg Barker fletch@fletchowns.net wrote:

I want to use the AWS cookbook to attach EBS volumes. I wasn't sure what
the best way to ensure that it only happens when the node is actually an
EC2 instance, and not a local Vagrant VM. Separate roles? Separate
environments and use env_run_lists? Is there something like if
node['is_ec2']?

Thanks!
Greg

3

On Jan 14, 2015, at 11:47 AM, Greg Barker fletch@fletchowns.net wrote:

I want to use the AWS cookbook to attach EBS volumes. I wasn't sure what the best way to ensure that it only happens when the node is actually an EC2 instance, and not a local Vagrant VM. Separate roles? Separate environments and use env_run_lists? Is there something like if node['is_ec2']?

Yep, you can do this:

if node['ec2']
include_recipe 'myebsstuff'
end

--Noah

4

Awesome!! Thank you!

On Wed, Jan 14, 2015 at 11:54 AM, Noah Kantrowitz noah@coderanger.net
wrote:

On Jan 14, 2015, at 11:47 AM, Greg Barker fletch@fletchowns.net wrote:

I want to use the AWS cookbook to attach EBS volumes. I wasn't sure what
the best way to ensure that it only happens when the node is actually an
EC2 instance, and not a local Vagrant VM. Separate roles? Separate
environments and use env_run_lists? Is there something like if
node['is_ec2']?

Yep, you can do this:

if node['ec2']
include_recipe 'myebsstuff'
end

--Noah

5

So I added this to my metadata.rb:

depends 'aws'

Added this to my run_list:

"recipe[aws]",
"recipe[aws::ec2_hints]",

Added this to my recipe:

if node["ec2"]
include_recipe "mycookbook::ec2_specific_thing"
end

Works great when I ran it on an EC2 instance. But when I tried a vagrant up for a local VM, it hits an error in my ec2_specific_thing recipe. I
thought it wasn't going to be executed at all though. Any ideas?

Thanks!
Greg

On Wed, Jan 14, 2015 at 12:03 PM, Greg Barker fletch@fletchowns.net wrote:

Awesome!! Thank you!

On Wed, Jan 14, 2015 at 11:54 AM, Noah Kantrowitz noah@coderanger.net
wrote:

On Jan 14, 2015, at 11:47 AM, Greg Barker fletch@fletchowns.net wrote:

I want to use the AWS cookbook to attach EBS volumes. I wasn't sure
what the best way to ensure that it only happens when the node is actually
an EC2 instance, and not a local Vagrant VM. Separate roles? Separate
environments and use env_run_lists? Is there something like if
node['is_ec2']?

Yep, you can do this:

if node['ec2']
include_recipe 'myebsstuff'
end

--Noah

6

Should I be relying on the node['cloud']['provider'] instead?

On Wed, Jan 21, 2015 at 7:07 PM, Greg Barker fletch@fletchowns.net wrote:

So I added this to my metadata.rb:

depends 'aws'

Added this to my run_list:

"recipe[aws]",
"recipe[aws::ec2_hints]",

Added this to my recipe:

if node["ec2"]
include_recipe "mycookbook::ec2_specific_thing"
end

Works great when I ran it on an EC2 instance. But when I tried a vagrant up for a local VM, it hits an error in my ec2_specific_thing recipe. I
thought it wasn't going to be executed at all though. Any ideas?

Thanks!
Greg

On Wed, Jan 14, 2015 at 12:03 PM, Greg Barker fletch@fletchowns.net
wrote:

Awesome!! Thank you!

On Wed, Jan 14, 2015 at 11:54 AM, Noah Kantrowitz noah@coderanger.net
wrote:

On Jan 14, 2015, at 11:47 AM, Greg Barker fletch@fletchowns.net wrote:

I want to use the AWS cookbook to attach EBS volumes. I wasn't sure
what the best way to ensure that it only happens when the node is actually
an EC2 instance, and not a local Vagrant VM. Separate roles? Separate
environments and use env_run_lists? Is there something like if
node['is_ec2']?

Yep, you can do this:

if node['ec2']
include_recipe 'myebsstuff'
end

--Noah

7

On Wed, Jan 21, 2015 at 10:07 PM, Greg Barker fletch@fletchowns.net wrote:

So I added this to my metadata.rb:

depends 'aws'

Added this to my run_list:

"recipe[aws]",
"recipe[aws::ec2_hints]",

Added this to my recipe:

if node["ec2"]
include_recipe "mycookbook::ec2_specific_thing"
end

Works great when I ran it on an EC2 instance. But when I tried a vagrant up for a local VM, it hits an error in my ec2_specific_thing recipe. I
thought it wasn't going to be executed at all though. Any ideas?

That shouldn't happen. Can you paste the error?

  • Julian
8

Julian,

That’s the expected behavior, right? aws::ec2_hints adds the ec2 hints file for ohai which causes the node to be recognized as EC2 even if it isn’t. It doesn’t have any normal attributes populated but the node[‘ec2’] key exists.

This is a CentOS 6.5 node running in VirtualBox

chef-shell -z

...
chef > File.exists?('/etc/chef/ohai/hints/ec2.json')
=> false
chef > node['ec2']
=> nil
chef > exit

mkdir -p /etc/chef/ohai/hints

touch ohai/hints/ec2.json

chef-shell -z

....
chef > File.exists?('/etc/chef/ohai/hints/ec2.json')
=> true
chef > node['ec2']
=> #<Chef::Node::Attribute @normal={...}, @current_normal=nil, @default={...}, @current_default=nil, @override={...}, @current_override=nil, @automatic={...}, @current_automatic={}, @current_nesting_level=["ec2"], @auto_vivifiy_on_read=false, @set_unless_value_present=false, @set_type=nil, @has_been_read=false>

On Friday, January 23, 2015 at 7:11 PM, Julian C. Dunn wrote:

On Wed, Jan 21, 2015 at 10:07 PM, Greg Barker <fletch@fletchowns.net (mailto:fletch@fletchowns.net)> wrote:

So I added this to my metadata.rb:

depends 'aws'

Added this to my run_list:

"recipe[aws]",
"recipe[aws::ec2_hints]",

Added this to my recipe:

if node["ec2"]
include_recipe "mycookbook::ec2_specific_thing"
end

Works great when I ran it on an EC2 instance. But when I tried a vagrant up for a local VM, it hits an error in my ec2_specific_thing recipe. I
thought it wasn't going to be executed at all though. Any ideas?

That shouldn't happen. Can you paste the error?

  • Julian
9

Ah yes. I missed the first part of Greg's email that stated that his
run_list contains recipe[aws::ec2_hints]. So... maybe not a good thing
to do when running under Vagrant :slight_smile:

  • Julian

On Fri, Jan 23, 2015 at 7:29 PM, Daniel Condomitti
daniel@condomitti.com wrote:

Julian,

That’s the expected behavior, right? aws::ec2_hints adds the ec2 hints file
for ohai which causes the node to be recognized as EC2 even if it isn’t. It
doesn’t have any normal attributes populated but the node[‘ec2’] key exists.

This is a CentOS 6.5 node running in VirtualBox

chef-shell -z

...
chef > File.exists?('/etc/chef/ohai/hints/ec2.json')
=> false
chef > node['ec2']
=> nil
chef > exit

mkdir -p /etc/chef/ohai/hints

touch ohai/hints/ec2.json

chef-shell -z

....
chef > File.exists?('/etc/chef/ohai/hints/ec2.json')
=> true
chef > node['ec2']
=> #<Chef::Node::Attribute @normal={...}, @current_normal=nil,
@default={...}, @current_default=nil, @override={...},
@current_override=nil, @automatic={...}, @current_automatic={},
@current_nesting_level=["ec2"], @auto_vivifiy_on_read=false,
@set_unless_value_present=false, @set_type=nil, @has_been_read=false>

On Friday, January 23, 2015 at 7:11 PM, Julian C. Dunn wrote:

On Wed, Jan 21, 2015 at 10:07 PM, Greg Barker fletch@fletchowns.net wrote:

So I added this to my metadata.rb:

depends 'aws'

Added this to my run_list:

"recipe[aws]",
"recipe[aws::ec2_hints]",

Added this to my recipe:

if node["ec2"]
include_recipe "mycookbook::ec2_specific_thing"
end

Works great when I ran it on an EC2 instance. But when I tried a vagrant up for a local VM, it hits an error in my ec2_specific_thing recipe. I
thought it wasn't going to be executed at all though. Any ideas?

That shouldn't happen. Can you paste the error?

  • Julian

--
[ Julian C. Dunn jdunn@aquezada.com * Sorry, I'm ]
[ WWW: Julian Dunn's Blog - Commentary on media, technology, and everything in between. * only Web 1.0 ]
[ gopher://sdf.org/1/users/keymaker/ * compliant! ]
[ PGP: 91B3 7A9D 683C 7C16 715F 442C 6065 D533 FDC2 05B9 ]

10

I was hoping to avoid adding complexity to the run list by creating a new
environment for EC2 and having to specify the env_run_lists in my base
role. If I'm going down that route I would probably just put
mycookbook::ec2_specific_
thing on the run_list there, which negates the necessity for using if
node["ec2"].

It seems like a much simpler way to solve it would be something like:

if !node["ec2"].empty?
include_recipe "mycookbook::ec2_specific_thing"
end

Or maybe relying on another attribute like node['cloud']['provider']?

Or am I crazy for trying to avoid env_run_lists here?

Greg

On Sat, Jan 24, 2015 at 9:46 AM, Julian C. Dunn jdunn@aquezada.com wrote:

Ah yes. I missed the first part of Greg's email that stated that his
run_list contains recipe[aws::ec2_hints]. So... maybe not a good thing
to do when running under Vagrant :slight_smile:

  • Julian

On Fri, Jan 23, 2015 at 7:29 PM, Daniel Condomitti
daniel@condomitti.com wrote:

Julian,

That’s the expected behavior, right? aws::ec2_hints adds the ec2 hints
file
for ohai which causes the node to be recognized as EC2 even if it isn’t.
It
doesn’t have any normal attributes populated but the node[‘ec2’] key
exists.

This is a CentOS 6.5 node running in VirtualBox

chef-shell -z

...
chef > File.exists?('/etc/chef/ohai/hints/ec2.json')
=> false
chef > node['ec2']
=> nil
chef > exit

mkdir -p /etc/chef/ohai/hints

touch ohai/hints/ec2.json

chef-shell -z

....
chef > File.exists?('/etc/chef/ohai/hints/ec2.json')
=> true
chef > node['ec2']
=> #<Chef::Node::Attribute @normal={...}, @current_normal=nil,
@default={...}, @current_default=nil, @override={...},
@current_override=nil, @automatic={...}, @current_automatic={},
@current_nesting_level=["ec2"], @auto_vivifiy_on_read=false,
@set_unless_value_present=false, @set_type=nil, @has_been_read=false>

On Friday, January 23, 2015 at 7:11 PM, Julian C. Dunn wrote:

On Wed, Jan 21, 2015 at 10:07 PM, Greg Barker fletch@fletchowns.net
wrote:

So I added this to my metadata.rb:

depends 'aws'

Added this to my run_list:

"recipe[aws]",
"recipe[aws::ec2_hints]",

Added this to my recipe:

if node["ec2"]
include_recipe "mycookbook::ec2_specific_thing"
end

Works great when I ran it on an EC2 instance. But when I tried a vagrant up for a local VM, it hits an error in my ec2_specific_thing recipe. I
thought it wasn't going to be executed at all though. Any ideas?

That shouldn't happen. Can you paste the error?

  • Julian

--
[ Julian C. Dunn jdunn@aquezada.com * Sorry, I'm ]
[ WWW: Julian Dunn's Blog - Commentary on media, technology, and everything in between. * only Web 1.0 ]
[ gopher://sdf.org/1/users/keymaker/ * compliant! ]
[ PGP: 91B3 7A9D 683C 7C16 715F 442C 6065 D533 FDC2 05B9 ]

11

The problem is that there’s no reliable way to detect if you’re running in EC2, Rackspace Cloud or on bare metal hardware so hints are required[0]. We work around this by adding a provider attribute to our bootstrap JSON:

<% if ['openstack', 'ec2' ].include? first_boot['provider'] %>
mkdir -p /etc/chef/ohai/hints
touch /etc/chef/ohai/hints/<%= first_boot['provider'] %>.json
<% end %>

This is in our bootstrap template and our tools that kick off EC2 provisioning or vSphere VM creation sets provider to one of openstack, vsphere, ec2, or leaves it out if it’s bare metal.

Dan

[0] https://tickets.opscode.com/browse/OHAI-310

On Monday, January 26, 2015 at 8:47 PM, Greg Barker wrote:

I was hoping to avoid adding complexity to the run list by creating a new environment for EC2 and having to specify the env_run_lists in my base role. If I'm going down that route I would probably just put mycookbook::ec2_specific_
thing on the run_list there, which negates the necessity for using if node["ec2"].

It seems like a much simpler way to solve it would be something like:
if !node["ec2"].empty? include_recipe "mycookbook::ec2_specific_thing" endOr maybe relying on another attribute like node['cloud']['provider']?

Or am I crazy for trying to avoid env_run_lists here?

Greg

On Sat, Jan 24, 2015 at 9:46 AM, Julian C. Dunn <jdunn@aquezada.com (mailto:jdunn@aquezada.com)> wrote:

Ah yes. I missed the first part of Greg's email that stated that his
run_list contains recipe[aws::ec2_hints]. So... maybe not a good thing
to do when running under Vagrant :slight_smile:

  • Julian

On Fri, Jan 23, 2015 at 7:29 PM, Daniel Condomitti
<daniel@condomitti.com (mailto:daniel@condomitti.com)> wrote:

Julian,

That’s the expected behavior, right? aws::ec2_hints adds the ec2 hints file
for ohai which causes the node to be recognized as EC2 even if it isn’t. It
doesn’t have any normal attributes populated but the node[‘ec2’] key exists.

This is a CentOS 6.5 node running in VirtualBox

chef-shell -z

...
chef > File.exists?('/etc/chef/ohai/hints/ec2.json')
=> false
chef > node['ec2']
=> nil
chef > exit

mkdir -p /etc/chef/ohai/hints

touch ohai/hints/ec2.json

chef-shell -z

....
chef > File.exists?('/etc/chef/ohai/hints/ec2.json')
=> true
chef > node['ec2']
=> #<Chef::Node::Attribute @normal={...}, @current_normal=nil,
@default={...}, @current_default=nil, @override={...},
@current_override=nil, @automatic={...}, @current_automatic={},
@current_nesting_level=["ec2"], @auto_vivifiy_on_read=false,
@set_unless_value_present=false, @set_type=nil, @has_been_read=false>

On Friday, January 23, 2015 at 7:11 PM, Julian C. Dunn wrote:

On Wed, Jan 21, 2015 at 10:07 PM, Greg Barker <fletch@fletchowns.net (mailto:fletch@fletchowns.net)> wrote:

So I added this to my metadata.rb:

depends 'aws'

Added this to my run_list:

"recipe[aws]",
"recipe[aws::ec2_hints]",

Added this to my recipe:

if node["ec2"]
include_recipe "mycookbook::ec2_specific_thing"
end

Works great when I ran it on an EC2 instance. But when I tried a vagrant up for a local VM, it hits an error in my ec2_specific_thing recipe. I
thought it wasn't going to be executed at all though. Any ideas?

That shouldn't happen. Can you paste the error?

  • Julian

--
[ Julian C. Dunn <jdunn@aquezada.com (mailto:jdunn@aquezada.com)> * Sorry, I'm ]
[ WWW: Julian Dunn's Blog - Commentary on media, technology, and everything in between. * only Web 1.0 ]
[ gopher://sdf.org/1/users/keymaker/ (http://sdf.org/1/users/keymaker/) * compliant! ]
[ PGP: 91B3 7A9D 683C 7C16 715F 442C 6065 D533 FDC2 05B9 ]

12

I think you're down a rabbit-hole of circular preconditions here.

The purpose of the aws::ec2_hints recipe is to create the hints file
when you, the operator, are sure that the box is running on EC2. You
can't just unconditionally run that recipe because it would just cause
the rest of the Chef run to assume you're always running under EC2.

Using another attribute (node['cloud']['provider'] or whatnot) won't
really help, because the point is that there's no reliable way to
automatically detect if a box is in EC2 or not. Hence the need for the
hints file.

A couple points of advice:

  1. You may or may not even need the hints file if Ohai can actually
    detect that your box is in EC2.
  2. If that doesn't work, drop the file off during the initial
    bootstrap, outside of a recipe. knife-ec2 will do this for you.
  3. Alternately, use some other condition that you trust to determine
    whether the box is in EC2, and set the hint and/or invoke
    aws::ec2_hints appropriately. e.g.

if some_connection_to_the_ec2_metadata_service_worked
include_recipe('aws::ec2_hints')
end

  1. env_run_lists are, more often than not, an unnecessary complication
    and I'd avoid them where possible.
  • Julian

On Mon, Jan 26, 2015 at 8:47 PM, Greg Barker fletch@fletchowns.net wrote:

I was hoping to avoid adding complexity to the run list by creating a new
environment for EC2 and having to specify the env_run_lists in my base role.
If I'm going down that route I would probably just put
mycookbook::ec2_specific_
thing on the run_list there, which negates the necessity for using if
node["ec2"].

It seems like a much simpler way to solve it would be something like:

if !node["ec2"].empty?
include_recipe "mycookbook::ec2_specific_thing"
end

Or maybe relying on another attribute like node['cloud']['provider']?

Or am I crazy for trying to avoid env_run_lists here?

Greg

On Sat, Jan 24, 2015 at 9:46 AM, Julian C. Dunn jdunn@aquezada.com wrote:

Ah yes. I missed the first part of Greg's email that stated that his
run_list contains recipe[aws::ec2_hints]. So... maybe not a good thing
to do when running under Vagrant :slight_smile:

  • Julian

On Fri, Jan 23, 2015 at 7:29 PM, Daniel Condomitti
daniel@condomitti.com wrote:

Julian,

That’s the expected behavior, right? aws::ec2_hints adds the ec2 hints
file
for ohai which causes the node to be recognized as EC2 even if it isn’t.
It
doesn’t have any normal attributes populated but the node[‘ec2’] key
exists.

This is a CentOS 6.5 node running in VirtualBox

chef-shell -z

...
chef > File.exists?('/etc/chef/ohai/hints/ec2.json')
=> false
chef > node['ec2']
=> nil
chef > exit

mkdir -p /etc/chef/ohai/hints

touch ohai/hints/ec2.json

chef-shell -z

....
chef > File.exists?('/etc/chef/ohai/hints/ec2.json')
=> true
chef > node['ec2']
=> #<Chef::Node::Attribute @normal={...}, @current_normal=nil,
@default={...}, @current_default=nil, @override={...},
@current_override=nil, @automatic={...}, @current_automatic={},
@current_nesting_level=["ec2"], @auto_vivifiy_on_read=false,
@set_unless_value_present=false, @set_type=nil, @has_been_read=false>

On Friday, January 23, 2015 at 7:11 PM, Julian C. Dunn wrote:

On Wed, Jan 21, 2015 at 10:07 PM, Greg Barker fletch@fletchowns.net
wrote:

So I added this to my metadata.rb:

depends 'aws'

Added this to my run_list:

"recipe[aws]",
"recipe[aws::ec2_hints]",

Added this to my recipe:

if node["ec2"]
include_recipe "mycookbook::ec2_specific_thing"
end

Works great when I ran it on an EC2 instance. But when I tried a
vagrant up for a local VM, it hits an error in my ec2_specific_thing recipe. I
thought it wasn't going to be executed at all though. Any ideas?

That shouldn't happen. Can you paste the error?

  • Julian

--
[ Julian C. Dunn jdunn@aquezada.com * Sorry, I'm ]
[ WWW: Julian Dunn's Blog - Commentary on media, technology, and everything in between. * only Web 1.0 ]
[ gopher://sdf.org/1/users/keymaker/ * compliant! ]
[ PGP: 91B3 7A9D 683C 7C16 715F 442C 6065 D533 FDC2 05B9 ]

--
[ Julian C. Dunn jdunn@aquezada.com * Sorry, I'm ]
[ WWW: Julian Dunn's Blog - Commentary on media, technology, and everything in between. * only Web 1.0 ]
[ gopher://sdf.org/1/users/keymaker/ * compliant! ]
[ PGP: 91B3 7A9D 683C 7C16 715F 442C 6065 D533 FDC2 05B9 ]