Knife-ec2 instance creation and management

Chef Infra (archive)
1

Hello,

I am new to Chef and am evaluating it for future use at my company. Over the last week I have found a few useful tutorials to get me going. One in-particular is the EC2 Bootstrap guide which I’ve used with varied success.

http://wiki.opscode.com/display/chef/EC2+Bootstrap+Fast+Start+Guide

I have followed the instructions from this and have successfully spun up a small ubuntu server using the following options in my knife.rb file

knife[:image] = 'ami-70f96e40’
knife[:user] = 'ubuntu’
knife[:availability_zone] = 'us-west-2b’
knife[:region] = 'us-west-2’
knife[:aws_access_key_id] = "secret"
knife[:aws_secret_access_key] = "secret"
knife[:aws_ssh_key_id] = "secret"
knife[:identity_file] = "/Users/jandrews/.ssh/secret.pem"
knife[:ssh_user] = “ubuntu”

Next I create a run list to install the apache2 cookbook since that is the web server I am most comfortable with. The run list looks good so I move to Step 6 of the tutorial which allows me to execute “chef-client” remotely for the node I spun up. The command it says to use is:

knife ssh name:NODENAME -x ubuntu “sudo chef-client” -a ec2.public_hostname

The “-a ec2.public_hostname” is suppose to tell knife to use the public hostname of the server since the FDQN is set to the internal ip address. The problem is that it does not work. I instead get the following error

“FATAL: 1 node found, but does not have the required attribute to establish the connection. Try setting another attribute to open the connection using --attribute.”

Googling has given me no useful information. I looked at the node attributes via the web browser on opscode.com and notice an ec2 node attributes with no value, and wonder if that’s suppose to be a json valued key/pair list.

So it seems either something is broken with knife-ec2 or I am missing something minor. I would really appreciate any insight anyone has on how to get this command to work.

Thanks
James

2

Hi James,

What node attributes do you see when you run:

knife search node 'name:NODENAME' -a ec2

You should see all of the ec2 related attributes cataloged by ohai...
do you see a valid public_hostname?

-K.

On Wed, Jun 19, 2013 at 4:54 PM, James Andrews james@openbridge.com wrote:

Hello,

I am new to Chef and am evaluating it for future use at my company. Over the last week I have found a few useful tutorials to get me going. One in-particular is the EC2 Bootstrap guide which I've used with varied success.

http://wiki.opscode.com/display/chef/EC2+Bootstrap+Fast+Start+Guide

I have followed the instructions from this and have successfully spun up a small ubuntu server using the following options in my knife.rb file

knife[:image] = 'ami-70f96e40'
knife[:user] = 'ubuntu'
knife[:availability_zone] = 'us-west-2b'
knife[:region] = 'us-west-2'
knife[:aws_access_key_id] = "secret"
knife[:aws_secret_access_key] = "secret"
knife[:aws_ssh_key_id] = "secret"
knife[:identity_file] = "/Users/jandrews/.ssh/secret.pem"
knife[:ssh_user] = "ubuntu"

Next I create a run list to install the apache2 cookbook since that is the web server I am most comfortable with. The run list looks good so I move to Step 6 of the tutorial which allows me to execute "chef-client" remotely for the node I spun up. The command it says to use is:

knife ssh name:NODENAME -x ubuntu "sudo chef-client" -a ec2.public_hostname

The "-a ec2.public_hostname" is suppose to tell knife to use the public hostname of the server since the FDQN is set to the internal ip address. The problem is that it does not work. I instead get the following error

"FATAL: 1 node found, but does not have the required attribute to establish the connection. Try setting another attribute to open the connection using --attribute."

Googling has given me no useful information. I looked at the node attributes via the web browser on opscode.com and notice an ec2 node attributes with no value, and wonder if that's suppose to be a json valued key/pair list.

So it seems either something is broken with knife-ec2 or I am missing something minor. I would really appreciate any insight anyone has on how to get this command to work.

Thanks
James

--
Kevin Karwaski
Operations Engineer @ {Fiksu}

3

Hi Kevin,

I’ve listed it below. Seems the public ip and host name are not included.

Thanks,
James

i-22371517:
ec2:
ami_id: ami-70f96e40
ami_launch_index: 0
ami_manifest_path: (unknown)
block_device_mapping_ami: /dev/sda1
block_device_mapping_ephemeral0: sdb
block_device_mapping_root: /dev/sda1
block_device_mapping_swap: sda3
hostname: ip-172-31-41-147
instance_action: none
instance_id: i-22371517
instance_type: m1.small
kernel_id: aki-fc37bacc
local_hostname: ip-172-31-41-147
local_ipv4: 172.31.41.147
mac: 0a:1f:63:c8:c9:bc
metrics_vhostmd: <?xml version="1.0" encoding="UTF-8"?>
network_interfaces_macs:
0a:1f:63:c8:c9:bc:
device_number: 0
interface_id: eni-ee3aa786
local_hostname: ip-172-31-41-147
local_ipv4s: 172.31.41.147
mac: 0a:1f:63:c8:c9:bc
owner_id: 120268158211
security_group_ids: sg-abe106c4
security_groups: default
subnet_id: subnet-aac148c2
subnet_ipv4_cidr_block: 172.31.32.0/20
vpc_id: vpc-a9c148c1
vpc_ipv4_cidr_block: 172.31.0.0/16
placement_availability_zone: us-west-2b
profile: default-paravirtual
public_keys_0_openssh_key: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCwASllyHCUrLiXkFvE8Eg7DixbtdJIiR+UOTcOmg5hP19kkN8MQDoW3FnR+X92JHtG/ArmzYJGGAHPv6Y9kbeKjkOnlucDLAveH7X9PhjWmxrycl9qVOnHIEvbFoPjNds/sshdh9A96RggGhjVkKkGuwE1Ttjx/1tcOkii+ypsT24PDYr9eGF5TI9eKcjmBiAPAWejBjcSZltzxr0eVjPql80Z+6moe3G4J145dDiqKdN4eqTtOP9Hi+g3Bj8BqdkNZAfEyvkh/tkebkGQBfGwLpb4ws/UnQLsbn09tBtdsXAV34ywJrBRQKz3hngjsBkXfwHedllxTPBUlG6GqqU/ awsjames

reservation_id:                  r-f49941c1
security_groups:                 default
userdata:
4

Also, I think you need ticks around the search query and the command:

knife ssh 'name:NODENAME' 'sudo chef-client' -x ubuntu -a ec2.public_hostname

-K.

On Wed, Jun 19, 2013 at 5:05 PM, Kevin Karwaski kkarwaski@fiksu.com wrote:

Hi James,

What node attributes do you see when you run:

knife search node 'name:NODENAME' -a ec2

You should see all of the ec2 related attributes cataloged by ohai...
do you see a valid public_hostname?

-K.

On Wed, Jun 19, 2013 at 4:54 PM, James Andrews james@openbridge.com wrote:

Hello,

I am new to Chef and am evaluating it for future use at my company. Over the last week I have found a few useful tutorials to get me going. One in-particular is the EC2 Bootstrap guide which I've used with varied success.

http://wiki.opscode.com/display/chef/EC2+Bootstrap+Fast+Start+Guide

I have followed the instructions from this and have successfully spun up a small ubuntu server using the following options in my knife.rb file

knife[:image] = 'ami-70f96e40'
knife[:user] = 'ubuntu'
knife[:availability_zone] = 'us-west-2b'
knife[:region] = 'us-west-2'
knife[:aws_access_key_id] = "secret"
knife[:aws_secret_access_key] = "secret"
knife[:aws_ssh_key_id] = "secret"
knife[:identity_file] = "/Users/jandrews/.ssh/secret.pem"
knife[:ssh_user] = "ubuntu"

Next I create a run list to install the apache2 cookbook since that is the web server I am most comfortable with. The run list looks good so I move to Step 6 of the tutorial which allows me to execute "chef-client" remotely for the node I spun up. The command it says to use is:

knife ssh name:NODENAME -x ubuntu "sudo chef-client" -a ec2.public_hostname

The "-a ec2.public_hostname" is suppose to tell knife to use the public hostname of the server since the FDQN is set to the internal ip address. The problem is that it does not work. I instead get the following error

"FATAL: 1 node found, but does not have the required attribute to establish the connection. Try setting another attribute to open the connection using --attribute."

Googling has given me no useful information. I looked at the node attributes via the web browser on opscode.com and notice an ec2 node attributes with no value, and wonder if that's suppose to be a json valued key/pair list.

So it seems either something is broken with knife-ec2 or I am missing something minor. I would really appreciate any insight anyone has on how to get this command to work.

Thanks
James

--
Kevin Karwaski
Operations Engineer @ {Fiksu}

--
Kevin Karwaski
Operations Engineer @ {Fiksu}

5

I had added the ticks before, and got the same error. What I was using was copied from the wiki article.

Thanks
James

6

I think the issue may be related to the use of a VPC. I believe you
have to assign an EIP to an instance in a VPC to have a public
resolvable CNAME associated with the instance.

On Wed, Jun 19, 2013 at 5:17 PM, James Andrews james@openbridge.com wrote:

I had added the ticks before, and got the same error. What I was using was copied from the wiki article.

Thanks
James

--
Kevin Karwaski
Operations Engineer @ {Fiksu}

7

It looks like EC2 instances launched in non-default VPCs do not
receive public addressing:

Hope this helps!

On Wed, Jun 19, 2013 at 5:25 PM, Kevin Karwaski kkarwaski@fiksu.com wrote:

I think the issue may be related to the use of a VPC. I believe you
have to assign an EIP to an instance in a VPC to have a public
resolvable CNAME associated with the instance.

Forums | AWS re:Post

On Wed, Jun 19, 2013 at 5:17 PM, James Andrews james@openbridge.com wrote:

I had added the ticks before, and got the same error. What I was using was copied from the wiki article.

Thanks
James

--
Kevin Karwaski
Operations Engineer @ {Fiksu}

--
Kevin Karwaski
Operations Engineer @ {Fiksu}

8

It does have a public CNAME though. If I go to the control panel and click on the instance The instance info has.

Public DNS:ec2-54-218-44-44.us-west-2.compute.amazonaws.com

So it is getting a host name and an IP address. I am fully able to ssh into the machine and run "chef-client" manually, so it's not a problem of it not getting an IP address it's a problem of Chef not retaining that information.

Thanks
James

On Jun 20, 2013, at 6:25 AM, Kevin Karwaski wrote:

I think the issue may be related to the use of a VPC. I believe you
have to assign an EIP to an instance in a VPC to have a public
resolvable CNAME associated with the instance.

Forums | AWS re:Post

On Wed, Jun 19, 2013 at 5:17 PM, James Andrews james@openbridge.com wrote:

I had added the ticks before, and got the same error. What I was using was copied from the wiki article.

Thanks
James

--
Kevin Karwaski
Operations Engineer @ {Fiksu}

9

Also, Looking at the AWS panel I have no assigned Elastic IPs, so it's not a vpc server even though it says vpc_id I believe that all regular ec2 instances run on an amazon owned vpc and not a customer configured vpc if that makes any sense. Meaning Amazon itself uses a non customer managed vpc to manage non vpc instances.

james

On Jun 20, 2013, at 6:31 AM, Kevin Karwaski wrote:

It looks like EC2 instances launched in non-default VPCs do not
receive public addressing:

Virtual private clouds - Amazon Elastic Compute Cloud

Hope this helps!

On Wed, Jun 19, 2013 at 5:25 PM, Kevin Karwaski kkarwaski@fiksu.com wrote:

I think the issue may be related to the use of a VPC. I believe you
have to assign an EIP to an instance in a VPC to have a public
resolvable CNAME associated with the instance.

Forums | AWS re:Post

On Wed, Jun 19, 2013 at 5:17 PM, James Andrews james@openbridge.com wrote:

I had added the ticks before, and got the same error. What I was using was copied from the wiki article.

Thanks
James

--
Kevin Karwaski
Operations Engineer @ {Fiksu}

--
Kevin Karwaski
Operations Engineer @ {Fiksu}

10

Does the public_hostname appear in the on-instance metadata? Is the
external hostname returned properly when you run this on your
instance?

curl http://169.254.169.254/latest/meta-data/public-hostname

On Wed, Jun 19, 2013 at 5:40 PM, James Andrews james@openbridge.com wrote:

Also, Looking at the AWS panel I have no assigned Elastic IPs, so it's not a vpc server even though it says vpc_id I believe that all regular ec2 instances run on an amazon owned vpc and not a customer configured vpc if that makes any sense. Meaning Amazon itself uses a non customer managed vpc to manage non vpc instances.

james

On Jun 20, 2013, at 6:31 AM, Kevin Karwaski wrote:

It looks like EC2 instances launched in non-default VPCs do not
receive public addressing:

Virtual private clouds - Amazon Elastic Compute Cloud

Hope this helps!

On Wed, Jun 19, 2013 at 5:25 PM, Kevin Karwaski kkarwaski@fiksu.com wrote:

I think the issue may be related to the use of a VPC. I believe you
have to assign an EIP to an instance in a VPC to have a public
resolvable CNAME associated with the instance.

Forums | AWS re:Post

On Wed, Jun 19, 2013 at 5:17 PM, James Andrews james@openbridge.com wrote:

I had added the ticks before, and got the same error. What I was using was copied from the wiki article.

Thanks
James

--
Kevin Karwaski
Operations Engineer @ {Fiksu}

--
Kevin Karwaski
Operations Engineer @ {Fiksu}

--
Kevin Karwaski
Operations Engineer @ {Fiksu}

11

I get the following after running that curl command.

curl http://169.254.169.254/latest/meta-data/public-hostname

<?xml version="1.0" encoding="iso-8859-1"?> 404 - Not Found

404 - Not Found

I ran “knife ec2 server list” I get

Instance ID Name Public IP Private IP Flavor Image SSH Key Security Groups State
i-22371517 i-22371517 54.218.44.44 172.31.41.147 m1.small ami-70f96e40 awsjames default running

So it sees the server sees the public IP address I just don’t understand why it’s not in the meta data.

James

12

Ohai should be capturing this info from the on-instance EC2 metadata
store; from the output of your curl command above it looks like that
info is not present on your instance. You ran that curl command from
the EC2 instance, right?

Here's what ohai is grabbing:

This is what I see when I hit the metadata store on an EC2 instance:

curl http://169.254.169.254/latest/meta-data/

ami-id
ami-launch-index
ami-manifest-path
ancestor-ami-ids
block-device-mapping/
hostname
instance-action
instance-id
instance-type
kernel-id
local-hostname
local-ipv4
mac
metrics/
network/
placement/
profile
public-hostname
public-ipv4
public-keys/
reservation-id

curl http://169.254.169.254/latest/meta-data/public-hostname

BLAH-BLAH-BLAH.compute-1.amazonaws.com

Do you see anything cataloged in the EC2 metadata store?

On Wed, Jun 19, 2013 at 5:52 PM, James Andrews james@openbridge.com wrote:

I get the following after running that curl command.

curl http://169.254.169.254/latest/meta-data/public-hostname

<?xml version="1.0" encoding="iso-8859-1"?> 404 - Not Found

404 - Not Found

I ran "knife ec2 server list" I get

Instance ID Name Public IP Private IP Flavor Image SSH Key Security Groups State
i-22371517 i-22371517 54.218.44.44 172.31.41.147 m1.small ami-70f96e40 awsjames default running

So it sees the server sees the public IP address I just don't understand why it's not in the meta data.

James

--
Kevin Karwaski
Operations Engineer @ {Fiksu}

13

Hi Kevin,

Thank you for all your help. I did some digging and I got it to work. It was related to vpc, and I'll go ahead an explain it in detail so others don't have to spend 3 days banging their heads against the wall.

Amazon sets "default" vpcs for all regions except "N. Virginia". I use using "Oregon" (us-west2). I tried to delete the vpc but got some nasty warning. It seems that you are unable to delete these VPCs because if you do you won't be able to create EC2 instances in those regions anymore.

After changing my knife.rb file to use us-east-1c and generating a new pem file. I was able to execute the knife file to remotely execute chef-client. So the moral of the story is use "us-east" region until there is a work around for these other regions.

Again, thank you very much.
James

On Jun 20, 2013, at 7:02 AM, Kevin Karwaski wrote:

Ohai should be capturing this info from the on-instance EC2 metadata
store; from the output of your curl command above it looks like that
info is not present on your instance. You ran that curl command from
the EC2 instance, right?

Here's what ohai is grabbing:
ohai/lib/ohai/mixin/ec2_metadata.rb at main · chef/ohai · GitHub

This is what I see when I hit the metadata store on an EC2 instance:

curl http://169.254.169.254/latest/meta-data/

ami-id
ami-launch-index
ami-manifest-path
ancestor-ami-ids
block-device-mapping/
hostname
instance-action
instance-id
instance-type
kernel-id
local-hostname
local-ipv4
mac
metrics/
network/
placement/
profile
public-hostname
public-ipv4
public-keys/
reservation-id

curl http://169.254.169.254/latest/meta-data/public-hostname

BLAH-BLAH-BLAH.compute-1.amazonaws.com

Do you see anything cataloged in the EC2 metadata store?

On Wed, Jun 19, 2013 at 5:52 PM, James Andrews james@openbridge.com wrote:

I get the following after running that curl command.

curl http://169.254.169.254/latest/meta-data/public-hostname

<?xml version="1.0" encoding="iso-8859-1"?> 404 - Not Found

404 - Not Found

I ran "knife ec2 server list" I get

Instance ID Name Public IP Private IP Flavor Image SSH Key Security Groups State
i-22371517 i-22371517 54.218.44.44 172.31.41.147 m1.small ami-70f96e40 awsjames default running

So it sees the server sees the public IP address I just don't understand why it's not in the meta data.

James

--
Kevin Karwaski
Operations Engineer @ {Fiksu}

14

Your welcome! Thanks for sharing some of your findings!

-K.

On Wed, Jun 19, 2013 at 6:36 PM, James Andrews james@openbridge.com wrote:

Hi Kevin,

Thank you for all your help. I did some digging and I got it to work. It was related to vpc, and I'll go ahead an explain it in detail so others don't have to spend 3 days banging their heads against the wall.

Amazon sets "default" vpcs for all regions except "N. Virginia". I use using "Oregon" (us-west2). I tried to delete the vpc but got some nasty warning. It seems that you are unable to delete these VPCs because if you do you won't be able to create EC2 instances in those regions anymore.

After changing my knife.rb file to use us-east-1c and generating a new pem file. I was able to execute the knife file to remotely execute chef-client. So the moral of the story is use "us-east" region until there is a work around for these other regions.

Again, thank you very much.
James

On Jun 20, 2013, at 7:02 AM, Kevin Karwaski wrote:

Ohai should be capturing this info from the on-instance EC2 metadata
store; from the output of your curl command above it looks like that
info is not present on your instance. You ran that curl command from
the EC2 instance, right?

Here's what ohai is grabbing:
ohai/lib/ohai/mixin/ec2_metadata.rb at main · chef/ohai · GitHub

This is what I see when I hit the metadata store on an EC2 instance:

curl http://169.254.169.254/latest/meta-data/

ami-id
ami-launch-index
ami-manifest-path
ancestor-ami-ids
block-device-mapping/
hostname
instance-action
instance-id
instance-type
kernel-id
local-hostname
local-ipv4
mac
metrics/
network/
placement/
profile
public-hostname
public-ipv4
public-keys/
reservation-id

curl http://169.254.169.254/latest/meta-data/public-hostname

BLAH-BLAH-BLAH.compute-1.amazonaws.com

Do you see anything cataloged in the EC2 metadata store?

On Wed, Jun 19, 2013 at 5:52 PM, James Andrews james@openbridge.com wrote:

I get the following after running that curl command.

curl http://169.254.169.254/latest/meta-data/public-hostname

<?xml version="1.0" encoding="iso-8859-1"?> 404 - Not Found

404 - Not Found

I ran "knife ec2 server list" I get

Instance ID Name Public IP Private IP Flavor Image SSH Key Security Groups State
i-22371517 i-22371517 54.218.44.44 172.31.41.147 m1.small ami-70f96e40 awsjames default running

So it sees the server sees the public IP address I just don't understand why it's not in the meta data.

James

--
Kevin Karwaski
Operations Engineer @ {Fiksu}

--
Kevin Karwaski
Operations Engineer @ {Fiksu}

15

FWIW, OHAI discovery works for me on regions other than us-east-1 (I have
some instances in us-west-1 and in eu-west-1). I don't have any VPCs there,
though. Maybe this works the way you describe only for VPC-enabled accounts?

-- M

On 20 June 2013 16:03, Kevin Karwaski kkarwaski@fiksu.com wrote:

Your welcome! Thanks for sharing some of your findings!

-K.

On Wed, Jun 19, 2013 at 6:36 PM, James Andrews james@openbridge.com
wrote:

Hi Kevin,

Thank you for all your help. I did some digging and I got it to work.
It was related to vpc, and I'll go ahead an explain it in detail so others
don't have to spend 3 days banging their heads against the wall.

Amazon sets "default" vpcs for all regions except "N. Virginia". I use
using "Oregon" (us-west2). I tried to delete the vpc but got some nasty
warning. It seems that you are unable to delete these VPCs because if you
do you won't be able to create EC2 instances in those regions anymore.

After changing my knife.rb file to use us-east-1c and generating a new
pem file. I was able to execute the knife file to remotely execute
chef-client. So the moral of the story is use "us-east" region until there
is a work around for these other regions.

Again, thank you very much.
James

On Jun 20, 2013, at 7:02 AM, Kevin Karwaski wrote:

Ohai should be capturing this info from the on-instance EC2 metadata
store; from the output of your curl command above it looks like that
info is not present on your instance. You ran that curl command from
the EC2 instance, right?

Here's what ohai is grabbing:

ohai/lib/ohai/mixin/ec2_metadata.rb at main · chef/ohai · GitHub

This is what I see when I hit the metadata store on an EC2 instance:

curl http://169.254.169.254/latest/meta-data/

ami-id
ami-launch-index
ami-manifest-path
ancestor-ami-ids
block-device-mapping/
hostname
instance-action
instance-id
instance-type
kernel-id
local-hostname
local-ipv4
mac
metrics/
network/
placement/
profile
public-hostname
public-ipv4
public-keys/
reservation-id

curl http://169.254.169.254/latest/meta-data/public-hostname

BLAH-BLAH-BLAH.compute-1.amazonaws.com

Do you see anything cataloged in the EC2 metadata store?

On Wed, Jun 19, 2013 at 5:52 PM, James Andrews james@openbridge.com
wrote:

I get the following after running that curl command.

curl http://169.254.169.254/latest/meta-data/public-hostname

<?xml version="1.0" encoding="iso-8859-1"?> 404 - Not Found

404 - Not Found

I ran "knife ec2 server list" I get

Instance ID Name Public IP Private IP Flavor Image
SSH Key Security Groups State
i-22371517 i-22371517 54.218.44.44 172.31.41.147 m1.small
ami-70f96e40 awsjames default running

So it sees the server sees the public IP address I just don't
understand why it's not in the meta data.

James

--
Kevin Karwaski
Operations Engineer @ {Fiksu}

--
Kevin Karwaski
Operations Engineer @ {Fiksu}

16

I don't ever remember asking for vpc enabled account, how do I turn that off? It made me real miserable for 3 days.

Thanks
James

On Jun 20, 2013, at 11:25 PM, Maciej Pasternacki wrote:

FWIW, OHAI discovery works for me on regions other than us-east-1 (I have some instances in us-west-1 and in eu-west-1). I don't have any VPCs there, though. Maybe this works the way you describe only for VPC-enabled accounts?

-- M

On 20 June 2013 16:03, Kevin Karwaski kkarwaski@fiksu.com wrote:
Your welcome! Thanks for sharing some of your findings!

-K.

On Wed, Jun 19, 2013 at 6:36 PM, James Andrews james@openbridge.com wrote:

Hi Kevin,

Thank you for all your help. I did some digging and I got it to work. It was related to vpc, and I'll go ahead an explain it in detail so others don't have to spend 3 days banging their heads against the wall.

Amazon sets "default" vpcs for all regions except "N. Virginia". I use using "Oregon" (us-west2). I tried to delete the vpc but got some nasty warning. It seems that you are unable to delete these VPCs because if you do you won't be able to create EC2 instances in those regions anymore.

After changing my knife.rb file to use us-east-1c and generating a new pem file. I was able to execute the knife file to remotely execute chef-client. So the moral of the story is use "us-east" region until there is a work around for these other regions.

Again, thank you very much.
James

On Jun 20, 2013, at 7:02 AM, Kevin Karwaski wrote:

Ohai should be capturing this info from the on-instance EC2 metadata
store; from the output of your curl command above it looks like that
info is not present on your instance. You ran that curl command from
the EC2 instance, right?

Here's what ohai is grabbing:
ohai/lib/ohai/mixin/ec2_metadata.rb at main · chef/ohai · GitHub

This is what I see when I hit the metadata store on an EC2 instance:

curl http://169.254.169.254/latest/meta-data/

ami-id
ami-launch-index
ami-manifest-path
ancestor-ami-ids
block-device-mapping/
hostname
instance-action
instance-id
instance-type
kernel-id
local-hostname
local-ipv4
mac
metrics/
network/
placement/
profile
public-hostname
public-ipv4
public-keys/
reservation-id

curl http://169.254.169.254/latest/meta-data/public-hostname

BLAH-BLAH-BLAH.compute-1.amazonaws.com

Do you see anything cataloged in the EC2 metadata store?

On Wed, Jun 19, 2013 at 5:52 PM, James Andrews james@openbridge.com wrote:

I get the following after running that curl command.

curl http://169.254.169.254/latest/meta-data/public-hostname

<?xml version="1.0" encoding="iso-8859-1"?> 404 - Not Found

404 - Not Found

I ran "knife ec2 server list" I get

Instance ID Name Public IP Private IP Flavor Image SSH Key Security Groups State
i-22371517 i-22371517 54.218.44.44 172.31.41.147 m1.small ami-70f96e40 awsjames default running

So it sees the server sees the public IP address I just don't understand why it's not in the meta data.

James

--
Kevin Karwaski
Operations Engineer @ {Fiksu}

--
Kevin Karwaski
Operations Engineer @ {Fiksu}

17

I'm fairly certain it's default behavior for new AWS accounts.

On Thu, Jun 20, 2013 at 10:32 AM, James Andrews james@openbridge.com wrote:

I don't ever remember asking for vpc enabled account, how do I turn that
off? It made me real miserable for 3 days.

Thanks
James

On Jun 20, 2013, at 11:25 PM, Maciej Pasternacki wrote:

FWIW, OHAI discovery works for me on regions other than us-east-1 (I have
some instances in us-west-1 and in eu-west-1). I don't have any VPCs there,
though. Maybe this works the way you describe only for VPC-enabled accounts?

-- M

On 20 June 2013 16:03, Kevin Karwaski kkarwaski@fiksu.com wrote:

Your welcome! Thanks for sharing some of your findings!

-K.

On Wed, Jun 19, 2013 at 6:36 PM, James Andrews james@openbridge.com
wrote:

Hi Kevin,

Thank you for all your help. I did some digging and I got it to work.
It was related to vpc, and I'll go ahead an explain it in detail so others
don't have to spend 3 days banging their heads against the wall.

Amazon sets "default" vpcs for all regions except "N. Virginia". I use
using "Oregon" (us-west2). I tried to delete the vpc but got some nasty
warning. It seems that you are unable to delete these VPCs because if you
do you won't be able to create EC2 instances in those regions anymore.

After changing my knife.rb file to use us-east-1c and generating a new
pem file. I was able to execute the knife file to remotely execute
chef-client. So the moral of the story is use "us-east" region until there
is a work around for these other regions.

Again, thank you very much.
James

On Jun 20, 2013, at 7:02 AM, Kevin Karwaski wrote:

Ohai should be capturing this info from the on-instance EC2 metadata
store; from the output of your curl command above it looks like that
info is not present on your instance. You ran that curl command from
the EC2 instance, right?

Here's what ohai is grabbing:

ohai/lib/ohai/mixin/ec2_metadata.rb at main · chef/ohai · GitHub

This is what I see when I hit the metadata store on an EC2 instance:

curl http://169.254.169.254/latest/meta-data/

ami-id
ami-launch-index
ami-manifest-path
ancestor-ami-ids
block-device-mapping/
hostname
instance-action
instance-id
instance-type
kernel-id
local-hostname
local-ipv4
mac
metrics/
network/
placement/
profile
public-hostname
public-ipv4
public-keys/
reservation-id

curl http://169.254.169.254/latest/meta-data/public-hostname

BLAH-BLAH-BLAH.compute-1.amazonaws.com

Do you see anything cataloged in the EC2 metadata store?

On Wed, Jun 19, 2013 at 5:52 PM, James Andrews james@openbridge.com
wrote:

I get the following after running that curl command.

curl http://169.254.169.254/latest/meta-data/public-hostname

<?xml version="1.0" encoding="iso-8859-1"?> 404 - Not Found

404 - Not Found

I ran "knife ec2 server list" I get

Instance ID Name Public IP Private IP Flavor Image
SSH Key Security Groups State
i-22371517 i-22371517 54.218.44.44 172.31.41.147 m1.small
ami-70f96e40 awsjames default running

So it sees the server sees the public IP address I just don't
understand why it's not in the meta data.

James

--
Kevin Karwaski
Operations Engineer @ {Fiksu}

--
Kevin Karwaski
Operations Engineer @ {Fiksu}

--
Kevin Karwaski
Operations Engineer @ {Fiksu}

18

You're probably right. I'll have to figure out a way around that then if I want to use other regions.

Thanks.
James

On Jun 20, 2013, at 11:33 PM, Kevin Karwaski wrote:

I'm fairly certain it's default behavior for new AWS accounts.

On Thu, Jun 20, 2013 at 10:32 AM, James Andrews james@openbridge.com wrote:

I don't ever remember asking for vpc enabled account, how do I turn that
off? It made me real miserable for 3 days.

Thanks
James

On Jun 20, 2013, at 11:25 PM, Maciej Pasternacki wrote:

FWIW, OHAI discovery works for me on regions other than us-east-1 (I have
some instances in us-west-1 and in eu-west-1). I don't have any VPCs there,
though. Maybe this works the way you describe only for VPC-enabled accounts?

-- M

On 20 June 2013 16:03, Kevin Karwaski kkarwaski@fiksu.com wrote:

Your welcome! Thanks for sharing some of your findings!

-K.

On Wed, Jun 19, 2013 at 6:36 PM, James Andrews james@openbridge.com
wrote:

Hi Kevin,

Thank you for all your help. I did some digging and I got it to work.
It was related to vpc, and I'll go ahead an explain it in detail so others
don't have to spend 3 days banging their heads against the wall.

Amazon sets "default" vpcs for all regions except "N. Virginia". I use
using "Oregon" (us-west2). I tried to delete the vpc but got some nasty
warning. It seems that you are unable to delete these VPCs because if you
do you won't be able to create EC2 instances in those regions anymore.

After changing my knife.rb file to use us-east-1c and generating a new
pem file. I was able to execute the knife file to remotely execute
chef-client. So the moral of the story is use "us-east" region until there
is a work around for these other regions.

Again, thank you very much.
James

On Jun 20, 2013, at 7:02 AM, Kevin Karwaski wrote:

Ohai should be capturing this info from the on-instance EC2 metadata
store; from the output of your curl command above it looks like that
info is not present on your instance. You ran that curl command from
the EC2 instance, right?

Here's what ohai is grabbing:

ohai/lib/ohai/mixin/ec2_metadata.rb at main · chef/ohai · GitHub

This is what I see when I hit the metadata store on an EC2 instance:

curl http://169.254.169.254/latest/meta-data/

ami-id
ami-launch-index
ami-manifest-path
ancestor-ami-ids
block-device-mapping/
hostname
instance-action
instance-id
instance-type
kernel-id
local-hostname
local-ipv4
mac
metrics/
network/
placement/
profile
public-hostname
public-ipv4
public-keys/
reservation-id

curl http://169.254.169.254/latest/meta-data/public-hostname

BLAH-BLAH-BLAH.compute-1.amazonaws.com

Do you see anything cataloged in the EC2 metadata store?

On Wed, Jun 19, 2013 at 5:52 PM, James Andrews james@openbridge.com
wrote:

I get the following after running that curl command.

curl http://169.254.169.254/latest/meta-data/public-hostname

<?xml version="1.0" encoding="iso-8859-1"?> 404 - Not Found

404 - Not Found

I ran "knife ec2 server list" I get

Instance ID Name Public IP Private IP Flavor Image
SSH Key Security Groups State
i-22371517 i-22371517 54.218.44.44 172.31.41.147 m1.small
ami-70f96e40 awsjames default running

So it sees the server sees the public IP address I just don't
understand why it's not in the meta data.

James

--
Kevin Karwaski
Operations Engineer @ {Fiksu}

--
Kevin Karwaski
Operations Engineer @ {Fiksu}

--
Kevin Karwaski
Operations Engineer @ {Fiksu}

19

Here's a blog post from Amazon describing the new VPC behavior and
probably why you are seeing different behavior with different regions.
I haven't dealt with it yet so, I'm not sure how to address the
changes when using knife.

http://aws.typepad.com/aws/2013/03/amazon-ec2-update-virtual-private-clouds-for-everyone.html

"The default VPC features are available to new AWS customers and
existing customers launching instances in a Region for the first time.
If you’ve previously launched an EC2 instance in a Region or
provisioned ELB, RDS, or ElastiCache in a Region, we won’t create a
default VPC for you in that Region.

If you are an existing AWS customer and you want to start gaining
experience with this new behavior, you have two options. You can
create a new AWS account or you can pick a Region that you haven't
used (as defined above). You can see the set of available platforms in
the AWS Management Console (this information is also available through
the EC2 APIs and from the command line). Be sure to check the
Supported Platforms and Default VPC values for your account to see how
your account is configured in a specific Region."

~ Alex

On Thu, Jun 20, 2013 at 9:37 AM, James Andrews james@openbridge.com wrote:

You're probably right. I'll have to figure out a way around that then if I want to use other regions.

Thanks.
James

On Jun 20, 2013, at 11:33 PM, Kevin Karwaski wrote:

I'm fairly certain it's default behavior for new AWS accounts.

On Thu, Jun 20, 2013 at 10:32 AM, James Andrews james@openbridge.com wrote:

I don't ever remember asking for vpc enabled account, how do I turn that
off? It made me real miserable for 3 days.

Thanks
James

On Jun 20, 2013, at 11:25 PM, Maciej Pasternacki wrote:

FWIW, OHAI discovery works for me on regions other than us-east-1 (I have
some instances in us-west-1 and in eu-west-1). I don't have any VPCs there,
though. Maybe this works the way you describe only for VPC-enabled accounts?

-- M

On 20 June 2013 16:03, Kevin Karwaski kkarwaski@fiksu.com wrote:

Your welcome! Thanks for sharing some of your findings!

-K.

On Wed, Jun 19, 2013 at 6:36 PM, James Andrews james@openbridge.com
wrote:

Hi Kevin,

Thank you for all your help. I did some digging and I got it to work.
It was related to vpc, and I'll go ahead an explain it in detail so others
don't have to spend 3 days banging their heads against the wall.

Amazon sets "default" vpcs for all regions except "N. Virginia". I use
using "Oregon" (us-west2). I tried to delete the vpc but got some nasty
warning. It seems that you are unable to delete these VPCs because if you
do you won't be able to create EC2 instances in those regions anymore.

After changing my knife.rb file to use us-east-1c and generating a new
pem file. I was able to execute the knife file to remotely execute
chef-client. So the moral of the story is use "us-east" region until there
is a work around for these other regions.

Again, thank you very much.
James

On Jun 20, 2013, at 7:02 AM, Kevin Karwaski wrote:

Ohai should be capturing this info from the on-instance EC2 metadata
store; from the output of your curl command above it looks like that
info is not present on your instance. You ran that curl command from
the EC2 instance, right?

Here's what ohai is grabbing:

ohai/lib/ohai/mixin/ec2_metadata.rb at main · chef/ohai · GitHub

This is what I see when I hit the metadata store on an EC2 instance:

curl http://169.254.169.254/latest/meta-data/

ami-id
ami-launch-index
ami-manifest-path
ancestor-ami-ids
block-device-mapping/
hostname
instance-action
instance-id
instance-type
kernel-id
local-hostname
local-ipv4
mac
metrics/
network/
placement/
profile
public-hostname
public-ipv4
public-keys/
reservation-id

curl http://169.254.169.254/latest/meta-data/public-hostname

BLAH-BLAH-BLAH.compute-1.amazonaws.com

Do you see anything cataloged in the EC2 metadata store?

On Wed, Jun 19, 2013 at 5:52 PM, James Andrews james@openbridge.com
wrote:

I get the following after running that curl command.

curl http://169.254.169.254/latest/meta-data/public-hostname

<?xml version="1.0" encoding="iso-8859-1"?> 404 - Not Found

404 - Not Found

I ran "knife ec2 server list" I get

Instance ID Name Public IP Private IP Flavor Image
SSH Key Security Groups State
i-22371517 i-22371517 54.218.44.44 172.31.41.147 m1.small
ami-70f96e40 awsjames default running

So it sees the server sees the public IP address I just don't
understand why it's not in the meta data.

James

--
Kevin Karwaski
Operations Engineer @ {Fiksu}

--
Kevin Karwaski
Operations Engineer @ {Fiksu}

--
Kevin Karwaski
Operations Engineer @ {Fiksu}

20

Thank you Alex, some good information there. I am not sure how to handle the changes, but for the moment as long as I know what the problem is I can avoid it during the learning process. and revisit it when we are ready to start spinning our services.

Thanks
James

On Jun 20, 2013, at 11:45 PM, Alex Koch wrote:

Here's a blog post from Amazon describing the new VPC behavior and
probably why you are seeing different behavior with different regions.
I haven't dealt with it yet so, I'm not sure how to address the
changes when using knife.

http://aws.typepad.com/aws/2013/03/amazon-ec2-update-virtual-private-clouds-for-everyone.html

"The default VPC features are available to new AWS customers and
existing customers launching instances in a Region for the first time.
If you’ve previously launched an EC2 instance in a Region or
provisioned ELB, RDS, or ElastiCache in a Region, we won’t create a
default VPC for you in that Region.

If you are an existing AWS customer and you want to start gaining
experience with this new behavior, you have two options. You can
create a new AWS account or you can pick a Region that you haven't
used (as defined above). You can see the set of available platforms in
the AWS Management Console (this information is also available through
the EC2 APIs and from the command line). Be sure to check the
Supported Platforms and Default VPC values for your account to see how
your account is configured in a specific Region."

~ Alex

On Thu, Jun 20, 2013 at 9:37 AM, James Andrews james@openbridge.com wrote:

You're probably right. I'll have to figure out a way around that then if I want to use other regions.

Thanks.
James

On Jun 20, 2013, at 11:33 PM, Kevin Karwaski wrote:

I'm fairly certain it's default behavior for new AWS accounts.

On Thu, Jun 20, 2013 at 10:32 AM, James Andrews james@openbridge.com wrote:

I don't ever remember asking for vpc enabled account, how do I turn that
off? It made me real miserable for 3 days.

Thanks
James

On Jun 20, 2013, at 11:25 PM, Maciej Pasternacki wrote:

FWIW, OHAI discovery works for me on regions other than us-east-1 (I have
some instances in us-west-1 and in eu-west-1). I don't have any VPCs there,
though. Maybe this works the way you describe only for VPC-enabled accounts?

-- M

On 20 June 2013 16:03, Kevin Karwaski kkarwaski@fiksu.com wrote:

Your welcome! Thanks for sharing some of your findings!

-K.

On Wed, Jun 19, 2013 at 6:36 PM, James Andrews james@openbridge.com
wrote:

Hi Kevin,

Thank you for all your help. I did some digging and I got it to work.
It was related to vpc, and I'll go ahead an explain it in detail so others
don't have to spend 3 days banging their heads against the wall.

Amazon sets "default" vpcs for all regions except "N. Virginia". I use
using "Oregon" (us-west2). I tried to delete the vpc but got some nasty
warning. It seems that you are unable to delete these VPCs because if you
do you won't be able to create EC2 instances in those regions anymore.

After changing my knife.rb file to use us-east-1c and generating a new
pem file. I was able to execute the knife file to remotely execute
chef-client. So the moral of the story is use "us-east" region until there
is a work around for these other regions.

Again, thank you very much.
James

On Jun 20, 2013, at 7:02 AM, Kevin Karwaski wrote:

Ohai should be capturing this info from the on-instance EC2 metadata
store; from the output of your curl command above it looks like that
info is not present on your instance. You ran that curl command from
the EC2 instance, right?

Here's what ohai is grabbing:

ohai/lib/ohai/mixin/ec2_metadata.rb at main · chef/ohai · GitHub

This is what I see when I hit the metadata store on an EC2 instance:

curl http://169.254.169.254/latest/meta-data/

ami-id
ami-launch-index
ami-manifest-path
ancestor-ami-ids
block-device-mapping/
hostname
instance-action
instance-id
instance-type
kernel-id
local-hostname
local-ipv4
mac
metrics/
network/
placement/
profile
public-hostname
public-ipv4
public-keys/
reservation-id

curl http://169.254.169.254/latest/meta-data/public-hostname

BLAH-BLAH-BLAH.compute-1.amazonaws.com

Do you see anything cataloged in the EC2 metadata store?

On Wed, Jun 19, 2013 at 5:52 PM, James Andrews james@openbridge.com
wrote:

I get the following after running that curl command.

curl http://169.254.169.254/latest/meta-data/public-hostname

<?xml version="1.0" encoding="iso-8859-1"?> 404 - Not Found

404 - Not Found

I ran "knife ec2 server list" I get

Instance ID Name Public IP Private IP Flavor Image
SSH Key Security Groups State
i-22371517 i-22371517 54.218.44.44 172.31.41.147 m1.small
ami-70f96e40 awsjames default running

So it sees the server sees the public IP address I just don't
understand why it's not in the meta data.

James

--
Kevin Karwaski
Operations Engineer @ {Fiksu}

--
Kevin Karwaski
Operations Engineer @ {Fiksu}

--
Kevin Karwaski
Operations Engineer @ {Fiksu}