Knife + ec2

Chef Infra (archive)
1

Hi,

I’m trying to bootstrap nodes in EC2 using knife as follows:

knife ec2 server create --flavor m1.large -I ami-1b814f72 -G DevOps -S

DevOps -N foobar -x ec2-user -i /tmp/DevOps.pem -V -V

but it hangs with the following output:

DEBUG: Using configuration from /root/.chef/knife.rb
Instance ID: i-638eea06
Flavor: m1.large
Image: ami-1b814f72
Region: us-east-1
Availability Zone: us-east-1b
Security Groups: DevOps
SSH Key: DevOps

Waiting for server…
Public DNS Name: ec2-50-19-18-3.compute-1.amazonaws.com
Public IP Address: 50.19.18.3
Private DNS Name: ip-10-87-1-167.ec2.internal
Private IP Address: 10.87.1.167

Waiting for sshd.

If, however, I manually log in to the instance everything seems available:

$ ssh -i DevOps.pem ec2-user@ec2-50-19-18-3.compute-1.amazonaws.com
The authenticity of host
’ec2-50-19-18-3.compute-1.amazonaws.com(50.19.18.3)’ can’t be
established.
RSA key fingerprint is 0e:da:5e:4b:8d:80:7f:ba:63:89:25:ad:dd:b4:ba:a0.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘ec2-50-19-18-3.compute-1.amazonaws.com,50.19.18.3’
(RSA) to the list of known hosts.

   __|  __|_  )
   _|  (     /   Amazon Linux AMI
  ___|\___|___|

See /usr/share/doc/system-release/ for latest release notes.
There are 23 security update(s) out of 32 total update(s) available
[ec2-user@ip-10-87-1-167 ~]$

Is there any way to get more debugging information from knife? Does anyone
know what could be going wrong?

Thanks,
Mark

2

I think this can also be caused by the sshd daemon taking longer to start
than the timeout allows. I see it in roughly 1/20 ec2 nodes. You could
bootstrap the instance, but I just delete them and re-create.
On Feb 9, 2012 5:51 AM, "mark bradley" gopearls42@gmail.com wrote:

Hi,

I'm trying to bootstrap nodes in EC2 using knife as follows:

knife ec2 server create --flavor m1.large -I ami-1b814f72 -G DevOps -S

DevOps -N foobar -x ec2-user -i /tmp/DevOps.pem -V -V

but it hangs with the following output:

DEBUG: Using configuration from /root/.chef/knife.rb
Instance ID: i-638eea06
Flavor: m1.large
Image: ami-1b814f72
Region: us-east-1
Availability Zone: us-east-1b
Security Groups: DevOps
SSH Key: DevOps

Waiting for server.......................
Public DNS Name: ec2-50-19-18-3.compute-1.amazonaws.com
Public IP Address: 50.19.18.3
Private DNS Name: ip-10-87-1-167.ec2.internal
Private IP Address: 10.87.1.167

Waiting for sshd.

If, however, I manually log in to the instance everything seems available:

$ ssh -i DevOps.pem ec2-user@ec2-50-19-18-3.compute-1.amazonaws.com
The authenticity of host 'ec2-50-19-18-3.compute-1.amazonaws.com(50.19.18.3)' can't be established.
RSA key fingerprint is 0e:da:5e:4b:8d:80:7f:ba:63:89:25:ad:dd:b4:ba:a0.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ec2-50-19-18-3.compute-1.amazonaws.com,50.19.18.3'
(RSA) to the list of known hosts.

   __|  __|_  )
   _|  (     /   Amazon Linux AMI
  ___|\___|___|

See /usr/share/doc/system-release/ for latest release notes.
There are 23 security update(s) out of 32 total update(s) available
[ec2-user@ip-10-87-1-167 ~]$

Is there any way to get more debugging information from knife? Does anyone
know what could be going wrong?

Thanks,
Mark

3

How long are you waiting for sshd to come up there? It can take a minute or
more.

Kevin Harvey
Web Developer
kcharvey@gmail.com

On Thu, Feb 9, 2012 at 8:50 AM, mark bradley gopearls42@gmail.com wrote:

Hi,

I'm trying to bootstrap nodes in EC2 using knife as follows:

knife ec2 server create --flavor m1.large -I ami-1b814f72 -G DevOps -S

DevOps -N foobar -x ec2-user -i /tmp/DevOps.pem -V -V

but it hangs with the following output:

DEBUG: Using configuration from /root/.chef/knife.rb
Instance ID: i-638eea06
Flavor: m1.large
Image: ami-1b814f72
Region: us-east-1
Availability Zone: us-east-1b
Security Groups: DevOps
SSH Key: DevOps

Waiting for server.......................
Public DNS Name: ec2-50-19-18-3.compute-1.amazonaws.com
Public IP Address: 50.19.18.3
Private DNS Name: ip-10-87-1-167.ec2.internal
Private IP Address: 10.87.1.167

Waiting for sshd.

If, however, I manually log in to the instance everything seems available:

$ ssh -i DevOps.pem ec2-user@ec2-50-19-18-3.compute-1.amazonaws.com
The authenticity of host 'ec2-50-19-18-3.compute-1.amazonaws.com(50.19.18.3)' can't be established.
RSA key fingerprint is 0e:da:5e:4b:8d:80:7f:ba:63:89:25:ad:dd:b4:ba:a0.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ec2-50-19-18-3.compute-1.amazonaws.com,50.19.18.3'
(RSA) to the list of known hosts.

   __|  __|_  )
   _|  (     /   Amazon Linux AMI
  ___|\___|___|

See /usr/share/doc/system-release/ for latest release notes.
There are 23 security update(s) out of 32 total update(s) available
[ec2-user@ip-10-87-1-167 ~]$

Is there any way to get more debugging information from knife? Does anyone
know what could be going wrong?

Thanks,
Mark

4

Total guess: knife blocked on unknown host SSL key?
On Feb 9, 2012 5:54 AM, "Kevin Harvey" kcharvey@gmail.com wrote:

How long are you waiting for sshd to come up there? It can take a minute
or more.

Kevin Harvey
Web Developer
kcharvey@gmail.com

On Thu, Feb 9, 2012 at 8:50 AM, mark bradley gopearls42@gmail.com wrote:

Hi,

I'm trying to bootstrap nodes in EC2 using knife as follows:

knife ec2 server create --flavor m1.large -I ami-1b814f72 -G DevOps -S

DevOps -N foobar -x ec2-user -i /tmp/DevOps.pem -V -V

but it hangs with the following output:

DEBUG: Using configuration from /root/.chef/knife.rb
Instance ID: i-638eea06
Flavor: m1.large
Image: ami-1b814f72
Region: us-east-1
Availability Zone: us-east-1b
Security Groups: DevOps
SSH Key: DevOps

Waiting for server.......................
Public DNS Name: ec2-50-19-18-3.compute-1.amazonaws.com
Public IP Address: 50.19.18.3
Private DNS Name: ip-10-87-1-167.ec2.internal
Private IP Address: 10.87.1.167

Waiting for sshd.

If, however, I manually log in to the instance everything seems available:

$ ssh -i DevOps.pem ec2-user@ec2-50-19-18-3.compute-1.amazonaws.com
The authenticity of host 'ec2-50-19-18-3.compute-1.amazonaws.com(50.19.18.3)' can't be established.
RSA key fingerprint is 0e:da:5e:4b:8d:80:7f:ba:63:89:25:ad:dd:b4:ba:a0.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ec2-50-19-18-3.compute-1.amazonaws.com,50.19.18.3'
(RSA) to the list of known hosts.

   __|  __|_  )
   _|  (     /   Amazon Linux AMI
  ___|\___|___|

See /usr/share/doc/system-release/ for latest release notes.
There are 23 security update(s) out of 32 total update(s) available
[ec2-user@ip-10-87-1-167 ~]$

Is there any way to get more debugging information from knife? Does
anyone know what could be going wrong?

Thanks,
Mark

5

I suggest adding an entry to your ~/.ssh/config file. Something similar to
the following.

Host compute.amazonaws.com
ForwardAgent yes
User csullivan-emr
CheckHostIP no
StrictHostKeyChecking no
UserKnownHostsFile /dev/null
KeepAlive yes
IdentityFile=~/.ssh/id_rsa

Also, look into this project to ease your SSH key usage.

http://www.funtoo.org/wiki/Keychain

On Thu, Feb 9, 2012 at 7:57 AM, Bryan Stenson bryan.stenson@gmail.comwrote:

Total guess: knife blocked on unknown host SSL key?
On Feb 9, 2012 5:54 AM, "Kevin Harvey" kcharvey@gmail.com wrote:

How long are you waiting for sshd to come up there? It can take a minute
or more.

Kevin Harvey
Web Developer
kcharvey@gmail.com

On Thu, Feb 9, 2012 at 8:50 AM, mark bradley gopearls42@gmail.comwrote:

Hi,

I'm trying to bootstrap nodes in EC2 using knife as follows:

knife ec2 server create --flavor m1.large -I ami-1b814f72 -G DevOps -S

DevOps -N foobar -x ec2-user -i /tmp/DevOps.pem -V -V

but it hangs with the following output:

DEBUG: Using configuration from /root/.chef/knife.rb
Instance ID: i-638eea06
Flavor: m1.large
Image: ami-1b814f72
Region: us-east-1
Availability Zone: us-east-1b
Security Groups: DevOps
SSH Key: DevOps

Waiting for server.......................
Public DNS Name: ec2-50-19-18-3.compute-1.amazonaws.com
Public IP Address: 50.19.18.3
Private DNS Name: ip-10-87-1-167.ec2.internal
Private IP Address: 10.87.1.167

Waiting for sshd.

If, however, I manually log in to the instance everything seems
available:

$ ssh -i DevOps.pem ec2-user@ec2-50-19-18-3.compute-1.amazonaws.com
The authenticity of host 'ec2-50-19-18-3.compute-1.amazonaws.com(50.19.18.3)' can't be established.
RSA key fingerprint is 0e:da:5e:4b:8d:80:7f:ba:63:89:25:ad:dd:b4:ba:a0.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ec2-50-19-18-3.compute-1.amazonaws.com,50.19.18.3'
(RSA) to the list of known hosts.

   __|  __|_  )
   _|  (     /   Amazon Linux AMI
  ___|\___|___|

See /usr/share/doc/system-release/ for latest release notes.
There are 23 security update(s) out of 32 total update(s) available
[ec2-user@ip-10-87-1-167 ~]$

Is there any way to get more debugging information from knife? Does
anyone know what could be going wrong?

Thanks,
Mark

--
Charles Sullivan
charlie.sullivan@gmail.com

6

Well, my accident I left it waiting overnight :frowning:

Mark

On Thu, Feb 9, 2012 at 8:53 AM, Kevin Harvey kcharvey@gmail.com wrote:

How long are you waiting for sshd to come up there? It can take a minute
or more.

Kevin Harvey
Web Developer
kcharvey@gmail.com

On Thu, Feb 9, 2012 at 8:50 AM, mark bradley gopearls42@gmail.com wrote:

Hi,

I'm trying to bootstrap nodes in EC2 using knife as follows:

knife ec2 server create --flavor m1.large -I ami-1b814f72 -G DevOps -S

DevOps -N foobar -x ec2-user -i /tmp/DevOps.pem -V -V

but it hangs with the following output:

DEBUG: Using configuration from /root/.chef/knife.rb
Instance ID: i-638eea06
Flavor: m1.large
Image: ami-1b814f72
Region: us-east-1
Availability Zone: us-east-1b
Security Groups: DevOps
SSH Key: DevOps

Waiting for server.......................
Public DNS Name: ec2-50-19-18-3.compute-1.amazonaws.com
Public IP Address: 50.19.18.3
Private DNS Name: ip-10-87-1-167.ec2.internal
Private IP Address: 10.87.1.167

Waiting for sshd.

If, however, I manually log in to the instance everything seems available:

$ ssh -i DevOps.pem ec2-user@ec2-50-19-18-3.compute-1.amazonaws.com
The authenticity of host 'ec2-50-19-18-3.compute-1.amazonaws.com(50.19.18.3)' can't be established.
RSA key fingerprint is 0e:da:5e:4b:8d:80:7f:ba:63:89:25:ad:dd:b4:ba:a0.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ec2-50-19-18-3.compute-1.amazonaws.com,50.19.18.3'
(RSA) to the list of known hosts.

   __|  __|_  )
   _|  (     /   Amazon Linux AMI
  ___|\___|___|

See /usr/share/doc/system-release/ for latest release notes.
There are 23 security update(s) out of 32 total update(s) available
[ec2-user@ip-10-87-1-167 ~]$

Is there any way to get more debugging information from knife? Does
anyone know what could be going wrong?

Thanks,
Mark

7

Doublecheck that your DevOps security group has your IP authorized for SSH.

I ran "knife ec2 server create -f m1.large -I ami-1b814f72 -G default -S mykey -N foobar -x ec2-user -V -V" and it hung on SSH, then I remembered that my default group only allows inter-ec2 communication.

This worked for me "knife ec2 server create -f m1.large -I ami-1b814f72 -G default,management -S mykey -N foobar -x ec2-user -V -V"

I use ssh-add to store my ssh key for ease of use.

On Feb 9, 2012, at 8:39 AM, mark bradley wrote:

Well, my accident I left it waiting overnight :frowning:

Mark

On Thu, Feb 9, 2012 at 8:53 AM, Kevin Harvey kcharvey@gmail.com wrote:
How long are you waiting for sshd to come up there? It can take a minute or more.

Kevin Harvey
Web Developer
kcharvey@gmail.com

On Thu, Feb 9, 2012 at 8:50 AM, mark bradley gopearls42@gmail.com wrote:
Hi,

I'm trying to bootstrap nodes in EC2 using knife as follows:

knife ec2 server create --flavor m1.large -I ami-1b814f72 -G DevOps -S DevOps -N foobar -x ec2-user -i /tmp/DevOps.pem -V -V

but it hangs with the following output:

DEBUG: Using configuration from /root/.chef/knife.rb
Instance ID: i-638eea06
Flavor: m1.large
Image: ami-1b814f72
Region: us-east-1
Availability Zone: us-east-1b
Security Groups: DevOps
SSH Key: DevOps

Waiting for server.......................
Public DNS Name: ec2-50-19-18-3.compute-1.amazonaws.com
Public IP Address: 50.19.18.3
Private DNS Name: ip-10-87-1-167.ec2.internal
Private IP Address: 10.87.1.167

Waiting for sshd.

If, however, I manually log in to the instance everything seems available:

$ ssh -i DevOps.pem ec2-user@ec2-50-19-18-3.compute-1.amazonaws.com
The authenticity of host 'ec2-50-19-18-3.compute-1.amazonaws.com (50.19.18.3)' can't be established.
RSA key fingerprint is 0e:da:5e:4b:8d:80:7f:ba:63:89:25:ad:dd:b4:ba:a0.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ec2-50-19-18-3.compute-1.amazonaws.com,50.19.18.3' (RSA) to the list of known hosts.

   __|  __|_  )
   _|  (     /   Amazon Linux AMI
  ___|\___|___|

See /usr/share/doc/system-release/ for latest release notes.
There are 23 security update(s) out of 32 total update(s) available
[ec2-user@ip-10-87-1-167 ~]$

Is there any way to get more debugging information from knife? Does anyone know what could be going wrong?

Thanks,
Mark