Knife ec2 failing on sshd


#1

Hello everyone…

I’m attempting my first knife ec2 create commands for building instances
inside a VPC.

The problem I’m facing is that “Waiting for sshd access to become
available…” Never completes. What’s weird (and flies in the face of
what I thought was going on) is that while I’m watching little dots appear,
in another terminal window on the same workstation, using the same identity
file, user, and I think hostname (EIP) that the knife command is using, I
CAN ssh into the instance.
Any ideas what might cause this?

  • Michael

#2

Hi Michael,

I generally use Cloudformation for all my VPC instances, so have not
attempted a direct bootstrap. Have you confirmed that the inbound and
outbound security group and ACL rules allow traffic both directions on 22,
80, 443, and the ephemeral ports?


Michael F. Weinberg | Director of Operations
http://heavywaterops.com | @heavywaterops

On Fri, Mar 6, 2015 at 2:16 PM, Michael Lindsay mlindsay@metrodigi.com
wrote:

Hello everyone…

I’m attempting my first knife ec2 create commands for building instances
inside a VPC.

The problem I’m facing is that “Waiting for sshd access to become
available…” Never completes. What’s weird (and flies in the face of
what I thought was going on) is that while I’m watching little dots appear,
in another terminal window on the same workstation, using the same identity
file, user, and I think hostname (EIP) that the knife command is using, I
CAN ssh into the instance.
Any ideas what might cause this?

  • Michael

#3

Hi again Michael :slight_smile: I’m new to VPC, but I feel like I’ve poured over all
the settings there. Would the behavior be different with knife than ssh on
the same workstation in this regard?

On Fri, Mar 6, 2015 at 2:19 PM, Michael Weinberg michael@hw-ops.com wrote:

Hi Michael,

I generally use Cloudformation for all my VPC instances, so have not
attempted a direct bootstrap. Have you confirmed that the inbound and
outbound security group and ACL rules allow traffic both directions on 22,
80, 443, and the ephemeral ports?


Michael F. Weinberg | Director of Operations
http://heavywaterops.com | @heavywaterops

On Fri, Mar 6, 2015 at 2:16 PM, Michael Lindsay mlindsay@metrodigi.com
wrote:

Hello everyone…

I’m attempting my first knife ec2 create commands for building instances
inside a VPC.

The problem I’m facing is that “Waiting for sshd access to become
available…” Never completes. What’s weird (and flies in the face of
what I thought was going on) is that while I’m watching little dots appear,
in another terminal window on the same workstation, using the same identity
file, user, and I think hostname (EIP) that the knife command is using, I
CAN ssh into the instance.
Any ideas what might cause this?

  • Michael

#4

No, it shouldn’t be. But I don’t know what is actually happening during
’waiting for sshd,’ so on the off chance there’s some additional
communication there I thought I’d suggest it.


Michael F. Weinberg | Director of Operations
http://heavywaterops.com | @heavywaterops

On Fri, Mar 6, 2015 at 2:29 PM, Michael Lindsay mlindsay@metrodigi.com
wrote:

Hi again Michael :slight_smile: I’m new to VPC, but I feel like I’ve poured over all
the settings there. Would the behavior be different with knife than ssh on
the same workstation in this regard?

On Fri, Mar 6, 2015 at 2:19 PM, Michael Weinberg michael@hw-ops.com
wrote:

Hi Michael,

I generally use Cloudformation for all my VPC instances, so have not
attempted a direct bootstrap. Have you confirmed that the inbound and
outbound security group and ACL rules allow traffic both directions on 22,
80, 443, and the ephemeral ports?


Michael F. Weinberg | Director of Operations
http://heavywaterops.com | @heavywaterops

On Fri, Mar 6, 2015 at 2:16 PM, Michael Lindsay mlindsay@metrodigi.com
wrote:

Hello everyone…

I’m attempting my first knife ec2 create commands for building instances
inside a VPC.

The problem I’m facing is that “Waiting for sshd access to become
available…” Never completes. What’s weird (and flies in the face of
what I thought was going on) is that while I’m watching little dots appear,
in another terminal window on the same workstation, using the same identity
file, user, and I think hostname (EIP) that the knife command is using, I
CAN ssh into the instance.
Any ideas what might cause this?

  • Michael

#5

It’s not doing anything besides trying to open a TCP connection to the SSH port:


and

  • Julian

On Fri, Mar 6, 2015 at 4:32 PM, Michael Weinberg michael@hw-ops.com wrote:

No, it shouldn’t be. But I don’t know what is actually happening during
’waiting for sshd,’ so on the off chance there’s some additional
communication there I thought I’d suggest it.


Michael F. Weinberg | Director of Operations
http://heavywaterops.com | @heavywaterops

On Fri, Mar 6, 2015 at 2:29 PM, Michael Lindsay mlindsay@metrodigi.com
wrote:

Hi again Michael :slight_smile: I’m new to VPC, but I feel like I’ve poured over all
the settings there. Would the behavior be different with knife than ssh on
the same workstation in this regard?

On Fri, Mar 6, 2015 at 2:19 PM, Michael Weinberg michael@hw-ops.com
wrote:

Hi Michael,

I generally use Cloudformation for all my VPC instances, so have not
attempted a direct bootstrap. Have you confirmed that the inbound and
outbound security group and ACL rules allow traffic both directions on 22,
80, 443, and the ephemeral ports?


Michael F. Weinberg | Director of Operations
http://heavywaterops.com | @heavywaterops

On Fri, Mar 6, 2015 at 2:16 PM, Michael Lindsay mlindsay@metrodigi.com
wrote:

Hello everyone…

I’m attempting my first knife ec2 create commands for building instances
inside a VPC.

The problem I’m facing is that “Waiting for sshd access to become
available…” Never completes. What’s weird (and flies in the face of what
I thought was going on) is that while I’m watching little dots appear, in
another terminal window on the same workstation, using the same identity
file, user, and I think hostname (EIP) that the knife command is using, I
CAN ssh into the instance.

Any ideas what might cause this?

  • Michael


[ Julian C. Dunn jdunn@aquezada.com * Sorry, I’m ]
[ WWW: http://www.aquezada.com/staff/julian * only Web 1.0 ]
[ gopher://sdf.org/1/users/keymaker/ * compliant! ]
[ PGP: 91B3 7A9D 683C 7C16 715F 442C 6065 D533 FDC2 05B9 ]


#6

Hello, are you trying to configure a node on a private subnet? If so are
you supplying the --ssh-gateway option/configure ssh config proxies?

On Fri, Mar 6, 2015 at 5:32 PM, Michael Weinberg michael@hw-ops.com wrote:

No, it shouldn’t be. But I don’t know what is actually happening during
’waiting for sshd,’ so on the off chance there’s some additional
communication there I thought I’d suggest it.


Michael F. Weinberg | Director of Operations
http://heavywaterops.com | @heavywaterops

On Fri, Mar 6, 2015 at 2:29 PM, Michael Lindsay mlindsay@metrodigi.com
wrote:

Hi again Michael :slight_smile: I’m new to VPC, but I feel like I’ve poured over all
the settings there. Would the behavior be different with knife than ssh on
the same workstation in this regard?

On Fri, Mar 6, 2015 at 2:19 PM, Michael Weinberg michael@hw-ops.com
wrote:

Hi Michael,

I generally use Cloudformation for all my VPC instances, so have not
attempted a direct bootstrap. Have you confirmed that the inbound and
outbound security group and ACL rules allow traffic both directions on 22,
80, 443, and the ephemeral ports?


Michael F. Weinberg | Director of Operations
http://heavywaterops.com | @heavywaterops

On Fri, Mar 6, 2015 at 2:16 PM, Michael Lindsay mlindsay@metrodigi.com
wrote:

Hello everyone…

I’m attempting my first knife ec2 create commands for building
instances inside a VPC.

The problem I’m facing is that “Waiting for sshd access to become
available…” Never completes. What’s weird (and flies in the face of
what I thought was going on) is that while I’m watching little dots appear,
in another terminal window on the same workstation, using the same identity
file, user, and I think hostname (EIP) that the knife command is using, I
CAN ssh into the instance.
Any ideas what might cause this?

  • Michael

#7

@Julian: hadn’t thought to look at the source code, that’s good info (and I
can see why the dots roll out more slowly :wink: ). I’m not connecting to an
ubuntu instance so looks like it wouldn’t be TCP connection in the second
git file but instead something from here?

@Darren: It’s a public subnet, so no proxy required

On Fri, Mar 6, 2015 at 2:48 PM, Darren Dyke dwdyke@gmail.com wrote:

Hello, are you trying to configure a node on a private subnet? If so are
you supplying the --ssh-gateway option/configure ssh config proxies?

On Fri, Mar 6, 2015 at 5:32 PM, Michael Weinberg michael@hw-ops.com
wrote:

No, it shouldn’t be. But I don’t know what is actually happening during
’waiting for sshd,’ so on the off chance there’s some additional
communication there I thought I’d suggest it.


Michael F. Weinberg | Director of Operations
http://heavywaterops.com | @heavywaterops

On Fri, Mar 6, 2015 at 2:29 PM, Michael Lindsay mlindsay@metrodigi.com
wrote:

Hi again Michael :slight_smile: I’m new to VPC, but I feel like I’ve poured over
all the settings there. Would the behavior be different with knife than
ssh on the same workstation in this regard?

On Fri, Mar 6, 2015 at 2:19 PM, Michael Weinberg michael@hw-ops.com
wrote:

Hi Michael,

I generally use Cloudformation for all my VPC instances, so have not
attempted a direct bootstrap. Have you confirmed that the inbound and
outbound security group and ACL rules allow traffic both directions on 22,
80, 443, and the ephemeral ports?


Michael F. Weinberg | Director of Operations
http://heavywaterops.com | @heavywaterops

On Fri, Mar 6, 2015 at 2:16 PM, Michael Lindsay <mlindsay@metrodigi.com

wrote:

Hello everyone…

I’m attempting my first knife ec2 create commands for building
instances inside a VPC.

The problem I’m facing is that “Waiting for sshd access to become
available…” Never completes. What’s weird (and flies in the face of
what I thought was going on) is that while I’m watching little dots appear,
in another terminal window on the same workstation, using the same identity
file, user, and I think hostname (EIP) that the knife command is using, I
CAN ssh into the instance.
Any ideas what might cause this?

  • Michael

#8

@Julian, it did find it’s way to tcp_test_ssh after all. Running with
Debug enabled I can see ‘No ssh gateway found’. Furthermore I can see it
is trying to connect to the Private IP Address and not the EIP that I
provided. Which explains the difference between my ssh and knife’s… Just
looking for a fix for that now.

  • M

On Fri, Mar 6, 2015 at 3:04 PM, Michael Lindsay mlindsay@metrodigi.com
wrote:

@Julian: hadn’t thought to look at the source code, that’s good info (and
I can see why the dots roll out more slowly :wink: ). I’m not connecting to
an ubuntu instance so looks like it wouldn’t be TCP connection in the
second git file but instead something from here?

https://github.com/chef/knife-ec2/blob/master/lib/chef/knife/ec2_server_create.rb#L748-L769

@Darren: It’s a public subnet, so no proxy required

On Fri, Mar 6, 2015 at 2:48 PM, Darren Dyke dwdyke@gmail.com wrote:

Hello, are you trying to configure a node on a private subnet? If so are
you supplying the --ssh-gateway option/configure ssh config proxies?

On Fri, Mar 6, 2015 at 5:32 PM, Michael Weinberg michael@hw-ops.com
wrote:

No, it shouldn’t be. But I don’t know what is actually happening during
’waiting for sshd,’ so on the off chance there’s some additional
communication there I thought I’d suggest it.


Michael F. Weinberg | Director of Operations
http://heavywaterops.com | @heavywaterops

On Fri, Mar 6, 2015 at 2:29 PM, Michael Lindsay mlindsay@metrodigi.com
wrote:

Hi again Michael :slight_smile: I’m new to VPC, but I feel like I’ve poured over
all the settings there. Would the behavior be different with knife than
ssh on the same workstation in this regard?

On Fri, Mar 6, 2015 at 2:19 PM, Michael Weinberg michael@hw-ops.com
wrote:

Hi Michael,

I generally use Cloudformation for all my VPC instances, so have not
attempted a direct bootstrap. Have you confirmed that the inbound and
outbound security group and ACL rules allow traffic both directions on 22,
80, 443, and the ephemeral ports?


Michael F. Weinberg | Director of Operations
http://heavywaterops.com | @heavywaterops

On Fri, Mar 6, 2015 at 2:16 PM, Michael Lindsay <
mlindsay@metrodigi.com> wrote:

Hello everyone…

I’m attempting my first knife ec2 create commands for building
instances inside a VPC.

The problem I’m facing is that “Waiting for sshd access to become
available…” Never completes. What’s weird (and flies in the face of
what I thought was going on) is that while I’m watching little dots appear,
in another terminal window on the same workstation, using the same identity
file, user, and I think hostname (EIP) that the knife command is using, I
CAN ssh into the instance.
Any ideas what might cause this?

  • Michael

#9

@Michael L.

If you are using the --associate-eip flag, try setting
"–server-connect-attribute public_ip_address" See below from the ec2
server create:

option :server_connect_attribute, :long => “–server-connect-attribute
ATTRIBUTE”, :short => “-a ATTRIBUTE”, :description => “The EC2 server
attribute to use for SSH connection. Use this attr for creating VPC
instances along with --associate-eip”, :default => nil

On Fri, Mar 6, 2015 at 6:19 PM, Michael Lindsay mlindsay@metrodigi.com
wrote:

@Julian, it did find it’s way to tcp_test_ssh after all. Running with
Debug enabled I can see ‘No ssh gateway found’. Furthermore I can see it
is trying to connect to the Private IP Address and not the EIP that I
provided. Which explains the difference between my ssh and knife’s… Just
looking for a fix for that now.

  • M

On Fri, Mar 6, 2015 at 3:04 PM, Michael Lindsay mlindsay@metrodigi.com
wrote:

@Julian: hadn’t thought to look at the source code, that’s good info (and
I can see why the dots roll out more slowly :wink: ). I’m not connecting to
an ubuntu instance so looks like it wouldn’t be TCP connection in the
second git file but instead something from here?

https://github.com/chef/knife-ec2/blob/master/lib/chef/knife/ec2_server_create.rb#L748-L769

@Darren: It’s a public subnet, so no proxy required

On Fri, Mar 6, 2015 at 2:48 PM, Darren Dyke dwdyke@gmail.com wrote:

Hello, are you trying to configure a node on a private subnet? If so are
you supplying the --ssh-gateway option/configure ssh config proxies?

On Fri, Mar 6, 2015 at 5:32 PM, Michael Weinberg michael@hw-ops.com
wrote:

No, it shouldn’t be. But I don’t know what is actually happening during
’waiting for sshd,’ so on the off chance there’s some additional
communication there I thought I’d suggest it.


Michael F. Weinberg | Director of Operations
http://heavywaterops.com | @heavywaterops

On Fri, Mar 6, 2015 at 2:29 PM, Michael Lindsay <mlindsay@metrodigi.com

wrote:

Hi again Michael :slight_smile: I’m new to VPC, but I feel like I’ve poured over
all the settings there. Would the behavior be different with knife than
ssh on the same workstation in this regard?

On Fri, Mar 6, 2015 at 2:19 PM, Michael Weinberg michael@hw-ops.com
wrote:

Hi Michael,

I generally use Cloudformation for all my VPC instances, so have not
attempted a direct bootstrap. Have you confirmed that the inbound and
outbound security group and ACL rules allow traffic both directions on 22,
80, 443, and the ephemeral ports?


Michael F. Weinberg | Director of Operations
http://heavywaterops.com | @heavywaterops

On Fri, Mar 6, 2015 at 2:16 PM, Michael Lindsay <
mlindsay@metrodigi.com> wrote:

Hello everyone…

I’m attempting my first knife ec2 create commands for building
instances inside a VPC.

The problem I’m facing is that “Waiting for sshd access to become
available…” Never completes. What’s weird (and flies in the face of
what I thought was going on) is that while I’m watching little dots appear,
in another terminal window on the same workstation, using the same identity
file, user, and I think hostname (EIP) that the knife command is using, I
CAN ssh into the instance.
Any ideas what might cause this?

  • Michael

#10

Things are definitely improving:

First attempt with this debug statement indicates that it’s using the EID
to connect, but got this: Net::SSH::HostKeyMismatch. Realized since I’ve
been putting so many different instances behind this EID, I had to kill
it’s known_hosts entry (shouldn’t --no-host-key-verify be doing this for
me?)

Now I’m getting “Authentication failed for user @[EIP].
(Net::SSH::AuthenticationFailed)” Which appears to be missing the username.

This is the stack trace:

…/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/net-ssh-2.9.2/lib/net/ssh.rb:224:in
`start’: Authentication failed for user @[EIP]
(Net::SSH::AuthenticationFailed)

from
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/net-ssh-gateway-1.2.0/lib/net/ssh/gateway.rb:74:in
`initialize’

from
/Users/mnl/.chefdk/gem/ruby/2.1.0/gems/knife-ec2-0.10.0/lib/chef/knife/ec2_server_create.rb:815:in
`new’

from
/Users/mnl/.chefdk/gem/ruby/2.1.0/gems/knife-ec2-0.10.0/lib/chef/knife/ec2_server_create.rb:815:in
`configure_ssh_gateway’

from
/Users/mnl/.chefdk/gem/ruby/2.1.0/gems/knife-ec2-0.10.0/lib/chef/knife/ec2_server_create.rb:779:in
`tunnel_test_ssh’

from
/Users/mnl/.chefdk/gem/ruby/2.1.0/gems/knife-ec2-0.10.0/lib/chef/knife/ec2_server_create.rb:766:in
`wait_for_tunnelled_sshd’

from
/Users/mnl/.chefdk/gem/ruby/2.1.0/gems/knife-ec2-0.10.0/lib/chef/knife/ec2_server_create.rb:730:in
`wait_for_sshd’

from
/Users/mnl/.chefdk/gem/ruby/2.1.0/gems/knife-ec2-0.10.0/lib/chef/knife/ec2_server_create.rb:406:in
`run’

from /opt/chefdk/embedded/apps/chef/lib/chef/knife.rb:417:in `block in
run_with_pretty_exceptions’

from /opt/chefdk/embedded/apps/chef/lib/chef/local_mode.rb:38:in
`with_server_connectivity’

from /opt/chefdk/embedded/apps/chef/lib/chef/knife.rb:416:in
`run_with_pretty_exceptions’

from /opt/chefdk/embedded/apps/chef/lib/chef/knife.rb:213:in `run’

from /opt/chefdk/embedded/apps/chef/lib/chef/application/knife.rb:139:in
`run’

from /opt/chefdk/embedded/apps/chef/bin/knife:25:in `<top (required)>’

from /opt/chefdk/bin/knife:40:in `load’

from /opt/chefdk/bin/knife:40:in `’

On Fri, Mar 6, 2015 at 3:36 PM, Darren Dyke dwdyke@gmail.com wrote:

@Michael L.

If you are using the --associate-eip flag, try setting
"–server-connect-attribute public_ip_address" See below from the ec2
server create:
https://github.com/chef/knife-ec2/blob/master/lib/chef/knife/ec2_server_create.rb#L260-L264

option :server_connect_attribute, :long => “–server-connect-attribute
ATTRIBUTE”, :short => “-a ATTRIBUTE”, :description => “The EC2 server
attribute to use for SSH connection. Use this attr for creating VPC
instances along with --associate-eip”, :default => nil

On Fri, Mar 6, 2015 at 6:19 PM, Michael Lindsay mlindsay@metrodigi.com
wrote:

@Julian, it did find it’s way to tcp_test_ssh after all. Running with
Debug enabled I can see ‘No ssh gateway found’. Furthermore I can see it
is trying to connect to the Private IP Address and not the EIP that I
provided. Which explains the difference between my ssh and knife’s… Just
looking for a fix for that now.

  • M

On Fri, Mar 6, 2015 at 3:04 PM, Michael Lindsay mlindsay@metrodigi.com
wrote:

@Julian: hadn’t thought to look at the source code, that’s good info
(and I can see why the dots roll out more slowly :wink: ). I’m not connecting
to an ubuntu instance so looks like it wouldn’t be TCP connection in the
second git file but instead something from here?

https://github.com/chef/knife-ec2/blob/master/lib/chef/knife/ec2_server_create.rb#L748-L769

@Darren: It’s a public subnet, so no proxy required

On Fri, Mar 6, 2015 at 2:48 PM, Darren Dyke dwdyke@gmail.com wrote:

Hello, are you trying to configure a node on a private subnet? If so
are you supplying the --ssh-gateway option/configure ssh config proxies?

On Fri, Mar 6, 2015 at 5:32 PM, Michael Weinberg michael@hw-ops.com
wrote:

No, it shouldn’t be. But I don’t know what is actually happening
during ‘waiting for sshd,’ so on the off chance there’s some additional
communication there I thought I’d suggest it.


Michael F. Weinberg | Director of Operations
http://heavywaterops.com | @heavywaterops

On Fri, Mar 6, 2015 at 2:29 PM, Michael Lindsay <
mlindsay@metrodigi.com> wrote:

Hi again Michael :slight_smile: I’m new to VPC, but I feel like I’ve poured over
all the settings there. Would the behavior be different with knife than
ssh on the same workstation in this regard?

On Fri, Mar 6, 2015 at 2:19 PM, Michael Weinberg michael@hw-ops.com
wrote:

Hi Michael,

I generally use Cloudformation for all my VPC instances, so have not
attempted a direct bootstrap. Have you confirmed that the inbound and
outbound security group and ACL rules allow traffic both directions on 22,
80, 443, and the ephemeral ports?


Michael F. Weinberg | Director of Operations
http://heavywaterops.com | @heavywaterops

On Fri, Mar 6, 2015 at 2:16 PM, Michael Lindsay <
mlindsay@metrodigi.com> wrote:

Hello everyone…

I’m attempting my first knife ec2 create commands for building
instances inside a VPC.

The problem I’m facing is that “Waiting for sshd access to become
available…” Never completes. What’s weird (and flies in the face of
what I thought was going on) is that while I’m watching little dots appear,
in another terminal window on the same workstation, using the same identity
file, user, and I think hostname (EIP) that the knife command is using, I
CAN ssh into the instance.
Any ideas what might cause this?

  • Michael

#11

Are you passing --ssh-user ?

On Fri, Mar 6, 2015 at 6:54 PM, Michael Lindsay mlindsay@metrodigi.com
wrote:

Things are definitely improving:

First attempt with this debug statement indicates that it’s using the EID
to connect, but got this: Net::SSH::HostKeyMismatch. Realized since I’ve
been putting so many different instances behind this EID, I had to kill
it’s known_hosts entry (shouldn’t --no-host-key-verify be doing this for
me?)

Now I’m getting “Authentication failed for user @[EIP].
(Net::SSH::AuthenticationFailed)” Which appears to be missing the username.

This is the stack trace:

…/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/net-ssh-2.9.2/lib/net/ssh.rb:224:in
`start’: Authentication failed for user @[EIP]
(Net::SSH::AuthenticationFailed)

from
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/net-ssh-gateway-1.2.0/lib/net/ssh/gateway.rb:74:in
`initialize’

from
/Users/mnl/.chefdk/gem/ruby/2.1.0/gems/knife-ec2-0.10.0/lib/chef/knife/ec2_server_create.rb:815:in
`new’

from
/Users/mnl/.chefdk/gem/ruby/2.1.0/gems/knife-ec2-0.10.0/lib/chef/knife/ec2_server_create.rb:815:in
`configure_ssh_gateway’

from
/Users/mnl/.chefdk/gem/ruby/2.1.0/gems/knife-ec2-0.10.0/lib/chef/knife/ec2_server_create.rb:779:in
`tunnel_test_ssh’

from
/Users/mnl/.chefdk/gem/ruby/2.1.0/gems/knife-ec2-0.10.0/lib/chef/knife/ec2_server_create.rb:766:in
`wait_for_tunnelled_sshd’

from
/Users/mnl/.chefdk/gem/ruby/2.1.0/gems/knife-ec2-0.10.0/lib/chef/knife/ec2_server_create.rb:730:in
`wait_for_sshd’

from
/Users/mnl/.chefdk/gem/ruby/2.1.0/gems/knife-ec2-0.10.0/lib/chef/knife/ec2_server_create.rb:406:in
`run’

from /opt/chefdk/embedded/apps/chef/lib/chef/knife.rb:417:in `block in
run_with_pretty_exceptions’

from /opt/chefdk/embedded/apps/chef/lib/chef/local_mode.rb:38:in
`with_server_connectivity’

from /opt/chefdk/embedded/apps/chef/lib/chef/knife.rb:416:in
`run_with_pretty_exceptions’

from /opt/chefdk/embedded/apps/chef/lib/chef/knife.rb:213:in `run’

from /opt/chefdk/embedded/apps/chef/lib/chef/application/knife.rb:139:in
`run’

from /opt/chefdk/embedded/apps/chef/bin/knife:25:in `<top (required)>’

from /opt/chefdk/bin/knife:40:in `load’

from /opt/chefdk/bin/knife:40:in `’

On Fri, Mar 6, 2015 at 3:36 PM, Darren Dyke dwdyke@gmail.com wrote:

@Michael L.

If you are using the --associate-eip flag, try setting
"–server-connect-attribute public_ip_address" See below from the ec2
server create:
https://github.com/chef/knife-ec2/blob/master/lib/chef/knife/ec2_server_create.rb#L260-L264

option :server_connect_attribute, :long => “–server-connect-attribute
ATTRIBUTE”, :short => “-a ATTRIBUTE”, :description => “The EC2 server
attribute to use for SSH connection. Use this attr for creating VPC
instances along with --associate-eip”, :default => nil

On Fri, Mar 6, 2015 at 6:19 PM, Michael Lindsay mlindsay@metrodigi.com
wrote:

@Julian, it did find it’s way to tcp_test_ssh after all. Running with
Debug enabled I can see ‘No ssh gateway found’. Furthermore I can see it
is trying to connect to the Private IP Address and not the EIP that I
provided. Which explains the difference between my ssh and knife’s… Just
looking for a fix for that now.

  • M

On Fri, Mar 6, 2015 at 3:04 PM, Michael Lindsay mlindsay@metrodigi.com
wrote:

@Julian: hadn’t thought to look at the source code, that’s good info
(and I can see why the dots roll out more slowly :wink: ). I’m not connecting
to an ubuntu instance so looks like it wouldn’t be TCP connection in the
second git file but instead something from here?

https://github.com/chef/knife-ec2/blob/master/lib/chef/knife/ec2_server_create.rb#L748-L769

@Darren: It’s a public subnet, so no proxy required

On Fri, Mar 6, 2015 at 2:48 PM, Darren Dyke dwdyke@gmail.com wrote:

Hello, are you trying to configure a node on a private subnet? If so
are you supplying the --ssh-gateway option/configure ssh config proxies?

On Fri, Mar 6, 2015 at 5:32 PM, Michael Weinberg michael@hw-ops.com
wrote:

No, it shouldn’t be. But I don’t know what is actually happening
during ‘waiting for sshd,’ so on the off chance there’s some additional
communication there I thought I’d suggest it.


Michael F. Weinberg | Director of Operations
http://heavywaterops.com | @heavywaterops

On Fri, Mar 6, 2015 at 2:29 PM, Michael Lindsay <
mlindsay@metrodigi.com> wrote:

Hi again Michael :slight_smile: I’m new to VPC, but I feel like I’ve poured
over all the settings there. Would the behavior be different with knife
than ssh on the same workstation in this regard?

On Fri, Mar 6, 2015 at 2:19 PM, Michael Weinberg <michael@hw-ops.com

wrote:

Hi Michael,

I generally use Cloudformation for all my VPC instances, so have
not attempted a direct bootstrap. Have you confirmed that the inbound and
outbound security group and ACL rules allow traffic both directions on 22,
80, 443, and the ephemeral ports?


Michael F. Weinberg | Director of Operations
http://heavywaterops.com | @heavywaterops

On Fri, Mar 6, 2015 at 2:16 PM, Michael Lindsay <
mlindsay@metrodigi.com> wrote:

Hello everyone…

I’m attempting my first knife ec2 create commands for building
instances inside a VPC.

The problem I’m facing is that “Waiting for sshd access to become
available…” Never completes. What’s weird (and flies in the face of
what I thought was going on) is that while I’m watching little dots appear,
in another terminal window on the same workstation, using the same identity
file, user, and I think hostname (EIP) that the knife command is using, I
CAN ssh into the instance.
Any ideas what might cause this?

  • Michael

#12

I’d left --ssh-gateway [EID] in there from a previous experiment. Running
without that appears to have fixed the problem. Opened up chef server’s
443 to the instance’s security group and now I’m off and (very verbosely)
running.

Thanks Darren & Michael!

On Fri, Mar 6, 2015 at 3:57 PM, Darren Dyke dwdyke@gmail.com wrote:

Are you passing --ssh-user ?

On Fri, Mar 6, 2015 at 6:54 PM, Michael Lindsay mlindsay@metrodigi.com
wrote:

Things are definitely improving:

First attempt with this debug statement indicates that it’s using the EID
to connect, but got this: Net::SSH::HostKeyMismatch. Realized since I’ve
been putting so many different instances behind this EID, I had to kill
it’s known_hosts entry (shouldn’t --no-host-key-verify be doing this for
me?)

Now I’m getting “Authentication failed for user @[EIP].
(Net::SSH::AuthenticationFailed)” Which appears to be missing the username.

This is the stack trace:

…/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/net-ssh-2.9.2/lib/net/ssh.rb:224:in
`start’: Authentication failed for user @[EIP]
(Net::SSH::AuthenticationFailed)

from
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/net-ssh-gateway-1.2.0/lib/net/ssh/gateway.rb:74:in
`initialize’

from
/Users/mnl/.chefdk/gem/ruby/2.1.0/gems/knife-ec2-0.10.0/lib/chef/knife/ec2_server_create.rb:815:in
`new’

from
/Users/mnl/.chefdk/gem/ruby/2.1.0/gems/knife-ec2-0.10.0/lib/chef/knife/ec2_server_create.rb:815:in
`configure_ssh_gateway’

from
/Users/mnl/.chefdk/gem/ruby/2.1.0/gems/knife-ec2-0.10.0/lib/chef/knife/ec2_server_create.rb:779:in
`tunnel_test_ssh’

from
/Users/mnl/.chefdk/gem/ruby/2.1.0/gems/knife-ec2-0.10.0/lib/chef/knife/ec2_server_create.rb:766:in
`wait_for_tunnelled_sshd’

from
/Users/mnl/.chefdk/gem/ruby/2.1.0/gems/knife-ec2-0.10.0/lib/chef/knife/ec2_server_create.rb:730:in
`wait_for_sshd’

from
/Users/mnl/.chefdk/gem/ruby/2.1.0/gems/knife-ec2-0.10.0/lib/chef/knife/ec2_server_create.rb:406:in
`run’

from /opt/chefdk/embedded/apps/chef/lib/chef/knife.rb:417:in `block in
run_with_pretty_exceptions’

from /opt/chefdk/embedded/apps/chef/lib/chef/local_mode.rb:38:in
`with_server_connectivity’

from /opt/chefdk/embedded/apps/chef/lib/chef/knife.rb:416:in
`run_with_pretty_exceptions’

from /opt/chefdk/embedded/apps/chef/lib/chef/knife.rb:213:in `run’

from /opt/chefdk/embedded/apps/chef/lib/chef/application/knife.rb:139:in
`run’

from /opt/chefdk/embedded/apps/chef/bin/knife:25:in `<top (required)>’

from /opt/chefdk/bin/knife:40:in `load’

from /opt/chefdk/bin/knife:40:in `’

On Fri, Mar 6, 2015 at 3:36 PM, Darren Dyke dwdyke@gmail.com wrote:

@Michael L.

If you are using the --associate-eip flag, try setting
"–server-connect-attribute public_ip_address" See below from the ec2
server create:
https://github.com/chef/knife-ec2/blob/master/lib/chef/knife/ec2_server_create.rb#L260-L264

option :server_connect_attribute, :long => “–server-connect-attribute
ATTRIBUTE”, :short => “-a ATTRIBUTE”, :description => “The EC2 server
attribute to use for SSH connection. Use this attr for creating VPC
instances along with --associate-eip”, :default => nil

On Fri, Mar 6, 2015 at 6:19 PM, Michael Lindsay mlindsay@metrodigi.com
wrote:

@Julian, it did find it’s way to tcp_test_ssh after all. Running with
Debug enabled I can see ‘No ssh gateway found’. Furthermore I can see it
is trying to connect to the Private IP Address and not the EIP that I
provided. Which explains the difference between my ssh and knife’s… Just
looking for a fix for that now.

  • M

On Fri, Mar 6, 2015 at 3:04 PM, Michael Lindsay <mlindsay@metrodigi.com

wrote:

@Julian: hadn’t thought to look at the source code, that’s good info
(and I can see why the dots roll out more slowly :wink: ). I’m not connecting
to an ubuntu instance so looks like it wouldn’t be TCP connection in the
second git file but instead something from here?

https://github.com/chef/knife-ec2/blob/master/lib/chef/knife/ec2_server_create.rb#L748-L769

@Darren: It’s a public subnet, so no proxy required

On Fri, Mar 6, 2015 at 2:48 PM, Darren Dyke dwdyke@gmail.com wrote:

Hello, are you trying to configure a node on a private subnet? If so
are you supplying the --ssh-gateway option/configure ssh config proxies?

On Fri, Mar 6, 2015 at 5:32 PM, Michael Weinberg michael@hw-ops.com
wrote:

No, it shouldn’t be. But I don’t know what is actually happening
during ‘waiting for sshd,’ so on the off chance there’s some additional
communication there I thought I’d suggest it.


Michael F. Weinberg | Director of Operations
http://heavywaterops.com | @heavywaterops

On Fri, Mar 6, 2015 at 2:29 PM, Michael Lindsay <
mlindsay@metrodigi.com> wrote:

Hi again Michael :slight_smile: I’m new to VPC, but I feel like I’ve poured
over all the settings there. Would the behavior be different with knife
than ssh on the same workstation in this regard?

On Fri, Mar 6, 2015 at 2:19 PM, Michael Weinberg <
michael@hw-ops.com> wrote:

Hi Michael,

I generally use Cloudformation for all my VPC instances, so have
not attempted a direct bootstrap. Have you confirmed that the inbound and
outbound security group and ACL rules allow traffic both directions on 22,
80, 443, and the ephemeral ports?


Michael F. Weinberg | Director of Operations
http://heavywaterops.com | @heavywaterops

On Fri, Mar 6, 2015 at 2:16 PM, Michael Lindsay <
mlindsay@metrodigi.com> wrote:

Hello everyone…

I’m attempting my first knife ec2 create commands for building
instances inside a VPC.

The problem I’m facing is that “Waiting for sshd access to become
available…” Never completes. What’s weird (and flies in the face of
what I thought was going on) is that while I’m watching little dots appear,
in another terminal window on the same workstation, using the same identity
file, user, and I think hostname (EIP) that the knife command is using, I
CAN ssh into the instance.
Any ideas what might cause this?

  • Michael

#13

Wicked man! Glad you got it working

On Fri, Mar 6, 2015 at 7:05 PM, Michael Lindsay mlindsay@metrodigi.com
wrote:

I’d left --ssh-gateway [EID] in there from a previous experiment. Running
without that appears to have fixed the problem. Opened up chef server’s
443 to the instance’s security group and now I’m off and (very verbosely)
running.

Thanks Darren & Michael!

On Fri, Mar 6, 2015 at 3:57 PM, Darren Dyke dwdyke@gmail.com wrote:

Are you passing --ssh-user ?

On Fri, Mar 6, 2015 at 6:54 PM, Michael Lindsay mlindsay@metrodigi.com
wrote:

Things are definitely improving:

First attempt with this debug statement indicates that it’s using the
EID to connect, but got this: Net::SSH::HostKeyMismatch. Realized since
I’ve been putting so many different instances behind this EID, I had to
kill it’s known_hosts entry (shouldn’t --no-host-key-verify be doing this
for me?)

Now I’m getting “Authentication failed for user @[EIP].
(Net::SSH::AuthenticationFailed)” Which appears to be missing the username.

This is the stack trace:

…/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/net-ssh-2.9.2/lib/net/ssh.rb:224:in
`start’: Authentication failed for user @[EIP]
(Net::SSH::AuthenticationFailed)

from
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/net-ssh-gateway-1.2.0/lib/net/ssh/gateway.rb:74:in
`initialize’

from
/Users/mnl/.chefdk/gem/ruby/2.1.0/gems/knife-ec2-0.10.0/lib/chef/knife/ec2_server_create.rb:815:in
`new’

from
/Users/mnl/.chefdk/gem/ruby/2.1.0/gems/knife-ec2-0.10.0/lib/chef/knife/ec2_server_create.rb:815:in
`configure_ssh_gateway’

from
/Users/mnl/.chefdk/gem/ruby/2.1.0/gems/knife-ec2-0.10.0/lib/chef/knife/ec2_server_create.rb:779:in
`tunnel_test_ssh’

from
/Users/mnl/.chefdk/gem/ruby/2.1.0/gems/knife-ec2-0.10.0/lib/chef/knife/ec2_server_create.rb:766:in
`wait_for_tunnelled_sshd’

from
/Users/mnl/.chefdk/gem/ruby/2.1.0/gems/knife-ec2-0.10.0/lib/chef/knife/ec2_server_create.rb:730:in
`wait_for_sshd’

from
/Users/mnl/.chefdk/gem/ruby/2.1.0/gems/knife-ec2-0.10.0/lib/chef/knife/ec2_server_create.rb:406:in
`run’

from /opt/chefdk/embedded/apps/chef/lib/chef/knife.rb:417:in `block in
run_with_pretty_exceptions’

from /opt/chefdk/embedded/apps/chef/lib/chef/local_mode.rb:38:in
`with_server_connectivity’

from /opt/chefdk/embedded/apps/chef/lib/chef/knife.rb:416:in
`run_with_pretty_exceptions’

from /opt/chefdk/embedded/apps/chef/lib/chef/knife.rb:213:in `run’

from /opt/chefdk/embedded/apps/chef/lib/chef/application/knife.rb:139:in
`run’

from /opt/chefdk/embedded/apps/chef/bin/knife:25:in `<top (required)>’

from /opt/chefdk/bin/knife:40:in `load’

from /opt/chefdk/bin/knife:40:in `’

On Fri, Mar 6, 2015 at 3:36 PM, Darren Dyke dwdyke@gmail.com wrote:

@Michael L.

If you are using the --associate-eip flag, try setting
"–server-connect-attribute public_ip_address" See below from the ec2
server create:
https://github.com/chef/knife-ec2/blob/master/lib/chef/knife/ec2_server_create.rb#L260-L264

option :server_connect_attribute, :long => “–server-connect-attribute
ATTRIBUTE”, :short => “-a ATTRIBUTE”, :description => “The EC2 server
attribute to use for SSH connection. Use this attr for creating VPC
instances along with --associate-eip”, :default => nil

On Fri, Mar 6, 2015 at 6:19 PM, Michael Lindsay <mlindsay@metrodigi.com

wrote:

@Julian, it did find it’s way to tcp_test_ssh after all. Running
with Debug enabled I can see ‘No ssh gateway found’. Furthermore I can see
it is trying to connect to the Private IP Address and not the EIP that I
provided. Which explains the difference between my ssh and knife’s… Just
looking for a fix for that now.

  • M

On Fri, Mar 6, 2015 at 3:04 PM, Michael Lindsay <
mlindsay@metrodigi.com> wrote:

@Julian: hadn’t thought to look at the source code, that’s good info
(and I can see why the dots roll out more slowly :wink: ). I’m not connecting
to an ubuntu instance so looks like it wouldn’t be TCP connection in the
second git file but instead something from here?

https://github.com/chef/knife-ec2/blob/master/lib/chef/knife/ec2_server_create.rb#L748-L769

@Darren: It’s a public subnet, so no proxy required

On Fri, Mar 6, 2015 at 2:48 PM, Darren Dyke dwdyke@gmail.com wrote:

Hello, are you trying to configure a node on a private subnet? If so
are you supplying the --ssh-gateway option/configure ssh config proxies?

On Fri, Mar 6, 2015 at 5:32 PM, Michael Weinberg <michael@hw-ops.com

wrote:

No, it shouldn’t be. But I don’t know what is actually happening
during ‘waiting for sshd,’ so on the off chance there’s some additional
communication there I thought I’d suggest it.


Michael F. Weinberg | Director of Operations
http://heavywaterops.com | @heavywaterops

On Fri, Mar 6, 2015 at 2:29 PM, Michael Lindsay <
mlindsay@metrodigi.com> wrote:

Hi again Michael :slight_smile: I’m new to VPC, but I feel like I’ve poured
over all the settings there. Would the behavior be different with knife
than ssh on the same workstation in this regard?

On Fri, Mar 6, 2015 at 2:19 PM, Michael Weinberg <
michael@hw-ops.com> wrote:

Hi Michael,

I generally use Cloudformation for all my VPC instances, so have
not attempted a direct bootstrap. Have you confirmed that the inbound and
outbound security group and ACL rules allow traffic both directions on 22,
80, 443, and the ephemeral ports?


Michael F. Weinberg | Director of Operations
http://heavywaterops.com | @heavywaterops

On Fri, Mar 6, 2015 at 2:16 PM, Michael Lindsay <
mlindsay@metrodigi.com> wrote:

Hello everyone…

I’m attempting my first knife ec2 create commands for building
instances inside a VPC.

The problem I’m facing is that “Waiting for sshd access to
become available…” Never completes. What’s weird (and flies in the face
of what I thought was going on) is that while I’m watching little dots
appear, in another terminal window on the same workstation, using the same
identity file, user, and I think hostname (EIP) that the knife command is
using, I CAN ssh into the instance.
Any ideas what might cause this?

  • Michael