Knife bootstrap Net::SSH::ConnectionTimeout (chekdk)

Chef Infra (archive)
1

Bootstrapping a node with:
knife bootstrap 192.168.1.116 -x root -N fedtest -i ~/.ssh/id_rsa
immediately fails with:
ERROR: Net::SSH::ConnectionTimeout: Net::SSH::ConnectionTimeout

Simple virtualbox lab env, with Chef Server, workstation, and test node.
Workstation properly communicates with Chef Server. Starter kit used for config.
Workstation connects with SSH properly to Node (ssh-copy-id, etc. all done and confirmed).
Also checked underlying Ruby net:ssh with script below - works as expected.

$ chef --version
Chef Development Kit Version: 1.3.16
chef-client version: 12.18.31
delivery version: master (fc66beb4e2ee47d488fe63b49453483ee104f89b)
berks version: 5.6.3
kitchen version: 1.15.0

Platforms I’ve tried:
Wkstation: Fedora 25, Archlinux
Server: CentOS
Node: Fedora 25, CentOS

knife command verbose output (snipped)

Connecting to 192.168.1.116
DEBUG: Looking for bootstrap template in /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-12.18.31/lib/chef/knife/bootstrap/templates
DEBUG: Found bootstrap template in /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-12.18.31/lib/chef/knife/bootstrap/templates
DEBUG: Adding 192.168.1.116
DEBUG: establishing connection to 192.168.1.116:22
/opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/transport/session.rb:90:in `rescue in initialize': Net::SSH::ConnectionTimeout (Net::SSH::ConnectionTimeout)
	from /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh/transport/session.rb:57:in `initialize'
	from /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh.rb:232:in `new'
	from /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/net-ssh-3.2.0/lib/net/ssh.rb:232:in `start'
	from /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/net-ssh-multi-1.2.1/lib/net/ssh/multi/server.rb:186:in `new_session'
	from /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/net-ssh-multi-1.2.1/lib/net/ssh/multi/session.rb:488:in `next_session'
	from /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/net-ssh-multi-1.2.1/lib/net/ssh/multi/server.rb:138:in `session'
	from /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/net-ssh-multi-1.2.1/lib/net/ssh/multi/session_actions.rb:36:in `block (2 levels) in sessions'
	from /opt/chefdk/embedded/lib/ruby/gems/2.3.0/gems/logging-2.1.0/lib/logging/diagnostic_context.rb:450:in `block in create_with_logging_context'

Script used to test NET:SSH directly (Note: ruby is not otherwise installed)

$ which ruby
/opt/chefdk/embedded/bin/ruby

#!/usr/bin/env ruby

require 'net/ssh'

puts "opening connection.\n"
new_connection = Net::SSH.start('192.168.1.116', 'root', {:keys => ['~/.ssh/id_rsa'], :keepalive => true, :keepalive_interval => 60, :timeout => 60}) 
puts "connection established, run uptime.\n"
puts new_connection.exec!('uptime')
puts "running uname -a\n"
puts new_connection.exec!('uname -a')
puts "sleeping for 300 seconds.\n"
(1..5).each do |iterator|
  sleep_seconds = iterator * 60
  sleep 60
  puts "#{sleep_seconds}\n"
end
puts "running uptime.\n"
puts new_connection.exec!('uptime')
puts "running uname -a\n"
puts new_connection.exec!('uname -a')
puts "closing connection.\n"
new_connection.close
puts "done.\n"

$ ruby ./test.rb 
opening connection.
connection established, run uptime.
 06:58:37 up  1:10,  0 users,  load average: 0.00, 0.00, 0.00
running uname -a
Linux fedtest.xxxxx.com 4.8.6-300.fc25.x86_64 #1 SMP Tue Nov 1 12:36:38 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
sleeping for 300 seconds.

I’d appreciate any help / thoughts. This info was also posted to Github, however after 10 days of no repsonse thought I’d try here.

Rgrds

2

Does it bootstrap without adding “-i ~/.ssh/id_rsa” ?

3

No it does not, password auth attempts fails with the same immediate response as well. When monitoring with wireshark - nothing hits the wire.

4

can you ssh from the work station to the node? scan it with nmap to see what ports are open? is there a proxy in between?

5

As stated above:

Yes - SSH to the node works.
Yes - SSH through ruby outside of knife works.

There is no proxy.
It’s all on a local PC using VirtualBox VMs or the host as a workstation.
I’ve disabled firewalld on the Node as well to be overall “sure”.

I do appreciate you trying to check basics but it appears to be clear that the problem is within the “knife” call itself.

7

While I’ve not had any real progress, it seems that this condition may be a confluence of Gnome desktop and ssh-agent with the underlying net::ssh ruby implementation.

I’m not sure why the pure ruby script would work and knife calling the gem wouldn’t - if anyone has any thoughts I welcome them . .