Kitchen Action - stopped waiting after 160 attempts without success while launching windows instance

Chef Infra (archive)
1

Hi,

I was trying to launch windows-2012 instance using kitchen-ec2 in my cookbook. I observe the instance gets created when i run kitchen create.But, eventually the instance gets terminated after 160 attempts. I am not sure what could be missing in the config, that’s causing the instance not to create in my first attempt. In the second attempt, I have also increased the retryable_tries to 1500 and that caused a different error where it seemed to fail with authentication error. Any help on where i could be miss configuring things in my kitchen.yml file or any additional values to be included. Thanks

Kitchen.yml:

driver:
name: ec2
associate_public_ip: false
shared_credentials_profile: test_user
aws_ssh_key_id: windows_test
instance_type: t2.medium
region: us-east-1
availability_zone: us-east-1c
subnet_id: subnet-a8989
security_group_ids:

  • “sg-3433”
  • "sg-abcdef"
    retryable_tries: 160
    retryable_sleep: 5

transport:
name: winrm
ssh_key: windowstest.pem
associate_public_ip: false
connection_timeout: 200
connection_retries: 10
username: Administrator

provisioner:
name: chef_zero
http_proxy: http://proxy.abc.com:8091
https_proxy: http://proxy.abc.com:8091
no_proxy: "169.254.169.254,amazonaws.com"
environments_path: environments
roles_path: roles
require_chef_omnibus: true
client_rb:
ssl_verify_mode: :verify_none
verify_api_cert: false

platforms:

  • name: windows-2012
    driver:
    image_id: ami-09udlj
    interface: private
    block_device_mappings:
    - device_name: /dev/sda1
    ebs:
    volume_size: 50
    delete_on_termination: true
    volume_type: gp2
    snapshot_id: snap-dlk0uere
    - device_name: /dev/xvdg
    ebs:
    volume_size: 30
    delete_on_termination: true
    volume_type: gp2
    snapshot_id: snap-0dfudf
    tags:
    Name: windows-2012
    suites:
  • name: default
    run_list:
    • recipe[testing-kitchen]
      kitchen:

    log_level: debug

    attributes:

verifier:
http_proxy: http://proxy.abc.com:8091
https_proxy: http://proxy.abc.com:8091
no_proxy: “169.254.169.254,amazonaws.com”

Error -1

D ------Exception------- D Class: Kitchen::ActionFailed D Message: Failed to complete #create action: [stopped waiting after 160 attempts without success] D ---Nested Exception--- D Class: Aws::Waiters::Errors::TooManyAttemptsError D Message: stopped waiting after 160 attempts without success

With increasing retryable_tries: 1500, i got this below error where it seemed to retrieve the password but couldn’t succeed.

Error-2

Waited 0/1500s for instance <i-0f3a7b92> to fetch windows admin password. Retrieved Windows password for instance <i-0f3a7b92>. EC2 instance <i-0f3a7b92> ready. [WinRM] connection failed. retrying in 3 seconds: #<WinRM::WinRMAuthorizationError: WinRM::WinRMAuthorizationError> [WinRM] connection failed. retrying in 3 seconds: #<WinRM::WinRMAuthorizationError: WinRM::WinRMAuthorizationError>

2

What version of Test Kitchen and EC2 drivers are you using?

Have you tried changing the transport settings?

  transport:
      max_threads: 5
      connection_timeout: 10
      connection_retries: 36
      connection_retry_sleep: 10
      max_wait_until_ready: 1200

Our instances also timed out because we use a custom AMI that requires a reboot so it takes a while to come up!

3

Thanks for looking into the error. I have been using the following versions of Test Kitchen & EC2.
kitchen version: 1.6.0
Test Kitchen version 1.6.0
kitchen-ec2 1.0.0

I have modified the transport values (posted below) before launching instance using kitchen create -l debug and observed the same error, as listed above. However, when I looked into aws console for the instance, I observed the Instance Status Checks present in Status Checks to be displaying this message

Instance reachability check failed at May 5, 2016 at 12:05:00 PM UTC-5 (3 minutes ago)

I am not familiar what the above message means, does it means something with network connectivity?

values used:

retryable_tries: 200 retryable_sleep: 5

Error
` D Console output: ---
2016/05/05 16:59:08Z: EC2ConfigMonitorState: 0
2016/05/05 16:59:08Z: Windows is being configured. SysprepState=IMAGE_STATE_UNDEPLOYABLE
2016/05/05 16:59:14Z: Difficulty detecting PV driver details for Console Output
2016/05/05 17:00:10Z: Windows is still being configured. SysprepState=IMAGE_STATE_UNDEPLOYABLE
2016/05/05 17:01:10Z: Windows is still being configured. SysprepState=IMAGE_STATE_UNDEPLOYABLE
2016/05/05 17:02:13Z: Windows is still being configured. SysprepState=IMAGE_STATE_SPECIALIZE_RESEAL_TO_OOBE
2016/05/05 17:03:18Z: Windows is still being configured. SysprepState=IMAGE_STATE_SPECIALIZE_RESEAL_TO_OOBE
2016/05/05 17:03:54Z: Message: EC2Config stopping.

Waited 0/1000s for instance to fetch windows admin password.
Retrieved Windows password for instance .
EC2 instance ready.
[WinRM] connection failed. retrying in 3 seconds: #<WinRM::WinRMAuthorizationError: WinRM::WinRMAuthorizationError>`

4

@chris_sullivan It looks like the username under transport seemed to have changed from the regular Administrator for our windows instance to different one causing WinRM::WinRMAuthorizationError error earlier. I have corrected the username and now with kitchen create -l debug i see the instance created and in ready state and when it move forwards it fails with some encoding method definition failure.

D ------Exception-------
D Class: Kitchen::ActionFailed
D Message: Failed to complete #create action: [undefined method `encoding' for nil:NilClass]
D ---Nested Exception---
D Class: NoMethodError
D Message: undefined method `encoding' for nil:NilClass
D ------Backtrace-------
D /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/rubyntlm-0.6.0/lib/net/ntlm/encode_util.rb:42:in `encode_utf16le'
D /opt/chefdk/embedded/lib/ruby/ge

5

Sorry I can’t recall seeing this error before, a quick look at the code at the top of the backtrace

      def self.encode_utf16le(str)
        str = str.force_encoding('UTF-8') if [::Encoding::ASCII_8BIT,::Encoding::US_ASCII].include?(str.encoding)
        str.dup.force_encoding('UTF-8').encode(Encoding::UTF_16LE, Encoding::UTF_8).force_encoding('UTF-8')
      end
```

Would suggest that str is nil, but the issue isn't with rubyntlm, might be missing test kitchen configuration, can you supply the chef --version and backtrace.
6

I am currently using the following version of chef.

Chef Development Kit Version: 0.12.0

back trace

D      ------Exception-------
D      Class: Kitchen::ActionFailed
D      Message: Failed to complete #create action: [undefined method `encoding' for nil:NilClass]
D      ---Nested Exception---
D      Class: NoMethodError
D      Message: undefined method `encoding' for nil:NilClass
D      ------Backtrace-------
D      /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/rubyntlm-0.6.0/lib/net/ntlm/encode_util.rb:42:in `encode_utf16le'
D      /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/rubyntlm-0.6.0/lib/net/ntlm/client/session.rb:187:in `oem_or_unicode_str'
D      /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/rubyntlm-0.6.0/lib/net/ntlm/client/session.rb:172:in `password'
D      /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/rubyntlm-0.6.0/lib/net/ntlm/client/session.rb:192:in `ntlmv2_hash'
D      /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/rubyntlm-0.6.0/lib/net/ntlm/client/session.rb:196:in `calculate_user_session_key!'
D      /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/rubyntlm-0.6.0/lib/net/ntlm/client/session.rb:27:in `authenticate!'
D      /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/rubyntlm-0.6.0/lib/net/ntlm/client.rb:36:in `init_context'
D      /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/winrm-1.7.2/lib/winrm/http/transport.rb:228:in `init_auth'
D      /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/winrm-1.7.2/lib/winrm/http/transport.rb:166:in `send_request'
D      /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/winrm-1.7.2/lib/winrm/winrm_service.rb:489:in `send_message'
D      /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/winrm-1.7.2/lib/winrm/winrm_service.rb:390:in `run_wql'
D      /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/winrm-1.7.2/lib/winrm/command_executor.rb:171:in `os_version'
D      /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/winrm-1.7.2/lib/winrm/command_executor.rb:130:in `code_page'
D      /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/winrm-1.7.2/lib/winrm/command_executor.rb:72:in `block in open'
D      /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/winrm-1.7.2/lib/winrm/command_executor.rb:203:in `retryable'
D      /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/winrm-1.7.2/lib/winrm/command_executor.rb:71:in `open'
D      /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/winrm-1.7.2/lib/winrm/winrm_service.rb:356:in `create_executor'
D      /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.6.0/lib/kitchen/transport/winrm.rb:321:in `session'
D      /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.6.0/lib/kitchen/transport/winrm.rb:135:in `wait_until_ready'
D      /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/kitchen-ec2-1.0.0/lib/kitchen/driver/ec2.rb:205:in `create'
D      /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.6.0/lib/kitchen/instance.rb:449:in `public_send'
D      /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.6.0/lib/kitchen/instance.rb:449:in `block in perform_action'
D      /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.6.0/lib/kitchen/instance.rb:513:in `call'
D      /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.6.0/lib/kitchen/instance.rb:513:in `synchronize_or_call'
D      /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.6.0/lib/kitchen/instance.rb:478:in `block in action'
D      /opt/chefdk/embedded/lib/ruby/2.1.0/benchmark.rb:279:in `measure'
D      /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.6.0/lib/kitchen/instance.rb:477:in `action'
D      /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.6.0/lib/kitchen/instance.rb:449:in `perform_action'
D      /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.6.0/lib/kitchen/instance.rb:359:in `create_action'
D      /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.6.0/lib/kitchen/instance.rb:348:in `block in transition_to'
D      /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.6.0/lib/kitchen/instance.rb:347:in `each'
D      /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.6.0/lib/kitchen/instance.rb:347:in `transition_to'
D      /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.6.0/lib/kitchen/instance.rb:127:in `create'
D      /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.6.0/lib/kitchen/command.rb:176:in `public_send'
D      /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.6.0/lib/kitchen/command.rb:176:in `block (2 levels) in run_action'
D      /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/logging-2.1.0/lib/logging/diagnostic_context.rb:450:in `call'
D      /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/logging-2.1.0/lib/logging/diagnostic_context.rb:450:in `block in create_with_logging_context'
7

That stacktrace indicates that a nil password is sent to winrm

8

@Matt_Wrock I was using my key-par in kitchen.yml for authentication(it worked perfectly for other platforms) and i am not familiar if windows require a password to be configured in kitchen.yml(this was my first time,no idea on that) that can be sent to winrm. could we configure a random password for the username or is there a way to generate it randomly and pass it during the connection.Thanks

9

I have rarely used he ec2 driver but my understanding is that it automagically retrieves the amazon generated password for you.