Kitchen test fails after Domain Controller install

haug · September 15, 2022, 5:34am

Hi,
When running kitchen converge on a cookbook that install windows active directory services so the node becomes a domain controller we get an
"Failed to complete #converge action: [WinRM::WinRMAuthorizationError]"

The recipe installs the domain controller functionality, sets two registry keys and then a reboot (with a 1 minute delay).
From the log file after setting the registry keys:
D, [2022-09-15T06:30:15.223863 #18966] DEBUG -- dc1-windows2019: [WinRM] Waiting for output...
D, [2022-09-15T06:31:05.271217 #18966] DEBUG -- dc1-windows2019: [WinRM] cleaning up command_id: D109ADD0-266F-4AAD-B918-2E67A129A73C on shell_id 59ADB9C4-BAEC-40BA-8B8F-93E1D1F9B3E3
D, [2022-09-15T06:32:19.231680 #18966] DEBUG -- dc1-windows2019: Cleaning up local sandbox in /tmp/dc1-windows2019-sandbox-20220915-18966-vrmr2s
E, [2022-09-15T06:32:19.238861 #18966] ERROR -- dc1-windows2019: Converge failed on instance .
E, [2022-09-15T06:32:19.239016 #18966] ERROR -- dc1-windows2019: ------Exception-------
E, [2022-09-15T06:32:19.239036 #18966] ERROR -- dc1-windows2019: Class: WinRM::WinRMAuthorizationError
E, [2022-09-15T06:32:19.239051 #18966] ERROR -- dc1-windows2019: Message: WinRM::WinRMAuthorizationError
E, [2022-09-15T06:32:19.239061 #18966] ERROR -- dc1-windows2019: ----------------------
E, [2022-09-15T06:32:19.239071 #18966] ERROR -- dc1-windows2019: ------Backtrace-------
E, [2022-09-15T06:32:19.239081 #18966] ERROR -- dc1-windows2019: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/winrm-2.3.6/lib/winrm/http/response_handler.rb:59:in raise_if_auth_error' E, [2022-09-15T06:32:19.239092 #18966] ERROR -- dc1-windows2019: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/winrm-2.3.6/lib/winrm/http/response_handler.rb:51:in raise_if_error'
E, [2022-09-15T06:32:19.239132 #18966] ERROR -- dc1-windows2019: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/winrm-2.3.6/lib/winrm/http/response_handler.rb:34:in parse_to_xml' E, [2022-09-15T06:32:19.239157 #18966] ERROR -- dc1-windows2019: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/winrm-2.3.6/lib/winrm/http/transport.rb:186:in send_request'
E, [2022-09-15T06:32:19.239168 #18966] ERROR -- dc1-windows2019: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/winrm-2.3.6/lib/winrm/shells/power_shell.rb:42:in close_shell' E, [2022-09-15T06:32:19.239178 #18966] ERROR -- dc1-windows2019: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/winrm-2.3.6/lib/winrm/shells/base.rb:89:in close'
E, [2022-09-15T06:32:19.239188 #18966] ERROR -- dc1-windows2019: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/test-kitchen-2.11.2/lib/kitchen/transport/winrm.rb:94:in close' E, [2022-09-15T06:32:19.239198 #18966] ERROR -- dc1-windows2019: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/test-kitchen-2.11.2/lib/kitchen/transport/winrm.rb:262:in ensure in execute_with_exit_code'
E, [2022-09-15T06:32:19.239208 #18966] ERROR -- dc1-windows2019: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/test-kitchen-2.11.2/lib/kitchen/transport/winrm.rb:262:in execute_with_exit_code' E, [2022-09-15T06:32:19.239218 #18966] ERROR -- dc1-windows2019: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/test-kitchen-2.11.2/lib/kitchen/transport/winrm.rb:108:in execute'
E, [2022-09-15T06:32:19.239228 #18966] ERROR -- dc1-windows2019: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/test-kitchen-2.11.2/lib/kitchen/transport/base.rb:130:in execute_with_retry' E, [2022-09-15T06:32:19.239239 #18966] ERROR -- dc1-windows2019: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/test-kitchen-2.11.2/lib/kitchen/provisioner/base.rb:86:in block in call'
E, [2022-09-15T06:32:19.239249 #18966] ERROR -- dc1-windows2019: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/test-kitchen-2.11.2/lib/kitchen/transport/base.rb:100:in initialize' E, [2022-09-15T06:32:19.239260 #18966] ERROR -- dc1-windows2019: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/test-kitchen-2.11.2/lib/kitchen/transport/winrm.rb:87:in initialize'
E, [2022-09-15T06:32:19.239270 #18966] ERROR -- dc1-windows2019: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/test-kitchen-2.11.2/lib/kitchen/transport/winrm.rb:479:in new' E, [2022-09-15T06:32:19.239280 #18966] ERROR -- dc1-windows2019: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/test-kitchen-2.11.2/lib/kitchen/transport/winrm.rb:479:in create_new_connection'
E, [2022-09-15T06:32:19.239290 #18966] ERROR -- dc1-windows2019: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/test-kitchen-2.11.2/lib/kitchen/transport/winrm.rb:74:in connection' E, [2022-09-15T06:32:19.239299 #18966] ERROR -- dc1-windows2019: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/test-kitchen-2.11.2/lib/kitchen/provisioner/base.rb:75:in call'
E, [2022-09-15T06:32:19.239309 #18966] ERROR -- dc1-windows2019: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/test-kitchen-2.11.2/lib/kitchen/instance.rb:419:in block in converge_action' E, [2022-09-15T06:32:19.239319 #18966] ERROR -- dc1-windows2019: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/test-kitchen-2.11.2/lib/kitchen/instance.rb:563:in synchronize_or_call'
E, [2022-09-15T06:32:19.239329 #18966] ERROR -- dc1-windows2019: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/test-kitchen-2.11.2/lib/kitchen/instance.rb:524:in block in action' E, [2022-09-15T06:32:19.239340 #18966] ERROR -- dc1-windows2019: /opt/chef-workstation/embedded/lib/ruby/2.7.0/benchmark.rb:293:in measure'
E, [2022-09-15T06:32:19.239350 #18966] ERROR -- dc1-windows2019: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/test-kitchen-2.11.2/lib/kitchen/instance.rb:523:in action' E, [2022-09-15T06:32:19.239360 #18966] ERROR -- dc1-windows2019: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/test-kitchen-2.11.2/lib/kitchen/instance.rb:414:in converge_action'
E, [2022-09-15T06:32:19.239370 #18966] ERROR -- dc1-windows2019: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/test-kitchen-2.11.2/lib/kitchen/instance.rb:392:in block (2 levels) in transition_to' E, [2022-09-15T06:32:19.239384 #18966] ERROR -- dc1-windows2019: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/test-kitchen-2.11.2/lib/kitchen/lifecycle_hooks.rb:47:in run_with_hooks'
E, [2022-09-15T06:32:19.239396 #18966] ERROR -- dc1-windows2019: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/test-kitchen-2.11.2/lib/kitchen/instance.rb:391:in block in transition_to' E, [2022-09-15T06:32:19.239406 #18966] ERROR -- dc1-windows2019: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/test-kitchen-2.11.2/lib/kitchen/instance.rb:390:in each'
E, [2022-09-15T06:32:19.239416 #18966] ERROR -- dc1-windows2019: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/test-kitchen-2.11.2/lib/kitchen/instance.rb:390:in transition_to' E, [2022-09-15T06:32:19.239427 #18966] ERROR -- dc1-windows2019: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/test-kitchen-2.11.2/lib/kitchen/instance.rb:139:in converge'
E, [2022-09-15T06:32:19.239436 #18966] ERROR -- dc1-windows2019: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/test-kitchen-2.11.2/lib/kitchen/command.rb:195:in public_send' E, [2022-09-15T06:32:19.239446 #18966] ERROR -- dc1-windows2019: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/test-kitchen-2.11.2/lib/kitchen/command.rb:195:in run_action_in_thread'
E, [2022-09-15T06:32:19.239457 #18966] ERROR -- dc1-windows2019: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/test-kitchen-2.11.2/lib/kitchen/command.rb:166:in block (2 levels) in run_action' E, [2022-09-15T06:32:19.239466 #18966] ERROR -- dc1-windows2019: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in block in create_with_logging_context'
E, [2022-09-15T06:32:19.239476 #18966] ERROR -- dc1-windows2019: ----End Backtrace-----

We don't get any information that the node is going to be rebooted.

On the node I can see that it reboots approx. 1 minute later and when it is starting up it does all of the final settings for the domain controller installation.
If I run the kitchen converge immediately after the error it runs the converge just fine, so the username/password used still works.

Anyone seen this before?

Best regards

byteflyer · September 19, 2022, 1:10pm

Do you have the following in your kitchen.yml file in the provisioner section?

provisioner:
  ...
  retry_on_exit_code:
    - 35
    - 213
  wait_for_retry: 45
  max_retries: 6

These exit codes are used on windows to signal a reboot, and should allow TK to pick up and re-run chef after the reboot.

haug · September 19, 2022, 1:16pm

I will test with 213 (never seen that one before)
The other two options we have.
I will get back once I have tested

haug · September 19, 2022, 1:28pm

Still the same WinRM::WinRMAuthorizationError when the server is responding again, and no log output about the server restart that we initiate in the recipe.
It seems that kitchen (or winrm) is not able to reauthenticate after the server becomes a domain controller.
If I run kitchen converge right after the error message it runs fine.

Topic		Replies	Views
Machine_batch fails with WinRM Authorization Error Chef Infra (archive)	2	1274	October 21, 2016
Issues with kitchen converge using windows boxs Chef Infra (archive)	22	6218	June 14, 2016
[RESOLVED] Chef workstation failed to connect Chef Client Chef Infra (archive)	1	1509	August 27, 2018
Kitchen converge Windows 2012r2 Chef Infra (archive)	5	2074	February 2, 2017
Getting error while executing kitchen converge Chef Infra (archive)	0	744	December 22, 2020

Kitchen test fails after Domain Controller install

Related topics