Machine_batch fails with WinRM Authorization Error

Chef Infra (archive)
1

Hello All,

So I am facing this error when converging Windows 2012 R2 VMs using the machine_batch resource on AWS. Everything works as expected. The only thing is that my first recipe has a reboot in the end of it's execution and as soon as that step completes, I see the Error 35 Reboot_Scheduled and some information around it with RFC 062. Then I tried to modify my chef recipe for machine_batch to ignore error code 35 with option ignore_failure: [35] under convergence options but I still see the same error.

The machines reboot and then again start execution as I create a scheduled task in my first recipe to perform chef runs on reboot. But since the convergence on machine_batch fails on the provisioning node, it fails to create the Load Balancer which is defined after the machine_batch resource.

One more thing to note here is that I do change the local "Administrator" user name to something else in my first recipe and I use the same "Administrator" user name and its password under machine_options. But if this was the error then I should see "Waiting for ... to become connectible" loop that it does when password or user name is wrong?

I am not able to get around this problem. Anybody have seen such a thing or any clue what can be done here?

Any sort of help would be appreciated!

Please see the stacktrace output below:
Generated at 2016-10-19 00:02:56 +0000
Chef::Provider::MachineBatch::MachineBatchError: machine_batch[default] (server_stack::server_stack_dedicated line 100) had an error: Chef::Provider::MachineBatch::MachineBatchError: WinRM::WinRMAuthorizationError
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-2.0.1/lib/chef/provider/machine_batch.rb:114:in rescue in block (2 levels) in with_ready_machines' C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-2.0.1/lib/chef/provider/machine_batch.rb:117:in block (2 levels) in with_ready_machines'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-2.0.1/lib/chef/provisioning/driver.rb:251:in block in ready_machines' C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-12.14.89-universal-mingw32/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:263:in process_input'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-12.14.89-universal-mingw32/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:253:in process_one' C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-12.14.89-universal-mingw32/lib/chef/chef_fs/parallelizer.rb:93:in call'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-12.14.89-universal-mingw32/lib/chef/chef_fs/parallelizer.rb:93:in `worker_loop'

Caused by WinRM::WinRMAuthorizationError: WinRM::WinRMAuthorizationError
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/winrm-2.0.2/lib/winrm/http/response_handler.rb:58:in raise_if_auth_error' C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/winrm-2.0.2/lib/winrm/http/response_handler.rb:51:in raise_if_error'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/winrm-2.0.2/lib/winrm/http/response_handler.rb:36:in parse_to_xml' C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/winrm-2.0.2/lib/winrm/http/transport.rb:54:in send_request'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/winrm-2.0.2/lib/winrm/shells/base.rb:146:in cleanup_command' C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/winrm-2.0.2/lib/winrm/shells/base.rb:128:in with_command_shell'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/winrm-2.0.2/lib/winrm/shells/base.rb:75:in run' C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-2.0.1/lib/chef/provisioning/transport/winrm.rb:42:in block in execute'
C:/opscode/chefdk/embedded/lib/ruby/2.3.0/timeout.rb:91:in block in timeout' C:/opscode/chefdk/embedded/lib/ruby/2.3.0/timeout.rb:33:in block in catch'
C:/opscode/chefdk/embedded/lib/ruby/2.3.0/timeout.rb:33:in catch' C:/opscode/chefdk/embedded/lib/ruby/2.3.0/timeout.rb:33:in catch'
C:/opscode/chefdk/embedded/lib/ruby/2.3.0/timeout.rb:106:in timeout' C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-2.0.1/lib/chef/provisioning/transport.rb:92:in with_execute_timeout'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-2.0.1/lib/chef/provisioning/transport/winrm.rb:40:in execute' C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-2.0.1/lib/chef/provisioning/machine/basic_machine.rb:30:in block in execute'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-12.14.89-universal-mingw32/lib/chef/mixin/why_run.rb:52:in add_action' C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-12.14.89-universal-mingw32/lib/chef/provider.rb:176:in converge_by'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-2.0.1/lib/chef/provisioning/chef_provider_action_handler.rb:54:in perform_action' C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-2.0.1/lib/chef/provisioning/add_prefix_action_handler.rb:31:in perform_action'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-2.0.1/lib/chef/provisioning/machine/basic_machine.rb:29:in execute' C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-2.0.1/lib/chef/provisioning/convergence_strategy/install_msi.rb:60:in block (2 levels) in converge'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-12.14.89-universal-mingw32/lib/chef/run_context.rb:522:in open_stream' C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-2.0.1/lib/chef/provisioning/chef_provider_action_handler.rb:59:in open_stream'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-2.0.1/lib/chef/provisioning/convergence_strategy/install_msi.rb:55:in block in converge' C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-12.14.89-universal-mingw32/lib/chef/run_context.rb:522:in open_stream'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-2.0.1/lib/chef/provisioning/chef_provider_action_handler.rb:59:in open_stream' C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-2.0.1/lib/chef/provisioning/convergence_strategy/install_msi.rb:54:in converge'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-2.0.1/lib/chef/provisioning/convergence_strategy/ignore_convergence_failure.rb:39:in converge' C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-2.0.1/lib/chef/provisioning/machine/basic_machine.rb:21:in converge'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-2.0.1/lib/chef/provider/machine_batch.rb:61:in block (2 levels) in <class:MachineBatch>' C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-2.0.1/lib/chef/provider/machine_batch.rb:111:in block (2 levels) in with_ready_machines'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-provisioning-2.0.1/lib/chef/provisioning/driver.rb:251:in block in ready_machines' C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-12.14.89-universal-mingw32/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:263:in process_input'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-12.14.89-universal-mingw32/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:253:in process_one' C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-12.14.89-universal-mingw32/lib/chef/chef_fs/parallelizer.rb:93:in call'
C:/opscode/chefdk/embedded/lib/ruby/gems/2.3.0/gems/chef-12.14.89-universal-mingw32/lib/chef/chef_fs/parallelizer.rb:93:in `worker_loop'

2

This looks to me like a byproduct of the user change. The stack trace shows the WinRMAuthorizationError coming from the winrm cleanup_command. Winrm must send a command termination message after a command completes. The command was likely the user change and then when winrm cleans that up, winrm returns a 401 error code and since the user change invalidated the current auth token.

You’d either want to ensure your base images have this user already set or have your recipie create a scheduled task to perform the change after the run completes.

1 Like
3

Yes that worked!. Thanks man!