Bootstrap Windows with Proxy - "Failed download: download completed, but downloaded file not found"

The tutorial I'm trying to follow

The command I'm running:

knife bootstrap windows winrm cheftest01 --winrm-user MYCREDS --winrm-passw
ord MYPASS --node-name "ChefTest01"

The error that comes out of ChefDK (my powershell window)

cheftest01 Attempting to download client package using PowerShell if available...
cheftest01 powershell.exe -ExecutionPolicy Unrestricted -InputFormat None -NoProfile -NonInteractive -File C:\chef\wget.ps1 "https://www.chef.io/chef/download?p=windows&pv=2008r2&m=x86_64&DownloadContext=PowerShell&v=12" "C:\Users\username\AppData\Local\Temp\chef-client-latest.msi"
cheftest01 Exception calling "DownloadFile" with "2" argument(s): "Unable to connect to the remote server"
cheftest01 At C:\chef\wget.ps1:14 char:1
cheftest01 + $webClient.DownloadFile($remoteUrl, $localPath);
cheftest01 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cheftest01 + CategoryInfo : NotSpecified: (:slight_smile: , MethodInvocationException
cheftest01 + FullyQualifiedErrorId : WebException
cheftest01
cheftest01 Failed download: download completed, but downloaded file not found
cheftest01 Warning: Failed to download "https://www.chef.io/chef/download?p=windows&pv=2008r2&m=x86_64&DownloadContext=PowerShell&v=12" to "C:\Users\username\AppData\Local\Temp\chef-client-latest.msi"
cheftest01 Warning: Retrying download with cscript ...
cheftest01 C:\chef\wget.vbs(47, 1) msxml3.dll: The operation timed out
cheftest01
cheftest01
cheftest01 Failed download: download completed, but downloaded file not found
cheftest01 Exiting without bootstrapping due to download failure.
ERROR: Failed to execute command on cheftest01 return code 1
ERROR: Bootstrap command returned 1


Soooooo I've been troubleshooting the past few days and I can confirm:

  1. If I browse to the site locally from the machine, I can download the file manually. I can install it manually as well but the script never seems to make it that far anyhow so I don't think its relevant.
  2. I was able to tweak wget.ps1 to get it to accept my credentials for the proxy and successfully run and download the file HOWEVER the stock version of wget.ps1 unsurprisingly fails with the same error if I login to the server and try to run it locally.

If you'd like to see the changes I had to make to wget.ps1 to get it to run I can provide it.

I'd be okay with modifying wget.ps1 and push that when I try to bootstrap instead but I can't seem to figure out how to do that or find any documentation on it.

For sure the problem is related to proxy authentication, but I'm not sure how to provide knife/chef with the proper info to get bootstrap to work correctly.

Help?

You can use the --bootstrap-proxy argument and set it to the proxy url which will cause wget.ps1 to use the proxy. However, if your proxy requires authentication, it looks like wget.ps1 does not account for that. Is that the change you had t omake to wget.ps1?

I had thought that was the case - sadly it looks like I simply overlooked that argument you provided. Trying it with the argument appears to have worked, so thank you! I spent far too much time fiddling around with that wget.ps1 file ughh.

I encountered a different error though now that we seem to have gotten past the proxy issue.

cheftest01 ================================================================================
cheftest01 Chef encountered an error attempting to create the client "ChefTest01"
cheftest01 ================================================================================
cheftest01
cheftest01 System Info:
cheftest01 ------------
cheftest01 chef_version=13.1.31
cheftest01 ruby=ruby 2.4.1p111 (2017-03-22 revision 58053) [x64-mingw32]
cheftest01 program_name=C:/opscode/chef/bin/chef-client
cheftest01 executable=C:/opscode/chef/bin/chef-client
cheftest01
cheftest01
cheftest01 Running handlers:
cheftest01 [2017-06-06T08:17:34-04:00] ERROR: Running exception handlers
cheftest01 Running handlers complete
cheftest01 [2017-06-06T08:17:34-04:00] ERROR: Exception handlers complete
cheftest01 Chef Client failed. 0 resources updated in 11 seconds
cheftest01 [2017-06-06T08:17:34-04:00] FATAL: Stacktrace dumped to c:/chef/cache/chef-stacktrace.out
cheftest01 [2017-06-06T08:17:34-04:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
cheftest01 [2017-06-06T08:17:34-04:00] FATAL: OpenSSL::PKey::RSAError: private key needed.
ERROR: Failed to execute command on cheftest01 return code 1
ERROR: Bootstrap command returned 1

I watched this video to help understand the bootstrap process better

I guess my question is

  1. Is the problem the fact that I don't have a client.pem file present?

and if so

  1. Why wasn't that file created during the bootstrap process? Did I miss something in the documentation/tutorial I've been following again? The only thing I can figure is I'm supposed to login to Chef Server first and "Create Client" then my node will recognize itself on the Chef Server and grab the proper client.pem? That is merely a guess though, I haven't seen that instruction anywhere in what I've been following.

Extra Credit - is there anyway to change the bootstrap process to provide a different wget.ps1 file, or some other modification, if I ever want to modify the process in the future?

A lack of a client.pem would not cause problems creating the client because the client must be created before a client.pem can be produced which is what a successful bootstrap would do. It will authenticate to the server using your key and create a client and key to be sent to the node. Common problems are that knife is unable to authenticate your own local user to the chef server or that your user lacks permissions to create new clients.

The private key needed error is likely indicative that knife cannot find your own user .pem. Do you have that available locally and does your knife.rb point to it?

There is no mechanism to alter the wget.ps1.

Interesting. I used the stuff here and was able to upload a cookbook

Wouldn't this below indicate that my workstation is communicating with Chef Server correctly?

PS C:\users\matt.keller\chef-repo> knife cookbook list
learn_chef_iis 0.1.0

So basically I was able to upload my cookbook and then query the server for the cookbook list.

Also, from the next module/step, I was able to successfully run this

PS C:\users\matt.keller\chef-repo> knife wsman test cheftest01 --manual-list
Connected successfully to cheftest01 at http://cheftest01:5985/wsman.

Finally, I checked permissions and made sure that my Chef user account is in the "admins" group and that group appears to have full access (read, update, delete, grant)

Would you suggest that I setup my workstation again from scratch or something? The environment thing seemed to be the only thing I struggled with but I thought it was working fine....

Yeah the fact that you can upload cookbooks certainly indicates that knife can find your key.

It may be helpful to see the stacktrace.out file mentioned in your error log:

cheftest01 [2017-06-06T08:17:34-04:00] FATAL: Stacktrace dumped to c:/chef/cache/chef-stacktrace.out
cheftest01 [2017-06-06T08:17:34-04:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report

Generated at 2017-06-06 15:07:29 -0400
OpenSSL::PKey::RSAError: private key needed.
C:/opscode/chef/embedded/lib/ruby/gems/2.4.0/gems/mixlib-authentication-1.4.1/lib/mixlib/authentication/signedheaderauth.rb:254:in private_encrypt' C:/opscode/chef/embedded/lib/ruby/gems/2.4.0/gems/mixlib-authentication-1.4.1/lib/mixlib/authentication/signedheaderauth.rb:254:in do_sign'
C:/opscode/chef/embedded/lib/ruby/gems/2.4.0/gems/mixlib-authentication-1.4.1/lib/mixlib/authentication/signedheaderauth.rb:111:in sign' C:/opscode/chef/embedded/lib/ruby/gems/2.4.0/gems/chef-13.1.31-universal-mingw32/lib/chef/http/auth_credentials.rb:51:in signature_headers'
C:/opscode/chef/embedded/lib/ruby/gems/2.4.0/gems/chef-13.1.31-universal-mingw32/lib/chef/http/authenticator.rb:111:in authentication_headers' C:/opscode/chef/embedded/lib/ruby/gems/2.4.0/gems/chef-13.1.31-universal-mingw32/lib/chef/http/authenticator.rb:50:in handle_request'
C:/opscode/chef/embedded/lib/ruby/gems/2.4.0/gems/chef-13.1.31-universal-mingw32/lib/chef/http.rb:331:in block in apply_request_middleware' C:/opscode/chef/embedded/lib/ruby/gems/2.4.0/gems/chef-13.1.31-universal-mingw32/lib/chef/http.rb:329:in each'
C:/opscode/chef/embedded/lib/ruby/gems/2.4.0/gems/chef-13.1.31-universal-mingw32/lib/chef/http.rb:329:in inject' C:/opscode/chef/embedded/lib/ruby/gems/2.4.0/gems/chef-13.1.31-universal-mingw32/lib/chef/http.rb:329:in apply_request_middleware'
C:/opscode/chef/embedded/lib/ruby/gems/2.4.0/gems/chef-13.1.31-universal-mingw32/lib/chef/http.rb:147:in request' C:/opscode/chef/embedded/lib/ruby/gems/2.4.0/gems/chef-13.1.31-universal-mingw32/lib/chef/http.rb:131:in post'
C:/opscode/chef/embedded/lib/ruby/gems/2.4.0/gems/chef-13.1.31-universal-mingw32/lib/chef/api_client/registration.rb:96:in create' C:/opscode/chef/embedded/lib/ruby/gems/2.4.0/gems/chef-13.1.31-universal-mingw32/lib/chef/api_client/registration.rb:87:in create_or_update'
C:/opscode/chef/embedded/lib/ruby/gems/2.4.0/gems/chef-13.1.31-universal-mingw32/lib/chef/api_client/registration.rb:58:in run' C:/opscode/chef/embedded/lib/ruby/gems/2.4.0/gems/chef-13.1.31-universal-mingw32/lib/chef/client.rb:666:in register'
C:/opscode/chef/embedded/lib/ruby/gems/2.4.0/gems/chef-13.1.31-universal-mingw32/lib/chef/client.rb:267:in run' C:/opscode/chef/embedded/lib/ruby/gems/2.4.0/gems/chef-13.1.31-universal-mingw32/lib/chef/application.rb:273:in run_with_graceful_exit_option'
C:/opscode/chef/embedded/lib/ruby/gems/2.4.0/gems/chef-13.1.31-universal-mingw32/lib/chef/application.rb:249:in block in run_chef_client' C:/opscode/chef/embedded/lib/ruby/gems/2.4.0/gems/chef-13.1.31-universal-mingw32/lib/chef/local_mode.rb:44:in with_server_connectivity'
C:/opscode/chef/embedded/lib/ruby/gems/2.4.0/gems/chef-13.1.31-universal-mingw32/lib/chef/application.rb:232:in run_chef_client' C:/opscode/chef/embedded/lib/ruby/gems/2.4.0/gems/chef-13.1.31-universal-mingw32/lib/chef/application/client.rb:464:in sleep_then_run_chef_client'
C:/opscode/chef/embedded/lib/ruby/gems/2.4.0/gems/chef-13.1.31-universal-mingw32/lib/chef/application/client.rb:451:in block in interval_run_chef_client' C:/opscode/chef/embedded/lib/ruby/gems/2.4.0/gems/chef-13.1.31-universal-mingw32/lib/chef/application/client.rb:450:in loop'
C:/opscode/chef/embedded/lib/ruby/gems/2.4.0/gems/chef-13.1.31-universal-mingw32/lib/chef/application/client.rb:450:in interval_run_chef_client' C:/opscode/chef/embedded/lib/ruby/gems/2.4.0/gems/chef-13.1.31-universal-mingw32/lib/chef/application/client.rb:434:in run_application'
C:/opscode/chef/embedded/lib/ruby/gems/2.4.0/gems/chef-13.1.31-universal-mingw32/lib/chef/application.rb:59:in run' C:/opscode/chef/embedded/lib/ruby/gems/2.4.0/gems/chef-13.1.31-universal-mingw32/bin/chef-client:26:in <top (required)>'
C:/opscode/chef/bin/chef-client:58:in load' C:/opscode/chef/bin/chef-client:58:in '

ok thats helpful and tells me the problem is indeed related to a missing client.pem on the node. It would be interesting to see if you can find a client and node object on the server for ChefTest01. One thing to try would be to delete those objects from the server and try again. If that still does not work then I would file an issue on github.

Delete the objects from… Chef Server? How? From Windows Server/my node? Where?

I attempted removing everything from the nodes (I have 2 that I’m testing on.)

Basically I deleted

C:\Chef
C:\opscode

and made sure that it was removed from Programs and Features

Is there somewhere else, perhaps in the registry or something that I need to remove as well?

If there isn’t, then I’m stumped because it still failed after removing everything.

If I wanted to submit a bug on GIT, where exactly would I do that? I’m sure there are multiple branches/forks… would it be for Knife specifically or?

My aplogies for not being clear. I would delete them from chef server. Deleting c:\opscode and c:\chef probably isn’t a bad idea either. That would bring you back to a clean slate.

https://github.com/chef/chef is the git repo where I would submit an issue.

Bummer.

I’m going to try having someone else setup a workstation and have them try to bootstrap a fresh 2012r2 install and see what happens but I have my doubts. I’m pretty bummed. I guess I can try manually installing for nodes but that will be too intensive for our environment long term.

Update:

I got this working by uninstalling/reinstalling ChefDK for my workstation. I do think the original problem was somehow rooted in the user.pem file. I was able to sort it out and finally bootstrap a server - however, the original intent of this post is still very much a question. I cannot get Chef/knife to bootstrap through our proxy.

I added a proxy exclusion for one of my test boxes and was able to get it to bootstrap just fine. Small victory I suppose. Bootstrapping with the --bootstrap-proxy though? Nope. It does 2 things

  1. It throws what I assume is the generic proxy authentication failure error

cheftest01 ================================================================================
cheftest01 Chef encountered an error attempting to load the node data for "cheftest01"
cheftest01 ================================================================================
cheftest01
cheftest01 Unexpected API Request Failure:
cheftest01 -------------------------------
cheftest01 Cannot fetch the contents of the response.
cheftest01
cheftest01 Platform:
cheftest01 ---------
cheftest01 x64-mingw32
cheftest01
cheftest01 [2017-06-09T10:18:12-04:00] ERROR: Running exception handlers
cheftest01 [2017-06-09T10:18:12-04:00] ERROR: Exception handlers complete
cheftest01 [2017-06-09T10:18:12-04:00] FATAL: Stacktrace dumped to c:/chef/cache/chef-stacktrace.out
cheftest01 [2017-06-09T10:18:12-04:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
cheftest01 [2017-06-09T10:18:12-04:00] FATAL: Net::HTTPServerException: 407 "Proxy Authentication Required"
DEBUG: [WinRM] cleaning up command_id: CD60B2C3-ABA7-4E97-8230-BB2DEC78276F on shell_id 12838DCF-6B26-4FF6-9AA8-7E009693A6F2
ERROR: Failed to execute command on cheftest01 return code 1
ERROR: Bootstrap command returned 1

  1. If I run the command a second time, the command/powershell window just hangs. It doesn't quit or end up not responding but the command just looks like its sitting there executing but it never finishes (I tried letting it run over night for example.) Even more strange, it will add the node to the Chef Server but it clearly isn't added correctly because Chef Server won't show the platform or IP Address or report the node as ever having checked in. So I end up just going in and deleting the node manually. I even tried running the command with -VV to try and get a little more info out of it but no dice.

Here is the stacktrace.out in case anyone has any ideas to handling the proxy authentication.

Generated at 2017-06-09 10:18:12 -0400
Net::HTTPServerException: 407 "Proxy Authentication Required"
C:/opscode/chef/embedded/lib/ruby/2.3.0/net/http/response.rb:120:in error!' C:/opscode/chef/embedded/lib/ruby/2.3.0/net/http/response.rb:129:in value'
C:/opscode/chef/embedded/lib/ruby/2.3.0/net/http.rb:920:in connect' C:/opscode/chef/embedded/lib/ruby/2.3.0/net/http.rb:863:in do_start'
C:/opscode/chef/embedded/lib/ruby/2.3.0/net/http.rb:852:in start' C:/opscode/chef/embedded/lib/ruby/2.3.0/net/http.rb:1398:in request'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.20.3-universal-mingw32/lib/chef/http/basic_client.rb:70:in request' C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.20.3-universal-mingw32/lib/chef/http.rb:340:in block in send_http_request'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.20.3-universal-mingw32/lib/chef/http.rb:372:in block in retrying_http_errors' C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.20.3-universal-mingw32/lib/chef/http.rb:370:in loop'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.20.3-universal-mingw32/lib/chef/http.rb:370:in retrying_http_errors' C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.20.3-universal-mingw32/lib/chef/http.rb:333:in send_http_request'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.20.3-universal-mingw32/lib/chef/http.rb:148:in request' C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.20.3-universal-mingw32/lib/chef/http.rb:115:in get'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.20.3-universal-mingw32/lib/chef/node.rb:581:in load' C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.20.3-universal-mingw32/lib/chef/node.rb:565:in find_or_create'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.20.3-universal-mingw32/lib/chef/policy_builder/dynamic.rb:72:in load_node' C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.20.3-universal-mingw32/lib/chef/client.rb:470:in load_node'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.20.3-universal-mingw32/lib/chef/client.rb:270:in run' C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.20.3-universal-mingw32/lib/chef/application.rb:277:in run_with_graceful_exit_option'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.20.3-universal-mingw32/lib/chef/application.rb:253:in block in run_chef_client' C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.20.3-universal-mingw32/lib/chef/local_mode.rb:44:in with_server_connectivity'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.20.3-universal-mingw32/lib/chef/application.rb:236:in run_chef_client' C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.20.3-universal-mingw32/lib/chef/application/client.rb:464:in sleep_then_run_chef_client'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.20.3-universal-mingw32/lib/chef/application/client.rb:451:in block in interval_run_chef_client' C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.20.3-universal-mingw32/lib/chef/application/client.rb:450:in loop'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.20.3-universal-mingw32/lib/chef/application/client.rb:450:in interval_run_chef_client' C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.20.3-universal-mingw32/lib/chef/application/client.rb:434:in run_application'
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.20.3-universal-mingw32/lib/chef/application.rb:59:in run' C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.20.3-universal-mingw32/bin/chef-client:26:in <top (required)>'
C:/opscode/chef/bin/chef-client:68:in load' C:/opscode/chef/bin/chef-client:68:in '

Update again, for those following along or who encounter this in the future

ERROR: No response received from remote node after 2.37 minutes, giving up.
ERROR: Exception: Unable to parse authorization header. Headers: {"Proxy-Authenticate"=>"NTLM", "Cache-Control"=>"no-cache", "Pragma"=>"no-cache", "Content-Type"=>"text/html; charset=utf-8", "Proxy-Connection"=>"close", "Set-Cookie"=>"BCSI-CS-477914e091836061=2; Path=/", "Connection"=>"close", "Content-Length"=>"1106"}
Body:

Access Denied

Your credentials could not be authenticated: "General authentication failure due to bad user ID or authentication token." You will not be permitted access until your credentials can be verified.

This is typically caused by an incorrect username and/or password, but could also be caused by network problems.

If you have any questions please contact desktopsupport@blah.com

That looks like I'm reaching the proxy but the authentication is just failing so I think I'm close. I found this page - Working with Proxies - which makes reference to updating your knife.rb to include these entries

http_proxy          myproxyip:proxyport
http_proxy_pass     mywindowscredsforproxy
http_proxy_user     mywindowspass

Our proxy uses NTLM I think so its just a matter of formatting at this point. I'm working with our networking team that manages the proxy but I think once I get it added to knife.rb correctly, we should be in business. Once I get it working I plan on recapping the exact changes here for everyone's sake in the future.