Adam,
I am definitely setting the winrm settings as specified in the docs. I have
tried increasing the “MaxMemoryPerShellMb” without any success, but not
changed the “MaxTimeoutMs” beyond what the documentation recommends. I will
try to look into that soon but right now I do not have time to go further
with this.
Cheers,
Florian
From: Adam Edwards [mailto:adamed@opscode.com]
Sent: 07 August 2013 16:11
To: chef@lists.opscode.com
Subject: [chef] RE: RE: Re: better windows client download script
I think what is happening is that you haven’t set the configuration to
increase the quota for process memory (default is 150mb, because of the
"let’s ready everything into memory and copy it" behavior I described below
that will generally be exceeded) and the config to increase the timeout for
command execution. See the docs below:
http://docs.opscode.com/plugin_knife_windows.html
The section on “MaxMemoryPerShellMb” and “MaxTimeoutMs” are the settings
involved here. You’ll need to increase those to get things to work in many
cases. The first setting is an absolute requirement related to the size of
the MSI being downloaded. The second value is needed depending on your
bandwidth - by the time the bootstrap has hit that, it has already
downloaded the file once (and then failed trying to make an in-memory copy),
then it retries with powershell, which downloads it again. So downloading a
70mb file on some connections may take more than the default timeout for the
command to complete, thus causing the failure.
Please let us know if configuring those settings makes the default
knife-windows template succeed for you.
What’s interesting to me is that you’ve found a purely vbscript workaround
to avoid the retried download - I spent a lot of time trying to develop a
vbscript version but every approach I examined would end up creating a
second in memory copy even if you tried to write directly to disk. It looks
like the “ADODB.Stream” object might be the key here - the first part of the
script is still inefficient and reads the entire file in to memory, but
that’s still within the 150mb limit, the ADODB.Stream might be smart enough
to write chunks the way that the .Net webclient in the powershell approach
does it. You can validate that it’s doing this if you monitor peak memory
usage of the script - if it never goes above, say 100mb, then that would be
consistent.
Assuming the ADODB.stream object is available out of the box on all the
versions of Windows where Chef runs (down to 2003 R2), it would be worth
submitting a patch if you’ve filled out a Contributor License Agreement -
while the PowerShell approach is more efficient and concise, it only works
on 2k8R2 and above, and currently only gets invoked after you’ve already
wasted a download. If your script works on all systems the first time, we’d
save bandwidth and time.
Thanks.
-Adam
From: Florian Hehlen [mailto:florian.hehlen@gmail.com]
Sent: Wednesday, August 7, 2013 1:22 AM
To: chef@lists.opscode.com
Subject: [chef] RE: Re: better windows client download script
Adam,
My issue is that both the vbs and powershell scripts fail on some machines.
The vbs script fails quickly with a very obscure error(can’t find a log of
it at moment) while the powershell script runs for a long time and then
fails quietly.
I have been encountering this issue on some machines running win2k8R2 as
well as on some win7 workstations.
I have to admit that I am not a windows or powershell expert. Can you expand
on how to change the memory settings that you recommend? Or maybe you can
point to some documentation.
Cheers,
Florian
From: Adam Edwards [mailto:adamed@opscode.com]
Sent: 06 August 2013 19:33
To: chef@lists.opscode.com
Subject: [chef] Re: better windows client download script
Florian, can you tell us more about the specific issues that this script
solves for you?
A challenge with using default objects in Windows from vbscript to do the
download has been their memory usage - they tend to read the entire msi
package into memory, then make another copy in memory, and write it to disk.
When running under WinRM, this fails if you don’t increase shell memory
quotas (and on Win2k12, there is a bug in Windows itself where overriding
this value does not work :)).
The best way to do this that I’ve found is to use powershell and access the
.net WebClient class - like wget or curl, it streams the download in chunks
and writes chunks to disk so that memory utilization is low and you never
hit these limits. Unfortunately, powershell is only available by default on
win2k8r2 and above, so if you have to work on other systems, this isn’t a
good option. However, this is how we worked around the WinRM shell
limitation bug on Win2k12.
-Adam
From: Florian Hehlen florian.hehlen@gmail.com
Reply-To: "chef@lists.opscode.com" chef@lists.opscode.com
Date: Tuesday, August 6, 2013 10:25 AM
To: "chef@lists.opscode.com" chef@lists.opscode.com
Subject: [chef] better windows client download script
Hi all,
I just wanted to share a vb script that I have found that seems to be better
than the default one provided with the chef client. I have been struggling
for a while now with the script that fails on some machines and not others.
I admit the windows setup is not ideal where I am but it is likely that I am
not the one.
The point is that this script tries a few different approaches before giving
up. I have not fully appreciated how it does it. I was mostly happy that it
works!
I found the script here:
http://www.ericphelps.com/scripting/samples/BinaryDownload/
http://www.ericphelps.com/scripting/samples/BinaryDownload/
Below is the drop-in replacement script that I slightly adapted to work with
the chef code in
knife-windows-0.5.12\lib\chef\knife\core\windows_bootstrap_context.rb
strUrl = WScript.Arguments.Named(“url”)
strFile = WScript.Arguments.Named(“path”)
Const adTypeBinary = 1
Const adSaveCreateOverWrite = 2
Const ForWriting = 2
Dim web, varByteArray, strData, strBuffer, lngCounter, ado
On Error Resume Next
'Download the file with any available object
Err.Clear
Set web = Nothing
Set web = CreateObject("WinHttp.WinHttpRequest.5.1")
If web Is Nothing Then Set web = CreateObject("WinHttp.WinHttpRequest")
If web Is Nothing Then Set web = CreateObject("MSXML2.ServerXMLHTTP")
If web Is Nothing Then Set web = CreateObject("Microsoft.XMLHTTP")
web.Open "GET", strURL, False
web.Send
If Err.Number <> 0 Then
SaveWebBinary = False
Set web = Nothing
Wscript.Quit
End If
If web.Status <> "200" Then
SaveWebBinary = False
Set web = Nothing
Wscript.Quit
End If
varByteArray = web.ResponseBody
Set web = Nothing
'Now save the file with any available method
On Error Resume Next
Set ado = Nothing
Set ado = CreateObject("ADODB.Stream")
If ado Is Nothing Then
Set fs = CreateObject("Scripting.FileSystemObject")
Set ts = fs.OpenTextFile(strFile, ForWriting, True)
strData = ""
strBuffer = ""
For lngCounter = 0 to UBound(varByteArray)
ts.Write Chr(255 And Ascb(Midb(varByteArray,lngCounter + 1, 1)))
Next
ts.Close
Else
ado.Type = adTypeBinary
ado.Open
ado.Write varByteArray
ado.SaveToFile strFile, adSaveCreateOverWrite
ado.Close
End If
SaveWebBinary = True