Ohai,
I’ve recently seen some corrupted remote_file downloads and was wondering
how people are handling this?
Afaik, the checksum property is only used to determine if a download should
occur, not verify when the download is complete?
Would be nice with a simple way to tell remote_file to verify the data.
/Jeppe
Yes, I think this is currently a shortcoming in Chef.
https://tickets.opscode.com/browse/CHEF-4502
https://tickets.opscode.com/browse/CHEF-4647
On Fri, Jan 31, 2014 at 6:56 AM, Jeppe Nejsum Madsen jeppe@ingolfs.dk wrote:
Ohai,
I've recently seen some corrupted remote_file downloads and was wondering
how people are handling this?Afaik, the checksum property is only used to determine if a download should
occur, not verify when the download is complete?Would be nice with a simple way to tell remote_file to verify the data.
/Jeppe
--
[ Julian C. Dunn jdunn@aquezada.com * Sorry, I'm ]
[ WWW: http://www.aquezada.com/staff/julian * only Web 1.0 ]
[ gopher://sdf.org/1/users/keymaker/ * compliant! ]
[ PGP: 91B3 7A9D 683C 7C16 715F 442C 6065 D533 FDC2 05B9 ]
I had this issue in the past and wrote up a workaround:
On Fri, Jan 31, 2014 at 10:45 AM, Julian C. Dunn jdunn@aquezada.com wrote:
Yes, I think this is currently a shortcoming in Chef.
https://tickets.opscode.com/browse/CHEF-4502
https://tickets.opscode.com/browse/CHEF-4647
- Julian
On Fri, Jan 31, 2014 at 6:56 AM, Jeppe Nejsum Madsen jeppe@ingolfs.dk
wrote:Ohai,
I've recently seen some corrupted remote_file downloads and was wondering
how people are handling this?Afaik, the checksum property is only used to determine if a download
should
occur, not verify when the download is complete?Would be nice with a simple way to tell remote_file to verify the data.
/Jeppe
--
[ Julian C. Dunn jdunn@aquezada.com * Sorry, I'm ]
[ WWW: http://www.aquezada.com/staff/julian * only Web 1.0 ]
[ gopher://sdf.org/1/users/keymaker/ * compliant! ]
[ PGP: 91B3 7A9D 683C 7C16 715F 442C 6065 D533 FDC2 05B9 ]
So CHEF-4502 just got merged to master last week and will be released
with 11.10.0 and was in the 11.10.0.rc.0 released yesterday (I just
updated the ticket correctly which was mistakenly listing it as 11.12.0).
That should mitigate at least a lot of the truncated downloads problem
(I think that it may mitigate all truncated downloads in the
remote_file resource
since we should be doing HTTP/1.1 and not using chunked encoding or
anything like that which would not have a Content-Length header, but i'll
admit that's getting into the weeds of HTTP further than my knowledge
goes). There's also still the possibility of corrupted bit flipping in
the stream which a checksum would address (or the file being transferred
into RAM correctly but then having RAM->disk corruption or whatever due
to crappy RAM/MB/chipset/etc), but fixing CHEF-4502 should knock down
the probability of a corrupted download by an order of magnitude or so.
On 1/31/14 7:45 AM, Julian C. Dunn wrote:
Yes, I think this is currently a shortcoming in Chef.
https://tickets.opscode.com/browse/CHEF-4502
https://tickets.opscode.com/browse/CHEF-4647
- Julian
On Fri, Jan 31, 2014 at 6:56 AM, Jeppe Nejsum Madsen jeppe@ingolfs.dk wrote:
Ohai,
I've recently seen some corrupted remote_file downloads and was wondering
how people are handling this?Afaik, the checksum property is only used to determine if a download should
occur, not verify when the download is complete?Would be nice with a simple way to tell remote_file to verify the data.
/Jeppe
Lamont Granquist lamont@opscode.com writes:
So CHEF-4502 just got merged to master last week and will be released
with 11.10.0 and was in the 11.10.0.rc.0 released yesterday (I just
updated the ticket correctly which was mistakenly listing it as
11.12.0).That should mitigate at least a lot of the truncated downloads problem
(I think that it may mitigate all truncated downloads in the
remote_file resource
since we should be doing HTTP/1.1 and not using chunked encoding or
anything like that which would not have a Content-Length header, but
i'll
admit that's getting into the weeds of HTTP further than my knowledge
goes). There's also still the possibility of corrupted bit flipping
in the stream which a checksum would address (or the file being
transferred into RAM correctly but then having RAM->disk corruption or
whatever due to crappy RAM/MB/chipset/etc), but fixing CHEF-4502
should knock down the probability of a corrupted download by an order
of magnitude or so.
Thanks for the update!
Seems like it will solve the problems we've seen.
/Jeppe