How to specify multiple paths for remote_file resource

Chef Infra (archive)
1

I just want to download multiple files with different URL and place them in a folder I know we can provide an Array for remote_file resource but how we are going to specify multiple paths for each and every file that we download.

2

You probably need multiple remote_file resources. If you want to start with an Array of URIs, you can mangle them with ruby's uri library and maybe some of the file stuff. For example:

[1] pry(main)> u = "https://foo.example/path/to/one"
=> "https://foo.example/path/to/one"
[2] pry(main)> require 'uri'
=> true
[3] pry(main)> URI(u).path
=> "/path/to/one"
[4] pry(main)> File.basename(URI(u).path)
=> "one"
3

So it means that we need to manage it with ruby code only instead of Chef resources.

4

You can use ruby code to efficiently declare a lot of resources at once, when the resources are very similar. The canonical example of this used to be packages, e.g., ["one", "two"].each { |pkg_name| package pkg_name } but now there's the multipackage stuff that will optimize for doing less of the things your package manager is slow at. But the basic idea is still useful. In your case you could do:

["uri1", "uri2"].each do |uri|
  path = # code to get the path you want from the uri
  remote_file # options
end
5

You could do something like:

urllist = {
  { 'url': 'http://some.url/', 'path': '/some/path/', 'filename': 'some.file' },
  { 'url': 'https://someother.tld/', 'path': '/some/other/path', 'filename': 'another.file'}
}

urllist.each do |urlinfo|
  remote_file "#{urlinfo['path']}/#{urlinfo['filename']}" do
    source "#{urlinfo['url']}/#{urlinfo['filename']}"
    owner 'someowner'
    group 'somegroup'
    mode 0755
  end
end

You could probably put the filename in the url and use a variable with split to get just the filename, but if you want to customize the owners and groups you can also just add those attributes to the array and adjust the code inside the loop.

One of the nice things about Chef is that it’s very easy to take advantage of native Ruby functions while still leveraging the Chef DSL at the same time, like creating loops to create resources dynamically. I don’t know about other folks, but I do that a lot in a lot of my cookbooks.

6

Thanks for your response but now I got stuck with another problem. I have provided the remote_file with action create_if_missing but it is taking long time which seems it is downloading the file again but the file which I am trying to download is already available in that location. I can skip it with condition but I File.exists but I don’t want to use it. Could anyone suggest me something on this.

7

It would probably be best to supply the file checksum. See here: https://docs.chef.io/resource_remote_file.html#prevent-re-downloads

8

If you control the server that hosts the file, you should configure it to correctly handle conditional GET. On any relatively recent version of Chef Client, remote_file does conditional GET with a cache to reduce downloads.