Remote file considered harmful


#1

I’d like to examine the division between the “file”, “remote_file” and
"tempate" resources.

First of all, they are badly named. Why isn’t there a remote_template to
correspond with remote_file? What’s “remote” about remote_file in Chef Solo?
Can we at least rename the little-used “file” to be “file_touch” so we can use
"file" instead of “remote_file” everywhere?

Second, the 3 types are similar in my mind because the ‘flow’ of data is 99.9%
the same: “I need a file to exist on disk right HERE (with or without an ERB
step)”.

My current workflow looks like this: I start off by copying a config file into
"/files" and create a “remote_file” resource. But then I discover I need an
attribute in the config file, so I have to move the config file (from “/files"
to “/templates”) and change the resource type (from “remote_file” to
"template”) and sometimes edit the name (from “config” to “config.erb”).
Sometimes I break up the config file to sub-configs. But these go into
different directories depending on if they need ERB attributes or not.
Logically, the sub-configs should be siblings, but physically they go in
different directories. Sometimes I find myself moving the files back and forth
as I refactor the config file and sub-config files and move attributes around.

It would make more sense to set an attribute (“oh, by the way, that file needs
ERB now”) on the existing resource. I also think it’s better if I don’t have to
move the file from /files to /templates.

Imagine merging these resources into one “file” resource:

touch a file

file “/tmp/touch_me” do
action :touch
end

copy file

file “/tmp/blob” do

source “blob” is implicit. Looks in /files/default/blob

end

Some files are so small, it’s silly to create a

new file in a new directory just to hold the contents

file “/tmp/blob” do
contents "dude"
end

Some generated files don’t benifit from ERB.

file “database.yml” do
contents do @node.database_settings.to_yaml; end
end

switching from a static file to a template

shouldn’t require a new resource.

file “/tmp/blob” do
filter :erb # process source file with ERB

Not sure if it should default to source ‘blob’ or ‘blob.erb’

also not sure if it should look in /templates or just in /files

end

I know it’s a major change, but hey, now is the time for big changes, right?
What do you think?

-=Dan=-


#2

I strongly agree with everything here. This sounds like a huge
improvement to me. That would also open up the remote_file resource
to actually refer to a remote file on a different server, which could
be downloaded or updated as necessary.

  • Ian

On Oct 21, 2009, at 1:40 PM, dan@animoto.com wrote:

I’d like to examine the division between the “file”, “remote_file” and
"tempate" resources.

First of all, they are badly named. Why isn’t there a
remote_template to
correspond with remote_file? What’s “remote” about remote_file in
Chef Solo?
Can we at least rename the little-used “file” to be “file_touch” so
we can use
"file" instead of “remote_file” everywhere?

Second, the 3 types are similar in my mind because the ‘flow’ of
data is 99.9%
the same: “I need a file to exist on disk right HERE (with or
without an ERB
step)”.

My current workflow looks like this: I start off by copying a config
file into
"/files" and create a “remote_file” resource. But then I discover I
need an
attribute in the config file, so I have to move the config file
(from “/files"
to “/templates”) and change the resource type (from “remote_file” to
"template”) and sometimes edit the name (from “config” to
"config.erb").
Sometimes I break up the config file to sub-configs. But these go into
different directories depending on if they need ERB attributes or not.
Logically, the sub-configs should be siblings, but physically they
go in
different directories. Sometimes I find myself moving the files back
and forth
as I refactor the config file and sub-config files and move
attributes around.

It would make more sense to set an attribute (“oh, by the way, that
file needs
ERB now”) on the existing resource. I also think it’s better if I
don’t have to
move the file from /files to /templates.

Imagine merging these resources into one “file” resource:

touch a file

file “/tmp/touch_me” do
action :touch
end

copy file

file “/tmp/blob” do

source “blob” is implicit. Looks in /files/default/blob

end

Some files are so small, it’s silly to create a

new file in a new directory just to hold the contents

file “/tmp/blob” do
contents "dude"
end

Some generated files don’t benifit from ERB.

file “database.yml” do
contents do @node.database_settings.to_yaml; end
end

switching from a static file to a template

shouldn’t require a new resource.

file “/tmp/blob” do
filter :erb # process source file with ERB

Not sure if it should default to source ‘blob’ or ‘blob.erb’

also not sure if it should look in /templates or just in /files

end

I know it’s a major change, but hey, now is the time for big
changes, right?
What do you think?

-=Dan=-


#3

On Wed, Oct 21, 2009 at 11:40 AM, dan@animoto.com wrote:

First of all, they are badly named. Why isn’t there a remote_template to
correspond with remote_file? What’s “remote” about remote_file in Chef
Solo?
Can we at least rename the little-used “file” to be “file_touch” so we can
use
"file" instead of “remote_file” everywhere?

From an development perspective, you actually use File all the time, because
Template and RemoteFile extend File, which prevents redundant code between
the latter two. I feel like there used to be an attribute to File to specify
the contents of the file, but I’m pretty sure that’s a memory from using
Puppet. Maybe we should add that, that is something else though.

I agree that the semantics of remote_file don’t fit Chef Solo when you’re
assuming that you’re getting the file from a Chef server, but the
remote_file provider supports URIs in the source attribute as well, which is
certainly likely to be remote even in Chef Solo.

So I agree only insofar as the naming can be a little tricky, but I feel
like it is architected well and disagree there.

Bryan


#4

On Wed, Oct 21, 2009 at 11:40 AM, dan@animoto.com wrote:

I’d like to examine the division between the “file”, “remote_file” and
"tempate" resources.

<… snip the explanation …>

In talking about this at Opscode HQ, and a bit on IRC, we agree.
Essentially, we should make the ‘file’ resource capable of doing the
job of both remote_file and template, and leave the existing resources
for backwards compatability. Your explanation makes sense, and it
will make things more clear - plus, I can imagine having multiple
processors for files, making the templating system even nifiter.
(Imagine an XSLT one, for example, on XML files.)

Dan, since you wrote such a great message, would you do us the honor
of filing the ticket and letting us know the number?

Best,
Adam


Opscode, Inc.
Adam Jacob, CTO
T: (206) 508-7449 E: adam@opscode.com


#5

On Wed, Oct 21, 2009 at 11:40 AM, dan@animoto.com wrote:

I’d like to examine the division between the “file”, “remote_file” and
"tempate" resources.

yes, good stuff. forcing naming to .erb, etc. might be handy for syntax
checking before deployment, etc.

I would also add ‘response_file’ to the list and add support for a filter
that allows ‘response_template’ along the lines of <
http://tickets.opscode.com/browse/CHEF-450>


Sander van Zoest
sander@vanzoest.com
San Diego, CA, USA
http://Sander.vanZoest.com/


#6

Ok, I am honored to open CHEF-651. Didn’t mean to make it a "major"
priority, but there you go.

On a related but different note: We could incorporate some shell
script syntax in the File resource – for scripts that output exactly
one file:

file “/etc/hosts” do
execute “make_hosts.rb”

maybe it looks in /files/default/make_hosts.rb?

maybe the STDOUT of make_hosts.rb becomes /etc/hosts?

but not if the script is unhappy?

end

Just thinking out loud.

-=Dan=-