File Provider Changes in 11.6

(Gist form: https://gist.github.com/danielsdeleo/5732547#file-ft-announce-md)

Hi Chefs,

We’ve recently merged a substantial rewrite of Chef’s file provider
family to master, and it will be released in 11.6.0. As part of these
changes, we’ve standardized the way the different providers create files
and update content, and added some cool improvements and features. But
first, a word of caution:

Changes to Previously Undefined Behaviors

Prior to this patch, file providers did not have defined behavior for
some situations, but instead relied on the behavior of the underlying
ruby implementation. We feel that defining and standardizing this
behavior will be a huge benefit to you, but there is some risk of
breakage, so please read on:

  • Chef previously did not define what file permissions it would set if a
    file resource did not specify them, and was inconsistent between
    different providers and in some cases differed based on the version of
    ruby used. All file providers will now create files with default
    permissions determined by the OS and filesystem default behavior. In
    general this is governed by your umask setting, but may also be affected
    by filesystem type or mount options.

    This can be a problem when depending on the default file mode when
    managing ssh keys or config with cookbook_file or template
    resources. In previous chef versions, cookbook_file would set the mode
    of a file to 0600 if not explicitly specified, and template
    resources would set the mode to 0600 if not specified when running
    on ruby 1.9.3, but 0644 on ruby 1.9.2 and earlier. In Chef 11.6,
    these are likely to be created with 0644 permissions (assuming a
    022 umask), which can cause SSH operations to error out.

  • Chef previously did not have a defined behavior if a file provider
    encountered something other than a file when attempting to update
    content. In particular, chef would follow symlinks and overwrite the
    symlink target’s content; other dir entry types (such as devices, named
    pipes, etc.) would fail in strange ways. Chef will now raise an error
    instead of overwriting the content of a symlink. If you wish, you can
    delete whatever’s in your way by setting force_unlink true on your
    resource. Note that there is no longer any built in means for managing
    the content of a file via a symlink–you must manage the target file
    instead.

Other Risks

File providers have been significantly refactored, with many methods
deprecated. If you’ve written custom providers that subclass a file
provider, we recommend you test them against master (or an upcoming beta
when available). We’ve made every effort to ensure that code using the
now-deprecated methods will continue to work, but it’s possible that the
new code may not set internal state in a way your code expects.

New Features and Enhancements

By rewriting the file providers to comprise a set of reusable
components, we’re able to deliver a lot of enhancements we think you’ll
like:

  • SELinux support: Chef will restorecon files after modifying them.

  • Configurable atomic file updates. Chef lets you choose between atomic
    (mv-based) or non-atomic (cp-based) file updates. Defaults to atomic.
    Atomic:

    • will not fail updating important file when out of disk space
    • will not fail updating a running binary
    • may alter file permissions when running as non-root user in some
      cases
    • files will temporarily have incorrect selinux permissions (until
      restorecon runs) or windows ACLs (until an ACL restore step runs)

    Non atomic is basically converse of the above.

    Global config: file_atomic_update, per-resource: atomic_update

    NOTE: At the moment, we don’t know if this will automatically make
    community cookbooks work with selinux enabled; it’s likely that the
    debian style file hierarchies used by some community cookbooks
    conflict with default selinux policy.

  • All file providers will now create files such that ACL inheritance
    works as expected on Windows.

  • Remote file can automatically send HTTP conditional GET requests,
    using ETags and If-Modified-Since. This is enabled by defualt, use
    use_conditional_get false to disable, or use_etags false or
    use_last_modified false to disable individual headers.

  • Remote file now supports FTP and local files, using “ftp://” and
    "file://" URIs, respectively.

  • Remote file supports custom headers.

Template Helper Methods

Template resources can now define helper methods or modules for use in
the template context.

In the template resource, there are three interfaces to extend the
template context. The simplest way is to declare a single method inline,
like this:

template "/path" do

  # The classic "hello world"
  helper(:hello_world) { "hello world" }

  # You can reference the node object to clean up repeated calls to
  # your cookbook's attributes:
  helper(:app) { node["app"] }

  # Helpers can take arguments.
  helper(:app_conf) { |setting| node["app"][setting] }
end

Your template can then use the methods you’ve created like this:

Say hello: <%= hello_world %> 

node["app"]["listen_port"] is: <%= app["listen_port"] %>

node["app"]["log_location"] is: <%= app_conf("log_location") %>

If your logic gets more complicated or your dentist told you to avoid
syntax sugar, you can define a module inline instead. This may also be
handy as an intermediate step between the inline method syntax above and
the full-blown module approach below when refactoring. The following code
is functionally identical to the first example:

template "/path" do
  helpers do
    # Now you're in the context of a module that will extend your
    # template

    def hello_world
      "hello world"
    end

    def app
      node["app"]
    end

    def app_conf(setting)
      node["app"][setting]
    end
  end
end

Finally, you can simply name a module to extend your template. If you
need to reuse extensions in multiple templates, or your template
extension code grows too unwieldy for the inline approaches above, you
can define a module in a library and use it in your templates like this:

template "/path" do
  helpers(MyHelperModule)
end


Daniel DeLeo

Hi Chefs,

A quick update on the changes coming in Chef 11.6:

On Friday, June 7, 2013 at 2:35 PM, Daniel DeLeo wrote:

(Gist form: File Refactor Announcement · GitHub)

Changes to Previously Undefined Behaviors

  • Chef previously did not have a defined behavior if a file provider
    encountered something other than a file when attempting to update
    content. In particular, chef would follow symlinks and overwrite the
    symlink target's content; other dir entry types (such as devices, named
    pipes, etc.) would fail in strange ways. Chef will now raise an error
    instead of overwriting the content of a symlink. If you wish, you can
    delete whatever's in your way by setting force_unlink true on your
    resource. Note that there is no longer any built in means for managing
    the content of a file via a symlink--you must manage the target file
    instead.

Our in-house testing revealed that it is more common than we believed for a file/template/etc. resource to manage a file via a symlink. For example, in Ubuntu 13.04 resolv.conf is a symlink, and we'd been happily using a cookbook we'd developed against previous Ubuntu versions to manage that file. As originally written, the file provider changes in 11.6 would have caused our cookbook to start throwing errors when managing resolv.conf.

To avoid a breaking change for this case, we've added an option called manage_symlink_source to file (and derivative) resources. When enabled, Chef will detect a symlink and manage the symlink's source file. By default, this option is enabled, but will emit a warning. We plan to change the default behavior to disabled in Chef 12. The option can be permanently enabled (with no warning) by adding manage_symlink_source true to your file/template/etc. resource.

Thanks,

Dan DeLeo

p.s., We'll have a release candidate available for you very soon.

Just as workaround for manage_symlink_source absence in older versions of
Chef do:

manage_symlink_source(true) if respond_to?(: manage_symlink_source)

This is awesome! So excited for template methods. This and partials have been things I wanted to see for a long time. Thanks to everyone who worked on this stuff.

Dan is there an easy way to detect file resources that have not specified modes? Might be nice to have an option flag that raises this. Maybe a simple cook that will emit all resources that might be effected on the upcoming change.

Now I have to go rewrite heaps of stuff!

On Jun 7, 2013, at 2:35 PM, Daniel DeLeo dan@kallistec.com wrote:

(Gist form: File Refactor Announcement · GitHub)

Hi Chefs,

We've recently merged a substantial rewrite of Chef's file provider
family to master, and it will be released in 11.6.0. As part of these
changes, we've standardized the way the different providers create files
and update content, and added some cool improvements and features. But
first, a word of caution:

Changes to Previously Undefined Behaviors

Prior to this patch, file providers did not have defined behavior for
some situations, but instead relied on the behavior of the underlying
ruby implementation. We feel that defining and standardizing this
behavior will be a huge benefit to you, but there is some risk of
breakage, so please read on:

  • Chef previously did not define what file permissions it would set if a
    file resource did not specify them, and was inconsistent between
    different providers and in some cases differed based on the version of
    ruby used. All file providers will now create files with default
    permissions determined by the OS and filesystem default behavior. In
    general this is governed by your umask setting, but may also be affected
    by filesystem type or mount options.

    This can be a problem when depending on the default file mode when
    managing ssh keys or config with cookbook_file or template
    resources. In previous chef versions, cookbook_file would set the mode
    of a file to 0600 if not explicitly specified, and template
    resources would set the mode to 0600 if not specified when running
    on ruby 1.9.3, but 0644 on ruby 1.9.2 and earlier. In Chef 11.6,
    these are likely to be created with 0644 permissions (assuming a
    022 umask), which can cause SSH operations to error out.

  • Chef previously did not have a defined behavior if a file provider
    encountered something other than a file when attempting to update
    content. In particular, chef would follow symlinks and overwrite the
    symlink target's content; other dir entry types (such as devices, named
    pipes, etc.) would fail in strange ways. Chef will now raise an error
    instead of overwriting the content of a symlink. If you wish, you can
    delete whatever's in your way by setting force_unlink true on your
    resource. Note that there is no longer any built in means for managing
    the content of a file via a symlink--you must manage the target file
    instead.

Other Risks

File providers have been significantly refactored, with many methods
deprecated. If you've written custom providers that subclass a file
provider, we recommend you test them against master (or an upcoming beta
when available). We've made every effort to ensure that code using the
now-deprecated methods will continue to work, but it's possible that the
new code may not set internal state in a way your code expects.

New Features and Enhancements

By rewriting the file providers to comprise a set of reusable
components, we're able to deliver a lot of enhancements we think you'll
like:

  • SELinux support: Chef will restorecon files after modifying them.

  • Configurable atomic file updates. Chef lets you choose between atomic
    (mv-based) or non-atomic (cp-based) file updates. Defaults to atomic.
    Atomic:

    • will not fail updating important file when out of disk space
    • will not fail updating a running binary
    • may alter file permissions when running as non-root user in some
      cases
    • files will temporarily have incorrect selinux permissions (until
      restorecon runs) or windows ACLs (until an ACL restore step runs)

    Non atomic is basically converse of the above.

    Global config: file_atomic_update, per-resource: atomic_update

    NOTE: At the moment, we don't know if this will automatically make
    community cookbooks work with selinux enabled; it's likely that the
    debian style file hierarchies used by some community cookbooks
    conflict with default selinux policy.

  • All file providers will now create files such that ACL inheritance
    works as expected on Windows.

  • Remote file can automatically send HTTP conditional GET requests,
    using ETags and If-Modified-Since. This is enabled by defualt, use
    use_conditional_get false to disable, or use_etags false or
    use_last_modified false to disable individual headers.

  • Remote file now supports FTP and local files, using "ftp://" and
    "file://" URIs, respectively.

  • Remote file supports custom headers.

Template Helper Methods

Template resources can now define helper methods or modules for use in
the template context.

In the template resource, there are three interfaces to extend the
template context. The simplest way is to declare a single method inline,
like this:

template "/path" do

  # The classic "hello world"
  helper(:hello_world) { "hello world" }

  # You can reference the node object to clean up repeated calls to
  # your cookbook's attributes:
  helper(:app) { node["app"] }

  # Helpers can take arguments.
  helper(:app_conf) { |setting| node["app"][setting] }
end

Your template can then use the methods you've created like this:

Say hello: <%= hello_world %> 

node["app"]["listen_port"] is: <%= app["listen_port"] %>

node["app"]["log_location"] is: <%= app_conf("log_location") %>

If your logic gets more complicated or your dentist told you to avoid
syntax sugar, you can define a module inline instead. This may also be
handy as an intermediate step between the inline method syntax above and
the full-blown module approach below when refactoring. The following code
is functionally identical to the first example:

template "/path" do
  helpers do
    # Now you're in the context of a module that will extend your
    # template

    def hello_world
      "hello world"
    end

    def app
      node["app"]
    end

    def app_conf(setting)
      node["app"][setting]
    end
  end
end

Finally, you can simply name a module to extend your template. If you
need to reuse extensions in multiple templates, or your template
extension code grows too unwieldy for the inline approaches above, you
can define a module in a library and use it in your templates like this:

template "/path" do
  helpers(MyHelperModule)
end

--
Daniel DeLeo

I believe that the users cookbook already explicitly specifies the modes
for .ssh and .ssh/authorized_keys if working with the users via the data
bag? I've always specified the modes natively myself (control-freak at
heart), so that's the only thing that I'm really concerned about breaking.

--
~~ StormeRider ~~

"Every world needs its heroes [...] They inspire us to be better than we
are. And they protect from the darkness that's just around the corner."

(from Smallville Season 6x1: "Zod")

On why I hate the phrase "that's so lame"... http://bit.ly/Ps3uSS

On Fri, Jun 7, 2013 at 2:48 PM, Jesse Nelson spheromak@gmail.com wrote:

This is awesome! So excited for template methods. This and partials have
been things I wanted to see for a long time. Thanks to everyone who worked
on this stuff.

Dan is there an easy way to detect file resources that have not specified
modes? Might be nice to have an option flag that raises this. Maybe a
simple cook that will emit all resources that might be effected on the
upcoming change.

Now I have to go rewrite heaps of stuff!

On Jun 7, 2013, at 2:35 PM, Daniel DeLeo dan@kallistec.com wrote:

(Gist form:
File Refactor Announcement · GitHub)

Hi Chefs,

We've recently merged a substantial rewrite of Chef's file provider
family to master, and it will be released in 11.6.0. As part of these
changes, we've standardized the way the different providers create files
and update content, and added some cool improvements and features. But
first, a word of caution:

Changes to Previously Undefined Behaviors

Prior to this patch, file providers did not have defined behavior for
some situations, but instead relied on the behavior of the underlying
ruby implementation. We feel that defining and standardizing this
behavior will be a huge benefit to you, but there is some risk of
breakage, so please read on:

  • Chef previously did not define what file permissions it would set if a
    file resource did not specify them, and was inconsistent between
    different providers and in some cases differed based on the version of
    ruby used. All file providers will now create files with default
    permissions determined by the OS and filesystem default behavior. In
    general this is governed by your umask setting, but may also be
    affected
    by filesystem type or mount options.

    This can be a problem when depending on the default file mode when
    managing ssh keys or config with cookbook_file or template
    resources. In previous chef versions, cookbook_file would set the
    mode
    of a file to 0600 if not explicitly specified, and template
    resources would set the mode to 0600 if not specified when running
    on ruby 1.9.3, but 0644 on ruby 1.9.2 and earlier. In Chef 11.6,
    these are likely to be created with 0644 permissions (assuming a
    022 umask), which can cause SSH operations to error out.

  • Chef previously did not have a defined behavior if a file provider
    encountered something other than a file when attempting to update
    content. In particular, chef would follow symlinks and overwrite the
    symlink target's content; other dir entry types (such as devices, named
    pipes, etc.) would fail in strange ways. Chef will now raise an error
    instead of overwriting the content of a symlink. If you wish, you can
    delete whatever's in your way by setting force_unlink true on your
    resource. Note that there is no longer any built in means for managing
    the content of a file via a symlink--you must manage the target file
    instead.

Other Risks

File providers have been significantly refactored, with many methods
deprecated. If you've written custom providers that subclass a file
provider, we recommend you test them against master (or an upcoming beta
when available). We've made every effort to ensure that code using the
now-deprecated methods will continue to work, but it's possible that the
new code may not set internal state in a way your code expects.

New Features and Enhancements

By rewriting the file providers to comprise a set of reusable
components, we're able to deliver a lot of enhancements we think you'll
like:

  • SELinux support: Chef will restorecon files after modifying them.

  • Configurable atomic file updates. Chef lets you choose between atomic
    (mv-based) or non-atomic (cp-based) file updates. Defaults to atomic.
    Atomic:

    • will not fail updating important file when out of disk space
    • will not fail updating a running binary
    • may alter file permissions when running as non-root user in some
      cases
    • files will temporarily have incorrect selinux permissions (until
      restorecon runs) or windows ACLs (until an ACL restore step runs)

    Non atomic is basically converse of the above.

    Global config: file_atomic_update, per-resource: atomic_update

    NOTE: At the moment, we don't know if this will automatically make
    community cookbooks work with selinux enabled; it's likely that the
    debian style file hierarchies used by some community cookbooks
    conflict with default selinux policy.

  • All file providers will now create files such that ACL inheritance
    works as expected on Windows.

  • Remote file can automatically send HTTP conditional GET requests,
    using ETags and If-Modified-Since. This is enabled by defualt, use
    use_conditional_get false to disable, or use_etags false or
    use_last_modified false to disable individual headers.

  • Remote file now supports FTP and local files, using "ftp://" and
    "file://" URIs, respectively.

  • Remote file supports custom headers.

Template Helper Methods

Template resources can now define helper methods or modules for use in
the template context.

In the template resource, there are three interfaces to extend the
template context. The simplest way is to declare a single method inline,
like this:

template "/path" do

  # The classic "hello world"
  helper(:hello_world) { "hello world" }

  # You can reference the node object to clean up repeated calls to
  # your cookbook's attributes:
  helper(:app) { node["app"] }

  # Helpers can take arguments.
  helper(:app_conf) { |setting| node["app"][setting] }
end

Your template can then use the methods you've created like this:

Say hello: <%= hello_world %>

node["app"]["listen_port"] is: <%= app["listen_port"] %>

node["app"]["log_location"] is: <%= app_conf("log_location") %>

If your logic gets more complicated or your dentist told you to avoid
syntax sugar, you can define a module inline instead. This may also be
handy as an intermediate step between the inline method syntax above and
the full-blown module approach below when refactoring. The following code
is functionally identical to the first example:

template "/path" do
  helpers do
    # Now you're in the context of a module that will extend your
    # template

    def hello_world
      "hello world"
    end

    def app
      node["app"]
    end

    def app_conf(setting)
      node["app"][setting]
    end
  end
end

Finally, you can simply name a module to extend your template. If you
need to reuse extensions in multiple templates, or your template
extension code grows too unwieldy for the inline approaches above, you
can define a module in a library and use it in your templates like this:

template "/path" do
  helpers(MyHelperModule)
end

--
Daniel DeLeo

That might be a good foodcritic test...

--
~~ StormeRider ~~

"Every world needs its heroes [...] They inspire us to be better than we
are. And they protect from the darkness that's just around the corner."

(from Smallville Season 6x1: "Zod")

On why I hate the phrase "that's so lame"... http://bit.ly/Ps3uSS

On Fri, Jun 7, 2013 at 2:48 PM, Jesse Nelson spheromak@gmail.com wrote:

This is awesome! So excited for template methods. This and partials have
been things I wanted to see for a long time. Thanks to everyone who worked
on this stuff.

Dan is there an easy way to detect file resources that have not specified
modes? Might be nice to have an option flag that raises this. Maybe a
simple cook that will emit all resources that might be effected on the
upcoming change.

Now I have to go rewrite heaps of stuff!

On Jun 7, 2013, at 2:35 PM, Daniel DeLeo dan@kallistec.com wrote:

(Gist form:
File Refactor Announcement · GitHub)

Hi Chefs,

We've recently merged a substantial rewrite of Chef's file provider
family to master, and it will be released in 11.6.0. As part of these
changes, we've standardized the way the different providers create files
and update content, and added some cool improvements and features. But
first, a word of caution:

Changes to Previously Undefined Behaviors

Prior to this patch, file providers did not have defined behavior for
some situations, but instead relied on the behavior of the underlying
ruby implementation. We feel that defining and standardizing this
behavior will be a huge benefit to you, but there is some risk of
breakage, so please read on:

  • Chef previously did not define what file permissions it would set if a
    file resource did not specify them, and was inconsistent between
    different providers and in some cases differed based on the version of
    ruby used. All file providers will now create files with default
    permissions determined by the OS and filesystem default behavior. In
    general this is governed by your umask setting, but may also be
    affected
    by filesystem type or mount options.

    This can be a problem when depending on the default file mode when
    managing ssh keys or config with cookbook_file or template
    resources. In previous chef versions, cookbook_file would set the
    mode
    of a file to 0600 if not explicitly specified, and template
    resources would set the mode to 0600 if not specified when running
    on ruby 1.9.3, but 0644 on ruby 1.9.2 and earlier. In Chef 11.6,
    these are likely to be created with 0644 permissions (assuming a
    022 umask), which can cause SSH operations to error out.

  • Chef previously did not have a defined behavior if a file provider
    encountered something other than a file when attempting to update
    content. In particular, chef would follow symlinks and overwrite the
    symlink target's content; other dir entry types (such as devices, named
    pipes, etc.) would fail in strange ways. Chef will now raise an error
    instead of overwriting the content of a symlink. If you wish, you can
    delete whatever's in your way by setting force_unlink true on your
    resource. Note that there is no longer any built in means for managing
    the content of a file via a symlink--you must manage the target file
    instead.

Other Risks

File providers have been significantly refactored, with many methods
deprecated. If you've written custom providers that subclass a file
provider, we recommend you test them against master (or an upcoming beta
when available). We've made every effort to ensure that code using the
now-deprecated methods will continue to work, but it's possible that the
new code may not set internal state in a way your code expects.

New Features and Enhancements

By rewriting the file providers to comprise a set of reusable
components, we're able to deliver a lot of enhancements we think you'll
like:

  • SELinux support: Chef will restorecon files after modifying them.

  • Configurable atomic file updates. Chef lets you choose between atomic
    (mv-based) or non-atomic (cp-based) file updates. Defaults to atomic.
    Atomic:

    • will not fail updating important file when out of disk space
    • will not fail updating a running binary
    • may alter file permissions when running as non-root user in some
      cases
    • files will temporarily have incorrect selinux permissions (until
      restorecon runs) or windows ACLs (until an ACL restore step runs)

    Non atomic is basically converse of the above.

    Global config: file_atomic_update, per-resource: atomic_update

    NOTE: At the moment, we don't know if this will automatically make
    community cookbooks work with selinux enabled; it's likely that the
    debian style file hierarchies used by some community cookbooks
    conflict with default selinux policy.

  • All file providers will now create files such that ACL inheritance
    works as expected on Windows.

  • Remote file can automatically send HTTP conditional GET requests,
    using ETags and If-Modified-Since. This is enabled by defualt, use
    use_conditional_get false to disable, or use_etags false or
    use_last_modified false to disable individual headers.

  • Remote file now supports FTP and local files, using "ftp://" and
    "file://" URIs, respectively.

  • Remote file supports custom headers.

Template Helper Methods

Template resources can now define helper methods or modules for use in
the template context.

In the template resource, there are three interfaces to extend the
template context. The simplest way is to declare a single method inline,
like this:

template "/path" do

  # The classic "hello world"
  helper(:hello_world) { "hello world" }

  # You can reference the node object to clean up repeated calls to
  # your cookbook's attributes:
  helper(:app) { node["app"] }

  # Helpers can take arguments.
  helper(:app_conf) { |setting| node["app"][setting] }
end

Your template can then use the methods you've created like this:

Say hello: <%= hello_world %>

node["app"]["listen_port"] is: <%= app["listen_port"] %>

node["app"]["log_location"] is: <%= app_conf("log_location") %>

If your logic gets more complicated or your dentist told you to avoid
syntax sugar, you can define a module inline instead. This may also be
handy as an intermediate step between the inline method syntax above and
the full-blown module approach below when refactoring. The following code
is functionally identical to the first example:

template "/path" do
  helpers do
    # Now you're in the context of a module that will extend your
    # template

    def hello_world
      "hello world"
    end

    def app
      node["app"]
    end

    def app_conf(setting)
      node["app"][setting]
    end
  end
end

Finally, you can simply name a module to extend your template. If you
need to reuse extensions in multiple templates, or your template
extension code grows too unwieldy for the inline approaches above, you
can define a module in a library and use it in your templates like this:

template "/path" do
  helpers(MyHelperModule)
end

--
Daniel DeLeo

On Friday, June 7, 2013 at 2:48 PM, Jesse Nelson wrote:

This is awesome! So excited for template methods. This and partials have been things I wanted to see for a long time. Thanks to everyone who worked on this stuff.

Dan is there an easy way to detect file resources that have not specified modes? Might be nice to have an option flag that raises this. Maybe a simple cook that will emit all resources that might be effected on the upcoming change.

Here's a more detailed breakdown of the changes:

  • cookbook_file: some 0600 files created 0644 for umask 022 users
  • directory: no change
  • file: no change
  • remote_directory: no change
  • remote_file: old ruby, no change. new_ruby, some 0600 files created 0644 for umask 022 users
  • template: old ruby, no change. new_ruby, some 0600 files created 0644 for umask 022 users

Where "old ruby" is 1.9.2 and lower.

It should be pretty straightforward to check the RUBY_VERSION constant to figure out what you should look for, then iterate through run_context.resource_collection to find resources where the mode is nil.

--
Daniel DeLeo