Newlines in templates


#1

I’m reusing the simple_iptables cookbook and working on a Windows PC for my workstation. My rules file is failing when I converge the node, and it tries to restart iptables. The problem seems to be the newlines in the file are DOS. For my own cookbooks I can set the newlines to always be UNIX, but I don’t want to edit a community cookbook. Is there a Chef/knife trick for ensuring that the template newlines are platform appropriate?


#2

Joshua, which file has the DOS newlines – are you saying a template file in
the simple_iptables cookbook has it? Or is the issue that your editor is
doing DOS style newlines?

If it’s the latter, I think the simplest thing is to just configure the
editor to do Unix line-endings. I get that that can still be challenging if
there are multiple people working on these things and you need to enforce
Unix-line endings for everyone.

-Adam

From: THARP, JOSHUA L [mailto:jt3518@att.com]
Sent: Tuesday, November 5, 2013 10:28 AM
To: chef@lists.opscode.com
Subject: [chef] Newlines in templates

I’m reusing the simple_iptables cookbook and working on a Windows PC for my
workstation. My rules file is failing when I converge the node, and it
tries to restart iptables. The problem seems to be the newlines in the file
are DOS. For my own cookbooks I can set the newlines to always be UNIX, but
I don’t want to edit a community cookbook. Is there a Chef/knife trick for
ensuring that the template newlines are platform appropriate?


#3

Let me try to be more precise.

I’m reusing the simple_iptables community cookbook. That cookbook has a template that it uses to create the actual iptables configuration file. If I “knife cookbook upload simple_iptables” from a Windows machine and then use its LWRP from my cookbook (which has no templates of its own) the resulting iptables configuration file ends up with DOS newlines.

It seems like Chef should alter the type of newlines when a node is converged so that if you are converging a Linux box you get LF and a Windows box you get CR-LF.

From: Adam Edwards [mailto:adamed@opscode.com]
Sent: Tuesday, November 05, 2013 10:55 AM
To: chef@lists.opscode.com
Subject: [chef] RE: Newlines in templates

Joshua, which file has the DOS newlines - are you saying a template file in the simple_iptables cookbook has it? Or is the issue that your editor is doing DOS style newlines?

If it’s the latter, I think the simplest thing is to just configure the editor to do Unix line-endings. I get that that can still be challenging if there are multiple people working on these things and you need to enforce Unix-line endings for everyone.

-Adam

From: THARP, JOSHUA L [mailto:jt3518@att.commailto:jt3518@att.com]
Sent: Tuesday, November 5, 2013 10:28 AM
To: chef@lists.opscode.commailto:chef@lists.opscode.com
Subject: [chef] Newlines in templates

I’m reusing the simple_iptables cookbook and working on a Windows PC for my workstation. My rules file is failing when I converge the node, and it tries to restart iptables. The problem seems to be the newlines in the file are DOS. For my own cookbooks I can set the newlines to always be UNIX, but I don’t want to edit a community cookbook. Is there a Chef/knife trick for ensuring that the template newlines are platform appropriate?


#4

Are you using Git? If so, it is converting the checked out files for you. See:

https://help.github.com/articles/dealing-with-line-endings

On Nov 5, 2013, at 21:17, “THARP, JOSHUA L” jt3518@att.com wrote:

Let me try to be more precise.

I’m reusing the simple_iptables community cookbook. That cookbook has a template that it uses to create the actual iptables configuration file. If I “knife cookbook upload simple_iptables” from a Windows machine and then use its LWRP from my cookbook (which has no templates of its own) the resulting iptables configuration file ends up with DOS newlines.

It seems like Chef should alter the type of newlines when a node is converged so that if you are converging a Linux box you get LF and a Windows box you get CR-LF.

From: Adam Edwards [mailto:adamed@opscode.com]
Sent: Tuesday, November 05, 2013 10:55 AM
To: chef@lists.opscode.com
Subject: [chef] RE: Newlines in templates

Joshua, which file has the DOS newlines – are you saying a template file in the simple_iptables cookbook has it? Or is the issue that your editor is doing DOS style newlines?

If it’s the latter, I think the simplest thing is to just configure the editor to do Unix line-endings. I get that that can still be challenging if there are multiple people working on these things and you need to enforce Unix-line endings for everyone.

-Adam

From: THARP, JOSHUA L [mailto:jt3518@att.com]
Sent: Tuesday, November 5, 2013 10:28 AM
To: chef@lists.opscode.com
Subject: [chef] Newlines in templates

I’m reusing the simple_iptables cookbook and working on a Windows PC for my workstation. My rules file is failing when I converge the node, and it tries to restart iptables. The problem seems to be the newlines in the file are DOS. For my own cookbooks I can set the newlines to always be UNIX, but I don’t want to edit a community cookbook. Is there a Chef/knife trick for ensuring that the template newlines are platform appropriate?


#5

I am using Git, and with a little work (and a global setting that affects all repositories) I can have it change the newlines. However, the question remains if it is possible for Chef to do newline conversions. If I’m editing on a Windows PC, and I upload my cookbook – targeting a *nix platform – the cookbook is going to have the wrong line endings, and many Linux configuration files as well as shell interpreters are sensitive to the line endings. I would think mine is a fairly common scenario.

From: Martha Greenberg [mailto:marthag@MIT.EDU]
Sent: Tuesday, November 05, 2013 3:02 PM
To: chef@lists.opscode.com
Subject: [chef] Re: RE: RE: Newlines in templates

Are you using Git? If so, it is converting the checked out files for you. See:

https://help.github.com/articles/dealing-with-line-endings

On Nov 5, 2013, at 21:17, “THARP, JOSHUA L” <jt3518@att.commailto:jt3518@att.com> wrote:
Let me try to be more precise.

I’m reusing the simple_iptables community cookbook. That cookbook has a template that it uses to create the actual iptables configuration file. If I “knife cookbook upload simple_iptables” from a Windows machine and then use its LWRP from my cookbook (which has no templates of its own) the resulting iptables configuration file ends up with DOS newlines.

It seems like Chef should alter the type of newlines when a node is converged so that if you are converging a Linux box you get LF and a Windows box you get CR-LF.

From: Adam Edwards [mailto:adamed@opscode.com]
Sent: Tuesday, November 05, 2013 10:55 AM
To: chef@lists.opscode.commailto:chef@lists.opscode.com
Subject: [chef] RE: Newlines in templates

Joshua, which file has the DOS newlines – are you saying a template file in the simple_iptables cookbook has it? Or is the issue that your editor is doing DOS style newlines?

If it’s the latter, I think the simplest thing is to just configure the editor to do Unix line-endings. I get that that can still be challenging if there are multiple people working on these things and you need to enforce Unix-line endings for everyone.

-Adam

From: THARP, JOSHUA L [mailto:jt3518@att.commailto:jt3518@att.com]
Sent: Tuesday, November 5, 2013 10:28 AM
To: chef@lists.opscode.commailto:chef@lists.opscode.com
Subject: [chef] Newlines in templates

I’m reusing the simple_iptables cookbook and working on a Windows PC for my workstation. My rules file is failing when I converge the node, and it tries to restart iptables. The problem seems to be the newlines in the file are DOS. For my own cookbooks I can set the newlines to always be UNIX, but I don’t want to edit a community cookbook. Is there a Chef/knife trick for ensuring that the template newlines are platform appropriate?


#6

On Wed, Nov 6, 2013 at 9:03 AM, THARP, JOSHUA L jt3518@att.com wrote:

I am using Git, and with a little work (and a global setting that affects
all repositories) I can have it change the newlines. However, the question
remains if it is possible for Chef to do newline conversions. If I’m editing
on a Windows PC, and I upload my cookbook – targeting a *nix platform – the
cookbook is going to have the wrong line endings, and many Linux
configuration files as well as shell interpreters are sensitive to the line
endings. I would think mine is a fairly common scenario.

Joshua,

While that’s true, there’s no sensible default that Chef could take.
What if you’re managing a mixed Linux/Windows infrastructure with
Chef, and some of the templates need Windows-format line-endings and
others need Unix-format ones?

  • Julian


[ Julian C. Dunn jdunn@aquezada.com * Sorry, I’m ]
[ WWW: http://www.aquezada.com/staff/julian * only Web 1.0 ]
[ gopher://sdf.org/1/users/keymaker/ * compliant! ]
[ PGP: 91B3 7A9D 683C 7C16 715F 442C 6065 D533 FDC2 05B9 ]


#7

Could it be done at the time that the template is actually parsed and written to disk based on the platform type?

On Wednesday, November 6, 2013 at 7:02 PM, Julian C. Dunn wrote:

On Wed, Nov 6, 2013 at 9:03 AM, THARP, JOSHUA L <jt3518@att.com (mailto:jt3518@att.com)> wrote:

I am using Git, and with a little work (and a global setting that affects
all repositories) I can have it change the newlines. However, the question
remains if it is possible for Chef to do newline conversions. If I’m editing
on a Windows PC, and I upload my cookbook – targeting a *nix platform – the
cookbook is going to have the wrong line endings, and many Linux
configuration files as well as shell interpreters are sensitive to the line
endings. I would think mine is a fairly common scenario.

Joshua,

While that’s true, there’s no sensible default that Chef could take.
What if you’re managing a mixed Linux/Windows infrastructure with
Chef, and some of the templates need Windows-format line-endings and
others need Unix-format ones?

  • Julian


[ Julian C. Dunn <jdunn@aquezada.com (mailto:jdunn@aquezada.com)> * Sorry, I’m ]
[ WWW: http://www.aquezada.com/staff/julian * only Web 1.0 ]
[ gopher://sdf.org/1/users/keymaker/ (http://sdf.org/1/users/keymaker/) * compliant! ]
[ PGP: 91B3 7A9D 683C 7C16 715F 442C 6065 D533 FDC2 05B9 ]


#8

FWIW:

I think the file format and encoding is the responsability
of the cookbook/template writer.

There’s plenty of editors allowing to
set the file encoding and line endings the way you wish.

I think about
gvim and notepad++ for thoose I actually use for this purpose.

At
least it must be an option as magically converting a file at rendering
could become confusing.

I think about rendering config files on a
server to push them toward a switch later, you may wish to keep the
encoding of the file even if it is not the server base encoding.

Tensibai

Le 2013-11-07 04:40, Daniel Condomitti a écrit :

Could
it be done at the time that the template is actually parsed and written
to disk based on the platform type?

On Wednesday, November 6, 2013
at 7:02 PM, Julian C. Dunn wrote:

On Wed, Nov 6, 2013 at 9:03 AM,
THARP, JOSHUA L jt3518@att.com wrote:

I am using Git, and
with a little work (and a global setting that affects

all
repositories) I can have it change the newlines. However, the question

remains if it is possible for Chef to do newline conversions. If
I’m editing

on a Windows PC, and I upload my cookbook - targeting a
*nix platform - the

cookbook is going to have the wrong line
endings, and many Linux

configuration files as well as shell
interpreters are sensitive to the line

endings. I would think mine
is a fairly common scenario.

Joshua,

While that’s true,
there’s no sensible default that Chef could take.

What if you’re
managing a mixed Linux/Windows infrastructure with

Chef, and some of
the templates need Windows-format line-endings and

others need
Unix-format ones?

  • Julian


[ Julian C. Dunn
jdunn@aquezada.com * Sorry, I’m ]

[ WWW:
http://www.aquezada.com/staff/julian [1] * only Web 1.0 ]

[
gopher://sdf.org/1/users/keymaker/ [2] * compliant! ]

[ PGP: 91B3
7A9D 683C 7C16 715F 442C 6065 D533 FDC2 05B9 ]

Links:

[1]


[2]
http://sdf.org/1/users/keymaker/


#9

I’m with Daniel. The node that downloads the files from the Chef server is in the best position to know the required end of line character. Julian’s point supports mine nicely. How can I tell Git or my editor to always use or when some cookbooks (and therefore templates) might be used on more than one platform? Perhaps Opscode could add a tag to templates that flags chef-client to make the file’s end of line characters conform?

@Tensibai I too use gvim and can control the line endings. However, what set this off was the use of a community cookbook. I could edit the community cookbook to change the line endings, but I shouldn’t have to.

From: Tensibai [mailto:tensibai@iabis.net]
Sent: Thursday, November 07, 2013 1:16 AM
To: chef@lists.opscode.com
Subject: [chef] Re: Re: Re: RE: Re: RE: RE: Newlines in templates

FWIW:

I think the file format and encoding is the responsability of the cookbook/template writer.

There’s plenty of editors allowing to set the file encoding and line endings the way you wish.

I think about gvim and notepad++ for thoose I actually use for this purpose.

At least it must be an option as magically converting a file at rendering could become confusing.

I think about rendering config files on a server to push them toward a switch later, you may wish to keep the encoding of the file even if it is not the server base encoding.

Tensibai

Le 2013-11-07 04:40, Daniel Condomitti a écrit :
Could it be done at the time that the template is actually parsed and written to disk based on the platform type?

On Wednesday, November 6, 2013 at 7:02 PM, Julian C. Dunn wrote:
On Wed, Nov 6, 2013 at 9:03 AM, THARP, JOSHUA L <jt3518@att.commailto:jt3518@att.com> wrote:
I am using Git, and with a little work (and a global setting that affects
all repositories) I can have it change the newlines. However, the question
remains if it is possible for Chef to do newline conversions. If I’m editing
on a Windows PC, and I upload my cookbook – targeting a *nix platform – the
cookbook is going to have the wrong line endings, and many Linux
configuration files as well as shell interpreters are sensitive to the line
endings. I would think mine is a fairly common scenario.

Joshua,

While that’s true, there’s no sensible default that Chef could take.
What if you’re managing a mixed Linux/Windows infrastructure with
Chef, and some of the templates need Windows-format line-endings and
others need Unix-format ones?

  • Julian


[ Julian C. Dunn <jdunn@aquezada.commailto:jdunn@aquezada.com> * Sorry, I’m ]
[ WWW: http://www.aquezada.com/staff/julian * only Web 1.0 ]
[ gopher://sdf.org/1/users/keymaker/http://sdf.org/1/users/keymaker/ * compliant! ]
[ PGP: 91B3 7A9D 683C 7C16 715F 442C 6065 D533 FDC2 05B9 ]


#10

I really don’t agree saying the node is in the best position, it may
depend on the purpose of the file rendered.

So yes, it could be of
some help in limited cases to have an attribut like line_ending:
(cr|cr-lf|lf|target-ending) into the template ressource.

Indeed, you
should not have to convert a community cookbook (but when needed,
dos2unix/unix2dos or (to|fro)dos under ubuntu are great help for that).

The files and template sources dir in a cookbook can be specified
toward the target os, the most precise wins, so files within a windows
dir should have cr-lf endings and files under a redhat or ubuntu or
linux dir should have a lf ending (I won’t go about OS X as I don’t
manage this kind of systems).

A file within the default dir is
generally os agnostic and depends more on the way the app read the file
than the node os on which the app run.

In brief: I think it could be
an attribute into template, but it sounds like a real overhead, maybe a
hwrp extending the template ressource which can be included or not would
fit better, but I may be wrong in this assumption ?

Maybe the debate
worth a ticket on chef jira.

Le 2013-11-07 15:27, THARP, JOSHUA L a
écrit :

I’m with Daniel. The node that downloads the files from the
Chef server is in the best position to know the required end of line
character. Julian’s point supports mine nicely. How can I tell Git or my
editor to always use or when some cookbooks (and therefore
templates) might be used on more than one platform? Perhaps Opscode
could add a tag to templates that flags chef-client to make the file’s
end of line characters conform?

@Tensibai I too use gvim and can
control the line endings. However, what set this off was the use of a
community cookbook. I could edit the community cookbook to change the
line endings, but I shouldn’t have to.

FROM: Tensibai
[mailto:tensibai@iabis.net]
SENT: Thursday, November 07, 2013 1:16
AM
TO: chef@lists.opscode.com
SUBJECT: [chef] Re: Re: Re: RE: Re:
RE: RE: Newlines in templates

FWIW:

I think the file format
and encoding is the responsability of the cookbook/template writer.

There’s plenty of editors allowing to set the file encoding and line
endings the way you wish.

I think about gvim and notepad++ for
thoose I actually use for this purpose.

At least it must be an
option as magically converting a file at rendering could become
confusing.

I think about rendering config files on a server to
push them toward a switch later, you may wish to keep the encoding of
the file even if it is not the server base encoding.

Tensibai

Le 2013-11-07 04:40, Daniel Condomitti a écrit :

Could it be
done at the time that the template is actually parsed and written to
disk based on the platform type?

On Wednesday, November 6, 2013
at 7:02 PM, Julian C. Dunn wrote:

On Wed, Nov 6, 2013 at 9:03
AM, THARP, JOSHUA L jt3518@att.com wrote:

I am using Git,
and with a little work (and a global setting that affects

all repositories) I can have it change the newlines. However, the
question

remains if it is possible for Chef to do newline
conversions. If I’m editing

on a Windows PC, and I upload my
cookbook - targeting a *nix platform - the

cookbook is going
to have the wrong line endings, and many Linux

configuration
files as well as shell interpreters are sensitive to the line

endings. I would think mine is a fairly common scenario.

Joshua,

While that’s true, there’s no sensible default that
Chef could take.

What if you’re managing a mixed Linux/Windows
infrastructure with

Chef, and some of the templates need
Windows-format line-endings and

others need Unix-format ones?

  • Julian

[ Julian C. Dunn
jdunn@aquezada.com * Sorry, I’m ]

[ WWW:
http://www.aquezada.com/staff/julian [1] * only Web 1.0 ]

[
gopher://sdf.org/1/users/keymaker/ [2] * compliant! ]

[ PGP:
91B3 7A9D 683C 7C16 715F 442C 6065 D533 FDC2 05B9 ]

Links:

[1] http://www.aquezada.com/staff/julian
[2]
http://sdf.org/1/users/keymaker/


#11

Good suggestion on the Jira ticket: https://tickets.opscode.com/browse/CHEF-4746

From: Tensibai [mailto:tensibai@iabis.net]
Sent: Thursday, November 07, 2013 7:12 AM
To: chef@lists.opscode.com
Subject: [chef] Re: RE: Re: Re: Re: RE: Re: RE: RE: Newlines in templates

I really don’t agree saying the node is in the best position, it may depend on the purpose of the file rendered.

So yes, it could be of some help in limited cases to have an attribut like line_ending: (cr|cr-lf|lf|target-ending) into the template ressource.

Indeed, you should not have to convert a community cookbook (but when needed, dos2unix/unix2dos or (to|fro)dos under ubuntu are great help for that).

The files and template sources dir in a cookbook can be specified toward the target os, the most precise wins, so files within a windows dir should have cr-lf endings and files under a redhat or ubuntu or linux dir should have a lf ending (I won’t go about OS X as I don’t manage this kind of systems).

A file within the default dir is generally os agnostic and depends more on the way the app read the file than the node os on which the app run.

In brief: I think it could be an attribute into template, but it sounds like a real overhead, maybe a hwrp extending the template ressource which can be included or not would fit better, but I may be wrong in this assumption ?

Maybe the debate worth a ticket on chef jira.

Le 2013-11-07 15:27, THARP, JOSHUA L a écrit :
I’m with Daniel. The node that downloads the files from the Chef server is in the best position to know the required end of line character. Julian’s point supports mine nicely. How can I tell Git or my editor to always use or when some cookbooks (and therefore templates) might be used on more than one platform? Perhaps Opscode could add a tag to templates that flags chef-client to make the file’s end of line characters conform?

@Tensibai I too use gvim and can control the line endings. However, what set this off was the use of a community cookbook. I could edit the community cookbook to change the line endings, but I shouldn’t have to.

From: Tensibai [mailto:tensibai@iabis.net]
Sent: Thursday, November 07, 2013 1:16 AM
To: chef@lists.opscode.commailto:chef@lists.opscode.com
Subject: [chef] Re: Re: Re: RE: Re: RE: RE: Newlines in templates

FWIW:

I think the file format and encoding is the responsability of the cookbook/template writer.

There’s plenty of editors allowing to set the file encoding and line endings the way you wish.

I think about gvim and notepad++ for thoose I actually use for this purpose.

At least it must be an option as magically converting a file at rendering could become confusing.

I think about rendering config files on a server to push them toward a switch later, you may wish to keep the encoding of the file even if it is not the server base encoding.

Tensibai

Le 2013-11-07 04:40, Daniel Condomitti a écrit :
Could it be done at the time that the template is actually parsed and written to disk based on the platform type?

On Wednesday, November 6, 2013 at 7:02 PM, Julian C. Dunn wrote:
On Wed, Nov 6, 2013 at 9:03 AM, THARP, JOSHUA L <jt3518@att.commailto:jt3518@att.com> wrote:
I am using Git, and with a little work (and a global setting that affects
all repositories) I can have it change the newlines. However, the question
remains if it is possible for Chef to do newline conversions. If I’m editing
on a Windows PC, and I upload my cookbook – targeting a *nix platform – the
cookbook is going to have the wrong line endings, and many Linux
configuration files as well as shell interpreters are sensitive to the line
endings. I would think mine is a fairly common scenario.

Joshua,

While that’s true, there’s no sensible default that Chef could take.
What if you’re managing a mixed Linux/Windows infrastructure with
Chef, and some of the templates need Windows-format line-endings and
others need Unix-format ones?

  • Julian


[ Julian C. Dunn <jdunn@aquezada.commailto:jdunn@aquezada.com> * Sorry, I’m ]
[ WWW: http://www.aquezada.com/staff/julian * only Web 1.0 ]
[ gopher://sdf.org/1/users/keymaker/http://sdf.org/1/users/keymaker/ * compliant! ]
[ PGP: 91B3 7A9D 683C 7C16 715F 442C 6065 D533 FDC2 05B9 ]