Appending to an existing file, i.e nginx site


#1

Hi list,

I have a use case where I need to append a location to an existing nginx
site (in /etc/nginx/sites-enabled/sitename), this site was also deployed by
another recipe of mine.

Why? Well, basically it’s a static site componente that should be accessed
as a path, but the main app should remain intact. They should share the
same domain
.

I should be pretty straightforward to do, but I’d like to know if anyone
else had a similar use case before? If so, how did you approach it?

Thanks in advance!

M>


#2

If the existing template used to create the site contains the line you need
to modify, but you want to modify the value, then you should be able to
modify the nginx site resource (it should be on the resource path) to add
the required information… just use the same resource name and only
specify what you want to change.

If it doesn’t yet contain that line, you’ll need to use a different
template (you could override which template gets used by modifying the
resource, for example) or modify the existing template to add the line.

On Fri, Mar 15, 2013 at 3:20 AM, Marcelo de Moraes Serpa <
celoserpa@gmail.com> wrote:

Hi list,

I have a use case where I need to append a location to an existing nginx
site (in /etc/nginx/sites-enabled/sitename), this site was also deployed by
another recipe of mine.

Why? Well, basically it’s a static site componente that should be accessed
as a path, but the main app should remain intact. They should share the
same domain
.

I should be pretty straightforward to do, but I’d like to know if anyone
else had a similar use case before? If so, how did you approach it?

Thanks in advance!

M>


#3

How does this come up every other week? You guys heard of Google?
Mailing list search?

–AJ

On 19 March 2013 07:18, Jesse Campbell hikeit@gmail.com wrote:

If the existing template used to create the site contains the line you need
to modify, but you want to modify the value, then you should be able to
modify the nginx site resource (it should be on the resource path) to add
the required information… just use the same resource name and only specify
what you want to change.

If it doesn’t yet contain that line, you’ll need to use a different template
(you could override which template gets used by modifying the resource, for
example) or modify the existing template to add the line.

On Fri, Mar 15, 2013 at 3:20 AM, Marcelo de Moraes Serpa
celoserpa@gmail.com wrote:

Hi list,

I have a use case where I need to append a location to an existing nginx
site (in /etc/nginx/sites-enabled/sitename), this site was also deployed by
another recipe of mine.

Why? Well, basically it’s a static site componente that should be accessed
as a path, but the main app should remain intact. They should share the
same domain
.

I should be pretty straightforward to do, but I’d like to know if anyone
else had a similar use case before? If so, how did you approach it?

Thanks in advance!

M>


#4

Sorry for my initial reply.

Please don’t edit files that already exist. Please never do this.

If you really must:

http://community.opscode.com/cookbooks/line


http://augeas.net/

Hope this helps, thanks in advance, and please use the mailing list
search functionality next time before posting a new thread.

Cheers,

AJ

On 15 March 2013 20:20, Marcelo de Moraes Serpa celoserpa@gmail.com wrote:

Hi list,

I have a use case where I need to append a location to an existing nginx
site (in /etc/nginx/sites-enabled/sitename), this site was also deployed by
another recipe of mine.

Why? Well, basically it’s a static site componente that should be accessed
as a path, but the main app should remain intact. They should share the
same domain
.

I should be pretty straightforward to do, but I’d like to know if anyone
else had a similar use case before? If so, how did you approach it?

Thanks in advance!

M>


#5

@Jesse, that makes sense. Having a new template based off the old one, but
with the added data. Thanks for the idea.

@AJ Sorry if I got you upset, wasn’t my intention. I tried googling for it
before posting. This is a very specific need for an automation script I’m
setting up, and will only be used for adding “static” components to an
existing domain (a landing page for example, for a promo) in a specific
path.

Of course, I could just create a small application that could serve those
static files from a specific route. I was just wondering someone out there
had a similar use case and used Chef to implement it, since it just seemed
easir to serve them from nginx (and I didn’t want to add the path conf
manually, hence the post).

And thank you very much for the URLs, very interesting resources!

  • Marcelo.

On Mon, Mar 18, 2013 at 2:24 PM, AJ Christensen aj@junglist.gen.nz wrote:

Sorry for my initial reply.

Please don’t edit files that already exist. Please never do this.

If you really must:

http://community.opscode.com/cookbooks/line
https://github.com/opscode/chef/blob/master/lib/chef/util/file_edit.rb
http://augeas.net/

Hope this helps, thanks in advance, and please use the mailing list
search functionality next time before posting a new thread.

Cheers,

AJ

On 15 March 2013 20:20, Marcelo de Moraes Serpa celoserpa@gmail.com
wrote:

Hi list,

I have a use case where I need to append a location to an existing nginx
site (in /etc/nginx/sites-enabled/sitename), this site was also deployed
by
another recipe of mine.

Why? Well, basically it’s a static site componente that should be
accessed
as a path, but the main app should remain intact. They should share the
same domain
.

I should be pretty straightforward to do, but I’d like to know if anyone
else had a similar use case before? If so, how did you approach it?

Thanks in advance!

M>


#6

On Mon, Mar 18, 2013 at 4:24 PM, AJ Christensen aj@junglist.gen.nz wrote:

Please don’t edit files that already exist. Please never do this.

Is that really a practical edict? What about files like
/etc/sysctl.conf needing to be written to by multiple recipes?

FileEdit works, but it gets a bad rap on this list. Chef’s going to
need a canonical and supported way to do this…

Michael Della Bitta


Appinions
18 East 41st Street, 2nd Floor
New York, NY 10017-6271

www.appinions.com

Where Influence Isn’t a Game


#7

Im not sure if we can have a standard resource for this type of action. For
specific tasks, we can have dedicated resource, for example the sysctl
cookbook does provide a resource/provider[1], which is much cleaner/safer
to use across multiple cookbooks. Given that all config files does not
follow same patterns, its difficult to generalize this. Albeit these
resources can use file edit behind the scenes (for cases where sysctl like
interface is absent).

What might make sense have a set of common patterns (like gitconfig, ini
etc) implemented and then subclass it from there.

[1]

On Tue, Mar 19, 2013 at 7:29 AM, Michael Della Bitta <
michael.della.bitta@appinions.com> wrote:

On Mon, Mar 18, 2013 at 4:24 PM, AJ Christensen aj@junglist.gen.nz
wrote:

Please don’t edit files that already exist. Please never do this.

Is that really a practical edict? What about files like
/etc/sysctl.conf needing to be written to by multiple recipes?

FileEdit works, but it gets a bad rap on this list. Chef’s going to
need a canonical and supported way to do this…

Michael Della Bitta


Appinions
18 East 41st Street, 2nd Floor
New York, NY 10017-6271

www.appinions.com

Where Influence Isn’t a Game


#8

I do this a lot in my recipes using bash, albeit for single line items such
as in hosts files and such. Moving it up to something that supports
multi-line using ruby I’m assuming wouldn’t be too difficult. I’m not a
ruby developer though, so maybe I’m wrong?

The logic is simply

  • grep the file to see if what I want to add is already there or not
  • if not, put it in

As an example, I need loopback on the localhost line for our backup
application

ash “add_loopback” do
code <<-EOH
LOOPBACKCHECK=$(grep loopback /etc/hosts | wc -l)
if [ $LOOPBACKCHECK -lt 1 ];
then
sed -i ‘s/::1[
\t]*localhost/::1\tlocalhost\tloopback/g’ /etc/hosts
fi
EOH
end

Or, I also need the hostname assigned to the default outbound ip for the
host which is always eth0 for us

bash “ensure_hostname” do
code <<-EOH
HOSTNAMECHECK=$(grep $HOSTNAME /etc/hosts | wc -l)
if [ HOSTNAMECHECK -lt 1 ]; then HOSTIP=(grep IPADDR
/etc/sysconfig/network-scripts/ifcfg-eth0 | sed s/‘IPADDR=’//)
echo “$HOSTIP $HOSTNAME” >> /etc/hosts
fi
EOH
end

On Tue, Mar 19, 2013 at 10:29 AM, Michael Della Bitta <
michael.della.bitta@appinions.com> wrote:

On Mon, Mar 18, 2013 at 4:24 PM, AJ Christensen aj@junglist.gen.nz
wrote:

Please don’t edit files that already exist. Please never do this.

Is that really a practical edict? What about files like
/etc/sysctl.conf needing to be written to by multiple recipes?

FileEdit works, but it gets a bad rap on this list. Chef’s going to
need a canonical and supported way to do this…

Michael Della Bitta


Appinions
18 East 41st Street, 2nd Floor
New York, NY 10017-6271

www.appinions.com

Where Influence Isn’t a Game


#9

+1

No, it’s not a practical edict, and in many cases it can be very counterproductive and even cause security problems.

The chef philosophy is that you don’t edit partial files, but only generate a complete copy of a configuration file. I can understand where that’s coming from. In many cases, it does make sense - but in my mind this approach is vastly overused in chef. In a few cases, chef itself actually breaks the philosophy - that’s why there are resources to manage /etc/fstab and /etc/exports, for instance.

The main issue I have with it isn’t even with global files such as /etc/sysctl.conf or /etc/hosts.

The far bigger elephant in the room is that distribution vendors usually provide well-tuned configuration files that shouldn’t be clobbered, but rather surgically manipulated.

That said, the surgical approach can be very difficult to implement in a generalized way. Multiline context-sensitive edits are non-trivial; for instance, the nginx configuration file. You have to be able to detect not just whether the appropriate entry exists with all the multiple lines in place, but also that it actually is in the right place; you can’t just append it to the end of a file and expect it to work.

To make matters even more complicated, nginx configuration file support including other files.

So to make this work, you have three options:

  • Create a one-off solution that encodes a lot of assumptions about how your particular configuration file is structured.
  • Implement a complete parser that understands nginix configuration files and translates it into something like json/XML/DOM, then allows you to manipulate it, and write out only the manipulated parts.
  • Generate the complete configuration file.

So, in this specific scenario, and many similar ones, I can understand where AJ’s statement came from. It’s just vastly overused.

-----Original message-----

From:Michael Della Bitta <michael.della.bitta@appinions.com mailto:michael.della.bitta@appinions.com >
Sent: Tuesday 19th March 2013 7:29
To: chef@lists.opscode.com mailto:chef@lists.opscode.com
Subject: [chef] Re: Re: Appending to an existing file, i.e nginx site

On Mon, Mar 18, 2013 at 4:24 PM, AJ Christensen <aj@junglist.gen.nz mailto:aj@junglist.gen.nz > wrote:

Please don’t edit files that already exist. Please never do this.

Is that really a practical edict? What about files like
/etc/sysctl.conf needing to be written to by multiple recipes?

FileEdit works, but it gets a bad rap on this list. Chef’s going to
need a canonical and supported way to do this…

Michael Della Bitta


Appinions
18 East 41st Street, 2nd Floor
New York, NY 10017-6271

www.appinions.com http://www.appinions.com

Where Influence Isn’t a Game


#10

Just a thought, can’t

/etc/nginx/sites-enabled/sitename

Be a template or file deployed via chef?

For apache we have a standard build from source I wrote, which includes the
httpd.conf.

For the local.conf I store it in templates, with a template for each host
we have. The are deployed via this recipe.

template “/usr/local/apache2/conf.d/local.conf” do
source "configs/${node[‘fqdn’]}_local.conf.erb"
end

So for www.myhost.com I have a template file www.myhost.com_local.conf.erb

I took this approach because then we have a nice system for keeping our web
server configs in a system we already have version control set up for. Also
a nice central place to examine and edit apache configs when we need to.

On Fri, Mar 15, 2013 at 3:20 AM, Marcelo de Moraes Serpa <
celoserpa@gmail.com> wrote:

Hi list,

I have a use case where I need to append a location to an existing nginx
site (in /etc/nginx/sites-enabled/sitename), this site was also deployed by
another recipe of mine.

Why? Well, basically it’s a static site componente that should be accessed
as a path, but the main app should remain intact. They should share the
same domain
.

I should be pretty straightforward to do, but I’d like to know if anyone
else had a similar use case before? If so, how did you approach it?

Thanks in advance!

M>


#11

All,

In the case of Ubuntu (and Debian, I suppose), this probably trickles
down to Mint as well, Heavy Water operations often find that the
defaults are poorly constructed examples copied straight from the
source code of the original software; even moreso could cause great
damage if deployed and used in a system without understanding the
implications of defaults.

I definitely don’t hold my vendor in such high regard! Frankly; I’d
like to strip or bypass config files (& service definitions) out of
packages altogether. The asset-bundle or omnibus-stack packages we
build are generally created in this manner.

–skip-config-files --skip-services --skip-users --skip-groups
–dont-pretend-to-be-config-management

If the vendor default configuration file never existed; the service
would not start. The service would not even exist!

You wouldn’t have to edit single lines of files contained in the
software you are deploying if it was part of the procedure for
deploying your software (this mirrors 90+% of ours):

  • research config
  • construct a configuration file
  • instrument configuration for automation
  • install stripped packages
  • deploy users and groups
  • deploy configuration
  • deploy services

I hope this offers some insight in-to how we have solved countless
deployments over disparate time-zones totaling Node counts in the
thousands across the prime Unixes.

A final point I would like to make: it is possible to re-open a
template Resource in the Compile DSL and add additional data to the
’variables’ parameter; in rare occasions we use this to modify a
template from multiple sources during compile phase, i.e.: credentials
for a database, additional URLs for a site, etc. You can use Bryan
Berry’s chef-rewind [0] or the old-school form for this:

r = begin
  resources(:template => 'the_resource_name')
rescue Chef::Exceptions::ResourceNotFound
   template('the_resource_name') { .... }
end

r.variables(
 foo: 'bar',
 baz: 'bong )

Cheers, HTH!

AJ

[0] https://github.com/bryanwb/chef-rewind

On 20 March 2013 04:26, Kevin Keane Subscription
subscription@kkeane.com wrote:

+1

No, it’s not a practical edict, and in many cases it can be very
counterproductive and even cause security problems.

The chef philosophy is that you don’t edit partial files, but only generate
a complete copy of a configuration file. I can understand where that’s
coming from. In many cases, it does make sense - but in my mind this
approach is vastly overused in chef. In a few cases, chef itself actually
breaks the philosophy - that’s why there are resources to manage /etc/fstab
and /etc/exports, for instance.

The main issue I have with it isn’t even with global files such as
/etc/sysctl.conf or /etc/hosts.

The far bigger elephant in the room is that distribution vendors usually
provide well-tuned configuration files that shouldn’t be clobbered, but
rather surgically manipulated.

That said, the surgical approach can be very difficult to implement in a
generalized way. Multiline context-sensitive edits are non-trivial; for
instance, the nginx configuration file. You have to be able to detect not
just whether the appropriate entry exists with all the multiple lines in
place, but also that it actually is in the right place; you can’t just
append it to the end of a file and expect it to work.

To make matters even more complicated, nginx configuration file support
including other files.

So to make this work, you have three options:

  • Create a one-off solution that encodes a lot of assumptions about how your
    particular configuration file is structured.
  • Implement a complete parser that understands nginix configuration files
    and translates it into something like json/XML/DOM, then allows you to
    manipulate it, and write out only the manipulated parts.
  • Generate the complete configuration file.

So, in this specific scenario, and many similar ones, I can understand where
AJ’s statement came from. It’s just vastly overused.

-----Original message-----

From:Michael Della Bitta michael.della.bitta@appinions.com
Sent: Tuesday 19th March 2013 7:29
To: chef@lists.opscode.com
Subject: [chef] Re: Re: Appending to an existing file, i.e nginx site

On Mon, Mar 18, 2013 at 4:24 PM, AJ Christensen aj@junglist.gen.nz
wrote:

Please don’t edit files that already exist. Please never do this.

Is that really a practical edict? What about files like
/etc/sysctl.conf needing to be written to by multiple recipes?

FileEdit works, but it gets a bad rap on this list. Chef’s going to
need a canonical and supported way to do this…

Michael Della Bitta


Appinions
18 East 41st Street, 2nd Floor
New York, NY 10017-6271

www.appinions.com

Where Influence Isn’t a Game


#12

I suspect there might simply be different philosophies (and markets) behind different distributions here. As far as I can tell, RedHat seems to do a very good job in this area, probably because of their different market. When they put Enterprise in their name, and expect people to pay for support, this is one of the areas they pay particular attention to.

Ubuntu tries to stay at the very cutting edge of technology, and targets more desktops than servers (although plenty people do use it on servers). Given this different focus, what you are saying doesn’t completely surprise me.

I haven’t played around with Fedora much lately, but I suspect it is more like Ubuntu; it is where these types of things get shaken out before it makes it into the Enterprise Linux version.

Another consideration is simply manpower. Can the Opscode+chef community in the long run sustain rewriting all kinds of configuration files from scratch (and maintain that across version updates!), when Canonical+Ubuntu community apprently doesn’t do a good job despite close to ten times the number of employees? And RedHat has yet 10 times as many employees. That concern is why I would much rather see chef stay as close to the native configuration as possible.

-----Original message-----

From:AJ Christensen <aj@junglist.gen.nz mailto:aj@junglist.gen.nz >
Sent: Tuesday 19th March 2013 12:53
To: chef <chef@lists.opscode.com mailto:chef@lists.opscode.com >
Subject: [chef] Re: RE: Re: Re: Appending to an existing file, i.e nginx site

All,

In the case of Ubuntu (and Debian, I suppose), this probably trickles
down to Mint as well, Heavy Water operations often find that the
defaults are poorly constructed examples copied straight from the
source code of the original software; even moreso could cause great
damage if deployed and used in a system without understanding the
implications of defaults.

I definitely don’t hold my vendor in such high regard! Frankly; I’d
like to strip or bypass config files (& service definitions) out of
packages altogether. The asset-bundle or omnibus-stack packages we
build are generally created in this manner.

–skip-config-files --skip-services --skip-users --skip-groups
–dont-pretend-to-be-config-management

If the vendor default configuration file never existed; the service
would not start. The service would not even exist!

You wouldn’t have to edit single lines of files contained in the
software you are deploying if it was part of the procedure for
deploying your software (this mirrors 90+% of ours):

  • research config
  • construct a configuration file
  • instrument configuration for automation
  • install stripped packages
  • deploy users and groups
  • deploy configuration
  • deploy services

I hope this offers some insight in-to how we have solved countless
deployments over disparate time-zones totaling Node counts in the
thousands across the prime Unixes.

A final point I would like to make: it is possible to re-open a
template Resource in the Compile DSL and add additional data to the
’variables’ parameter; in rare occasions we use this to modify a
template from multiple sources during compile phase, i.e.: credentials
for a database, additional URLs for a site, etc. You can use Bryan
Berry’s chef-rewind [0] or the old-school form for this:

r = begin
  resources(:template => 'the_resource_name')
rescue Chef::Exceptions::ResourceNotFound
   template('the_resource_name') { .... }
end

r.variables(
 foo: 'bar',
 baz: 'bong )

Cheers, HTH!

AJ

[0] https://github.com/bryanwb/chef-rewind

On 20 March 2013 04:26, Kevin Keane Subscription
<subscription@kkeane.com mailto:subscription@kkeane.com > wrote:

+1

No, it’s not a practical edict, and in many cases it can be very
counterproductive and even cause security problems.

The chef philosophy is that you don’t edit partial files, but only generate
a complete copy of a configuration file. I can understand where that’s
coming from. In many cases, it does make sense - but in my mind this
approach is vastly overused in chef. In a few cases, chef itself actually
breaks the philosophy - that’s why there are resources to manage /etc/fstab
and /etc/exports, for instance.

The main issue I have with it isn’t even with global files such as
/etc/sysctl.conf or /etc/hosts.

The far bigger elephant in the room is that distribution vendors usually
provide well-tuned configuration files that shouldn’t be clobbered, but
rather surgically manipulated.

That said, the surgical approach can be very difficult to implement in a
generalized way. Multiline context-sensitive edits are non-trivial; for
instance, the nginx configuration file. You have to be able to detect not
just whether the appropriate entry exists with all the multiple lines in
place, but also that it actually is in the right place; you can’t just
append it to the end of a file and expect it to work.

To make matters even more complicated, nginx configuration file support
including other files.

So to make this work, you have three options:

  • Create a one-off solution that encodes a lot of assumptions about how your
    particular configuration file is structured.
  • Implement a complete parser that understands nginix configuration files
    and translates it into something like json/XML/DOM, then allows you to
    manipulate it, and write out only the manipulated parts.
  • Generate the complete configuration file.

So, in this specific scenario, and many similar ones, I can understand where
AJ’s statement came from. It’s just vastly overused.

-----Original message-----

From:Michael Della Bitta <michael.della.bitta@appinions.com mailto:michael.della.bitta@appinions.com >
Sent: Tuesday 19th March 2013 7:29
To: chef@lists.opscode.com mailto:chef@lists.opscode.com
Subject: [chef] Re: Re: Appending to an existing file, i.e nginx site

On Mon, Mar 18, 2013 at 4:24 PM, AJ Christensen <aj@junglist.gen.nz mailto:aj@junglist.gen.nz >
wrote:

Please don’t edit files that already exist. Please never do this.

Is that really a practical edict? What about files like
/etc/sysctl.conf needing to be written to by multiple recipes?

FileEdit works, but it gets a bad rap on this list. Chef’s going to
need a canonical and supported way to do this…

Michael Della Bitta


Appinions
18 East 41st Street, 2nd Floor
New York, NY 10017-6271

www.appinions.com http://www.appinions.com

Where Influence Isn’t a Game


#13

On Tue, Mar 19, 2013 at 8:53 PM, AJ Christensen aj@junglist.gen.nz wrote:

All,

In the case of Ubuntu (and Debian, I suppose), this probably trickles
down to Mint as well, Heavy Water operations often find that the
defaults are poorly constructed examples copied straight from the
source code of the original software; even moreso could cause great
damage if deployed and used in a system without understanding the
implications of defaults.

I definitely don’t hold my vendor in such high regard! Frankly; I’d
like to strip or bypass config files (& service definitions) out of
packages altogether. The asset-bundle or omnibus-stack packages we
build are generally created in this manner.

Hear hear! I definitely trust more the Chef community to come up with
reasonable defaults–and still expect to have to tweak it.

Andrea