Need cfengine like copy to achieve Self Healing


#1

I’m trying to make my systems self healing with chef. I’ve been doing
this for many years with cfengine. I think chef needs a cfengine like
copy resource to make it easier to maintain a self healing environment.

In cfengine 2, the copy action has a recursive option that lets you
preserve permissions and create classes if files were changed. There
doesn’t appear to be any resource to do this in chef. You could use a
tar file to preserve permissions, but then the copy isn’t idempotent.
There would be no easy way to tell if files on the destination system
have changed. You could put the files into a package like an rpm, but
those packages are more difficult to maintain. I have a lot of
directories that I need to copy and maintain. I don’t want to create an
rpm for every directory.

It would also be nice if chef didn’t have to import all of the files you
want to copy. I’d like to be able to change a directory tree without
having to tell Chef about the change.

To get around this limitation, I’m going to try to create an "rsync"
LWRP in chef.

Steve Caissie
Director of Operations

Vlingo

Vlingo’s http://www.vlingo.com/ intelligent voice applications turn
your words into action. Simply speak to your phone to do just about
anything on the go! Visit www.Vlingo.com http://www.Vlingo.com to get
started.


#2

On Sat, Feb 5, 2011 at 9:59 AM, Steve Caissie stevec@vlingo.com wrote:

I’m trying to make my systems self healing with chef. I’ve been doing this
for many years with cfengine. I think chef needs a cfengine like copy
resource to make it easier to maintain a self healing environment.

In cfengine 2, the copy action has a recursive option that lets you preserve
permissions and create classes if files were changed. There doesn’t appear
to be any resource to do this in chef. You could use a tar file to preserve
permissions, but then the copy isn’t idempotent. There would be no easy way
to tell if files on the destination system have changed. You could put the
files into a package like an rpm, but those packages are more difficult to
maintain. I have a lot of directories that I need to copy and maintain. I
don’t want to create an rpm for every directory.

It would also be nice if chef didn’t have to import all of the files you
want to copy. I’d like to be able to change a directory tree without having
to tell Chef about the change.

To get around this limitation, I’m going to try to create an “rsync” LWRP in
chef.

Can you explain a bit more about how what you’re looking for differs
from the existing remote directory[1] resource? Do you just want a
remote directory that uses a local (i.e., already exists elsewhere on
the filesystem) source? Do you need permissions that are more varied
than remote directory provides?

Steve Caissie
Director of Operations

Vlingo

Thanks,
Dan DeLeo


#3

I can apply the same mode to all of the files in a directory with remote_directory, but I don’t see any way to preserve the source directory’s permissions.

Ideally, I’d like to be able to copy from a local or remote source and preserve permissions. Cfengine offers this with the copy action.

Steve

-----Original Message-----
From: ddeleo@kallistec.com [mailto:ddeleo@kallistec.com] On Behalf Of Daniel DeLeo
Sent: Saturday, February 05, 2011 1:24 PM
To: chef@lists.opscode.com
Subject: [chef] Re: Need cfengine like copy to achieve Self Healing

On Sat, Feb 5, 2011 at 9:59 AM, Steve Caissie stevec@vlingo.com wrote:

I’m trying to make my systems self healing with chef. I’ve been doing this
for many years with cfengine. I think chef needs a cfengine like copy
resource to make it easier to maintain a self healing environment.

In cfengine 2, the copy action has a recursive option that lets you preserve
permissions and create classes if files were changed. There doesn’t appear
to be any resource to do this in chef. You could use a tar file to preserve
permissions, but then the copy isn’t idempotent. There would be no easy way
to tell if files on the destination system have changed. You could put the
files into a package like an rpm, but those packages are more difficult to
maintain. I have a lot of directories that I need to copy and maintain. I
don’t want to create an rpm for every directory.

It would also be nice if chef didn’t have to import all of the files you
want to copy. I’d like to be able to change a directory tree without having
to tell Chef about the change.

To get around this limitation, I’m going to try to create an “rsync” LWRP in
chef.

Can you explain a bit more about how what you’re looking for differs
from the existing remote directory[1] resource? Do you just want a
remote directory that uses a local (i.e., already exists elsewhere on
the filesystem) source? Do you need permissions that are more varied
than remote directory provides?

Steve Caissie
Director of Operations

Vlingo

Thanks,
Dan DeLeo


#4

I’m confused about how recursively copying a directory and preserving
permissions provides “self healing”. Can you elaborate on how that
feature of cfengine provides self healing in your environment? It is
possible there is a more idiomatic way to accomplish what you are
after in chef.

On Sat, Feb 5, 2011 at 2:06 PM, Steve Caissie stevec@vlingo.com wrote:

I can apply the same mode to all of the files in a directory with remote_directory, but I don’t see any way to preserve the source directory’s permissions.

Ideally, I’d like to be able to copy from a local or remote source and preserve permissions. Cfengine offers this with the copy action.

Steve

-----Original Message-----
From: ddeleo@kallistec.com [mailto:ddeleo@kallistec.com] On Behalf Of Daniel DeLeo
Sent: Saturday, February 05, 2011 1:24 PM
To: chef@lists.opscode.com
Subject: [chef] Re: Need cfengine like copy to achieve Self Healing

On Sat, Feb 5, 2011 at 9:59 AM, Steve Caissie stevec@vlingo.com wrote:

I’m trying to make my systems self healing with chef. I’ve been doing this
for many years with cfengine. I think chef needs a cfengine like copy
resource to make it easier to maintain a self healing environment.

In cfengine 2, the copy action has a recursive option that lets you preserve
permissions and create classes if files were changed. There doesn’t appear
to be any resource to do this in chef. You could use a tar file to preserve
permissions, but then the copy isn’t idempotent. There would be no easy way
to tell if files on the destination system have changed. You could put the
files into a package like an rpm, but those packages are more difficult to
maintain. I have a lot of directories that I need to copy and maintain. I
don’t want to create an rpm for every directory.

It would also be nice if chef didn’t have to import all of the files you
want to copy. I’d like to be able to change a directory tree without having
to tell Chef about the change.

To get around this limitation, I’m going to try to create an “rsync” LWRP in
chef.

Can you explain a bit more about how what you’re looking for differs
from the existing remote directory[1] resource? Do you just want a
remote directory that uses a local (i.e., already exists elsewhere on
the filesystem) source? Do you need permissions that are more varied
than remote directory provides?

Steve Caissie
Director of Operations

Vlingo

Thanks,
Dan DeLeo


#5

Sorry, I’ll try to provide a clear example:

I’ve got a 3rd party monitoring application that requires me to deploy the agents and plugins to all of the systems on my network. Here’s the cfengine syntax to do that:

monitoring_agent_class::
    /opt/agent-software
        dest=/opt/agent-software
        owner=root
        group=root
        recurse=inf
        purge=true
        server=monitoring_server
        define=restart_agent
    /var/lib/agent-plugins
        dest=/opt/agent-plugins
        owner=root
        group=root
        recurse=inf
        purge=true
        server=monitoring_server
        define=restart_agent

This tells each system on the network to copy the monitoring agent and plugins from “monitoring_server”. There are multiple levels of directories under /opt/agent-software. It is “self healing”, because any missing or changed files on the destination servers are fixed next time cfengine runs. Only the files that need to change are updated. Cfengine guarantees that the source and destination directories are an exact match. I can make a quick change to an agent-plugin and next time cfengine runs, that change is installed on each server.

I think these are my choices if I want to do this in Chef:

  1. remote_directory resource - It doesn’t preserve file permissions. If I set files_mode=755 on the files in /opt/agent-software, I’d have execute permissions on configuration files that should be read-only.

  2. tar the directories and use the cookbook_file resource - If I want to be 100% sure that the destination isn’t corrupt, I’d have to untar the tar file on every chef run and restart the agent. With the cfengine configuration above, the agent would only be restarted if an individual file gets updated.

  3. Create an rpm for agent-software and another for agent-plugins - To make this “self healing”, chef would need a “verify” action in the package resource. I’d use the “verify” action to verify that the target system’s install matches the contents of the rpm. If it doesn’t match, I’d install the rpm with a --force option. I also think creating and maintaining an rpm is a lot more complex that the cfengine setup above.

Are there any other Chef options for doing this?

Steve

-----Original Message-----
From: mike@hales.ws [mailto:mike@hales.ws] On Behalf Of Michael Hale
Sent: Saturday, February 05, 2011 2:12 PM
To: chef
Subject: [chef] Re: RE: Re: Need cfengine like copy to achieve Self Healing

I’m confused about how recursively copying a directory and preserving
permissions provides “self healing”. Can you elaborate on how that
feature of cfengine provides self healing in your environment? It is
possible there is a more idiomatic way to accomplish what you are
after in chef.

On Sat, Feb 5, 2011 at 2:06 PM, Steve Caissie stevec@vlingo.com wrote:

I can apply the same mode to all of the files in a directory with remote_directory, but I don’t see any way to preserve the source directory’s permissions.

Ideally, I’d like to be able to copy from a local or remote source and preserve permissions. Cfengine offers this with the copy action.

Steve

-----Original Message-----
From: ddeleo@kallistec.com [mailto:ddeleo@kallistec.com] On Behalf Of Daniel DeLeo
Sent: Saturday, February 05, 2011 1:24 PM
To: chef@lists.opscode.com
Subject: [chef] Re: Need cfengine like copy to achieve Self Healing

On Sat, Feb 5, 2011 at 9:59 AM, Steve Caissie stevec@vlingo.com wrote:

I’m trying to make my systems self healing with chef. I’ve been doing this
for many years with cfengine. I think chef needs a cfengine like copy
resource to make it easier to maintain a self healing environment.

In cfengine 2, the copy action has a recursive option that lets you preserve
permissions and create classes if files were changed. There doesn’t appear
to be any resource to do this in chef. You could use a tar file to preserve
permissions, but then the copy isn’t idempotent. There would be no easy way
to tell if files on the destination system have changed. You could put the
files into a package like an rpm, but those packages are more difficult to
maintain. I have a lot of directories that I need to copy and maintain. I
don’t want to create an rpm for every directory.

It would also be nice if chef didn’t have to import all of the files you
want to copy. I’d like to be able to change a directory tree without having
to tell Chef about the change.

To get around this limitation, I’m going to try to create an “rsync” LWRP in
chef.

Can you explain a bit more about how what you’re looking for differs
from the existing remote directory[1] resource? Do you just want a
remote directory that uses a local (i.e., already exists elsewhere on
the filesystem) source? Do you need permissions that are more varied
than remote directory provides?

Steve Caissie
Director of Operations

Vlingo

Thanks,
Dan DeLeo


#6

RSync might also do this, so you could run that from Chef. You’d need a network connection at the time, of course.

— On Sat, 2/5/11, Steve Caissie stevec@vlingo.com wrote:

From: Steve Caissie stevec@vlingo.com
Subject: [chef] RE: Re: RE: Re: Need cfengine like copy to achieve Self Healing
To: chef@lists.opscode.com
Date: Saturday, February 5, 2011, 1:44 PM
Sorry, I’ll try to provide a clear
example:

I’ve got a 3rd party monitoring application that requires
me to deploy the agents and plugins to all of the systems on
my network. Here’s the cfengine syntax to do that:

monitoring_agent_class::
    /opt/agent-software

dest=/opt/agent-software
owner=root
group=root
recurse=inf
purge=true

server=monitoring_server

define=restart_agent
/var/lib/agent-plugins

dest=/opt/agent-plugins
owner=root
group=root
recurse=inf
purge=true

server=monitoring_server

define=restart_agent

This tells each system on the network to copy the
monitoring agent and plugins from “monitoring_server”. There
are multiple levels of directories under
/opt/agent-software. It is “self healing”, because any
missing or changed files on the destination servers are
fixed next time cfengine runs. Only the files that need to
change are updated. Cfengine guarantees that the source and
destination directories are an exact match. I can make
a quick change to an agent-plugin and next time cfengine
runs, that change is installed on each server.

I think these are my choices if I want to do this in Chef:

  1. remote_directory resource - It doesn’t preserve file
    permissions. If I set files_mode=755 on the files in
    /opt/agent-software, I’d have execute permissions on
    configuration files that should be read-only.

  2. tar the directories and use the cookbook_file resource -
    If I want to be 100% sure that the destination isn’t
    corrupt, I’d have to untar the tar file on every chef run
    and restart the agent. With the cfengine configuration
    above, the agent would only be restarted if an individual
    file gets updated.

  3. Create an rpm for agent-software and another for
    agent-plugins - To make this “self healing”, chef would need
    a “verify” action in the package resource. I’d use the
    "verify" action to verify that the target system’s install
    matches the contents of the rpm. If it doesn’t match,
    I’d install the rpm with a --force option. I also
    think creating and maintaining an rpm is a lot more complex
    that the cfengine setup above.

Are there any other Chef options for doing this?

Steve

-----Original Message-----
From: mike@hales.ws
[mailto:mike@hales.ws] On
Behalf Of Michael Hale
Sent: Saturday, February 05, 2011 2:12 PM
To: chef
Subject: [chef] Re: RE: Re: Need cfengine like copy to
achieve Self Healing

I’m confused about how recursively copying a directory and
preserving
permissions provides “self healing”. Can you elaborate on
how that
feature of cfengine provides self healing in your
environment? It is
possible there is a more idiomatic way to accomplish what
you are
after in chef.

On Sat, Feb 5, 2011 at 2:06 PM, Steve Caissie stevec@vlingo.com
wrote:

I can apply the same mode to all of the files in a
directory with remote_directory, but I don’t see any way to
preserve the source directory’s permissions.

Ideally, I’d like to be able to copy from a local or
remote source and preserve permissions. Cfengine offers this
with the copy action.

Steve

-----Original Message-----
From: ddeleo@kallistec.com
[mailto:ddeleo@kallistec.com]
On Behalf Of Daniel DeLeo
Sent: Saturday, February 05, 2011 1:24 PM
To: chef@lists.opscode.com
Subject: [chef] Re: Need cfengine like copy to achieve
Self Healing

On Sat, Feb 5, 2011 at 9:59 AM, Steve Caissie stevec@vlingo.com
wrote:

I’m trying to make my systems self healing with
chef. I’ve been doing this

for many years with cfengine. I think chef needs a
cfengine like copy

resource to make it easier to maintain a self
healing environment.

In cfengine 2, the copy action has a recursive
option that lets you preserve

permissions and create classes if files were
changed. There doesn’t appear

to be any resource to do this in chef. You could
use a tar file to preserve

permissions, but then the copy isn’t idempotent.
There would be no easy way

to tell if files on the destination system have
changed. You could put the

files into a package like an rpm, but those
packages are more difficult to

maintain. I have a lot of directories that I
need to copy and maintain. I

don’t want to create an rpm for every directory.

It would also be nice if chef didn’t have to
import all of the files you

want to copy. I’d like to be able to change a
directory tree without having

to tell Chef about the change.

To get around this limitation, I’m going to try to
create an “rsync” LWRP in

chef.

Can you explain a bit more about how what you’re
looking for differs
from the existing remote directory[1] resource? Do you
just want a
remote directory that uses a local (i.e., already
exists elsewhere on
the filesystem) source? Do you need permissions that
are more varied
than remote directory provides?

Steve Caissie
Director of Operations

Vlingo

Thanks,
Dan DeLeo


#7

I’ve been playing around with this kind of stuff myself.

Inspired by reading this:

And more specifically this:

Check out this cookbook here for a (not quite all the way working) example:

The “copying” functionality you’re asking about is achieved by
allowing the root accounts within the closure generate and place their
pub keys in authorized_keys via an ohai plugin and a small utility
cookbook

Then, within the actual server.rb, use node attributes and searching
to “negotiate” for master and go into an “if master” , "unless master"
construct to decide what to do.

In the event that the “master” disappears, others within the system
will take over its duties, achieving self healing via convergence
within the system, as long as something deletes the master node from
chef server. (nagios trigger or manual intervention)

YMMV

-s


#8

If these configuration files are managed completely outside chef (not
via template resources) then rsync sounds like the perfect tool for
the job. As another poster indicated you could add some smarts to your
cookbooks to be able to recover from a defunct master node.

On Sat, Feb 5, 2011 at 4:44 PM, Steve Caissie stevec@vlingo.com wrote:

Sorry, I’ll try to provide a clear example:

I’ve got a 3rd party monitoring application that requires me to deploy the agents and plugins to all of the systems on my network. Here’s the cfengine syntax to do that:

monitoring_agent_class::
/opt/agent-software
dest=/opt/agent-software
owner=root
group=root
recurse=inf
purge=true
server=monitoring_server
define=restart_agent
/var/lib/agent-plugins
dest=/opt/agent-plugins
owner=root
group=root
recurse=inf
purge=true
server=monitoring_server
define=restart_agent

This tells each system on the network to copy the monitoring agent and plugins from “monitoring_server”. There are multiple levels of directories under /opt/agent-software. It is “self healing”, because any missing or changed files on the destination servers are fixed next time cfengine runs. Only the files that need to change are updated. Cfengine guarantees that the source and destination directories are an exact match. I can make a quick change to an agent-plugin and next time cfengine runs, that change is installed on each server.

I think these are my choices if I want to do this in Chef:

  1. remote_directory resource - It doesn’t preserve file permissions. If I set files_mode=755 on the files in /opt/agent-software, I’d have execute permissions on configuration files that should be read-only.

  2. tar the directories and use the cookbook_file resource - If I want to be 100% sure that the destination isn’t corrupt, I’d have to untar the tar file on every chef run and restart the agent. With the cfengine configuration above, the agent would only be restarted if an individual file gets updated.

  3. Create an rpm for agent-software and another for agent-plugins - To make this “self healing”, chef would need a “verify” action in the package resource. I’d use the “verify” action to verify that the target system’s install matches the contents of the rpm. If it doesn’t match, I’d install the rpm with a --force option. I also think creating and maintaining an rpm is a lot more complex that the cfengine setup above.

Are there any other Chef options for doing this?

Steve

-----Original Message-----
From: mike@hales.ws [mailto:mike@hales.ws] On Behalf Of Michael Hale
Sent: Saturday, February 05, 2011 2:12 PM
To: chef
Subject: [chef] Re: RE: Re: Need cfengine like copy to achieve Self Healing

I’m confused about how recursively copying a directory and preserving
permissions provides “self healing”. Can you elaborate on how that
feature of cfengine provides self healing in your environment? It is
possible there is a more idiomatic way to accomplish what you are
after in chef.

On Sat, Feb 5, 2011 at 2:06 PM, Steve Caissie stevec@vlingo.com wrote:

I can apply the same mode to all of the files in a directory with remote_directory, but I don’t see any way to preserve the source directory’s permissions.

Ideally, I’d like to be able to copy from a local or remote source and preserve permissions. Cfengine offers this with the copy action.

Steve

-----Original Message-----
From: ddeleo@kallistec.com [mailto:ddeleo@kallistec.com] On Behalf Of Daniel DeLeo
Sent: Saturday, February 05, 2011 1:24 PM
To: chef@lists.opscode.com
Subject: [chef] Re: Need cfengine like copy to achieve Self Healing

On Sat, Feb 5, 2011 at 9:59 AM, Steve Caissie stevec@vlingo.com wrote:

I’m trying to make my systems self healing with chef. I’ve been doing this
for many years with cfengine. I think chef needs a cfengine like copy
resource to make it easier to maintain a self healing environment.

In cfengine 2, the copy action has a recursive option that lets you preserve
permissions and create classes if files were changed. There doesn’t appear
to be any resource to do this in chef. You could use a tar file to preserve
permissions, but then the copy isn’t idempotent. There would be no easy way
to tell if files on the destination system have changed. You could put the
files into a package like an rpm, but those packages are more difficult to
maintain. I have a lot of directories that I need to copy and maintain. I
don’t want to create an rpm for every directory.

It would also be nice if chef didn’t have to import all of the files you
want to copy. I’d like to be able to change a directory tree without having
to tell Chef about the change.

To get around this limitation, I’m going to try to create an “rsync” LWRP in
chef.

Can you explain a bit more about how what you’re looking for differs
from the existing remote directory[1] resource? Do you just want a
remote directory that uses a local (i.e., already exists elsewhere on
the filesystem) source? Do you need permissions that are more varied
than remote directory provides?

Steve Caissie
Director of Operations

Vlingo

Thanks,
Dan DeLeo


#9

Sean,

Your cookbook looks interesting. Please keep me posted on the progress.
For now, I think I’m going to put the source files on load balanced
servers running the rsync daemon. The security isn’t great, but it’s a
good first step.

Thanks for the pointers to the papers.

Steve

-----Original Message-----
From: Sean OMeara [mailto:someara@gmail.com]
Sent: Saturday, February 05, 2011 6:53 PM
To: chef@lists.opscode.com
Subject: [chef] Re: Need cfengine like copy to achieve Self Healing

I’ve been playing around with this kind of stuff myself.

Inspired by reading this:

And more specifically this:

Check out this cookbook here for a (not quite all the way working)
example:

https://github.com/someara/cookbooks-affs/blob/master/freeipa/recipes/se
rver.rb

The “copying” functionality you’re asking about is achieved by
allowing the root accounts within the closure generate and place their
pub keys in authorized_keys via an ohai plugin and a small utility
cookbook


/plugins/rootuser_user_rsa_public.rb

https://github.com/someara/cookbooks-affs/blob/master/sshroot2rootssh/re
cipes/default.rb

Then, within the actual server.rb, use node attributes and searching
to “negotiate” for master and go into an “if master” , "unless master"
construct to decide what to do.

In the event that the “master” disappears, others within the system
will take over its duties, achieving self healing via convergence
within the system, as long as something deletes the master node from
chef server. (nagios trigger or manual intervention)

YMMV

-s