What is a simple method to arrange e-mail notification of failed convergence?


#1

What is a simple method of arranging to receive e-mail notification
whenever chef-client attempts and fails to converge a node?

I know Chef provides a flexible framework for various handlers,
including exception handlers. At this time, I don’t need flexibility; I
need simplicity. I just want to say, “For any of these nodes, when
convergence fails, send a message to ops-notifications@example.com.” I
hoped I could override in a Chef environment definition a node attribute
used by the chef-client cookbook so that it would configure nodes
as described, but after various searches of the Chef docs, the whole
Web, and this list, I’ve yet to find indication that such is a possibility.

References:


Phil Mocek
https://mocek.org


#2

On Monday, November 24, 2014 at 11:47 AM, Phil Mocek wrote:

What is a simple method of arranging to receive e-mail notification
whenever chef-client attempts and fails to converge a node?

I know Chef provides a flexible framework for various handlers,
including exception handlers. At this time, I don’t need flexibility; I
need simplicity. I just want to say, “For any of these nodes, when
convergence fails, send a message to ops-notifications@example.com (mailto:ops-notifications@example.com).” I
hoped I could override in a Chef environment definition a node attribute
used by the chef-client cookbook so that it would configure nodes
as described, but after various searches of the Chef docs, the whole
Web, and this list, I’ve yet to find indication that such is a possibility.

References:


Phil Mocek
https://mocek.org

Chef delegates all of this kind of thing to handlers (this means the core team doesn’t need to maintain handlers for e.g., chat systems we don’t use), but there are already ones written that do what you’re asking, like this one: https://github.com/kisoku/chef-handler-mail

A super simple example is provided in the docs you linked: https://docs.getchef.com/essentials_handlers.html#syntax if you just want to copy pasta that code.

If you’re interested in other community-contributed handlers, there’s a list at the bottom of the documentation page: https://docs.getchef.com/essentials_handlers.html#community-handlers

Also, there is a chef_handler cookbook that will manage handlers for you, some documentation on that is here: https://docs.getchef.com/resource_chef_handler.html (GH: https://github.com/opscode-cookbooks/chef_handler)

HTH,


Daniel DeLeo


#3

Daniel DeLeo wrote:

there are already ones written that do what you’re asking, like
this one: https://github.com/kisoku/chef-handler-mail

That’s not quite what I’m asking, which is for a simple method to
configure chef-client to send an e-mail to a particular address if
convergence fails. Apparently, to use chef-handler-email, I need
to install one or more gems on client systems (that sounds like a
half-day of dependency hell), or use the provided LWRP in a recipe
of my creation. If that’s the simplest way to get an e-mail from
chef-client on failure, I’m disappointed, and concerned that I
have miscommunicated my goal.

A super simple example is provided in the docs you linked:
https://docs.getchef.com/essentials_handlers.html#syntax if you
just want to copy pasta that code.

That appears to be a simple example of how to write a custom
handler
. I do not want to write a custom handler. All I really
need to customize is the e-mail address to which notifications
should be sent, like cron’s MAILTO setting.

If you’re interested in other community-contributed handlers,
there’s a list at the bottom of the documentation page:
https://docs.getchef.com/essentials_handlers.html#community-handlers

That is preferable to writing my own handler. It still appears to
require writing a recipe. I’m busy writing recipes to deploy and
configure software and virtual infrastructure. I just want to
tell chef-client to send an e-mail when it fails.

Also, there is a chef_handler cookbook that will manage handlers
for you, some documentation on that is here:
https://docs.getchef.com/resource_chef_handler.html (GH:
https://github.com/opscode-cookbooks/chef_handler)

That’s also useful, and also far more complicated than I hoped it
would be just to get chef-client to send an e-mail on failure.

I feel like I must be missing something. My goal, here, seems
likely to be a common one, and Chef generally handles those well
without much work on the user’s part.

Thanks for the suggestions, Daniel. I suspect I will end up using
some of what you referenced.


Phil Mocek
https://mocek.org


#4

Adding a chef handler is the simple method for configuring email notifications from Chef runs. The client doesn’t have a built in mechanism for sending emails considering you may not have even installed an MTA when the Chef run fails.

If you want to rely on something like cron’s MAILTO you can run chef-client on a cron but that won’t help you during the initial bootstrap process. We post our chef failures to our internal chat and Github gists using nothing more than a handler which performs HTTP posts through the Ruby stdlib; you may want to look into doing the same as you don’t need a working MTA for that.

Installing gems on systems is actually very easy given the chef_gem resource. Try using that handler, it shouldn’t take you very long to get working assuming you’re using omnibus.

On Tuesday, November 25, 2014 at 10:54 AM, Phil Mocek wrote:

Daniel DeLeo wrote:

there are already ones written that do what you’re asking, like
this one: https://github.com/kisoku/chef-handler-mail

That’s not quite what I’m asking, which is for a simple method to
configure chef-client to send an e-mail to a particular address if
convergence fails. Apparently, to use chef-handler-email, I need
to install one or more gems on client systems (that sounds like a
half-day of dependency hell), or use the provided LWRP in a recipe
of my creation. If that’s the simplest way to get an e-mail from
chef-client on failure, I’m disappointed, and concerned that I
have miscommunicated my goal.

A super simple example is provided in the docs you linked:
https://docs.getchef.com/essentials_handlers.html#syntax if you
just want to copy pasta that code.

That appears to be a simple example of how to write a custom
handler
. I do not want to write a custom handler. All I really
need to customize is the e-mail address to which notifications
should be sent, like cron’s MAILTO setting.

If you’re interested in other community-contributed handlers,
there’s a list at the bottom of the documentation page:
https://docs.getchef.com/essentials_handlers.html#community-handlers

That is preferable to writing my own handler. It still appears to
require writing a recipe. I’m busy writing recipes to deploy and
configure software and virtual infrastructure. I just want to
tell chef-client to send an e-mail when it fails.

Also, there is a chef_handler cookbook that will manage handlers
for you, some documentation on that is here:
https://docs.getchef.com/resource_chef_handler.html (GH:
https://github.com/opscode-cookbooks/chef_handler)

That’s also useful, and also far more complicated than I hoped it
would be just to get chef-client to send an e-mail on failure.

I feel like I must be missing something. My goal, here, seems
likely to be a common one, and Chef generally handles those well
without much work on the user’s part.

Thanks for the suggestions, Daniel. I suspect I will end up using
some of what you referenced.


Phil Mocek
https://mocek.org


#5

Yo,

All of the suggestions are trivial, less than 5 minutes of work. It’s
your relative knowledge (“Chef skill”) with regard to the ecosystem
that makes them seem “complicated”. I’ve heard this a lot recently,
but you’re the first I’m going to call out on it.

I would probably suggest not being so aggressive in your replies to
the mailing list. Building a chef handler is simple. Building one that
shells to ‘mail’ with a pipe is simple as well. It is not complicated.
@kallistec took the time to respond rapidly to your question with a
handler that @kisoku had already built for you. Many of us build this
software for a living.

I would add a recommendation for using the Pony gem. It looks like
@kisoku’s chef-handler-mail software supports a custom template for
emailing, which is something you’d end up doing eventually anyway: I
bet you’ll want to send data based on the converge itself (updated
resources, which resources, updated resources of a certain type, run
length info, blah blah.)

The “half day of dependency hell” to which you so promptly refer is
solved with a few Chef resources in a cookbook, probably designed for
installing the email handler:


chef_gem 'chef-handler-mail'
require 'chef/handler/mail'

chef_handler 'MailHandler' do
   source 'chef/handler/mail'
   arguments :to_address => "root"
   action :nothing
end.run_action(:enable)

The dependency on ‘pony’ is handled through the chef_gem installation
of chef-handler-mail. There are no additional deps. [0]

Please, do the needful.

cheers,

–aj

[0] https://rubygems.org/gems/chef-handler-mail

On Wed, Nov 26, 2014 at 7:54 AM, Phil Mocek phil-lists@mocek.org wrote:

Daniel DeLeo wrote:

there are already ones written that do what you’re asking, like
this one: https://github.com/kisoku/chef-handler-mail

That’s not quite what I’m asking, which is for a simple method to
configure chef-client to send an e-mail to a particular address if
convergence fails. Apparently, to use chef-handler-email, I need
to install one or more gems on client systems (that sounds like a
half-day of dependency hell), or use the provided LWRP in a recipe
of my creation. If that’s the simplest way to get an e-mail from
chef-client on failure, I’m disappointed, and concerned that I
have miscommunicated my goal.

A super simple example is provided in the docs you linked:
https://docs.getchef.com/essentials_handlers.html#syntax if you
just want to copy pasta that code.

That appears to be a simple example of how to write a custom
handler
. I do not want to write a custom handler. All I really
need to customize is the e-mail address to which notifications
should be sent, like cron’s MAILTO setting.

If you’re interested in other community-contributed handlers,
there’s a list at the bottom of the documentation page:
https://docs.getchef.com/essentials_handlers.html#community-handlers

That is preferable to writing my own handler. It still appears to
require writing a recipe. I’m busy writing recipes to deploy and
configure software and virtual infrastructure. I just want to
tell chef-client to send an e-mail when it fails.

Also, there is a chef_handler cookbook that will manage handlers
for you, some documentation on that is here:
https://docs.getchef.com/resource_chef_handler.html (GH:
https://github.com/opscode-cookbooks/chef_handler)

That’s also useful, and also far more complicated than I hoped it
would be just to get chef-client to send an e-mail on failure.

I feel like I must be missing something. My goal, here, seems
likely to be a common one, and Chef generally handles those well
without much work on the user’s part.

Thanks for the suggestions, Daniel. I suspect I will end up using
some of what you referenced.


Phil Mocek
https://mocek.org


#6

On Nov 25, 2014, at 12:54 PM, Phil Mocek phil-lists@mocek.org wrote:

Daniel DeLeo wrote:

there are already ones written that do what you’re asking, like
this one: https://github.com/kisoku/chef-handler-mail

That’s not quite what I’m asking, which is for a simple method to
configure chef-client to send an e-mail to a particular address if
convergence fails.

For chef-client to do that, the node would have to have some sort of SMTP server installed, and a whole bunch of other dependencies. That kind of thing isn’t built into Chef – if you want notifications sent on behalf of a chef-client run, then you need some sort of chef-handler.

Keep in mind that chef-client is not a standalone program. It’s part of a much larger system, and there’s huge chunks to that system that chef-client doesn’t make any attempt to address – it relies on those other parts of the system for that. When you’re managing a cluster of thousands of systems, it doesn’t make sense to duplicate that infrastructure on each and every one of them – instead, you implement that part of the system in one place, and let all the other parts of the system leverage that.


Brad Knowles brad@shub-internet.org
LinkedIn Profile: http://tinyurl.com/y8kpxu


#7

What we use at Rosetta is the airbrake handler. This sends a message to our
internal Hoptoad (now called Airbrake for the cloud version IIUC) server,
which in turn sends out an email. Hoptoad also handles correlation of
events so it can glob up a bunch of items together (albeit with multiple
notifications, which is nice) and you can say that all the events for that
type of error are fixed by one issue.


~~ StormeRider ~~

“Every world needs its heroes […] They inspire us to be better than we
are. And they protect from the darkness that’s just around the corner.”

(from Smallville Season 6x1: “Zod”)

On why I hate the phrase “that’s so lame”… http://bit.ly/Ps3uSS

On Mon, Nov 24, 2014 at 11:47 AM, Phil Mocek phil-lists@mocek.org wrote:

What is a simple method of arranging to receive e-mail notification
whenever chef-client attempts and fails to converge a node?

I know Chef provides a flexible framework for various handlers,
including exception handlers. At this time, I don’t need flexibility; I
need simplicity. I just want to say, “For any of these nodes, when
convergence fails, send a message to ops-notifications@example.com.” I
hoped I could override in a Chef environment definition a node attribute
used by the chef-client cookbook so that it would configure nodes
as described, but after various searches of the Chef docs, the whole
Web, and this list, I’ve yet to find indication that such is a possibility.

References:


Phil Mocek
https://mocek.org