Sending via SMTP the fully log of a node when chef run failed

Chef Infra (archive)
1

Hi partners,

I am trying to send the log of a node when chef run failed with this code:
mail_to = node["mail"]["to"]

Chef.event_handler do
  on :run_failed do
    HandlerSendEmail::Helper.new.send_email mail_to
  end
end

The HandlerSendEmail::Helper.new.send_email code is:

module HandlerSendEmail
  require 'net/smtp'
  class Helper
    def unindent string
      first = string[/\A\s*/]
      string.gsub /^#{first}/, ''
    end

    def send_email(to,opts={})
      opts[:server]      ||= 'ipaddress'
      opts[:from]        ||= 'sender'
      opts[:from_alias]  ||= 'Chef Reporter'
      opts[:subject]     ||= "Chef Run on Node #{Chef.run_context.node.name}"
      opts[:message]     ||= "Chef run failed at #{Time.now.iso8601}, check the attached log."

      filename = "C:\\chef\\log-#{Chef.run_context.node.name}"
      # Read a file and encode it into base64 format
      encodedcontent = [File.read(filename)].pack("m")   # base64

      marker = "AUNIQUEMARKER"

      # Define the main headers.
      header = <<-HEADER
        From: #{opts[:from_alias]} <#{opts[:from]}>
        To: <#{to}>
        Subject: #{opts[:subject]}
        MIME-Version: 1.0
        Content-Type: multipart/mixed; boundary=#{marker}
        --#{marker}
      HEADER

      # Define the message action
      body = <<-BODY
        Content-Type: text/plain
        Content-Transfer-Encoding:8bit

        #{opts[:message]}
        --#{marker}
      BODY

      # Define the attachment section
      attached = <<-ATTACHED
        Content-Type: multipart/mixed; name=\"#{filename}\"
        Content-Transfer-Encoding:base64
        Content-Disposition: attachment; filename="#{filename}"

        #{encodedcontent}
        --#{marker}--
      ATTACHED

      mailtext = unindent header + body + attached
      Net::SMTP.start(opts[:server], 25) do |smtp|
         smtp.send_message mailtext, opts[:from], to
      end
    end
  end
end

The module work but it send the log immediately and the log’s content don’t show me what is the error. For example:

INFO: *** Chef 12.11.18 ***
INFO: Platform: x64-mingw32
INFO: Chef-client pid: 1252
INFO: Run List is [recipe[deploy_war]]
INFO: Run List expands to [deploy_war]
INFO: Starting Chef Run for PosW10
INFO: Running start handlers
INFO: Start handlers complete.
INFO: Loading cookbooks [deploy_war@1.0.3, windows@1.44.1, chef_handler@1.4.0]
INFO: Storing updated cookbooks/deploy_war/recipes/default.rb in the cache.
INFO: Processing chef_gem[mechanize] action install (deploy_war::prepare line 12)
INFO: Processing file[C:\chef\redsis-validator.pem] action delete (deploy_war::prepare line 18)
INFO: Processing powershell_script[Start SQLSERVER] action run (deploy_war::prepare line 24)
INFO: powershell_script[Start SQLSERVER] ran successfully
INFO: Processing ruby_block[wait for sql service to start] action run (deploy_war::prepare line 29)
INFO: Running queued delayed notifications before re-raising exception
ERROR: Running exception handlers
ERROR: Exception handlers complete
INFO: Sending resource update report (run-id: 6706d426-8cb3-45b1-a72b-d9d0cf5c276f)

As you can see don’t show me the error like normal log, which is:

INFO: *** Chef 12.11.18 ***
INFO: Platform: x64-mingw32
INFO: Chef-client pid: 1252
INFO: Run List is [recipe[deploy_war]]
INFO: Run List expands to [deploy_war]
INFO: Starting Chef Run for PosW10
INFO: Running start handlers
INFO: Start handlers complete.
INFO: Loading cookbooks [deploy_war@1.0.3, windows@1.44.1, chef_handler@1.4.0]
INFO: Storing updated cookbooks/deploy_war/recipes/default.rb in the cache.
INFO: Processing chef_gem[mechanize] action install (deploy_war::prepare line 12)
INFO: Processing file[C:\chef\redsis-validator.pem] action delete (deploy_war::prepare line 18)
INFO: Processing powershell_script[Start SQLSERVER] action run (deploy_war::prepare line 24)
INFO: powershell_script[Start SQLSERVER] ran successfully
INFO: Processing ruby_block[wait for sql service to start] action run (deploy_war::prepare line 29)
INFO: Running queued delayed notifications before re-raising exception
ERROR: Running exception handlers
ERROR: Exception handlers complete
INFO: Sending resource update report (run-id: 6706d426-8cb3-45b1-a72b-d9d0cf5c276f)
FATAL: Stacktrace dumped to C:/chef/cache/chef-stacktrace.out
FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
FATAL: NameError: ruby_block[wait for sql service to start] (deploy_war::prepare line 29) had an error: NameError: uninitialized constant Chef::Recipe::Tols

Some help to send the log when it’s fully completed ? (The error is intentional to test the module)

2

Handlers necessarily run before the exception is printed, because exceptions can happen in the handlers themselves and Chef needs to report all of this. The handler will have access to the exception that failed the run via some methods it inherits. Those are defined here: https://github.com/chef/chef/blob/6a0927c62e8c6d6e466a6159d82b6f2be8b5c52d/lib/chef/handler.rb#L192-L204

There is a simple example in the code that shows how you might use them here:

HTH

3

Hi @kallistec,

When I use run_status.formatted_exception and Array(backtrace).join("\n") get this:

FATAL: NoMethodError: undefined method formatted_exception for nil:NilClass

4

Can you share a code snippet? Can't tell if you're possibly calling that in a context where the method receiver is wrong or if the handler DSL is missing some code to pass that in for you.

5

This is the code that I have in helpers.rb into /libraries:

module HandlerSendEmail
  require 'net/smtp'
  class Helper < Chef::Handler
    def unindent string
      first = string[/\A\s*/]
      string.gsub /^#{first}/, ''
    end

    def send_email(to,opts={})
      opts[:server]      ||= 'ipAddress'
      opts[:from]        ||= 'sender'
      opts[:from_alias]  ||= 'Chef Reporter'
      opts[:subject]     ||= "Chef Run on Node #{Chef.run_context.node.name}"
      opts[:message]     ||= "Chef run failed at #{Time.now.iso8601}, check the attached log.\n#{run_status.formatted_exception}\n#{Array(run_status.backtrace).join('\n')}"

      filename = "C:\\chef\\log-#{Chef.run_context.node.name}"
      # Read a file and encode it into base64 format
      encodedcontent = [File.read(filename)].pack("m")   # base64

      marker = "AUNIQUEMARKER"

      # Define the main headers.
      header = <<-HEADER
        From: #{opts[:from_alias]} <#{opts[:from]}>
        To: <#{to}>
        Subject: #{opts[:subject]}
        MIME-Version: 1.0
        Content-Type: multipart/mixed; boundary=#{marker}
        --#{marker}
      HEADER

      # Define the message action
      body = <<-BODY
        Content-Type: text/plain
        Content-Transfer-Encoding:8bit

        #{opts[:message]}
        --#{marker}
      BODY

      # Define the attachment section
      attached = <<-ATTACHED
        Content-Type: multipart/mixed; name=\"#{filename}\"
        Content-Transfer-Encoding:base64
        Content-Disposition: attachment; filename="#{filename}"

        #{encodedcontent}
        --#{marker}--
      ATTACHED

      mailtext = unindent header + body + attached
      Net::SMTP.start(opts[:server], 25) do |smtp|
         smtp.send_message mailtext, opts[:from], to
      end
    end
  end
end
6

@kallistec, you check the code ?

7

Compared with the code in your first snippet, it looks like you might be getting confused between the two kinds of hooks we have. One is the start/exeception/report handlers, and the other are the event handlers. There are two kinds because the exception/report handlers were added first and later we added a bunch of instrumentation into the chef code and exposed that as a plugin/extension point.

For your use case, the exception handler type is probably the best fit. Your Helper class already subclasses from the main handler class, you just need to add a report method that takes no arguments and configure Chef to use that class as an exception handler.

8

Some example ?

9

See https://docs.chef.io/handlers.html#run-from-client-rb

There’s also https://github.com/chef-cookbooks/chef_handler that can do it for you.

FWIW, there are probably one or several existing handlers that can email you the details of run failures that you might want to google around for.

Daniel DeLeo

10

Thanks @kallistec, the log stills being incompleted however I get the detail of the error in the email message.