Monitoring knife-lastrun?

StephenKing · June 9, 2012, 4:17pm

Hi,

I want to monitor the output of knife-lastrun (to check execution time + failed runs).

Is anyone of you already doing this (we’re using Zabbix, but I would say this fact doesn’t matter…).

Do you see any other way than:
*) give the client on our monitoring station chef admin privileges to retrieve the values via knife (<- bad idea…)
*) modify the knife-lastrun gem to write stats into a local file (however I have no clue, how to do this - I’m not a ruby guy :().
*) or does knife-lastrun already write stats to a local file? However the code looks like it only writes to log and saves to the node.

Would be nice, if someone could help me!

Yours
Steffen

tetsu_soh · June 9, 2012, 4:45pm

Hi,

May be different from what you are asking, we are using Grill to monitor
Chef run status.
It has a good GUI with alter and report feature and very easy to setup.

You can get a trial version to try Grill out.

Regards,

Tetsu

On 6/10/12 1:17 AM, Steffen Gebert wrote:

Hi,

I want to monitor the output of knife-lastrun (to check execution time + failed runs).

Is anyone of you already doing this (we're using Zabbix, but I would say this fact doesn't matter..).

Do you see any other way than:
*) give the client on our monitoring station chef admin privileges to retrieve the values via knife (<- bad idea..)
*) modify the knife-lastrun gem to write stats into a local file (however I have no clue, how to do this - I'm not a ruby guy :().
*) or does knife-lastrun already write stats to a local file? However the code looks like it only writes to log and saves to the node.

Would be nice, if someone could help me!

Yours
Steffen

Daniel_Schauenberg · June 9, 2012, 4:50pm

Hi,

wouldn't report and exception handlers be the better choice here? Sounds like using knife lastrun is just adding complexity?

Cheers,
Daniel

On Jun 9, 2012, at 12:45 PM, Tetsu Soh wrote:

Hi,

May be different from what you are asking, we are using Grill to monitor Chef run status.
It has a good GUI with alter and report feature and very easy to setup.

You can get a trial version to try Grill out.

Regards,

Tetsu

On 6/10/12 1:17 AM, Steffen Gebert wrote:

Hi,

I want to monitor the output of knife-lastrun (to check execution time + failed runs).

Is anyone of you already doing this (we're using Zabbix, but I would say this fact doesn't matter..).

Do you see any other way than:
*) give the client on our monitoring station chef admin privileges to retrieve the values via knife (<- bad idea..)
*) modify the knife-lastrun gem to write stats into a local file (however I have no clue, how to do this - I'm not a ruby guy :().
*) or does knife-lastrun already write stats to a local file? However the code looks like it only writes to log and saves to the node.

Would be nice, if someone could help me!

Yours
Steffen

StephenKing · June 9, 2012, 5:00pm

Hi,

thanks for your quick replies!

I'm unsure, whether knife-lastrun is a perfect start. The optimum solution would be, if it wouldn't only save to node data, but also to write results into files. I would then pick these up through zabbix agent.

Do you know some report handler that does already exactly this?

@Tetsu
I read about Grill and also the other dashboard here in the lists. But I prefer to have it integrated in our existing monitoring.

Thanks
Steffen

On 09.06.2012, at 18:50, Daniel Schauenberg wrote:

Hi,

wouldn't report and exception handlers be the better choice here? Sounds like using knife lastrun is just adding complexity?

Cheers,
Daniel

On Jun 9, 2012, at 12:45 PM, Tetsu Soh wrote:

Hi,

May be different from what you are asking, we are using Grill to monitor Chef run status.
It has a good GUI with alter and report feature and very easy to setup.

You can get a trial version to try Grill out.

Regards,

Tetsu

On 6/10/12 1:17 AM, Steffen Gebert wrote:

Hi,

I want to monitor the output of knife-lastrun (to check execution time + failed runs).

Is anyone of you already doing this (we're using Zabbix, but I would say this fact doesn't matter..).

Do you see any other way than:
*) give the client on our monitoring station chef admin privileges to retrieve the values via knife (<- bad idea..)
*) modify the knife-lastrun gem to write stats into a local file (however I have no clue, how to do this - I'm not a ruby guy :().
*) or does knife-lastrun already write stats to a local file? However the code looks like it only writes to log and saves to the node.

Would be nice, if someone could help me!

Yours
Steffen

Kallen · June 10, 2012, 6:07am

On Sat, 09 Jun 2012, Steffen Gebert wrote:

Hi,

thanks for your quick replies!

I'm unsure, whether knife-lastrun is a perfect start. The optimum solution would be, if it wouldn't only save to node data, but also to write results into files. I would then pick these up through zabbix agent.

Do you know some report handler that does already exactly this?

hi. we use zabbix where i work. from each node i'm reporting the chef-client
status to zabbux using zabbix-sender from a chef report handler. i based my
code on this gentleman's handler:

https://github.com/ranjibd/nsca_handler/blob/master/nsca_handler.rb

i modified it to fit zabbix bits. when the client node sends its status,
0 good, non-zero bad, it does basically:

zabbix-sender_binary -s host001 -c /etc/zabbix.conf -z zabbix-server.foo.com -k chef_client[status] -o $CHEF_CLIENT_EXIT_STATUS

on the zabbix server i also do a freshness check. if the server hasn't heard
from a client in X period, alert.

HTH,
kallen

@Tetsu
I read about Grill and also the other dashboard here in the lists. But I prefer to have it integrated in our existing monitoring.

Thanks
Steffen

On 09.06.2012, at 18:50, Daniel Schauenberg wrote:

Hi,

wouldn't report and exception handlers be the better choice here? Sounds like using knife lastrun is just adding complexity?

Cheers,
Daniel

On Jun 9, 2012, at 12:45 PM, Tetsu Soh wrote:

Hi,

May be different from what you are asking, we are using Grill to monitor Chef run status.
It has a good GUI with alter and report feature and very easy to setup.

You can get a trial version to try Grill out.

Regards,

Tetsu

On 6/10/12 1:17 AM, Steffen Gebert wrote:

Hi,

I want to monitor the output of knife-lastrun (to check execution time + failed runs).

Is anyone of you already doing this (we're using Zabbix, but I would say this fact doesn't matter..).

Do you see any other way than:
*) give the client on our monitoring station chef admin privileges to retrieve the values via knife (<- bad idea..)
*) modify the knife-lastrun gem to write stats into a local file (however I have no clue, how to do this - I'm not a ruby guy :().
*) or does knife-lastrun already write stats to a local file? However the code looks like it only writes to log and saves to the node.

Would be nice, if someone could help me!

Yours
Steffen

StephenKing · June 10, 2012, 7:34pm

Thanks kallen, that was an excellent starting point!

I've never used zabbix_sender, but - of course - it's absolutely the right tool for this job!

I managed to write a handler that reports execution times and status of chef-client runs to zabbix:

github.com

StephenKing/chef-zabbix-custom-checks/blob/master/templates/default/chef-client/chef-client-handler.rb

require 'rubygems'
require 'chef/log'
require 'chef'
require 'tempfile'

module Zabbix
  class Report < Chef::Handler
    def report

      Chef::Log.info "Zabbix::Report handler started"

      prefix = "custom.chef-client.last_run"
      file = Tempfile.new('client-handler-zabbix-report')
      host_name = node[:zabbix][:agent][:hostname] || node[:fqdn]
      
      message = [
        "#{host_name} #{prefix}.success #{run_status.success? ? 1 : 0}",
        "#{host_name} #{prefix}.elapsed_time #{run_status.elapsed_time}",
        "#{host_name} #{prefix}.start_time #{run_status.start_time.to_i}",
        "#{host_name} #{prefix}.end_time #{run_status.end_time.to_i}",

This file has been truncated. show original

Just a question: Do you log more than plain numbers? (exception stack trace, updated resources etc?). Found nothing out, how to log multiple lines.

Nevertheless, thanks a lot for your help!

Steffen

On 10.06.2012, at 08:07, kallen@groknaut.net wrote:

On Sat, 09 Jun 2012, Steffen Gebert wrote:

Hi,

thanks for your quick replies!

I'm unsure, whether knife-lastrun is a perfect start. The optimum solution would be, if it wouldn't only save to node data, but also to write results into files. I would then pick these up through zabbix agent.

Do you know some report handler that does already exactly this?

hi. we use zabbix where i work. from each node i'm reporting the chef-client
status to zabbux using zabbix-sender from a chef report handler. i based my
code on this gentleman's handler:

https://github.com/ranjibd/nsca_handler/blob/master/nsca_handler.rb

i modified it to fit zabbix bits. when the client node sends its status,
0 good, non-zero bad, it does basically:

zabbix-sender_binary -s host001 -c /etc/zabbix.conf -z zabbix-server.foo.com -k chef_client[status] -o $CHEF_CLIENT_EXIT_STATUS

on the zabbix server i also do a freshness check. if the server hasn't heard
from a client in X period, alert.

HTH,
kallen

@Tetsu
I read about Grill and also the other dashboard here in the lists. But I prefer to have it integrated in our existing monitoring.

Thanks
Steffen

On 09.06.2012, at 18:50, Daniel Schauenberg wrote:

Hi,

wouldn't report and exception handlers be the better choice here? Sounds like using knife lastrun is just adding complexity?

Cheers,
Daniel

On Jun 9, 2012, at 12:45 PM, Tetsu Soh wrote:

Hi,

May be different from what you are asking, we are using Grill to monitor Chef run status.
It has a good GUI with alter and report feature and very easy to setup.

You can get a trial version to try Grill out.

Regards,

Tetsu

On 6/10/12 1:17 AM, Steffen Gebert wrote:

Hi,

I want to monitor the output of knife-lastrun (to check execution time + failed runs).

Is anyone of you already doing this (we're using Zabbix, but I would say this fact doesn't matter..).

Do you see any other way than:
*) give the client on our monitoring station chef admin privileges to retrieve the values via knife (<- bad idea..)
*) modify the knife-lastrun gem to write stats into a local file (however I have no clue, how to do this - I'm not a ruby guy :().
*) or does knife-lastrun already write stats to a local file? However the code looks like it only writes to log and saves to the node.

Would be nice, if someone could help me!

Yours
Steffen

Kallen · June 11, 2012, 5:27am

On Sun, 10 Jun 2012, Steffen Gebert wrote:

Thanks kallen, that was an excellent starting point!

I've never used zabbix_sender, but - of course - it's absolutely the right tool for this job!

I managed to write a handler that reports execution times and status of chef-client runs to zabbix:
chef-zabbix-custom-checks/templates/default/chef-client/chef-client-handler.rb at master · StephenKing/chef-zabbix-custom-checks · GitHub

Just a question: Do you log more than plain numbers? (exception stack trace, updated resources etc?). Found nothing out, how to log multiple lines.

nope, not at this time. for now, for me, just knowing my clients are
succeeding or failing is good enough. those that fail, i'll ssh in and take
a look.

glad it was helpful! :>

kallen

Nevertheless, thanks a lot for your help!

Steffen

Torben_Knerr · June 11, 2012, 6:51am

On Mon, Jun 11, 2012 at 7:27 AM, kallen@groknaut.net wrote:

On Sun, 10 Jun 2012, Steffen Gebert wrote:

Thanks kallen, that was an excellent starting point!

I've never used zabbix_sender, but - of course - it's absolutely the
right tool for this job!

I managed to write a handler that reports execution times and status of
chef-client runs to zabbix:

chef-zabbix-custom-checks/templates/default/chef-client/chef-client-handler.rb at master · StephenKing/chef-zabbix-custom-checks · GitHub

Just a question: Do you log more than plain numbers? (exception stack
trace, updated resources etc?). Found nothing out, how to log multiple
lines.

I remember there was a presentation from Etsy at ChefConf mentioning they
were using knife-lastrun to report Chef runs (including stacktrace, etc):

Not sure if that helps...

Torben

nope, not at this time. for now, for me, just knowing my clients are

succeeding or failing is good enough. those that fail, i'll ssh in and take
a look.

glad it was helpful! :>

kallen

Nevertheless, thanks a lot for your help!

Steffen

StephenKing · June 11, 2012, 7:15am

Hi Torben,

I saw that presentation, too, (online) and knife-lastrun was my starting point.

Besides the report handler for zabbix, I've also added it to our hosts yesterday. So it's pretty easy to inspect the reason for a failure through knife.

Yours
Steffen

On 11.06.2012, at 07:27, kallen@groknaut.net wrote:

On Sun, 10 Jun 2012, Steffen Gebert wrote:

Thanks kallen, that was an excellent starting point!

I've never used zabbix_sender, but - of course - it's absolutely the right tool for this job!

I managed to write a handler that reports execution times and status of chef-client runs to zabbix:
chef-zabbix-custom-checks/templates/default/chef-client/chef-client-handler.rb at master · StephenKing/chef-zabbix-custom-checks · GitHub

Just a question: Do you log more than plain numbers? (exception stack trace, updated resources etc?). Found nothing out, how to log multiple lines.

nope, not at this time. for now, for me, just knowing my clients are
succeeding or failing is good enough. those that fail, i'll ssh in and take
a look.

glad it was helpful! :>

kallen

Nevertheless, thanks a lot for your help!

Steffen

Topic		Replies	Views
Record the output of chef-client in nodes Chef Infra (archive)	13	534	September 26, 2014
How to track chef-client execution remotely? Chef Infra (archive)	4	682	September 22, 2016
Best practice for measuring and monitoring chef-client runs? Chef Infra (archive)	10	2403	September 17, 2014
Where I can find in my chef server the reports sent by chef-client Chef Infra (archive)	2	509	September 12, 2016
Log storage of chef client Chef Infra (archive)	2	497	September 21, 2017

Monitoring knife-lastrun?

Related Topics