Monitoring knife-lastrun?


#1

Hi,

I want to monitor the output of knife-lastrun (to check execution time + failed runs).

Is anyone of you already doing this (we’re using Zabbix, but I would say this fact doesn’t matter…).

Do you see any other way than:
*) give the client on our monitoring station chef admin privileges to retrieve the values via knife (<- bad idea…)
*) modify the knife-lastrun gem to write stats into a local file (however I have no clue, how to do this - I’m not a ruby guy :().
*) or does knife-lastrun already write stats to a local file? However the code looks like it only writes to log and saves to the node.

Would be nice, if someone could help me!

Yours
Steffen


#2

Hi,

May be different from what you are asking, we are using Grill to monitor
Chef run status.
It has a good GUI with alter and report feature and very easy to setup.

You can get a trial version to try Grill out.

Regards,

Tetsu

On 6/10/12 1:17 AM, Steffen Gebert wrote:

Hi,

I want to monitor the output of knife-lastrun (to check execution time + failed runs).

Is anyone of you already doing this (we’re using Zabbix, but I would say this fact doesn’t matter…).

Do you see any other way than:
*) give the client on our monitoring station chef admin privileges to retrieve the values via knife (<- bad idea…)
*) modify the knife-lastrun gem to write stats into a local file (however I have no clue, how to do this - I’m not a ruby guy :().
*) or does knife-lastrun already write stats to a local file? However the code looks like it only writes to log and saves to the node.

Would be nice, if someone could help me!

Yours
Steffen


#3

Hi,

wouldn’t report and exception handlers be the better choice here? Sounds like using knife lastrun is just adding complexity?

Cheers,
Daniel

On Jun 9, 2012, at 12:45 PM, Tetsu Soh wrote:

Hi,

May be different from what you are asking, we are using Grill to monitor Chef run status.
It has a good GUI with alter and report feature and very easy to setup.

You can get a trial version to try Grill out.

Regards,

Tetsu

On 6/10/12 1:17 AM, Steffen Gebert wrote:

Hi,

I want to monitor the output of knife-lastrun (to check execution time + failed runs).

Is anyone of you already doing this (we’re using Zabbix, but I would say this fact doesn’t matter…).

Do you see any other way than:
*) give the client on our monitoring station chef admin privileges to retrieve the values via knife (<- bad idea…)
*) modify the knife-lastrun gem to write stats into a local file (however I have no clue, how to do this - I’m not a ruby guy :().
*) or does knife-lastrun already write stats to a local file? However the code looks like it only writes to log and saves to the node.

Would be nice, if someone could help me!

Yours
Steffen


#4

Hi,

thanks for your quick replies!

I’m unsure, whether knife-lastrun is a perfect start. The optimum solution would be, if it wouldn’t only save to node data, but also to write results into files. I would then pick these up through zabbix agent.

Do you know some report handler that does already exactly this?

@Tetsu
I read about Grill and also the other dashboard here in the lists. But I prefer to have it integrated in our existing monitoring.

Thanks
Steffen

On 09.06.2012, at 18:50, Daniel Schauenberg wrote:

Hi,

wouldn’t report and exception handlers be the better choice here? Sounds like using knife lastrun is just adding complexity?

Cheers,
Daniel

On Jun 9, 2012, at 12:45 PM, Tetsu Soh wrote:

Hi,

May be different from what you are asking, we are using Grill to monitor Chef run status.
It has a good GUI with alter and report feature and very easy to setup.

You can get a trial version to try Grill out.

Regards,

Tetsu

On 6/10/12 1:17 AM, Steffen Gebert wrote:

Hi,

I want to monitor the output of knife-lastrun (to check execution time + failed runs).

Is anyone of you already doing this (we’re using Zabbix, but I would say this fact doesn’t matter…).

Do you see any other way than:
*) give the client on our monitoring station chef admin privileges to retrieve the values via knife (<- bad idea…)
*) modify the knife-lastrun gem to write stats into a local file (however I have no clue, how to do this - I’m not a ruby guy :().
*) or does knife-lastrun already write stats to a local file? However the code looks like it only writes to log and saves to the node.

Would be nice, if someone could help me!

Yours
Steffen


#5

On Sat, 09 Jun 2012, Steffen Gebert wrote:

Hi,

thanks for your quick replies!

I’m unsure, whether knife-lastrun is a perfect start. The optimum solution would be, if it wouldn’t only save to node data, but also to write results into files. I would then pick these up through zabbix agent.

Do you know some report handler that does already exactly this?

hi. we use zabbix where i work. from each node i’m reporting the chef-client
status to zabbux using zabbix-sender from a chef report handler. i based my
code on this gentleman’s handler:

https://github.com/ranjibd/nsca_handler/blob/master/nsca_handler.rb

i modified it to fit zabbix bits. when the client node sends its status,
0 good, non-zero bad, it does basically:

zabbix-sender_binary -s host001 -c /etc/zabbix.conf -z zabbix-server.foo.com -k chef_client[status] -o $CHEF_CLIENT_EXIT_STATUS

on the zabbix server i also do a freshness check. if the server hasn’t heard
from a client in X period, alert.

HTH,
kallen

@Tetsu
I read about Grill and also the other dashboard here in the lists. But I prefer to have it integrated in our existing monitoring.

Thanks
Steffen

On 09.06.2012, at 18:50, Daniel Schauenberg wrote:

Hi,

wouldn’t report and exception handlers be the better choice here? Sounds like using knife lastrun is just adding complexity?

Cheers,
Daniel

On Jun 9, 2012, at 12:45 PM, Tetsu Soh wrote:

Hi,

May be different from what you are asking, we are using Grill to monitor Chef run status.
It has a good GUI with alter and report feature and very easy to setup.

You can get a trial version to try Grill out.

Regards,

Tetsu

On 6/10/12 1:17 AM, Steffen Gebert wrote:

Hi,

I want to monitor the output of knife-lastrun (to check execution time + failed runs).

Is anyone of you already doing this (we’re using Zabbix, but I would say this fact doesn’t matter…).

Do you see any other way than:
*) give the client on our monitoring station chef admin privileges to retrieve the values via knife (<- bad idea…)
*) modify the knife-lastrun gem to write stats into a local file (however I have no clue, how to do this - I’m not a ruby guy :().
*) or does knife-lastrun already write stats to a local file? However the code looks like it only writes to log and saves to the node.

Would be nice, if someone could help me!

Yours
Steffen


#6

Thanks kallen, that was an excellent starting point!

I’ve never used zabbix_sender, but - of course - it’s absolutely the right tool for this job!

I managed to write a handler that reports execution times and status of chef-client runs to zabbix:

Just a question: Do you log more than plain numbers? (exception stack trace, updated resources etc?). Found nothing out, how to log multiple lines.

Nevertheless, thanks a lot for your help!

Steffen

On 10.06.2012, at 08:07, kallen@groknaut.net wrote:

On Sat, 09 Jun 2012, Steffen Gebert wrote:

Hi,

thanks for your quick replies!

I’m unsure, whether knife-lastrun is a perfect start. The optimum solution would be, if it wouldn’t only save to node data, but also to write results into files. I would then pick these up through zabbix agent.

Do you know some report handler that does already exactly this?

hi. we use zabbix where i work. from each node i’m reporting the chef-client
status to zabbux using zabbix-sender from a chef report handler. i based my
code on this gentleman’s handler:

https://github.com/ranjibd/nsca_handler/blob/master/nsca_handler.rb

i modified it to fit zabbix bits. when the client node sends its status,
0 good, non-zero bad, it does basically:

zabbix-sender_binary -s host001 -c /etc/zabbix.conf -z zabbix-server.foo.com -k chef_client[status] -o $CHEF_CLIENT_EXIT_STATUS

on the zabbix server i also do a freshness check. if the server hasn’t heard
from a client in X period, alert.

HTH,
kallen

@Tetsu
I read about Grill and also the other dashboard here in the lists. But I prefer to have it integrated in our existing monitoring.

Thanks
Steffen

On 09.06.2012, at 18:50, Daniel Schauenberg wrote:

Hi,

wouldn’t report and exception handlers be the better choice here? Sounds like using knife lastrun is just adding complexity?

Cheers,
Daniel

On Jun 9, 2012, at 12:45 PM, Tetsu Soh wrote:

Hi,

May be different from what you are asking, we are using Grill to monitor Chef run status.
It has a good GUI with alter and report feature and very easy to setup.

You can get a trial version to try Grill out.

Regards,

Tetsu

On 6/10/12 1:17 AM, Steffen Gebert wrote:

Hi,

I want to monitor the output of knife-lastrun (to check execution time + failed runs).

Is anyone of you already doing this (we’re using Zabbix, but I would say this fact doesn’t matter…).

Do you see any other way than:
*) give the client on our monitoring station chef admin privileges to retrieve the values via knife (<- bad idea…)
*) modify the knife-lastrun gem to write stats into a local file (however I have no clue, how to do this - I’m not a ruby guy :().
*) or does knife-lastrun already write stats to a local file? However the code looks like it only writes to log and saves to the node.

Would be nice, if someone could help me!

Yours
Steffen


#7

On Sun, 10 Jun 2012, Steffen Gebert wrote:

Thanks kallen, that was an excellent starting point!

I’ve never used zabbix_sender, but - of course - it’s absolutely the right tool for this job!

I managed to write a handler that reports execution times and status of chef-client runs to zabbix:
https://github.com/StephenKing/chef-zabbix-custom-checks/blob/master/templates/default/chef-client/chef-client-handler.rb

Just a question: Do you log more than plain numbers? (exception stack trace, updated resources etc?). Found nothing out, how to log multiple lines.

nope, not at this time. for now, for me, just knowing my clients are
succeeding or failing is good enough. those that fail, i’ll ssh in and take
a look.

glad it was helpful! :>

kallen

Nevertheless, thanks a lot for your help!

Steffen


#8

On Mon, Jun 11, 2012 at 7:27 AM, kallen@groknaut.net wrote:

On Sun, 10 Jun 2012, Steffen Gebert wrote:

Thanks kallen, that was an excellent starting point!

I’ve never used zabbix_sender, but - of course - it’s absolutely the
right tool for this job!

I managed to write a handler that reports execution times and status of
chef-client runs to zabbix:

https://github.com/StephenKing/chef-zabbix-custom-checks/blob/master/templates/default/chef-client/chef-client-handler.rb

Just a question: Do you log more than plain numbers? (exception stack
trace, updated resources etc?). Found nothing out, how to log multiple
lines.

I remember there was a presentation from Etsy at ChefConf mentioning they
were using knife-lastrun to report Chef runs (including stacktrace, etc):

Lessons from Etsy: Avoiding Kitchen Nightmares - #ChefConf 2012 from Patrick McDonnell

Not sure if that helps…

Torben

nope, not at this time. for now, for me, just knowing my clients are

succeeding or failing is good enough. those that fail, i’ll ssh in and take
a look.

glad it was helpful! :>

kallen

Nevertheless, thanks a lot for your help!

Steffen


#9

Hi Torben,

I saw that presentation, too, (online) and knife-lastrun was my starting point.

Besides the report handler for zabbix, I’ve also added it to our hosts yesterday. So it’s pretty easy to inspect the reason for a failure through knife.

Yours
Steffen

On 11.06.2012, at 07:27, kallen@groknaut.net wrote:

On Sun, 10 Jun 2012, Steffen Gebert wrote:

Thanks kallen, that was an excellent starting point!

I’ve never used zabbix_sender, but - of course - it’s absolutely the right tool for this job!

I managed to write a handler that reports execution times and status of chef-client runs to zabbix:
https://github.com/StephenKing/chef-zabbix-custom-checks/blob/master/templates/default/chef-client/chef-client-handler.rb

Just a question: Do you log more than plain numbers? (exception stack trace, updated resources etc?). Found nothing out, how to log multiple lines.

nope, not at this time. for now, for me, just knowing my clients are
succeeding or failing is good enough. those that fail, i’ll ssh in and take
a look.

glad it was helpful! :>

kallen

Nevertheless, thanks a lot for your help!

Steffen