Knife monitoring


#1

Hey all,

As our chef repos grow and becomes more collaborative, a need to track activity on the chef server is rising in priority for me.

We’ve wrapped the knife util to track activity, but can’t enforce that people will use the wrapped util. Is there any way anyone is doing this? I’d like to implement tracking at the chef server side. Any plugin or advice would be appreciated.

Gregory Patmore


#2

We tried to tackle that problem and had similar issues. Instead, we’re
going the automated route, only Jenkins can make changes to chef: automated
pushes of cookbooks, environments, roles, etc. that tracking and history
are built into Jenkins and only occur after a proper merge has been done in
GitHub.

I know that’s not exactly what you’re asking but just something to think
about.

Good luck!

Mike

On Friday, May 23, 2014, Gregory Patmore gregorypatmore@gmail.com wrote:

Hey all,

As our chef repos grow and becomes more collaborative, a need to track
activity on the chef server is rising in priority for me.

We’ve wrapped the knife util to track activity, but can’t enforce that
people will use the wrapped util. Is there any way anyone is doing this?
I’d like to implement tracking at the chef server side. Any plugin or
advice would be appreciated.

Gregory Patmore


#3

We went a similar route, though we aren’t using Jenkins, but instead, have “chefhooks” bit running on a worker server. It will apply any change that’s pushed to our git repo, so it’s very trusting, but since it’s so easy to update environments, data bags, etc., nobody does it using knife (with the exception of encrypted data bags, which must be edited via knife).

We don’t, however, handle cookbooks that way; those are all driven by a Berkshelf workflow that relies on Travis to test things using Test Kitchen & the EC2 driver.


Jeff Byrnes
@berkleebassist
Operations Engineer
EverTrue
704.516.4628

On May 23, 2014 at 8:08:04 AM, Mike Splain (mike.splain@gmail.com) wrote:

We tried to tackle that problem and had similar issues. Instead, we’re going the automated route, only Jenkins can make changes to chef: automated pushes of cookbooks, environments, roles, etc. that tracking and history are built into Jenkins and only occur after a proper merge has been done in GitHub.

I know that’s not exactly what you’re asking but just something to think about.

Good luck!

Mike

On Friday, May 23, 2014, Gregory Patmore gregorypatmore@gmail.com wrote:
Hey all,

As our chef repos grow and becomes more collaborative, a need to track activity on the chef server is rising in priority for me.

We’ve wrapped the knife util to track activity, but can’t enforce that people will use the wrapped util. Is there any way anyone is doing this? I’d like to implement tracking at the chef server side. Any plugin or advice would be appreciated.

Gregory Patmore


#4

Hi Gregory,

We were running into the same problem and wrote also some knife wrappers, but indeed this is not solving anything.
Then we wrote some tooling that would scan the nginx log on who/what has changed something and pull the information on the backend out of chef and commit it in git.
But this was very chatty, error prone and not scalable… (but worked for some months)

Then we started with the development of chef-guard (written in Go)
We use the chef-guard between erchef and bookshelf on the frontend and communicate from the guard to the backend.
It scans who is doing what and commit every change (plain json) into github, so you will endup with a repo with all your change information.
It integrates with foodcritic and rubocop and can scan every cookbook upload on syntax/style and even block uploads that do not comply. (if you want)
Idea is to even check your uploads against already uploaded cookbooks and block you if you have made changes but not updated the version.
And we can disabled the --force overwrite on frozen cookbooks, because what’s the use of freezing if you can do that :wink:

Next week svanharmelen@schubergphilis.com who wrote the tool will opensource it on github, so you can develop on it yourself.
We have been running with this in production since Chef-Conf 2014 and squashed all the minor bugs…
So keep in contact and we will help you out on making your environment 100% …

Kind regards,
Sander Botman

Schuberg Philis
Boeingavenue 271
1119 PD Schiphol-Rijk
http://www.schubergphilis.com

-----Original Message-----
From: Gregory Patmore [mailto:gregorypatmore@gmail.com]
Sent: vrijdag 23 mei 2014 14:03
To: chef
Subject: [chef] Knife monitoring

Hey all,

As our chef repos grow and becomes more collaborative, a need to track activity on the chef server is rising in priority for me.

We’ve wrapped the knife util to track activity, but can’t enforce that people will use the wrapped util. Is there any way anyone is doing this? I’d like to implement tracking at the chef server side. Any plugin or advice would be appreciated.

Gregory Patmore


#5

Check out Schuberg Phillis’s chef-monitor, a way that you can tap the
RabbitMQ bus in Chef Server to grab events of interest.

Chef Actions (a for-pay addon to EC that we announced at ChefConf and
that will ship soon) has a similar architecture, though, of course,
many more features.

  • Julian

On Fri, May 23, 2014 at 8:02 AM, Gregory Patmore
gregorypatmore@gmail.com wrote:

Hey all,

As our chef repos grow and becomes more collaborative, a need to track activity on the chef server is rising in priority for me.

We’ve wrapped the knife util to track activity, but can’t enforce that people will use the wrapped util. Is there any way anyone is doing this? I’d like to implement tracking at the chef server side. Any plugin or advice would be appreciated.

Gregory Patmore


[ Julian C. Dunn jdunn@aquezada.com * Sorry, I’m ]
[ WWW: http://www.aquezada.com/staff/julian * only Web 1.0 ]
[ gopher://sdf.org/1/users/keymaker/ * compliant! ]
[ PGP: 91B3 7A9D 683C 7C16 715F 442C 6065 D533 FDC2 05B9 ]


#6

We’re looking into how to do something similar in using git to be the
primary record of state (including audit-ability & changelog), i.e.
cookbooks in separate repos, and a chef-data repo for databags,
environments, etc. (we don’t use roles).

But storing nodes troubles me - a lot of tools & cookbooks expect to
store state in the “normal” priority attributes, which are persisted in
Chef Server’s node database. Transient state is generated by cookbooks at
the other priority levels, which can be used to implement Search and other
data publishing/metrics activity. All that would get blitzed if we store
all the nodes in the same chef-data repo, and did a blanket upload of all
nodes (along with databags & environments) from the git repo whenever
someone pushed a single new node/databag/whatever.

Any suggestions on how do to cope with this?

Tristan.

On 23 May 2014 14:38, Jeff Byrnes jeff@evertrue.com wrote:

We went a similar route, though we aren’t using Jenkins, but instead, have
“chefhooks” bit running on a worker server. It will apply any change that’s
pushed to our git repo, so it’s very trusting, but since it’s so easy to
update environments, data bags, etc., nobody does it using knife (with the
exception of encrypted data bags, which must be edited via knife).

We don’t, however, handle cookbooks that way; those are all driven by a
Berkshelf workflow that relies on Travis to test things using Test Kitchen
& the EC2 driver.


Jeff Byrnes
@berkleebassist http://twitter.com/berkleebassist
Operations Engineer
EverTrue http://www.evertrue.com/
704.516.4628

On May 23, 2014 at 8:08:04 AM, Mike Splain (mike.splain@gmail.com) wrote:

We tried to tackle that problem and had similar issues. Instead, we’re
going the automated route, only Jenkins can make changes to chef: automated
pushes of cookbooks, environments, roles, etc. that tracking and history
are built into Jenkins and only occur after a proper merge has been done in
GitHub.

I know that’s not exactly what you’re asking but just something to think
about.

Good luck!

Mike

On Friday, May 23, 2014, Gregory Patmore gregorypatmore@gmail.com wrote:

Hey all,

As our chef repos grow and becomes more collaborative, a need to track
activity on the chef server is rising in priority for me.

We’ve wrapped the knife util to track activity, but can’t enforce that
people will use the wrapped util. Is there any way anyone is doing this?
I’d like to implement tracking at the chef server side. Any plugin or
advice would be appreciated.

Gregory Patmore


#7

we have a private github server, so the issue isn’t as critical for us since we have control over access to the server.

Revision control is relatively simple on the source side, and as long as you can guarantee that anyone with a knife account will no go off reservation it will suffice, but as the team, infrastructure, and various environments grow it becomes more of an exercise in faith then a reliable system (this is just my experience).

I’ve been pondering the same questions you have lately. Employing tools like chef-zero and other chef helpers require local access to node information, which, as you point out, can be represented incompletely.

The only idea I’ve had that I haven’t dismissed is tracking the node objects in a different repo then your cookbooks, but even that doesn’t quite cut it if I understand you’re meaning.

Sent from my iPhone

On May 23, 2014, at 3:14 PM, Tristan Keen tristan.keen@gmail.com wrote:

We’re looking into how to do something similar in using git to be the primary record of state (including audit-ability & changelog), i.e. cookbooks in separate repos, and a chef-data repo for databags, environments, etc. (we don’t use roles).

But storing nodes troubles me - a lot of tools & cookbooks expect to store state in the “normal” priority attributes, which are persisted in Chef Server’s node database. Transient state is generated by cookbooks at the other priority levels, which can be used to implement Search and other data publishing/metrics activity. All that would get blitzed if we store all the nodes in the same chef-data repo, and did a blanket upload of all nodes (along with databags & environments) from the git repo whenever someone pushed a single new node/databag/whatever.

Any suggestions on how do to cope with this?

Tristan.

On 23 May 2014 14:38, Jeff Byrnes jeff@evertrue.com wrote:
We went a similar route, though we aren’t using Jenkins, but instead, have “chefhooks” bit running on a worker server. It will apply any change that’s pushed to our git repo, so it’s very trusting, but since it’s so easy to update environments, data bags, etc., nobody does it using knife (with the exception of encrypted data bags, which must be edited via knife).

We don’t, however, handle cookbooks that way; those are all driven by a Berkshelf workflow that relies on Travis to test things using Test Kitchen & the EC2 driver.


Jeff Byrnes
@berkleebassist
Operations Engineer
EverTrue
704.516.4628

On May 23, 2014 at 8:08:04 AM, Mike Splain (mike.splain@gmail.com) wrote:

We tried to tackle that problem and had similar issues. Instead, we’re going the automated route, only Jenkins can make changes to chef: automated pushes of cookbooks, environments, roles, etc. that tracking and history are built into Jenkins and only occur after a proper merge has been done in GitHub.

I know that’s not exactly what you’re asking but just something to think about.

Good luck!

Mike

On Friday, May 23, 2014, Gregory Patmore gregorypatmore@gmail.com wrote:
Hey all,

As our chef repos grow and becomes more collaborative, a need to track activity on the chef server is rising in priority for me.

We’ve wrapped the knife util to track activity, but can’t enforce that people will use the wrapped util. Is there any way anyone is doing this? I’d like to implement tracking at the chef server side. Any plugin or advice would be appreciated.

Gregory Patmore


#8

Thanks Julian,

This is more in line with what I’m after. Tracking changes in 4 different environments with up to 20 knife-ers across 3 other teams besides my own is getting to be a cumbersome obligation. It forces me to be somewhat inflexible on my production environment, and with multiple dev processes happening in the non prod environments it’s starting to turn into the Wild Wild West.

Luckily I caught it but almost re-introduced the heart bleed bug when an uninformed DBA trying to push through OpenSSL upgrades by way of a dependency for Postgres.

I want to be the voice of DevOps philosophy, sourcing config management from all tech staff, but without strict change control and issue forensics on the chef side, I’m forced to either throttle back software production too much or become the Operations Ogre defending his bridge lol.

Sent from my iPhone

On May 23, 2014, at 3:00 PM, “Julian C. Dunn” jdunn@aquezada.com wrote:

Check out Schuberg Phillis’s chef-monitor, a way that you can tap the
RabbitMQ bus in Chef Server to grab events of interest.

https://www.cupfighter.net/2014/02/chef-monitoring

Chef Actions (a for-pay addon to EC that we announced at ChefConf and
that will ship soon) has a similar architecture, though, of course,
many more features.

  • Julian

On Fri, May 23, 2014 at 8:02 AM, Gregory Patmore
gregorypatmore@gmail.com wrote:

Hey all,

As our chef repos grow and becomes more collaborative, a need to track activity on the chef server is rising in priority for me.

We’ve wrapped the knife util to track activity, but can’t enforce that people will use the wrapped util. Is there any way anyone is doing this? I’d like to implement tracking at the chef server side. Any plugin or advice would be appreciated.

Gregory Patmore


[ Julian C. Dunn jdunn@aquezada.com * Sorry, I’m ]
[ WWW: http://www.aquezada.com/staff/julian * only Web 1.0 ]
[ gopher://sdf.org/1/users/keymaker/ * compliant! ]
[ PGP: 91B3 7A9D 683C 7C16 715F 442C 6065 D533 FDC2 05B9 ]