Cookbook dependency graphing

Not sure if anyone's already doing something like this, but I made some
REALLY MINOR modifications to the most ancient and venerable Ruby depgraph
tool and got it parsing cookbook metadata instead of C# code and installed
gems.

Default behavior is to parse, fire up graphviz and show you a dependency
graph. That's fine but I wanted it as a JSON object.

I added the -json switch to allow you to generate an object that looks like
this:

{ "cookbook" : [ "dependency1", "dependency2"],
...
}

Hope someone out there finds this useful.

A while back, I did something similar for roles only:
http://miketheman.github.io/knife-role-spaghetti/

Does this make my (albiet, not really touched this code in a while)
code obsolete?

-M

On Tue, Jul 9, 2013 at 8:00 PM, steve . leftathome@gmail.com wrote:

Not sure if anyone's already doing something like this, but I made some
REALLY MINOR modifications to the most ancient and venerable Ruby depgraph
tool and got it parsing cookbook metadata instead of C# code and installed
gems.

GitHub - leftathome/depgraph: Modified version of Ruby depgraph that parses Chef cookbook metadata.

Default behavior is to parse, fire up graphviz and show you a dependency
graph. That's fine but I wanted it as a JSON object.

I added the -json switch to allow you to generate an object that looks like
this:

{ "cookbook" : [ "dependency1", "dependency2"],
...
}

Hope someone out there finds this useful.

Oh, this looks really nice! I especially like how the cookbooks are
rendered vs roles :slight_smile:

Actually I'm working with application cookbooks rather than roles, and I'm
working from inside a cookbook repo rather than a Chef repo.

A combination of both would be perfect :wink:

On Wed, Jul 10, 2013 at 7:07 AM, Mike miketheman@gmail.com wrote:

A while back, I did something similar for roles only:
knife-role-spaghetti by miketheman

Does this make my (albiet, not really touched this code in a while)
code obsolete?

-M

On Tue, Jul 9, 2013 at 8:00 PM, steve . leftathome@gmail.com wrote:

Not sure if anyone's already doing something like this, but I made some
REALLY MINOR modifications to the most ancient and venerable Ruby
depgraph
tool and got it parsing cookbook metadata instead of C# code and
installed
gems.

GitHub - leftathome/depgraph: Modified version of Ruby depgraph that parses Chef cookbook metadata.

Default behavior is to parse, fire up graphviz and show you a dependency
graph. That's fine but I wanted it as a JSON object.

I added the -json switch to allow you to generate an object that looks
like
this:

{ "cookbook" : [ "dependency1", "dependency2"],
...
}

Hope someone out there finds this useful.

Guess I'm too stupid to install it...

In my Gemfile:

source 'https://rubygems.org'

gem 'depgraph', git: 'https://github.com/leftathome/depgraph.git'

Then I run:

D:\Repos\_github\_cookbooks\gitlab>bundle install
Fetching https://github.com/leftathome/depgraph.git
remote: Counting objects: 316, done.
remote: Compressing objects: 100% (124/124), done.
rRemote: Total 316 (delta 171), reused 311 (delta 166)
Receiving objects: 100% (316/316), 69.82 KiB | 73 KiB/s, done.
Resolving deltas: 100% (171/171), done.
Fetching gem metadata from https://rubygems.org/.......
Fetching gem metadata from https://rubygems.org/..
Resolving dependencies...
Could not find gem 'depgraph (>= 0) x86-mingw32' in
https://github.com/leftathome/depgraph.git (at master).
Source does not contain any versions of 'depgraph (>= 0) x86-mingw32'

Is it supposed to be installed as a gem?

Not a Ruby expert, but I don't see a .gemspec there. Might this be the
problem?

On Wed, Jul 10, 2013 at 2:00 AM, steve . leftathome@gmail.com wrote:

Not sure if anyone's already doing something like this, but I made some
REALLY MINOR modifications to the most ancient and venerable Ruby depgraph
tool and got it parsing cookbook metadata instead of C# code and installed
gems.

GitHub - leftathome/depgraph: Modified version of Ruby depgraph that parses Chef cookbook metadata.

Default behavior is to parse, fire up graphviz and show you a dependency
graph. That's fine but I wanted it as a JSON object.

I added the -json switch to allow you to generate an object that looks
like this:

{ "cookbook" : [ "dependency1", "dependency2"],
...
}

Hope someone out there finds this useful.

Great tool!

I used:
gem install specific_install
gem specific_install -l git@github.com:leftathome/depgraph.git

--
Roman B

On Wednesday, 10 July, 2013 at 8:38 AM, Torben Knerr wrote:

Guess I'm too stupid to install it...

In my Gemfile:


source 'https://rubygems.org'

gem 'depgraph', git: 'https://github.com/leftathome/depgraph.git'

Then I run:


D:\Repos\_github\_cookbooks\gitlab>bundle install
Fetching https://github.com/leftathome/depgraph.git
remote: Counting objects: 316, done.
remote: Compressing objects: 100% (124/124), done.
rRemote: Total 316 (delta 171), reused 311 (delta 166)
Receiving objects: 100% (316/316), 69.82 KiB | 73 KiB/s, done.
Resolving deltas: 100% (171/171), done.
Fetching gem metadata from https://rubygems.org/.......
Fetching gem metadata from https://rubygems.org/..
Resolving dependencies...
Could not find gem 'depgraph (>= 0) x86-mingw32' in
https://github.com/leftathome/depgraph.git (at master).
Source does not contain any versions of 'depgraph (>= 0) x86-mingw32'


Is it supposed to be installed as a gem?

Not a Ruby expert, but I don't see a .gemspec there. Might this be the problem?

On Wed, Jul 10, 2013 at 2:00 AM, steve . <leftathome@gmail.com (mailto:leftathome@gmail.com)> wrote:

Not sure if anyone's already doing something like this, but I made some REALLY MINOR modifications to the most ancient and venerable Ruby depgraph tool and got it parsing cookbook metadata instead of C# code and installed gems.

GitHub - leftathome/depgraph: Modified version of Ruby depgraph that parses Chef cookbook metadata.

Default behavior is to parse, fire up graphviz and show you a dependency graph. That's fine but I wanted it as a JSON object.

I added the -json switch to allow you to generate an object that looks like this:

{ "cookbook" : [ "dependency1", "dependency2"],
...
}

Hope someone out there finds this useful.

Well, Mike, the code I wrote today doesn't look at roles at all. I didn't
think too hard about mapping cookbook changes to roles, as I'd decided that
I'd implement that as a pretty simple grep. In other words:

Inbound change set includes changes to "cookbook1" -> Look up "cookbook1"
in dependency graph, find dependencies on "cookbook2" -> Look for roles
containing either "cookbook1" or "cookbook2" -> Test those roles first.

That's basically the logic I am looking to implement.

The other fun thing, which I'd thought would somehow involve me writing an
entirely new Jenkins plugin but now appears to be implementable via
ScriptTrigger, is going to be "Run Librarian, build aggregate change set."
Then run through the above parsing logic.

I'm currently supporting a unified Chef repo but hope to tool things in
such a way that individual cookbook repos get updated with each central
release. I don't know if this sounds fancy or not but the implementation
is actually not at all fancy and I feel a bit stupid for not trying this
approach earlier (basically involves a merciless application of "git
filter-branch" as part of the release job).

Honestly, the visualization came about on my way towards getting to the
JSON object, but internally everyone seems to really like the graphviz
renders. Someone even mentioned that they were using your knife plugin,
Mike!

Hmm, maybe I should be rendering the dependency graph and including it in
the yard-generated documentation site.

(working on getting the OK to contribute that back, BTW - works just as
well with individual cookbooks as it does with whole repositories...)

On Tue, Jul 9, 2013 at 10:07 PM, Mike miketheman@gmail.com wrote:

A while back, I did something similar for roles only:
knife-role-spaghetti by miketheman

Does this make my (albiet, not really touched this code in a while)
code obsolete?

-M

On Tue, Jul 9, 2013 at 8:00 PM, steve . leftathome@gmail.com wrote:

Not sure if anyone's already doing something like this, but I made some
REALLY MINOR modifications to the most ancient and venerable Ruby
depgraph
tool and got it parsing cookbook metadata instead of C# code and
installed
gems.

GitHub - leftathome/depgraph: Modified version of Ruby depgraph that parses Chef cookbook metadata.

Default behavior is to parse, fire up graphviz and show you a dependency
graph. That's fine but I wanted it as a JSON object.

I added the -json switch to allow you to generate an object that looks
like
this:

{ "cookbook" : [ "dependency1", "dependency2"],
...
}

Hope someone out there finds this useful.

Hi Torbin,

I threw this together today on my Macbook, so I'm not surprised to find
that you're having trouble getting it to work on Windows. I'm pretty sure
at least some of the depgraph codebase pre-dates Chef and it definitely
wasn't designed with Bundler in mind. The gemspec is actually declared
in the Rakefile.

However, I can attest that "rake gem" and "sudo gem install
pkg/depgraph-VERSION.gem" got it installed for me. No idea how easy it's
going to be for you to get graphviz / ruby-graphviz functioning on Windows,
though.

I don't know if the original gem is actively maintained at this point --
haven't put together a pull request yet -- but if you get it working on
Windows, send me a PR and I'll merge it on in...

On Tue, Jul 9, 2013 at 10:38 PM, Torben Knerr ukio@gmx.de wrote:

Guess I'm too stupid to install it...

In my Gemfile:

source 'https://rubygems.org'

gem 'depgraph', git: 'https://github.com/leftathome/depgraph.git'

Then I run:

D:\Repos\_github\_cookbooks\gitlab>bundle install
Fetching https://github.com/leftathome/depgraph.git
remote: Counting objects: 316, done.
remote: Compressing objects: 100% (124/124), done.
rRemote: Total 316 (delta 171), reused 311 (delta 166)
Receiving objects: 100% (316/316), 69.82 KiB | 73 KiB/s, done.
Resolving deltas: 100% (171/171), done.
Fetching gem metadata from https://rubygems.org/.......
Fetching gem metadata from https://rubygems.org/..
Resolving dependencies...
Could not find gem 'depgraph (>= 0) x86-mingw32' in
https://github.com/leftathome/depgraph.git (at master).
Source does not contain any versions of 'depgraph (>= 0) x86-mingw32'

Is it supposed to be installed as a gem?

Not a Ruby expert, but I don't see a .gemspec there. Might this be the
problem?

On Wed, Jul 10, 2013 at 2:00 AM, steve . leftathome@gmail.com wrote:

Not sure if anyone's already doing something like this, but I made some
REALLY MINOR modifications to the most ancient and venerable Ruby depgraph
tool and got it parsing cookbook metadata instead of C# code and installed
gems.

GitHub - leftathome/depgraph: Modified version of Ruby depgraph that parses Chef cookbook metadata.

Default behavior is to parse, fire up graphviz and show you a dependency
graph. That's fine but I wanted it as a JSON object.

I added the -json switch to allow you to generate an object that looks
like this:

{ "cookbook" : [ "dependency1", "dependency2"],
...
}

Hope someone out there finds this useful.

On Wed, Jul 10, 2013 at 7:41 AM, Roman B rombob@gmail.com wrote:

Great tool!

I used:
gem install specific_install
gem specific_install -l git@github.com:leftathome/depgraph.git

Thanks Roman, that did the trick! :slight_smile:

Hi Steve,

thanks! It works on Windows as well, you just need graphviz installed (and
in the path), and you need to install it either as you or Roman described.

I just understood that it simply looks for all the metadata.rb files in
the current directory, so it works both with librarian and berkshelf (just
make sure the cookbooks are vendored e.g. to the ./cookbooks directory).
That is cool! :slight_smile:

One improvement would be to show the version number of the resolved
cookbook in the graph nodes in addition to only the name. One step further
you could also label the edges with the version constraints, if any.

P.S.: for the use case you describe (i.e. finding out which other cookbooks
depend on an updated cookbook) you could also argue that this is the job of
a dependency manager to find out. I'm not sure if librarian can do that,
but berkshelf supports this exactly via berks contingent COOKBOOK -F json

On Wed, Jul 10, 2013 at 7:48 AM, steve . leftathome@gmail.com wrote:

Hi Torbin,

I threw this together today on my Macbook, so I'm not surprised to find
that you're having trouble getting it to work on Windows. I'm pretty sure
at least some of the depgraph codebase pre-dates Chef and it definitely
wasn't designed with Bundler in mind. The gemspec is actually declared
in the Rakefile.

However, I can attest that "rake gem" and "sudo gem install
pkg/depgraph-VERSION.gem" got it installed for me. No idea how easy it's
going to be for you to get graphviz / ruby-graphviz functioning on Windows,
though.

I don't know if the original gem is actively maintained at this point --
haven't put together a pull request yet -- but if you get it working on
Windows, send me a PR and I'll merge it on in...

On Tue, Jul 9, 2013 at 10:38 PM, Torben Knerr ukio@gmx.de wrote:

Guess I'm too stupid to install it...

In my Gemfile:

source 'https://rubygems.org'

gem 'depgraph', git: 'https://github.com/leftathome/depgraph.git'

Then I run:

D:\Repos\_github\_cookbooks\gitlab>bundle install
Fetching https://github.com/leftathome/depgraph.git
remote: Counting objects: 316, done.
remote: Compressing objects: 100% (124/124), done.
rRemote: Total 316 (delta 171), reused 311 (delta 166)
Receiving objects: 100% (316/316), 69.82 KiB | 73 KiB/s, done.
Resolving deltas: 100% (171/171), done.
Fetching gem metadata from https://rubygems.org/.......
Fetching gem metadata from https://rubygems.org/..
Resolving dependencies...
Could not find gem 'depgraph (>= 0) x86-mingw32' in
https://github.com/leftathome/depgraph.git (at master).
Source does not contain any versions of 'depgraph (>= 0) x86-mingw32'

Is it supposed to be installed as a gem?

Not a Ruby expert, but I don't see a .gemspec there. Might this be the
problem?

On Wed, Jul 10, 2013 at 2:00 AM, steve . leftathome@gmail.com wrote:

Not sure if anyone's already doing something like this, but I made some
REALLY MINOR modifications to the most ancient and venerable Ruby depgraph
tool and got it parsing cookbook metadata instead of C# code and installed
gems.

GitHub - leftathome/depgraph: Modified version of Ruby depgraph that parses Chef cookbook metadata.

Default behavior is to parse, fire up graphviz and show you a dependency
graph. That's fine but I wanted it as a JSON object.

I added the -json switch to allow you to generate an object that looks
like this:

{ "cookbook" : [ "dependency1", "dependency2"],
...
}

Hope someone out there finds this useful.

P.S.: just added a pull requrest so it can be installed via bundler as well:

On Wed, Jul 10, 2013 at 9:55 AM, Torben Knerr ukio@gmx.de wrote:

Hi Steve,

thanks! It works on Windows as well, you just need graphviz installed (and
in the path), and you need to install it either as you or Roman described.

I just understood that it simply looks for all the metadata.rb files in
the current directory, so it works both with librarian and berkshelf (just
make sure the cookbooks are vendored e.g. to the ./cookbooks directory).
That is cool! :slight_smile:

One improvement would be to show the version number of the resolved
cookbook in the graph nodes in addition to only the name. One step further
you could also label the edges with the version constraints, if any.

P.S.: for the use case you describe (i.e. finding out which other
cookbooks depend on an updated cookbook) you could also argue that this is
the job of a dependency manager to find out. I'm not sure if librarian can
do that, but berkshelf supports this exactly via berks contingent COOKBOOK -F json

On Wed, Jul 10, 2013 at 7:48 AM, steve . leftathome@gmail.com wrote:

Hi Torbin,

I threw this together today on my Macbook, so I'm not surprised to find
that you're having trouble getting it to work on Windows. I'm pretty sure
at least some of the depgraph codebase pre-dates Chef and it definitely
wasn't designed with Bundler in mind. The gemspec is actually declared
in the Rakefile.

However, I can attest that "rake gem" and "sudo gem install
pkg/depgraph-VERSION.gem" got it installed for me. No idea how easy it's
going to be for you to get graphviz / ruby-graphviz functioning on Windows,
though.

I don't know if the original gem is actively maintained at this point --
haven't put together a pull request yet -- but if you get it working on
Windows, send me a PR and I'll merge it on in...

On Tue, Jul 9, 2013 at 10:38 PM, Torben Knerr ukio@gmx.de wrote:

Guess I'm too stupid to install it...

In my Gemfile:

source 'https://rubygems.org'

gem 'depgraph', git: 'https://github.com/leftathome/depgraph.git'

Then I run:

D:\Repos\_github\_cookbooks\gitlab>bundle install
Fetching https://github.com/leftathome/depgraph.git
remote: Counting objects: 316, done.
remote: Compressing objects: 100% (124/124), done.
rRemote: Total 316 (delta 171), reused 311 (delta 166)
Receiving objects: 100% (316/316), 69.82 KiB | 73 KiB/s, done.
Resolving deltas: 100% (171/171), done.
Fetching gem metadata from https://rubygems.org/.......
Fetching gem metadata from https://rubygems.org/..
Resolving dependencies...
Could not find gem 'depgraph (>= 0) x86-mingw32' in
https://github.com/leftathome/depgraph.git (at master).
Source does not contain any versions of 'depgraph (>= 0) x86-mingw32'

Is it supposed to be installed as a gem?

Not a Ruby expert, but I don't see a .gemspec there. Might this be the
problem?

On Wed, Jul 10, 2013 at 2:00 AM, steve . leftathome@gmail.com wrote:

Not sure if anyone's already doing something like this, but I made some
REALLY MINOR modifications to the most ancient and venerable Ruby depgraph
tool and got it parsing cookbook metadata instead of C# code and installed
gems.

GitHub - leftathome/depgraph: Modified version of Ruby depgraph that parses Chef cookbook metadata.

Default behavior is to parse, fire up graphviz and show you a
dependency graph. That's fine but I wanted it as a JSON object.

I added the -json switch to allow you to generate an object that looks
like this:

{ "cookbook" : [ "dependency1", "dependency2"],
...
}

Hope someone out there finds this useful.

Pull request accepted! Thank you for choosing the open source development
model. :wink:

On Wed, Jul 10, 2013 at 10:52 PM, Torben Knerr ukio@gmx.de wrote:

P.S.: just added a pull requrest so it can be installed via bundler as
well:
Switch to external depgraph.gemspec so DepGraph can be used with Bundler by tknerr · Pull Request #1 · leftathome/depgraph · GitHub

On Wed, Jul 10, 2013 at 9:55 AM, Torben Knerr ukio@gmx.de wrote:

Hi Steve,

thanks! It works on Windows as well, you just need graphviz installed
(and in the path), and you need to install it either as you or Roman
described.

I just understood that it simply looks for all the metadata.rb files in
the current directory, so it works both with librarian and berkshelf (just
make sure the cookbooks are vendored e.g. to the ./cookbooks directory).
That is cool! :slight_smile:

One improvement would be to show the version number of the resolved
cookbook in the graph nodes in addition to only the name. One step further
you could also label the edges with the version constraints, if any.

P.S.: for the use case you describe (i.e. finding out which other
cookbooks depend on an updated cookbook) you could also argue that this is
the job of a dependency manager to find out. I'm not sure if librarian can
do that, but berkshelf supports this exactly via berks contingent COOKBOOK -F json

On Wed, Jul 10, 2013 at 7:48 AM, steve . leftathome@gmail.com wrote:

Hi Torbin,

I threw this together today on my Macbook, so I'm not surprised to find
that you're having trouble getting it to work on Windows. I'm pretty sure
at least some of the depgraph codebase pre-dates Chef and it definitely
wasn't designed with Bundler in mind. The gemspec is actually declared
in the Rakefile.

However, I can attest that "rake gem" and "sudo gem install
pkg/depgraph-VERSION.gem" got it installed for me. No idea how easy it's
going to be for you to get graphviz / ruby-graphviz functioning on Windows,
though.

I don't know if the original gem is actively maintained at this point --
haven't put together a pull request yet -- but if you get it working on
Windows, send me a PR and I'll merge it on in...

On Tue, Jul 9, 2013 at 10:38 PM, Torben Knerr ukio@gmx.de wrote:

Guess I'm too stupid to install it...

In my Gemfile:

source 'https://rubygems.org'

gem 'depgraph', git: 'https://github.com/leftathome/depgraph.git'

Then I run:

D:\Repos\_github\_cookbooks\gitlab>bundle install
Fetching https://github.com/leftathome/depgraph.git
remote: Counting objects: 316, done.
remote: Compressing objects: 100% (124/124), done.
rRemote: Total 316 (delta 171), reused 311 (delta 166)
Receiving objects: 100% (316/316), 69.82 KiB | 73 KiB/s, done.
Resolving deltas: 100% (171/171), done.
Fetching gem metadata from https://rubygems.org/.......
Fetching gem metadata from https://rubygems.org/..
Resolving dependencies...
Could not find gem 'depgraph (>= 0) x86-mingw32' in
https://github.com/leftathome/depgraph.git (at master).
Source does not contain any versions of 'depgraph (>= 0) x86-mingw32'

Is it supposed to be installed as a gem?

Not a Ruby expert, but I don't see a .gemspec there. Might this be the
problem?

On Wed, Jul 10, 2013 at 2:00 AM, steve . leftathome@gmail.com wrote:

Not sure if anyone's already doing something like this, but I made
some REALLY MINOR modifications to the most ancient and venerable Ruby
depgraph tool and got it parsing cookbook metadata instead of C# code and
installed gems.

GitHub - leftathome/depgraph: Modified version of Ruby depgraph that parses Chef cookbook metadata.

Default behavior is to parse, fire up graphviz and show you a
dependency graph. That's fine but I wanted it as a JSON object.

I added the -json switch to allow you to generate an object that looks
like this:

{ "cookbook" : [ "dependency1", "dependency2"],
...
}

Hope someone out there finds this useful.

\m/
do it!

On Tue, Jul 9, 2013 at 5:00 PM, steve . leftathome@gmail.com wrote:

Not sure if anyone's already doing something like this, but I made some
REALLY MINOR modifications to the most ancient and venerable Ruby depgraph
tool and got it parsing cookbook metadata instead of C# code and installed
gems.

GitHub - leftathome/depgraph: Modified version of Ruby depgraph that parses Chef cookbook metadata.

Default behavior is to parse, fire up graphviz and show you a dependency
graph. That's fine but I wanted it as a JSON object.

I added the -json switch to allow you to generate an object that looks
like this:

{ "cookbook" : [ "dependency1", "dependency2"],
...
}

Hope someone out there finds this useful.

Like! :slight_smile:
On Jul 10, 2013 2:30 AM, "Ranjib Dey" dey.ranjib@gmail.com wrote:

\m/
do it!

On Tue, Jul 9, 2013 at 5:00 PM, steve . leftathome@gmail.com wrote:

Not sure if anyone's already doing something like this, but I made some
REALLY MINOR modifications to the most ancient and venerable Ruby depgraph
tool and got it parsing cookbook metadata instead of C# code and installed
gems.

GitHub - leftathome/depgraph: Modified version of Ruby depgraph that parses Chef cookbook metadata.

Default behavior is to parse, fire up graphviz and show you a dependency
graph. That's fine but I wanted it as a JSON object.

I added the -json switch to allow you to generate an object that looks
like this:

{ "cookbook" : [ "dependency1", "dependency2"],
...
}

Hope someone out there finds this useful.