GitHub service hooks for chef server?


#1

Ohai Chefs,

Has anyone considered building a GitHub service hook for chef repos? The idea being, you could turn it on, plug in your Chef server’s URL and your credentials, and then pushes to your GitHub repo will automatically be uploaded to your Chef server (i.e. cookbooks, roles, environments, data bags, etc.). So git push would replace knife [thingy] from file / knife cookbook upload.

It seems like it would be a very convenient way to do the “files in git are the canonical representation of my Chef universe” pattern.

I know it’s better to have a CI server do this after running the cookbooks, etc. through various tests. But that is a pretty high bar to get setup, and there is inherent value in having your Git repo and you Chef server in sync all the time.

Thoughts?

Wes


#2

+1


@millisami
~ Sachin Sagar Rai
Ruby on Rails Developer
http://tfm.com.np
http://nepalonrails.tumblr.com
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)

On Saturday, October 6, 2012 at 1:14 AM, Wes Morgan wrote:

Ohai Chefs,

Has anyone considered building a GitHub service hook for chef repos? The idea being, you could turn it on, plug in your Chef server’s URL and your credentials, and then pushes to your GitHub repo will automatically be uploaded to your Chef server (i.e. cookbooks, roles, environments, data bags, etc.). So git push would replace knife [thingy] from file / knife cookbook upload.

It seems like it would be a very convenient way to do the “files in git are the canonical representation of my Chef universe” pattern.

I know it’s better to have a CI server do this after running the cookbooks, etc. through various tests. But that is a pretty high bar to get setup, and there is inherent value in having your Git repo and you Chef server in sync all the time.

Thoughts?

Wes


#3

It has been talked about a lot, would love to see it :slight_smile:

Sachin Sagar Rai millisami@gmail.com wrote:

+1


@millisami
~ Sachin Sagar Rai
Ruby on Rails Developer
http://tfm.com.np
http://nepalonrails.tumblr.com
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)

On Saturday, October 6, 2012 at 1:14 AM, Wes Morgan wrote:

Ohai Chefs,

Has anyone considered building a GitHub service hook for chef repos?
The idea being, you could turn it on, plug in your Chef server’s URL
and your credentials, and then pushes to your GitHub repo will
automatically be uploaded to your Chef server (i.e. cookbooks, roles,
environments, data bags, etc.). So git push would replace knife
[thingy] from file / knife cookbook upload.

It seems like it would be a very convenient way to do the “files in
git are the canonical representation of my Chef universe” pattern.

I know it’s better to have a CI server do this after running the
cookbooks, etc. through various tests. But that is a pretty high bar to
get setup, and there is inherent value in having your Git repo and you
Chef server in sync all the time.

Thoughts?

Wes


#4

For any Chef GitHub service hook to work, it will need to parse out what files were modified, added, and removed; determine the correct actions to take (uploading / removing cookbooks, nodes, roles, etc.); clone/pull the repo to get the files to act on (or just grab the modified/added files from their raw download URLs); and finally sync up with the Chef server.

I did some investigating into implementing that as a service hook. Here’s what I found:

The GitHub service hooks are all defined in this repo: https://github.com/github/github-services

They are written in Ruby, unsurprisingly.

It looks like the most common approach taken by service hooks is to forward the GitHub payload (metadata about the commits that were pushed and who pushed them, etc.) to a remote service that understands how to parse and act on it. Some hooks translate the payload themselves, but they are much simpler than what would be required to go from GitHub payload to Chef REST API calls. It also doesn’t appear that the service hook API gives you filesystem access to the repo, which we would need to be able to upload things directly from the service hook.

So I guess this would involve two parts:

  1. A web service that can accept GitHub service hook requests and take the appropriate actions.
  2. The GitHub service hook itself that would just forward the payload to your Chef server (or hosted Chef) using some secure method such as mixlib-authentication.

Ideally #1 would be built-in to Chef Server so using the GitHub service hook wouldn’t require setting up yet another service. But that’s a usability issue rather than an architectural one.

It shouldn’t be too difficult to implement all this, but I’m curious if Opscode has any opinion on the architecture and if they would be willing to accept a new component in Chef Server (including hosted Chef, since that is what I use) that can respond to GitHub payloads.

Wes

On Oct 5, 2012, at 1:57 PM, Noah Kantrowitz noah@coderanger.net wrote:

It has been talked about a lot, would love to see it :slight_smile:

Sachin Sagar Rai millisami@gmail.com wrote:
+1


@millisami
~ Sachin Sagar Rai
Ruby on Rails Developer
http://tfm.com.np
http://nepalonrails.tumblr.com
Sent with Sparrow

On Saturday, October 6, 2012 at 1:14 AM, Wes Morgan wrote:

Ohai Chefs,

Has anyone considered building a GitHub service hook for chef repos? The idea being, you could turn it on, plug in your Chef server’s URL and your credentials, and then pushes to your GitHub repo will automatically be uploaded to your Chef server (i.e. cookbooks, roles, environments, data bags, etc.). So git push would replace knife [thingy] from file / knife cookbook upload.

It seems like it would be a very convenient way to do the “files in git are the canonical representation of my Chef universe” pattern.

I know it’s better to have a CI server do this after running the cookbooks, etc. through various tests. But that is a pretty high bar to get setup, and there is inherent value in having your Git repo and you Chef server in sync all the time.

Thoughts?

Wes


#5

So, very experimental (like, I wrote it but haven’t even ever executed it) code to handle this:

You would need to run it on your own kit, on a system with a copy of your chef-repo git repository. Point github at it, and in theory, it will sync.

Pull requests happily accepted.

I doubt that something like this would become a component of Opscode’s Chef Server, given the requirements it places on workflow - but that wouldn’t be my call, it would be Christopher Brown’s. That said, this code would work just fine with Hosted Chef.

Best,
Adam

On Oct 5, 2012, at 3:24 PM, Wes Morgan wrote:

For any Chef GitHub service hook to work, it will need to parse out what files were modified, added, and removed; determine the correct actions to take (uploading / removing cookbooks, nodes, roles, etc.); clone/pull the repo to get the files to act on (or just grab the modified/added files from their raw download URLs); and finally sync up with the Chef server.

I did some investigating into implementing that as a service hook. Here’s what I found:

The GitHub service hooks are all defined in this repo: https://github.com/github/github-services

They are written in Ruby, unsurprisingly.

It looks like the most common approach taken by service hooks is to forward the GitHub payload (metadata about the commits that were pushed and who pushed them, etc.) to a remote service that understands how to parse and act on it. Some hooks translate the payload themselves, but they are much simpler than what would be required to go from GitHub payload to Chef REST API calls. It also doesn’t appear that the service hook API gives you filesystem access to the repo, which we would need to be able to upload things directly from the service hook.

So I guess this would involve two parts:

  1. A web service that can accept GitHub service hook requests and take the appropriate actions.
  2. The GitHub service hook itself that would just forward the payload to your Chef server (or hosted Chef) using some secure method such as mixlib-authentication.

Ideally #1 would be built-in to Chef Server so using the GitHub service hook wouldn’t require setting up yet another service. But that’s a usability issue rather than an architectural one.

It shouldn’t be too difficult to implement all this, but I’m curious if Opscode has any opinion on the architecture and if they would be willing to accept a new component in Chef Server (including hosted Chef, since that is what I use) that can respond to GitHub payloads.

Wes

On Oct 5, 2012, at 1:57 PM, Noah Kantrowitz <noah@coderanger.netmailto:noah@coderanger.net> wrote:

It has been talked about a lot, would love to see it :slight_smile:

Sachin Sagar Rai <millisami@gmail.commailto:millisami@gmail.com> wrote:
+1


@millisami
~ Sachin Sagar Rai
Ruby on Rails Developer
http://tfm.com.nphttp://tfm.com.np/
http://nepalonrails.tumblr.comhttp://nepalonrails.tumblr.com/
Sent with Sparrowhttp://www.sparrowmailapp.com/?sig

On Saturday, October 6, 2012 at 1:14 AM, Wes Morgan wrote:

Ohai Chefs,

Has anyone considered building a GitHub service hook for chef repos? The idea being, you could turn it on, plug in your Chef server’s URL and your credentials, and then pushes to your GitHub repo will automatically be uploaded to your Chef server (i.e. cookbooks, roles, environments, data bags, etc.). So git push would replace knife [thingy] from file / knife cookbook upload.

It seems like it would be a very convenient way to do the “files in git are the canonical representation of my Chef universe” pattern.

I know it’s better to have a CI server do this after running the cookbooks, etc. through various tests. But that is a pretty high bar to get setup, and there is inherent value in having your Git repo and you Chef server in sync all the time.

Thoughts?

Wes