# Deleting objects from the chef server when deleted from git repo

**URL:** <https://discourse.chef.io/t/deleting-objects-from-the-chef-server-when-deleted-from-git-repo/3292>\
**Category:** Chef Infra (archive)\
**Created:** [November 14, 2012, 9:11pm UTC](https://discourse.chef.io/t/deleting-objects-from-the-chef-server-when-deleted-from-git-repo/3292 "2012-11-14T21:11:39Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Wes\_Morgan](https://avatars.discourse-cdn.com/v4/letter/w/3be4f8/32.png) [@Wes\_Morgan](https://discourse.chef.io/u/Wes_Morgan)\
**Post date:** [November 14, 2012, 9:11pm UTC](https://discourse.chef.io/t/deleting-objects-from-the-chef-server-when-deleted-from-git-repo/3292/1 "2012-11-14T21:11:39Z")

</div>

I finally got around to setting up a Jenkins job to upload changes to the Chef server when I push them to the git repo. So far it’s _great_ having one source of truth for my Chef data.

However, the one remaining hole is deleting things from the Chef server when they’re deleted from the git repo. Is anyone already doing that in an automated fashion? If so, what is your favorite approach?

It seems there a few options out there for how to do it. Everything from parsing the git log to using a knife plugin to see what’s on the server but not in the local repo working copy.

I kind of wish there were just a “knife sync” command. 🙂

Wes

---

<div class="post-metadata">

**Author:** ![realityforge](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/realityforge/32/395_2.png) [@realityforge](https://discourse.chef.io/u/realityforge)\
**Post date:** [November 14, 2012, 10:14pm UTC](https://discourse.chef.io/t/deleting-objects-from-the-chef-server-when-deleted-from-git-repo/3292/2 "2012-11-14T22:14:32Z")

</div>

Hi,

On Thu, Nov 15, 2012 at 8:11 AM, Wes Morgan [cap10morgan@gmail.com](mailto:cap10morgan@gmail.com) wrote:

> I finally got around to setting up a Jenkins job to upload changes to the Chef server when I push them to the git repo. So far it's _great_ having one source of truth for my Chef data.
> 
> However, the one remaining hole is deleting things from the Chef server when they're deleted from the git repo. Is anyone already doing that in an automated fashion? If so, what is your favorite approach?

We actually use a fancy bit of ruby script using the spice gem that  
does uploads, deletes and runs a bunch of auditing and validation  
functions on the chef server. I have heard people have also done  
something similar with jclouds but never looked into it.

However a quick and dirty script may work just as well. Something like

knife cookbook list | awk '{print $1}' | sort \> server.txt  
ls cookbooks | sort \> local.txt  
diff server.txt local.txt | grep '\<' | awk '{print $2}' | xargs  
--verbose knife cookbook delete -y

--  
Cheers,

Peter Donald

---

<div class="post-metadata">

**Author:** ![Wes\_Morgan](https://avatars.discourse-cdn.com/v4/letter/w/3be4f8/32.png) [@Wes\_Morgan](https://discourse.chef.io/u/Wes_Morgan)\
**Post date:** [November 14, 2012, 10:31pm UTC](https://discourse.chef.io/t/deleting-objects-from-the-chef-server-when-deleted-from-git-repo/3292/3 "2012-11-14T22:31:49Z")

</div>

On Nov 14, 2012, at 3:14 PM, Peter Donald [peter@realityforge.org](mailto:peter@realityforge.org) wrote:

> Hi,
> 
> On Thu, Nov 15, 2012 at 8:11 AM, Wes Morgan [cap10morgan@gmail.com](mailto:cap10morgan@gmail.com) wrote:
> 
> > I finally got around to setting up a Jenkins job to upload changes to the Chef server when I push them to the git repo. So far it's _great_ having one source of truth for my Chef data.
> > 
> > However, the one remaining hole is deleting things from the Chef server when they're deleted from the git repo. Is anyone already doing that in an automated fashion? If so, what is your favorite approach?
> 
> We actually use a fancy bit of ruby script using the spice gem that  
> does uploads, deletes and runs a bunch of auditing and validation  
> functions on the chef server. I have heard people have also done  
> something similar with jclouds but never looked into it.

> However a quick and dirty script may work just as well. Something like
> 
> knife cookbook list | awk '{print $1}' | sort \> server.txt  
> ls cookbooks | sort \> local.txt  
> diff server.txt local.txt | grep '\<' | awk '{print $2}' | xargs  
> --verbose knife cookbook delete -y

Brilliant, thanks.

Running this made me remember that cookbooks aren't segregated by organization in hosted Chef, so this would delete things that I don't want to delete. Hmm, maybe this needs some more thought. I think deletes would have to be triggered via git log parsing to get around that.

> --  
> Cheers,
> 
> Peter Donald

---

<div class="post-metadata">

**Author:** ![Torben\_Knerr](https://avatars.discourse-cdn.com/v4/letter/t/ba8739/32.png) [@Torben\_Knerr](https://discourse.chef.io/u/Torben_Knerr)\
**Post date:** [November 20, 2012, 7:19pm UTC](https://discourse.chef.io/t/deleting-objects-from-the-chef-server-when-deleted-from-git-repo/3292/4 "2012-11-20T19:19:03Z")

</div>

Many times I thought of having only the git repo as the single source of  
truth, but since there seems to be no out-of-the-box solution for it kept  
just a thought.

For one thing I want the single source of truth in a git repository. If I  
can't have that, I'd like to have an audit trail at least...

@Peter, Wes: would you mind sharing your scripts?

Cheers, Torben  
Am 14.11.2012 23:31 schrieb "Wes Morgan" [cap10morgan@gmail.com](mailto:cap10morgan@gmail.com):

> On Nov 14, 2012, at 3:14 PM, Peter Donald [peter@realityforge.org](mailto:peter@realityforge.org) wrote:
> 
> > Hi,
> > 
> > On Thu, Nov 15, 2012 at 8:11 AM, Wes Morgan [cap10morgan@gmail.com](mailto:cap10morgan@gmail.com)  
> > wrote:
> > 
> > > I finally got around to setting up a Jenkins job to upload changes to  
> > > the Chef server when I push them to the git repo. So far it's _great_  
> > > having one source of truth for my Chef data.
> > > 
> > > However, the one remaining hole is deleting things from the Chef server  
> > > when they're deleted from the git repo. Is anyone already doing that in an  
> > > automated fashion? If so, what is your favorite approach?
> > 
> > We actually use a fancy bit of ruby script using the spice gem that  
> > does uploads, deletes and runs a bunch of auditing and validation  
> > functions on the chef server. I have heard people have also done  
> > something similar with jclouds but never looked into it.
> 
> > However a quick and dirty script may work just as well. Something like
> > 
> > knife cookbook list | awk '{print $1}' | sort \> server.txt  
> > ls cookbooks | sort \> local.txt  
> > diff server.txt local.txt | grep '\<' | awk '{print $2}' | xargs  
> > --verbose knife cookbook delete -y
> 
> Brilliant, thanks.
> 
> Running this made me remember that cookbooks aren't segregated by  
> organization in hosted Chef, so this would delete things that I don't want  
> to delete. Hmm, maybe this needs some more thought. I think deletes would  
> have to be triggered via git log parsing to get around that.
> 
> > --  
> > Cheers,
> > 
> > Peter Donald

---

<div class="post-metadata">

**Author:** ![O\_T\_Suarez](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/o_t_suarez/32/621_2.png) [@O\_T\_Suarez](https://discourse.chef.io/u/O_T_Suarez)\
**Post date:** [November 20, 2012, 7:32pm UTC](https://discourse.chef.io/t/deleting-objects-from-the-chef-server-when-deleted-from-git-repo/3292/5 "2012-11-20T19:32:15Z")

</div>

Hi,  
I wrote this post-receive hook trying to achieve the same goal:

> <https://gist.github.com/devops-ar/4075735>

To deploy a new server, you create a databag json file, commit and  
push the file. Same for cookbooks, roles, etc. The hook uploads data  
to the chef server and takes care of executing scripts, reporting  
everything to an irc channel.  
For the audit trail part, the knife execution is done by rundeck, the  
url for the output is available on the irc channel.  
The workflow is detailed on this blog post (sry, link in spanish,  
english version coming up soon)  
[http://www.devops.com.ar/blog/2012/11/20/instalando-servidores-con-un-commit/](http://www.devops.com.ar/blog/2012/11/20/instalando-servidores-con-un-commit/)  
regards,  
osvaldo

On Tue, Nov 20, 2012 at 4:19 PM, Torben Knerr [ukio@gmx.de](mailto:ukio@gmx.de) wrote:

> Many times I thought of having only the git repo as the single source of  
> truth, but since there seems to be no out-of-the-box solution for it kept  
> just a thought.
> 
> For one thing I want the single source of truth in a git repository. If I  
> can't have that, I'd like to have an audit trail at least...
> 
> @Peter, Wes: would you mind sharing your scripts?
> 
> Cheers, Torben
> 
> Am 14.11.2012 23:31 schrieb "Wes Morgan" [cap10morgan@gmail.com](mailto:cap10morgan@gmail.com):
> 
> > On Nov 14, 2012, at 3:14 PM, Peter Donald [peter@realityforge.org](mailto:peter@realityforge.org) wrote:
> > 
> > > Hi,
> > > 
> > > On Thu, Nov 15, 2012 at 8:11 AM, Wes Morgan [cap10morgan@gmail.com](mailto:cap10morgan@gmail.com)  
> > > wrote:
> > > 
> > > > I finally got around to setting up a Jenkins job to upload changes to  
> > > > the Chef server when I push them to the git repo. So far it's _great_ having  
> > > > one source of truth for my Chef data.
> > > > 
> > > > However, the one remaining hole is deleting things from the Chef server  
> > > > when they're deleted from the git repo. Is anyone already doing that in an  
> > > > automated fashion? If so, what is your favorite approach?
> > > 
> > > We actually use a fancy bit of ruby script using the spice gem that  
> > > does uploads, deletes and runs a bunch of auditing and validation  
> > > functions on the chef server. I have heard people have also done  
> > > something similar with jclouds but never looked into it.
> > 
> > > However a quick and dirty script may work just as well. Something like
> > > 
> > > knife cookbook list | awk '{print $1}' | sort \> server.txt  
> > > ls cookbooks | sort \> local.txt  
> > > diff server.txt local.txt | grep '\<' | awk '{print $2}' | xargs  
> > > --verbose knife cookbook delete -y
> > 
> > Brilliant, thanks.
> > 
> > Running this made me remember that cookbooks aren't segregated by  
> > organization in hosted Chef, so this would delete things that I don't want  
> > to delete. Hmm, maybe this needs some more thought. I think deletes would  
> > have to be triggered via git log parsing to get around that.
> > 
> > > --  
> > > Cheers,
> > > 
> > > Peter Donald

---

<div class="post-metadata">

**Author:** ![Torben\_Knerr](https://avatars.discourse-cdn.com/v4/letter/t/ba8739/32.png) [@Torben\_Knerr](https://discourse.chef.io/u/Torben_Knerr)\
**Post date:** [November 20, 2012, 7:54pm UTC](https://discourse.chef.io/t/deleting-objects-from-the-chef-server-when-deleted-from-git-repo/3292/6 "2012-11-20T19:54:17Z")

</div>

Awesome, thanks for sharing!  
Am 20.11.2012 20:32 schrieb "O. T. Suarez" [otsuarez@gmail.com](mailto:otsuarez@gmail.com):

> Hi,  
> I wrote this post-receive hook trying to achieve the same goal:  
> [git hook for chef deployments · GitHub](https://gist.github.com/4075735#file_post_receive)  
> To deploy a new server, you create a databag json file, commit and  
> push the file. Same for cookbooks, roles, etc. The hook uploads data  
> to the chef server and takes care of executing scripts, reporting  
> everything to an irc channel.  
> For the audit trail part, the knife execution is done by rundeck, the  
> url for the output is available on the irc channel.  
> The workflow is detailed on this blog post (sry, link in spanish,  
> english version coming up soon)
> 
> [http://www.devops.com.ar/blog/2012/11/20/instalando-servidores-con-un-commit/](http://www.devops.com.ar/blog/2012/11/20/instalando-servidores-con-un-commit/)  
> regards,  
> osvaldo
> 
> On Tue, Nov 20, 2012 at 4:19 PM, Torben Knerr [ukio@gmx.de](mailto:ukio@gmx.de) wrote:
> 
> > Many times I thought of having only the git repo as the single source of  
> > truth, but since there seems to be no out-of-the-box solution for it kept  
> > just a thought.
> > 
> > For one thing I want the single source of truth in a git repository. If I  
> > can't have that, I'd like to have an audit trail at least...
> > 
> > @Peter, Wes: would you mind sharing your scripts?
> > 
> > Cheers, Torben
> > 
> > Am 14.11.2012 23:31 schrieb "Wes Morgan" [cap10morgan@gmail.com](mailto:cap10morgan@gmail.com):
> > 
> > > On Nov 14, 2012, at 3:14 PM, Peter Donald [peter@realityforge.org](mailto:peter@realityforge.org)  
> > > wrote:
> > > 
> > > > Hi,
> > > > 
> > > > On Thu, Nov 15, 2012 at 8:11 AM, Wes Morgan [cap10morgan@gmail.com](mailto:cap10morgan@gmail.com)  
> > > > wrote:
> > > > 
> > > > > I finally got around to setting up a Jenkins job to upload changes to  
> > > > > the Chef server when I push them to the git repo. So far it's  
> > > > > _great_ having  
> > > > > one source of truth for my Chef data.
> > > > > 
> > > > > However, the one remaining hole is deleting things from the Chef  
> > > > > server  
> > > > > when they're deleted from the git repo. Is anyone already doing that  
> > > > > in an  
> > > > > automated fashion? If so, what is your favorite approach?
> > > > 
> > > > We actually use a fancy bit of ruby script using the spice gem that  
> > > > does uploads, deletes and runs a bunch of auditing and validation  
> > > > functions on the chef server. I have heard people have also done  
> > > > something similar with jclouds but never looked into it.
> > > 
> > > > However a quick and dirty script may work just as well. Something like
> > > > 
> > > > knife cookbook list | awk '{print $1}' | sort \> server.txt  
> > > > ls cookbooks | sort \> local.txt  
> > > > diff server.txt local.txt | grep '\<' | awk '{print $2}' | xargs  
> > > > --verbose knife cookbook delete -y
> > > 
> > > Brilliant, thanks.
> > > 
> > > Running this made me remember that cookbooks aren't segregated by  
> > > organization in hosted Chef, so this would delete things that I don't  
> > > want  
> > > to delete. Hmm, maybe this needs some more thought. I think deletes  
> > > would  
> > > have to be triggered via git log parsing to get around that.
> > > 
> > > > --  
> > > > Cheers,
> > > > 
> > > > Peter Donald

---

<div class="post-metadata">

**Author:** ![steve](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/steve/32/294_2.png) [@steve](https://discourse.chef.io/u/steve)\
**Post date:** [November 20, 2012, 11:22pm UTC](https://discourse.chef.io/t/deleting-objects-from-the-chef-server-when-deleted-from-git-repo/3292/7 "2012-11-20T23:22:29Z")

</div>

We've built a system around Jenkins that attempts to shore up trust in the  
Git repo. It uses a volatile Chef server for testing purposes -- one  
testing assumption is that only the latest and greatest versions of  
everything will be tested, so version constraints are ignored.

I'm now looking at refactoring this system to use Librarian, but the  
biggest hurdle I see is determining when and what changed in one of the  
dependent Git repositories as a Jenkins build trigger. I'm pretty sure  
this is going to end with me writing a Jenkins build trigger plugin and I  
have the feeling I am going to find that very unpleasant.

If this does, in fact, come to pass, I'll see about open-sourcing the build  
trigger. (Sorry, I don't think I can offer up the entire system at present  
... test-kitchen offers a subset of this functionality, though!)

On Tue, Nov 20, 2012 at 11:54 AM, Torben Knerr [ukio@gmx.de](mailto:ukio@gmx.de) wrote:

> Awesome, thanks for sharing!  
> Am 20.11.2012 20:32 schrieb "O. T. Suarez" [otsuarez@gmail.com](mailto:otsuarez@gmail.com):
> 
> Hi,
> 
> > I wrote this post-receive hook trying to achieve the same goal:  
> > [git hook for chef deployments · GitHub](https://gist.github.com/4075735#file_post_receive)  
> > To deploy a new server, you create a databag json file, commit and  
> > push the file. Same for cookbooks, roles, etc. The hook uploads data  
> > to the chef server and takes care of executing scripts, reporting  
> > everything to an irc channel.  
> > For the audit trail part, the knife execution is done by rundeck, the  
> > url for the output is available on the irc channel.  
> > The workflow is detailed on this blog post (sry, link in spanish,  
> > english version coming up soon)
> > 
> > [http://www.devops.com.ar/blog/2012/11/20/instalando-servidores-con-un-commit/](http://www.devops.com.ar/blog/2012/11/20/instalando-servidores-con-un-commit/)  
> > regards,  
> > osvaldo
> > 
> > On Tue, Nov 20, 2012 at 4:19 PM, Torben Knerr [ukio@gmx.de](mailto:ukio@gmx.de) wrote:
> > 
> > > Many times I thought of having only the git repo as the single source of  
> > > truth, but since there seems to be no out-of-the-box solution for it  
> > > kept  
> > > just a thought.
> > > 
> > > For one thing I want the single source of truth in a git repository. If  
> > > I  
> > > can't have that, I'd like to have an audit trail at least...
> > > 
> > > @Peter, Wes: would you mind sharing your scripts?
> > > 
> > > Cheers, Torben
> > > 
> > > Am 14.11.2012 23:31 schrieb "Wes Morgan" [cap10morgan@gmail.com](mailto:cap10morgan@gmail.com):
> > > 
> > > > On Nov 14, 2012, at 3:14 PM, Peter Donald [peter@realityforge.org](mailto:peter@realityforge.org)  
> > > > wrote:
> > > > 
> > > > > Hi,
> > > > > 
> > > > > On Thu, Nov 15, 2012 at 8:11 AM, Wes Morgan [cap10morgan@gmail.com](mailto:cap10morgan@gmail.com)  
> > > > > wrote:
> > > > > 
> > > > > > I finally got around to setting up a Jenkins job to upload changes  
> > > > > > to  
> > > > > > the Chef server when I push them to the git repo. So far it's  
> > > > > > _great_ having  
> > > > > > one source of truth for my Chef data.
> > > > > > 
> > > > > > However, the one remaining hole is deleting things from the Chef  
> > > > > > server  
> > > > > > when they're deleted from the git repo. Is anyone already doing  
> > > > > > that in an  
> > > > > > automated fashion? If so, what is your favorite approach?
> > > > > 
> > > > > We actually use a fancy bit of ruby script using the spice gem that  
> > > > > does uploads, deletes and runs a bunch of auditing and validation  
> > > > > functions on the chef server. I have heard people have also done  
> > > > > something similar with jclouds but never looked into it.
> > > > 
> > > > > However a quick and dirty script may work just as well. Something  
> > > > > like
> > > > > 
> > > > > knife cookbook list | awk '{print $1}' | sort \> server.txt  
> > > > > ls cookbooks | sort \> local.txt  
> > > > > diff server.txt local.txt | grep '\<' | awk '{print $2}' | xargs  
> > > > > --verbose knife cookbook delete -y
> > > > 
> > > > Brilliant, thanks.
> > > > 
> > > > Running this made me remember that cookbooks aren't segregated by  
> > > > organization in hosted Chef, so this would delete things that I don't  
> > > > want  
> > > > to delete. Hmm, maybe this needs some more thought. I think deletes  
> > > > would  
> > > > have to be triggered via git log parsing to get around that.
> > > > 
> > > > > --  
> > > > > Cheers,
> > > > > 
> > > > > Peter Donald