# An deep topic

**URL:** <https://discourse.chef.io/t/an-deep-topic/5596>\
**Category:** Chef Infra (archive)\
**Created:** [July 8, 2014, 3:05am UTC](https://discourse.chef.io/t/an-deep-topic/5596 "2014-07-08T03:05:20Z")\
**Posts on this page:** 4\
**Page:** 2

<div class="post-metadata">

**Author:** ![Lamont\_Granquist\_OLD](https://avatars.discourse-cdn.com/v4/letter/l/7feea3/32.png) [@Lamont\_Granquist\_OLD](https://discourse.chef.io/u/Lamont_Granquist_OLD)\
**Post date:** [July 8, 2014, 7:32pm UTC](https://discourse.chef.io/t/an-deep-topic/5596/21 "2014-07-08T19:32:52Z")

</div>

On 7/8/14, 11:16 AM, John E. Vincent (lusis) wrote:

> So that post from Lamont is awesome but it kinda points to the thing I  
> was bringing up. This section right here:
> 
> "This is the correct way to start building Chef Cookbooks"
> 
> And then
> 
> "Not that we're not using Test Kitchen to actually test our cookbook."
> 
> Wait what? Then why is it "Test" Kitchen?
> 
> When I asked all the `why` questions, this is a good example. Why  
> shouldn't I use the documented `knife cookbook create`? Is it not  
> acceptable to the job of creating a cookbook? What makes this more  
> confusing is that this is coming from someone who works for Chef.

The "knife cookbook create" command was probably written 5 years ago.  
ChefDK started being written a few months ago and the documentation  
clearly hasn't caught up. As you like to point out, ChefDK is currently  
only an 0.1.0 version. If you'd like a clear statement I think its  
fairly obvious that "chef generate cookbook" will become the way to do  
this and will entirely replace "knife cookbook create", "berks  
cookbook", and the  
knife-plugin-that-does-the-same-thing-with-the-funny-name-that-i-cant-think-of-right-now.  
We will not go out of our way to break the other commands, but we're  
putting our effort into ChefDK and 'chef generate' and there's 3 or 4  
software devs working on it right now, where 'knife cookbook create' is  
going to have to largely user driven for fixes and 'berks cookbook' was  
trying to solve this problem while being bolted onto the side of a  
dependency solver. We're set up much better to be successful at  
producing a good cookbook skeleton generator, so its very likely that  
'chef generate cookbook' becomes the dominant way to solve that problem  
in the future.

And the problem here is that we have to balance being able to throw our  
effort at solving a problem against being 'arrogant'. We do control the  
learnchef documentation and are the first place people go to and we have  
a lot of software devs working for us and can throw human resources at  
coding problems, which should set us up to be successful at tackling  
problems that we focus on. At the same time we don't want to state that  
'berks cookbook is shit, don't use it' or something which would really  
be horribly arrogant. We also recognize that we don't always do the  
right thing. If 'chef generate cookbook' turns out to be some overly  
engineered terribleness that is difficult to use then we don't want to  
preclude someone in the community with a better way of doing things to  
go do it and show us a bitter/simpler/easier/cleaner/whatever way.  
That's largely what happened with test-kitchen and berkshelf, which were  
projects spun up entirely in the community.

And right now we problem can't say to replace whatever cookbook  
generators you're using with 'chef generate cookbook'. It still is only  
a 0.1.0 version so it almost certainly doesn't cover all the use cases,  
and may not be "production ready". In fact, I documented in that write  
up a major bug making it not even useful in that trivial case without  
hand editing your .kitchen.yml to fix something. So we can't state that  
everyone should use it, because clearly everyone should not use it right  
now. But I wrote that HOWTO up because we're getting there to the point  
where its useful in the simplest case to bootstrap someone from zero  
into writing a simple cookbook -- which is the first baby step.

And the problem with 'test-kitchen' is just in naming the thing. In  
order to solve the TDD problem, test-kitchen had to solve the problem of  
spinning up and destroying virts, installing chef on the virts, using  
berkshelf to sync the cookbook under test and all its dependencies,  
supporting vagrant and various other clouds -- in addition to the goal  
of running tests. Well, turns out that you can just not run the tests  
in test-kitchen and what you get is a nice facade where a single command  
can take the cookbook that you are developing and fire up a virt,  
install omnibus-chef, sync all the cookbooks and converge the node.  
Also if you use that utility as part of your cookbook development cycle  
then it becomes a no-brainer to later add tests to your cookbooks. It  
wouldn't make any sense at this point to write something else to do  
everything that TK does _except_ running tests. It also doesn't make a  
whole lot of sense to rename TK at this point just because there's this  
edge case of TK-without-tests doing lots of incredibly useful work.

And one more thing I'll say on this thread is that it sounds like you've  
got a whole lot of frustration with the TDD cookbook design paradigm.  
And I can related because my initial impression was bad. I've been a  
actual software developer for only about 3 years now where my background  
is in operations and perl hacking. And I initially hated TDD cookbooks  
-- Berksfile, Rakefile, Thorfile, various different spec directories,  
and a half dozen .yml dotfile turds and what does it get me? So, fuck  
all these hippie kids and their ruby TDD shit (tableflip). I can easily  
view TDD cookbooks from the perspective of a non-ruby non-programming  
system administrator coming to Chef for the first time, and its a  
completely horrific vision, and the casual way that you are expected to  
be already familiar with bundler, rake, thor, rubocop, etc and pretty  
much can't use TDD cookbooks unless you really grok ruby already. Since  
I work here, I had to knuckle down and start using it, and found that it  
was _extremely_ powerful when done right. It is the 21st century way of  
doing infrastructure engineering and configuration management. But its  
still very convoluted and with sharp edges all over the place. The  
whole concept of ChefDK is trying to address all of that frustration and  
minimize it. And that HOWTO I wrote up is an example of that. And  
consider what it buys you -- I probably ran into the idea of spinning up  
a virt and testing CFEngine code against it back in 2003 when VMWare was  
first becoming popular and scoffed at it. I used to just blow code into  
a master branch (that was sync'd to everything in production) protected  
by some CFEngine classes to only apply it to some workstations and then  
do "Testing" by gradually pushing the changes out to more and more  
servers across the enterprise until I crashed something and had to back  
off and fix it (but with experience I could anticipate and minimize that  
from happening and grow my virtual neckbeard out longer than anyone  
else). Now I can spin up a virt on my laptop and throw production  
configuration code against it and see what happens. In retrospect that  
is seriously fucking amazing.

---

<div class="post-metadata">

**Author:** ![adamhjk](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/adamhjk/32/190_2.png) [@adamhjk](https://discourse.chef.io/u/adamhjk)\
**Post date:** [July 8, 2014, 8:11pm UTC](https://discourse.chef.io/t/an-deep-topic/5596/22 "2014-07-08T20:11:07Z")

</div>

I think this is clearly something we need to add to the roadmap and the  
server by default.

Adam

On Tue, Jul 8, 2014 at 10:51 AM, Jeff Byrnes [jeff@evertrue.com](mailto:jeff@evertrue.com) wrote:

> It’d be absolutely stellar if the Chef Server also provided a universe  
> endpoint; that would obviate our need for a Berkshelf API server entirely,  
> especially since we don’t have any particular need for our own Supermarket  
> right now. The alternative of having Supermarket baked-in to the Chef  
> Server would also be welcome.
> 
> On the larger topic of Chef providing us with The Way to author cookbooks,  
> I agree with Dylan’s points that we don’t necessarily need a One True Way,  
> but some more direction on a Few Good Ways would be very welcome. And I  
> wholeheartedly agree that the website examples should be solid examples.
> 
> --  
> Jeff Byrnes  
> @berkleebassist [http://twitter.com/berkleebassist](http://twitter.com/berkleebassist)  
> Operations Engineer  
> EverTrue [http://www.evertrue.com/](http://www.evertrue.com/)  
> 704.516.4628
> 
> On July 8, 2014 at 1:20:57 PM, Lamont Granquist ([lamont@opscode.com](mailto:lamont@opscode.com))  
> wrote:
> 
> On 7/7/14, 8:05 PM, John E. Vincent (lusis) wrote:
> 
> > FWIW I didn't have a problem with Berkshelf originally. One: I didn't  
> > use it and didn't HAVE to use it. Two: It didn't have this entirely  
> > new service it depended on.
> > 
> > Now I can't avoid it and it requires a separate internet service  
> > (though now apparently rolled into the supermarket) just to be used.  
> > Just addressing this one issue...
> 
> I hated the Berkshelf API with a passion initially. I still think its a  
> hack, but its a hack around problems that needed to be solved. The  
> first problem is just speed. Berkshelf needs to depsolve against all  
> the cookbooks on the community site, and doing that with a single  
> request for every cookbook was slow. So the /universe endpoint was  
> created to slurp it all down. That endpoint is now integrated with the  
> community site, and what is going on is essentially exactly what happens  
> when you use 'gem install' and it goes and hits rubygems to pull down  
> the list of current gems so that the rubygems depsolver can figure out  
> what gems to install. Its that problem, and its now solved in a very  
> similar fashion. There's another obvious source of cookbooks -- the  
> chef server -- that you should be able to depsolve against as well, and  
> we should add a universe endpoint (probably  
> /organizations//cookbook\_universe or something) to the chef-server  
> to solve it.
> 
> The other problem is one dealing with metadata.rb and metadata.json that  
> has its roots in configuration-as-code. The problem is that if you ship  
> around metadata.rb and it has pure-ruby code in it that references  
> external data you will not necessarily get the same metadata.json result  
> when you evaluate it. There are two chef tickets on this:
> 
> [https://tickets.opscode.com/browse/CHEF-4810](https://tickets.opscode.com/browse/CHEF-4810)  
> [https://tickets.opscode.com/browse/CHEF-4811](https://tickets.opscode.com/browse/CHEF-4811)
> 
> There's also Jamie's blog post on it:
> 
> [http://blog.vialstudios.com/the-importance-of-compiled-metadata/](http://blog.vialstudios.com/the-importance-of-compiled-metadata/)
> 
> And there's my issue that I opened against berkshelf where you can see  
> it tied all together:
> 
> [Respect transitive dependencies on git branches over community site · Issue #1126 · berkshelf/berkshelf · GitHub](https://github.com/berkshelf/berkshelf/issues/1126)
> 
> The result of that leads to the lack of being able to use transitive  
> github dependencies and the need to serve your own API server. The use  
> most important case which is driving it is an important one which is  
> that people are using ruby code in metadata.rb to drive their version  
> numbers off of CI/CD (e.g. jenkins) and in that case the artifacts need  
> to have compiled metadata.json so that all the actors involved in the  
> system agree on the version number in the metadata. Since CI/CD is  
> important, we don't want to break that.
> 
> Eliminating the need for 'finalized artifacts' and API servers may not  
> be a use case that you care about, but we can't just break the users  
> that do care about it because you don't. My initial reaction to the  
> problem is just to do a (tableflip) on configuration-as-code and think  
> to myself that we should deprecate metadata.rb and mandate using  
> metadata.json (back-to-the-future time). There might be a better way to  
> address the problem, though, which looks at the CD/CI use case and fixes  
> their version numbering use case a better way. I'm somewhat optimistic  
> that in the future that the berks API server will have completely gone  
> away.
> 
> I'd also be interested in solutions like baking supermarket into every  
> chef server deploy. Then everyone would have a local repo to push  
> cookbooks to and a /universe endpoint. Then that just becomes your  
> local berks API server.
> 
> So, if I've got any point here its that I'm very grouchy about the berks  
> API server because I see it as something which is a barrier to entry of  
> berkshelf adoption and its a problem that need to get solved. But  
> being grouchy about it without understanding why its there (Jamie is  
> pretty smart, he didn't just write it because he felt like writing some  
> server code) is not being productive. We need to understand the reasons  
> why things are there, and the use cases that drive them and come up with  
> solutions to the problems, rather than just complain. Just complaining  
> about the berks API is bad signalling since it just causes flak to the  
> people working on berkshelf. What you want to signal is the chef  
> server team that they should integrate the /universe endpoint, or that  
> supermarket should be bundled with chef server, or other solutions like  
> that. What the chef server team hears right now is that Berkshelf sucks,  
> which sounds entirely like a Berkshelf problem. That's not useful in  
> helping them to determine what they could do better with the server code  
> or packaging to help solve that problem -- and I'm fairly convinced that  
> this is an ecosystem problem, and that Berkshelf is what it is because  
> its been developed in isolation from work going on in the client and  
> server, and that needs to change in order to make it better.

---

<div class="post-metadata">

**Author:** ![adamhjk](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/adamhjk/32/190_2.png) [@adamhjk](https://discourse.chef.io/u/adamhjk)\
**Post date:** [July 8, 2014, 8:18pm UTC](https://discourse.chef.io/t/an-deep-topic/5596/23 "2014-07-08T20:18:16Z")

</div>

We should clearly ship a /universe endpoint in the chef server.

As stewards of Berkshelf, we didn't take over the project from its authors

- that was, I hope, quite clear. What we did commit to doing was helping  
develop it, maintain it, cover some of its costs, and start integrating it  
tightly into the Chef DK.

With vagrant-berkshelf, the reality is that it was a project created by  
people in our community, all of whom moved on from using it (and instead  
focused on Test Kitchen.) They then decided to no longer maintain it.

Within Chef Software, the Official Deal (tm) is that the tooling in the  
Chef DK is the tooling we're having our developers fine tune, because it's  
the best way forward (I love me some Vagrant, but supporting Docker, cloud  
providers, etc in Test Kitchen is better for cookbook development that  
needs it.) So there is no incentive to dedicate our resources to doing paid  
maintenance of vagrant-berkshelf, rather than shaving the yaks needed to  
bring some sanity to an ecosystem that had rapidly evolved. Hence, it needs  
a new maintainer who cares and loves it, just like its original authors did.

Lets get that API into Chef Server.

This message was dope, and you should write more of them without fear of  
hurting anyones feelings.

Adam

On Tue, Jul 8, 2014 at 11:25 AM, Zac Stevens [zts@cryptocracy.com](mailto:zts@cryptocracy.com) wrote:

> On Tue, Jul 8, 2014 at 6:20 PM, Lamont Granquist [lamont@opscode.com](mailto:lamont@opscode.com)  
> wrote:
> 
> > Just complaining about the berks API is bad signalling since it just  
> > causes flak to the people working on berkshelf. What you want to signal  
> > is the chef server team that they should integrate the /universe endpoint,  
> > or that supermarket should be bundled with chef server, or other solutions  
> > like that. What the chef server team hears right now is that Berkshelf  
> > sucks, which sounds entirely like a Berkshelf problem. That's not useful  
> > in helping them to determine what they could do better with the server code  
> > or packaging to help solve that problem -- and I'm fairly convinced that  
> > this is an ecosystem problem, and that Berkshelf is what it is because its  
> > been developed in isolation from work going on in the client and server,  
> > and that needs to change in order to make it better.
> 
> I've refrained from saying much publicly about the berks API precisely  
> because I don't want to give flak to the people working on Berkshelf - that  
> said, I would very much like to give a little flack to Chef (the company)  
> about it. Although the PR[1] announcing corporate stewardship of the  
> project didn't say much about what that actually _meant_, I figured that it  
> would mean a little less itch-scratching and a little more thought about  
> how users would be adversely affected by changes.
> 
> Then the vagrant-berkshelf plugin was deprecated[2]. I understand why an  
> open source team would do so, and I don't fault them for it - on the other  
> hand, I would criticise Chef (the company) for that same decision. I can  
> only assume that Chef felt that their stewardship of Berkshelf didn't  
> extend to vagrant-berkshelf - which is a shame, since test-kitchen isn't a  
> replacement for every use case.
> 
> Onto Berkshelf 3, the first major release under Chef's stewardship.  
> Previously, I was perfectly happy using Hosted Chef as an artifact  
> repository in my Berkshelf workflow. With Berks 3, I now have to run my  
> own berks API server to preserve that functionality. I chose Hosted Chef  
> to avoid running my own Chef server, so this really grates[3]. Again, I  
> understand this decision by an open source team, but I'm left wondering  
> where the corporate stewardship comes into it.
> 
> Here's the thing: Berkshelf 3 is great upgrade. I put off updating to it  
> for ages because of the amount of work involved, I still grumble about the  
> API server every other day, but it's a net win all the same. The  
> individuals working on it deserve a great big round of applause.
> 
> So, here's my message to the Chef Server team (and Chef, the company, at  
> large): Berkshelf 3 is a great tool, but it's let down by the fact that the  
> API server as a separate dependency. Please roll the API server into Chef  
> Server - especially Hosted Chef - so that I can stop saying "Berkshelf 3 is  
> a great too, but..." and start saying "Berkshelf 3 is a great tool!"
> 
> Thankyou,
> 
> Zac
> 
> 1: [Opscode To Steward Berkshelf - Chef Blog | Chef](http://www.getchef.com/blog/2013/11/12/opscode-to-steward-berkshelf/)  
> 2: [The future of Vagrant Berkshelf | Seth Vargo](https://sethvargo.com/the-future-of-vagrant-berkshelf/)  
> 3: for the moment, I only use Hosted Chef as an artifact repository - and  
> pay nothing for the privilege. I'd have made a lot more noise about this  
> if I felt entitled to complain.

---

<div class="post-metadata">

**Author:** ![coderanger](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/coderanger/32/5_2.png) [@coderanger](https://discourse.chef.io/u/coderanger)\
**Post date:** [July 8, 2014, 8:33pm UTC](https://discourse.chef.io/t/an-deep-topic/5596/24 "2014-07-08T20:33:19Z")

</div>

On Jul 8, 2014, at 11:21 AM, "John E. Vincent (lusis)" [lusis.org+chef-list@gmail.com](mailto:lusis.org+chef-list@gmail.com) wrote:

> Something I wanted to bring up here is that much of the Berks API could have been clarified with a communicated roadmap.
> 
> "Hey folks, we're going to start pushing the Berkshelf API as the canonical depsolver for cookbooks. The plan is to run a copy on Heroku, then run a copy as part of supermarket and eventually move it into Chef Server"

I think it is worth pointing out that berks-api (and now supermarket like it) are _not_ depsolvers. The only API used by berkshelf is /universe which just returns a list of all cookbooks that exist, their versions, and their dependencies. All depsolving still happens in berks and/or chef-server just like before berks3.

--Noah

[Previous page](https://discourse.chef.io/t/an-deep-topic/5596.md?page=1)