Hi everybody,
I want to start a new project with chef automate.
I actually have a bunch of existing cookbooks, sometimes community, some of are privates.
I built a private supermarket too. I am looking for best practices, best way to restart from scratch.
What sort of cookbooks are you puting in your supermarket ?
Do i need to knife upload some cookbooks manually or do i need to git add every cookbooks one by one ?
Thanks for sharing your experiences with me 
For cookbook publishing, first you need to decide where your source of truth lies. Is it going to be:
Each of these has their pros and cons. (One thing you will hit quickly is where the source of truth for secrets should be and will it be different than cookbook source).
Ideally it’s source control (yes, I store my secrets encrypted in source control), and if you’re starting from scratch this is the recommended way, but not all shops are configured this way. Facebook has some great tools to support this - they have a tool called grocery delivery that can keep chef servers all synced up with source control. This can also be adapted to do the same for a private supermarket.
For my cookbook pipeline, yes I just have a pipeline trigger on any source code change, and the deliver stage runs “knife supermarket share”.
One problem that you will hit when you have a chef-repo style cookbook repository is figuring out which cookbook changed. You might have to write some code to support this. So you may want to go with the “one-cookbook-per-repo” style instead if that works for you.
Bootstrapping a private supermarket can be tough to automate if you are using community cookbooks. One problem you will hit immediately if you try to git add every cookbook one by one is that there will be a dependency order in “depends” for non-trivial cookbooks. If you don’t have all the dependencies for a cookbook, it will fail to publish to the supermarket. So you want something to scan for all dependencies and publish those. Unfortunately Chef Automate does not provide any tooling to help bootstrap a supermarket. You may need to “prime" the supermarket once by hand to get all the dependencies, but after that publishing cookbook-by-cookbook should be fine. (As after the initial, you should always have a cookbook’s dependencies, even if they might be a little older, and that at least won’t block publishing by knife supermarket share). You can write code to scan a cookbook’s dependencies, but the problem is getting the order right for the first time.
Also, hopefully you never forked or modified a chef cookbook with the same name as an upstream supermarket cookbook. In my shop, we take care to preface all our cookbook names with a unique prefix so we never conflict with upstream cookbook names. But that’s another consideration with a private supermarket.
Another issue that you will hit almost immediately with cookbooks is version bumping in the metadata. Some have a “delivery local” step for devs to check to see that the cookbook version hasn’t already been published, to remind them to bump the version. In my shop, we follow the Facebook approach and go versionless with our cookbooks and our devs use attributes as “feature flags”. But this requires some rigor, and Chef Automate doesn’t have great tooling to support feature flags. I supplement this with a service called “LaunchDarkly”.
Finally if you don’t have this already, you probably want to make sure you have some sort of binary store for your Chef Automate workflows to store state/files between stages/steps. Artifactory is a popular choice, and it does a great job to support the “source of truth” being source control. I’ve been experimenting with using Artifactory as the Chef cookbook store myself to replace supermarket. One great feature it has is the ability to have “virtual” repos that combine together different sources. So it solves the bootstrapping problem for a supermarket above by letting you mirror the upstream supermarket on the net, and combine that together with your local cookbooks. Unfortunately I’m still working through some speed issues with JFrog’s implementation in my pilot implementation - for my cookbook set, I’ve discovered that it is an order of magnitude slower than the official private supermarket, which makes it almost unusable. I’m working with JFrog to hopefully get this issue resolved (I also notice that in the latest Berkshelf it now has Artifactory as a source, which might help with this).
These are some things I can think of off the top of my head.
Oh, one other thing that you may be wondering with a private supermarket - can you have multiple sources in your Berksfile
Short answer - no, not really
It doesn’t work like you would expect, going to the first source, trying that, and if it fails going to the second source.
Unless the behavior of Berkshelf/Supermarket changes, expect to have only one cookbook repository in your Berksfile (why Artifactory looks so promising to me, as it supports layered/federated supermarkets).
——
Check out my Chef training video on O’Reilly Safari: https://www.safaribooksonline.com/library/view/learning-chef-for/9781491959442/ https://www.safaribooksonline.com/library/view/learning-chef-for/9781491959442/
Hi Misheska,
Thank you for sharing your experience.
Managing community cookbooks is a mess because you're not in charge of the code.
This morning, i am in this kind of use case...
I am trying to integrate 'cron' cookbook which is a direct dependency for chef-client.
I choosed to git clone the cookbook and integrate it as a freezed pizza in my automate ecosystem.
15 files inspected, 38 offenses detected
So now 2 solutions, find a way to bypass all errors, or fix the "bad practices".
I tried to mentalize what is a supermarket, to understand why people at Chef designed this.
Maybe we need to think about the words. In a supermarket you can buy freezed pizzas but you can also buy pizzas from a local guy. If you buy freeze pizzas you can do like my mum when i was kid, put fresh and tasties ingredients on it 
To be honest, i need to mentally "digest" this
Yes, unfortunately I end up using community cookbooks more for inspiration rather than in direct reuse. Ignoring the issue of passing cookstyle/foodcritic as that can be a contentious topic as some view it more of a style check than a functionality check, it can be very difficult to make reusable cookbooks that are at the right level of abstraction to be used by different companies/groups.
In my experience, I’ve only been able to directly reuse Facebook’s open source cookbooks in my shop, and that’s been about it. But even they use a model that might not work for everyone.
I wish I could spend more resources on my team to contribute back to the community, but I have to get every open source commit we make reviewed by a committee that only meets twice per year, so it takes a long time to provide contributions and by that time the upstream code has likely changed long since. I expect many other teams face similar obstacles.
We’ve used multiple sources in our Berksfiles for years without any issues. It works exactly like you stated above, if it doesn’t find the dependency in one source then it searches the other.