Berkshelf with chef-repo: am I doing it right?

Chef Infra (archive)
1

Hi all,

Although I’m used to maintaining a standard chef-repo for my organisation’s cookbooks and their dependencies, I’ve decided to adopt Berkshelf because it makes it easier to track “unofficial” cookbooks, and forks of “official” cookbooks that have tweaks or fixes that I need.

Although I have it pretty much working for me, I have the niggling sense that I’m not doing it quite right. And all of the documentation and guidance I have found online seems to assume you are using Berkshelf to manage the dependencies of a cookbook you are working on in isolation. I have found almost nothing in the way of guidance on how to use it in the context of a chef-repo.

Here’s what I’ve done:

  1. Delete from my chef-repo all third-party cookbooks that I am simply using, not maintaining for my organisation.
  2. Create a Berksfile in the root of my chef-repo, and list all of the cookbooks I have just deleted from my chef-repo.
  3. ‘berks install’ to install them as Berkshelf-managed cookbooks.
  4. ‘berks upload’ to upload all of the Berkshelf-managed cookbooks to my Chef Server.

While this works, it seems a little ungainly. Depending on whether I am uploading a berkshelf-managed cookbook or one of my own application’s cookbooks, I need to upload it to my Chef Server by different means (‘berks upload’ or ‘knife cookbook upload’).

Is this how others are using Berkshelf with a chef-repo, or have I missed something?

I get the sense many Berkshelf devotees choose to do without a chef-repo entirely, and instead maintain just a cookbook for their application. And while this may work fine for a chef-solo based workflow, I don’t see how you could do without node roles, environments, and data bags if you’re managing clusters of servers with Chef Server as I am.

Any guidance, pointers or thoughts appreciated.

–
Kevin Yank
Chief Technology Officer, Avalanche Technology Group
http://avalanche.com.au/

ph: +61 4 2241 0083

2

29.11.2013 07:27, Kevin Yank ?????:

Hi all,

Although I’m used to maintaining a standard chef-repo for my organisation’s cookbooks and their dependencies, I’ve decided to adopt Berkshelf because it makes it easier to track “unofficial” cookbooks, and forks of “official” cookbooks that have tweaks or fixes that I need.

Although I have it pretty much working for me, I have the niggling sense that I’m not doing it quite right. And all of the documentation and guidance I have found online seems to assume you are using Berkshelf to manage the dependencies of a cookbook you are working on in isolation. I have found almost nothing in the way of guidance on how to use it in the context of a chef-repo.

Here’s what I’ve done:

  1. Delete from my chef-repo all third-party cookbooks that I am simply using, not maintaining for my organisation.
  2. Create a Berksfile in the root of my chef-repo, and list all of the cookbooks I have just deleted from my chef-repo.
  3. ‘berks install’ to install them as Berkshelf-managed cookbooks.
  4. ‘berks upload’ to upload all of the Berkshelf-managed cookbooks to my Chef Server.

While this works, it seems a little ungainly. Depending on whether I am uploading a berkshelf-managed cookbook or one of my own application’s cookbooks, I need to upload it to my Chef Server by different means (‘berks upload’ or ‘knife cookbook upload’).
Only store you application cookbooks in separate repo, doesn't use git
submodules.
Is this how others are using Berkshelf with a chef-repo, or have I missed something?
How I understood, everyone does as he wants )))
I get the sense many Berkshelf devotees choose to do without a chef-repo entirely, and instead maintain just a cookbook for their application. And while this may work fine for a chef-solo based workflow,
It's horrible. ((
I don’t see how you could do without node roles, environments, and data bags if you’re managing clusters of servers with Chef Server as I am.
As far as I know berkshelf developer (Jamie Winsor and riot games)
doesn't using chef-repo ) They are used chef-solo for provision systems.

And berkshelf is developed mainly for chef-solo use case.

See this article.

Hope this help something for you

Any guidance, pointers or thoughts appreciated.

--
Kevin Yank
Chief Technology Officer, Avalanche Technology Group
http://avalanche.com.au/

ph: +61 4 2241 0083

I agree that there is small amount of information about this.

--
Best regards,

CVision Lab System Administrator
Vladmir Skubriev

3

On 29 Nov 2013, at 2:27 pm, Kevin Yank kyank@avalanche.com.au wrote:

Is this how others are using Berkshelf with a chef-repo, or have I missed something?

Answering my own question, I finally stumbled on a GitHub Issue that explains it all. If only the docs did as well!

I’ve posted my own summary for the curious:

--
Kevin Yank
Chief Technology Officer, Avalanche Technology Group
http://avalanche.com.au/

4

29.11.2013 14:47, Kevin Yank пишет:

On 29 Nov 2013, at 2:27 pm, Kevin Yank kyank@avalanche.com.au wrote:

Is this how others are using Berkshelf with a chef-repo, or have I missed something?

Answering my own question, I finally stumbled on a GitHub Issue that explains it all. If only the docs did as well!

I’ve posted my own summary for the curious:
How to use Berkshelf to manage Organization repo like the Librarian-chef does? · Issue #535 · berkshelf/berkshelf · GitHub

--
Kevin Yank
Chief Technology Officer, Avalanche Technology Group
http://avalanche.com.au/

Thank you. I see you last comment and this is good explanation for problem.

I started with a my own "community" chef-server and work with it a year ago.

I have a very long time for decide: "I need it(chef-server) or not"

I could not decide using only chef-server, using chef-solo with berks or
use server with berks.

What stopped you? What do you choose?

--
Best regards,

CVision Lab System Administrator
Vladmir Skubriev

5

I have more than one Chef Server, each with their own set of data bags, roles, nodes, cookbooks, etc.

While I don’t commit my cookbooks to the chef-repo’ VCS, I still maintain a cookbooks/ directory in it (which is on .gitignore), and keep everything that’s part of that Chef infrastructure under the chef-repo root path.

Cookbooks and their dependencies are managed and uploaded using Berkshelf, so I only have the cookbooks I’m working on under chef-repo/cookbooks.

I don’t ever use "knife cookbook upload”. Instead I use “berks upload” from the cookbook’s root.

--Cassiano Leal
http://cassianoleal.com
http://twitter.com/cassianoleal

On Friday, November 29, 2013 at 09:54, Vladimir Skubriev wrote:

29.11.2013 14:47, Kevin Yank пишет:

On 29 Nov 2013, at 2:27 pm, Kevin Yank <kyank@avalanche.com.au (mailto:kyank@avalanche.com.au)> wrote:

Is this how others are using Berkshelf with a chef-repo, or have I missed something?

Answering my own question, I finally stumbled on a GitHub Issue that explains it all. If only the docs did as well!

I’ve posted my own summary for the curious:
How to use Berkshelf to manage Organization repo like the Librarian-chef does? · Issue #535 · berkshelf/berkshelf · GitHub

--
Kevin Yank
Chief Technology Officer, Avalanche Technology Group
http://avalanche.com.au/

Thank you. I see you last comment and this is good explanation for problem.

I started with a my own "community" chef-server and work with it a year ago.

I have a very long time for decide: "I need it(chef-server) or not"

I could not decide using only chef-server, using chef-solo with berks or
use server with berks.

What stopped you? What do you choose?

--
Best regards,

CVision Lab System Administrator
Vladmir Skubriev

6

29.11.2013 16:22, Cassiano Leal пишет:

I have more than one Chef Server, each with their own set of data
bags, roles, nodes, cookbooks, etc.

While I don’t commit my cookbooks to the chef-repo’ VCS, I still
maintain a cookbooks/ directory in it (which is on .gitignore), and
keep everything that’s part of that Chef infrastructure under the
chef-repo root path.

Cookbooks and their dependencies are managed and uploaded using
Berkshelf, so I only have the cookbooks I’m working on under
chef-repo/cookbooks.

I don’t ever use "knife cookbook upload”. Instead I use “berks upload”
from the cookbook’s root.

--
Cassiano Leal
http://cassianoleal.com
http://twitter.com/cassianoleal

On Friday, November 29, 2013 at 09:54, Vladimir Skubriev wrote:

29.11.2013 14:47, Kevin Yank пишет:

On 29 Nov 2013, at 2:27 pm, Kevin Yank <kyank@avalanche.com.au
mailto:kyank@avalanche.com.au> wrote:

Is this how others are using Berkshelf with a chef-repo, or have I
missed something?

Answering my own question, I finally stumbled on a GitHub Issue that
explains it all. If only the docs did as well!

I’ve posted my own summary for the curious:
How to use Berkshelf to manage Organization repo like the Librarian-chef does? · Issue #535 · berkshelf/berkshelf · GitHub

--
Kevin Yank
Chief Technology Officer, Avalanche Technology Group
http://avalanche.com.au/
Thank you. I see you last comment and this is good explanation for
problem.

I started with a my own "community" chef-server and work with it a
year ago.

I have a very long time for decide: "I need it(chef-server) or not"

I could not decide using only chef-server, using chef-solo with berks or
use server with berks.

What stopped you? What do you choose?

--
Best regards,

CVision Lab System Administrator
Vladmir Skubriev

I'am very interested in "How you are resolv situations when cookbook
deleted from berksfile, but but remains on the server ?

How to clean up all cookbooks on the chef-server ?

In other words: "How to maintain order on the server with cookbooks, and
not only with cookbooks, but also roles, databags, etc"

May be there are a common command of a knife to clean up all server from
existing data and then upload a new infrastructure by knife upload and
berks upload ?

--
Best regards,

CVision Lab System Administrator
Vladmir Skubriev

7

That would be best done by a Rake set of knife tasks. However, any clients
trying to run against the server during that time will likely fail. If you
have a static infrastructure, that's probably not an issue, but if you
dynamically bootstrap things like compute instances, it could be an issue.

On Friday, November 29, 2013, Vladimir Skubriev wrote:

29.11.2013 16:22, Cassiano Leal пишет:

I have more than one Chef Server, each with their own set of data bags,
roles, nodes, cookbooks, etc.

While I don’t commit my cookbooks to the chef-repo’ VCS, I still
maintain a cookbooks/ directory in it (which is on .gitignore), and keep
everything that’s part of that Chef infrastructure under the chef-repo root
path.

Cookbooks and their dependencies are managed and uploaded using
Berkshelf, so I only have the cookbooks I’m working on under
chef-repo/cookbooks.

I don’t ever use "knife cookbook upload”. Instead I use “berks upload”
from the cookbook’s root.

--
Cassiano Leal
http://cassianoleal.com
http://twitter.com/cassianoleal

On Friday, November 29, 2013 at 09:54, Vladimir Skubriev wrote:

29.11.2013 14:47, Kevin Yank пишет:

On 29 Nov 2013, at 2:27 pm, Kevin Yank <kyank@avalanche.com.au<javascript:_e({}, 'cvml', 'kyank@avalanche.com.au');>>
wrote:

Is this how others are using Berkshelf with a chef-repo, or have I
missed something?

Answering my own question, I finally stumbled on a GitHub Issue that
explains it all. If only the docs did as well!

I’ve posted my own summary for the curious:
How to use Berkshelf to manage Organization repo like the Librarian-chef does? · Issue #535 · berkshelf/berkshelf · GitHub

--
Kevin Yank
Chief Technology Officer, Avalanche Technology Group
http://avalanche.com.au/

Thank you. I see you last comment and this is good explanation for problem.

I started with a my own "community" chef-server and work with it a year
ago.

I have a very long time for decide: "I need it(chef-server) or not"

I could not decide using only chef-server, using chef-solo with berks or
use server with berks.

What stopped you? What do you choose?

--
Best regards,

CVision Lab System Administrator
Vladmir Skubriev

I'am very interested in "How you are resolv situations when cookbook
deleted from berksfile, but but remains on the server ?

How to clean up all cookbooks on the chef-server ?

In other words: "How to maintain order on the server with cookbooks, and
not only with cookbooks, but also roles, databags, etc"

May be there are a common command of a knife to clean up all server from
existing data and then upload a new infrastructure by knife upload and
berks upload ?

--
Best regards,

CVision Lab System Administrator
Vladmir Skubriev

--

~~ StormeRider ~~

"Every world needs its heroes [...] They inspire us to be better than we
are. And they protect from the darkness that's just around the corner."

(from Smallville Season 6x1: "Zod")

On why I hate the phrase "that's so lame"... http://bit.ly/Ps3uSS

8

The Chef solo / Vagrant / Berkshelf / Application Cookbooks workflow that
works well for me described here:
http://lists.opscode.com/sympa/arc/chef/2013-10/msg00307.html

The problem with a top-level Berksfile in the chef-repo is that you are
assuming a single consistent set of cookbooks across all the nodes you
manage via that chef-repo.

I tend to stick with the "one application cookbook per node" paradigm and
resolve each application cookbook's dependencies into a dedicated
sub-directory so that their transitive dependencies don't clash.

Finally, I'm calling the "infrastructure repo" because of using one
chef-repo like structure per "infrastructure", where infrastrcture means
just a set of correlated nodes.

See the example here:

Cheers,
Torben
On Nov 29, 2013 1:23 PM, "Cassiano Leal" cassianoleal@gmail.com wrote:

I have more than one Chef Server, each with their own set of data bags,
roles, nodes, cookbooks, etc.

While I don’t commit my cookbooks to the chef-repo’ VCS, I still maintain
a cookbooks/ directory in it (which is on .gitignore), and keep everything
that’s part of that Chef infrastructure under the chef-repo root path.

Cookbooks and their dependencies are managed and uploaded using Berkshelf,
so I only have the cookbooks I’m working on under chef-repo/cookbooks.

I don’t ever use "knife cookbook upload”. Instead I use “berks upload”
from the cookbook’s root.

--
Cassiano Leal
http://cassianoleal.com
http://twitter.com/cassianoleal

On Friday, November 29, 2013 at 09:54, Vladimir Skubriev wrote:

29.11.2013 14:47, Kevin Yank пишет:

On 29 Nov 2013, at 2:27 pm, Kevin Yank kyank@avalanche.com.au wrote:

Is this how others are using Berkshelf with a chef-repo, or have I missed
something?

Answering my own question, I finally stumbled on a GitHub Issue that
explains it all. If only the docs did as well!

I’ve posted my own summary for the curious:
How to use Berkshelf to manage Organization repo like the Librarian-chef does? · Issue #535 · berkshelf/berkshelf · GitHub

--
Kevin Yank
Chief Technology Officer, Avalanche Technology Group
http://avalanche.com.au/

Thank you. I see you last comment and this is good explanation for problem.

I started with a my own "community" chef-server and work with it a year
ago.

I have a very long time for decide: "I need it(chef-server) or not"

I could not decide using only chef-server, using chef-solo with berks or
use server with berks.

What stopped you? What do you choose?

--
Best regards,

CVision Lab System Administrator
Vladmir Skubriev

9

I wrote a blog post awhile back1 about my own personal chef-repo
conversion to use Berkshelf. The information is not all still 100%
relevant/accurate given the entropy of tool development and my
personal use patterns, but it's pretty close.

I still have a ./cookbooks directory, because I don't see a reason to
publish and manage 20+ additional cookbook repositories, especially
private ones. The main difference between the post and now is have a
helper method like this in the Berksfile:

def internal_cookbook(name, version = '>= 0.0.0', options = {})
cookbook name, version, { :path => "./cookbooks/#{name}" }.merge(options)
end

Then for all the "internal" cookbooks, in the ./cookbooks directory, I
have entries like this:

internal_cookbook "housepub-workstation"
internal_cookbook "users"

Internally at Opscode, we're currently doing something like this, too.
We're working to move to a single-repo-per-cookbook as a longer term
goal.

Hope this helps,
Joshua

  • apparently a year ago in fact :).

On Fri, Nov 29, 2013 at 3:47 AM, Kevin Yank kyank@avalanche.com.au wrote:

On 29 Nov 2013, at 2:27 pm, Kevin Yank kyank@avalanche.com.au wrote:

Is this how others are using Berkshelf with a chef-repo, or have I missed something?

Answering my own question, I finally stumbled on a GitHub Issue that explains it all. If only the docs did as well!

I’ve posted my own summary for the curious:
How to use Berkshelf to manage Organization repo like the Librarian-chef does? · Issue #535 · berkshelf/berkshelf · GitHub

--
Kevin Yank
Chief Technology Officer, Avalanche Technology Group
http://avalanche.com.au/

--
Joshua Timberman, Opscode, Inc.

10

02.12.2013 23:09, Joshua Timberman пишет:

I wrote a blog post awhile back1 about my own personal chef-repo
conversion to use Berkshelf. The information is not all still 100%
relevant/accurate given the entropy of tool development and my
personal use patterns, but it's pretty close.

I still have a ./cookbooks directory, because I don't see a reason to
publish and manage 20+ additional cookbook repositories, especially
private ones. The main difference between the post and now is have a
helper method like this in the Berksfile:

def internal_cookbook(name, version = '>= 0.0.0', options = {})
cookbook name, version, { :path => "./cookbooks/#{name}" }.merge(options)
end

Then for all the "internal" cookbooks, in the ./cookbooks directory, I
have entries like this:

internal_cookbook "housepub-workstation"
internal_cookbook "users"

Internally at Opscode, we're currently doing something like this, too.
We're working to move to a single-repo-per-cookbook as a longer term
goal.

Hope this helps,
Joshua

  • apparently a year ago in fact :).
    Okey.

I know that now you switch to using chef-vault. And recommend it to me
in last mail.

As are you did it before know about chef-vault ?

For example you need to use community mysql cookbook in
"housepub-cookbook" application cookbook.

If you use mysql cookbook you need to define attribute with a password
of mysql root, before use of cookbook in application (your's internal
cookbook) cookbook.

node['mysql']['server_root_password']

Do you define this node attribute in application (internal) cookbook
"housepub-workstation"

in plaintext | or write a code to extract passwords from databags with
you preference ?

--
Best regards,

CVision Lab System Administrator
Vladmir Skubriev