Bershelf / git configuration issue?

Chef Infra (archive)
1

I posted this in stackoverflow http://stackoverflow.com/questions/28996765/how-do-i-setup-cookbooks-in-git-for-berkshelf, but I thought I’d post it here too.
I’m trying to use Berkshelf, and am trying the following simple test.
We have an internal git server using Gitlab, say, http://git-server.domain.com. I created a “project” group, and pushed mycookbook.git repo so, according to git, the cookbook is accessible via http://git-server.domain.com/project/mycookbook.git.
Now I have an environment cookbook called myapp_dev, and its Berksfile looks like this
source "https://supermarket.chef.io"
source “http://git-server.domain.com/project”

metadata
and its metadata.rb looks like this
name ‘myapp_dev’
(snip)
version ‘0.1.0’

depends ‘mycookbook’, '~> 1.0.0’
When I do a berks install, I get
Required artifacts do not exist at the desired version
Missing artifacts: mycookbook
Unable to find a solution for demands: mycookbook(>= 0.0.0), myapp_dev (0.1.0)
What do I have configured wrong?
Chris

2

On Thu, Mar 12, 2015 at 10:35 AM, Fouts, Chris Chris.Fouts@sensus.com wrote:

I posted this in stackoverflow
chef infra - How do I setup cookbooks in git for berkshelf? - Stack Overflow,
but I thought I’d post it here too.

It'd be great if you'd stick to one or the other so that the answer
shows up in a single place.

I’m trying to use Berkshelf, and am trying the following simple test.

We have an internal git server using Gitlab, say,
http://git-server.domain.com. I created a "project" group, and pushed
mycookbook.git repo so, according to git, the cookbook is accessible via
http://git-server.domain.com/project/mycookbook.git.

Now I have an environment cookbook called myapp_dev, and its Berksfile looks
like this

source "https://supermarket.chef.io"
source "http://git-server.domain.com/project"

I don't think this is going to do what you want because your Git
server doesn't have a /universe endpoint for Berkshelf to use.

You can specify the source of cookbooks individually on the 'depends'
lines (the syntax is similar to that in a Gemfile and it's documented
on berkshelf.com) or you can run your own supermarket internally.

  • Julian

--
[ Julian C. Dunn jdunn@aquezada.com * Sorry, I'm ]
[ WWW: Julian Dunn's Blog - Commentary on media, technology, and everything in between. * only Web 1.0 ]
[ gopher://sdf.org/1/users/keymaker/ * compliant! ]
[ PGP: 91B3 7A9D 683C 7C16 715F 442C 6065 D533 FDC2 05B9 ]

3

Thanks Julian.

I resisted cross-posting here but I got no traction in stackoverflow. I will post your answer there.

You said...
"
I don't think this is going to do what you want because your Git server doesn't have a /universe endpoint for Berkshelf to use.

You can specify the source of cookbooks individually on the 'depends'
lines (the syntax is similar to that in a Gemfile and it's documented on berkshelf.com) or you can run your own supermarket internally.
"

This makes sense. I will explore specifying the git repo per cookbook in my Berksfile as an initial attempt, and then look into running a supermarket internally.

Thanks,
Chris

-----Original Message-----
From: Julian C. Dunn [mailto:jdunn@aquezada.com]
Sent: Thursday, March 12, 2015 10:43 AM
To: chef@lists.opscode.com
Subject: [chef] Re: Bershelf / git configuration issue?

On Thu, Mar 12, 2015 at 10:35 AM, Fouts, Chris Chris.Fouts@sensus.com wrote:

I posted this in stackoverflow
chef infra - How do I setup cookbooks in git for berkshelf? - Stack Overflow
n-git-for-berkshelf,
but I thought I’d post it here too.

It'd be great if you'd stick to one or the other so that the answer shows up in a single place.

I’m trying to use Berkshelf, and am trying the following simple test.

We have an internal git server using Gitlab, say,
http://git-server.domain.com. I created a "project" group, and pushed
mycookbook.git repo so, according to git, the cookbook is accessible
via http://git-server.domain.com/project/mycookbook.git.

Now I have an environment cookbook called myapp_dev, and its Berksfile
looks like this

source "https://supermarket.chef.io"
source "http://git-server.domain.com/project"

I don't think this is going to do what you want because your Git server doesn't have a /universe endpoint for Berkshelf to use.

You can specify the source of cookbooks individually on the 'depends'
lines (the syntax is similar to that in a Gemfile and it's documented on berkshelf.com) or you can run your own supermarket internally.

  • Julian

--
[ Julian C. Dunn jdunn@aquezada.com * Sorry, I'm ]
[ WWW: Julian Dunn's Blog - Commentary on media, technology, and everything in between. * only Web 1.0 ]
[ gopher://sdf.org/1/users/keymaker/ * compliant! ]
[ PGP: 91B3 7A9D 683C 7C16 715F 442C 6065 D533 FDC2 05B9 ]

4

You can't set your git server as a source in Berksfile as Julian said but
you can target cookbooks in your git repo:

cookbook 'mycookbook', git: 'https://git-server.domain.com/mycookbook.git',
tag: 'v1.0.0'

or maybe with a commit hash

cookbook 'mycookbook', git: 'https://git-server.domain.com/mycookbook.git',
ref: 'eef7e65806e7ff3bdbe148e27c447ef4a8bc3881'

There's a couple other ways you can find at http://berkshelf.com

berkshelf-api might be an option (GitHub - berkshelf/berkshelf-api: Berkshelf dependency API server)
if you're running github enterprise. I thought it was going away but I see
recent commits. Some time ago I wanted to point it at Gitlab but it wasn't
supported. README says it only support github enterprise (and then
proceeds to discourage actually doing it). You can run your own
supermarket but I'm pretty sure you have to also run chef server or you
can't authenticate against it. Though the README (
GitHub - chef/supermarket: Chef's community platform) indicates there may be a hack coming
down the pipe soon:

"NOTE: Authentication currently requires a live chef server running oc-id.
We are working on a solution which would allow a developer to run the
authentication locally, stay tuned"

I personally hope so because I think it's pretty lame they make you run a
chef server (If I remember right berkshelf was originally created for
workflows and orgs that were NOT running a chef server which makes it even
more frustrating).

Sorry I hijacked your thread to rant about this particular issue about an
otherwise awesome platform (Chef <3). It's just always really bothered me
how that shook out and your question brought it all back to the surface.
Hopefully you got some value from technical part of my response :slight_smile:

MG

On Thu, Mar 12, 2015 at 7:56 AM, Fouts, Chris Chris.Fouts@sensus.com
wrote:

Thanks Julian.

I resisted cross-posting here but I got no traction in stackoverflow. I
will post your answer there.

You said...
"
I don't think this is going to do what you want because your Git server
doesn't have a /universe endpoint for Berkshelf to use.

You can specify the source of cookbooks individually on the 'depends'
lines (the syntax is similar to that in a Gemfile and it's documented on
berkshelf.com) or you can run your own supermarket internally.
"

This makes sense. I will explore specifying the git repo per cookbook in
my Berksfile as an initial attempt, and then look into running a
supermarket internally.

Thanks,
Chris

-----Original Message-----
From: Julian C. Dunn [mailto:jdunn@aquezada.com]
Sent: Thursday, March 12, 2015 10:43 AM
To: chef@lists.opscode.com
Subject: [chef] Re: Bershelf / git configuration issue?

On Thu, Mar 12, 2015 at 10:35 AM, Fouts, Chris Chris.Fouts@sensus.com
wrote:

I posted this in stackoverflow
chef infra - How do I setup cookbooks in git for berkshelf? - Stack Overflow
n-git-for-berkshelf,
but I thought I’d post it here too.

It'd be great if you'd stick to one or the other so that the answer shows
up in a single place.

I’m trying to use Berkshelf, and am trying the following simple test.

We have an internal git server using Gitlab, say,
http://git-server.domain.com. I created a "project" group, and pushed
mycookbook.git repo so, according to git, the cookbook is accessible
via http://git-server.domain.com/project/mycookbook.git.

Now I have an environment cookbook called myapp_dev, and its Berksfile
looks like this

source "https://supermarket.chef.io"
source "http://git-server.domain.com/project"

I don't think this is going to do what you want because your Git server
doesn't have a /universe endpoint for Berkshelf to use.

You can specify the source of cookbooks individually on the 'depends'
lines (the syntax is similar to that in a Gemfile and it's documented on
berkshelf.com) or you can run your own supermarket internally.

  • Julian

--
[ Julian C. Dunn jdunn@aquezada.com * Sorry, I'm ]
[ WWW: Julian Dunn's Blog - Commentary on media, technology, and everything in between. * only Web 1.0 ]
[ gopher://sdf.org/1/users/keymaker/ * compliant! ]
[ PGP: 91B3 7A9D 683C 7C16 715F 442C 6065 D533 FDC2 05B9 ]

5

Yep that’s what I ended up doing, i.e., adding the git repo on the cookbook line.

I appreciate all the responses, and they gave me more avenues to explore.

Chris

From: Michael Glenney [mailto:mike.glenney@gmail.com]
Sent: Thursday, March 12, 2015 12:39 PM
To: chef@lists.opscode.com
Subject: [chef] Re: RE: Re: Bershelf / git configuration issue?

You can't set your git server as a source in Berksfile as Julian said but you can target cookbooks in your git repo:

cookbook 'mycookbook', git: 'https://git-server.domain.com/mycookbook.git', tag: 'v1.0.0'

or maybe with a commit hash

cookbook 'mycookbook', git: 'https://git-server.domain.com/mycookbook.git', ref: 'eef7e65806e7ff3bdbe148e27c447ef4a8bc3881'

There's a couple other ways you can find at http://berkshelf.com

berkshelf-api might be an option (GitHub - berkshelf/berkshelf-api: Berkshelf dependency API server) if you're running github enterprise. I thought it was going away but I see recent commits. Some time ago I wanted to point it at Gitlab but it wasn't supported. README says it only support github enterprise (and then proceeds to discourage actually doing it). You can run your own supermarket but I'm pretty sure you have to also run chef server or you can't authenticate against it. Though the README (GitHub - chef/supermarket: Chef's community platform) indicates there may be a hack coming down the pipe soon:

"NOTE: Authentication currently requires a live chef server running oc-id. We are working on a solution which would allow a developer to run the authentication locally, stay tuned"

I personally hope so because I think it's pretty lame they make you run a chef server (If I remember right berkshelf was originally created for workflows and orgs that were NOT running a chef server which makes it even more frustrating).

Sorry I hijacked your thread to rant about this particular issue about an otherwise awesome platform (Chef <3). It's just always really bothered me how that shook out and your question brought it all back to the surface. Hopefully you got some value from technical part of my response :slight_smile:

MG

On Thu, Mar 12, 2015 at 7:56 AM, Fouts, Chris <Chris.Fouts@sensus.commailto:Chris.Fouts@sensus.com> wrote:
Thanks Julian.

I resisted cross-posting here but I got no traction in stackoverflow. I will post your answer there.

You said...
"
I don't think this is going to do what you want because your Git server doesn't have a /universe endpoint for Berkshelf to use.

You can specify the source of cookbooks individually on the 'depends'
lines (the syntax is similar to that in a Gemfile and it's documented on berkshelf.comhttp://berkshelf.com) or you can run your own supermarket internally.
"

This makes sense. I will explore specifying the git repo per cookbook in my Berksfile as an initial attempt, and then look into running a supermarket internally.

Thanks,
Chris

-----Original Message-----
From: Julian C. Dunn [mailto:jdunn@aquezada.commailto:jdunn@aquezada.com]
Sent: Thursday, March 12, 2015 10:43 AM
To: chef@lists.opscode.commailto:chef@lists.opscode.com
Subject: [chef] Re: Bershelf / git configuration issue?

On Thu, Mar 12, 2015 at 10:35 AM, Fouts, Chris <Chris.Fouts@sensus.commailto:Chris.Fouts@sensus.com> wrote:

I posted this in stackoverflow
chef infra - How do I setup cookbooks in git for berkshelf? - Stack Overflow
n-git-for-berkshelf,
but I thought I’d post it here too.

It'd be great if you'd stick to one or the other so that the answer shows up in a single place.

I’m trying to use Berkshelf, and am trying the following simple test.

We have an internal git server using Gitlab, say,
http://git-server.domain.com. I created a "project" group, and pushed
mycookbook.git repo so, according to git, the cookbook is accessible
via http://git-server.domain.com/project/mycookbook.git.

Now I have an environment cookbook called myapp_dev, and its Berksfile
looks like this

source "https://supermarket.chef.io"
source "http://git-server.domain.com/project"

I don't think this is going to do what you want because your Git server doesn't have a /universe endpoint for Berkshelf to use.

You can specify the source of cookbooks individually on the 'depends'
lines (the syntax is similar to that in a Gemfile and it's documented on berkshelf.comhttp://berkshelf.com) or you can run your own supermarket internally.

  • Julian

--
[ Julian C. Dunn <jdunn@aquezada.commailto:jdunn@aquezada.com> * Sorry, I'm ]
[ WWW: Julian Dunn's Blog - Commentary on media, technology, and everything in between. * only Web 1.0 ]
[ gopher://sdf.org/1/users/keymaker/http://sdf.org/1/users/keymaker/ * compliant! ]
[ PGP: 91B3 7A9D 683C 7C16 715F 442C 6065 D533 FDC2 05B9 ]

6

On Thu, Mar 12, 2015 at 12:39 PM, Michael Glenney
mike.glenney@gmail.com wrote:

Though the README (GitHub - chef/supermarket: Chef's community platform) indicates there may
be a hack coming down the pipe soon:

"NOTE: Authentication currently requires a live chef server running oc-id.
We are working on a solution which would allow a developer to run the
authentication locally, stay tuned"

I personally hope so because I think it's pretty lame they make you run a
chef server (If I remember right berkshelf was originally created for
workflows and orgs that were NOT running a chef server which makes it even
more frustrating).

We're contemplating a solution like making oc-id be able to use
different auth backends (so kind of like an Atlassian Crowd) -- would
this be useful?

Another thought I had was what if we just made plugins for artifact
servers like Nexus or Artifactory to be able to "speak" the
supermarket protocol, and that way you'd get all the rights management
out-of-the-box. Is that of interest to folks?

  • Julian
7

+1 for making nexus, artifactory, bintray etc speak the supermarket protocol.

They seem to be quite common and widespread artifact repositories (at
least among the customers I worked with).

Would love to see them speaking chef.

Cheers, Torben

On Fri, Mar 13, 2015 at 9:22 PM, Julian C. Dunn jdunn@aquezada.com wrote:

On Thu, Mar 12, 2015 at 12:39 PM, Michael Glenney
mike.glenney@gmail.com wrote:

Though the README (GitHub - chef/supermarket: Chef's community platform) indicates there may
be a hack coming down the pipe soon:

"NOTE: Authentication currently requires a live chef server running oc-id.
We are working on a solution which would allow a developer to run the
authentication locally, stay tuned"

I personally hope so because I think it's pretty lame they make you run a
chef server (If I remember right berkshelf was originally created for
workflows and orgs that were NOT running a chef server which makes it even
more frustrating).

We're contemplating a solution like making oc-id be able to use
different auth backends (so kind of like an Atlassian Crowd) -- would
this be useful?

Another thought I had was what if we just made plugins for artifact
servers like Nexus or Artifactory to be able to "speak" the
supermarket protocol, and that way you'd get all the rights management
out-of-the-box. Is that of interest to folks?

  • Julian
8

+1 - that would be very, very convenient.

Cheers,

am

On 13 Mar 2015, at 23:40, Torben Knerr mail@tknerr.de wrote:

+1 for making nexus, artifactory, bintray etc speak the supermarket protocol.

They seem to be quite common and widespread artifact repositories (at
least among the customers I worked with).

Would love to see them speaking chef.

Cheers, Torben

On Fri, Mar 13, 2015 at 9:22 PM, Julian C. Dunn jdunn@aquezada.com wrote:

On Thu, Mar 12, 2015 at 12:39 PM, Michael Glenney
mike.glenney@gmail.com wrote:

Though the README (GitHub - chef/supermarket: Chef's community platform) indicates there may
be a hack coming down the pipe soon:

"NOTE: Authentication currently requires a live chef server running oc-id.
We are working on a solution which would allow a developer to run the
authentication locally, stay tuned"

I personally hope so because I think it's pretty lame they make you run a
chef server (If I remember right berkshelf was originally created for
workflows and orgs that were NOT running a chef server which makes it even
more frustrating).

We're contemplating a solution like making oc-id be able to use
different auth backends (so kind of like an Atlassian Crowd) -- would
this be useful?

Another thought I had was what if we just made plugins for artifact
servers like Nexus or Artifactory to be able to "speak" the
supermarket protocol, and that way you'd get all the rights management
out-of-the-box. Is that of interest to folks?

  • Julian
9

-----Original Message-----
From: Julian C. Dunn [mailto:jdunn@aquezada.com]
Sent: Friday, March 13, 2015 4:23 PM
To: chef@lists.opscode.com
Subject: [chef] Re: Re: RE: Re: Bershelf / git configuration issue?

On Thu, Mar 12, 2015 at 12:39 PM, Michael Glenney
mike.glenney@gmail.com wrote:

Though the README (GitHub - chef/supermarket: Chef's community platform) indicates
there may be a hack coming down the pipe soon:

"NOTE: Authentication currently requires a live chef server running oc-id.
We are working on a solution which would allow a developer to run the
authentication locally, stay tuned"

I personally hope so because I think it's pretty lame they make you
run a chef server (If I remember right berkshelf was originally
created for workflows and orgs that were NOT running a chef server
which makes it even more frustrating).

We're contemplating a solution like making oc-id be able to use different auth
backends (so kind of like an Atlassian Crowd) -- would this be useful?

Another thought I had was what if we just made plugins for artifact servers like
Nexus or Artifactory to be able to "speak" the supermarket protocol, and that
way you'd get all the rights management out-of-the-box. Is that of interest to
folks?

I personally gave up on the chef server requirement for small environments some time ago, and use Berkshelf with chef-solo. It doesn't allow environment auditing, but most tools allow manual listing of cluster members and other configurations. It's also invaluable for configuring, or bootstrapping, the initial chef server to support the other environments.

Nico Kadel-Garcia
Lead DevOps Engineer
nkadel@skyhookwireless.com