Anybody uses git subtree in a chef repo to development cookbooks separately from the production chef-repo?

Chef Infra (archive)
1

Now I decide to choiche best method for creating and maintain a chef-repo.

When I started working with a chef I didn’t using a chef-repo and starts
development of my cookbooks in same directory in home.

For example: ~/chef-cookbooks

And there are my own cookbooks and community cookbooks.

Which is loading to the server via knife cookbook upload foldername.

My own cookbooks development with mercurial cvs.

Site cookbooks git cloned or unpacked from tar.gz and has not modified.

Now I have come to the conclusion that I needed a chef-repo to store all
my data in one place.

How I can store community cookbooks and my own cookbooks in chef-repo
(In which folders they should be) ?

I can convert my mercurial repositories with a cookbooks to git repos
and include them in the chef repo.

What do you think about this ?

Can I revice more flexibiliy and usabilty instead of many problems in
this use case ?

And how to include community git clonned cookbooks in my future
chef-repo(I want to use a berkshelf) ? Using a git sub trees or
something else?

In general, I want to get a fully managed and flex system.

Anybody uses git subtree in a chef repo to development cookbooks
separately from the production chef-repo ?

Excusme for bad english.

–
Best regards,

CVision Lab System Administrator
Vladmir Skubriev

2

Hi Vladimir,

we use Git to manage our repo cookbooks.
We have 2 types of cookbooks, one that we call "base cookbooks", that we
use for common installations, like Java, Tomcat, HAProxy, etc..(we do not
change the community cookbooks, we use then as is and we extend if we need
to change something). For these type of cookbooks, we use a repo in Git
with the same name of cookbook, and the upload is directly. For the other
type, that we call "project cookbooks", we use a directory inside of
project repo, in our case, a directory named /project/build/cookbook (the
build directory has others stuffs related to our Jenkins build system). To
upload the cookbook we have a script in Ruby (you can use any script that
you prefer) that copy the build cookbook directory to a temporary directory
with the cookbook name and then, we execute the knife upload command.
It is a simple (not the best solution) but works.

I hope to had helped.
Sorry also, for my bad English.

Regards

Eduardo.

2013/10/29 Vladimir Skubriev skubriev@cvisionlab.com

Now I decide to choiche best method for creating and maintain a chef-repo.

When I started working with a chef I didn't using a chef-repo and starts
development of my cookbooks in same directory in home.

For example: ~/chef-cookbooks

And there are my own cookbooks and community cookbooks.

Which is loading to the server via knife cookbook upload foldername.

My own cookbooks development with mercurial cvs.

Site cookbooks git cloned or unpacked from tar.gz and has not modified.

Now I have come to the conclusion that I needed a chef-repo to store all
my data in one place.

How I can store community cookbooks and my own cookbooks in chef-repo (In
which folders they should be) ?

I can convert my mercurial repositories with a cookbooks to git repos and
include them in the chef repo.

What do you think about this ?

Can I revice more flexibiliy and usabilty instead of many problems in this
use case ?

And how to include community git clonned cookbooks in my future
chef-repo(I want to use a berkshelf) ? Using a git sub trees or something
else?

In general, I want to get a fully managed and flex system.

Anybody uses git subtree in a chef repo to development cookbooks
separately from the production chef-repo ?

Excusme for bad english.

--
Best regards,

CVision Lab System Administrator
Vladmir Skubriev

--

Atenciosamente,

Eduardo

3

29.10.2013 14:57, Eduardo Dias ?????:

Hi Vladimir,

we use Git to manage our repo cookbooks.
We have 2 types of cookbooks, one that we call "base cookbooks", that
we use for common installations, like Java, Tomcat, HAProxy, etc..(we
do not change the community cookbooks, we use then as is and we extend
if we need to change something). For these type of cookbooks, we use a
repo in Git with the same name of cookbook, and the upload is
directly. For the other type, that we call "project cookbooks", we use
a directory inside of project repo, in our case, a directory named
/project/build/cookbook (the build directory has others stuffs related
to our Jenkins build system). To upload the cookbook we have a script
in Ruby (you can use any script that you prefer) that copy the build
cookbook directory to a temporary directory with the cookbook name and
then, we execute the knife upload command.
It is a simple (not the best solution) but works.

Now I also use this strategy, but I manually controls my knife cookbook
upload command. Doing cookbook updates in server in manual mode.

But also heard about chef-repo and therefore I have a question.

Because I would to store all my work in one place. But can't understand
how to do it. )

The general idea of chef-repo. How I can understand is "Store all in
chef-repo, and upload all to chef-server at all and in one knife command"

I hope to had helped.
That's help os course. In "How this do others".
Sorry also, for my bad English.

NO problem ) Of course )

Regards

Eduardo.

--

Atenciosamente,

Eduardo

--
Best regards,

CVision Lab System Administrator
Vladmir Skubriev

4

Well,

I never mind to upload a bunch of cookbooks in one time, this it is not
usual to us, since we have a Jenkins pipeline that test and then upload
each cookbook, to guarantee that we are not uploading cookbook with errors.
But is a interesting approach.
If I see some related to this I send to you.

Regards,

Eduardo

2013/10/29 Vladimir Skubriev skubriev@cvisionlab.com

29.10.2013 14:57, Eduardo Dias пишет:

Hi Vladimir,

we use Git to manage our repo cookbooks.
We have 2 types of cookbooks, one that we call "base cookbooks", that we
use for common installations, like Java, Tomcat, HAProxy, etc..(we do not
change the community cookbooks, we use then as is and we extend if we need
to change something). For these type of cookbooks, we use a repo in Git
with the same name of cookbook, and the upload is directly. For the other
type, that we call "project cookbooks", we use a directory inside of
project repo, in our case, a directory named /project/build/cookbook (the
build directory has others stuffs related to our Jenkins build system). To
upload the cookbook we have a script in Ruby (you can use any script that
you prefer) that copy the build cookbook directory to a temporary directory
with the cookbook name and then, we execute the knife upload command.
It is a simple (not the best solution) but works.

Now I also use this strategy, but I manually controls my knife cookbook
upload command. Doing cookbook updates in server in manual mode.

But also heard about chef-repo and therefore I have a question.

Because I would to store all my work in one place. But can't understand
how to do it. )

The general idea of chef-repo. How I can understand is "Store all in
chef-repo, and upload all to chef-server at all and in one knife command"

I hope to had helped.

That's help os course. In "How this do others".

Sorry also, for my bad English.

NO problem ) Of course )

Regards

Eduardo.

--

Atenciosamente,

Eduardo

--
Best regards,

CVision Lab System Administrator
Vladmir Skubriev

--

Atenciosamente,

Eduardo

5

29.10.2013 15:23, Eduardo Dias пишет:

Well,

I never mind to upload a bunch of cookbooks in one time, this it is
not usual to us, since we have a Jenkins pipeline that test and then
upload each cookbook, to guarantee that we are not uploading cookbook
with errors.
But is a interesting approach.

I doesn't mean that I download all site cookbooks, for example from
GitHub - chef-boneyard/cookbooks: DEPRECATED: This repository has been split up into separate repositories by cookbook under the "opscode-cookbooks" organization. in a one subtree.

I want to create a separate subtree for each site (opscode community)
cookbook in my chef-repo.

Or may be clone alien(foreign) cookbooks in a separate directory for
example to ~/chef-cookbooks/foreign_cookbook_name.

Do a development in this cloned repo, and if I want to upload this
cookbook to my chef server - then create a subtree in chef-repo, which
is with origin from ~chef-cookbooks/foreign_cookbook_name.

Because as I understand chef-repo is created not for development
cookbooks. chef-repo is for store chef-server configuration in one place
under control version system.

Right way to doing development of cookbooks is a standalone cloned
separate repository (one for one cookbook).

This is quite and more easy to contribute authors of cookbooks which is
used by you. (I hope that that you understand me)

Or may be I something wrong ?

If I see some related to this I send to you.

Regards,

Eduardo
Thank you. I will be wait )))

--
Best regards,

CVision Lab System Administrator
Vladmir Skubriev

6

hummmm.
I think that I understood you.
I never used chef-repo before, I have read quickly some minutes before, I
nee to read more to try to help you.
Basically, if I am not wrong, it makes a local repo with all cookbook that
you want to manage, with the same structure of chef server.
Sorry but I need to learn more.

Eduardo

2013/10/29 Vladimir Skubriev skubriev@cvisionlab.com

29.10.2013 15:23, Eduardo Dias пишет:

Well,

I never mind to upload a bunch of cookbooks in one time, this it is not
usual to us, since we have a Jenkins pipeline that test and then upload
each cookbook, to guarantee that we are not uploading cookbook with errors.
But is a interesting approach.

I doesn't mean that I download all site cookbooks, for example from
https://github.com/opscode/**cookbooks.githttps://github.com/opscode/cookbooks.gitin a one subtree.

I want to create a separate subtree for each site (opscode community)
cookbook in my chef-repo.

Or may be clone alien(foreign) cookbooks in a separate directory for
example to ~/chef-cookbooks/foreign_**cookbook_name.

Do a development in this cloned repo, and if I want to upload this
cookbook to my chef server - then create a subtree in chef-repo, which is
with origin from ~chef-cookbooks/foreign_**cookbook_name.

Because as I understand chef-repo is created not for development
cookbooks. chef-repo is for store chef-server configuration in one place
under control version system.

Right way to doing development of cookbooks is a standalone cloned
separate repository (one for one cookbook).

This is quite and more easy to contribute authors of cookbooks which is
used by you. (I hope that that you understand me)

Or may be I something wrong ?

If I see some related to this I send to you.

Regards,

Eduardo

Thank you. I will be wait )))

--
Best regards,

CVision Lab System Administrator
Vladmir Skubriev

--

Atenciosamente,

Eduardo

7

29.10.2013 15:52, Eduardo Dias пишет:

hummmm.
I think that I understood you.
I never used chef-repo before, I have read quickly some minutes
before, I nee to read more to try to help you.
Basically, if I am not wrong, it makes a local repo with all cookbook
that you want to manage, with the same structure of chef server.
Sorry but I need to learn more.

No sorry, because I also learned about it just now))

But I am using chef-server over a halt a year ))

I am no hurry. I will be wait, communicate and study for this task too.

Wait for a guru )

--
Best regards,

CVision Lab System Administrator
Vladmir Skubriev

8

Great.

I have a nice day.

Eduardo

2013/10/29 Vladimir Skubriev skubriev@cvisionlab.com

29.10.2013 15:52, Eduardo Dias пишет:

hummmm.

I think that I understood you.
I never used chef-repo before, I have read quickly some minutes before, I
nee to read more to try to help you.
Basically, if I am not wrong, it makes a local repo with all cookbook
that you want to manage, with the same structure of chef server.
Sorry but I need to learn more.

No sorry, because I also learned about it just now))

But I am using chef-server over a halt a year ))

I am no hurry. I will be wait, communicate and study for this task too.

Wait for a guru )

--
Best regards,

CVision Lab System Administrator
Vladmir Skubriev

--

Atenciosamente,

Eduardo

9

Storing all of your cookbooks in the chef-repo has become somewhat of an
anti-pattern. Instead, the recommendation is that you have a "chef-data"
folder, which is source controlled, where you store all your roles,
environments and data_bags. Next, you use a tool like Berkshelf to manage
dependencies for your project cookbooks. This encourages the use of one
git-repo per cookbook.

So the file structure would look something like this:

chef-data
environments
roles
data_bags
cookbooks
projectcookbook1
projectcookbook2

Instead of downloading and owning the community cookbooks, you'd use
Berkshelf (which some describe as bundler/maven for Chef cookbooks). You
can find out more about Berkshelf at berkshelf.com or by watching this
ChefConf video where Jamie Winsor, the creator of Berkshelf, describes why
Berkshelf was made:
http://www.youtube.com/watch?v=hYt0E84kYUI&feature=c4-overview-vl&list=PLrmstJpucjzXNMLcI5X-EjirpDd-SITd3

Tom Duffield — Automation Consulting Engineer

651.769.7497 – tom@opscode.com – *my:
*Linkedinhttp://www.linkedin.com/in/thomasduffield/
Twitter https://twitter.com/tomduffield

OPSCODE

CODE CAN
opscode.com http://www.opscode.com Blog http://www.opscode.com/blog/
Facebook http://www.facebook.com/opscode
Twitterhttp://www.twitter.com/opscode
YouTube http://www.youtube.com/opscode

On Tue, Oct 29, 2013 at 7:02 AM, Eduardo Dias eduardodiasbr@gmail.comwrote:

Great.

I have a nice day.

Eduardo

2013/10/29 Vladimir Skubriev skubriev@cvisionlab.com

29.10.2013 15:52, Eduardo Dias пишет:

hummmm.

I think that I understood you.
I never used chef-repo before, I have read quickly some minutes before,
I nee to read more to try to help you.
Basically, if I am not wrong, it makes a local repo with all cookbook
that you want to manage, with the same structure of chef server.
Sorry but I need to learn more.

No sorry, because I also learned about it just now))

But I am using chef-server over a halt a year ))

I am no hurry. I will be wait, communicate and study for this task too.

Wait for a guru )

--
Best regards,

CVision Lab System Administrator
Vladmir Skubriev

--

Atenciosamente,

Eduardo

10

29.10.2013 17:24, Tom Duffield пишет:

Storing all of your cookbooks in the chef-repo has become somewhat of
an anti-pattern. Instead, the recommendation is that you have a
"chef-data" folder, which is source controlled, where you store all
your roles, environments and data_bags. Next, you use a tool like
Berkshelf to manage dependencies for your project cookbooks. This
encourages the use of one git-repo per cookbook.

So the file structure would look something like this:

chef-dataЩхен сай about
environments
roles
data_bags
cookbooks
projectcookbook1
projectcookbook2

Instead of downloading and owning the community cookbooks, you'd use
Berkshelf (which some describe as bundler/maven for Chef cookbooks).
You can find out more about Berkshelf at berkshelf.com
http://berkshelf.com or by watching this ChefConf video where Jamie
Winsor, the creator of Berkshelf, describes why Berkshelf was made:
http://www.youtube.com/watch?v=hYt0E84kYUI&feature=c4-overview-vl&list=PLrmstJpucjzXNMLcI5X-EjirpDd-SITd3

Thank you very much for the answer.

I will learned today and next days. And may be ask a some small
questions in this thread.

How I can understand now I was wrong when I said about using git sub
tree for store in a chef repo cookbook separately. Because cookbook is a
separate program which must be controlled in a separate version control
system repository.

But official documentation about chef-repo
(http://docs.opscode.com/essentials_repository.html)
will report otherwise:

cookbooks/ Contains cookbooks that have been downloaded from the
https://cookbooks.opscode.com or created locally.

This is confusing. Because at the one hand we must to store cookbooks in
a chef repo. But other hand we must store cookbooks separately and using
a berkshelf or librarian or other script for download cookbooks to the
chef server.

How I can understand now:

I must create a separate git repo (Now I'am doing this right now this)
and upload cookbooks i.e. my own application cookbooks and community
cookbooks with a tool named librarian or berkshelf?

I.e all cookbooks must be uploaded automatically and should be avaiable
from my admin workstation where is chef-repo, berkshelf and other
developmnet works.

Ideally I must store my bare git repos of my own cookbooks in the
separate server in local network or using github.

But push only production ready snapshots to the git server - best
practice of course.

Summing up:

Git subtree is not required because cookbooks is developing separated.

Is I am right ?

Or may be something wrong ?

--
Best regards,

CVision Lab System Administrator
Vladmir Skubriev

11

You are on the right track. If you are using Berkshelf/Librarian you don't
need Git submodules and you don't need to store your cookbooks in your
chef-repo.

Tom Duffield — Automation Consulting Engineer

651.769.7497 – tom@opscode.com – *my:
*Linkedinhttp://www.linkedin.com/in/thomasduffield/
Twitter https://twitter.com/tomduffield

OPSCODE

CODE CAN
opscode.com http://www.opscode.com Blog http://www.opscode.com/blog/
Facebook http://www.facebook.com/opscode
Twitterhttp://www.twitter.com/opscode
YouTube http://www.youtube.com/opscode

On Tue, Oct 29, 2013 at 9:18 AM, Vladimir Skubriev
skubriev@cvisionlab.comwrote:

29.10.2013 17:24, Tom Duffield пишет:

Storing all of your cookbooks in the chef-repo has become somewhat of an
anti-pattern. Instead, the recommendation is that you have a "chef-data"
folder, which is source controlled, where you store all your roles,
environments and data_bags. Next, you use a tool like Berkshelf to manage
dependencies for your project cookbooks. This encourages the use of one
git-repo per cookbook.

So the file structure would look something like this:

chef-dataЩхен сай about
environments
roles
data_bags
cookbooks
projectcookbook1
projectcookbook2

Instead of downloading and owning the community cookbooks, you'd use
Berkshelf (which some describe as bundler/maven for Chef cookbooks). You
can find out more about Berkshelf at berkshelf.com or by watching this
ChefConf video where Jamie Winsor, the creator of Berkshelf, describes why
Berkshelf was made:
http://www.youtube.com/watch?v=hYt0E84kYUI&feature=c4-overview-vl&list=PLrmstJpucjzXNMLcI5X-EjirpDd-SITd3

Thank you very much for the answer.

I will learned today and next days. And may be ask a some small questions
in this thread.

How I can understand now I was wrong when I said about using git sub tree
for store in a chef repo cookbook separately. Because cookbook is a
separate program which must be controlled in a separate version control
system repository.

But official documentation about chef-repo (
http://docs.opscode.com/essentials_repository.html)
will report otherwise:

cookbooks/ Contains cookbooks that have been downloaded from the
https://cookbooks.opscode.com or created locally.

This is confusing. Because at the one hand we must to store cookbooks in a
chef repo. But other hand we must store cookbooks separately and using a
berkshelf or librarian or other script for download cookbooks to the chef
server.

How I can understand now:

I must create a separate git repo (Now I'am doing this right now this) and
upload cookbooks i.e. my own application cookbooks and community cookbooks
with a tool named librarian or berkshelf?

I.e all cookbooks must be uploaded automatically and should be avaiable
from my admin workstation where is chef-repo, berkshelf and other
developmnet works.

Ideally I must store my bare git repos of my own cookbooks in the separate
server in local network or using github.

But push only production ready snapshots to the git server - best practice
of course.

Summing up:

Git subtree is not required because cookbooks is developing separated.

Is I am right ?

Or may be something wrong ?

--
Best regards,

CVision Lab System Administrator
Vladmir Skubriev

12

Hi Vladimir,

If you use berkshelf or librarian you usually .gitignore the cookbooks
directory where berkshelf/librarian download the cookbooks to, thus you
don't need submodules.

Concerning the chef-repo or "chef-data" repository, consider that they are
not necessarily in a 1:1 relation with chef server. For example, you can
have multiple chef-repo like repositories with databags/environments/roles
but still configured against the same chef server. I tend to call them
"infrastructure repositories", but "project repositories" or "chef-data
repository" are good names for them as well. Chef-repo is just a directory
structure, you can have many instances of them.

Cheers,
Torben
On Oct 29, 2013 3:41 PM, "Tom Duffield" tom@opscode.com wrote:

You are on the right track. If you are using Berkshelf/Librarian you don't
need Git submodules and you don't need to store your cookbooks in your
chef-repo.

Tom Duffield — Automation Consulting Engineer

651.769.7497 – tom@opscode.com – *my: *Linkedinhttp://www.linkedin.com/in/thomasduffield/
Twitter https://twitter.com/tomduffield

OPSCODE

CODE CAN
opscode.com http://www.opscode.com Bloghttp://www.opscode.com/blog/
Facebook http://www.facebook.com/opscode Twitterhttp://www.twitter.com/opscode
YouTube http://www.youtube.com/opscode

On Tue, Oct 29, 2013 at 9:18 AM, Vladimir Skubriev <
skubriev@cvisionlab.com> wrote:

29.10.2013 17:24, Tom Duffield пишет:

Storing all of your cookbooks in the chef-repo has become somewhat of an
anti-pattern. Instead, the recommendation is that you have a "chef-data"
folder, which is source controlled, where you store all your roles,
environments and data_bags. Next, you use a tool like Berkshelf to manage
dependencies for your project cookbooks. This encourages the use of one
git-repo per cookbook.

So the file structure would look something like this:

chef-dataЩхен сай about
environments
roles
data_bags
cookbooks
projectcookbook1
projectcookbook2

Instead of downloading and owning the community cookbooks, you'd use
Berkshelf (which some describe as bundler/maven for Chef cookbooks). You
can find out more about Berkshelf at berkshelf.com or by watching this
ChefConf video where Jamie Winsor, the creator of Berkshelf, describes why
Berkshelf was made:
http://www.youtube.com/watch?v=hYt0E84kYUI&feature=c4-overview-vl&list=PLrmstJpucjzXNMLcI5X-EjirpDd-SITd3

Thank you very much for the answer.

I will learned today and next days. And may be ask a some small questions
in this thread.

How I can understand now I was wrong when I said about using git sub tree
for store in a chef repo cookbook separately. Because cookbook is a
separate program which must be controlled in a separate version control
system repository.

But official documentation about chef-repo (
http://docs.opscode.com/essentials_repository.html)
will report otherwise:

cookbooks/ Contains cookbooks that have been downloaded from the
https://cookbooks.opscode.com or created locally.

This is confusing. Because at the one hand we must to store cookbooks in
a chef repo. But other hand we must store cookbooks separately and using a
berkshelf or librarian or other script for download cookbooks to the chef
server.

How I can understand now:

I must create a separate git repo (Now I'am doing this right now this)
and upload cookbooks i.e. my own application cookbooks and community
cookbooks with a tool named librarian or berkshelf?

I.e all cookbooks must be uploaded automatically and should be avaiable
from my admin workstation where is chef-repo, berkshelf and other
developmnet works.

Ideally I must store my bare git repos of my own cookbooks in the
separate server in local network or using github.

But push only production ready snapshots to the git server - best
practice of course.

Summing up:

Git subtree is not required because cookbooks is developing separated.

Is I am right ?

Or may be something wrong ?

--
Best regards,

CVision Lab System Administrator
Vladmir Skubriev

13

29.10.2013 18:41, Tom Duffield пишет:

You are on the right track. If you are using Berkshelf/Librarian you
don't need Git submodules and you don't need to store your cookbooks
in your chef-repo.

Thank you very much. Now I understand my way.

--
Best regards,

CVision Lab System Administrator
Vladmir Skubriev

14

29.10.2013 21:14, Torben Knerr пишет:

Hi Vladimir,

If you use berkshelf or librarian you usually .gitignore the cookbooks
directory where berkshelf/librarian download the cookbooks to, thus
you don't need submodules.

Concerning the chef-repo or "chef-data" repository, consider that they
are not necessarily in a 1:1 relation with chef server. For example,
you can have multiple chef-repo like repositories with
databags/environments/roles but still configured against the same chef
server. I tend to call them "infrastructure repositories", but
"project repositories" or "chef-data repository" are good names for
them as well. Chef-repo is just a directory structure, you can have
many instances of them.

Cheers,
Torben

Thank you for a tip and answer

--
Best regards,

CVision Lab System Administrator
Vladmir Skubriev