Better workflow for actively-developed cookbooks

I have yet to find an ideal setup for this, but librarian-chef is oh so close.

When you are actively developing a cookbook (or modifying a fork, etc.), you need a git clone from which you can push and pull. This should be the same directory where knife expects to find the cookbook so things like “knife cookbook metadata” will work as expected (knife should arguably be less tied to a particular directory layout, but that’s another issue).

I tried to make this work by cloning my active cookbook repos into vendor/cookbooks and then pointing the Cheffile at them via :path => ‘vendor/cookbooks/foo’, but for some reason that copies the cookbook into the main cookbooks/ dir when you run librarian-chef install. Why is that? Knife then operates on that cookbook rather than the one in vendor/cookbooks and changes would need to be manually kept in sync with the git working copy. So that’s a dead end.

The ideal scenario would be if librarian-chef could give me a cloned git repo from which I can push and pull (and properly isolate it from the parent repo in which it is operating, via submodules or something else), which is also in knife’s cookbook path. Bonus points if it can do that from the :git => … option rather than :path => …

If there’s a way to easily do this that I’m missing, I’d love to hear about it.

Thanks chefs!

Wes

Wes,

Librarian-Chef takes over your repository's ./cookbooks/ directory by
design. It is no longer your directory. It is no longer source code. It no
longer belongs to the source repository. It is now a build artifact that is
produced by Librarian-Chef on your behalf, on-command, as per the
instructions you give to it on the command line and in your Cheffile. The
readme talks more about this and gives some tips for dealing with cookbooks
you are working on: GitHub - applicationsonline/librarian: Librarian - A Framework for Bundlers. Librarian-Chef is at: https://github.com/applicationsonline/librarian-chef..

A future version of Librarian-Chef may allow you to configure the choice of
directory.

Librarian-Chef assumes that you have a way to develop and test the
underlying cookbooks independently of any consuming infrastructure
repository. Sadly, the state of the world is not yet such that most
cookbooks are developed this way. So there will be some friction over the
near term, friction like the kind you have discovered, as the state of the
world progresses in that direction.

Until then, the options available may seem like a choice between, on the
one hand, a whole lot of work in testing a cookbook fork independently and
in isolation and, on the other hand, hacks with the git and path sources in
your Cheffile.

Cheers,
Jay

On Fri, Jun 22, 2012 at 12:05 PM, Wes Morgan cap10morgan@gmail.com wrote:

I have yet to find an ideal setup for this, but librarian-chef is oh so
close.

When you are actively developing a cookbook (or modifying a fork, etc.),
you need a git clone from which you can push and pull. This should be the
same directory where knife expects to find the cookbook so things like
"knife cookbook metadata" will work as expected (knife should arguably be
less tied to a particular directory layout, but that's another issue).

I tried to make this work by cloning my active cookbook repos into
vendor/cookbooks and then pointing the Cheffile at them via :path =>
'vendor/cookbooks/foo', but for some reason that copies the cookbook into
the main cookbooks/ dir when you run librarian-chef install. Why is that?
Knife then operates on that cookbook rather than the one in
vendor/cookbooks and changes would need to be manually kept in sync with
the git working copy. So that's a dead end.

The ideal scenario would be if librarian-chef could give me a cloned git
repo from which I can push and pull (and properly isolate it from the
parent repo in which it is operating, via submodules or something else),
which is also in knife's cookbook path. Bonus points if it can do that from
the :git => ... option rather than :path => ...

If there's a way to easily do this that I'm missing, I'd love to hear
about it.

Thanks chefs!

Wes

Thanks for your clarifications. I understand most of it, and I'm not actually asking for those things to be changed. I don't think. :slight_smile:

What I really need, and what I should have said first:

With bundler, when I specify a :path => option, it just uses the code in that path directly. Is there a reason Librarian-Chef can't do that and instead copies it into the cookbooks dir? If there were just the one directory for the path'd cookbook, that would pretty much solve my problem. It sounded like the L-C knife integration could manipulate knife's cookbook path, so I was surprised it didn't do that and just point it at the directory specified in the :path => option.

Wes

On Jun 22, 2012, at 1:04 PM, Jay Feldblum wrote:

Wes,

Librarian-Chef takes over your repository's ./cookbooks/ directory by design. It is no longer your directory. It is no longer source code. It no longer belongs to the source repository. It is now a build artifact that is produced by Librarian-Chef on your behalf, on-command, as per the instructions you give to it on the command line and in your Cheffile. The readme talks more about this and gives some tips for dealing with cookbooks you are working on: GitHub - applicationsonline/librarian: Librarian - A Framework for Bundlers. Librarian-Chef is at: https://github.com/applicationsonline/librarian-chef..

A future version of Librarian-Chef may allow you to configure the choice of directory.

Librarian-Chef assumes that you have a way to develop and test the underlying cookbooks independently of any consuming infrastructure repository. Sadly, the state of the world is not yet such that most cookbooks are developed this way. So there will be some friction over the near term, friction like the kind you have discovered, as the state of the world progresses in that direction.

Until then, the options available may seem like a choice between, on the one hand, a whole lot of work in testing a cookbook fork independently and in isolation and, on the other hand, hacks with the git and path sources in your Cheffile.

Cheers,
Jay

On Fri, Jun 22, 2012 at 12:05 PM, Wes Morgan cap10morgan@gmail.com wrote:
I have yet to find an ideal setup for this, but librarian-chef is oh so close.

When you are actively developing a cookbook (or modifying a fork, etc.), you need a git clone from which you can push and pull. This should be the same directory where knife expects to find the cookbook so things like "knife cookbook metadata" will work as expected (knife should arguably be less tied to a particular directory layout, but that's another issue).

I tried to make this work by cloning my active cookbook repos into vendor/cookbooks and then pointing the Cheffile at them via :path => 'vendor/cookbooks/foo', but for some reason that copies the cookbook into the main cookbooks/ dir when you run librarian-chef install. Why is that? Knife then operates on that cookbook rather than the one in vendor/cookbooks and changes would need to be manually kept in sync with the git working copy. So that's a dead end.

The ideal scenario would be if librarian-chef could give me a cloned git repo from which I can push and pull (and properly isolate it from the parent repo in which it is operating, via submodules or something else), which is also in knife's cookbook path. Bonus points if it can do that from the :git => ... option rather than :path => ...

If there's a way to easily do this that I'm missing, I'd love to hear about it.

Thanks chefs!

Wes

Hey Wes,

There was a great conversation related to Librarian-Chef earlier in April:

http://lists.opscode.com/sympa/arc/chef/2012-04/msg00326.html

Not sure if this is useful for you but hope it helps!

Cheers,
-K.

On Fri, Jun 22, 2012 at 1:24 PM, Wes Morgan cap10morgan@gmail.com wrote:

Thanks for your clarifications. I understand most of it, and I'm not
actually asking for those things to be changed. I don't think. :slight_smile:

What I really need, and what I should have said first:

With bundler, when I specify a :path => option, it just uses the code in
that path directly. Is there a reason Librarian-Chef can't do that and
instead copies it into the cookbooks dir? If there were just the one
directory for the path'd cookbook, that would pretty much solve my problem.
It sounded like the L-C knife integration could manipulate knife's cookbook
path, so I was surprised it didn't do that and just point it at the
directory specified in the :path => option.

Wes

On Jun 22, 2012, at 1:04 PM, Jay Feldblum wrote:

Wes,

Librarian-Chef takes over your repository's ./cookbooks/ directory by
design. It is no longer your directory. It is no longer source code. It no
longer belongs to the source repository. It is now a build artifact that is
produced by Librarian-Chef on your behalf, on-command, as per the
instructions you give to it on the command line and in your Cheffile. The
readme talks more about this and gives some tips for dealing with cookbooks
you are working on: GitHub - applicationsonline/librarian: Librarian - A Framework for Bundlers. Librarian-Chef is at: https://github.com/applicationsonline/librarian-chef..

A future version of Librarian-Chef may allow you to configure the choice of
directory.

Librarian-Chef assumes that you have a way to develop and test the
underlying cookbooks independently of any consuming infrastructure
repository. Sadly, the state of the world is not yet such that most
cookbooks are developed this way. So there will be some friction over the
near term, friction like the kind you have discovered, as the state of the
world progresses in that direction.

Until then, the options available may seem like a choice between, on the one
hand, a whole lot of work in testing a cookbook fork independently and in
isolation and, on the other hand, hacks with the git and path sources in
your Cheffile.

Cheers,
Jay

On Fri, Jun 22, 2012 at 12:05 PM, Wes Morgan cap10morgan@gmail.com wrote:

I have yet to find an ideal setup for this, but librarian-chef is oh so
close.

When you are actively developing a cookbook (or modifying a fork, etc.),
you need a git clone from which you can push and pull. This should be the
same directory where knife expects to find the cookbook so things like
"knife cookbook metadata" will work as expected (knife should arguably be
less tied to a particular directory layout, but that's another issue).

I tried to make this work by cloning my active cookbook repos into
vendor/cookbooks and then pointing the Cheffile at them via :path =>
'vendor/cookbooks/foo', but for some reason that copies the cookbook into
the main cookbooks/ dir when you run librarian-chef install. Why is that?
Knife then operates on that cookbook rather than the one in vendor/cookbooks
and changes would need to be manually kept in sync with the git working
copy. So that's a dead end.

The ideal scenario would be if librarian-chef could give me a cloned git
repo from which I can push and pull (and properly isolate it from the parent
repo in which it is operating, via submodules or something else), which is
also in knife's cookbook path. Bonus points if it can do that from the :git
=> ... option rather than :path => ...

If there's a way to easily do this that I'm missing, I'd love to hear
about it.

Thanks chefs!

Wes

--
Kevin Karwaski

I've still been really pleased with the way I use Librarian-Chef. I let it
take over my cookbooks directory within my chef-repo and make sure I don't
version control that directory so instead it is just a "build artifact" as
Jay calls it. I keep the chef-repo/cookbooks directory as the only
cookbook_path in my knife config so I can easily 'knife cookbook upload -a'.

There are two locations for cookbooks I am developing.

  1. Forked Cookbooks which I put outside my chef-repo directory under
    ../cookbooks and then I use :path in Librarian-Chef Cheffile to bring those
    into the chef-repo/cookbooks "build" directory.
    When I'm done developing these cookbooks I push them to the git server
    and change their entries in Cheffile to :git entries.
  2. Private? Cookbooks which I put inside my chef-repo/site-cookbooks
    directory so they get version controlled with the rest of the chef-repo.
    These Cheffile entries are permanently :path entries regardless of whether
    or not I'm making changes to them.

If I make changes to either of these two types of cookbooks and I want to
test those changes I just run "librarian-chef install" and the
chef-repo/cookbooks build directory picks up all the changes.
Then I can either run "chef-solo", "vagrant provision" or "knife cookbook
upload -a" depending on how I'm testing.

I understand what you mean but I wouldn't say knife is "tied to a
particular directory layout". You can add the -o option to knife
subcommands to specify what paths to look for the cookbook.
I find that is a small inconvenience compared to the flexibility otherwise
provided by Librarian-Chef and my usage.

I don't bother with the Librarian-Chef knife integration pretty much just
because I don't like that I can't run many (all?) knife commands unless I'm
in the chef-repo where librarian-chef has been initialized. The knife
commands just break and I find that to be too restrictive.

Jeremiah

On Fri, Jun 22, 2012 at 1:24 PM, Wes Morgan cap10morgan@gmail.com wrote:

Thanks for your clarifications. I understand most of it, and I'm not
actually asking for those things to be changed. I don't think. :slight_smile:

What I really need, and what I should have said first:

With bundler, when I specify a :path => option, it just uses the code in
that path directly. Is there a reason Librarian-Chef can't do that and
instead copies it into the cookbooks dir? If there were just the one
directory for the path'd cookbook, that would pretty much solve my problem.
It sounded like the L-C knife integration could manipulate knife's cookbook
path, so I was surprised it didn't do that and just point it at the
directory specified in the :path => option.

Wes

On Jun 22, 2012, at 1:04 PM, Jay Feldblum wrote:

Wes,

Librarian-Chef takes over your repository's ./cookbooks/ directory by
design. It is no longer your directory. It is no longer source code. It no
longer belongs to the source repository. It is now a build artifact that is
produced by Librarian-Chef on your behalf, on-command, as per the
instructions you give to it on the command line and in your Cheffile. The
readme talks more about this and gives some tips for dealing with cookbooks
you are working on: GitHub - applicationsonline/librarian: Librarian - A Framework for Bundlers. Librarian-Chef is at: https://github.com/applicationsonline/librarian-chef..

A future version of Librarian-Chef may allow you to configure the choice
of directory.

Librarian-Chef assumes that you have a way to develop and test the
underlying cookbooks independently of any consuming infrastructure
repository. Sadly, the state of the world is not yet such that most
cookbooks are developed this way. So there will be some friction over the
near term, friction like the kind you have discovered, as the state of the
world progresses in that direction.

Until then, the options available may seem like a choice between, on the
one hand, a whole lot of work in testing a cookbook fork independently and
in isolation and, on the other hand, hacks with the git and path sources in
your Cheffile.

Cheers,
Jay

On Fri, Jun 22, 2012 at 12:05 PM, Wes Morgan cap10morgan@gmail.comwrote:

I have yet to find an ideal setup for this, but librarian-chef is oh so
close.

When you are actively developing a cookbook (or modifying a fork, etc.),
you need a git clone from which you can push and pull. This should be the
same directory where knife expects to find the cookbook so things like
"knife cookbook metadata" will work as expected (knife should arguably be
less tied to a particular directory layout, but that's another issue).

I tried to make this work by cloning my active cookbook repos into
vendor/cookbooks and then pointing the Cheffile at them via :path =>
'vendor/cookbooks/foo', but for some reason that copies the cookbook into
the main cookbooks/ dir when you run librarian-chef install. Why is that?
Knife then operates on that cookbook rather than the one in
vendor/cookbooks and changes would need to be manually kept in sync with
the git working copy. So that's a dead end.

The ideal scenario would be if librarian-chef could give me a cloned git
repo from which I can push and pull (and properly isolate it from the
parent repo in which it is operating, via submodules or something else),
which is also in knife's cookbook path. Bonus points if it can do that from
the :git => ... option rather than :path => ...

If there's a way to easily do this that I'm missing, I'd love to hear
about it.

Thanks chefs!

Wes

“A future version of Librarian-Chef may allow you to configure the choice
of directory.”

When I first started using Librarian-Chef I ran "librarian-chef init"
within the site-cookbooks directory and experimented with it’s use there.
I didn’t seem to notice any problems running in another directory. Was I
missing something?

Jeremiah

I'm more or less using your approach, Jeremiah. The friction I ran into was when I wanted to roll a new release of a cookbook I manage (linode) from a git repo. The "knife cookbook metadata [ linode | . | -o ./ linode | metadata.rb]" commands wouldn't generate a metadata.json file for me. It wanted the cookbook to be in cookbooks/linode relative to cwd. Hence my "tied to a particular directory layout" comment.

So that sent me down the librarian-chef :path => (sorry, I couldn't resist), which then had different behavior than I expected given its similarity to the bundler option of the same name.

For the specific case of metadata.json generation, it would be nice if knife would work directly on a metadata.rb file. That would at least solve my immediate issue. And I'm sure I'll learn to live with the way L-C works. Overall it's a big improvement over what I was doing before.

Wes

On Jun 22, 2012, at 1:52 PM, Jeremiah Snapp wrote:

I've still been really pleased with the way I use Librarian-Chef. I let it take over my cookbooks directory within my chef-repo and make sure I don't version control that directory so instead it is just a "build artifact" as Jay calls it. I keep the chef-repo/cookbooks directory as the only cookbook_path in my knife config so I can easily 'knife cookbook upload -a'.

There are two locations for cookbooks I am developing.

Forked Cookbooks which I put outside my chef-repo directory under ../cookbooks and then I use :path in Librarian-Chef Cheffile to bring those into the chef-repo/cookbooks "build" directory.
When I'm done developing these cookbooks I push them to the git server and change their entries in Cheffile to :git entries.
Private? Cookbooks which I put inside my chef-repo/site-cookbooks directory so they get version controlled with the rest of the chef-repo. These Cheffile entries are permanently :path entries regardless of whether or not I'm making changes to them.
If I make changes to either of these two types of cookbooks and I want to test those changes I just run "librarian-chef install" and the chef-repo/cookbooks build directory picks up all the changes.
Then I can either run "chef-solo", "vagrant provision" or "knife cookbook upload -a" depending on how I'm testing.

I understand what you mean but I wouldn't say knife is "tied to a particular directory layout". You can add the -o option to knife subcommands to specify what paths to look for the cookbook.
I find that is a small inconvenience compared to the flexibility otherwise provided by Librarian-Chef and my usage.

I don't bother with the Librarian-Chef knife integration pretty much just because I don't like that I can't run many (all?) knife commands unless I'm in the chef-repo where librarian-chef has been initialized. The knife commands just break and I find that to be too restrictive.

Jeremiah

On Fri, Jun 22, 2012 at 1:24 PM, Wes Morgan cap10morgan@gmail.com wrote:
Thanks for your clarifications. I understand most of it, and I'm not actually asking for those things to be changed. I don't think. :slight_smile:

What I really need, and what I should have said first:

With bundler, when I specify a :path => option, it just uses the code in that path directly. Is there a reason Librarian-Chef can't do that and instead copies it into the cookbooks dir? If there were just the one directory for the path'd cookbook, that would pretty much solve my problem. It sounded like the L-C knife integration could manipulate knife's cookbook path, so I was surprised it didn't do that and just point it at the directory specified in the :path => option.

Wes

On Jun 22, 2012, at 1:04 PM, Jay Feldblum wrote:

Wes,

Librarian-Chef takes over your repository's ./cookbooks/ directory by design. It is no longer your directory. It is no longer source code. It no longer belongs to the source repository. It is now a build artifact that is produced by Librarian-Chef on your behalf, on-command, as per the instructions you give to it on the command line and in your Cheffile. The readme talks more about this and gives some tips for dealing with cookbooks you are working on: GitHub - applicationsonline/librarian: Librarian - A Framework for Bundlers. Librarian-Chef is at: https://github.com/applicationsonline/librarian-chef..

A future version of Librarian-Chef may allow you to configure the choice of directory.

Librarian-Chef assumes that you have a way to develop and test the underlying cookbooks independently of any consuming infrastructure repository. Sadly, the state of the world is not yet such that most cookbooks are developed this way. So there will be some friction over the near term, friction like the kind you have discovered, as the state of the world progresses in that direction.

Until then, the options available may seem like a choice between, on the one hand, a whole lot of work in testing a cookbook fork independently and in isolation and, on the other hand, hacks with the git and path sources in your Cheffile.

Cheers,
Jay

On Fri, Jun 22, 2012 at 12:05 PM, Wes Morgan cap10morgan@gmail.com wrote:
I have yet to find an ideal setup for this, but librarian-chef is oh so close.

When you are actively developing a cookbook (or modifying a fork, etc.), you need a git clone from which you can push and pull. This should be the same directory where knife expects to find the cookbook so things like "knife cookbook metadata" will work as expected (knife should arguably be less tied to a particular directory layout, but that's another issue).

I tried to make this work by cloning my active cookbook repos into vendor/cookbooks and then pointing the Cheffile at them via :path => 'vendor/cookbooks/foo', but for some reason that copies the cookbook into the main cookbooks/ dir when you run librarian-chef install. Why is that? Knife then operates on that cookbook rather than the one in vendor/cookbooks and changes would need to be manually kept in sync with the git working copy. So that's a dead end.

The ideal scenario would be if librarian-chef could give me a cloned git repo from which I can push and pull (and properly isolate it from the parent repo in which it is operating, via submodules or something else), which is also in knife's cookbook path. Bonus points if it can do that from the :git => ... option rather than :path => ...

If there's a way to easily do this that I'm missing, I'd love to hear about it.

Thanks chefs!

Wes

I could be missing something but I can run “knife cookbook metadata
vagrant_extras -o …/cookbooks/” from within my chef-repo directory and it
works fine. That reaches outside my chef-repo to my cookbooks directory
with forked cookbooks. I can also run “knife cookbook metadata -a -o
…/cookbooks/” and it would run against all forked cookbooks in that
directory. I think the problem might be that you are putting your options
(-o) before your cookbook name. The options have to come after the
cookbook name. Let me know if I’m totally misunderstanding your problem.

Wes,

It's a good question.

Right now, the reason is that the Knife integration is a little too
simplistic for its ambitions.

The Knife integration does not manipulate the cookbook_path. Instead, you
do that yourself. The Knife integration just tells you the path to use when
set the cookbook_path.

There are certainly better ways out there to do it, ways which might
address what you are asking about, and which I may explore in the future.

Cheers,
Jay

On Fri, Jun 22, 2012 at 1:24 PM, Wes Morgan cap10morgan@gmail.com wrote:

Thanks for your clarifications. I understand most of it, and I'm not
actually asking for those things to be changed. I don't think. :slight_smile:

What I really need, and what I should have said first:

With bundler, when I specify a :path => option, it just uses the code in
that path directly. Is there a reason Librarian-Chef can't do that and
instead copies it into the cookbooks dir? If there were just the one
directory for the path'd cookbook, that would pretty much solve my problem.
It sounded like the L-C knife integration could manipulate knife's cookbook
path, so I was surprised it didn't do that and just point it at the
directory specified in the :path => option.

Wes

alright, I have a pretty heterodox idea of how I would like to use
librarian-chef so that a team of infrastructure devs can work in sync

I am on a team of 3 infrastructure devs, i am the (relative) expert, the
other 2 guys are smart but n00bs

I want us to have one common Cheffile in our one common chef-repo

the cookbooks we develop independently each have their own git repo,
unfortunately, private ones for the most part

When I create a new git repository for application-foo, I want to add the
git repository link, branch name/tag/commit name, to Cheffile so that when
the other guys are working, they can easily pull in the cookbooks that I am
working on and vice versa

However, I dont want to do the git add . && git commit -am 'foo' && git push origin master && librarian-chef update dance when I am actively
working on a cookbook that is within an "active" cookbook.

I don't need librarian to resolve any dependencies for my active cookbooks,
I just want a common file w/ the list of all cookbooks we are working on as
a team and i want librarian to download them if they don't exist already.

Perhaps this is a perversion of all things bundler but this is what I want.
It also would get much more complicated if we didn't have git repos w/
shared commit access.

Is this crazy or a good idea or both?

On Fri, Jun 22, 2012 at 11:01 PM, Jay Feldblum y_feldblum@yahoo.com wrote:

Wes,

It's a good question.

Right now, the reason is that the Knife integration is a little too
simplistic for its ambitions.

The Knife integration does not manipulate the cookbook_path. Instead, you
do that yourself. The Knife integration just tells you the path to use when
set the cookbook_path.

There are certainly better ways out there to do it, ways which might
address what you are asking about, and which I may explore in the future.

Cheers,
Jay

On Fri, Jun 22, 2012 at 1:24 PM, Wes Morgan cap10morgan@gmail.com wrote:

Thanks for your clarifications. I understand most of it, and I'm not
actually asking for those things to be changed. I don't think. :slight_smile:

What I really need, and what I should have said first:

With bundler, when I specify a :path => option, it just uses the code in
that path directly. Is there a reason Librarian-Chef can't do that and
instead copies it into the cookbooks dir? If there were just the one
directory for the path'd cookbook, that would pretty much solve my problem.
It sounded like the L-C knife integration could manipulate knife's cookbook
path, so I was surprised it didn't do that and just point it at the
directory specified in the :path => option.

Wes

Hey Wes / Bryan, et all

We (Riot Games) have a software solution for this very problem that we will be announcing to the community in the next few days. We've been using it internally and just want to get our messaging and documentation setup before we do so.

We have every single Cookbook in it's a separate Git repository of it's own and each one of those maps to a job on our build server. The job on the build server calls this software solution and it gathers dependencies of the Cookbook we're building and allows you to then publish it, and it's dependencies, to your Chef server.

Every single application project at Riot also uses this dependency getter to figure out what Cookbook Versions should be used to deploy the built artifact by either reading a Lockfile or coming up with a solution based on the metadata of the Cookbooks found in the dependency file and their dependencies.

With this tool we're able to iterate quickly and version our Cookbooks in a more sane manner. It allows you to throw out the idea that you should be keeping Cookbooks in your Chef-Repo.

I'll be sending an email to the list when we're ready.

--
Jamie Winsor
@resetexistence

On Friday, June 22, 2012 at 2:28 PM, Bryan Berry wrote:

alright, I have a pretty heterodox idea of how I would like to use librarian-chef so that a team of infrastructure devs can work in sync

I am on a team of 3 infrastructure devs, i am the (relative) expert, the other 2 guys are smart but n00bs

I want us to have one common Cheffile in our one common chef-repo

the cookbooks we develop independently each have their own git repo, unfortunately, private ones for the most part

When I create a new git repository for application-foo, I want to add the git repository link, branch name/tag/commit name, to Cheffile so that when the other guys are working, they can easily pull in the cookbooks that I am working on and vice versa

However, I dont want to do the git add . && git commit -am 'foo' && git push origin master && librarian-chef update dance when I am actively working on a cookbook that is within an "active" cookbook.

I don't need librarian to resolve any dependencies for my active cookbooks, I just want a common file w/ the list of all cookbooks we are working on as a team and i want librarian to download them if they don't exist already.

Perhaps this is a perversion of all things bundler but this is what I want. It also would get much more complicated if we didn't have git repos w/ shared commit access.

Is this crazy or a good idea or both?

On Fri, Jun 22, 2012 at 11:01 PM, Jay Feldblum <y_feldblum@yahoo.com (mailto:y_feldblum@yahoo.com)> wrote:

Wes,

It's a good question.

Right now, the reason is that the Knife integration is a little too simplistic for its ambitions.

The Knife integration does not manipulate the cookbook_path. Instead, you do that yourself. The Knife integration just tells you the path to use when set the cookbook_path.

There are certainly better ways out there to do it, ways which might address what you are asking about, and which I may explore in the future.

Cheers,
Jay

On Fri, Jun 22, 2012 at 1:24 PM, Wes Morgan <cap10morgan@gmail.com (mailto:cap10morgan@gmail.com)> wrote:

Thanks for your clarifications. I understand most of it, and I'm not actually asking for those things to be changed. I don't think. :slight_smile:

What I really need, and what I should have said first:

With bundler, when I specify a :path => option, it just uses the code in that path directly. Is there a reason Librarian-Chef can't do that and instead copies it into the cookbooks dir? If there were just the one directory for the path'd cookbook, that would pretty much solve my problem. It sounded like the L-C knife integration could manipulate knife's cookbook path, so I was surprised it didn't do that and just point it at the directory specified in the :path => option.

Wes

Bryan,

The various rigorous practices that developers have are often
well-supported by their tools.

The same practices are not as well-supported by the devops tools because
these tools are still being built and because the ideas and practices are
still coming across. Devops as a field is still under construction.

In the meantime: you mix and match, you pick your battles, and you do what
you have to do.

Cheers,
Jay

On Fri, Jun 22, 2012 at 5:28 PM, Bryan Berry bryan.berry@gmail.com wrote:

alright, I have a pretty heterodox idea of how I would like to use
librarian-chef so that a team of infrastructure devs can work in sync

I am on a team of 3 infrastructure devs, i am the (relative) expert, the
other 2 guys are smart but n00bs

I want us to have one common Cheffile in our one common chef-repo

the cookbooks we develop independently each have their own git repo,
unfortunately, private ones for the most part

When I create a new git repository for application-foo, I want to add the
git repository link, branch name/tag/commit name, to Cheffile so that when
the other guys are working, they can easily pull in the cookbooks that I am
working on and vice versa

However, I dont want to do the git add . && git commit -am 'foo' && git push origin master && librarian-chef update dance when I am actively
working on a cookbook that is within an "active" cookbook.

I don't need librarian to resolve any dependencies for my active
cookbooks, I just want a common file w/ the list of all cookbooks we are
working on as a team and i want librarian to download them if they don't
exist already.

Perhaps this is a perversion of all things bundler but this is what I
want. It also would get much more complicated if we didn't have git repos
w/ shared commit access.

Is this crazy or a good idea or both?

On Saturday, June 23, 2012, Wes Morgan wrote:

I'm more or less using your approach, Jeremiah. The friction I ran into
was when I wanted to roll a new release of a cookbook I manage (linode)
from a git repo. The "knife cookbook metadata [ linode | . | -o ./ linode |
metadata.rb]" commands wouldn't generate a metadata.json file for me.

I don't have my code base in front of me but I typically run a command like

knife cookbook metadata from file somepath/metadata.rb

It wanted the cookbook to be in cookbooks/linode relative to cwd. Hence my
"tied to a particular directory layout" comment.

So that sent me down the librarian-chef :path => (sorry, I couldn't
resist), which then had different behavior than I expected given its
similarity to the bundler option of the same name.

For the specific case of metadata.json generation, it would be nice if
knife would work directly on a metadata.rb file. That would at least solve
my immediate issue. And I'm sure I'll learn to live with the way L-C works.
Overall it's a big improvement over what I was doing before.

Wes

On Jun 22, 2012, at 1:52 PM, Jeremiah Snapp wrote:

I've still been really pleased with the way I use Librarian-Chef. I let
it take over my cookbooks directory within my chef-repo and make sure I
don't version control that directory so instead it is just a "build
artifact" as Jay calls it. I keep the chef-repo/cookbooks directory as the
only cookbook_path in my knife config so I can easily 'knife cookbook
upload -a'.

There are two locations for cookbooks I am developing.

  1. Forked Cookbooks which I put outside my chef-repo directory under
    ../cookbooks and then I use :path in Librarian-Chef Cheffile to bring those
    into the chef-repo/cookbooks "build" directory.
    When I'm done developing these cookbooks I push them to the git server
    and change their entries in Cheffile to :git entries.
  2. Private? Cookbooks which I put inside my chef-repo/site-cookbooks
    directory so they get version controlled with the rest of the chef-repo.
    These Cheffile entries are permanently :path entries regardless of whether
    or not I'm making changes to them.

If I make changes to either of these two types of cookbooks and I want to
test those changes I just run "librarian-chef install" and the
chef-repo/cookbooks build directory picks up all the changes.
Then I can either run "chef-solo", "vagrant provision" or "knife cookbook
upload -a" depending on how I'm testing.

I understand what you mean but I wouldn't say knife is "tied to a
particular directory layout". You can add the -o option to knife
subcommands to specify what paths to look for the cookbook.
I find that is a small inconvenience compared to the flexibility otherwise
provided by Librarian-Chef and my usage.

I don't bother with the Librarian-Chef knife integration pretty much just
because I don't like that I can't run many (all?) knife commands unless I'm
in the chef-repo where librarian-chef has been initialized. The knife
commands just break and I find that to be too restrictive.

Jeremiah

On Fri, Jun 22, 2012 at 1:24 PM, Wes Morgan cap10morgan@gmail.com wrote:

Thanks for your clarifications. I understand most of it, and I'm not
actually asking for those things to be changed. I don't think. :slight_smile:

What I really need, and what I should have said first:

With bundler, when I specify a :path => option, it just uses the code in
that path directly. Is there a reason Librarian-Chef can't do that and
instead copies it into the cookbooks dir? If there were just the one
directory for the path'd cookbook, that would pretty much solve my problem.
It sounded like the L-C knife integration could manipulate knife's cookbook
path, so I was surprised it didn't do that and just point it at the
directory specified in the :path => option.

Wes

On Jun 22, 2012, at 1:04 PM, Jay Feldblum wrote:

Wes,

Librarian-Chef takes over your repository's ./cookbooks/ directory by
design. It is no longer your directory. It is no longer source code. It no
longer belongs to the source repository. It is now a build artifact that is
produced by Librarian-Chef on your behalf, on-command, as per the
instructions you give to it on the command line and in your Cheffile. The
readme talks more about this and gives some tips for dealing with cookbooks
you are working on: GitHub - applicationsonline/librarian: Librarian - A Framework for Bundlers. Librarian-Chef is at: https://github.com/applicationsonline/librarian-chef..

A future version of Librarian-Chef may allow you to configure the choice
of directory.

Librarian-Chef assumes that you have a way to develop and test the
underlying cookbooks independently of any consuming infrastructure
repository. Sadly, the state of the world is not yet such that most
cookbooks are developed this way. So there will be some friction over the
near term, friction like the kind you have discovered, as the state of the
world progresses in that direction.

Until then, the options available may seem like a choice between, on the
one hand, a whole lot of work in testing a cookbook fork independently and
in isolation and, on the other hand, hacks with the git and

--
Cheers,

Peter Donald

Jay,

I don't quite understand your point. Are you are saying that
Librarian-chef isn't meant to meet the particular use case I have described
and should be looking for a separate tool?

On Fri, Jun 22, 2012 at 11:44 PM, Jay Feldblum y_feldblum@yahoo.com wrote:

Bryan,

The various rigorous practices that developers have are often
well-supported by their tools.

The same practices are not as well-supported by the devops tools because
these tools are still being built and because the ideas and practices are
still coming across. Devops as a field is still under construction.

In the meantime: you mix and match, you pick your battles, and you do what
you have to do.

Cheers,
Jay

On Fri, Jun 22, 2012 at 5:28 PM, Bryan Berry bryan.berry@gmail.comwrote:

alright, I have a pretty heterodox idea of how I would like to use
librarian-chef so that a team of infrastructure devs can work in sync

I am on a team of 3 infrastructure devs, i am the (relative) expert, the
other 2 guys are smart but n00bs

I want us to have one common Cheffile in our one common chef-repo

the cookbooks we develop independently each have their own git repo,
unfortunately, private ones for the most part

When I create a new git repository for application-foo, I want to add the
git repository link, branch name/tag/commit name, to Cheffile so that when
the other guys are working, they can easily pull in the cookbooks that I am
working on and vice versa

However, I dont want to do the git add . && git commit -am 'foo' && git push origin master && librarian-chef update dance when I am actively
working on a cookbook that is within an "active" cookbook.

I don't need librarian to resolve any dependencies for my active
cookbooks, I just want a common file w/ the list of all cookbooks we are
working on as a team and i want librarian to download them if they don't
exist already.

Perhaps this is a perversion of all things bundler but this is what I
want. It also would get much more complicated if we didn't have git repos
w/ shared commit access.

Is this crazy or a good idea or both?

ohai, just saw this:

On 23 June 2012 20:45, Bryan Berry bryan.berry@gmail.com wrote:

Jay,

I don't quite understand your point. Are you are saying that Librarian-chef
isn't meant to meet the particular use case I have described and should be
looking for a separate tool?

On Fri, Jun 22, 2012 at 11:44 PM, Jay Feldblum y_feldblum@yahoo.com wrote:

Bryan,

The various rigorous practices that developers have are often
well-supported by their tools.

The same practices are not as well-supported by the devops tools because
these tools are still being built and because the ideas and practices are
still coming across. Devops as a field is still under construction.

In the meantime: you mix and match, you pick your battles, and you do what
you have to do.

Cheers,
Jay

On Fri, Jun 22, 2012 at 5:28 PM, Bryan Berry bryan.berry@gmail.com
wrote:

alright, I have a pretty heterodox idea of how I would like to use
librarian-chef so that a team of infrastructure devs can work in sync

I am on a team of 3 infrastructure devs, i am the (relative) expert, the
other 2 guys are smart but n00bs

I want us to have one common Cheffile in our one common chef-repo

the cookbooks we develop independently each have their own git repo,
unfortunately, private ones for the most part

When I create a new git repository for application-foo, I want to add the
git repository link, branch name/tag/commit name, to Cheffile so that when
the other guys are working, they can easily pull in the cookbooks that I am
working on and vice versa

However, I dont want to do the git add . && git commit -am 'foo' && git push origin master && librarian-chef update dance when I am actively
working on a cookbook that is within an "active" cookbook.

I don't need librarian to resolve any dependencies for my active
cookbooks, I just want a common file w/ the list of all cookbooks we are
working on as a team and i want librarian to download them if they don't
exist already.

Perhaps this is a perversion of all things bundler but this is what I
want. It also would get much more complicated if we didn't have git repos w/
shared commit access.

Is this crazy or a good idea or both?

That's the piece that I mentioned earlier today. We'll make a better announcement once we've got the proper documentation and examples in place.

--
Jamie Winsor
@resetexistence

On Saturday, June 23, 2012 at 3:06 AM, AJ Christensen wrote:

ohai, just saw this:

GitHub - berkshelf/berkshelf: A Chef Cookbook manager

On 23 June 2012 20:45, Bryan Berry <bryan.berry@gmail.com (mailto:bryan.berry@gmail.com)> wrote:

Jay,

I don't quite understand your point. Are you are saying that Librarian-chef
isn't meant to meet the particular use case I have described and should be
looking for a separate tool?

On Fri, Jun 22, 2012 at 11:44 PM, Jay Feldblum <y_feldblum@yahoo.com (mailto:y_feldblum@yahoo.com)> wrote:

Bryan,

The various rigorous practices that developers have are often
well-supported by their tools.

The same practices are not as well-supported by the devops tools because
these tools are still being built and because the ideas and practices are
still coming across. Devops as a field is still under construction.

In the meantime: you mix and match, you pick your battles, and you do what
you have to do.

Cheers,
Jay

On Fri, Jun 22, 2012 at 5:28 PM, Bryan Berry <bryan.berry@gmail.com (mailto:bryan.berry@gmail.com)>
wrote:

alright, I have a pretty heterodox idea of how I would like to use
librarian-chef so that a team of infrastructure devs can work in sync

I am on a team of 3 infrastructure devs, i am the (relative) expert, the
other 2 guys are smart but n00bs

I want us to have one common Cheffile in our one common chef-repo

the cookbooks we develop independently each have their own git repo,
unfortunately, private ones for the most part

When I create a new git repository for application-foo, I want to add the
git repository link, branch name/tag/commit name, to Cheffile so that when
the other guys are working, they can easily pull in the cookbooks that I am
working on and vice versa

However, I dont want to do the git add . && git commit -am 'foo' && git push origin master && librarian-chef update dance when I am actively
working on a cookbook that is within an "active" cookbook.

I don't need librarian to resolve any dependencies for my active
cookbooks, I just want a common file w/ the list of all cookbooks we are
working on as a team and i want librarian to download them if they don't
exist already.

Perhaps this is a perversion of all things bundler but this is what I
want. It also would get much more complicated if we didn't have git repos w/
shared commit access.

Is this crazy or a good idea or both?

Bryan,

Not at all.

Librarian-Chef is meant to bring a particular practice of developers into
the devops fields. For full benefit, that practice depends on a number of
other tools and practices already being in place as well.

But the devops field is still under construction. While that remains so,
people will likely encounter some friction using Librarian-Chef when they
try to use it in the exact way that developers use Bundler, and will likely
need to do some hacks and workarounds. That's to be expected. As the devops
field progresses, and the tools and practices in the devops field mature,
using Librarian-Chef or any other bundler will become more natural.

Cheers,
Jay

On Sat, Jun 23, 2012 at 4:45 AM, Bryan Berry bryan.berry@gmail.com wrote:

Jay,

I don't quite understand your point. Are you are saying that
Librarian-chef isn't meant to meet the particular use case I have described
and should be looking for a separate tool?

On Fri, Jun 22, 2012 at 11:44 PM, Jay Feldblum y_feldblum@yahoo.comwrote:

Bryan,

The various rigorous practices that developers have are often
well-supported by their tools.

The same practices are not as well-supported by the devops tools because
these tools are still being built and because the ideas and practices are
still coming across. Devops as a field is still under construction.

In the meantime: you mix and match, you pick your battles, and you do
what you have to do.

Cheers,
Jay

On Fri, Jun 22, 2012 at 5:28 PM, Bryan Berry bryan.berry@gmail.comwrote:

alright, I have a pretty heterodox idea of how I would like to use
librarian-chef so that a team of infrastructure devs can work in sync

I am on a team of 3 infrastructure devs, i am the (relative) expert, the
other 2 guys are smart but n00bs

I want us to have one common Cheffile in our one common chef-repo

the cookbooks we develop independently each have their own git repo,
unfortunately, private ones for the most part

When I create a new git repository for application-foo, I want to add
the git repository link, branch name/tag/commit name, to Cheffile so that
when the other guys are working, they can easily pull in the cookbooks that
I am working on and vice versa

However, I dont want to do the git add . && git commit -am 'foo' && git push origin master && librarian-chef update dance when I am actively
working on a cookbook that is within an "active" cookbook.

I don't need librarian to resolve any dependencies for my active
cookbooks, I just want a common file w/ the list of all cookbooks we are
working on as a team and i want librarian to download them if they don't
exist already.

Perhaps this is a perversion of all things bundler but this is what I
want. It also would get much more complicated if we didn't have git repos
w/ shared commit access.

Is this crazy or a good idea or both?

I'm using librarian-chef and wouldn't know how else I should manage my
cookbook dependencies. It is great for resolving / downloading dependencies
from various sources (community site, git, local path) if you treat them as
"binaries" and I'm pretty happy with it.

But I'm also experiencing some friction when it comes to actively
developing on cookbooks which I want to upload to chef server during my
test / devlopment cycle. With librarian-chef I see only a few, all not
ideal, alternatives:

  • do the full git add . && git commit -am 'foo' && git push origin master && librarian-chef update cycle as mentioned by Bryan
  • do active development using a :path dependency, then when you have
    changes do the librarian-chef update cycle. When you are done testing
    do git add . && git commit -am 'foo' && git push origin master and switch
    to a :git dependency
  • do the changes directly in the $CHEF_REPO/cooksbooks directory
    (dangerous: don't do librarian-chef update / install or you will lose
    everything!!!), and when you are done testing replay the changes to your
    original cookbook source (might be suitable for tiny changes only)

@Jamie: can you give us a quick comparison on the differences of
librarian-chef and berkshelf? would berkshelf better support the scenario
above?

Cheers,
Torben

On Sat, Jun 23, 2012 at 12:13 PM, Jamie Winsor jamie@vialstudios.comwrote:

That's the piece that I mentioned earlier today. We'll make a better
announcement once we've got the proper documentation and examples in place.

--
Jamie Winsor
@resetexistence
reset (Jamie Stormbreaker) · GitHub

On Saturday, June 23, 2012 at 3:06 AM, AJ Christensen wrote:

ohai, just saw this:

GitHub - berkshelf/berkshelf: A Chef Cookbook manager

On 23 June 2012 20:45, Bryan Berry bryan.berry@gmail.com wrote:

Jay,

I don't quite understand your point. Are you are saying that
Librarian-chef
isn't meant to meet the particular use case I have described and should be
looking for a separate tool?

On Fri, Jun 22, 2012 at 11:44 PM, Jay Feldblum y_feldblum@yahoo.com
wrote:

Bryan,

The various rigorous practices that developers have are often
well-supported by their tools.

The same practices are not as well-supported by the devops tools because
these tools are still being built and because the ideas and practices are
still coming across. Devops as a field is still under construction.

In the meantime: you mix and match, you pick your battles, and you do what
you have to do.

Cheers,
Jay

On Fri, Jun 22, 2012 at 5:28 PM, Bryan Berry bryan.berry@gmail.com
wrote:

alright, I have a pretty heterodox idea of how I would like to use
librarian-chef so that a team of infrastructure devs can work in sync

I am on a team of 3 infrastructure devs, i am the (relative) expert, the
other 2 guys are smart but n00bs

I want us to have one common Cheffile in our one common chef-repo

the cookbooks we develop independently each have their own git repo,
unfortunately, private ones for the most part

When I create a new git repository for application-foo, I want to add the
git repository link, branch name/tag/commit name, to Cheffile so that when
the other guys are working, they can easily pull in the cookbooks that I am
working on and vice versa

However, I dont want to do the git add . && git commit -am 'foo' && git push origin master && librarian-chef update dance when I am actively
working on a cookbook that is within an "active" cookbook.

I don't need librarian to resolve any dependencies for my active
cookbooks, I just want a common file w/ the list of all cookbooks we are
working on as a team and i want librarian to download them if they don't
exist already.

Perhaps this is a perversion of all things bundler but this is what I
want. It also would get much more complicated if we didn't have git repos
w/
shared commit access.

Is this crazy or a good idea or both?

Absolutely, I'm finishing up the documentation today which will outline this in a better format.

Scenario 1: developing an application

Imagine you are working on an application called "pvpnet". Pvpnet contains a Berksfile and a Vagrantfile. The Berskfile contains a source with a path location to an application cookbook on disk (:path => "~/code/pvpnet-cookbook"). You run the command "knife berks install --shims". It installs the dependencies of the "pvpnet-cookbook" to a directory on disk at "~/.berkshelf" (by default) and creates a directory at cookbooks/ containing links to the cookbooks you just installed, including the one defined by the :path source.

In vagrant you have a "cookbooks" entry in your cookbooks_path which points to the shims directory made by Berkshelf. Now you are able to startup Vagrant and Vagrant will always read the changes you are making to the "pvpnet-cookbook" and it will pick up the dependencies of the "pvpnet-cookbook"

Scenario 2: developing a new cookbook

Imagine we aren't working on an application ourselves, but instead we're working on the "nginx" cookbook. The cookbook contains a Berksfile with an entry "metadata". This entry means to look at the current working directory to find a metadata file and use this directory as a cookbook. We also have a Vagrant file here pointing to "cookbooks" and, the same deal as the first time through, you run "knife berks install --shims" to create the cookbooks/ directory of shims. Now when you run vagrant provision you will be using the current contents of the cookbook you are developing.

Scenario 3: building your building your cookbooks

We use Jenkins at Riot, but this would go for any build server. Every cookbook lives in it's own repository. Every repository is hooked up to a Jenkins job that monitors changes. Every cookbook has a build script which does the following thing (you need to do this part).

  1. Runs a bootstrap (Installs dependent gems and software)
  2. Using Berkshelf, gather the cookbook's dependencies (knife berks install)
  3. Unit test the cookbook (if you have unit tests)
  4. Lint test the cookbook with food critic
  5. Bump the version number of the Cookbook
  6. Upload the cookbook and it's dependencies to your Chef server for later consumption via "knife berks upload"

This will setup your Cookbook with continuous delivery model to allow you to rapidly deliver incremental updates of a Cookbook to your organization.

Scenario 4: building an application

Every piece of software you are building should have a very similar life cycle to what I just outlined in Scenario 3. The difference is that you have a Berksfile containing a source for your application's cookbook. You can then gather it's dependencies and figure out what version of the cookbook this application code should be deployed with. Then you can mark your built artifact with metadata saying you want it deployed with X version of X cookbook and lock your environment to deploy the app code with that cookbook and that artifact version.

I can elaborate on Scenario 4 more, but it's less about Berkshelf at that point and more a blog post about how to do continuous delivery with Chef.

--
Jamie Winsor
@resetexistence

On Sunday, June 24, 2012 at 2:24 AM, Torben Knerr wrote:

I'm using librarian-chef and wouldn't know how else I should manage my cookbook dependencies. It is great for resolving / downloading dependencies from various sources (community site, git, local path) if you treat them as "binaries" and I'm pretty happy with it.

But I'm also experiencing some friction when it comes to actively developing on cookbooks which I want to upload to chef server during my test / devlopment cycle. With librarian-chef I see only a few, all not ideal, alternatives:

  • do the full git add . && git commit -am 'foo' && git push origin master && librarian-chef update cycle as mentioned by Bryan
  • do active development using a :path dependency, then when you have changes do the librarian-chef update cycle. When you are done testing do git add . && git commit -am 'foo' && git push origin master and switch to a :git dependency
  • do the changes directly in the $CHEF_REPO/cooksbooks directory (dangerous: don't do librarian-chef update / install or you will lose everything!!!), and when you are done testing replay the changes to your original cookbook source (might be suitable for tiny changes only)

@Jamie: can you give us a quick comparison on the differences of librarian-chef and berkshelf? would berkshelf better support the scenario above?

Cheers,
Torben

On Sat, Jun 23, 2012 at 12:13 PM, Jamie Winsor <jamie@vialstudios.com (mailto:jamie@vialstudios.com)> wrote:

That's the piece that I mentioned earlier today. We'll make a better announcement once we've got the proper documentation and examples in place.

--
Jamie Winsor
@resetexistence
reset (Jamie Stormbreaker) · GitHub

On Saturday, June 23, 2012 at 3:06 AM, AJ Christensen wrote:

ohai, just saw this:

GitHub - berkshelf/berkshelf: A Chef Cookbook manager

On 23 June 2012 20:45, Bryan Berry <bryan.berry@gmail.com (mailto:bryan.berry@gmail.com)> wrote:

Jay,

I don't quite understand your point. Are you are saying that Librarian-chef
isn't meant to meet the particular use case I have described and should be
looking for a separate tool?

On Fri, Jun 22, 2012 at 11:44 PM, Jay Feldblum <y_feldblum@yahoo.com (mailto:y_feldblum@yahoo.com)> wrote:

Bryan,

The various rigorous practices that developers have are often
well-supported by their tools.

The same practices are not as well-supported by the devops tools because
these tools are still being built and because the ideas and practices are
still coming across. Devops as a field is still under construction.

In the meantime: you mix and match, you pick your battles, and you do what
you have to do.

Cheers,
Jay

On Fri, Jun 22, 2012 at 5:28 PM, Bryan Berry <bryan.berry@gmail.com (mailto:bryan.berry@gmail.com)>
wrote:

alright, I have a pretty heterodox idea of how I would like to use
librarian-chef so that a team of infrastructure devs can work in sync

I am on a team of 3 infrastructure devs, i am the (relative) expert, the
other 2 guys are smart but n00bs

I want us to have one common Cheffile in our one common chef-repo

the cookbooks we develop independently each have their own git repo,
unfortunately, private ones for the most part

When I create a new git repository for application-foo, I want to add the
git repository link, branch name/tag/commit name, to Cheffile so that when
the other guys are working, they can easily pull in the cookbooks that I am
working on and vice versa

However, I dont want to do the git add . && git commit -am 'foo' && git push origin master && librarian-chef update dance when I am actively
working on a cookbook that is within an "active" cookbook.

I don't need librarian to resolve any dependencies for my active
cookbooks, I just want a common file w/ the list of all cookbooks we are
working on as a team and i want librarian to download them if they don't
exist already.

Perhaps this is a perversion of all things bundler but this is what I
want. It also would get much more complicated if we didn't have git repos w/
shared commit access.

Is this crazy or a good idea or both?

On Mon, Jun 25, 2012 at 5:49 AM, Jamie Winsor jamie@vialstudios.com wrote:

Absolutely, I'm finishing up the documentation today which will outline this
in a better format.

What Jamie Winsor outlines is very interesting and Berkshelf seems to
address a common use case.

I do wonder if this is not Chef-server centric.
Having setup Chef-server locally, via vagrant, in order to pursue
continuous deployment, I have since moved to use Chef-solo.
In this respect I think Librarian-Chef is still where-it-is-at wrt
cookbook deployment.

As Jay Feldblum notes, devops is still in a state of flux.
I've discussed some issues and possibilities with Jay in terms of the
production deployment of cookbooks, where they are obtained from and
how (protocol/conventions).
I can't say, from memory, that he has agreed, but the ability to
remove '.git' is nice and hopefully over time Librarian-Chef will
fulfill its potential.

As Jay notes the devops area is still very much decorated with
wet-paint notices (witness the opscode repo reorganization), and
developing/patching cookbooks is a universal need.
However, my view is that getting the production deployment nailed down
is the priority.
After that it will be possible to determine how best to handle
development/patched cookbooks.

For what it is worth right now I work on development cookbooks in a
separate local clone of my remote fork.
Push changes to my remote fork, and reference the remote fork in the Cheffile.

I think this is what is described by Bryan.
This is not the most convenient, but it has the substantial benefit
that my development environment, in terms of cookbooks/Cheffile, is
100% what my production environment is.
This is the reason I don't use :path in my Cheffile.

My 2c.

Scenario 1: developing an application

Imagine you are working on an application called "pvpnet". Pvpnet contains a
Berksfile and a Vagrantfile. The Berskfile contains a source with a path
location to an application cookbook on disk (:path =>
"~/code/pvpnet-cookbook"). You run the command "knife berks install
--shims". It installs the dependencies of the "pvpnet-cookbook" to a
directory on disk at "~/.berkshelf" (by default) and creates a directory at
cookbooks/ containing links to the cookbooks you just installed, including
the one defined by the :path source.

In vagrant you have a "cookbooks" entry in your cookbooks_path which points
to the shims directory made by Berkshelf. Now you are able to startup
Vagrant and Vagrant will always read the changes you are making to the
"pvpnet-cookbook" and it will pick up the dependencies of the
"pvpnet-cookbook"

Scenario 2: developing a new cookbook

Imagine we aren't working on an application ourselves, but instead we're
working on the "nginx" cookbook. The cookbook contains a Berksfile with an
entry "metadata". This entry means to look at the current working directory
to find a metadata file and use this directory as a cookbook. We also have a
Vagrant file here pointing to "cookbooks" and, the same deal as the first
time through, you run "knife berks install --shims" to create the cookbooks/
directory of shims. Now when you run vagrant provision you will be using the
current contents of the cookbook you are developing.

Scenario 3: building your building your cookbooks

We use Jenkins at Riot, but this would go for any build server. Every
cookbook lives in it's own repository. Every repository is hooked up to a
Jenkins job that monitors changes. Every cookbook has a build script which
does the following thing (you need to do this part).

  1. Runs a bootstrap (Installs dependent gems and software)
  2. Using Berkshelf, gather the cookbook's dependencies (knife berks install)
  3. Unit test the cookbook (if you have unit tests)
  4. Lint test the cookbook with food critic
  5. Bump the version number of the Cookbook
  6. Upload the cookbook and it's dependencies to your Chef server for later
    consumption via "knife berks upload"

This will setup your Cookbook with continuous delivery model to allow you to
rapidly deliver incremental updates of a Cookbook to your organization.

Scenario 4: building an application

Every piece of software you are building should have a very similar life
cycle to what I just outlined in Scenario 3. The difference is that you have
a Berksfile containing a source for your application's cookbook. You can
then gather it's dependencies and figure out what version of the cookbook
this application code should be deployed with. Then you can mark your built
artifact with metadata saying you want it deployed with X version of X
cookbook and lock your environment to deploy the app code with that cookbook
and that artifact version.

I can elaborate on Scenario 4 more, but it's less about Berkshelf at that
point and more a blog post about how to do continuous delivery with Chef.

--
Jamie Winsor
@resetexistence
reset (Jamie Stormbreaker) · GitHub

On Sunday, June 24, 2012 at 2:24 AM, Torben Knerr wrote:

I'm using librarian-chef and wouldn't know how else I should manage my
cookbook dependencies. It is great for resolving / downloading dependencies
from various sources (community site, git, local path) if you treat them as
"binaries" and I'm pretty happy with it.

But I'm also experiencing some friction when it comes to actively developing
on cookbooks which I want to upload to chef server during my test /
devlopment cycle. With librarian-chef I see only a few, all not ideal,
alternatives:

  • do the full git add . && git commit -am 'foo' && git push origin master && librarian-chef update cycle as mentioned by Bryan
  • do active development using a :path dependency, then when you have
    changes do the librarian-chef update cycle. When you are done testing
    do git add . && git commit -am 'foo' && git push origin master and switch
    to a :git dependency
  • do the changes directly in the $CHEF_REPO/cooksbooks directory
    (dangerous: don't do librarian-chef update / install or you will lose
    everything!!!), and when you are done testing replay the changes to your
    original cookbook source (might be suitable for tiny changes only)

@Jamie: can you give us a quick comparison on the differences of
librarian-chef and berkshelf? would berkshelf better support the scenario
above?

Cheers,
Torben

On Sat, Jun 23, 2012 at 12:13 PM, Jamie Winsor jamie@vialstudios.com
wrote:

That's the piece that I mentioned earlier today. We'll make a better
announcement once we've got the proper documentation and examples in place.

--
Jamie Winsor
@resetexistence
reset (Jamie Stormbreaker) · GitHub

On Saturday, June 23, 2012 at 3:06 AM, AJ Christensen wrote:

ohai, just saw this:

GitHub - berkshelf/berkshelf: A Chef Cookbook manager

On 23 June 2012 20:45, Bryan Berry bryan.berry@gmail.com wrote:

Jay,

I don't quite understand your point. Are you are saying that Librarian-chef
isn't meant to meet the particular use case I have described and should be
looking for a separate tool?

On Fri, Jun 22, 2012 at 11:44 PM, Jay Feldblum y_feldblum@yahoo.com wrote:

Bryan,

The various rigorous practices that developers have are often
well-supported by their tools.

The same practices are not as well-supported by the devops tools because
these tools are still being built and because the ideas and practices are
still coming across. Devops as a field is still under construction.

In the meantime: you mix and match, you pick your battles, and you do what
you have to do.

Cheers,
Jay

On Fri, Jun 22, 2012 at 5:28 PM, Bryan Berry bryan.berry@gmail.com
wrote:

alright, I have a pretty heterodox idea of how I would like to use
librarian-chef so that a team of infrastructure devs can work in sync

I am on a team of 3 infrastructure devs, i am the (relative) expert, the
other 2 guys are smart but n00bs

I want us to have one common Cheffile in our one common chef-repo

the cookbooks we develop independently each have their own git repo,
unfortunately, private ones for the most part

When I create a new git repository for application-foo, I want to add the
git repository link, branch name/tag/commit name, to Cheffile so that when
the other guys are working, they can easily pull in the cookbooks that I am
working on and vice versa

However, I dont want to do the git add . && git commit -am 'foo' && git push origin master && librarian-chef update dance when I am actively
working on a cookbook that is within an "active" cookbook.

I don't need librarian to resolve any dependencies for my active
cookbooks, I just want a common file w/ the list of all cookbooks we are
working on as a team and i want librarian to download them if they don't
exist already.

Perhaps this is a perversion of all things bundler but this is what I
want. It also would get much more complicated if we didn't have git repos w/
shared commit access.

Is this crazy or a good idea or both?

--
πόλλ' οἶδ ἀλώπηξ, ἀλλ' ἐχῖνος ἓν μέγα
[The fox knows many things, but the hedgehog knows one big thing.]
Archilochus, Greek poet (c. 680 BC – c. 645 BC)
http://hedgehogshiatus.com