Chef repository usage

Morgan_Blackthorne · July 23, 2012, 4:54am

Still getting up to speed with chef. I know that most people suggest
putting the Chef configuration into a VCS like SVN/Git/etc, but what I’m
not clear about is what files should be put into the repo and uploaded (say
with shef or knife) and what should only exist on the chef server. For day
to day operations, do you suggest using shef to edit and test
configurations, and then download the files into the VCS repo to archive,
or should everything be done through the VCS repo and pushed to the chef
server via knife for testing?

Is there a knife command that will take a full (properly formatted)
directory structure and upload it to chef in one go, or do you have to
upload everything individually?

Thanks for the help.

~~ StormeRider ~~

“With hearts immortal, we stand before our lives
A soul against oblivion, forever asking why
This upside down symphony in a paradox called life
Our hearts immortal
What you give to love, will never die”

AJ_Christensen · July 23, 2012, 5:26am

I'd suggest forking the opscode/chef-repo repository and building up
from there. Use knife to upload the contents of the chef-repo to the
chef-server (hosted chef or otherwise), or to drive an chef-solo
configuration (maybe with Vagrant)

Some tools to investigate that may assist:

Vagrant - http://vagrantup.com/
Librarian - GitHub - applicationsonline/librarian: Librarian - A Framework for Bundlers. Librarian-Chef is at: https://github.com/applicationsonline/librarian-chef.
Berkshelf - GitHub - berkshelf/berkshelf: A Chef Cookbook manager
guard-knife - GitHub - nistude/guard-knife: Guard for Chef using knife to upload files
knife-table - GitHub - heavywater/knife-table: Knife workflow plugin
knife-spork - GitHub - jonlives/knife-spork: A workflow plugin to help many devs work with cookbooks and environments at once

Cheers,

--AJ

On 23 July 2012 16:54, Morgan Blackthorne stormerider@gmail.com wrote:

Still getting up to speed with chef. I know that most people suggest putting
the Chef configuration into a VCS like SVN/Git/etc, but what I'm not clear
about is what files should be put into the repo and uploaded (say with shef
or knife) and what should only exist on the chef server. For day to day
operations, do you suggest using shef to edit and test configurations, and
then download the files into the VCS repo to archive, or should everything
be done through the VCS repo and pushed to the chef server via knife for
testing?

Is there a knife command that will take a full (properly formatted)
directory structure and upload it to chef in one go, or do you have to
upload everything individually?

Thanks for the help.

~~ StormeRider ~~

"With hearts immortal, we stand before our lives
A soul against oblivion, forever asking why
This upside down symphony in a paradox called life
Our hearts immortal
What you give to love, will never die"

Mike_Adolphs · July 23, 2012, 6:16am

The chef-repo AJ mentioned comes with some predefined rake tasks. 'rake install' does this all-you-can-upload thingy for your chef server.

Daily workflows are quite often discussed on the mailing list. For cookbook management some use berkshelf, others librarian-chef. Some use knife-guard for automatic updates on the chef-server, some don't. It's mostly about what fits best to your existing workflows and/or what you're familiar with.

For a start stick to the chef-repo layout and use the predefined rake tasks.

Regards,
Mike

Sent from my iPhone.

Am 23.07.2012 um 07:26 schrieb AJ Christensen aj@junglist.gen.nz:

I'd suggest forking the opscode/chef-repo repository and building up
from there. Use knife to upload the contents of the chef-repo to the
chef-server (hosted chef or otherwise), or to drive an chef-solo
configuration (maybe with Vagrant)

Some tools to investigate that may assist:

Vagrant - http://vagrantup.com/

Librarian - GitHub - applicationsonline/librarian: Librarian - A Framework for Bundlers. Librarian-Chef is at: https://github.com/applicationsonline/librarian-chef.

Berkshelf - GitHub - berkshelf/berkshelf: A Chef Cookbook manager

guard-knife - GitHub - nistude/guard-knife: Guard for Chef using knife to upload files

knife-table - GitHub - heavywater/knife-table: Knife workflow plugin

knife-spork - GitHub - jonlives/knife-spork: A workflow plugin to help many devs work with cookbooks and environments at once

Cheers,

--AJ

On 23 July 2012 16:54, Morgan Blackthorne stormerider@gmail.com wrote:

Still getting up to speed with chef. I know that most people suggest putting
the Chef configuration into a VCS like SVN/Git/etc, but what I'm not clear
about is what files should be put into the repo and uploaded (say with shef
or knife) and what should only exist on the chef server. For day to day
operations, do you suggest using shef to edit and test configurations, and
then download the files into the VCS repo to archive, or should everything
be done through the VCS repo and pushed to the chef server via knife for
testing?

Is there a knife command that will take a full (properly formatted)
directory structure and upload it to chef in one go, or do you have to
upload everything individually?

Thanks for the help.

~~ StormeRider ~~

"With hearts immortal, we stand before our lives
A soul against oblivion, forever asking why
This upside down symphony in a paradox called life
Our hearts immortal
What you give to love, will never die"

Sullivan_Jon_Paul · July 23, 2012, 10:32am

I really like knife-essentials for uploading to a chef server. It gives you the "knife upload" command that just pushes up what is different in your VCS.

From: Morgan Blackthorne [mailto:stormerider@gmail.com]
Sent: 23 July 2012 05:54
To: chef@lists.opscode.com
Subject: [chef] Chef repository usage...

Still getting up to speed with chef. I know that most people suggest putting the Chef configuration into a VCS like SVN/Git/etc, but what I'm not clear about is what files should be put into the repo and uploaded (say with shef or knife) and what should only exist on the chef server. For day to day operations, do you suggest using shef to edit and test configurations, and then download the files into the VCS repo to archive, or should everything be done through the VCS repo and pushed to the chef server via knife for testing?

Is there a knife command that will take a full (properly formatted) directory structure and upload it to chef in one go, or do you have to upload everything individually?

Thanks for the help.

~~ StormeRider ~~

"With hearts immortal, we stand before our lives
A soul against oblivion, forever asking why
This upside down symphony in a paradox called life
Our hearts immortal
What you give to love, will never die"

Thanks,
Jon-Paul Sullivan Cloud Services - @hpcloud

Postal Address: Hewlett-Packard Galway Limited, Ballybrit Business Park, Galway.
Registered Office: Hewlett-Packard Galway Limited, 63-74 Sir John Rogerson's Quay, Dublin 2.
Registered Number: 361933

The contents of this message and any attachments to it are confidential and may be legally privileged. If you have received this message in error you should delete it from your system immediately and advise the sender.

To any recipient of this message within HP, unless otherwise stated, you should consider this message and attachments as "HP CONFIDENTIAL".

Bryan_McLellan · July 23, 2012, 3:58pm

On Mon, Jul 23, 2012 at 6:32 AM, Sullivan, Jon Paul
JonPaul.Sullivan@hp.com wrote:

I really like knife-essentials for uploading to a chef server. It gives you
the “knife upload” command that just pushes up what is different in your
VCS.

GitHub - jkeiser/knife-essentials: Knife commands to treat the Chef server like a filesystem

And of course, this can be installed with Rubygems:

$ gem install knife-essentials

Bryan

Morgan_Blackthorne · July 23, 2012, 7:14pm

Thanks all.

I looked at the repo that had been set up by my cosysadmin (who is now at a
new company) and I see that there is a Rakefile and such configured. I went
and installed rake, confirmed it was pointing to the ruby1.9 version, and
tried doing rake install. I get back this error, which Googling did me no
favors for:

morgan@rain:~/svn/OPS/chef-repo$ rake install --trace
rake aborted!
cannot load such file -- mixlib/authentication/signedheaderauth
/usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in require' /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in require'
/usr/lib/ruby/vendor_ruby/chef/rest/auth_credentials.rb:25:in <top (required)>' /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in require'
/usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in require' /usr/lib/ruby/vendor_ruby/chef/rest.rb:28:in <top (required)>'
/usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in require' /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in require'
/usr/lib/ruby/vendor_ruby/chef/search/query.rb:21:in <top (required)>' /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in require'
/usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in require' /usr/lib/ruby/vendor_ruby/chef/mixin/language.rb:19:in <top (required)>'
/usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in require' /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in require'
/usr/lib/ruby/vendor_ruby/chef/resource.rb:22:in <top (required)>' /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in require'
/usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in require' /usr/lib/ruby/vendor_ruby/chef/resource/file.rb:20:in <top (required)>'
/usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in require' /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in require'
/usr/lib/ruby/vendor_ruby/chef/provider/file.rb:21:in <top (required)>' /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in require'
/usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in require' /usr/lib/ruby/vendor_ruby/chef/provider/cookbook_file.rb:20:in <top
(required)>'
/usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in require' /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in require'
/usr/lib/ruby/vendor_ruby/chef/providers.rb:20:in <top (required)>' /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in require'
/usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in require' /usr/lib/ruby/vendor_ruby/chef.rb:25:in <top (required)>'
/usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in require' /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in require'
/home/morgan/svn/OPS/chef-repo/Rakefile:22:in <top (required)>' /usr/lib/ruby/vendor_ruby/rake/rake_module.rb:25:in load'
/usr/lib/ruby/vendor_ruby/rake/rake_module.rb:25:in load_rakefile' /usr/lib/ruby/vendor_ruby/rake/application.rb:501:in raw_load_rakefile'
/usr/lib/ruby/vendor_ruby/rake/application.rb:82:in block in load_rakefile' /usr/lib/ruby/vendor_ruby/rake/application.rb:133:in standard_exception_handling'
/usr/lib/ruby/vendor_ruby/rake/application.rb:81:in load_rakefile' /usr/lib/ruby/vendor_ruby/rake/application.rb:65:in block in run'
/usr/lib/ruby/vendor_ruby/rake/application.rb:133:in
standard_exception_handling' /usr/lib/ruby/vendor_ruby/rake/application.rb:63:in run'
/usr/bin/rake:27:in `'
morgan@rain:~/svn/OPS/chef-repo$

I'm going to try to clone the chef repo mentioned in the first part of this
thread and switch to using my Hosted Chef sandbox (knife-block is very
handy for testing like this) and seeing if the error persists, but I don't
want to make a lot of changes to the existing repo until I know why it's
set up the way it's set up.

~~ StormeRider ~~

"With hearts immortal, we stand before our lives
A soul against oblivion, forever asking why
This upside down symphony in a paradox called life
Our hearts immortal
What you give to love, will never die"

On Mon, Jul 23, 2012 at 8:58 AM, Bryan McLellan btm@loftninjas.org wrote:

On Mon, Jul 23, 2012 at 6:32 AM, Sullivan, Jon Paul
JonPaul.Sullivan@hp.com wrote:

I really like knife-essentials for uploading to a chef server. It gives
you
the “knife upload” command that just pushes up what is different in your
VCS.

GitHub - jkeiser/knife-essentials: Knife commands to treat the Chef server like a filesystem

And of course, this can be installed with Rubygems:

$ gem install knife-essentials

Bryan

Bryan_McLellan · July 23, 2012, 7:26pm

On Mon, Jul 23, 2012 at 3:14 PM, Morgan Blackthorne
stormerider@gmail.com wrote:

cannot load such file -- mixlib/authentication/signedheaderauth

Do you have the mixlib-authentication gem installed?

Bryan

Morgan_Blackthorne · July 23, 2012, 7:40pm

I did not. Now it's asking for ohai. I set this Ubuntu 12.04 workstation up
using the opscode deb repositories, does that not include the necessary
gems?

~~ StormeRider ~~

"With hearts immortal, we stand before our lives
A soul against oblivion, forever asking why
This upside down symphony in a paradox called life
Our hearts immortal
What you give to love, will never die"

On Mon, Jul 23, 2012 at 12:26 PM, Bryan McLellan btm@loftninjas.org wrote:

On Mon, Jul 23, 2012 at 3:14 PM, Morgan Blackthorne
stormerider@gmail.com wrote:

cannot load such file -- mixlib/authentication/signedheaderauth

Do you have the mixlib-authentication gem installed?

Bryan

Bryan_McLellan · July 23, 2012, 8:03pm

On Mon, Jul 23, 2012 at 3:40 PM, Morgan Blackthorne
stormerider@gmail.com wrote:

I did not. Now it's asking for ohai. I set this Ubuntu 12.04 workstation up
using the opscode deb repositories, does that not include the necessary
gems?

We're getting a bit off track here, my bad.

You should use 'knife cookbook upload -a' rather than 'rake install'
to copy all of your cookbooks to the server.

In the long long long ago, rake install would physically copy your
cookbooks to the correct place on the server. We quickly added an API
to replace that, but haven't been so quick to get rid of all the old
rake tasks.

http://wiki.opscode.com/display/chef/Managing+Cookbooks+With+Knife#ManagingCookbooksWithKnife-upload

http://tickets.opscode.com/browse/CHEF-2353

Bryan

Torben_Knerr · July 24, 2012, 5:56am

On Mon, Jul 23, 2012 at 10:03 PM, Bryan McLellan btm@loftninjas.org wrote:

On Mon, Jul 23, 2012 at 3:40 PM, Morgan Blackthorne
stormerider@gmail.com wrote:

I did not. Now it's asking for ohai. I set this Ubuntu 12.04 workstation up
using the opscode deb repositories, does that not include the necessary
gems?

We're getting a bit off track here, my bad.

You should use 'knife cookbook upload -a' rather than 'rake install'
to copy all of your cookbooks to the server.

In the long long long ago, rake install would physically copy your
cookbooks to the correct place on the server. We quickly added an API
to replace that, but haven't been so quick to get rid of all the old
rake tasks.

http://wiki.opscode.com/display/chef/Managing+Cookbooks+With+Knife#ManagingCookbooksWithKnife-upload

http://tickets.opscode.com/browse/CHEF-2353

Bryan

@Bryan: I thought that rake install would not only upload all
cookbooks but also all the other stuff in the chef repo (databags,
roles, environments) via the REST API...?

@Morgan: is there a Gemfile? If so, then gem install bundler && bundle install will install all the gems listed in the Gemfile along
with their dependencies. You then may want to run bundle exec rake install instead of just rake install, but this is only to make sure
that Ruby sees the exact version of the gem as defined in the
Gemfile.lock (e.g. if you have multiple versions of the same gem
installed)

Cheers,
Torben

Joshua_Timberman · July 24, 2012, 6:52am

Ohai,

On Jul 23, 2012, at 11:56 PM, Torben Knerr wrote:

@Bryan: I thought that rake install would not only upload all
cookbooks but also all the other stuff in the chef repo (databags,
roles, environments) via the REST API...?

There are rake tasks that basically call knife to upload things to the Chef Server, but for the most part, several of the rake tasks are pretty much considered deprecated by the knife commands that replace them, as Bryan said we just haven't removed or otherwise done anything with the rake tasks.

rake databag:create[databag]
rake databag:create_item[databag,item]
rake databag:upload[databag]
rake databag:upload_all

Replaced with "knife data bag from file" or "knife data bag from file -a" (Chef 0.10.10+, via CHEF-1912)

rake default
rake test_cookbook[cookbook]
rake test_cookbooks

Replaced with "knife cookbook test" or "knife cookbook test -a".

rake install

No single replacement, other than the various knife commands.

rake metadata

Replaced with "knife cookbook metadata".

rake new_cookbook

Replaced with "knife cookbook create".

rake role[role_name]
rake roles

Replaced with "knife role from file ROLE1.rb ROLE2.rb ROLE3.rb ...".

rake upload_cookbook[cookbook]
rake upload_cookbooks

Replaced with "knife cookbook upload" or "knife cookbook upload -a".

The knife-essentials gem also provides overlapping functionality. As mentioned, install the gem and check it out and see if you find it useful.

--
Opscode, Inc
Joshua Timberman, Technical Program Manager
IRC, Skype, Twitter, Github: jtimberman

Topic		Replies	Views
Knife - upload cookbook from local repo to chef-server-webui doesn't work Chef Infra (archive)	2	542	February 18, 2011
Is it possible to version control Chef Repository when working from management workstation? Chef Infra (archive)	7	696	July 11, 2012
Chef repo question Chef Infra (archive)	9	971	December 5, 2016
Better workflow for actively-developed cookbooks Chef Infra (archive)	19	461	June 25, 2012
Put Chef Repo into Private Git Chef Infra (archive)	3	676	September 21, 2015

Chef repository usage

Related topics