Is it possible to version control Chef Repository when working from management workstation?


#1

Hi,

I work with Chef from workstation knife. I thought that I need to
clone Chef Repository, work with my local copy and then sync it with
the server. But it appears knife modifies that most things (like data
bags) directly on server through API not giving me a chance to record
the changes by committing them into version control system of my local
copy.

So basically if I want a version control over my configuration (Chef
Repository) I am limited to committing changes from server side only.
Did I miss something? How do people version control their
configuration?

It could help if knife had command to download the whole repo.

anatoly t.


#2

You commit before you run knife, or you run the knife commands as a
post-commit hook.

Best,
Adam

On Thu, Jul 5, 2012 at 5:51 PM, anatoly techtonik techtonik@gmail.com wrote:

Hi,

I work with Chef from workstation knife. I thought that I need to
clone Chef Repository, work with my local copy and then sync it with
the server. But it appears knife modifies that most things (like data
bags) directly on server through API not giving me a chance to record
the changes by committing them into version control system of my local
copy.

So basically if I want a version control over my configuration (Chef
Repository) I am limited to committing changes from server side only.
Did I miss something? How do people version control their
configuration?

It could help if knife had command to download the whole repo.

anatoly t.


Opscode, Inc.
Adam Jacob, Chief Customer Officer
T: (206) 619-7151 E: adam@opscode.com


#3

The real problem in your example is to get away from commands like

knife data bag edit
And
knife role edit

Use the chef-repo example and have roles and data bags in there. Edit the roles locally and then use standard source control options. When your ready to push to the chef server use the from file commands.

knife data bag from file users jmiller.json

knife role from file base.json

Hope that helps

On Jul 5, 2012, at 9:54 AM, Adam Jacob adam@opscode.com wrote:

You commit before you run knife, or you run the knife commands as a
post-commit hook.

Best,
Adam

On Thu, Jul 5, 2012 at 5:51 PM, anatoly techtonik techtonik@gmail.com wrote:

Hi,

I work with Chef from workstation knife. I thought that I need to
clone Chef Repository, work with my local copy and then sync it with
the server. But it appears knife modifies that most things (like data
bags) directly on server through API not giving me a chance to record
the changes by committing them into version control system of my local
copy.

So basically if I want a version control over my configuration (Chef
Repository) I am limited to committing changes from server side only.
Did I miss something? How do people version control their
configuration?

It could help if knife had command to download the whole repo.

anatoly t.


Opscode, Inc.
Adam Jacob, Chief Customer Officer
T: (206) 619-7151 E: adam@opscode.com


#4

On Thu, Jul 5, 2012 at 8:10 PM, Joshua Miller jassinpain@gmail.com wrote:

The real problem in your example is to get away from commands like

knife data bag edit
And
knife role edit

Use the chef-repo example and have roles and data bags in there. Edit the roles locally and then use standard source control options. When your ready to push to the chef server use the from file commands.

knife data bag from file users jmiller.json

knife role from file base.json

Hope that helps

Not really. I am trying to set a workflow and so far it is not very
satisfying as I hoped to manage my configuration as a source code. =)
I’ve made a cookbook with:
knife cookbook create newcook

It is created in my local cookbooks directory. I’ve committed it, then
uploaded to server.
Now I need to create data bag.
knife cookbook data bag create newbag

There are no files to commit. Neither before nor after. Your advice
basically mean that I manually need to keep layout in my local version
equal to the one the server maintains? That means I need to know the
layout and put .json file where it should belong on the server side.
knife data bag from file users ~/chef-repo/data-bags/jmiller.json

That’s not the version control I used to.

I’ve also noticed that knife doesn’t synchronize my local cookbooks
with remote. When I do
knife download newcook
it is not downloaded to my local cookbooks path - I have to specify
the path manually. Why?


#5

yes. they are not very tightly integrated.
For you the truth is git repo, but for chef its couch and for knife its
couch or local file system (depending upon the operation). As you see,
people try to automate any one of the step , or practice certain
methpds/workflows to use git as the master , and rest are based on that.

Chef is modular, if you want you can actually pull out the knife edit
options and enforce knife from file as the only way to push data, and
configure the system against a repo, where folks can push changes. The
only gotcha is the delay introduced .

On Thu, Jul 5, 2012 at 11:32 PM, anatoly techtonik techtonik@gmail.comwrote:

On Thu, Jul 5, 2012 at 8:10 PM, Joshua Miller jassinpain@gmail.com
wrote:

The real problem in your example is to get away from commands like

knife data bag edit
And
knife role edit

Use the chef-repo example and have roles and data bags in there. Edit
the roles locally and then use standard source control options. When your
ready to push to the chef server use the from file commands.

knife data bag from file users jmiller.json

knife role from file base.json

Hope that helps

Not really. I am trying to set a workflow and so far it is not very
satisfying as I hoped to manage my configuration as a source code. =)
I’ve made a cookbook with:
knife cookbook create newcook

It is created in my local cookbooks directory. I’ve committed it, then
uploaded to server.
Now I need to create data bag.
knife cookbook data bag create newbag

There are no files to commit. Neither before nor after. Your advice
basically mean that I manually need to keep layout in my local version
equal to the one the server maintains? That means I need to know the
layout and put .json file where it should belong on the server side.
knife data bag from file users ~/chef-repo/data-bags/jmiller.json

That’s not the version control I used to.

I’ve also noticed that knife doesn’t synchronize my local cookbooks
with remote. When I do
knife download newcook
it is not downloaded to my local cookbooks path - I have to specify
the path manually. Why?


#6

I see. In fact I don’t mind if knife executed commands on server and
had ability to sync (or download) Chef Repository. I could write a
wrapper script to around that, but at least I know the limitations,
which is huge advancement. Thanks for replies.

My ideal setup is like:

  1. Install HG on server side and make Chef Server automatically save
    Chef Repository filesystem tree there
  2. Every knife command triggers hg transaction(config change,
    pre-commit hook script to validate structure, save config to couch,
    commit with log message)
  3. Backup script pulls Chef Repository into some storage and sends me
    commit notifications about all activities

    anatoly t.

On Thu, Jul 5, 2012 at 9:11 PM, Ranjib Dey ranjibd@thoughtworks.com wrote:

yes. they are not very tightly integrated.
For you the truth is git repo, but for chef its couch and for knife its
couch or local file system (depending upon the operation). As you see,
people try to automate any one of the step , or practice certain
methpds/workflows to use git as the master , and rest are based on that.

Chef is modular, if you want you can actually pull out the knife edit
options and enforce knife from file as the only way to push data, and
configure the system against a repo, where folks can push changes. The only
gotcha is the delay introduced .

On Thu, Jul 5, 2012 at 11:32 PM, anatoly techtonik techtonik@gmail.com
wrote:

On Thu, Jul 5, 2012 at 8:10 PM, Joshua Miller jassinpain@gmail.com
wrote:

The real problem in your example is to get away from commands like

knife data bag edit
And
knife role edit

Use the chef-repo example and have roles and data bags in there. Edit
the roles locally and then use standard source control options. When your
ready to push to the chef server use the from file commands.

knife data bag from file users jmiller.json

knife role from file base.json

Hope that helps

Not really. I am trying to set a workflow and so far it is not very
satisfying as I hoped to manage my configuration as a source code. =)
I’ve made a cookbook with:
knife cookbook create newcook

It is created in my local cookbooks directory. I’ve committed it, then
uploaded to server.
Now I need to create data bag.
knife cookbook data bag create newbag

There are no files to commit. Neither before nor after. Your advice
basically mean that I manually need to keep layout in my local version
equal to the one the server maintains? That means I need to know the
layout and put .json file where it should belong on the server side.
knife data bag from file users ~/chef-repo/data-bags/jmiller.json

That’s not the version control I used to.

I’ve also noticed that knife doesn’t synchronize my local cookbooks
with remote. When I do
knife download newcook
it is not downloaded to my local cookbooks path - I have to specify
the path manually. Why?


#7

I believe your wanting something very similar to the git_to_chef script/cookbook I publicized last week: https://github.com/cjs226/git_to_chef

cjs

On Jul 5, 2012, at 1:02 PM, anatoly techtonik wrote:

On Thu, Jul 5, 2012 at 8:10 PM, Joshua Miller jassinpain@gmail.com wrote:

The real problem in your example is to get away from commands like

knife data bag edit
And
knife role edit

Use the chef-repo example and have roles and data bags in there. Edit the roles locally and then use standard source control options. When your ready to push to the chef server use the from file commands.

knife data bag from file users jmiller.json

knife role from file base.json

Hope that helps

Not really. I am trying to set a workflow and so far it is not very
satisfying as I hoped to manage my configuration as a source code. =)
I’ve made a cookbook with:
knife cookbook create newcook

It is created in my local cookbooks directory. I’ve committed it, then
uploaded to server.
Now I need to create data bag.
knife cookbook data bag create newbag

There are no files to commit. Neither before nor after. Your advice
basically mean that I manually need to keep layout in my local version
equal to the one the server maintains? That means I need to know the
layout and put .json file where it should belong on the server side.
knife data bag from file users ~/chef-repo/data-bags/jmiller.json

That’s not the version control I used to.

I’ve also noticed that knife doesn’t synchronize my local cookbooks
with remote. When I do
knife download newcook
it is not downloaded to my local cookbooks path - I have to specify
the path manually. Why?


#8

-----Original Message-----
From: anatoly techtonik [mailto:techtonik@gmail.com]
Sent: 05 July 2012 19:38
To: chef@lists.opscode.com
Subject: [chef] Re: Re: Re: Re: Re: Is it possible to version control
Chef Repository when working from management workstation?

I see. In fact I don’t mind if knife executed commands on server and
had ability to sync (or download) Chef Repository. I could write a
wrapper script to around that, but at least I know the limitations,
which is huge advancement. Thanks for replies.

If you want to download from chef to a structure that can be placed into source control I found the knife download plugin from knife-essentials to be the best so far.

It relies on your cookbook_path being set correctly. I use the following in my knife.rb for setting my cookbook path so that it is always correct for whatever directory I am in on the local filesystem.

Dynamically derive our cookbook directory from where we are in the filesystem

base = Dir.pwd
begin
cb_path = nil
Dir.foreach(base) do |dir|
if dir.eql? "cookbooks"
cb_path = "#{base}/#{dir}"
break
end
end
base = File.dirname(base)
end while base != “/” and cb_path.nil?
if cb_path.nil?
cb_path = "~/cookbooks"
end

Set the cookbook path

puts “Setting cookbook_path to #{cb_path}” if debug
cookbook_path cb_path if not cb_path.nil? and not cb_path.empty?

My ideal setup is like:

  1. Install HG on server side and make Chef Server automatically save
    Chef Repository filesystem tree there
  2. Every knife command triggers hg transaction(config change,
    pre-commit hook script to validate structure, save config to couch,
    commit with log message)
  3. Backup script pulls Chef Repository into some storage and sends me
    commit notifications about all activities

    anatoly t.

On Thu, Jul 5, 2012 at 9:11 PM, Ranjib Dey ranjibd@thoughtworks.com
wrote:

yes. they are not very tightly integrated.
For you the truth is git repo, but for chef its couch and for knife
its
couch or local file system (depending upon the operation). As you see,
people try to automate any one of the step , or practice certain
methpds/workflows to use git as the master , and rest are based on
that.

Chef is modular, if you want you can actually pull out the knife edit
options and enforce knife from file as the only way to push data, and
configure the system against a repo, where folks can push changes.
The only
gotcha is the delay introduced .

On Thu, Jul 5, 2012 at 11:32 PM, anatoly techtonik
techtonik@gmail.com
wrote:

On Thu, Jul 5, 2012 at 8:10 PM, Joshua Miller jassinpain@gmail.com
wrote:

The real problem in your example is to get away from commands like

knife data bag edit
And
knife role edit

Use the chef-repo example and have roles and data bags in there.
Edit

the roles locally and then use standard source control options.
When your

ready to push to the chef server use the from file commands.

knife data bag from file users jmiller.json

knife role from file base.json

Hope that helps

Not really. I am trying to set a workflow and so far it is not very
satisfying as I hoped to manage my configuration as a source code. =)
I’ve made a cookbook with:
knife cookbook create newcook

It is created in my local cookbooks directory. I’ve committed it,
then

uploaded to server.
Now I need to create data bag.
knife cookbook data bag create newbag

There are no files to commit. Neither before nor after. Your advice
basically mean that I manually need to keep layout in my local
version

equal to the one the server maintains? That means I need to know the
layout and put .json file where it should belong on the server side.
knife data bag from file users ~/chef-repo/data-bags/jmiller.json

That’s not the version control I used to.

I’ve also noticed that knife doesn’t synchronize my local cookbooks
with remote. When I do
knife download newcook
it is not downloaded to my local cookbooks path - I have to specify
the path manually. Why?

Thanks,
Jon-Paul Sullivan :slight_smile: Cloud Services - @hpcloud

Postal Address: Hewlett-Packard Galway Limited, Ballybrit Business Park, Galway.
Registered Office: Hewlett-Packard Galway Limited, 63-74 Sir John Rogerson’s Quay, Dublin 2.
Registered Number: 361933

The contents of this message and any attachments to it are confidential and may be legally privileged. If you have received this message in error you should delete it from your system immediately and advise the sender.

To any recipient of this message within HP, unless otherwise stated, you should consider this message and attachments as “HP CONFIDENTIAL”.