Chef-server 11 migration

Chef Infra (archive)
1

Folks,

I want to try chef-server 11. In order to do this I installed
chef-server 11 and want to migrate some clients to it preserving
certificates so that I could switch those clients back to old
chef-server. I couldn’t find any guidelines on migration and since
server 11 uses postgress instead of couchdb I expect migration should
have some peculiarities. May I ask you to share knowledge on this kind
of migration?

Thanks,
Kirill.

2

Wow, I hadn't caught that before... is 11 Postgres only, or are there other
DB options?

--
~~ StormeRider ~~

"Every world needs its heroes [...] They inspire us to be better than we
are. And they protect from the darkness that's just around the corner."

(from Smallville Season 6x1: "Zod")

On why I hate the phrase "that's so lame"... http://bit.ly/Ps3uSS

On Fri, Jan 4, 2013 at 5:12 PM, Kirill Timofeev kvt@hulu.com wrote:

Folks,

I want to try chef-server 11. In order to do this I installed chef-server
11 and want to migrate some clients to it preserving certificates so that I
could switch those clients back to old chef-server. I couldn't find any
guidelines on migration and since server 11 uses postgress instead of
couchdb I expect migration should have some peculiarities. May I ask you to
share knowledge on this kind of migration?

Thanks,
Kirill.

3

They've been saying MySQL or postgresql will both be supported.. my group
has been all-in MySQL, so I'm hoping that will still be the case.
On Jan 4, 2013 10:23 PM, "Morgan Blackthorne" stormerider@gmail.com wrote:

Wow, I hadn't caught that before... is 11 Postgres only, or are there
other DB options?

--
~~ StormeRider ~~

"Every world needs its heroes [...] They inspire us to be better than we
are. And they protect from the darkness that's just around the corner."

(from Smallville Season 6x1: "Zod")

On why I hate the phrase "that's so lame"... http://bit.ly/Ps3uSS

On Fri, Jan 4, 2013 at 5:12 PM, Kirill Timofeev kvt@hulu.com wrote:

Folks,

I want to try chef-server 11. In order to do this I installed chef-server
11 and want to migrate some clients to it preserving certificates so that I
could switch those clients back to old chef-server. I couldn't find any
guidelines on migration and since server 11 uses postgress instead of
couchdb I expect migration should have some peculiarities. May I ask you to
share knowledge on this kind of migration?

Thanks,
Kirill.

4

Hi there,

On Jan 5, 2013, at 5:50 AM, Jesse Campbell wrote:

They've been saying MySQL or postgresql will both be supported.. my group has been all-in MySQL, so I'm hoping that will still be the case.

The Chef 11 Server will ship with a Postgresql db and schema. We are not planning to support other database engines.

Initially we were targeting support for pg and mysql, but decided to concentrate efforts on a single db engine. The primary motivation being that we can spend time improving the server in terms of features, performance, and reliability rather than on cross db compatibility. It's a compromise. And that is to say, it is the sort of decision guaranteed not to make everyone happy.

One thing to be aware of is that with Chef 11 we are moving to an all-dependencies-included package model for the server (dubbed "omnibus"). So, for example, you will be able to install a complete Chef Server like:

dpkg -i chef-server.deb
sudo chef-server-ctl reconfigure

If you are a MySQL shop, you won't have to separately install postgresql.

On Fri, Jan 4, 2013 at 5:12 PM, Kirill Timofeev kvt@hulu.com wrote:
I want to try chef-server 11. In order to do this I installed chef-server 11 and want to migrate some clients to it preserving certificates so that I could switch those clients back to old chef-server. I couldn't find any guidelines on migration and since server 11 uses postgress instead of couchdb I expect migration should have some peculiarities. May I ask you to share knowledge on this kind of migration?

Thanks for giving the alpha builds a try! We don't yet have official migration tools. But I think I can help you with a couple of knife exec type scripts for experimenting with.

The only real trick is to know that in the Chef 11 server, you can set the public_key of a client when you create the client as an alternative to having the client generate a key pair for you.

On your existing Chef10 server, put the following in download.knife and run it as an admin client like knife exec download.knife.

disable JSON inflation

JSON.create_id = "no_thanks"

out_dir = "BACKUP"
Dir.mkdir(out_dir)

backup nodes

Dir.mkdir(File.join(out_dir, "nodes"))
node_names = api.get("nodes").keys
node_names.each do |name|
the_node = api.get("nodes/#{name}")
open(File.join(out_dir, "nodes", name), "w") do |f|
f.write(the_node.to_json)
end
end

backup clients for nodes

Dir.mkdir(File.join(out_dir, "clients"))
node_names.each do |name|
client = api.get("clients/#{name}")
open(File.join(out_dir, "clients", name), "w") do |f|
f.write(client.to_json)
end
end

Then in your Chef11 test environment, you can use the following to create matching clients and nodes. The rest of the data should be pretty easy to replicate (knife cookbook upload -a, etc)

disable JSON inflation

JSON.create_id = "no_thanks"

out_dir = "BACKUP"

load clients

Dir["#{out_dir}/clients/*"].each do |path|
client = JSON.parse(IO.read(path))
api.post("clients", {
:name => client['name'],
:public_key => client['public_key']
})
end

load nodes

Dir["#{out_dir}/nodes/*"].each do |path|
the_node = JSON.parse(IO.read(path))
api.post("nodes", the_node)
end

Hope that helps. Please report any problems you encounter with Chef11 to the list or file as bugs in the CHEF public bug tracker.

  • seth

--
Seth Falcon | Opscode

5

Thanks a lot for the information Seth!

On 01/06/2013 08:56 PM, Seth Falcon wrote:

Hi there,

On Jan 5, 2013, at 5:50 AM, Jesse Campbell wrote:

They've been saying MySQL or postgresql will both be supported.. my group has been all-in MySQL, so I'm hoping that will still be the case.
The Chef 11 Server will ship with a Postgresql db and schema. We are not planning to support other database engines.

Initially we were targeting support for pg and mysql, but decided to concentrate efforts on a single db engine. The primary motivation being that we can spend time improving the server in terms of features, performance, and reliability rather than on cross db compatibility. It's a compromise. And that is to say, it is the sort of decision guaranteed not to make everyone happy.

One thing to be aware of is that with Chef 11 we are moving to an all-dependencies-included package model for the server (dubbed "omnibus"). So, for example, you will be able to install a complete Chef Server like:

 dpkg -i chef-server.deb
 sudo chef-server-ctl reconfigure

If you are a MySQL shop, you won't have to separately install postgresql.

On Fri, Jan 4, 2013 at 5:12 PM, Kirill Timofeev kvt@hulu.com wrote:
I want to try chef-server 11. In order to do this I installed chef-server 11 and want to migrate some clients to it preserving certificates so that I could switch those clients back to old chef-server. I couldn't find any guidelines on migration and since server 11 uses postgress instead of couchdb I expect migration should have some peculiarities. May I ask you to share knowledge on this kind of migration?
Thanks for giving the alpha builds a try! We don't yet have official migration tools. But I think I can help you with a couple of knife exec type scripts for experimenting with.

The only real trick is to know that in the Chef 11 server, you can set the public_key of a client when you create the client as an alternative to having the client generate a key pair for you.

On your existing Chef10 server, put the following in download.knife and run it as an admin client like knife exec download.knife.

disable JSON inflation

JSON.create_id = "no_thanks"

out_dir = "BACKUP"
Dir.mkdir(out_dir)

backup nodes

Dir.mkdir(File.join(out_dir, "nodes"))
node_names = api.get("nodes").keys
node_names.each do |name|
the_node = api.get("nodes/#{name}")
open(File.join(out_dir, "nodes", name), "w") do |f|
f.write(the_node.to_json)
end
end

backup clients for nodes

Dir.mkdir(File.join(out_dir, "clients"))
node_names.each do |name|
client = api.get("clients/#{name}")
open(File.join(out_dir, "clients", name), "w") do |f|
f.write(client.to_json)
end
end

Then in your Chef11 test environment, you can use the following to create matching clients and nodes. The rest of the data should be pretty easy to replicate (knife cookbook upload -a, etc)

disable JSON inflation

JSON.create_id = "no_thanks"

out_dir = "BACKUP"

load clients

Dir["#{out_dir}/clients/*"].each do |path|
client = JSON.parse(IO.read(path))
api.post("clients", {
:name => client['name'],
:public_key => client['public_key']
})
end

load nodes

Dir["#{out_dir}/nodes/*"].each do |path|
the_node = JSON.parse(IO.read(path))
api.post("nodes", the_node)
end

Hope that helps. Please report any problems you encounter with Chef11 to the list or file as bugs in the CHEF public bug tracker.

  • seth

--
Seth Falcon | Opscode

6

IIRC, Chef 11 also includes a change in the encrypted data bag format.
Since nobody (outside of Opscode?) has tried to migrate from Chef 10.x to
Chef 11, I don't think anybody really knows what that process will look
like yet, but I took a stab at a knife data bag conversion plugin when this
was first mentioned. Here's the code:

It might help with some of your other migration tasks, as well... to form a
basis for those tasks, at the very least.

Good luck and let us all know how it goes!

On Mon, Jan 7, 2013 at 10:53 AM, Kirill Timofeev kvt@hulu.com wrote:

Thanks a lot for the information Seth!

On 01/06/2013 08:56 PM, Seth Falcon wrote:

Hi there,

On Jan 5, 2013, at 5:50 AM, Jesse Campbell wrote:

They've been saying MySQL or postgresql will both be supported.. my
group has been all-in MySQL, so I'm hoping that will still be the case.

The Chef 11 Server will ship with a Postgresql db and schema. We are not
planning to support other database engines.

Initially we were targeting support for pg and mysql, but decided to
concentrate efforts on a single db engine. The primary motivation being
that we can spend time improving the server in terms of features,
performance, and reliability rather than on cross db compatibility. It's a
compromise. And that is to say, it is the sort of decision guaranteed not
to make everyone happy.

One thing to be aware of is that with Chef 11 we are moving to an
all-dependencies-included package model for the server (dubbed "omnibus").
So, for example, you will be able to install a complete Chef Server like:

 dpkg -i chef-server.deb
 sudo chef-server-ctl reconfigure

If you are a MySQL shop, you won't have to separately install postgresql.

On Fri, Jan 4, 2013 at 5:12 PM, Kirill Timofeev kvt@hulu.com wrote:

I want to try chef-server 11. In order to do this I installed
chef-server 11 and want to migrate some clients to it preserving
certificates so that I could switch those clients back to old chef-server.
I couldn't find any guidelines on migration and since server 11 uses
postgress instead of couchdb I expect migration should have some
peculiarities. May I ask you to share knowledge on this kind of migration?

Thanks for giving the alpha builds a try! We don't yet have official
migration tools. But I think I can help you with a couple of knife exec
type scripts for experimenting with.

The only real trick is to know that in the Chef 11 server, you can set
the public_key of a client when you create the client as an alternative to
having the client generate a key pair for you.

On your existing Chef10 server, put the following in download.knife and
run it as an admin client like knife exec download.knife.

disable JSON inflation

JSON.create_id = "no_thanks"

out_dir = "BACKUP"
Dir.mkdir(out_dir)

backup nodes

Dir.mkdir(File.join(out_dir, "nodes"))
node_names = api.get("nodes").keys
node_names.each do |name|
the_node = api.get("nodes/#{name}")
open(File.join(out_dir, "nodes", name), "w") do |f|
f.write(the_node.to_json)
end
end

backup clients for nodes

Dir.mkdir(File.join(out_dir, "clients"))
node_names.each do |name|
client = api.get("clients/#{name}")
open(File.join(out_dir, "clients", name), "w") do |f|
f.write(client.to_json)
end
end

Then in your Chef11 test environment, you can use the following to create
matching clients and nodes. The rest of the data should be pretty easy to
replicate (knife cookbook upload -a, etc)

disable JSON inflation

JSON.create_id = "no_thanks"

out_dir = "BACKUP"

load clients

Dir["#{out_dir}/clients/*"].**each do |path|
client = JSON.parse(IO.read(path))
api.post("clients", {
:name => client['name'],
:public_key => client['public_key']
})
end

load nodes

Dir["#{out_dir}/nodes/*"].each do |path|
the_node = JSON.parse(IO.read(path))
api.post("nodes", the_node)
end

Hope that helps. Please report any problems you encounter with Chef11 to
the list or file as bugs in the CHEF public bug tracker.

  • seth

--
Seth Falcon | Opscode

7

On Monday, January 7, 2013 at 11:35 AM, steve . wrote:

IIRC, Chef 11 also includes a change in the encrypted data bag format. Since nobody (outside of Opscode?) has tried to migrate from Chef 10.x to Chef 11, I don't think anybody really knows what that process will look like yet, but I took a stab at a knife data bag conversion plugin when this was first mentioned. Here's the code:

GitHub - leftathome/knife-databag-upgrade: Quick and dirty data bag prototype encrypted data bag migrator from Chef 10.x to Chef 11

It might help with some of your other migration tasks, as well... to form a basis for those tasks, at the very least.

Good luck and let us all know how it goes!
It's not strictly necessary to migrate your data bags, as Chef 11 chef-client can read the older format encrypted data bags. That said, it's a good idea to do so, as the new format has improved security.

The key thing to be aware of is that only Chef 10.18+ (10.18.0 in testing now) will have forwards compatibility with Chef 11 format encrypted data bags so you need to use Chef 10.x to upload encrypted data bag items until all your chef clients are upgraded to at least 10.18.0.

--
Daniel DeLeo