Migrating from chef 11 to chef 12 server


#1

Is doing a full export of the data (using knife backup) from one server
running open source Chef Server 11 then importing the data into a new
instance running a fresh install of Chef Server 12 a supported upgrade
path? I’ve reviewed the docs here
http://docs.getchef.com/upgrade_server_open_source_notes.html, but if
possible, I would prefer to start fresh with a new instance of chef server
rather than upgrading the existing one in place.

thanks,
Bethany

Bethany Erskine
Senior Technical Operations Engineer
http://www.paperlesspost.com


#2

Hi Bethany,

An in-place upgrade from an open source Chef Server 11 to Chef Server 12
actually runs the knife-backup / knife-restore processes under the covers.
This will also have the end result of starting with a new instance of the
server, where the underlying database is completely fresh. Chef 12 installs
to /opt/opscode and the associated data dirs are in /var/opt/opscode where
Chef 11 was /opt/chef-server and /var/opt/chef-server.

If your desires of a fresh instance are more around wanting a new hardware
instance (or a new VM), then there are also commands that will allow you to
export the data and import it to the new server. The process around export
/ import is a bit more involved than simply a knife upload / download, so
I’ll find those docs and link you to them shortly.

Thanks!
Stephen

On Thu, Oct 9, 2014 at 8:08 AM, Bethany Erskine bethany@paperlesspost.com
wrote:

Is doing a full export of the data (using knife backup) from one server
running open source Chef Server 11 then importing the data into a new
instance running a fresh install of Chef Server 12 a supported upgrade
path? I’ve reviewed the docs here
http://docs.getchef.com/upgrade_server_open_source_notes.html, but if
possible, I would prefer to start fresh with a new instance of chef server
rather than upgrading the existing one in place.

thanks,
Bethany

Bethany Erskine
Senior Technical Operations Engineer
http://www.paperlesspost.com


Stephen Delano
Software Development Engineer
Opscode, Inc.
1008 Western Avenue
Suite 601
Seattle, WA 98104


#3

Hey Bethany,

I wanted to follow up on what Stephen said and make sure you have all
the information you might need. What Stephen said is correct. When you
upgrade from the open source Chef Server 11 to Chef Server 12 it is
essentially a clean install. The Chef Server 12 package installs into a
different location than the open source Chef Server 11 (/opt/opscode
instead of /opt/chef-server). The upgrade process makes use of this fact
to download the data from the Chef Server 11, transform it on disk, then
upload it to the Chef Server 12. Currently the automated process assumes
you’re doing a standalone install of one server to the next on the same
box, but this certainly doesn’t have to be the case.

To do an install from one box to another, you can use the upgrade
subcommands provided, which are documented here:
http://docs.getchef.com/upgrade_server_open_source_notes.html#subcommand-reference

These commands, if run serially, are exactly the same as running the
upgrade command. To have access to these commands, you need the Chef
Server 12 package installed on the box you are going to use them on (the
server does not have to be configured yet, the package just has to be
installed on the box). All the subcommands take the form
chef12-upgrade-*. So you could use the chef-server-ctl
chef12-upgrade-download option to download everything from the Chef
Server 11 (note that right now this will stop your Chef Server 11 after
it completes the download - if you don’t want this, you can run a knife
download yourself against the Chef Server 11 and it will produce the
same data as this command, so long as you tell knife download to
download everything. We’re working to make this experience nicer.).
After you run this command, it will point you to a tmp directory where
all the data was saved (you can also specify a directory to the command.
I recommend you run -h on all the subcommands to see the various options
available). If you transfer this data to the Chef Server 12 box, you can
then pick up with the chef12-upgrade-transform command and finish with
the chef12-upgrade-upload command and it will complete the upgrade for
you and bring up your Chef Server 12.

Given that we’ve open sourced all of the Chef server now, all of this
code is also open sourced, which is here:

In this case the code is all Ruby. The files in question you want to
view are upgrade.rb, which kicks off the process,
open_source_chef12_upgrade.rb which contains all the logic for doing the
upgrade, and then the three files chef12-upgrade-*, which serve as entry
points into the upgrade process if the subcommands talked about above
are used. Note that the subcommands just reuse the same code as the
upgrade process.

And just for completeness, I’ll note the docs for the normal upgrade
path are located here:
http://docs.getchef.com/server/upgrade_server.html#from-chef-server-osc

I’ll also note that when doing the upgrade, in the interest of ensuring
the process loses none of your data (we don’t expect it to, but this
involves computers, so the unexpected will happen), nothing is deleted
from disk. This means after an upgrade that the downloaded Chef Server
11 data will still be on disk, as will the transformed data that is
uploaded to the Chef Server 12. If you do this upgrade on a single box,
it is also up to you to remove the Chef Server 11 package from the
system. You can either backup or delete the data that is on disk as
you’d like. The upgrade process will tell you where this data is located
in a message at the end, or you can use the command line flags to
specify where you’d like it located. One benefit of not removing the
Chef Server 11 automatically is that if something does go wrong with the
upgrade to Chef Server 12, you can turn off the Chef Server 12 and
restart the Chef Server 11 (although you will need to point at the Chef
Server 11 ctl command yourself, since Chef Server 12 shares the same
chef-server-ctl naming. The Chef Server 11 ctl command is located at
/opt/chef-server/bin/chef-server-ctl. If you remove Chef Server 12 from
the system then this won’t be an issue, but as long as both share the
same system it will be).

That’s probably more information that you wanted, but if at any point
you have any questions, feel free to email this list or you can seek me
out in the #Chef IRC room. I’m mmzyk.

Thanks,

Mark Mzyk

Stephen Delano mailto:stephen@opscode.com
October 9, 2014 at 11:59 AM
Hi Bethany,

An in-place upgrade from an open source Chef Server 11 to Chef Server
12 actually runs the knife-backup / knife-restore processes under the
covers. This will also have the end result of starting with a new
instance of the server, where the underlying database is completely
fresh. Chef 12 installs to /opt/opscode and the associated data dirs
are in /var/opt/opscode where Chef 11 was /opt/chef-server and
/var/opt/chef-server.

If your desires of a fresh instance are more around wanting a new
hardware instance (or a new VM), then there are also commands that
will allow you to export the data and import it to the new server. The
process around export / import is a bit more involved than simply a
knife upload / download, so I’ll find those docs and link you to them
shortly.

Thanks!
Stephen


Stephen Delano
Software Development Engineer
Opscode, Inc.
1008 Western Avenue
Suite 601
Seattle, WA 98104
Bethany Erskine mailto:bethany@paperlesspost.com
October 9, 2014 at 11:08 AM
Is doing a full export of the data (using knife backup) from one
server running open source Chef Server 11 then importing the data into
a new instance running a fresh install of Chef Server 12 a supported
upgrade path? I’ve reviewed the docs here
http://docs.getchef.com/upgrade_server_open_source_notes.html, but
if possible, I would prefer to start fresh with a new instance of chef
server rather than upgrading the existing one in place.

thanks,
Bethany

Bethany Erskine
Senior Technical Operations Engineer
http://www.paperlesspost.com


#5

For what it worth here what I did when migrating from my old open source chef-server to chef12, I was not on chef 11.1 so I had some caveats to workaround when migrating.

The path is on the new chef12 server, get the key of an admin user to be able to download eveything:

chef-server-ctl chef12-upgrade-download -D -u user -k /path/to/user.pem -s https://chef11.server.url -d /tmp/chef11data

Next chef-server-ctl chef12-upgrade-data-transform -d /tmp/chef11data -e /tmp/chef12data It will ask you for an organization name

Last: chef-server-ctl chef12-upgrade-upload -e /tmp/chef12data

If the upload fails due to old cookbooks not valid anymore, you can have a look at my github repo where I stored what I used to fix my repo (more or less raw notes, so please forgive the ugly code and documentation): https://github.com/Tensibai/chef-server-migration


#6

I copied admin.pem file from chef11 server to /home/pzab/chef11backup/ on the new chef12 server (my “/tmp/chef11data” folder). Is it enough?
I got error
/opt/opscode/embedded/lib/ruby/2.2.0/fileutils.rb:1391:in `initialize’: No such file or directory @ rb_sysopen - /home/pzab/chef11backup/key_dump.json


#7

According to the error, you didn’t follow the steps.
Your key can be anywhere on the filesystem, I did my best to give silly path to avoid misunderstanding but I won’t hold your hand on the whole process.

If you need assistance consider helping the community by buying professional assistance on migration from chef enterprise at chef.io I’m the support tab.

Reminder: I’m not working for Chef enterprise, I’m a user like you, and happy to help when people show efforts to solve problems by themselves, if you don’t have enough time to dig, really spend what’s needed to get enterprise class support


#8

Tensibai, tnx for your steps. I got this script to get user key. Don’t know why I decided to change first step.
Then I used transform and upload.

Open source Chef 11 server data successfully uploaded to Chef 12 server

First time I had problem with one cookbook. Process was interrupted. I fixed cookbook and repeated upload. Now I have problem with all nodes.

ERROR: /nodes failed to create_child: Failure creating ‘example.com.json’: 400 “Bad Request” cause: {“error”:[“Invalid key apache in request body”]}

I can use knife commands on workstation.
As expected, knife node list return nothing, I can list cookbooks, clients, etc.


#9

Sorry, at this point there’s few I can do, check the nodes json files in the transformed (chef12) directory.
Maybe there’s a duplication somewhere or a node has bad attributes, hard to tell.

You can try using knife upload as does the chef-server-ctl upload with -VV to get more informations. (I don’t remember exactly the knife command, but IIRC the upload display it)


#10

I used both knife download / and knife backup extract
I took users from the first one and all other stuff from the second one, because knife backup downloads all cookbook versions.
I tried nodes from knife dowload and now I have up and running migrated chef-server 12
During chef12-upgrade-upload I had few

ERROR: Server returned error 500 for https://localhost/organizations/example/cookbooks/mysql/5.0.2, retrying 1/5 in 3s

but they where successfully uploaded.
Also i had this warning
Thanks alot for this thread.