RE: Re: Chef server update went wrong - how to repair?


#1

It does not seem to give me that option. If I tell it not to upgrade, it simply aborts. And if I tell it to upgrade, it seems to go right into the upgrade from chef 11 to chef 12 (even though chef11 does not even exist on that server any more, as far as I can tell)

chef-server-ctl upgrade

Upgrading with options #<OpenStruct skip_confirmation=false, chef11_server_url=“https://localhost”, chef12_server_url=“https://localhost”, upload_threads=10, chef11_admin_client_name=“admin”, chef11_admin_client_key="/etc/chef-server/admin.pem">
Open source Chef 11 server detected.
Would you like to upgrade? [yN]
n
Aborting upgrade.

chef-server-ctl upgrade

Upgrading with options #<OpenStruct skip_confirmation=false, chef11_server_url=“https://localhost”, chef12_server_url=“https://localhost”, upload_threads=10, chef11_admin_client_name=“admin”, chef11_admin_client_key="/etc/chef-server/admin.pem">
Open source Chef 11 server detected.
Would you like to upgrade? [yN]
y
Upgrading the open source Chef 11 server.
Chef 12 short organization name?
Chef 12 full organization name?
Creating /tmp/chef11-server-data20150628-5878-13zhtja as the location to save the open source Chef 11 server data
Ensuring Chef 12 server is stopped
ok: down: bookshelf: 0s, normally up
ok: down: nginx: 0s, normally up
ok: down: oc_bifrost: 1s, normally up
ok: down: oc_id: 0s, normally up
ok: down: opscode-chef-mover: 0s, normally up
ok: down: opscode-erchef: 1s, normally up
ok: down: opscode-expander: 0s, normally up
ok: down: opscode-expander-reindexer: 0s, normally up
ok: down: opscode-solr4: 0s, normally up
ok: down: postgresql: 1s, normally up
ok: down: rabbitmq: 0s, normally up
ok: down: redis_lb: 1s, normally up
sed: can’t read /opt/chef-server/bin/wait-for-rabbit: No such file or directory
Failed to write fix to wait-for-rabbit script

Kevin Keane

The NetTech

http://www.4nettech.com

Our values: Privacy, Liberty, Justice

See https://www.4nettech.com/corp/the-nettech-values.html

-----Original message-----
From: Stephen Delano stephen@opscode.com
Sent: Sunday 28th June 2015 12:02
To: chef@lists.opscode.com
Subject: [chef] Re: Chef server update went wrong - how to repair?

Hi Kevin,

Running chef-server-ctl upgrade and choosing not to upgrade from the Chef Server 11 data will fix your issue. In Chef Server 12.1.0, there are schema changes to the database that make the cookbook versions endpoint cause less of a load on the database. It appears that the errors you are seeing are a result of this schema not getting applied.


Stephen Delano - Engineering Lead, Chef

On Sun, Jun 28, 2015 at 11:51 AM, Kevin Keane Subscription <subscription@kkeane.com mailto:subscription@kkeane.com > wrote:

A few weeks ago, I finally updated from chef 11 to chef 12 (I believe it was 12.0.3 at the time). The upgrade went smoothly and without a hitch that I could tell.

A few days ago, I updated to chef-server-core 12.1.0, and all hell broke loose. Well, not all hell, but some critical parts of chef aren’t working.

The RPM told me that it detected chef 11, and to run chef-server-ctl update. I wasn’t sure why it still saw chef 11 since I already upgraded to 12 a while ago, but I followed the instruction.

Now my chef client runs are failing with an “internal service error”. knife node show works without a problem, so he node data actually exists.

I haven’t found anything that seemed relevant in any log files, but then I’m ot sure what I’d be looking for. I did see error 500s in the nginx logs for the URL: POST /organizations/myorganization/environments/_default/cookbook_versions HTTP/1.1

How do I go about troubleshooting and fixing this?

Here is what happens on one of my clients:

Starting Chef Client, version 12.3.0
resolving cookbooks for run list: [ <redacted - the run list is correct> ]

================================================================================
Error Resolving Cookbooks for Run List:

Unknown Server Error:

The server had a fatal error attempting to load the node data.

Server Response:

internal service error

Running handlers:
Running handlers complete
Chef Client failed. 0 resources updated in 108.207621603 seconds

Kevin Keane

The NetTech

760-721-8339

http://www.4nettech.com

Our values: Privacy, Liberty, Justice

See https://www.4nettech.com/corp/the-nettech-values.html


#2

Hi Kevin,

When you upgrade from the open source Chef 11 server to Chef server 12,
it doesn’t remove Chef server 11 from the system. The upgrade to Chef
server 12 is done next to Chef server 11 on the system, since they live
at different locations in the file system. It is done this way so you
can fall back to Chef server 11 if something goes wrong with the
upgrade. In your case, the upgrade went fine, but it seems like you
missed removing Chef server 11 from the system.

If you didn’t remove the old chef server package from the system the
files will still be present on disk and this is what the upgrade process
is detecting. Specifically, the upgrade process is picking up that
/opt/chef-server still exists and that is why it thinks that Chef server
11 is on the system (Chef server 12 installs to /opt/opscode, which is a
bit of legacy from it being based off the old enterprise code base).

So I recommend you remove the Chef server 11 package and then run the
upgrade command again. It should no longer pick up Chef server 11 and
will instead upgrade you from 12.0.3 to 12.1. Make sure you remove the
correct package, since they have similar names.

The reason that you likely don’t realize Chef server 11 is still on the
system is that all the processes for it are stopped, and on the upgrade
chef-server-ctl was switched to point at Chef server 12. It’s actually
still possible to access Chef server 11 if you use direct paths to the
tooling.

Hopefully that clears things up and gives you a way forward. Sorry for
the confusion on this.

Mark Mzyk
Chef Software Engineer

Kevin Keane Subscription mailto:subscription@kkeane.com
June 28, 2015 at 4:00 PM

It does not seem to give me that option. If I tell it not to upgrade,
it simply aborts. And if I tell it to upgrade, it seems to go right
into the upgrade from chef 11 to chef 12 (even though chef11 does not
even exist on that server any more, as far as I can tell)

chef-server-ctl upgrade

Upgrading with options #<OpenStruct skip_confirmation=false,
chef11_server_url=“https://localhost”,
chef12_server_url=“https://localhost”, upload_threads=10,
chef11_admin_client_name=“admin”,
chef11_admin_client_key="/etc/chef-server/admin.pem">
Open source Chef 11 server detected.
Would you like to upgrade? [yN]
n
Aborting upgrade.

chef-server-ctl upgrade

Upgrading with options #<OpenStruct skip_confirmation=false,
chef11_server_url=“https://localhost”,
chef12_server_url=“https://localhost”, upload_threads=10,
chef11_admin_client_name=“admin”,
chef11_admin_client_key="/etc/chef-server/admin.pem">
Open source Chef 11 server detected.
Would you like to upgrade? [yN]
y
Upgrading the open source Chef 11 server.
Chef 12 short organization name?
Chef 12 full organization name?
Creating /tmp/chef11-server-data20150628-5878-13zhtja as the location
to save the open source Chef 11 server data
Ensuring Chef 12 server is stopped
ok: down: bookshelf: 0s, normally up
ok: down: nginx: 0s, normally up
ok: down: oc_bifrost: 1s, normally up
ok: down: oc_id: 0s, normally up
ok: down: opscode-chef-mover: 0s, normally up
ok: down: opscode-erchef: 1s, normally up
ok: down: opscode-expander: 0s, normally up
ok: down: opscode-expander-reindexer: 0s, normally up
ok: down: opscode-solr4: 0s, normally up
ok: down: postgresql: 1s, normally up
ok: down: rabbitmq: 0s, normally up
ok: down: redis_lb: 1s, normally up
sed: can’t read /opt/chef-server/bin/wait-for-rabbit: No such file or
directory
Failed to write fix to wait-for-rabbit script

Kevin Keane

The NetTech

http://www.4nettech.com

Our values: Privacy, Liberty, Justice

See https://www.4nettech.com/corp/the-nettech-values.html