Chef server update went wrong - how to repair?


#1

A few weeks ago, I finally updated from chef 11 to chef 12 (I believe it was 12.0.3 at the time). The upgrade went smoothly and without a hitch that I could tell.

A few days ago, I updated to chef-server-core 12.1.0, and all hell broke loose. Well, not all hell, but some critical parts of chef aren’t working.

The RPM told me that it detected chef 11, and to run chef-server-ctl update. I wasn’t sure why it still saw chef 11 since I already upgraded to 12 a while ago, but I followed the instruction.

Now my chef client runs are failing with an “internal service error”. knife node show works without a problem, so he node data actually exists.

I haven’t found anything that seemed relevant in any log files, but then I’m ot sure what I’d be looking for. I did see error 500s in the nginx logs for the URL: POST /organizations/myorganization/environments/_default/cookbook_versions HTTP/1.1

How do I go about troubleshooting and fixing this?

Here is what happens on one of my clients:

Starting Chef Client, version 12.3.0
resolving cookbooks for run list: [ <redacted - the run list is correct> ]

================================================================================
Error Resolving Cookbooks for Run List:

Unknown Server Error:

The server had a fatal error attempting to load the node data.

Server Response:

internal service error

Running handlers:
Running handlers complete
Chef Client failed. 0 resources updated in 108.207621603 seconds

Kevin Keane

The NetTech

760-721-8339

http://www.4nettech.com

Our values: Privacy, Liberty, Justice

See https://www.4nettech.com/corp/the-nettech-values.html


#2

Hi Kevin,

Running chef-server-ctl upgrade and choosing not to upgrade from the Chef Server 11 data will fix your issue. In Chef Server 12.1.0, there are schema changes to the database that make the cookbook versions endpoint cause less of a load on the database. It appears that the errors you are seeing are a result of this schema not getting applied.


Stephen Delano - Engineering Lead, Chef

On Sun, Jun 28, 2015 at 11:51 AM, Kevin Keane Subscription
subscription@kkeane.com wrote:

A few weeks ago, I finally updated from chef 11 to chef 12 (I believe it was 12.0.3 at the time). The upgrade went smoothly and without a hitch that I could tell.
A few days ago, I updated to chef-server-core 12.1.0, and all hell broke loose. Well, not all hell, but some critical parts of chef aren’t working.
The RPM told me that it detected chef 11, and to run chef-server-ctl update. I wasn’t sure why it still saw chef 11 since I already upgraded to 12 a while ago, but I followed the instruction.
Now my chef client runs are failing with an “internal service error”. knife node show works without a problem, so he node data actually exists.
I haven’t found anything that seemed relevant in any log files, but then I’m ot sure what I’d be looking for. I did see error 500s in the nginx logs for the URL: POST /organizations/myorganization/environments/_default/cookbook_versions HTTP/1.1
How do I go about troubleshooting and fixing this?
Here is what happens on one of my clients:
Starting Chef Client, version 12.3.0
resolving cookbooks for run list: [ <redacted - the run list is correct> ]

Error Resolving Cookbooks for Run List:

Unknown Server Error:

The server had a fatal error attempting to load the node data.
Server Response:

internal service error
Running handlers:
Running handlers complete
Chef Client failed. 0 resources updated in 108.207621603 seconds
Kevin Keane
The NetTech
760-721-8339
http://www.4nettech.com
Our values: Privacy, Liberty, Justice
See https://www.4nettech.com/corp/the-nettech-values.html