Chef Infra Server 15.10.114 Upgrade Regression / Compatibility Issue

Chef Infra Server
1

Chef Infra Server 15.10.114 Upgrade Regression / Compatibility Issue

We encountered a reproducible issue when upgrading a standalone Chef Infra Server from 15.10.91 to 15.10.114.

The upgrade was performed strictly according to the documented upgrade procedure and completed successfully without any reported errors. After running chef-server-ctl reconfigure, all services appeared healthy from a process perspective.

Environment

  • Chef Infra Server 15.10.91 → 15.10.114
  • RHEL 8
  • Standalone deployment
  • Chef Automate integrated
  • Approximately 1300 managed nodes

Symptoms after upgrade

Immediately after upgrading to 15.10.114:

chef-server-ctl user-list

returned HTTP 500.

chef-server-ctl org-list

returned HTTP 500.

Chef Automate could no longer communicate correctly with Chef Infra Server.

The Erchef status endpoint reported:

{
  "status":"fail",
  "upstreams":{
    "chef_opensearch":"pong",
    "chef_sql":"fail",
    "oc_chef_authz":"pong",
    "data_collector":"pong"
  }
}

and opscode-erchef/current repeatedly showed:

{chef_sql,ping,throw,pang}

Troubleshooting performed

We verified the following components were healthy:

  • PostgreSQL running
  • OpenSearch running
  • Bifrost running
  • OC_ID running
  • Redis/Valkey running
  • Erchef listening on port 8000
  • Erchef SQL credentials valid

Direct PostgreSQL connectivity was successful:

PGPASSWORD="$(chef-server-ctl show-secret opscode_erchef sql_password)" \
psql -h 127.0.0.1 -U opscode_chef -d opscode_chef

Database contents were intact:

select count(*) from users;
select count(*) from orgs;

returned expected values.

Core tables such as:

users
orgs
nodes
clients
cookbooks

were present and queryable.

Therefore the issue did not appear to be caused by:

  • PostgreSQL availability
  • Authentication failure
  • Missing schema
  • Missing data
  • Secret mismatch

Recovery

We downgraded the server back to 15.10.91.

Steps performed:

rpm -Uvh chef-server-core-15.10.91-1.el8.x86_64.rpm

chef-server-ctl reconfigure

chef-server-ctl start

No backup restore was required.

Immediately after reverting:

chef-server-ctl user-list

worked normally.

chef-server-ctl org-list

worked normally.

Automate integration recovered.

/_status returned:

{
  "status":"pong",
  "upstreams":{
    "chef_opensearch":"pong",
    "chef_sql":"pong",
    "oc_chef_authz":"pong",
    "data_collector":"pong"
  }
}

Conclusion

The issue appears to be introduced or triggered by the move from 15.10.91 to 15.10.114.

We have not yet determined whether this is:

  • a regression in the Erchef SQL layer,
  • a migration issue,
  • a compatibility issue with existing data,
  • or an environment-specific condition.

However, the problem was fully reproducible on 15.10.114 and fully resolved by reverting to 15.10.91 using the same database, same secrets, and same server.

Would appreciate feedback from the Chef/Progress team if there are known issues, additional diagnostics, or existing cases related to:

chef_sql = fail
{chef_sql,ping,throw,pang}

following upgrade to 15.10.114.

2

are you able to submit a support ticket for this issue?

3

I can submit the support ticket, but I don't think I need to raise it because everything went fine after I downgraded. I didn't even restore the Database. I'm pretty sure the new version has some issues; post it here as a bug report for your reference.

4

in order to escalate it to engineering for a bug fix we need a support ticket and may need to investigate it further with support. I haven't seen this issue reported by any other users so we may need to verify if it's an actual bug or a side case based on environment or something else.