Chef Infra Server 15.10.114 Upgrade Regression / Compatibility Issue

Chef Infra Server
1

Chef Infra Server 15.10.114 Upgrade Regression / Compatibility Issue

We encountered a reproducible issue when upgrading a standalone Chef Infra Server from 15.10.91 to 15.10.114.

The upgrade was performed strictly according to the documented upgrade procedure and completed successfully without any reported errors. After running chef-server-ctl reconfigure, all services appeared healthy from a process perspective.

Environment

  • Chef Infra Server 15.10.91 → 15.10.114
  • RHEL 8
  • Standalone deployment
  • Chef Automate integrated
  • Approximately 1300 managed nodes

Symptoms after upgrade

Immediately after upgrading to 15.10.114:

chef-server-ctl user-list

returned HTTP 500.

chef-server-ctl org-list

returned HTTP 500.

Chef Automate could no longer communicate correctly with Chef Infra Server.

The Erchef status endpoint reported:

{
  "status":"fail",
  "upstreams":{
    "chef_opensearch":"pong",
    "chef_sql":"fail",
    "oc_chef_authz":"pong",
    "data_collector":"pong"
  }
}

and opscode-erchef/current repeatedly showed:

{chef_sql,ping,throw,pang}

Troubleshooting performed

We verified the following components were healthy:

  • PostgreSQL running
  • OpenSearch running
  • Bifrost running
  • OC_ID running
  • Redis/Valkey running
  • Erchef listening on port 8000
  • Erchef SQL credentials valid

Direct PostgreSQL connectivity was successful:

PGPASSWORD="$(chef-server-ctl show-secret opscode_erchef sql_password)" \
psql -h 127.0.0.1 -U opscode_chef -d opscode_chef

Database contents were intact:

select count(*) from users;
select count(*) from orgs;

returned expected values.

Core tables such as:

users
orgs
nodes
clients
cookbooks

were present and queryable.

Therefore the issue did not appear to be caused by:

  • PostgreSQL availability
  • Authentication failure
  • Missing schema
  • Missing data
  • Secret mismatch

Recovery

We downgraded the server back to 15.10.91.

Steps performed:

rpm -Uvh chef-server-core-15.10.91-1.el8.x86_64.rpm

chef-server-ctl reconfigure

chef-server-ctl start

No backup restore was required.

Immediately after reverting:

chef-server-ctl user-list

worked normally.

chef-server-ctl org-list

worked normally.

Automate integration recovered.

/_status returned:

{
  "status":"pong",
  "upstreams":{
    "chef_opensearch":"pong",
    "chef_sql":"pong",
    "oc_chef_authz":"pong",
    "data_collector":"pong"
  }
}

Conclusion

The issue appears to be introduced or triggered by the move from 15.10.91 to 15.10.114.

We have not yet determined whether this is:

  • a regression in the Erchef SQL layer,
  • a migration issue,
  • a compatibility issue with existing data,
  • or an environment-specific condition.

However, the problem was fully reproducible on 15.10.114 and fully resolved by reverting to 15.10.91 using the same database, same secrets, and same server.

Would appreciate feedback from the Chef/Progress team if there are known issues, additional diagnostics, or existing cases related to:

chef_sql = fail
{chef_sql,ping,throw,pang}

following upgrade to 15.10.114.

2

are you able to submit a support ticket for this issue?

3

I can submit the support ticket, but I don't think I need to raise it because everything went fine after I downgraded. I didn't even restore the Database. I'm pretty sure the new version has some issues; post it here as a bug report for your reference.

4

in order to escalate it to engineering for a bug fix we need a support ticket and may need to investigate it further with support. I haven't seen this issue reported by any other users so we may need to verify if it's an actual bug or a side case based on environment or something else.

5

Update: Findings from further investigation (15.10.114 / 15.10.125)

After additional troubleshooting, I wanted to share what we observed in our environment in case it helps others.

Environment

Upgrade path:

Chef Infra Server 15.10.91

->

Chef Infra Server 15.10.114

and later:

Chef Infra Server 15.10.91

->

Chef Infra Server 15.10.125

Both upgrades completed successfully and reported:

[Infra Server Upgrade] - Latest PostgreSQL Database Migration Available: 1.37

[Infra Server Upgrade] - PostgreSQL Database Migrations to Run: none

Chef Infra Server Upgraded!

However, after upgrade:

chef-server-ctl org-list

failed and:

curl http://127.0.0.1:8000/_status

reported:

"chef_sql":"fail"

Database logs

PostgreSQL logs showed:

SELECT value FROM kvpair WHERE key = $1

followed by:

ERROR: relation "kvpair" does not exist

Checking the database confirmed that the table did not exist:

select schemaname, tablename

from pg_tables

where tablename='kvpair';

Result:

0 rows

What we found in the package

The upgraded package contains:

/opt/opscode/embedded/service/opscode-erchef/schema/deploy/kvpair.sql

and:

/opt/opscode/embedded/service/opscode-erchef/schema/sqitch.plan

contains:

telemetry 2024-08-01

kvpair 2026-02-23

At that point neither migration appeared in:

select change_id, change, committed_at

from sqitch.changes

order by committed_at desc;

The newest entries in our database were still from 2021.

Recovery

We first manually created the table using the contents of:

deploy/kvpair.sql

After creating the table:

chef-server-ctl org-list

started working again.

Also:

curl http://127.0.0.1:8000/_status

returned:

{

"status":"pong",

"upstreams":{

"chef_sql":"pong"

}

}

Automate connectivity was restored as well.

Further investigation

We then executed:

su - opscode-pgsql -c "

cd /opt/opscode/embedded/service/opscode-erchef/schema &&

PGHOST=/var/opt/opscode/postgresql/13/data

PGPORT=5432

/opt/opscode/embedded/bin/sqitch deploy"

Output:

Deploying changes to opscode_chef

+ telemetry .. ok

+ kvpair ..... ok

After that:

select change_id, change, committed_at

from sqitch.changes

order by committed_at desc;

showed:

telemetry

kvpair

as newly deployed migrations.

Additional observation: PostgreSQL socket path

In our upgraded environment:

show unix_socket_directories;

returns:

/var/opt/opscode/postgresql/13/data

However several PostgreSQL utilities appeared to use the default socket path:

/var/run/postgresql/.s.PGSQL.5432

unless PGHOST was explicitly specified.

Affected commands included:

vacuumdb

reindexdb

pg_dumpall

sqitch deploy

For example:

export PGHOST=/var/opt/opscode/postgresql/13/data

export PGDATABASE=opscode_chef

allowed the following commands to work correctly:

/opt/opscode/embedded/bin/vacuumdb --all --full

/opt/opscode/embedded/bin/reindexdb --all

/opt/opscode/embedded/bin/pg_dumpall

``

and also allowed:

chef-server-ctl backup

to complete successfully.

Current status

Current status on 15.10.125:

chef_sql = pong

org-list works

Automate connectivity works

Cookbooks accessible

Data bags accessible

Profiles accessible

Backup works (with PGHOST exported)

At this point we have a working system.

I'm not yet claiming a definitive root cause, but the facts above may help anyone seeing:

chef_sql = fail

relation "kvpair" does not exist

after upgrading from older 15.10.x releases.

6

That {chef_sql,ping,throw,pang} pattern is interesting — have you checked the actual erchef logs to see what's blowing up with the SQL connection attempt? The status endpoint tells you something's failing but not really what. Since it's fully reproducible with the same data and secrets, it does feel like something in the code path between 91 and 114 rather than your setup.

7

Thanks. We did check the PostgreSQL side in more detail and found something more specific than the {chef_sql,ping,throw,pang} status.

PostgreSQL logs showed repeated failures like:

SELECT value FROM kvpair WHERE key = $1

with:

ERROR: relation "kvpair" does not exist

We verified that the table was indeed missing:

select schemaname, tablename

from pg_tables

where tablename='kvpair';

returned:

0 rows

The upgraded package contains:

/opt/opscode/embedded/service/opscode-erchef/schema/deploy/kvpair.sql

and sqitch.plan contains:

telemetry 2024-08-01

kvpair 2026-02-23

However neither migration had been recorded in our database:

select change

from sqitch.changes

where change in ('telemetry','kvpair');

returned no rows.

We then manually executed:

su - opscode-pgsql -c "

cd /opt/opscode/embedded/service/opscode-erchef/schema &&

PGHOST=/var/opt/opscode/postgresql/13/data

PGPORT=5432

/opt/opscode/embedded/bin/sqitch deploy"

Output:

+ telemetry .. ok

+ kvpair ..... ok

After that:

select change_id, change, committed_at

from sqitch.changes

order by committed_at desc;

showed both telemetry and kvpair as deployed.

Immediately afterwards:

chef-server-ctl org-list

succeeded, and:

curl http://127.0.0.1:8000/_status

changed from:

"chef_sql":"fail"

to:

"chef_sql":"pong"

Automate connectivity, cookbook access and databag access were also restored.

One additional observation is that in this upgraded environment PostgreSQL is using:

/var/opt/opscode/postgresql/13/data

for unix_socket_directories, while several tools (pg_dumpall, vacuumdb, reindexdb, and sqitch) required an explicit:

PGHOST=/var/opt/opscode/postgresql/13/data

to function correctly.

We have not yet determined whether the missing telemetry/kvpair migrations were skipped entirely during upgrade or whether the migration step failed because of the PostgreSQL connection/socket configuration, but applying the missing sqitch migrations restored the system completely.