Knife status Internal error 500

Chef Infra (archive)
1

Hi,
I’m facing unusual internal error firing the command knife status or knife search on the Chef workstation. The output is here below:

[admin@host011 chef-repo]$ knife status -sl
ERROR: Server returned error 500 for https://host012.domain.local/organizations/a2a/search/node?q=*:*&sort=X_CHEF_id_CHEF_X%20asc&start=0, retrying 1/5 in 3s
ERROR: Server returned error 500 for https://host012.domain.local/organizations/a2a/search/node?q=*:*&sort=X_CHEF_id_CHEF_X%20asc&start=0, retrying 2/5 in 7s
ERROR: Server returned error 500 for https://host012.domain.local/organizations/a2a/search/node?q=*:*&sort=X_CHEF_id_CHEF_X%20asc&start=0, retrying 3/5 in 11s
ERROR: Server returned error 500 for https://host012.domain.local/organizations/a2a/search/node?q=*:*&sort=X_CHEF_id_CHEF_X%20asc&start=0, retrying 4/5 in 27s
ERROR: Server returned error 500 for https://host012.domain.local/organizations/a2a/search/node?q=*:*&sort=X_CHEF_id_CHEF_X%20asc&start=0, retrying 5/5 in 39s
ERROR: internal server error
Response: internal service error
[admin@host011 chef-repo]$

chef-server-ctl status and chef-server-ctl test show everything is up and running.

Do you have any idea of the cause or faced the same problem?

Regards

2

You’ll need to look at the logs on the chef server (chef-server-ctl tail) to see what’s causing the 500s. Given the end points you mention it sounds like your solr instance is probably down.

3

I’ve found that the PostgreSQL DB has some errors on the pg_toast table. Do you have any tips to fix it?

2018-05-30_13:22:04.16106 ERROR:  unexpected chunk number 2 (expected 0) for toast value 6418290 in pg_toast_16490
2018-05-30_13:22:04.16107 STATEMENT:  SELECT id, authz_id, org_id, name, environment, last_updated_by, created_at, updated_at, serialized_object FROM nodes WHERE (org_id = $1 AND name = $2) LIMIT 1
2018-05-30_13:22:08.18023 ERROR:  unexpected chunk number 2 (expected 0) for toast value 6418290 in pg_toast_16490
2018-05-30_13:22:08.18026 STATEMENT:  SELECT id, authz_id, org_id, name, environment, last_updated_by, created_at, updated_at, serialized_object FROM nodes WHERE (org_id = $1 AND name = $2) LIMIT 1
2018-05-30_13:22:13.19949 ERROR:  unexpected chunk number 2 (expected 0) for toast value 6418290 in pg_toast_16490
2018-05-30_13:22:13.19951 STATEMENT:  SELECT id, authz_id, org_id, name, environment, last_updated_by, created_at, updated_at, serialized_object FROM nodes WHERE (org_id = $1 AND name = $2) LIMIT 1
2018-05-30_13:22:29.21815 ERROR:  unexpected chunk number 2 (expected 0) for toast value 6418290 in pg_toast_16490
2018-05-30_13:22:29.21816 STATEMENT:  SELECT id, authz_id, org_id, name, environment, last_updated_by, created_at, updated_at, serialized_object FROM nodes WHERE (org_id = $1 AND name = $2) LIMIT 1
2018-05-30_13:22:59.23807 ERROR:  unexpected chunk number 2 (expected 0) for toast value 6418290 in pg_toast_16490
2018-05-30_13:22:59.23810 STATEMENT:  SELECT id, authz_id, org_id, name, environment, last_updated_by, created_at, updated_at, serialized_object FROM nodes WHERE (org_id = $1 AND name = $2) LIMIT 1
2018-05-30_13:23:29.68538 ERROR:  unexpected chunk number 2 (expected 0) for toast value 6418549 in pg_toast_16490
2018-05-30_13:23:29.68539 STATEMENT:  SELECT id, authz_id, org_id, name, environment, last_updated_by, created_at, updated_at, serialized_object FROM nodes WHERE (org_id = $1 AND name = $2) LIMIT 1
2018-05-30_13:23:33.70498 ERROR:  unexpected chunk number 2 (expected 0) for toast value 6418549 in pg_toast_16490
2018-05-30_13:23:33.70500 STATEMENT:  SELECT id, authz_id, org_id, name, environment, last_updated_by, created_at, updated_at, serialized_object FROM nodes WHERE (org_id = $1 AND name = $2) LIMIT 1
2018-05-30_13:23:39.73248 ERROR:  unexpected chunk number 2 (expected 0) for toast value 6418549 in pg_toast_16490
2018-05-30_13:23:39.73249 STATEMENT:  SELECT id, authz_id, org_id, name, environment, last_updated_by, created_at, updated_at, serialized_object FROM nodes WHERE (org_id = $1 AND name = $2) LIMIT 1
2018-05-30_13:23:50.75113 ERROR:  unexpected chunk number 2 (expected 0) for toast value 6418549 in pg_toast_16490
2018-05-30_13:23:50.75114 STATEMENT:  SELECT id, authz_id, org_id, name, environment, last_updated_by, created_at, updated_at, serialized_object FROM nodes WHERE (org_id = $1 AND name = $2) LIMIT 1
2018-05-30_13:23:55.25771 ERROR:  unexpected chunk number 2 (expected 0) for toast value 6418290 in pg_toast_16490
2018-05-30_13:23:55.25772 STATEMENT:  SELECT id, authz_id, org_id, name, environment, last_updated_by, created_at, updated_at, serialized_object FROM nodes WHERE (org_id = $1 AND name = $2) LIMIT 1
2018-05-30_13:24:22.77656 ERROR:  unexpected chunk number 2 (expected 0) for toast value 6418549 in pg_toast_16490
2018-05-30_13:24:22.77657 STATEMENT:  SELECT id, authz_id, org_id, name, environment, last_updated_by, created_at, updated_at, serialized_object FROM nodes WHERE (org_id = $1 AND name = $2) LIMIT 1
2018-05-30_13:25:23.80193 ERROR:  unexpected chunk number 2 (expected 0) for toast value 6418549 in pg_toast_16490
2018-05-30_13:25:23.80196 STATEMENT:  SELECT id, authz_id, org_id, name, environment, last_updated_by, created_at, updated_at, serialized_object FROM nodes WHERE (org_id = $1 AND name = $2) LIMIT 1
2018-05-30_13:26:02.88014 ERROR:  unexpected chunk number 3 (expected 0) for toast value 6418553 in pg_toast_16490
2018-05-30_13:26:02.88017 STATEMENT:  SELECT id, authz_id, org_id, name, environment, last_updated_by, created_at, updated_at, serialized_object FROM nodes WHERE (org_id = $1 AND name = $2) LIMIT 1
2018-05-30_13:26:05.89823 ERROR:  unexpected chunk number 3 (expected 0) for toast value 6418553 in pg_toast_16490
2018-05-30_13:26:05.89824 STATEMENT:  SELECT id, authz_id, org_id, name, environment, last_updated_by, created_at, updated_at, serialized_object FROM nodes WHERE (org_id = $1 AND name = $2) LIMIT 1
2018-05-30_13:26:13.91671 ERROR:  unexpected chunk number 3 (expected 0) for toast value 6418553 in pg_toast_16490
2018-05-30_13:26:13.91673 STATEMENT:  SELECT id, authz_id, org_id, name, environment, last_updated_by, created_at, updated_at, serialized_object FROM nodes WHERE (org_id = $1 AND name = $2) LIMIT 1
2018-05-30_13:26:22.93942 ERROR:  unexpected chunk number 3 (expected 0) for toast value 6418553 in pg_toast_16490
2018-05-30_13:26:22.93944 STATEMENT:  SELECT id, authz_id, org_id, name, environment, last_updated_by, created_at, updated_at, serialized_object FROM nodes WHERE (org_id = $1 AND name = $2) LIMIT 1
2018-05-30_13:26:39.95792 ERROR:  unexpected chunk number 3 (expected 0) for toast value 6418553 in pg_toast_16490
2018-05-30_13:26:39.95794 STATEMENT:  SELECT id, authz_id, org_id, name, environment, last_updated_by, created_at, updated_at, serialized_object FROM nodes WHERE (org_id = $1 AND name = $2) LIMIT 1
2018-05-30_13:26:52.58160 ERROR:  unexpected chunk number 7 (expected 3) for toast value 6418260 in pg_toast_16490
2018-05-30_13:26:52.58162 STATEMENT:  SELECT id, authz_id, org_id, name, environment, last_updated_by, created_at, updated_at, serialized_object FROM nodes WHERE (org_id = $1 AND name = $2) LIMIT 1
2018-05-30_13:26:56.59889 ERROR:  unexpected chunk number 7 (expected 3) for toast value 6418260 in pg_toast_16490
2018-05-30_13:26:56.59891 STATEMENT:  SELECT id, authz_id, org_id, name, environment, last_updated_by, created_at, updated_at, serialized_object FROM nodes WHERE (org_id = $1 AND name = $2) LIMIT 1
2018-05-30_13:27:04.61614 ERROR:  unexpected chunk number 7 (expected 3) for toast value 6418260 in pg_toast_16490
2018-05-30_13:27:04.61617 STATEMENT:  SELECT id, authz_id, org_id, name, environment, last_updated_by, created_at, updated_at, serialized_object FROM nodes WHERE (org_id = $1 AND name = $2) LIMIT 1
2018-05-30_13:27:18.63380 ERROR:  unexpected chunk number 7 (expected 3) for toast value 6418260 in pg_toast_16490
2018-05-30_13:27:18.63382 STATEMENT:  SELECT id, authz_id, org_id, name, environment, last_updated_by, created_at, updated_at, serialized_object FROM nodes WHERE (org_id = $1 AND name = $2) LIMIT 1
2018-05-30_13:27:22.97591 ERROR:  unexpected chunk number 3 (expected 0) for toast value 6418553 in pg_toast_16490
2018-05-30_13:27:22.97593 STATEMENT:  SELECT id, authz_id, org_id, name, environment, last_updated_by, created_at, updated_at, serialized_object FROM nodes WHERE (org_id = $1 AND name = $2) LIMIT 1
2018-05-30_13:27:39.65102 ERROR:  unexpected chunk number 7 (expected 3) for toast value 6418260 in pg_toast_16490
2018-05-30_13:27:39.65105 STATEMENT:  SELECT id, authz_id, org_id, name, environment, last_updated_by, created_at, updated_at, serialized_object FROM nodes WHERE (org_id = $1 AND name = $2) LIMIT 1
2018-05-30_13:28:22.66937 ERROR:  unexpected chunk number 7 (expected 3) for toast value 6418260 in pg_toast_16490
2018-05-30_13:28:22.66940 STATEMENT:  SELECT id, authz_id, org_id, name, environment, last_updated_by, created_at, updated_at, serialized_object FROM nodes WHERE (org_id = $1 AND name = $2) LIMIT 1
2018-05-30_13:41:13.14698 ERROR:  unexpected chunk number 9 (expected 5) for toast value 6418282 in pg_toast_16490
2018-05-30_13:41:13.14700 STATEMENT:  SELECT id, authz_id, org_id, name, environment, last_updated_by, created_at, updated_at, serialized_object FROM nodes WHERE (org_id = $1 AND name = $2) LIMIT 1
2018-05-30_13:41:16.16640 ERROR:  unexpected chunk number 9 (expected 5) for toast value 6418282 in pg_toast_16490
2018-05-30_13:41:16.16642 STATEMENT:  SELECT id, authz_id, org_id, name, environment, last_updated_by, created_at, updated_at, serialized_object FROM nodes WHERE (org_id = $1 AND name = $2) LIMIT 1
2018-05-30_13:41:22.18491 ERROR:  unexpected chunk number 9 (expected 5) for toast value 6418282 in pg_toast_16490
2018-05-30_13:41:22.18493 STATEMENT:  SELECT id, authz_id, org_id, name, environment, last_updated_by, created_at, updated_at, serialized_object FROM nodes WHERE (org_id = $1 AND name = $2) LIMIT 1
2018-05-30_13:41:33.20929 ERROR:  unexpected chunk number 9 (expected 5) for toast value 6418282 in pg_toast_16490
2018-05-30_13:41:33.20931 STATEMENT:  SELECT id, authz_id, org_id, name, environment, last_updated_by, created_at, updated_at, serialized_object FROM nodes WHERE (org_id = $1 AND name = $2) LIMIT 1
2018-05-30_13:41:51.22776 ERROR:  unexpected chunk number 9 (expected 5) for toast value 6418282 in pg_toast_16490
2018-05-30_13:41:51.22778 STATEMENT:  SELECT id, authz_id, org_id, name, environment, last_updated_by, created_at, updated_at, serialized_object FROM nodes WHERE (org_id = $1 AND name = $2) LIMIT 1
2018-05-30_13:42:42.24558 ERROR:  unexpected chunk number 9 (expected 5) for toast value 6418282 in pg_toast_16490
2018-05-30_13:42:42.24560 STATEMENT:  SELECT id, authz_id, org_id, name, environment, last_updated_by, created_at, updated_at, serialized_object FROM nodes WHERE (org_id = $1 AND name = $2) LIMIT 1

Regards

4

https://newbiedba.wordpress.com/2015/07/07/postgresql-missing-chunk-0-for-toast-value-in-pg_toast/ appears to describe exactly the situation you’re in.

5

I’m reading just now. I’ve found the same article on my own and executed on my infrastructure. It looks like now the system has been restored, hoping no other errors will be shown.

Regards