Quorum Failure In Chef Delivery Push Jobs

Chef Automate
1

I am suddenly getting quorum failure on any run from converge from Delivery… I had been running fine up until last Friday.

I am able to log in to the nodes in any of the environments and can run chef-client locally.

Running a knife node list lists all of my nodes, however when running a knife node status, none of my Acceptance, Union, Rehearsal, or Delivered nodes are showing up.

Running knife node status returns the node as “unavailable”

I have tried removing a node and client from Chef, deleting and re-creating the VM and re-bootstrapping the node into Chef, and while the node is created in Chef, I still am getting the “unavailable” status on a Knife node status

Any Ideas of where I went pear shaped???

Update:

I have been reviewing a few other online resources and it seems that my push-jobs clients are failing to start… when I log into a node and attempt to start the pushy-client I get the following error upon attempting to retrieve the configuration from the Chef server:
ERROR: [] Exiting: No key returned from server; server may be using 1.x protocol. The config flag ‘allow_unencrypted’ disables encryption and allows use of 1.x server. Use with caution!

Anyone have any idea which configuration is causing the error???

2

All, just wanted to post that I figured this out. The issue began a few days ago when the push-jobs client updated because I had not pinned a version. That led to an issue with the clients not being able to converge. I was able to get around that problem by moving to the 2.7.0 version of the push-jobs client, until last Friday, when I started getting the “quorum failure” notices on the attempt to converge from Delivery, yet I was able to converge manually from a chef-client run on the node. After doing further research, I found that the push-jobs client had been again updated to solve this issue, with a default attribute to set a new config option. Updating to the new 2.8.1 client and writing a wrapper cookbook to override this new default attribute has resolved the issue.

Just in case anyone else has this issue and stumbles upon the original post…

1 Like
3

Thank you so much for this post/feedback! We really appreciate sharing the knowledge … If there is anything else you find please let us know!

Have a great week.
Salim Afiune