Quorum failed error


#1

Hi ,

I am trying to trigger a job using chef push job functionality.
But getting “quorum failed” error while trying to start a job using knife command.

[root@chef-workstation cookbooks]# knife job start java 'node-1’
Started. Job ID: 7296674996db01f7e3d86dd4c6ac0310
.Quorum_failed.
command: touch
created_at: Fri, 13 Jul 2018 12:07:37 GMT
env:
id: 7296674996db01f7e3d86dd4c6ac0310
nodes:
nacked: node-1
run_timeout: 3600
status: quorum_failed
updated_at: Fri, 13 Jul 2018 12:07:37 GMT
[root@chef-workstation cookbooks]#

Node is active and i can see it is able to communicate with push job server.

[root@chef-workstation cookbooks]# knife node status node-1
node-1 available
[root@chef-workstation cookbooks]#

From the push-job-client log, i could see below error:

INFO: [node-1] Received server heartbeat (sequence #302241) logging 0/3
INFO: [node-1] Received server heartbeat (sequence #302242) logging 1/3
INFO: [node-1] Received server heartbeat (sequence #302243) logging 2/3
INFO: [node-1] Received server heartbeat (sequence #302244) logging 3/3
INFO: [UNKNOWN] Job UNKNOWN: whitelist ‘java’ to ‘’
INFO: [UNKNOWN] Job UNKNOWN: whitelist ‘java’ to ‘’
ERROR: [node-1] Received commit adb43071447255a87b294ce313dc4e8a, but command ‘java’ is not in the whitelist!

But i have added the job to the whitelist attribute of node and i am able to see it from my chef-manage webui. Also added the details to push-job-client.rb file.

What could be possible reason for this error? Can anyone help me to understand the “quorum failed” error?


#2

You are getting quorum failed because server you specified is returning a nacked message for the job request. From the logs you posted that’s because it couldn’t find a java task in the whitelist in /etc/chef/push-job-client.rb

Can you confirm that there is a “java” whitelist entry in /etc/chef/push-job-client.rb. If so what does it look like? More than likely the task isn’t in the list or it’s not formatted correctly some how.

If you are using the push-jobs cookbook to configure your system and manually added the java task to push-job-client.rb it’s possible that it went through and removed the entry for you.


#3

Thanks for the reply.

i am using push-jobs cookbook .
by default the chef-client command was there and it was working fine.
knife job start chef-client ‘node-1’

when i set the push-jobs whitelist node attribute by running below command from workstation

knife node attribute set node-1 push_jobs.whitelist ‘[“java: chef-client -o recipe[java]”]’

it set the attribute successfully but when i extract the attribute value in json file the push-jobs attribute value will be empty. i think i am missing something the attribute value is not updating correctly. so i decided to update the push-jobs cookbook attribute default.rb file and uploaded.

when i run command knife job start java ‘node-1’ i am getting quorum failed error.

In /etc/chef/ push-job-client.rb file it has below entry

cat /etc/chef/push-job-client.rb
cat: /etc/chef/push-job-client.rb: No such file or directory
[root@node-1 ~]# cat /etc/chef/push-jobs-client.rb

Generated by Chef for node-1

Local modifications will be overwritten!

Chef::Config.from_file("/etc/chef/client.rb")

The whitelist comes from node[‘push_jobs’][‘whitelist’]

whitelist({“java”=>“chef-client -o recipe[java]”})

We’re under runit, so don’t output timestamp

Mixlib::Log::Formatter.show_time = false


#4

Hey. I want to job resturnant


#5

Hi

Please find job output details.

knife job start --capture java node-1
Started. Job ID: adb4307144722c5f5e7f4ef1578f7521
.Quorum_failed.
capture_output: true
command: java
created_at: Mon, 16 Jul 2018 12:29:02 GMT
env:
id: adb4307144722c5f5e7f4ef1578f7521
nodes:
nacked: node-1
run_timeout: 3600
status: quorum_failed
updated_at: Mon, 16 Jul 2018 12:29:02 GMT

[vagrant@lampnode pushjob] knife job output --channel stdout adb4307144722c5f5e7f4ef1578f7521 node-1 ERROR: The object you are looking for could not be found Response: <HTML><HEAD><TITLE>404 Not Found</TITLE></HEAD><BODY><H1>Not Found</H1>The requested document was not found on this server.<P><HR><ADDRESS>mochiweb+webmachine web server</ADDRESS></BODY></HTML> [vagrant@lampnode pushjob]


#6

When you are pasting the contents from the push-job-client.rb file please escape it using triple backticks or put it into a gist and link to it. It’s very hard to tell if there is possibly some sort of formatting error with the file otherwise since it’s getting evaluated as markdown content.