Hi ,
I am trying to trigger a job using chef push job functionality.
But getting “quorum failed” error while trying to start a job using knife command.
[root@chef-workstation cookbooks]# knife job start java 'node-1’
Started. Job ID: 7296674996db01f7e3d86dd4c6ac0310
.Quorum_failed.
command: touch
created_at: Fri, 13 Jul 2018 12:07:37 GMT
env:
id: 7296674996db01f7e3d86dd4c6ac0310
nodes:
nacked: node-1
run_timeout: 3600
status: quorum_failed
updated_at: Fri, 13 Jul 2018 12:07:37 GMT
[root@chef-workstation cookbooks]#
Node is active and i can see it is able to communicate with push job server.
[root@chef-workstation cookbooks]# knife node status node-1
node-1 available
[root@chef-workstation cookbooks]#
From the push-job-client log, i could see below error:
INFO: [node-1] Received server heartbeat (sequence #302241) logging 0/3
INFO: [node-1] Received server heartbeat (sequence #302242) logging 1/3
INFO: [node-1] Received server heartbeat (sequence #302243) logging 2/3
INFO: [node-1] Received server heartbeat (sequence #302244) logging 3/3
INFO: [UNKNOWN] Job UNKNOWN: whitelist ‘java’ to ‘’
INFO: [UNKNOWN] Job UNKNOWN: whitelist ‘java’ to ‘’
ERROR: [node-1] Received commit adb43071447255a87b294ce313dc4e8a, but command ‘java’ is not in the whitelist!
But i have added the job to the whitelist attribute of node and i am able to see it from my chef-manage webui. Also added the details to push-job-client.rb file.
What could be possible reason for this error? Can anyone help me to understand the “quorum failed” error?
You are getting quorum failed because server you specified is returning a nacked message for the job request. From the logs you posted that’s because it couldn’t find a java task in the whitelist in /etc/chef/push-job-client.rb
Can you confirm that there is a “java” whitelist entry in /etc/chef/push-job-client.rb. If so what does it look like? More than likely the task isn’t in the list or it’s not formatted correctly some how.
If you are using the push-jobs cookbook to configure your system and manually added the java task to push-job-client.rb it’s possible that it went through and removed the entry for you.
Thanks for the reply.
i am using push-jobs cookbook .
by default the chef-client command was there and it was working fine.
knife job start chef-client ‘node-1’
when i set the push-jobs whitelist node attribute by running below command from workstation
knife node attribute set node-1 push_jobs.whitelist ‘[“java: chef-client -o recipe[java]”]’
it set the attribute successfully but when i extract the attribute value in json file the push-jobs attribute value will be empty. i think i am missing something the attribute value is not updating correctly. so i decided to update the push-jobs cookbook attribute default.rb file and uploaded.
when i run command knife job start java ‘node-1’ i am getting quorum failed error.
In /etc/chef/ push-job-client.rb file it has below entry
cat /etc/chef/push-job-client.rb
cat: /etc/chef/push-job-client.rb: No such file or directory
[root@node-1 ~]# cat /etc/chef/push-jobs-client.rb
Generated by Chef for node-1
Local modifications will be overwritten!
Chef::Config.from_file("/etc/chef/client.rb")
The whitelist comes from node[‘push_jobs’][‘whitelist’]
whitelist({“java”=>“chef-client -o recipe[java]”})
We’re under runit, so don’t output timestamp
Mixlib::Log::Formatter.show_time = false
Hi
Please find job output details.
knife job start --capture java node-1
Started. Job ID: adb4307144722c5f5e7f4ef1578f7521
.Quorum_failed.
capture_output: true
command: java
created_at: Mon, 16 Jul 2018 12:29:02 GMT
env:
id: adb4307144722c5f5e7f4ef1578f7521
nodes:
nacked: node-1
run_timeout: 3600
status: quorum_failed
updated_at: Mon, 16 Jul 2018 12:29:02 GMT
[vagrant@lampnode pushjob] knife job output --channel stdout adb4307144722c5f5e7f4ef1578f7521 node-1
ERROR: The object you are looking for could not be found
Response: <HTML><HEAD><TITLE>404 Not Found</TITLE></HEAD><BODY><H1>Not Found</H1>The requested document was not found on this server.<P><HR><ADDRESS>mochiweb+webmachine web server</ADDRESS></BODY></HTML>
[vagrant@lampnode pushjob]
When you are pasting the contents from the push-job-client.rb file please escape it using triple backticks or put it into a gist and link to it. It’s very hard to tell if there is possibly some sort of formatting error with the file otherwise since it’s getting evaluated as markdown content.