I'm not sure what's happening here, but on newer version of hab:
hab 0.61.0/20180815171844
Without the keyring I get this error message when I try to join the ring:
db2e1ea142dc:~ # tailf nohup.out
→ Using core/libarchive/3.3.2/20171018164107
→ Using core/libsodium/1.0.13/20170905223149
→ Using core/linux-headers/4.3/20170513200956
→ Using core/openssl/1.0.2l/20171014213633
→ Using core/xz/5.2.2/20170513214327
→ Using core/zlib/1.2.8/20170513201911
✓ Installed core/hab-launcher/8133/20180810175521
★ Install of core/hab-launcher/8133/20180810175521 complete with 1 new packages installed.
hab-sup(ER)[components/sup/src/error.rs:488:9]: Crypto error: No revisions found for consulring sym key
ERROR 2018-08-30T17:16:21Z: hab_launch: Launcher exiting with code 86
When I add the keyring to /hab/cache/keys and rerun the supervisor, it joins.
I ignored the message I was getting above “secret key and nonce could not decrypt” and proceeded further to create the quorum with ring used for the for the first VM in other VMs. The quorum came up but the logs bother me a little. I see
Master one to which the other nodes peer has this log.
"Invalid size of nonce"))
ERROR 2018-08-31T13:28:40Z: habitat_butterfly::server::inbound: Error parsing protobuf: HabitatCore(CryptoError("Secret key and nonce could not decrypt ciphertext"))
consul.dev2(O): 2018/08/31 13:28:41 [WARN] raft: Failed to get previous log: 9265 log not found (last: 9264)
consul.dev2(O): 2018/08/31 13:28:41 [INFO] consul: New leader elected: ito002378
ERROR 2018-08-31T13:28:42Z: habitat_butterfly::server::inbound: Error parsing protobuf: HabitatCore(CryptoError("Secret key and nonce could not decrypt ciphertext"))
consul.dev2(O): 2018/08/31 13:28:42 [DEBUG] raft-net: 19.0.1.96:8300 accepted connection from: 19.0.1.88:37818
consul.dev2(O): 2018/08/31 13:28:43 [INFO] consul: New leader elected: ito002378
ERROR 2018-08-31T13:28:43Z: habitat_butterfly::server::inbound: Error parsing protobuf: HabitatCore(CryptoError("Invalid size of nonce"))
ERROR 2018-08-31T13:28:43Z: habitat_butterfly::server::inbound: Error parsing protobuf: HabitatCore(CryptoError("Secret key and nonce could not decrypt ciphertext"))
consul.dev2(O): 2018/08/31 13:28:44 [WARN] raft: Failed to get previous log: 9268 log not found (last: 9267)
consul.dev2(O): 2018/08/31 13:28:44 [INFO] consul: New leader elected: ito002378
consul.dev2(O): 2018/08/31 13:28:45 [ERR] agent: failed to sync remote state: rpc error making call: No cluster leader
consul.dev2(O): 2018/08/31 13:28:45 [WARN] raft: Failed to get previous log: 9269 log not found (last: 9268)
consul.dev2(O): 2018/08/31 13:28:45 [INFO] consul: New leader elected: ito002378
ERROR 2018-08-31T13:28:45Z: habitat_butterfly::server::inbound: Error parsing protobuf: HabitatCore(CryptoError("Secret key and nonce could not decrypt ciphertext"))
ERROR 2018-08-31T13:28:46Z: habitat_butterfly::server::inbound: Error parsing protobuf: HabitatCore(CryptoError("Invalid size of nonce"))
ERROR 2018-08-31T13:28:46Z: habitat_butterfly::server::inbound: Error parsing protobuf: HabitatCore(CryptoError("Secret key and nonce could not decrypt ciphertext"))
This below is the log from the second VM that is peered to the above.
consul.dev2(O): 2018/08/31 13:17:08 [WARN] raft: Failed to get previous log: 8555 log not found (last: 8554)
consul.dev2(O): 2018/08/31 13:17:08 [INFO] consul: New leader elected: ito002378
consul.dev2(O): 2018/08/31 13:17:10 [DEBUG] raft-net: 19.0.1.90:8300 accepted connection from: 19.0.1.88:59550
consul.dev2(O): 2018/08/31 13:17:10 [INFO] consul: New leader elected: ito002378
consul.dev2(O): 2018/08/31 13:17:11 [WARN] raft: Failed to get previous log: 8559 log not found (last: 8557)
consul.dev2(O): 2018/08/31 13:17:11 [INFO] consul: New leader elected: ito002378
consul.dev2(O): 2018/08/31 13:17:13 [WARN] raft: Failed to get previous log: 8560 log not found (last: 8559)
consul.dev2(O): 2018/08/31 13:17:13 [INFO] consul: New leader elected: ito002378
consul.dev2(O): 2018/08/31 13:17:15 [WARN] raft: Failed to get previous log: 8561 log not found (last: 8560)
consul.dev2(O): 2018/08/31 13:17:15 [INFO] consul: New leader elected: ito002378
consul.dev2(O): 2018/08/31 13:17:17 [WARN] raft: Failed to get previous log: 8562 log not found (last: 8561)
consul.dev2(O): 2018/08/31 13:17:17 [WARN] agent: Syncing node info failed. rpc error making call: leadership lost while committing log
consul.dev2(O): 2018/08/31 13:17:17 [ERR] agent: failed to sync remote state: rpc error making call: leadership lost while committing log
consul.dev2(O): 2018/08/31 13:17:17 [INFO] consul: New leader elected: ito002378
This is the third VM peered to the VM at the top(the first vm that generated the ring)
consul.dev2(O): 2018/08/31 13:28:14 [INFO] raft: pipelining replication to peer {Nonvoter fcf200b5-57ed-d409-bcf0-16c99a8af499 19.0.1.96:8300}
consul.dev2(O): 2018/08/31 13:28:14 [INFO] raft: pipelining replication to peer {Nonvoter 194bc586-76fa-d5e3-f1d9-5ac2db3c915f 19.0.1.90:8300}
consul.dev2(O): 2018/08/31 13:28:14 [INFO] raft: Node at 19.0.1.88:8300 [Follower] entering Follower state (Leader: "")
consul.dev2(O): 2018/08/31 13:28:14 [INFO] raft: aborting pipeline replication to peer {Nonvoter fcf200b5-57ed-d409-bcf0-16c99a8af499 19.0.1.96:8300}
consul.dev2(O): 2018/08/31 13:28:14 [INFO] raft: aborting pipeline replication to peer {Nonvoter 194bc586-76fa-d5e3-f1d9-5ac2db3c915f 19.0.1.90:8300}
consul.dev2(O): 2018/08/31 13:28:14 [ERR] consul: failed to wait for barrier: leadership lost while committing log
consul.dev2(O): 2018/08/31 13:28:14 [INFO] consul: cluster leadership lost