Chef client conflicts

Chef Infra (archive)
1

Hello,

We run two different Chef Server instances (11.x and 12.x). When running against the 11.x instance, when there’s a node conflict, the conflict is logged in the output, but the new node takes over the client and the node. When running against the 12.x instance, conflicts are logged in the output, and the Chef run fails:

[2016-02-08T19:37:27+00:00] INFO: *** Chef 12.5.1 ***
[2016-02-08T19:37:27+00:00] INFO: Chef-client pid: 13162
[2016-02-08T19:37:29+00:00] INFO: Client key /etc/chef/client.pem is not present - registering
[2016-02-08T19:37:29+00:00] INFO: HTTP Request Returned 409 Conflict: Client already exists
[2016-02-08T19:37:29+00:00] INFO: HTTP Request Returned 400 Bad Request: error
[2016-02-08T19:37:29+00:00] ERROR: Running exception handlers
[2016-02-08T19:37:29+00:00] ERROR: Exception handlers complete
[2016-02-08T19:37:29+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2016-02-08T19:37:29+00:00] ERROR: 400 "Bad Request"
[2016-02-08T19:37:29+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

Is it possible to change this behavior to “last writer wins” like before? Our bootstrap process triggers on host boot, and deletes its own node and client on a shutdown or reboot. Unfortunately, in the cloud, not all hosts die cleanly, so some zombie nodes are left behind. This wasn’t a huge concern before, but these failed bootstraps against 12.x require manual intervention, which doesn’t work at scale.

Any tips would be appreciated.

Thanks,
Ameir

2

Hi Ameir,

I’m not sure if it’s possible (it may be), but wouldn’t it be better to use a unique node name for nodes instead (that way you avoid conflicts)? Generally, we’ve used stuff like the instance id (in AWS) as part of the node name since it’s always unique.

Cheers
Kieran

3

Hi Kieran,

Thanks for your response. I’d be open to using the instance ID, although the hostname is preferable for obvious reasons. How do you enforce the instance ID? Do you query the metadata service and generate a client.rb with that node name populated?

Thanks!
-Ameir

4

That’s what we’ve always done, we use the metadata to either set node_name in client.rb or run chef-client for the first time with the -N option (e.g. chef-client -N i-123456).