Handle 403 Error if client with same name already exists on Chef Server

Chef Infra (archive)
1

Hello All,

So I have been using Chef to bootstrap the new EC2 Instances spun by AWS Autoscaling. The node names given to those servers are unique based on their IP addresses., this works all good. Now I need to address a case where Auto Scaling may spin a new instance with an IP address that was used earlier (in this case the node with the same name will still be on the Chef Server). Right now if a new node with the same name tries to check-in to Chef Server, it gives the 403 error and I need to manually remove the Node and Client from Chef Server to make it work.

What I need to know the best way to handle such a case?

Some ways that come into my head:

  1. Create a script that runs on the existing node to remove itself from Chef Server (when it is being scaled out by Auto Scaling group). I could also see if I can run this script on my Provisioner node based on the notifications I could get with the node name being scaled out.
  2. Alternative to scripting it out would be to set appropriate permissions for Clients group on Chef Server so that the new nodes can overwrite the old node object if it already exists. I want to know if this is a good idea and about the problems I might face in general if I do this.

Any help would be highly appreciated!

Thanks!

2

I haven’t implemented it yet but: https://aws.amazon.com/blogs/apn/automatically-delete-terminated-instances-in-chef-server-with-aws-lambda-and-cloudwatch-events/

3

We have implemented this with our organization and it works very well. I even expanded it to remove nodes from chef automate, active directory, dns, and other systems we have chef registering new servers to for monitoring etc…

4

There is also a project called aws_cleaner: https://github.com/eheydrick/aws-cleaner