Handling "deleted" servers on chef


#1

Hi Chefs,
I was wondering how some of you are handling deleted servers (bare metal or
VMs) which means nodes for the chef server.
For instance, if for some reason I suddenly kill 20 of my VMs/servers I
need to do some kind of housekeeping on chef-server, which in my case, a
dumb script does it.

Thoughts?

Thanks

Jorge E. Espada


#2

I had a similar script triggered by a Nagios host state change. When the
machine went down Nagios would figure out and execute the script.

2012/3/9 jorge espada espada.jorge@gmail.com

Hi Chefs,
I was wondering how some of you are handling deleted servers (bare metal
or VMs) which means nodes for the chef server.
For instance, if for some reason I suddenly kill 20 of my VMs/servers I
need to do some kind of housekeeping on chef-server, which in my case, a
dumb script does it.

Thoughts?

Thanks

Jorge E. Espada

Leonardo Gamas
Software Engineer
T +55 (71) 3494-3514
C +55 (75) 8134-7440
leogamas@jusbrasil.com.br


#3

One tip for ec2 users: you may also need/want to delete the chef client(i.e., not just the node) from the chef server.

The reason is that ec2 reuses private ip addresses which (at least in my case) are what get used as the client id when knife bootstrapping a node. Therefore, you can get a client id collision if a new ec2 instance reuses a private ip address the chef server has seen before.

  • Rob

On Mar 9, 2012, at 8:45 AM, Leonardo Gamas wrote:

I had a similar script triggered by a Nagios host state change. When the machine went down Nagios would figure out and execute the script.

2012/3/9 jorge espada espada.jorge@gmail.com
Hi Chefs,
I was wondering how some of you are handling deleted servers (bare metal or VMs) which means nodes for the chef server.
For instance, if for some reason I suddenly kill 20 of my VMs/servers I need to do some kind of housekeeping on chef-server, which in my case, a dumb script does it.

Thoughts?

Thanks

Jorge E. Espada

Leonardo Gamas
Software Engineer
T +55 (71) 3494-3514
C +55 (75) 8134-7440
leogamas@jusbrasil.com.br
www.jusbrasil.com.br


#4

On Mar 9, 2012, at 8:43 AM, Rob Guttman wrote:

One tip for ec2 users: you may also need/want to delete the chef client(i.e., not just the node) from the chef server.

The reason is that ec2 reuses private ip addresses which (at least in my case) are what get used as the client id when knife bootstrapping a node. Therefore, you can get a client id collision if a new ec2 instance reuses a private ip address the chef server has seen before.

Same deal for Rackspace. You want to delete the node, the client, and you want to delete the Rackspace instance as well. We also clean up our auto-registered DNS entries, and our ssh “known_hosts” files.

That’s a lot of stuff to have to remember to clean up every time you delete a node.
;-(


Brad Knowles bknowles@ihiji.com
SAGE Level IV, Chef Level 0.0.1


#5

the knife ec2 plugin has a --purge option for the knife ec2 vm delete command.
I’m thinking to copy that work and add it to the knive vsphere
command… rackspace could do it too.
Alternatively, deleting an instance could be a script that calls knife
vm delete, then knife client delete, then knife node delete.

Not sure which is better, adding it to the plugin, or deleting from a script…
-Jesse

On Fri, Mar 9, 2012 at 10:43, Brad Knowles bknowles@ihiji.com wrote:

On Mar 9, 2012, at 8:43 AM, Rob Guttman wrote:

One tip for ec2 users: you may also need/want to delete the chef client(i.e., not just the node) from the chef server.

The reason is that ec2 reuses private ip addresses which (at least in my case) are what get used as the client id when knife bootstrapping a node. Therefore, you can get a client id collision if a new ec2 instance reuses a private ip address the chef server has seen before.

Same deal for Rackspace. You want to delete the node, the client, and you want to delete the Rackspace instance as well. We also clean up our auto-registered DNS entries, and our ssh “known_hosts” files.

That’s a lot of stuff to have to remember to clean up every time you delete a node.
;-(


Brad Knowles bknowles@ihiji.com
SAGE Level IV, Chef Level 0.0.1


#6

The --purge option is still, iirc, in the master branch of the plugin source and not in the released plugin gem.

On Mar 9, 2012, at 9:31, Jesse Campbell hikeit@gmail.com wrote:

the knife ec2 plugin has a --purge option for the knife ec2 vm delete command.
I’m thinking to copy that work and add it to the knive vsphere
command… rackspace could do it too.
Alternatively, deleting an instance could be a script that calls knife
vm delete, then knife client delete, then knife node delete.

Not sure which is better, adding it to the plugin, or deleting from a script…
-Jesse

On Fri, Mar 9, 2012 at 10:43, Brad Knowles bknowles@ihiji.com wrote:

On Mar 9, 2012, at 8:43 AM, Rob Guttman wrote:

One tip for ec2 users: you may also need/want to delete the chef client(i.e., not just the node) from the chef server.

The reason is that ec2 reuses private ip addresses which (at least in my case) are what get used as the client id when knife bootstrapping a node. Therefore, you can get a client id collision if a new ec2 instance reuses a private ip address the chef server has seen before.

Same deal for Rackspace. You want to delete the node, the client, and you want to delete the Rackspace instance as well. We also clean up our auto-registered DNS entries, and our ssh “known_hosts” files.

That’s a lot of stuff to have to remember to clean up every time you delete a node.
;-(


Brad Knowles bknowles@ihiji.com
SAGE Level IV, Chef Level 0.0.1