Node recovery?


#1

I lost an EC2 instance over the weekend. Is there a proper method to replace it?

In the past, when I simply delete a client from chef, and bootstrap a client using the same node name, it registers itself, but wipes out the node run_list and attribute data.

Is there a way to bring up a new instance and assign it to be XYZ node and inherit the run_list & attributes?

Aaron


#2

did you reuse the client certificate of the old node? or re register it
using the validation cert? and keep the node name same
On Mon, Oct 17, 2011 at 9:43 PM, Aaron Abramson aabramson@wi-figuys.comwrote:

I lost an EC2 instance over the weekend. Is there a proper method to
replace it?

In the past, when I simply delete a client from chef, and bootstrap a
client using the same node name, it registers itself, but wipes out the node
run_list and attribute data.

Is there a way to bring up a new instance and assign it to be XYZ node and
inherit the run_list & attributes?

Aaron


#3

I did “knife client delete XYZ” and then “knife bootstrap IP -N XYZ”

On Oct 17, 2011, at 11:28 AM, Ranjib Dey wrote:

did you reuse the client certificate of the old node? or re register it using the validation cert? and keep the node name same
On Mon, Oct 17, 2011 at 9:43 PM, Aaron Abramson aabramson@wi-figuys.com wrote:
I lost an EC2 instance over the weekend. Is there a proper method to replace it?

In the past, when I simply delete a client from chef, and bootstrap a client using the same node name, it registers itself, but wipes out the node run_list and attribute data.

Is there a way to bring up a new instance and assign it to be XYZ node and inherit the run_list & attributes?

Aaron


#4

Hi Aaron,

On Mon, Oct 17, 2011 at 6:13 PM, Aaron Abramson aabramson@wi-figuys.comwrote:

I lost an EC2 instance over the weekend. Is there a proper method to
replace it?

In the past, when I simply delete a client from chef, and bootstrap a
client using the same node name, it registers itself, but wipes out the node
run_list and attribute data.

Is there a way to bring up a new instance and assign it to be XYZ node and
inherit the run_list & attributes?

Any reason why not specify the run_list in the bootstrap command (-r)? Do
you have spacial attributes for each host or can you define them in a role?

Bye


Haim


#5

Specifically, the node had several EBS volumes attached/mounted, and elastic IP, and those ID’s are stored as node attributes on the existing node within chef-server.

I want to fire up a new EC2 instance, and have it take-over for the down node, and re-attach/mount the EBS volumes, and elastic IP.

With amazon instances being treated as disposable, should we treat nodes as disposable as well? Or is there a way to have a new server assume ownership of an existing node?

Right now, I’m copy/pasting the current run_list data from chef-server, and grabbing any/all attributes, with the assumption that my only option is to manually rebuild the node data within chef.

Since knife has 2 methods, “node delete” and “client delete”, one would think that simply deleting the client would let a new server register itself as an existing node, and would inherit that nodes existing run_list and attributes. Otherwise, what is the point of deleting a node without deleting the client or vice versa.

On Oct 17, 2011, at 12:30 PM, Haim Ashkenazi wrote:

Hi Aaron,

On Mon, Oct 17, 2011 at 6:13 PM, Aaron Abramson aabramson@wi-figuys.com wrote:
I lost an EC2 instance over the weekend. Is there a proper method to replace it?

In the past, when I simply delete a client from chef, and bootstrap a client using the same node name, it registers itself, but wipes out the node run_list and attribute data.

Is there a way to bring up a new instance and assign it to be XYZ node and inherit the run_list & attributes?
Any reason why not specify the run_list in the bootstrap command (-r)? Do you have spacial attributes for each host or can you define them in a role?

Bye


Haim


#6

I use the chef server box or databag to store this type of attributes(those
you need beyond a nodes life cycle). And yes, i guess we should treat ec2
instances as disposable unit.

On Mon, Oct 17, 2011 at 11:13 PM, Aaron Abramson aabramson@wi-figuys.comwrote:

Specifically, the node had several EBS volumes attached/mounted, and
elastic IP, and those ID’s are stored as node attributes on the existing
node within chef-server.

I want to fire up a new EC2 instance, and have it take-over for the down
node, and re-attach/mount the EBS volumes, and elastic IP.

With amazon instances being treated as disposable, should we treat nodes as
disposable as well? Or is there a way to have a new server assume ownership
of an existing node?

Right now, I’m copy/pasting the current run_list data from chef-server, and
grabbing any/all attributes, with the assumption that my only option is to
manually rebuild the node data within chef.

Since knife has 2 methods, “node delete” and “client delete”, one would
think that simply deleting the client would let a new server register itself
as an existing node, and would inherit that nodes existing run_list and
attributes. Otherwise, what is the point of deleting a node without
deleting the client or vice versa.

On Oct 17, 2011, at 12:30 PM, Haim Ashkenazi wrote:

Hi Aaron,

On Mon, Oct 17, 2011 at 6:13 PM, Aaron Abramson aabramson@wi-figuys.comwrote:

I lost an EC2 instance over the weekend. Is there a proper method to
replace it?

In the past, when I simply delete a client from chef, and bootstrap a
client using the same node name, it registers itself, but wipes out the node
run_list and attribute data.

Is there a way to bring up a new instance and assign it to be XYZ node and
inherit the run_list & attributes?

Any reason why not specify the run_list in the bootstrap command (-r)? Do
you have spacial attributes for each host or can you define them in a role?

Bye


Haim


#7

On Mon, Oct 17, 2011 at 1:43 PM, Aaron Abramson aabramson@wi-figuys.com wrote:

With amazon instances being treated as disposable, should we treat nodes as
disposable as well? Or is there a way to have a new server assume ownership
of an existing node?

Yes. Stop caring about the node and start caring about the
service/role. There’s a certain amount of freedom you gain when you
treat everything as disposable.
The benefit of chef (or infrastructure as code in general) is that,
when ‘coded’, you have everything you need to rebuild a node from
scratch.
I’m obviously not privy to your environment but things like elastic
IP, EBS volumes and whatnot don’t belong to the node. They belong to
the role/service it’s providing. If it’s your production database, you
need to define those attributes in a data bag somewhere so that, when
you assign said role to a new node, they go with it.

Beyond that you should treat elastic IPs and EBS volumes themselves as
disposable. Amazon has had and will have again destructive EBS
problems. We’re actually moving away from using EBS entirely except
where we can’t get around it.

Everything fails. Embrace it.