Down scaling instances in AWS - how to run code on instance termination?


#1

Hi All :slight_smile:

I am using chef and autoscaling for aws ec2 instances. Installation of chef
and node’s registration process is handled mostly in user data so when an
instance is started because of autoscaling rule it properly registers in
chef.
When the instance is downscaling I would like to deregister it in chef
automatically. My problem is basically: how to run some code when aws
terminates an ec2 instance? I already saw some solution based on init
scripts run on shutdown but I think it is not the best choice :wink: I also
found info about terminating:wait functionality (
http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/AnalyzingAnAutoScalingInstance.html)
and am investigating it now.

Can someone point me in a proper direction? :slight_smile:

Thanks for the help!

BR,
Rafal.


#2

In the past I’ve done this with events and have a separate process that
receives the events and performs verification of them and, once verified,
runs cleanup steps. One of which was deleting the node and client entries
from the chef server.

Here’s some info on autoscaling events:

http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/ASGettingNotifications.html

Mike Glenney
New Context
On Feb 19, 2015 5:35 AM, “Rafał Radecki” radecki.rafal@gmail.com wrote:

Hi All :slight_smile:

I am using chef and autoscaling for aws ec2 instances. Installation of
chef and node’s registration process is handled mostly in user data so when
an instance is started because of autoscaling rule it properly registers in
chef.
When the instance is downscaling I would like to deregister it in chef
automatically. My problem is basically: how to run some code when aws
terminates an ec2 instance? I already saw some solution based on init
scripts run on shutdown but I think it is not the best choice :wink: I also
found info about terminating:wait functionality (
http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/AnalyzingAnAutoScalingInstance.html)
and am investigating it now.

Can someone point me in a proper direction? :slight_smile:

Thanks for the help!

BR,
Rafal.


#3

Hi,
Basically you need to create a script that wraps the knife delete command.
Put it on the /etc/init.d/ and then link it to the /etc/rc0.d/
Hope it helps
Michel

On Thu, Feb 19, 2015 at 10:35 AM, Rafał Radecki radecki.rafal@gmail.com
wrote:

Hi All :slight_smile:

I am using chef and autoscaling for aws ec2 instances. Installation of
chef and node’s registration process is handled mostly in user data so when
an instance is started because of autoscaling rule it properly registers in
chef.
When the instance is downscaling I would like to deregister it in chef
automatically. My problem is basically: how to run some code when aws
terminates an ec2 instance? I already saw some solution based on init
scripts run on shutdown but I think it is not the best choice :wink: I also
found info about terminating:wait functionality (
http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/AnalyzingAnAutoScalingInstance.html)
and am investigating it now.

Can someone point me in a proper direction? :slight_smile:

Thanks for the help!

BR,
Rafal.


#4

This is how we handled it as well. We configured our autoscaling groups
such that a termination event was sent to SNS, which landed in an SQS queue.

We then had an “admin” node that would wake up every 60 seconds and check
the SQS queue for any pending termination notices, and it would then
process them by deleting the node and client objects from the Chef server,
as well as some internal housekeeping items.

On Thu, Feb 19, 2015 at 8:21 AM, Michael Glenney mike.glenney@gmail.com
wrote:

In the past I’ve done this with events and have a separate process that
receives the events and performs verification of them and, once verified,
runs cleanup steps. One of which was deleting the node and client entries
from the chef server.

Here’s some info on autoscaling events:

http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/ASGettingNotifications.html

Mike Glenney
New Context
On Feb 19, 2015 5:35 AM, “Rafał Radecki” radecki.rafal@gmail.com wrote:

Hi All :slight_smile:

I am using chef and autoscaling for aws ec2 instances. Installation of
chef and node’s registration process is handled mostly in user data so when
an instance is started because of autoscaling rule it properly registers in
chef.
When the instance is downscaling I would like to deregister it in chef
automatically. My problem is basically: how to run some code when aws
terminates an ec2 instance? I already saw some solution based on init
scripts run on shutdown but I think it is not the best choice :wink: I also
found info about terminating:wait functionality (
http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/AnalyzingAnAutoScalingInstance.html)
and am investigating it now.

Can someone point me in a proper direction? :slight_smile:

Thanks for the help!

BR,
Rafal.


#5

Thanks for the tips :slight_smile: Notification hooks and SQS look very promising :slight_smile:
I assume that I would have to use aws-sdk for ruby (
https://aws.amazon.com/sdk-for-ruby/) in chef to connect to SQS and read
events from it? Are there any problems in using this sdk? Is it
stable/current? Are there any available howtos about SQS/aws-sdk/chef
integration?

BR,
Rafal.

2015-02-19 14:45 GMT+01:00 David Petzel davidpetzel@gmail.com:

This is how we handled it as well. We configured our autoscaling groups
such that a termination event was sent to SNS, which landed in an SQS queue.

We then had an “admin” node that would wake up every 60 seconds and check
the SQS queue for any pending termination notices, and it would then
process them by deleting the node and client objects from the Chef server,
as well as some internal housekeeping items.

On Thu, Feb 19, 2015 at 8:21 AM, Michael Glenney mike.glenney@gmail.com
wrote:

In the past I’ve done this with events and have a separate process that
receives the events and performs verification of them and, once verified,
runs cleanup steps. One of which was deleting the node and client entries
from the chef server.

Here’s some info on autoscaling events:

http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/ASGettingNotifications.html

Mike Glenney
New Context
On Feb 19, 2015 5:35 AM, “Rafał Radecki” radecki.rafal@gmail.com wrote:

Hi All :slight_smile:

I am using chef and autoscaling for aws ec2 instances. Installation of
chef and node’s registration process is handled mostly in user data so when
an instance is started because of autoscaling rule it properly registers in
chef.
When the instance is downscaling I would like to deregister it in chef
automatically. My problem is basically: how to run some code when aws
terminates an ec2 instance? I already saw some solution based on init
scripts run on shutdown but I think it is not the best choice :wink: I also
found info about terminating:wait functionality (
http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/AnalyzingAnAutoScalingInstance.html)
and am investigating it now.

Can someone point me in a proper direction? :slight_smile:

Thanks for the help!

BR,
Rafal.


#6

I just let my autoscaling nodes delete themselves as they terminate


On Feb 19, 2015 8:34 AM, “Rafał Radecki” radecki.rafal@gmail.com wrote:

Thanks for the tips :slight_smile: Notification hooks and SQS look very promising :slight_smile:
I assume that I would have to use aws-sdk for ruby (
https://aws.amazon.com/sdk-for-ruby/) in chef to connect to SQS and read
events from it? Are there any problems in using this sdk? Is it
stable/current? Are there any available howtos about SQS/aws-sdk/chef
integration?

BR,
Rafal.

2015-02-19 14:45 GMT+01:00 David Petzel davidpetzel@gmail.com:

This is how we handled it as well. We configured our autoscaling groups
such that a termination event was sent to SNS, which landed in an SQS queue.

We then had an “admin” node that would wake up every 60 seconds and check
the SQS queue for any pending termination notices, and it would then
process them by deleting the node and client objects from the Chef server,
as well as some internal housekeeping items.

On Thu, Feb 19, 2015 at 8:21 AM, Michael Glenney mike.glenney@gmail.com
wrote:

In the past I’ve done this with events and have a separate process that
receives the events and performs verification of them and, once verified,
runs cleanup steps. One of which was deleting the node and client entries
from the chef server.

Here’s some info on autoscaling events:

http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/ASGettingNotifications.html

Mike Glenney
New Context
On Feb 19, 2015 5:35 AM, “Rafał Radecki” radecki.rafal@gmail.com
wrote:

Hi All :slight_smile:

I am using chef and autoscaling for aws ec2 instances. Installation of
chef and node’s registration process is handled mostly in user data so when
an instance is started because of autoscaling rule it properly registers in
chef.
When the instance is downscaling I would like to deregister it in chef
automatically. My problem is basically: how to run some code when aws
terminates an ec2 instance? I already saw some solution based on init
scripts run on shutdown but I think it is not the best choice :wink: I also
found info about terminating:wait functionality (
http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/AnalyzingAnAutoScalingInstance.html)
and am investigating it now.

Can someone point me in a proper direction? :slight_smile:

Thanks for the help!

BR,
Rafal.