# Down scaling instances in AWS - how to run code on instance termination? **URL:** \ **Category:** Chef Infra (archive)\ **Created:** [February 19, 2015, 12:35pm UTC](https://discourse.chef.io/t/down-scaling-instances-in-aws-how-to-run-code-on-instance-termination/6327 "2015-02-19T12:35:36Z")\ **Posts on this page:** 6\ **Page:** 1

**Author:** ![Rafal\_Radecki](https://avatars.discourse-cdn.com/v4/letter/r/bcef8e/32.png) [@Rafal\_Radecki](https://discourse.chef.io/u/Rafal_Radecki)\ **Post date:** [February 19, 2015, 12:35pm UTC](https://discourse.chef.io/t/down-scaling-instances-in-aws-how-to-run-code-on-instance-termination/6327/1 "2015-02-19T12:35:36Z")

Hi All 🙂 I am using chef and autoscaling for aws ec2 instances. Installation of chef and node’s registration process is handled mostly in user data so when an instance is started because of autoscaling rule it properly registers in chef. When the instance is downscaling I would like to deregister it in chef automatically. My problem is basically: how to run some code when aws terminates an ec2 instance? I already saw some solution based on init scripts run on shutdown but I think it is not the best choice 😉 I also found info about terminating:wait functionality ( [http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/AnalyzingAnAutoScalingInstance.html](http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/AnalyzingAnAutoScalingInstance.html)) and am investigating it now. Can someone point me in a proper direction? 🙂 Thanks for the help! BR, Rafal. ---

**Author:** ![Michael\_Glenney](https://avatars.discourse-cdn.com/v4/letter/m/a9adbd/32.png) [@Michael\_Glenney](https://discourse.chef.io/u/Michael_Glenney)\ **Post date:** [February 19, 2015, 1:21pm UTC](https://discourse.chef.io/t/down-scaling-instances-in-aws-how-to-run-code-on-instance-termination/6327/2 "2015-02-19T13:21:14Z")

In the past I've done this with events and have a separate process that receives the events and performs verification of them and, once verified, runs cleanup steps. One of which was deleting the node and client entries from the chef server. Here's some info on autoscaling events: > **[What is Amazon EC2 Auto Scaling? - Amazon EC2 Auto Scaling](https://docs.aws.amazon.com/autoscaling/ec2/userguide/what-is-amazon-ec2-auto-scaling.html)** > > Automatically launch or terminate EC2 instances based on user-defined policies, health status checks, and schedules using Amazon EC2 Auto Scaling. Mike Glenney New Context On Feb 19, 2015 5:35 AM, "Rafał Radecki" [radecki.rafal@gmail.com](mailto:radecki.rafal@gmail.com) wrote: > Hi All 🙂 > > I am using chef and autoscaling for aws ec2 instances. Installation of > chef and node's registration process is handled mostly in user data so when > an instance is started because of autoscaling rule it properly registers in > chef. > When the instance is downscaling I would like to deregister it in chef > automatically. My problem is basically: how to run some code when aws > terminates an ec2 instance? I already saw some solution based on init > scripts run on shutdown but I think it is not the best choice 😉 I also > found info about terminating:wait functionality ( > [What is Amazon EC2 Auto Scaling? - Amazon EC2 Auto Scaling](http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/AnalyzingAnAutoScalingInstance.html)) > and am investigating it now. > > Can someone point me in a proper direction? 🙂 > > Thanks for the help! > > BR, > Rafal. ---

**Author:** ![Michel\_Blankleder](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/michel_blankleder/32/120_2.png) [@Michel\_Blankleder](https://discourse.chef.io/u/Michel_Blankleder)\ **Post date:** [February 19, 2015, 1:26pm UTC](https://discourse.chef.io/t/down-scaling-instances-in-aws-how-to-run-code-on-instance-termination/6327/3 "2015-02-19T13:26:28Z")

Hi, Basically you need to create a script that wraps the knife delete command. Put it on the /etc/init.d/ and then link it to the /etc/rc0.d/ Hope it helps Michel On Thu, Feb 19, 2015 at 10:35 AM, Rafał Radecki [radecki.rafal@gmail.com](mailto:radecki.rafal@gmail.com) wrote: > Hi All 🙂 > > I am using chef and autoscaling for aws ec2 instances. Installation of > chef and node's registration process is handled mostly in user data so when > an instance is started because of autoscaling rule it properly registers in > chef. > When the instance is downscaling I would like to deregister it in chef > automatically. My problem is basically: how to run some code when aws > terminates an ec2 instance? I already saw some solution based on init > scripts run on shutdown but I think it is not the best choice 😉 I also > found info about terminating:wait functionality ( > [What is Amazon EC2 Auto Scaling? - Amazon EC2 Auto Scaling](http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/AnalyzingAnAutoScalingInstance.html)) > and am investigating it now. > > Can someone point me in a proper direction? 🙂 > > Thanks for the help! > > BR, > Rafal. ---

**Author:** ![David\_Petzel](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/david_petzel/32/38_2.png) [@David\_Petzel](https://discourse.chef.io/u/David_Petzel)\ **Post date:** [February 19, 2015, 1:45pm UTC](https://discourse.chef.io/t/down-scaling-instances-in-aws-how-to-run-code-on-instance-termination/6327/4 "2015-02-19T13:45:02Z")

This is how we handled it as well. We configured our autoscaling groups such that a termination event was sent to SNS, which landed in an SQS queue. We then had an "admin" node that would wake up every 60 seconds and check the SQS queue for any pending termination notices, and it would then process them by deleting the node and client objects from the Chef server, as well as some internal housekeeping items. On Thu, Feb 19, 2015 at 8:21 AM, Michael Glenney [mike.glenney@gmail.com](mailto:mike.glenney@gmail.com) wrote: > In the past I've done this with events and have a separate process that > receives the events and performs verification of them and, once verified, > runs cleanup steps. One of which was deleting the node and client entries > from the chef server. > > Here's some info on autoscaling events: > > [What is Amazon EC2 Auto Scaling? - Amazon EC2 Auto Scaling](http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/ASGettingNotifications.html) > > Mike Glenney > New Context > On Feb 19, 2015 5:35 AM, "Rafał Radecki" [radecki.rafal@gmail.com](mailto:radecki.rafal@gmail.com) wrote: > > > Hi All 🙂 > > > > I am using chef and autoscaling for aws ec2 instances. Installation of > > chef and node's registration process is handled mostly in user data so when > > an instance is started because of autoscaling rule it properly registers in > > chef. > > When the instance is downscaling I would like to deregister it in chef > > automatically. My problem is basically: how to run some code when aws > > terminates an ec2 instance? I already saw some solution based on init > > scripts run on shutdown but I think it is not the best choice 😉 I also > > found info about terminating:wait functionality ( > > [What is Amazon EC2 Auto Scaling? - Amazon EC2 Auto Scaling](http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/AnalyzingAnAutoScalingInstance.html)) > > and am investigating it now. > > > > Can someone point me in a proper direction? 🙂 > > > > Thanks for the help! > > > > BR, > > Rafal. ---

**Author:** ![Rafal\_Radecki](https://avatars.discourse-cdn.com/v4/letter/r/bcef8e/32.png) [@Rafal\_Radecki](https://discourse.chef.io/u/Rafal_Radecki)\ **Post date:** [February 19, 2015, 3:33pm UTC](https://discourse.chef.io/t/down-scaling-instances-in-aws-how-to-run-code-on-instance-termination/6327/5 "2015-02-19T15:33:34Z")

Thanks for the tips 🙂 Notification hooks and SQS look very promising 🙂 I assume that I would have to use aws-sdk for ruby ( [AWS SDK for Ruby](https://aws.amazon.com/sdk-for-ruby/)) in chef to connect to SQS and read events from it? Are there any problems in using this sdk? Is it stable/current? Are there any available howtos about SQS/aws-sdk/chef integration? BR, Rafal. 2015-02-19 14:45 GMT+01:00 David Petzel [davidpetzel@gmail.com](mailto:davidpetzel@gmail.com): > This is how we handled it as well. We configured our autoscaling groups > such that a termination event was sent to SNS, which landed in an SQS queue. > > We then had an "admin" node that would wake up every 60 seconds and check > the SQS queue for any pending termination notices, and it would then > process them by deleting the node and client objects from the Chef server, > as well as some internal housekeeping items. > > On Thu, Feb 19, 2015 at 8:21 AM, Michael Glenney [mike.glenney@gmail.com](mailto:mike.glenney@gmail.com) > wrote: > > > In the past I've done this with events and have a separate process that > > receives the events and performs verification of them and, once verified, > > runs cleanup steps. One of which was deleting the node and client entries > > from the chef server. > > > > Here's some info on autoscaling events: > > > > [What is Amazon EC2 Auto Scaling? - Amazon EC2 Auto Scaling](http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/ASGettingNotifications.html) > > > > Mike Glenney > > New Context > > On Feb 19, 2015 5:35 AM, "Rafał Radecki" [radecki.rafal@gmail.com](mailto:radecki.rafal@gmail.com) wrote: > > > > > Hi All 🙂 > > > > > > I am using chef and autoscaling for aws ec2 instances. Installation of > > > chef and node's registration process is handled mostly in user data so when > > > an instance is started because of autoscaling rule it properly registers in > > > chef. > > > When the instance is downscaling I would like to deregister it in chef > > > automatically. My problem is basically: how to run some code when aws > > > terminates an ec2 instance? I already saw some solution based on init > > > scripts run on shutdown but I think it is not the best choice 😉 I also > > > found info about terminating:wait functionality ( > > > [What is Amazon EC2 Auto Scaling? - Amazon EC2 Auto Scaling](http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/AnalyzingAnAutoScalingInstance.html)) > > > and am investigating it now. > > > > > > Can someone point me in a proper direction? 🙂 > > > > > > Thanks for the help! > > > > > > BR, > > > Rafal. ---

**Author:** ![Jake\_Plimack](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/jake_plimack/32/145_2.png) [@Jake\_Plimack](https://discourse.chef.io/u/Jake_Plimack)\ **Post date:** [February 19, 2015, 3:37pm UTC](https://discourse.chef.io/t/down-scaling-instances-in-aws-how-to-run-code-on-instance-termination/6327/6 "2015-02-19T15:37:38Z")

I just let my autoscaling nodes delete themselves as they terminate > **[GitHub - kplimack/aws-asg-chefserver: init scripts to register and deregister...](https://github.com/kplimack/aws-asg-chefserver)** > > init scripts to register and deregister nodes with your chef-server - GitHub - kplimack/aws-asg-chefserver: init scripts to register and deregister nodes with your chef-server On Feb 19, 2015 8:34 AM, "Rafał Radecki" [radecki.rafal@gmail.com](mailto:radecki.rafal@gmail.com) wrote: > Thanks for the tips 🙂 Notification hooks and SQS look very promising 🙂 > I assume that I would have to use aws-sdk for ruby ( > [AWS SDK for Ruby](https://aws.amazon.com/sdk-for-ruby/)) in chef to connect to SQS and read > events from it? Are there any problems in using this sdk? Is it > stable/current? Are there any available howtos about SQS/aws-sdk/chef > integration? > > BR, > Rafal. > > 2015-02-19 14:45 GMT+01:00 David Petzel [davidpetzel@gmail.com](mailto:davidpetzel@gmail.com): > > > This is how we handled it as well. We configured our autoscaling groups > > such that a termination event was sent to SNS, which landed in an SQS queue. > > > > We then had an "admin" node that would wake up every 60 seconds and check > > the SQS queue for any pending termination notices, and it would then > > process them by deleting the node and client objects from the Chef server, > > as well as some internal housekeeping items. > > > > On Thu, Feb 19, 2015 at 8:21 AM, Michael Glenney [mike.glenney@gmail.com](mailto:mike.glenney@gmail.com) > > wrote: > > > > > In the past I've done this with events and have a separate process that > > > receives the events and performs verification of them and, once verified, > > > runs cleanup steps. One of which was deleting the node and client entries > > > from the chef server. > > > > > > Here's some info on autoscaling events: > > > > > > [What is Amazon EC2 Auto Scaling? - Amazon EC2 Auto Scaling](http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/ASGettingNotifications.html) > > > > > > Mike Glenney > > > New Context > > > On Feb 19, 2015 5:35 AM, "Rafał Radecki" [radecki.rafal@gmail.com](mailto:radecki.rafal@gmail.com) > > > wrote: > > > > > > > Hi All 🙂 > > > > > > > > I am using chef and autoscaling for aws ec2 instances. Installation of > > > > chef and node's registration process is handled mostly in user data so when > > > > an instance is started because of autoscaling rule it properly registers in > > > > chef. > > > > When the instance is downscaling I would like to deregister it in chef > > > > automatically. My problem is basically: how to run some code when aws > > > > terminates an ec2 instance? I already saw some solution based on init > > > > scripts run on shutdown but I think it is not the best choice 😉 I also > > > > found info about terminating:wait functionality ( > > > > [What is Amazon EC2 Auto Scaling? - Amazon EC2 Auto Scaling](http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/AnalyzingAnAutoScalingInstance.html)) > > > > and am investigating it now. > > > > > > > > Can someone point me in a proper direction? 🙂 > > > > > > > > Thanks for the help! > > > > > > > > BR, > > > > Rafal.