Newbie on autoscaling with Chef

Chef Infra (archive)
1

I’d like to use Chef to manage an application hosted on Amazon’s EC2
platform. There are three “components” to the appliction:

  1. Web application
  2. Database server
  3. Background worker/application

I’d like to have the 3rd component (background worker) automatically scale
based on CPU usage. So there might be one permanent background worker, but
if CPU levels on that worker remained above x% for over y seconds then a
couple of extra background workers would automatically be powered up.
Similarly, if CPU levels dropped below z% for y seconds, a couple of
background workers would automatically be terminated.

I think a major piece of the puzzle that is missing for me is how to
register new instances as nodes with the Chef server along with the
necessary cookbooks. All of the stuff I’ve read on using chef so far
appears to assume a static list of nodes that are being managed manually
from a workstation. What I’d like to do is have new nodes (not necessarily
workstations) register themselves to be configured by Chef… and also drop
off the list of managed nodes when they are terminated.

Is this possible using Chef?

Kind Regards,

James Crosswell
Founder | Tea Boy
www.mentaldesk.com

2

All of this is absolutely possible. It sounds complicated at first but when it’s broken down into the three different tasks it’s really not that bad:

Automatic scaling with ASGs is most effectively accomplished using custom metrics[1]. You can do this based off of the built in metrics for EC2 nodes for memory or CPU usage but normally queue depth is a more accurate measurement of when you need to bring up more workers. If you have a fixed number of workers running jobs they normally won’t dramatically increase CPU load on their own.

Registering the workers can be handled by either including your chef validator and client.rb (with the chef client preinstalled) in a custom AMI[2] or by using instance metadata to handle installing it[3]. Creating an AMI with your validator and chef-client already preinstalled would be the easiest to accomplish as the only metadata that needs to be passed in is a role or runlist.

Cleanup can be handled with shutdown hooks depending on the permissions on your chef server. At my org we use knife delete_[client,node] in orchestration scripts for cleaning up production/staging nodes but allow vagrant nodes to clean themselves up. There’s a good example for self-cleanup of nodes on cotap’s engineering blog [4] though if you’re using Ubutntu (we’re not) but a similar process can be built for any other distro.

[1] Autoscaling with custom metrics – That's Geeky
[2] Account Suspended
[3] aws advent
[4] Cotap Engineering — Instance Deregistration with Chef and Sensu

On Tuesday, November 26, 2013 at 2:25 AM, James Crosswell wrote:

I'd like to use Chef to manage an application hosted on Amazon's EC2 platform. There are three "components" to the appliction:
Web application
Database server
Background worker/application

I'd like to have the 3rd component (background worker) automatically scale based on CPU usage. So there might be one permanent background worker, but if CPU levels on that worker remained above x% for over y seconds then a couple of extra background workers would automatically be powered up. Similarly, if CPU levels dropped below z% for y seconds, a couple of background workers would automatically be terminated.

I think a major piece of the puzzle that is missing for me is how to register new instances as nodes with the Chef server along with the necessary cookbooks. All of the stuff I've read on using chef so far appears to assume a static list of nodes that are being managed manually from a workstation. What I'd like to do is have new nodes (not necessarily workstations) register themselves to be configured by Chef... and also drop off the list of managed nodes when they are terminated.

Is this possible using Chef?
Kind Regards,

James Crosswell
Founder | Tea Boy
www.mentaldesk.com (http://www.mentaldesk.com)

3

Hi,

I've just wrote sample code which let new servers register automatically to chef server and deregister when instances are terminated.

When creating autoscaling, you can set user_data at launch configuration and if user_data is set and user_data starts with #!, cloud-init will execute this user_data as executable script.

Another point is that it is not good to store validation key in the AMI. I recommend you to set IAM role to instances(which can be set at launch configuration) and you can get validation key from S3 bucket.

--
Ryutaro YOSHIBA

On 2013/11/26, at 19:25, James Crosswell james@mentaldesk.com wrote:

I'd like to use Chef to manage an application hosted on Amazon's EC2 platform. There are three "components" to the appliction:
Web application
Database server
Background worker/application
I'd like to have the 3rd component (background worker) automatically scale based on CPU usage. So there might be one permanent background worker, but if CPU levels on that worker remained above x% for over y seconds then a couple of extra background workers would automatically be powered up. Similarly, if CPU levels dropped below z% for y seconds, a couple of background workers would automatically be terminated.

I think a major piece of the puzzle that is missing for me is how to register new instances as nodes with the Chef server along with the necessary cookbooks. All of the stuff I've read on using chef so far appears to assume a static list of nodes that are being managed manually from a workstation. What I'd like to do is have new nodes (not necessarily workstations) register themselves to be configured by Chef... and also drop off the list of managed nodes when they are terminated.

Is this possible using Chef?

Kind Regards,

James Crosswell
Founder | Tea Boy
www.mentaldesk.com

4

Thanks guys. The nodes in my case are Windows nodes but I'm sure I can do
the same with Windows startup/shutdown scripts, until such a time as the
code can be ported to Linux. It looks like a fair amount of work will be
involved in setting this all up, but at least I know I'm not chasing wild
geese now :slight_smile:

Kind Regards,

James Crosswell
Founder | Tea Boy

On 26 November 2013 11:50, Daniel Condomitti daniel@condomitti.com wrote:

All of this is absolutely possible. It sounds complicated at first but
when it’s broken down into the three different tasks it’s really not that
bad:

Automatic scaling with ASGs is most effectively accomplished using custom
metrics[1]. You can do this based off of the built in metrics for EC2 nodes
for memory or CPU usage but normally queue depth is a more accurate
measurement of when you need to bring up more workers. If you have a fixed
number of workers running jobs they normally won’t dramatically increase
CPU load on their own.

Registering the workers can be handled by either including your chef
validator and client.rb (with the chef client preinstalled) in a custom
AMI[2] or by using instance metadata to handle installing it[3]. Creating
an AMI with your validator and chef-client already preinstalled would
be the easiest to accomplish as the only metadata that needs to be passed
in is a role or runlist.

Cleanup can be handled with shutdown hooks depending on the permissions on
your chef server. At my org we use knife delete_[client,node] in
orchestration scripts for cleaning up production/staging nodes but allow
vagrant nodes to clean themselves up. There’s a good example for
self-cleanup of nodes on cotap’s engineering blog [4] though if you’re
using Ubutntu (we’re not) but a similar process can be built for any other
distro.

[1] http://www.thatsgeeky.com/2012/01/autoscaling-with-custom-metrics/
[2]
Account Suspended
[3] aws advent
[4]
Cotap Engineering — Instance Deregistration with Chef and Sensu

On Tuesday, November 26, 2013 at 2:25 AM, James Crosswell wrote:

I'd like to use Chef to manage an application hosted on Amazon's EC2
platform. There are three "components" to the appliction:

  1. Web application
  2. Database server
  3. Background worker/application

I'd like to have the 3rd component (background worker) automatically scale
based on CPU usage. So there might be one permanent background worker, but
if CPU levels on that worker remained above x% for over y seconds then a
couple of extra background workers would automatically be powered up.
Similarly, if CPU levels dropped below z% for y seconds, a couple of
background workers would automatically be terminated.

I think a major piece of the puzzle that is missing for me is how to
register new instances as nodes with the Chef server along with the
necessary cookbooks. All of the stuff I've read on using chef so far
appears to assume a static list of nodes that are being managed manually
from a workstation. What I'd like to do is have new nodes (not necessarily
workstations) register themselves to be configured by Chef... and also drop
off the list of managed nodes when they are terminated.

Is this possible using Chef?

Kind Regards,

James Crosswell
Founder | Tea Boy
www.mentaldesk.com