Automatically starting a new EC2 instance with Chef


#1

Hi all,
What tools and services are available to automatically start a new EC2
instance with Chef?

My site has just one server and yesterday it failed. Unfortunately, I
was at my day job and couldn’t do anything about it. It’s configured
with Chef so starting up a new server with knife then using Opscode
Platform to put the roles on was reasonably easy (it could be easier
but I have some issues to fix).

What I would like is for some service to realise that my server is
down and then call Chef to setup a new one. Amazon Auto Scaling almost
does this - the issue is that it merely starts an AMI and doesn’t do
anything cleverer than that. My Chef setup assumes a brand new Ubuntu
image and goes from there. I suppose that, for future use, whatever
solution I use needs to also listen to the CloudWatch metrics to scale
up and down when required.

So:
Do I write an image which bootstraps chef automatically?
Or is there some other way to solve this problem?

Thanks,
Edward


#2

You can use AWS auto scaling. Create an AMI with chef-client on it and your
launch configuration for auto scaling should pass in the user data that you
would get from “knife ec2 instance data”. You can then run with a minimum
and desired instances of 1.

I would build a fully functional AMI that is fairly baked with your chef
run. This would minimize your downtime when that instance becomes
unhealthy. Chef then just acts as the enforcer of the configuration.

On Sat, May 14, 2011 at 8:55 AM, Edward Sargisson esarge@pobox.com wrote:

Hi all,
What tools and services are available to automatically start a new EC2
instance with Chef?

My site has just one server and yesterday it failed. Unfortunately, I
was at my day job and couldn’t do anything about it. It’s configured
with Chef so starting up a new server with knife then using Opscode
Platform to put the roles on was reasonably easy (it could be easier
but I have some issues to fix).

What I would like is for some service to realise that my server is
down and then call Chef to setup a new one. Amazon Auto Scaling almost
does this - the issue is that it merely starts an AMI and doesn’t do
anything cleverer than that. My Chef setup assumes a brand new Ubuntu
image and goes from there. I suppose that, for future use, whatever
solution I use needs to also listen to the CloudWatch metrics to scale
up and down when required.

So:
Do I write an image which bootstraps chef automatically?
Or is there some other way to solve this problem?

Thanks,
Edward


#3

If you want to achieve this goal with the minimum amount of work, create
a launch configuration with a user data script that installs ruby,
installs chef, write the validation certificate, chef config and
bootstrap json then launch chef. Ubuntu images are bundle with
cloud-init which is able to read shell scripts from user data. In other
words, edit the attached script and use that as the user data for new
instances or autoscaling groups.

BTW, a native chef plugin for cloud-init is waiting to be merged. If and
when it’s included, launching ubuntu images with chef will become
ridiculously easy.

Regards,
Avishai

On 14/05/11 16:55, Edward Sargisson wrote:

Hi all,
What tools and services are available to automatically start a new EC2
instance with Chef?

My site has just one server and yesterday it failed. Unfortunately, I
was at my day job and couldn’t do anything about it. It’s configured
with Chef so starting up a new server with knife then using Opscode
Platform to put the roles on was reasonably easy (it could be easier
but I have some issues to fix).

What I would like is for some service to realise that my server is
down and then call Chef to setup a new one. Amazon Auto Scaling almost
does this - the issue is that it merely starts an AMI and doesn’t do
anything cleverer than that. My Chef setup assumes a brand new Ubuntu
image and goes from there. I suppose that, for future use, whatever
solution I use needs to also listen to the CloudWatch metrics to scale
up and down when required.

So:
Do I write an image which bootstraps chef automatically?
Or is there some other way to solve this problem?

Thanks,
Edward


#4

Here’s how I do it:

On Sat, May 14, 2011 at 6:55 AM, Edward Sargisson esarge@pobox.com wrote:

Hi all,
What tools and services are available to automatically start a new EC2
instance with Chef?

My site has just one server and yesterday it failed. Unfortunately, I
was at my day job and couldn’t do anything about it. It’s configured
with Chef so starting up a new server with knife then using Opscode
Platform to put the roles on was reasonably easy (it could be easier
but I have some issues to fix).

What I would like is for some service to realise that my server is
down and then call Chef to setup a new one. Amazon Auto Scaling almost
does this - the issue is that it merely starts an AMI and doesn’t do
anything cleverer than that. My Chef setup assumes a brand new Ubuntu
image and goes from there. I suppose that, for future use, whatever
solution I use needs to also listen to the CloudWatch metrics to scale
up and down when required.

So:
Do I write an image which bootstraps chef automatically?
Or is there some other way to solve this problem?

Thanks,
Edward


#5

Hi all,
I followed the advice (and made great use of Avishai’s user data
script) and now have my site running so that AWS Elastic Load
Balancing will automatically detect an instance going down and bring
another back up - configured from the standard OS image using Chef.

I hope I don’t breach the standards of this list by including a link
to the blog post I’ve written which details all the steps.
http://www.trailhunger.com/blog/technical/2011/05/28/keeping-an-amazon-elastic-compute-cloud-ec2-instance-up-with-chef-and-auto-scaling/

Many thanks,
Edward

On Sat, May 14, 2011 at 12:36 PM, Avishai Ish-Shalom
avishai@fewbytes.com wrote:

If you want to achieve this goal with the minimum amount of work, create
a launch configuration with a user data script that installs ruby,
installs chef, write the validation certificate, chef config and
bootstrap json then launch chef. Ubuntu images are bundle with
cloud-init which is able to read shell scripts from user data. In other
words, edit the attached script and use that as the user data for new
instances or autoscaling groups.

BTW, a native chef plugin for cloud-init is waiting to be merged. If and
when it’s included, launching ubuntu images with chef will become
ridiculously easy.

Regards,
Avishai

On 14/05/11 16:55, Edward Sargisson wrote:

Hi all,
What tools and services are available to automatically start a new EC2
instance with Chef?

My site has just one server and yesterday it failed. Unfortunately, I
was at my day job and couldn’t do anything about it. It’s configured
with Chef so starting up a new server with knife then using Opscode
Platform to put the roles on was reasonably easy (it could be easier
but I have some issues to fix).

What I would like is for some service to realise that my server is
down and then call Chef to setup a new one. Amazon Auto Scaling almost
does this - the issue is that it merely starts an AMI and doesn’t do
anything cleverer than that. My Chef setup assumes a brand new Ubuntu
image and goes from there. I suppose that, for future use, whatever
solution I use needs to also listen to the CloudWatch metrics to scale
up and down when required.

So:
Do I write an image which bootstraps chef automatically?
Or is there some other way to solve this problem?

Thanks,
Edward


#6

Very nice. I presume that putting everything in the user data script is what makes ELB wait for the chef run to finish before it considers the instance in service?

I think your post may also answer a question I was about to pose to the group. I want to use chef/EC2 to build an on-demand regression test environment. When an instance gets tipped up, it doesn’t know what regression suite it will run, so it doesn’t know what software needs to be installed (the suite itself, plus needed libs and exes). The question is how to dynamically assign roles to nodes. If I understand your solution (and chef) correctly, whatever creates the user data script could dynamically set the value of role[] in the runlist. Then, when chef-client registers the instance with chef-server, it would say “hi, my name is regression-box-42, and my role is regression-suite-23”, which would cause chef to configure it with the appropriate software for “regression-suite-23”. Am I right?

On May 28, 2011, at 12:30 AM, Edward Sargisson wrote:

Hi all,
I followed the advice (and made great use of Avishai’s user data
script) and now have my site running so that AWS Elastic Load
Balancing will automatically detect an instance going down and bring
another back up - configured from the standard OS image using Chef.

I hope I don’t breach the standards of this list by including a link
to the blog post I’ve written which details all the steps.
http://www.trailhunger.com/blog/technical/2011/05/28/keeping-an-amazon-elastic-compute-cloud-ec2-instance-up-with-chef-and-auto-scaling/

Many thanks,
Edward

On Sat, May 14, 2011 at 12:36 PM, Avishai Ish-Shalom
avishai@fewbytes.com wrote:

If you want to achieve this goal with the minimum amount of work, create
a launch configuration with a user data script that installs ruby,
installs chef, write the validation certificate, chef config and
bootstrap json then launch chef. Ubuntu images are bundle with
cloud-init which is able to read shell scripts from user data. In other
words, edit the attached script and use that as the user data for new
instances or autoscaling groups.

BTW, a native chef plugin for cloud-init is waiting to be merged. If and
when it’s included, launching ubuntu images with chef will become
ridiculously easy.

Regards,
Avishai

On 14/05/11 16:55, Edward Sargisson wrote:

Hi all,
What tools and services are available to automatically start a new EC2
instance with Chef?

My site has just one server and yesterday it failed. Unfortunately, I
was at my day job and couldn’t do anything about it. It’s configured
with Chef so starting up a new server with knife then using Opscode
Platform to put the roles on was reasonably easy (it could be easier
but I have some issues to fix).

What I would like is for some service to realise that my server is
down and then call Chef to setup a new one. Amazon Auto Scaling almost
does this - the issue is that it merely starts an AMI and doesn’t do
anything cleverer than that. My Chef setup assumes a brand new Ubuntu
image and goes from there. I suppose that, for future use, whatever
solution I use needs to also listen to the CloudWatch metrics to scale
up and down when required.

So:
Do I write an image which bootstraps chef automatically?
Or is there some other way to solve this problem?

Thanks,
Edward


#7

Very nice. I presume that putting everything in the user data script is what makes ELB wait for the chef run to finish before it considers the instance in service?

No - the ELB is set with a grace period before it starts testing for
up-ness. Mine’s set for 12 minutes or something because a standard
chef run for my config is about 10 minutes.
However, for your requirements you probably don’t need the ELB unless
you particularly need load balancing in test.

The question is how to dynamically assign roles to nodes. If I understand your solution (and chef) correctly, whatever creates the user data script could dynamically set the value of role[] in the runlist. Then, when chef-client registers the instance with chef-server, it would say “hi, my name is regression-box-42, and my role is regression-suite-23”, which would cause chef to configure it with the appropriate software for “regression-suite-23”. Am I right?

Yes. If you look in the user data script you will see that it there is
a section where the roles are. At the end there is a setting to set
the environment on chef (-E qa).
So, for your requirements you could write a script to write the user
data script you want and then call the AWS command line client to
start it.

However, the easiest way would probably be to use knife bootstrap. You
can pass the run list into that - it will call AWS to launch the
instance, ssh in to configure chef then fire the chef-client.
Mind you, the user-data method has the advantage that you could write
in some node variables. i.e. setting a test version for your chef
config to load.

Feel free to ask more questions if I haven’t been clear.

Cheers,
Edward

On Tue, Jul 5, 2011 at 2:59 PM, Jeffrey E. Sussna jes@ingineering.it wrote:

Very nice. I presume that putting everything in the user data script is what makes ELB wait for the chef run to finish before it considers the instance in service?

I think your post may also answer a question I was about to pose to the group. I want to use chef/EC2 to build an on-demand regression test environment. When an instance gets tipped up, it doesn’t know what regression suite it will run, so it doesn’t know what software needs to be installed (the suite itself, plus needed libs and exes). The question is how to dynamically assign roles to nodes. If I understand your solution (and chef) correctly, whatever creates the user data script could dynamically set the value of role[] in the runlist. Then, when chef-client registers the instance with chef-server, it would say “hi, my name is regression-box-42, and my role is regression-suite-23”, which would cause chef to configure it with the appropriate software for “regression-suite-23”. Am I right?

On May 28, 2011, at 12:30 AM, Edward Sargisson wrote:

Hi all,
I followed the advice (and made great use of Avishai’s user data
script) and now have my site running so that AWS Elastic Load
Balancing will automatically detect an instance going down and bring
another back up - configured from the standard OS image using Chef.

I hope I don’t breach the standards of this list by including a link
to the blog post I’ve written which details all the steps.
http://www.trailhunger.com/blog/technical/2011/05/28/keeping-an-amazon-elastic-compute-cloud-ec2-instance-up-with-chef-and-auto-scaling/

Many thanks,
Edward

On Sat, May 14, 2011 at 12:36 PM, Avishai Ish-Shalom
avishai@fewbytes.com wrote:

If you want to achieve this goal with the minimum amount of work, create
a launch configuration with a user data script that installs ruby,
installs chef, write the validation certificate, chef config and
bootstrap json then launch chef. Ubuntu images are bundle with
cloud-init which is able to read shell scripts from user data. In other
words, edit the attached script and use that as the user data for new
instances or autoscaling groups.

BTW, a native chef plugin for cloud-init is waiting to be merged. If and
when it’s included, launching ubuntu images with chef will become
ridiculously easy.

Regards,
Avishai

On 14/05/11 16:55, Edward Sargisson wrote:

Hi all,
What tools and services are available to automatically start a new EC2
instance with Chef?

My site has just one server and yesterday it failed. Unfortunately, I
was at my day job and couldn’t do anything about it. It’s configured
with Chef so starting up a new server with knife then using Opscode
Platform to put the roles on was reasonably easy (it could be easier
but I have some issues to fix).

What I would like is for some service to realise that my server is
down and then call Chef to setup a new one. Amazon Auto Scaling almost
does this - the issue is that it merely starts an AMI and doesn’t do
anything cleverer than that. My Chef setup assumes a brand new Ubuntu
image and goes from there. I suppose that, for future use, whatever
solution I use needs to also listen to the CloudWatch metrics to scale
up and down when required.

So:
Do I write an image which bootstraps chef automatically?
Or is there some other way to solve this problem?

Thanks,
Edward


#8

Re ELB, I have other use cases in mind. Nice to be able to bring an instance up to date using Chef on an auto-scale event, instead of baking everything into the AMI. But relying on a grace period seems iffy.

Good to know about knife bootstrap.

On Jul 5, 2011, at 5:47 PM, Edward Sargisson wrote:

Very nice. I presume that putting everything in the user data script is what makes ELB wait for the chef run to finish before it considers the instance in service?

No - the ELB is set with a grace period before it starts testing for
up-ness. Mine’s set for 12 minutes or something because a standard
chef run for my config is about 10 minutes.
However, for your requirements you probably don’t need the ELB unless
you particularly need load balancing in test.

The question is how to dynamically assign roles to nodes. If I understand your solution (and chef) correctly, whatever creates the user data script could dynamically set the value of role[] in the runlist. Then, when chef-client registers the instance with chef-server, it would say “hi, my name is regression-box-42, and my role is regression-suite-23”, which would cause chef to configure it with the appropriate software for “regression-suite-23”. Am I right?

Yes. If you look in the user data script you will see that it there is
a section where the roles are. At the end there is a setting to set
the environment on chef (-E qa).
So, for your requirements you could write a script to write the user
data script you want and then call the AWS command line client to
start it.

However, the easiest way would probably be to use knife bootstrap. You
can pass the run list into that - it will call AWS to launch the
instance, ssh in to configure chef then fire the chef-client.
Mind you, the user-data method has the advantage that you could write
in some node variables. i.e. setting a test version for your chef
config to load.

Feel free to ask more questions if I haven’t been clear.

Cheers,
Edward

On Tue, Jul 5, 2011 at 2:59 PM, Jeffrey E. Sussna jes@ingineering.it wrote:

Very nice. I presume that putting everything in the user data script is what makes ELB wait for the chef run to finish before it considers the instance in service?

I think your post may also answer a question I was about to pose to the group. I want to use chef/EC2 to build an on-demand regression test environment. When an instance gets tipped up, it doesn’t know what regression suite it will run, so it doesn’t know what software needs to be installed (the suite itself, plus needed libs and exes). The question is how to dynamically assign roles to nodes. If I understand your solution (and chef) correctly, whatever creates the user data script could dynamically set the value of role[] in the runlist. Then, when chef-client registers the instance with chef-server, it would say “hi, my name is regression-box-42, and my role is regression-suite-23”, which would cause chef to configure it with the appropriate software for “regression-suite-23”. Am I right?

On May 28, 2011, at 12:30 AM, Edward Sargisson wrote:

Hi all,
I followed the advice (and made great use of Avishai’s user data
script) and now have my site running so that AWS Elastic Load
Balancing will automatically detect an instance going down and bring
another back up - configured from the standard OS image using Chef.

I hope I don’t breach the standards of this list by including a link
to the blog post I’ve written which details all the steps.
http://www.trailhunger.com/blog/technical/2011/05/28/keeping-an-amazon-elastic-compute-cloud-ec2-instance-up-with-chef-and-auto-scaling/

Many thanks,
Edward

On Sat, May 14, 2011 at 12:36 PM, Avishai Ish-Shalom
avishai@fewbytes.com wrote:

If you want to achieve this goal with the minimum amount of work, create
a launch configuration with a user data script that installs ruby,
installs chef, write the validation certificate, chef config and
bootstrap json then launch chef. Ubuntu images are bundle with
cloud-init which is able to read shell scripts from user data. In other
words, edit the attached script and use that as the user data for new
instances or autoscaling groups.

BTW, a native chef plugin for cloud-init is waiting to be merged. If and
when it’s included, launching ubuntu images with chef will become
ridiculously easy.

Regards,
Avishai

On 14/05/11 16:55, Edward Sargisson wrote:

Hi all,
What tools and services are available to automatically start a new EC2
instance with Chef?

My site has just one server and yesterday it failed. Unfortunately, I
was at my day job and couldn’t do anything about it. It’s configured
with Chef so starting up a new server with knife then using Opscode
Platform to put the roles on was reasonably easy (it could be easier
but I have some issues to fix).

What I would like is for some service to realise that my server is
down and then call Chef to setup a new one. Amazon Auto Scaling almost
does this - the issue is that it merely starts an AMI and doesn’t do
anything cleverer than that. My Chef setup assumes a brand new Ubuntu
image and goes from there. I suppose that, for future use, whatever
solution I use needs to also listen to the CloudWatch metrics to scale
up and down when required.

So:
Do I write an image which bootstraps chef automatically?
Or is there some other way to solve this problem?

Thanks,
Edward


#9

On Jul 5, 2011, at 5:47 PM, Edward Sargisson wrote:

Very nice. I presume that putting everything in the user data script is what makes ELB wait for the chef run to finish before it considers the instance in service?

No - the ELB is set with a grace period before it starts testing for
up-ness. Mine’s set for 12 minutes or something because a standard
chef run for my config is about 10 minutes.
However, for your requirements you probably don’t need the ELB unless
you particularly need load balancing in test.

Has anyone from the chef community talked to the Amazon folks about more robust ways for ELB to decide that an instance is fully up?


#10

For dynamic user-data, I use erubis templates.
I find that bootstrapping with user-data is far superior to knife
bootstrap in EC2 and when using autoscaling it’s really the only option.
As for ELB, if you use instance health checks configured to check the
application uri on the instance ELB will not use the instance if it’s
broken to the point healthcheck fail. If you absolutely have to make
sure chef has finished properly before the load balancer picks it up,
configure the autoscaling group WITHOUT elb then use an EC2 api call to
register the instance with the load balancer from a chef report handler.

Regards,
Avishai

On 06/07/11 03:06, Jeffrey E. Sussna wrote:

Re ELB, I have other use cases in mind. Nice to be able to bring an instance up to date using Chef on an auto-scale event, instead of baking everything into the AMI. But relying on a grace period seems iffy.

Good to know about knife bootstrap.

On Jul 5, 2011, at 5:47 PM, Edward Sargisson wrote:

Very nice. I presume that putting everything in the user data script is what makes ELB wait for the chef run to finish before it considers the instance in service?
No - the ELB is set with a grace period before it starts testing for
up-ness. Mine’s set for 12 minutes or something because a standard
chef run for my config is about 10 minutes.
However, for your requirements you probably don’t need the ELB unless
you particularly need load balancing in test.

The question is how to dynamically assign roles to nodes. If I understand your solution (and chef) correctly, whatever creates the user data script could dynamically set the value of role[] in the runlist. Then, when chef-client registers the instance with chef-server, it would say “hi, my name is regression-box-42, and my role is regression-suite-23”, which would cause chef to configure it with the appropriate software for “regression-suite-23”. Am I right?
Yes. If you look in the user data script you will see that it there is
a section where the roles are. At the end there is a setting to set
the environment on chef (-E qa).
So, for your requirements you could write a script to write the user
data script you want and then call the AWS command line client to
start it.

However, the easiest way would probably be to use knife bootstrap. You
can pass the run list into that - it will call AWS to launch the
instance, ssh in to configure chef then fire the chef-client.
Mind you, the user-data method has the advantage that you could write
in some node variables. i.e. setting a test version for your chef
config to load.

Feel free to ask more questions if I haven’t been clear.

Cheers,
Edward

On Tue, Jul 5, 2011 at 2:59 PM, Jeffrey E. Sussna jes@ingineering.it wrote:

Very nice. I presume that putting everything in the user data script is what makes ELB wait for the chef run to finish before it considers the instance in service?

I think your post may also answer a question I was about to pose to the group. I want to use chef/EC2 to build an on-demand regression test environment. When an instance gets tipped up, it doesn’t know what regression suite it will run, so it doesn’t know what software needs to be installed (the suite itself, plus needed libs and exes). The question is how to dynamically assign roles to nodes. If I understand your solution (and chef) correctly, whatever creates the user data script could dynamically set the value of role[] in the runlist. Then, when chef-client registers the instance with chef-server, it would say “hi, my name is regression-box-42, and my role is regression-suite-23”, which would cause chef to configure it with the appropriate software for “regression-suite-23”. Am I right?

On May 28, 2011, at 12:30 AM, Edward Sargisson wrote:

Hi all,
I followed the advice (and made great use of Avishai’s user data
script) and now have my site running so that AWS Elastic Load
Balancing will automatically detect an instance going down and bring
another back up - configured from the standard OS image using Chef.

I hope I don’t breach the standards of this list by including a link
to the blog post I’ve written which details all the steps.
http://www.trailhunger.com/blog/technical/2011/05/28/keeping-an-amazon-elastic-compute-cloud-ec2-instance-up-with-chef-and-auto-scaling/

Many thanks,
Edward

On Sat, May 14, 2011 at 12:36 PM, Avishai Ish-Shalom
avishai@fewbytes.com wrote:

If you want to achieve this goal with the minimum amount of work, create
a launch configuration with a user data script that installs ruby,
installs chef, write the validation certificate, chef config and
bootstrap json then launch chef. Ubuntu images are bundle with
cloud-init which is able to read shell scripts from user data. In other
words, edit the attached script and use that as the user data for new
instances or autoscaling groups.

BTW, a native chef plugin for cloud-init is waiting to be merged. If and
when it’s included, launching ubuntu images with chef will become
ridiculously easy.

Regards,
Avishai

On 14/05/11 16:55, Edward Sargisson wrote:

Hi all,
What tools and services are available to automatically start a new EC2
instance with Chef?

My site has just one server and yesterday it failed. Unfortunately, I
was at my day job and couldn’t do anything about it. It’s configured
with Chef so starting up a new server with knife then using Opscode
Platform to put the roles on was reasonably easy (it could be easier
but I have some issues to fix).

What I would like is for some service to realise that my server is
down and then call Chef to setup a new one. Amazon Auto Scaling almost
does this - the issue is that it merely starts an AMI and doesn’t do
anything cleverer than that. My Chef setup assumes a brand new Ubuntu
image and goes from there. I suppose that, for future use, whatever
solution I use needs to also listen to the CloudWatch metrics to scale
up and down when required.

So:
Do I write an image which bootstraps chef automatically?
Or is there some other way to solve this problem?

Thanks,
Edward


#11

Another way to deal with it is to suspend scaling operations while
your new instance comes up - then you re-enable them at the end.
Secondly, you could use Chef to configure an instance and then bake an
image of it. Then you can fire that instance up, which will be fast,
and it will automatically connect to the chef server and pull any
changes down.

To differentiate Avishai’s solution:
If you use the EC2 start instance API or knife to start an instance
then you can use the report handler to add the instance to the ELB.
If you rely on the ELB to detect that it’s down to start an instance
then you will need to rely on the grace period.

I haven’t tried it but potentially you could use Cloudwatch reporting
to send a message to a queue that started a new instance with
Avishai’s solution. That would probably be very reliable - i.e. not
relying on the non-determinism of waiting for the grace period to end.

Cheers,
Edward

On Wed, Jul 6, 2011 at 3:03 PM, Avishai Ish-Shalom avishai@fewbytes.com wrote:

For dynamic user-data, I use erubis templates.
I find that bootstrapping with user-data is far superior to knife
bootstrap in EC2 and when using autoscaling it’s really the only option.
As for ELB, if you use instance health checks configured to check the
application uri on the instance ELB will not use the instance if it’s
broken to the point healthcheck fail. If you absolutely have to make
sure chef has finished properly before the load balancer picks it up,
configure the autoscaling group WITHOUT elb then use an EC2 api call to
register the instance with the load balancer from a chef report handler.

Regards,
Avishai

On 06/07/11 03:06, Jeffrey E. Sussna wrote:

Re ELB, I have other use cases in mind. Nice to be able to bring an instance up to date using Chef on an auto-scale event, instead of baking everything into the AMI. But relying on a grace period seems iffy.

Good to know about knife bootstrap.

On Jul 5, 2011, at 5:47 PM, Edward Sargisson wrote:

Very nice. I presume that putting everything in the user data script is what makes ELB wait for the chef run to finish before it considers the instance in service?
No - the ELB is set with a grace period before it starts testing for
up-ness. Mine’s set for 12 minutes or something because a standard
chef run for my config is about 10 minutes.
However, for your requirements you probably don’t need the ELB unless
you particularly need load balancing in test.

The question is how to dynamically assign roles to nodes. If I understand your solution (and chef) correctly, whatever creates the user data script could dynamically set the value of role[] in the runlist. Then, when chef-client registers the instance with chef-server, it would say “hi, my name is regression-box-42, and my role is regression-suite-23”, which would cause chef to configure it with the appropriate software for “regression-suite-23”. Am I right?
Yes. If you look in the user data script you will see that it there is
a section where the roles are. At the end there is a setting to set
the environment on chef (-E qa).
So, for your requirements you could write a script to write the user
data script you want and then call the AWS command line client to
start it.

However, the easiest way would probably be to use knife bootstrap. You
can pass the run list into that - it will call AWS to launch the
instance, ssh in to configure chef then fire the chef-client.
Mind you, the user-data method has the advantage that you could write
in some node variables. i.e. setting a test version for your chef
config to load.

Feel free to ask more questions if I haven’t been clear.

Cheers,
Edward

On Tue, Jul 5, 2011 at 2:59 PM, Jeffrey E. Sussna jes@ingineering.it wrote:

Very nice. I presume that putting everything in the user data script is what makes ELB wait for the chef run to finish before it considers the instance in service?

I think your post may also answer a question I was about to pose to the group. I want to use chef/EC2 to build an on-demand regression test environment. When an instance gets tipped up, it doesn’t know what regression suite it will run, so it doesn’t know what software needs to be installed (the suite itself, plus needed libs and exes). The question is how to dynamically assign roles to nodes. If I understand your solution (and chef) correctly, whatever creates the user data script could dynamically set the value of role[] in the runlist. Then, when chef-client registers the instance with chef-server, it would say “hi, my name is regression-box-42, and my role is regression-suite-23”, which would cause chef to configure it with the appropriate software for “regression-suite-23”. Am I right?

On May 28, 2011, at 12:30 AM, Edward Sargisson wrote:

Hi all,
I followed the advice (and made great use of Avishai’s user data
script) and now have my site running so that AWS Elastic Load
Balancing will automatically detect an instance going down and bring
another back up - configured from the standard OS image using Chef.

I hope I don’t breach the standards of this list by including a link
to the blog post I’ve written which details all the steps.
http://www.trailhunger.com/blog/technical/2011/05/28/keeping-an-amazon-elastic-compute-cloud-ec2-instance-up-with-chef-and-auto-scaling/

Many thanks,
Edward

On Sat, May 14, 2011 at 12:36 PM, Avishai Ish-Shalom
avishai@fewbytes.com wrote:

If you want to achieve this goal with the minimum amount of work, create
a launch configuration with a user data script that installs ruby,
installs chef, write the validation certificate, chef config and
bootstrap json then launch chef. Ubuntu images are bundle with
cloud-init which is able to read shell scripts from user data. In other
words, edit the attached script and use that as the user data for new
instances or autoscaling groups.

BTW, a native chef plugin for cloud-init is waiting to be merged. If and
when it’s included, launching ubuntu images with chef will become
ridiculously easy.

Regards,
Avishai

On 14/05/11 16:55, Edward Sargisson wrote:

Hi all,
What tools and services are available to automatically start a new EC2
instance with Chef?

My site has just one server and yesterday it failed. Unfortunately, I
was at my day job and couldn’t do anything about it. It’s configured
with Chef so starting up a new server with knife then using Opscode
Platform to put the roles on was reasonably easy (it could be easier
but I have some issues to fix).

What I would like is for some service to realise that my server is
down and then call Chef to setup a new one. Amazon Auto Scaling almost
does this - the issue is that it merely starts an AMI and doesn’t do
anything cleverer than that. My Chef setup assumes a brand new Ubuntu
image and goes from there. I suppose that, for future use, whatever
solution I use needs to also listen to the CloudWatch metrics to scale
up and down when required.

So:
Do I write an image which bootstraps chef automatically?
Or is there some other way to solve this problem?

Thanks,
Edward


#12

This just in from Amazon (http://aws.typepad.com/aws/2011/07/auto-scaling-notifications-recurrence-and-more-control.html):

“With this release, you now have control of an additional process, AddToLoadBalancer. This can be particularly handy when newly launches EC2 instances must be initialized or verified in some way before they are ready to accept traffic.”

Sounds like just what the doctor ordered.

On Jul 6, 2011, at 6:45 PM, Edward Sargisson wrote:

Another way to deal with it is to suspend scaling operations while
your new instance comes up - then you re-enable them at the end.
Secondly, you could use Chef to configure an instance and then bake an
image of it. Then you can fire that instance up, which will be fast,
and it will automatically connect to the chef server and pull any
changes down.

To differentiate Avishai’s solution:
If you use the EC2 start instance API or knife to start an instance
then you can use the report handler to add the instance to the ELB.
If you rely on the ELB to detect that it’s down to start an instance
then you will need to rely on the grace period.

I haven’t tried it but potentially you could use Cloudwatch reporting
to send a message to a queue that started a new instance with
Avishai’s solution. That would probably be very reliable - i.e. not
relying on the non-determinism of waiting for the grace period to end.

Cheers,
Edward

On Wed, Jul 6, 2011 at 3:03 PM, Avishai Ish-Shalom avishai@fewbytes.com wrote:

For dynamic user-data, I use erubis templates.
I find that bootstrapping with user-data is far superior to knife
bootstrap in EC2 and when using autoscaling it’s really the only option.
As for ELB, if you use instance health checks configured to check the
application uri on the instance ELB will not use the instance if it’s
broken to the point healthcheck fail. If you absolutely have to make
sure chef has finished properly before the load balancer picks it up,
configure the autoscaling group WITHOUT elb then use an EC2 api call to
register the instance with the load balancer from a chef report handler.

Regards,
Avishai

On 06/07/11 03:06, Jeffrey E. Sussna wrote:

Re ELB, I have other use cases in mind. Nice to be able to bring an instance up to date using Chef on an auto-scale event, instead of baking everything into the AMI. But relying on a grace period seems iffy.

Good to know about knife bootstrap.

On Jul 5, 2011, at 5:47 PM, Edward Sargisson wrote:

Very nice. I presume that putting everything in the user data script is what makes ELB wait for the chef run to finish before it considers the instance in service?
No - the ELB is set with a grace period before it starts testing for
up-ness. Mine’s set for 12 minutes or something because a standard
chef run for my config is about 10 minutes.
However, for your requirements you probably don’t need the ELB unless
you particularly need load balancing in test.

The question is how to dynamically assign roles to nodes. If I understand your solution (and chef) correctly, whatever creates the user data script could dynamically set the value of role[] in the runlist. Then, when chef-client registers the instance with chef-server, it would say “hi, my name is regression-box-42, and my role is regression-suite-23”, which would cause chef to configure it with the appropriate software for “regression-suite-23”. Am I right?
Yes. If you look in the user data script you will see that it there is
a section where the roles are. At the end there is a setting to set
the environment on chef (-E qa).
So, for your requirements you could write a script to write the user
data script you want and then call the AWS command line client to
start it.

However, the easiest way would probably be to use knife bootstrap. You
can pass the run list into that - it will call AWS to launch the
instance, ssh in to configure chef then fire the chef-client.
Mind you, the user-data method has the advantage that you could write
in some node variables. i.e. setting a test version for your chef
config to load.

Feel free to ask more questions if I haven’t been clear.

Cheers,
Edward

On Tue, Jul 5, 2011 at 2:59 PM, Jeffrey E. Sussna jes@ingineering.it wrote:

Very nice. I presume that putting everything in the user data script is what makes ELB wait for the chef run to finish before it considers the instance in service?

I think your post may also answer a question I was about to pose to the group. I want to use chef/EC2 to build an on-demand regression test environment. When an instance gets tipped up, it doesn’t know what regression suite it will run, so it doesn’t know what software needs to be installed (the suite itself, plus needed libs and exes). The question is how to dynamically assign roles to nodes. If I understand your solution (and chef) correctly, whatever creates the user data script could dynamically set the value of role[] in the runlist. Then, when chef-client registers the instance with chef-server, it would say “hi, my name is regression-box-42, and my role is regression-suite-23”, which would cause chef to configure it with the appropriate software for “regression-suite-23”. Am I right?

On May 28, 2011, at 12:30 AM, Edward Sargisson wrote:

Hi all,
I followed the advice (and made great use of Avishai’s user data
script) and now have my site running so that AWS Elastic Load
Balancing will automatically detect an instance going down and bring
another back up - configured from the standard OS image using Chef.

I hope I don’t breach the standards of this list by including a link
to the blog post I’ve written which details all the steps.
http://www.trailhunger.com/blog/technical/2011/05/28/keeping-an-amazon-elastic-compute-cloud-ec2-instance-up-with-chef-and-auto-scaling/

Many thanks,
Edward

On Sat, May 14, 2011 at 12:36 PM, Avishai Ish-Shalom
avishai@fewbytes.com wrote:

If you want to achieve this goal with the minimum amount of work, create
a launch configuration with a user data script that installs ruby,
installs chef, write the validation certificate, chef config and
bootstrap json then launch chef. Ubuntu images are bundle with
cloud-init which is able to read shell scripts from user data. In other
words, edit the attached script and use that as the user data for new
instances or autoscaling groups.

BTW, a native chef plugin for cloud-init is waiting to be merged. If and
when it’s included, launching ubuntu images with chef will become
ridiculously easy.

Regards,
Avishai

On 14/05/11 16:55, Edward Sargisson wrote:

Hi all,
What tools and services are available to automatically start a new EC2
instance with Chef?

My site has just one server and yesterday it failed. Unfortunately, I
was at my day job and couldn’t do anything about it. It’s configured
with Chef so starting up a new server with knife then using Opscode
Platform to put the roles on was reasonably easy (it could be easier
but I have some issues to fix).

What I would like is for some service to realise that my server is
down and then call Chef to setup a new one. Amazon Auto Scaling almost
does this - the issue is that it merely starts an AMI and doesn’t do
anything cleverer than that. My Chef setup assumes a brand new Ubuntu
image and goes from there. I suppose that, for future use, whatever
solution I use needs to also listen to the CloudWatch metrics to scale
up and down when required.

So:
Do I write an image which bootstraps chef automatically?
Or is there some other way to solve this problem?

Thanks,
Edward