Knife ec2 plugin

Chef Infra (archive)
1

Hi all -

With knife’s ec2 plugin (which is awesome), I can create and instance from
an ami, but I cannot go the other way.

The use case is this:

virgin ubuntu 12.X ami -> knife clone/bootstrap with our product -> create
ami from this instance (for consumption elsewhere)

Before I start diving down this rabbit hole, is/was there a reason NOT to
do this? Time? Or was there a technical reason? Does this feature exist
and I don’t see it?

Just curious, seems like a good feature to have but don’t want to waste a
ton of time if there was a technical reason NOT to do this.

2

On Thursday, September 13, 2012 at 7:46 AM, Maven User wrote:

Hi all -

With knife's ec2 plugin (which is awesome), I can create and instance from an ami, but I cannot go the other way.

The use case is this:

virgin ubuntu 12.X ami -> knife clone/bootstrap with our product -> create ami from this instance (for consumption elsewhere)

Before I start diving down this rabbit hole, is/was there a reason NOT to do this? Time? Or was there a technical reason? Does this feature exist and I don't see it?

Just curious, seems like a good feature to have but don't want to waste a ton of time if there was a technical reason NOT to do this.
Creating an AMI is an involved process. Most people only follow the pattern you describe when their initial chef-client runs take a prohibitive amount of time and they need really fast instance creation for things like autoscaling.

--
Daniel DeLeo

3

On Sep 13, 2012, at 9:01 AM, Daniel DeLeo dan@kallistec.com wrote:

On Thursday, September 13, 2012 at 7:46 AM, Maven User wrote:

Hi all -

With knife's ec2 plugin (which is awesome), I can create and instance from an ami, but I cannot go the other way.

The use case is this:

virgin ubuntu 12.X ami -> knife clone/bootstrap with our product -> create ami from this instance (for consumption elsewhere)

Before I start diving down this rabbit hole, is/was there a reason NOT to do this? Time? Or was there a technical reason? Does this feature exist and I don't see it?

Just curious, seems like a good feature to have but don't want to waste a ton of time if there was a technical reason NOT to do this.
Creating an AMI is an involved process. Most people only follow the pattern you describe when their initial chef-client runs take a prohibitive amount of time and they need really fast instance creation for things like autoscaling.

Concur. From my experience, the two main reasons I use to create custom AMIs in AWS to use in conjunction with Chef:

To cut-down on instance boot up time (as described by Daniel)
You are using Autoscaling or CloudFormation where the knife API command can't be used

-john

4

Creating an AMI is tricky because there are 2 varieties of AMIs: 1 is the
instance-store, the other is the ebs-backed. They each require a different
set of actions. The instance-store requires that you place the amazon ssl
key and cert on the instance to be stamped, while the ebs method requires
that to get a consistent image you need to reboot the instance. Both
require that you then do some stuff after the snapshot is taken etc.

So it's not trivially simple, but the above will help you get started. I
have some ironfan-based clusters and recipes to make this work, but ironfan
isn't required to do this yourself. If you ping me I can probably provide
some guidance, but I'm short of bandwidth at the moment.

-Peter

On Thu, Sep 13, 2012 at 10:46 AM, Maven User maven.2.user@gmail.com wrote:

Hi all -

With knife's ec2 plugin (which is awesome), I can create and instance from
an ami, but I cannot go the other way.

The use case is this:

virgin ubuntu 12.X ami -> knife clone/bootstrap with our product -> create
ami from this instance (for consumption elsewhere)

Before I start diving down this rabbit hole, is/was there a reason NOT to
do this? Time? Or was there a technical reason? Does this feature exist
and I don't see it?

Just curious, seems like a good feature to have but don't want to waste a
ton of time if there was a technical reason NOT to do this.

5

Cool - thanks all for the feedback!

I’m new to most of this tech, so forgive my naivete.

In the AWS Console, I can right click on a running instance and create an
ami.

Essentially, what we’re doing with knife is perfecting a virginal Ubuntu
instance (apps/config/users/etc) then (currently) calling the AWS tools
(installed on build machines) to turn that instance into an AMI which in
turn we’re using Asgard to roll out to different ASGs.

If/when we want to make changes, we’ll roll out new AMI’s (the result of
our build process). The Instance that we create is transient (we’ll throw
it away when the AMI has been vetted).

We for sure want EBS backed for now.

Please keep the feedback coming!

6

One more recommendation: get familiar with the ec2-api tools.

On Sep 13, 2012, at 10:19 AM, Maven User maven.2.user@gmail.com wrote:

Cool - thanks all for the feedback!

I'm new to most of this tech, so forgive my naivete.

In the AWS Console, I can right click on a running instance and create an ami.

Essentially, what we're doing with knife is perfecting a virginal Ubuntu instance (apps/config/users/etc) then (currently) calling the AWS tools (installed on build machines) to turn that instance into an AMI which in turn we're using Asgard to roll out to different ASGs.

If/when we want to make changes, we'll roll out new AMI's (the result of our build process). The Instance that we create is transient (we'll throw it away when the AMI has been vetted).

We for sure want EBS backed for now.

Please keep the feedback coming!

7

Point 2 doesn't require a custom AMI if you don't want it to. You can do
autoscaling with a virgin AMI and use user-data bootstrapping to get
chef-client on there and running.

Details on my site here:
http://www.trailhunger.com/blog/technical/2011/05/28/keeping-an-amazon-elastic-compute-cloud-ec2-instance-up-with-chef-and-auto-scaling/

Cheers,
Edward

On Thu, Sep 13, 2012 at 9:15 AM, John Martinez john@johnmartinez.comwrote:

On Sep 13, 2012, at 9:01 AM, Daniel DeLeo dan@kallistec.com wrote:

On Thursday, September 13, 2012 at 7:46 AM, Maven User wrote:

Hi all -

With knife's ec2 plugin (which is awesome), I can create and instance from
an ami, but I cannot go the other way.

The use case is this:

virgin ubuntu 12.X ami -> knife clone/bootstrap with our product -> create
ami from this instance (for consumption elsewhere)

Before I start diving down this rabbit hole, is/was there a reason NOT to
do this? Time? Or was there a technical reason? Does this feature exist
and I don't see it?

Just curious, seems like a good feature to have but don't want to waste a
ton of time if there was a technical reason NOT to do this.

Creating an AMI is an involved process. Most people only follow the
pattern you describe when their initial chef-client runs take a prohibitive
amount of time and they need really fast instance creation for things like
autoscaling.

Concur. From my experience, the two main reasons I use to create custom
AMIs in AWS to use in conjunction with Chef:

  1. To cut-down on instance boot up time (as described by Daniel)
  2. You are using Autoscaling or CloudFormation where the knife API
    command can't be used

-john

8

I've posted a similar method here: https://gist.github.com/3722252

This uses cloud-init rather than a custom script (pros and cons there) in
the user-data and works on recent Ubuntu AMIs. I believe Amazon have also
forked cloud-init to make it work on Amazon Linux, CentOS, etc. I've only
ever used it with Ubuntu though.

Cheers

Jim

On 14 September 2012 15:04, Edward Sargisson esarge@pobox.com wrote:

Point 2 doesn't require a custom AMI if you don't want it to. You can do
autoscaling with a virgin AMI and use user-data bootstrapping to get
chef-client on there and running.

Details on my site here:
http://www.trailhunger.com/blog/technical/2011/05/28/keeping-an-amazon-elastic-compute-cloud-ec2-instance-up-with-chef-and-auto-scaling/

Cheers,
Edward

On Thu, Sep 13, 2012 at 9:15 AM, John Martinez john@johnmartinez.comwrote:

On Sep 13, 2012, at 9:01 AM, Daniel DeLeo dan@kallistec.com wrote:

On Thursday, September 13, 2012 at 7:46 AM, Maven User wrote:

Hi all -

With knife's ec2 plugin (which is awesome), I can create and instance
from an ami, but I cannot go the other way.

The use case is this:

virgin ubuntu 12.X ami -> knife clone/bootstrap with our product ->
create ami from this instance (for consumption elsewhere)

Before I start diving down this rabbit hole, is/was there a reason NOT to
do this? Time? Or was there a technical reason? Does this feature exist
and I don't see it?

Just curious, seems like a good feature to have but don't want to waste a
ton of time if there was a technical reason NOT to do this.

Creating an AMI is an involved process. Most people only follow the
pattern you describe when their initial chef-client runs take a prohibitive
amount of time and they need really fast instance creation for things like
autoscaling.

Concur. From my experience, the two main reasons I use to create custom
AMIs in AWS to use in conjunction with Chef:

  1. To cut-down on instance boot up time (as described by Daniel)
  2. You are using Autoscaling or CloudFormation where the knife API
    command can't be used

-john

--

Jim Croft
Systems Developer, Cloudreach Limited

Support Cloudreach at ByteNite http://goo.gl/dWWyT
Case Study: Transforming business at BRI http://goo.gl/ng4LJ

Cloudreach Limited is a limited company registered in England with registered number 06975407

The above terms reflect a potential business arrangement, are provided solely as a basis for further discussion,
and are not intended to be and do not constitute a legally binding obligation. No legally binding obligations
will be created, implied, or inferred until an agreement in final form is executed in writing by all parties involved.

This email may be confidential or privileged. If you received this communication by mistake, please don't forward
it to anyone else, please erase all copies and attachments, and please let us know that it has gone to the wrong person.