Auto scaling with chef questions

Chef Infra (archive)
1

The company I am working for is moving to AWS. I want to be able to have Chef pre-installed on our application server AMI’s so when they start up they will be configured correctly.

If one AMI can create many servers, how does the Chef client give them different client names?
How does Chef know to remove them when the server is deleted?

Thanks!
Bridger Larson

2

I had been thinking about the very same thing for a while, but haven't had the time to run some tests.

My guess so far is that every new machine that spins up will run chef-client, create a node and a client using the validator key and using the ec2-****** FQDN as client/node name. Have you tried running 'hostname -f' on an EC2 server?

--
Cassiano Leal

On Friday, December 7, 2012 at 21:43, Bridger Larson wrote:

The company I am working for is moving to AWS. I want to be able to have Chef pre-installed on our application server AMI’s so when they start up they will be configured correctly.

If one AMI can create many servers, how does the Chef client give them different client names?
How does Chef know to remove them when the server is deleted?

Thanks!
Bridger Larson

3

unless explicitly mentioned (via the -N flag) chef client will register new
node name as fqdn.
for the second question: no. chef does not delete the client and node
automatically when you terminate the ec2 instance. but knife-ec2 address
this.
regards
ranjib

On Fri, Dec 7, 2012 at 3:50 PM, Cassiano Leal cassianoleal@gmail.comwrote:

I had been thinking about the very same thing for a while, but haven't
had the time to run some tests.

My guess so far is that every new machine that spins up will run
chef-client, create a node and a client using the validator key and using
the ec2-****** FQDN as client/node name. Have you tried running 'hostname
-f' on an EC2 server?

--
Cassiano Leal

On Friday, December 7, 2012 at 21:43, Bridger Larson wrote:

The company I am working for is moving to AWS. I want to be able to
have Chef pre-installed on our application server AMI’s so when they start
up they will be configured correctly. ****

If one AMI can create many servers, how does the Chef client give them
different client names?****

How does Chef know to remove them when the server is deleted?****

Thanks!****

Bridger Larson****

4

In autoscaling groups you set a launch config which can contain a first-run.json that you can include to the chef-client.

{ "run_list" : "role[my_service]" }

chef-client -j /etc/chef/first-run.json

--
Jeffrey Hulten
Principal Consultant at Automated Labs, LLC
jeffh@automatedlabs.com 206-853-5216
Skype: jeffhulten

On Dec 7, 2012, at 3:50 PM, Cassiano Leal wrote:

I had been thinking about the very same thing for a while, but haven't had the time to run some tests.

My guess so far is that every new machine that spins up will run chef-client, create a node and a client using the validator key and using the ec2-****** FQDN as client/node name. Have you tried running 'hostname -f' on an EC2 server?

--
Cassiano Leal

On Friday, December 7, 2012 at 21:43, Bridger Larson wrote:

The company I am working for is moving to AWS. I want to be able to have Chef pre-installed on our application server AMI’s so when they start up they will be configured correctly.

If one AMI can create many servers, how does the Chef client give them different client names?
How does Chef know to remove them when the server is deleted?

Thanks!
Bridger Larson

5

Right now, we are using customized user data via the AS launch
configuration to insert instance IDs (collected from EC2's internal
metadata service) into the Chef node name.

--
Hector

On Fri, Dec 7, 2012 at 7:00 PM, Jeffrey Hulten jeffh@automatedlabs.com wrote:

In autoscaling groups you set a launch config which can contain a first-run.json that you can include to the chef-client.

{ "run_list" : "role[my_service]" }

chef-client -j /etc/chef/first-run.json

--
Jeffrey Hulten
Principal Consultant at Automated Labs, LLC
jeffh@automatedlabs.com 206-853-5216
Skype: jeffhulten

On Dec 7, 2012, at 3:50 PM, Cassiano Leal wrote:

I had been thinking about the very same thing for a while, but haven't had the time to run some tests.

My guess so far is that every new machine that spins up will run chef-client, create a node and a client using the validator key and using the ec2-****** FQDN as client/node name. Have you tried running 'hostname -f' on an EC2 server?

--
Cassiano Leal

On Friday, December 7, 2012 at 21:43, Bridger Larson wrote:

The company I am working for is moving to AWS. I want to be able to have Chef pre-installed on our application server AMI’s so when they start up they will be configured correctly.

If one AMI can create many servers, how does the Chef client give them different client names?
How does Chef know to remove them when the server is deleted?

Thanks!
Bridger Larson

6

I'd suggest using instance ID's as it's the main unique value you'll get
from AWS. On instance deregistration send a message to SQS. Have a
process on some stable node (the chef server?) that polls SQS and then
removes the node and client from the chef server.

I don't like passing in user data. We did that originally and it turned
out to be something we regretted and migrated away from. As Jeff pointed
out, use the first-run.json. Have a different AMI for each server role
type. Save yourself even potential future issues by fully baking the AMI
with everything on it. In run-state you'll still use chef but as the
enforcer of the state. If you have a critical service, you'll be thankful
at some point in time that you have fully baked AMI's if you want to scale.
If you're chef server fails, an external dependency fails or misc failure
happens you wouldn't be able to scale. Depending on how long your chef run
is you'll also save yourself lots of time waiting for nodes to come up when
you need them the most (you're scaling up for a reason).

Bryan
@agent462
Sr Manager of Operations, bestbuy.com

On Sun, Dec 9, 2012 at 3:05 PM, Hector Castro hectcastro@gmail.com wrote:

Right now, we are using customized user data via the AS launch
configuration to insert instance IDs (collected from EC2's internal
metadata service) into the Chef node name.

--
Hector

On Fri, Dec 7, 2012 at 7:00 PM, Jeffrey Hulten jeffh@automatedlabs.com
wrote:

In autoscaling groups you set a launch config which can contain a
first-run.json that you can include to the chef-client.

{ "run_list" : "role[my_service]" }

chef-client -j /etc/chef/first-run.json

--
Jeffrey Hulten
Principal Consultant at Automated Labs, LLC
jeffh@automatedlabs.com 206-853-5216
Skype: jeffhulten

On Dec 7, 2012, at 3:50 PM, Cassiano Leal wrote:

I had been thinking about the very same thing for a while, but haven't
had the time to run some tests.

My guess so far is that every new machine that spins up will run
chef-client, create a node and a client using the validator key and using
the ec2-****** FQDN as client/node name. Have you tried running 'hostname
-f' on an EC2 server?

--
Cassiano Leal

On Friday, December 7, 2012 at 21:43, Bridger Larson wrote:

The company I am working for is moving to AWS. I want to be able to
have Chef pre-installed on our application server AMI’s so when they start
up they will be configured correctly.

If one AMI can create many servers, how does the Chef client give them
different client names?
How does Chef know to remove them when the server is deleted?

Thanks!
Bridger Larson

7

That is what I was looking for. Thank you very much!

BRIDGER LARSON | Associate Build/Release Engineer | SirsiDynix
P: 801.223.5345 | bridger.larson@sirsidynix.commailto:bridger.larson@sirsidynix.com
www.sirsidynix.comhttp://www.sirsidynix.com/ | TOLL FREE: +1 800.284.3969

From: Ranjib Dey [mailto:dey.ranjib@gmail.com]
Sent: Friday, December 07, 2012 4:56 PM
To: chef@lists.opscode.com
Subject: [chef] Re: Re: Auto scaling with chef questions

unless explicitly mentioned (via the -N flag) chef client will register new node name as fqdn.
for the second question: no. chef does not delete the client and node automatically when you terminate the ec2 instance. but knife-ec2 address this.
regards
ranjib

On Fri, Dec 7, 2012 at 3:50 PM, Cassiano Leal <cassianoleal@gmail.commailto:cassianoleal@gmail.com> wrote:
I had been thinking about the very same thing for a while, but haven’t had the time to run some tests.

My guess so far is that every new machine that spins up will run chef-client, create a node and a client using the validator key and using the ec2-****** FQDN as client/node name. Have you tried running ‘hostname -f’ on an EC2 server?

–
Cassiano Leal

On Friday, December 7, 2012 at 21:43, Bridger Larson wrote:

The company I am working for is moving to AWS. I want to be able to have Chef pre-installed on our application server AMI’s so when they start up they will be configured correctly.

If one AMI can create many servers, how does the Chef client give them different client names?

How does Chef know to remove them when the server is deleted?

Thanks!

Bridger Larson

8

Does the first-run.json only run once or is it every time you specify it with "-j"?

BRIDGER LARSON | Associate Build/Release Engineer | SirsiDynix
P: 801.223.5345 | bridger.larson@sirsidynix.com
www.sirsidynix.com | TOLL FREE: +1 800.284.3969

-----Original Message-----
From: Jeffrey Hulten [mailto:jeffh@automatedlabs.com]
Sent: Friday, December 07, 2012 5:00 PM
To: chef@lists.opscode.com
Subject: [chef] Re: Auto scaling with chef questions

In autoscaling groups you set a launch config which can contain a first-run.json that you can include to the chef-client.

{ "run_list" : "role[my_service]" }

chef-client -j /etc/chef/first-run.json

--
Jeffrey Hulten
Principal Consultant at Automated Labs, LLC jeffh@automatedlabs.com 206-853-5216
Skype: jeffhulten

On Dec 7, 2012, at 3:50 PM, Cassiano Leal wrote:

I had been thinking about the very same thing for a while, but haven't had the time to run some tests.

My guess so far is that every new machine that spins up will run chef-client, create a node and a client using the validator key and using the ec2-****** FQDN as client/node name. Have you tried running 'hostname -f' on an EC2 server?

--
Cassiano Leal

On Friday, December 7, 2012 at 21:43, Bridger Larson wrote:

The company I am working for is moving to AWS. I want to be able to have Chef pre-installed on our application server AMI's so when they start up they will be configured correctly.

If one AMI can create many servers, how does the Chef client give them different client names?
How does Chef know to remove them when the server is deleted?

Thanks!
Bridger Larson

9

Every time you specify "-j" option. Also, as previously mentioned,
you'll need to clean up node and client on termination. I've written
my own shutdown script for this, and don't care if it's a reboot,
since the script to build up a node will recreate it on the server on
boot-up. We terminate more instances than reboot by A LOT due to
autoscale.

-john

On Dec 10, 2012, at 7:36 AM, Bridger Larson
Bridger.Larson@sirsidynix.com wrote:

Does the first-run.json only run once or is it every time you specify it with "-j"?

BRIDGER LARSON | Associate Build/Release Engineer | SirsiDynix
P: 801.223.5345 | bridger.larson@sirsidynix.com
www.sirsidynix.com | TOLL FREE: +1 800.284.3969

-----Original Message-----
From: Jeffrey Hulten [mailto:jeffh@automatedlabs.com]
Sent: Friday, December 07, 2012 5:00 PM
To: chef@lists.opscode.com
Subject: [chef] Re: Auto scaling with chef questions

In autoscaling groups you set a launch config which can contain a first-run.json that you can include to the chef-client.

{ "run_list" : "role[my_service]" }

chef-client -j /etc/chef/first-run.json

--
Jeffrey Hulten
Principal Consultant at Automated Labs, LLC jeffh@automatedlabs.com 206-853-5216
Skype: jeffhulten

On Dec 7, 2012, at 3:50 PM, Cassiano Leal wrote:

I had been thinking about the very same thing for a while, but haven't had the time to run some tests.

My guess so far is that every new machine that spins up will run chef-client, create a node and a client using the validator key and using the ec2-****** FQDN as client/node name. Have you tried running 'hostname -f' on an EC2 server?

--
Cassiano Leal

On Friday, December 7, 2012 at 21:43, Bridger Larson wrote:

The company I am working for is moving to AWS. I want to be able to have Chef pre-installed on our application server AMI's so when they start up they will be configured correctly.

If one AMI can create many servers, how does the Chef client give them different client names?
How does Chef know to remove them when the server is deleted?

Thanks!
Bridger Larson