Setting up hosts between multiple boxes on EC2


#1

Let’s say I provision two boxes on EC2: db1 and app1

What’s the best way to make each box aware of the other? I need to
tell app1 the IP/host/FQDN of the db1 machine so it can connect to it.

One option is to use hosts in the config files - for example, in the
app1 box the config references a host (db1) that I can set up either
in the hosts file or in the local DNS. But I still wouldn’t know the
IP of db1 to populate the hosts file with when running chef. At this
point, I need to change my data-bags once I find out the address of
db1 and then provision app1, running a recipe that adds the right
entry to the hosts file with the newly found db1 IP.

Just wondering if there is a better way.

Thanks,

Luis


#2

You’ll want to take a good look at how Chef utilizes search within
recipes. Rather than try to manage dns, you use search in the recipes
for your “app1” to find the machines that have the recipe or roles
that identify your “db1” machine (and vice versa). Many recipes do
this sort of work, load balancers looking for web apps, clients
looking for servers and workers looking for masters. IMHO, it’s one of
the cooler features of working with Chef.

http://wiki.opscode.com/display/chef/Search
and more specifically
http://wiki.opscode.com/display/chef/Search#Search-UsingSearchinRecipes

Thanks,
Matt Ray
Senior Technical Evangelist | Opscode Inc.
matt@opscode.com | (512) 731-2218
Twitter, IRC, GitHub: mattray

On Sun, Nov 20, 2011 at 1:32 PM, Luis Correa d’Almeida
luis.ca@gmail.com wrote:

Let’s say I provision two boxes on EC2: db1 and app1

What’s the best way to make each box aware of the other? I need to
tell app1 the IP/host/FQDN of the db1 machine so it can connect to it.

One option is to use hosts in the config files - for example, in the
app1 box the config references a host (db1) that I can set up either
in the hosts file or in the local DNS. But I still wouldn’t know the
IP of db1 to populate the hosts file with when running chef. At this
point, I need to change my data-bags once I find out the address of
db1 and then provision app1, running a recipe that adds the right
entry to the hosts file with the newly found db1 IP.

Just wondering if there is a better way.

Thanks,

Luis


#3

On Nov 20, 2011, at 8:31 AM, Matt Ray wrote:

You’ll want to take a good look at how Chef utilizes search within
recipes. Rather than try to manage dns, you use search in the recipes
for your “app1” to find the machines that have the recipe or roles
that identify your “db1” machine (and vice versa). Many recipes do
this sort of work, load balancers looking for web apps, clients
looking for servers and workers looking for masters. IMHO, it’s one of
the cooler features of working with Chef.

And I’ve recently gotten this sort of thing working with the “shorewall” firewall toolkit. I can tell you that it feels like a pretty good achievement when you finally get something like this working on your own, for the first time.

As I understand it, one thing to be concerned about with regards to using data bags is that you don’t want to have multiple nodes potentially writing to the same data bag at the same time – that could cause a race condition in writing the data bag contents back to the chef server, which might cause data to be lost.

So, if you were in a situation where you wanted to use search to populate the data bags and then be more circumspect about when information is purged from the data bags, then you would want to have that process done by a back-end monitoring server, and have read-only access to the data bags on all the rest of the nodes.


Brad Knowles bknowles@ihiji.com
SAGE Level IV, Chef Level 0.0.1


#4

Hi

I’ve actually implemented something like that with route53 (amazon’t DNS
service). For each important host (e.g, db) I’ve created a name (e.g.
db.domain.com) and added it as CNAME to the host’s internal address (this
does not work between regions though). The reason I chose route53 was that
I think (It’s not yet checked) that It helps reduces TTL since I already
query the amazon DNS.

The update script looks like this:

#!/usr/bin/env ruby

NODENAME = <%= %{’#{node.name}’} %>
HOSTNAME = <%= %{’#{node[:ec2][:hostname]}’} %>
DOMAINNAME = <%= %{’#{node[:company][:internal_domain]}’} %>
EC2_KEY_ID = <%= %{’#{@api[‘EC2_KEY_ID’]}’} %>
EC2_SECRET_KEY = <%= %{’#{@api[‘EC2_SECRET_KEY’]}’} %>

require 'rubygems’
require ‘right_aws’

module Route53Update
extend self

def r53
@r53 ||= RightAws::Route53Interface.new(EC2_KEY_ID, EC2_SECRET_KEY)
end

def zone_id
return @zone_id if @zone_id
zone = r53.list_hosted_zones.select { |zone|
zone[:name] =~ /^#{DOMAINNAME}.?$/
}.first
@zone_id = zone ? zone[:aws_id] :
raise(“Zone #{DOMAINNAME} doesn’t seem to be registered!”)
end

def create_cname
delete_cname
resource_record_sets = [{
:name => NODENAME,
:type => ‘CNAME’,
:ttl => 300,
:resource_records => [HOSTNAME]
}]
r53.create_resource_record_sets(zone_id, resource_record_sets, “”)
end

def delete_cname
r53.list_resource_record_sets(zone_id).each do |record|
if record[:name] == "#{NODENAME}."
r53.delete_resource_record_sets(zone_id, [record], “”)
break
end
end
end

def validate_cname
res = r53.list_resource_record_sets(zone_id).select { |record|
record[:name] =~ /#{NODENAME}.?$/
}.first
res && res[:resource_records].include?(HOSTNAME) ? true : false
end

def update_cname
return if validate_cname
create_cname
# TODO: validate sync!
end
end

if $0 == FILE
Route53Update.update_cname
end

You have to install right_aws gem and add an init script (so it’ll run on
boot). You can probably figure out the variables.

Using DNS helps with not having to restart application every time a host
changes it IP, but you have to consider DNS caching (e.g, Java caches dns
results depending on running environment).

HTH

Haim

On Sun, Nov 20, 2011 at 3:32 PM, Luis Correa d’Almeida luis.ca@gmail.comwrote:

Let’s say I provision two boxes on EC2: db1 and app1

What’s the best way to make each box aware of the other? I need to
tell app1 the IP/host/FQDN of the db1 machine so it can connect to it.

One option is to use hosts in the config files - for example, in the
app1 box the config references a host (db1) that I can set up either
in the hosts file or in the local DNS. But I still wouldn’t know the
IP of db1 to populate the hosts file with when running chef. At this
point, I need to change my data-bags once I find out the address of
db1 and then provision app1, running a recipe that adds the right
entry to the hosts file with the newly found db1 IP.

Just wondering if there is a better way.

Thanks,

Luis


Haim


#5

One more thing, In this setup the node is named with the general name (
db.domain.com). Otherwise change the NODENAME variable.

On Sun, Nov 20, 2011 at 5:40 PM, Haim Ashkenazi haim.ashkenazi@gmail.comwrote:

Hi

I’ve actually implemented something like that with route53 (amazon’t DNS
service). For each important host (e.g, db) I’ve created a name (e.g.
db.domain.com) and added it as CNAME to the host’s internal address (this
does not work between regions though). The reason I chose route53 was that
I think (It’s not yet checked) that It helps reduces TTL since I already
query the amazon DNS.

The update script looks like this:

#!/usr/bin/env ruby

NODENAME = <%= %{’#{node.name}’} %>
HOSTNAME = <%= %{’#{node[:ec2][:hostname]}’} %>
DOMAINNAME = <%= %{’#{node[:company][:internal_domain]}’} %>
EC2_KEY_ID = <%= %{’#{@api[‘EC2_KEY_ID’]}’} %>
EC2_SECRET_KEY = <%= %{’#{@api[‘EC2_SECRET_KEY’]}’} %>

require 'rubygems’
require ‘right_aws’

module Route53Update
extend self

def r53
@r53 ||= RightAws::Route53Interface.new(EC2_KEY_ID, EC2_SECRET_KEY)
end

def zone_id
return @zone_id if @zone_id
zone = r53.list_hosted_zones.select { |zone|
zone[:name] =~ /^#{DOMAINNAME}.?$/
}.first
@zone_id = zone ? zone[:aws_id] :
raise(“Zone #{DOMAINNAME} doesn’t seem to be registered!”)
end

def create_cname
delete_cname
resource_record_sets = [{
:name => NODENAME,
:type => ‘CNAME’,
:ttl => 300,
:resource_records => [HOSTNAME]
}]
r53.create_resource_record_sets(zone_id, resource_record_sets, “”)
end

def delete_cname
r53.list_resource_record_sets(zone_id).each do |record|
if record[:name] == "#{NODENAME}."
r53.delete_resource_record_sets(zone_id, [record], “”)
break
end
end
end

def validate_cname
res = r53.list_resource_record_sets(zone_id).select { |record|
record[:name] =~ /#{NODENAME}.?$/
}.first
res && res[:resource_records].include?(HOSTNAME) ? true : false
end

def update_cname
return if validate_cname
create_cname
# TODO: validate sync!
end
end

if $0 == FILE
Route53Update.update_cname
end

You have to install right_aws gem and add an init script (so it’ll run on
boot). You can probably figure out the variables.

Using DNS helps with not having to restart application every time a host
changes it IP, but you have to consider DNS caching (e.g, Java caches dns
results depending on running environment).

HTH

Haim

On Sun, Nov 20, 2011 at 3:32 PM, Luis Correa d’Almeida luis.ca@gmail.comwrote:

Let’s say I provision two boxes on EC2: db1 and app1

What’s the best way to make each box aware of the other? I need to
tell app1 the IP/host/FQDN of the db1 machine so it can connect to it.

One option is to use hosts in the config files - for example, in the
app1 box the config references a host (db1) that I can set up either
in the hosts file or in the local DNS. But I still wouldn’t know the
IP of db1 to populate the hosts file with when running chef. At this
point, I need to change my data-bags once I find out the address of
db1 and then provision app1, running a recipe that adds the right
entry to the hosts file with the newly found db1 IP.

Just wondering if there is a better way.

Thanks,

Luis


Haim


Haim


#6

Great, thanks. Using search works well for my setup.

On Sun, Nov 20, 2011 at 3:42 PM, Haim Ashkenazi
haim.ashkenazi@gmail.com wrote:

One more thing, In this setup the node is named with the general name
(db.domain.com). Otherwise change the NODENAME variable.

On Sun, Nov 20, 2011 at 5:40 PM, Haim Ashkenazi haim.ashkenazi@gmail.com
wrote:

Hi
I’ve actually implemented something like that with route53 (amazon’t DNS
service). For each important host (e.g, db) I’ve created a name (e.g.
db.domain.com) and added it as CNAME to the host’s internal address (this
does not work between regions though). The reason I chose route53 was that I
think (It’s not yet checked) that It helps reduces TTL since I already query
the amazon DNS.
The update script looks like this:
#!/usr/bin/env ruby
NODENAME = <%= %{’#{node.name}’} %>
HOSTNAME = <%= %{’#{node[:ec2][:hostname]}’} %>
DOMAINNAME = <%= %{’#{node[:company][:internal_domain]}’} %>
EC2_KEY_ID = <%= %{’#{@api[‘EC2_KEY_ID’]}’} %>
EC2_SECRET_KEY = <%= %{’#{@api[‘EC2_SECRET_KEY’]}’} %>
require 'rubygems’
require 'right_aws’
module Route53Update
extend self
def r53
@r53 ||= RightAws::Route53Interface.new(EC2_KEY_ID, EC2_SECRET_KEY)
end
def zone_id
return @zone_id if @zone_id
zone = r53.list_hosted_zones.select { |zone|
zone[:name] =~ /^#{DOMAINNAME}.?/ }.first @zone_id = zone ? zone[:aws_id] : raise("Zone #{DOMAINNAME} doesn't seem to be registered!") end def create_cname delete_cname resource_record_sets = [{ :name => NODENAME, :type => 'CNAME', :ttl => 300, :resource_records => [HOSTNAME] }] r53.create_resource_record_sets(zone_id, resource_record_sets, "") end def delete_cname r53.list_resource_record_sets(zone_id).each do |record| if record[:name] == "#{NODENAME}." r53.delete_resource_record_sets(zone_id, [record], "") break end end end def validate_cname res = r53.list_resource_record_sets(zone_id).select { |record| record[:name] =~ /#{NODENAME}\.?/
}.first
res && res[:resource_records].include?(HOSTNAME) ? true : false
end
def update_cname
return if validate_cname
create_cname
# TODO: validate sync!
end
end
if $0 == FILE
Route53Update.update_cname
end
You have to install right_aws gem and add an init script (so it’ll run on
boot). You can probably figure out the variables.
Using DNS helps with not having to restart application every time a host
changes it IP, but you have to consider DNS caching (e.g, Java caches dns
results depending on running environment).
HTH
Haim
On Sun, Nov 20, 2011 at 3:32 PM, Luis Correa d’Almeida luis.ca@gmail.com
wrote:

Let’s say I provision two boxes on EC2: db1 and app1

What’s the best way to make each box aware of the other? I need to
tell app1 the IP/host/FQDN of the db1 machine so it can connect to it.

One option is to use hosts in the config files - for example, in the
app1 box the config references a host (db1) that I can set up either
in the hosts file or in the local DNS. But I still wouldn’t know the
IP of db1 to populate the hosts file with when running chef. At this
point, I need to change my data-bags once I find out the address of
db1 and then provision app1, running a recipe that adds the right
entry to the hosts file with the newly found db1 IP.

Just wondering if there is a better way.

Thanks,

Luis


Haim


Haim


Luis Corrêa d’Almeida
tribe.fm/luis


#7

On Nov 20, 2011, at 9:40 AM, Haim Ashkenazi wrote:

I’ve actually implemented something like that with route53 (amazon’t DNS service). For each important host (e.g, db) I’ve created a name (e.g. db.domain.com) and added it as CNAME to the host’s internal address (this does not work between regions though). The reason I chose route53 was that I think (It’s not yet checked) that It helps reduces TTL since I already query the amazon DNS.

BTW, there’s already a cookbook for doing this kind of stuff with DynECT, and we have implemented similar functionality using a curl-based mechanism to Zerigo for our nodes on Rackspace Cloud.

I know that Fog is the “right” way to do this sort of thing (and I’m sure the DynECT cookbook uses this method), but the interface between Zerigo and Fog wasn’t obvious to us, and the existing cookbook from Zerigo for using their tools … well, let’s just say that we found it “sub-optimal”. So, we kit-bashed together the critical pieces we needed based on their documentation for their REST API and their curl-based mechanisms.

All of which gets us back to the mantra TIMTOWTDI.


Brad Knowles bknowles@ihiji.com
SAGE Level IV, Chef Level 0.0.1