How to handle the elastic_ip associate recipe to finish up smoothly?

Chef Infra (archive)
1

Hi Chefellas!

I fire up a ec2 instance on AWS as follows:

knife ec2 server create -E staging -N cf-worker -x ubuntu -G cf-worker -i

~/.ssh/deployer.pem -r ‘role[resque_worker]’

.....
ec2-50-16-125-18.compute-1.amazonaws.com update-initramfs: Generating

/boot/initrd.img-3.2.0-23-virtual
ec2-50-16-125-18.compute-1.amazonaws.com [Thu, 05 Jul 2012 12:10:15 +0000]
INFO: *** Chef 10.12.0 ***
ec2-50-16-125-18.compute-1.amazonaws.com [Thu, 05 Jul 2012 12:10:18 +0000]
INFO: Client key /etc/chef/client.pem is not present - registering
ec2-50-16-125-18.compute-1.amazonaws.com [Thu, 05 Jul 2012 12:10:27 +0000]
INFO: Setting the run_list to [“role[resque_worker]”] from JSON
ec2-50-16-125-18.compute-1.amazonaws.com [Thu, 05 Jul 2012 12:10:29 +0000]
INFO: Run List is [role[resque_worker]]
ec2-50-16-125-18.compute-1.amazonaws.com [Thu, 05 Jul 2012 12:10:29 +0000]
INFO: Run List expands to [chef-client::delete_validation, chef-client::config,
chef-client::service, apt, git, build-essential, timezone, openssh, postfix,
postfix::aliases, unattended_upgrades, ntp, base::install_ruby, vim, sudo,
base, base::sshd, aws, base::iptables, users::sysadmins, cf_users,
cf_worker::default, cf_worker::deploy, cf_worker::resque,
cf_worker::associate_eip]
…

The last recipe cf_worker::associate_eip in the run_list is as follows:

if node.has_key?("cloud")
  include_recipe "aws"

  aws = data_bag_item("aws", "main")
  elastic_ips = data_bag_item("aws", "elastic_ips")

  eip = elastic_ips['cf_worker'][node.chef_environment]

  aws_elastic_ip "eip_cf_worker" do
aws_access_key aws['aws_access_key_id']
aws_secret_access_key aws['aws_secret_access_key']
ip eip
action :associate
  end
end

If I exclude this last recipe, everything goes smoothly and well and the
chef-handler completes successfully.
But If I include it, doesn’t matter the order, whether I put it at first or
middle or last, the moment it associates the Elastic Ip, it keeps on hanging.

.....
ec2-50-16-125-18.compute-1.amazonaws.com [Thu, 05 Jul 2012 12:20:52 +0000]

INFO: Processing service[god-monitor] action restart (cf_worker::resque line
29)
ec2-50-16-125-18.compute-1.amazonaws.com [Thu, 05 Jul 2012 12:20:57 +0000]
INFO: service[god-monitor] restarted
ec2-50-16-125-18.compute-1.amazonaws.com [Thu, 05 Jul 2012 12:20:57 +0000]
INFO: Processing service[god-monitor] action start (cf_worker::resque line 29)
ec2-50-16-125-18.compute-1.amazonaws.com [Thu, 05 Jul 2012 12:21:03 +0000]
INFO: Processing aws_elastic_ip[eip_cf_worker] action associate
(cf_worker::associate_eip line 10)
ec2-50-16-125-18.compute-1.amazonaws.com [Thu, 05 Jul 2012 12:21:03 +0000]
INFO: New RightAws::Ec2 using shared connections mode
ec2-50-16-125-18.compute-1.amazonaws.com [Thu, 05 Jul 2012 12:21:03 +0000]
INFO: Opening new HTTPS connection to us-east-1.ec2.amazonaws.com:443
ec2-50-16-125-18.compute-1.amazonaws.com [Thu, 05 Jul 2012 12:21:07 +0000]
INFO: Attaching Elastic IP 23.23.85.126 to the instance
ec2-50-16-125-18.compute-1.amazonaws.com [Thu, 05 Jul 2012 12:21:07 +0000]
INFO: template[/etc/chef/client.rb] sending create action to
ruby_block[reload_client_config] (delayed)
ec2-50-16-125-18.compute-1.amazonaws.com [Thu, 05 Jul 2012 12:21:07 +0000]
INFO: Processing ruby_block[reload_client_config] action create
(chef-client::config line 53)
ec2-50-16-125-18.compute-1.amazonaws.com [Thu, 05 Jul 2012 12:21:07 +0000]
INFO: ruby_block[reload_client_config] called
ec2-50-16-125-18.compute-1.amazonaws.com [Thu, 05 Jul 2012 12:21:07 +0000]
INFO: template[/etc/init.d/chef-client] sending restart action to
service[chef-client] (delayed)
ec2-50-16-125-18.compute-1.amazonaws.com [Thu, 05 Jul 2012 12:21:07 +0000]
INFO: Processing service[chef-client] action restart (chef-client::service line
76)
…KEEPS ON HANGING UNTIL I FORCEFULLY (Ctrl + C) stop it…

Its so weird and I couldn’t figure out why this happens?
How you guys handle the elastic ip scenerio?

2

Hi,

Unfortunately if you are not running chef-solo under (screen and/or
tmux) your SSH connection will get interrupted once the new IP address
is attached to the instance.

The only real way to make this better is to either do one of the following:

  1. Run chef-client under screen/tmux on the server itself. Likely
    modifying the bootstrap so it installs screen (and or tmux) and then
    runs chef-client under that.
  2. Provision the servers first and then run knife bootstrap once the
    Elastic IP address is attached.
    • ec2-run-instances ami-.... then follow up with
      ec2-associate-addresses to the instance.

I hope that helps.

Scott

On 7/5/12 6:04 AM, Millisami wrote:

Hi Chefellas!

I fire up a ec2 instance on AWS as follows:

knife ec2 server create -E staging -N cf-worker -x ubuntu -G cf-worker -i

~/.ssh/deployer.pem -r 'role[resque_worker]'

.....
ec2-50-16-125-18.compute-1.amazonaws.com update-initramfs: Generating

/boot/initrd.img-3.2.0-23-virtual
ec2-50-16-125-18.compute-1.amazonaws.com [Thu, 05 Jul 2012 12:10:15 +0000]
INFO: *** Chef 10.12.0 ***
ec2-50-16-125-18.compute-1.amazonaws.com [Thu, 05 Jul 2012 12:10:18 +0000]
INFO: Client key /etc/chef/client.pem is not present - registering
ec2-50-16-125-18.compute-1.amazonaws.com [Thu, 05 Jul 2012 12:10:27 +0000]
INFO: Setting the run_list to ["role[resque_worker]"] from JSON
ec2-50-16-125-18.compute-1.amazonaws.com [Thu, 05 Jul 2012 12:10:29 +0000]
INFO: Run List is [role[resque_worker]]
ec2-50-16-125-18.compute-1.amazonaws.com [Thu, 05 Jul 2012 12:10:29 +0000]
INFO: Run List expands to [chef-client::delete_validation, chef-client::config,
chef-client::service, apt, git, build-essential, timezone, openssh, postfix,
postfix::aliases, unattended_upgrades, ntp, base::install_ruby, vim, sudo,
base, base::sshd, aws, base::iptables, users::sysadmins, cf_users,
cf_worker::default, cf_worker::deploy, cf_worker::resque,
cf_worker::associate_eip]
.....

The last recipe cf_worker::associate_eip in the run_list is as follows:

if node.has_key?("cloud")
  include_recipe "aws"

  aws = data_bag_item("aws", "main")
  elastic_ips = data_bag_item("aws", "elastic_ips")

  eip = elastic_ips['cf_worker'][node.chef_environment]

  aws_elastic_ip "eip_cf_worker" do

aws_access_key aws['aws_access_key_id']
aws_secret_access_key aws['aws_secret_access_key']
ip eip
action :associate
end
end

If I exclude this last recipe, everything goes smoothly and well and the
chef-handler completes successfully.
But If I include it, doesn't matter the order, whether I put it at first or
middle or last, the moment it associates the Elastic Ip, it keeps on hanging.

.....
ec2-50-16-125-18.compute-1.amazonaws.com [Thu, 05 Jul 2012 12:20:52 +0000]

INFO: Processing service[god-monitor] action restart (cf_worker::resque line
29)
ec2-50-16-125-18.compute-1.amazonaws.com [Thu, 05 Jul 2012 12:20:57 +0000]
INFO: service[god-monitor] restarted
ec2-50-16-125-18.compute-1.amazonaws.com [Thu, 05 Jul 2012 12:20:57 +0000]
INFO: Processing service[god-monitor] action start (cf_worker::resque line 29)
ec2-50-16-125-18.compute-1.amazonaws.com [Thu, 05 Jul 2012 12:21:03 +0000]
INFO: Processing aws_elastic_ip[eip_cf_worker] action associate
(cf_worker::associate_eip line 10)
ec2-50-16-125-18.compute-1.amazonaws.com [Thu, 05 Jul 2012 12:21:03 +0000]
INFO: New RightAws::Ec2 using shared connections mode
ec2-50-16-125-18.compute-1.amazonaws.com [Thu, 05 Jul 2012 12:21:03 +0000]
INFO: Opening new HTTPS connection to us-east-1.ec2.amazonaws.com:443
ec2-50-16-125-18.compute-1.amazonaws.com [Thu, 05 Jul 2012 12:21:07 +0000]
INFO: Attaching Elastic IP 23.23.85.126 to the instance
ec2-50-16-125-18.compute-1.amazonaws.com [Thu, 05 Jul 2012 12:21:07 +0000]
INFO: template[/etc/chef/client.rb] sending create action to
ruby_block[reload_client_config] (delayed)
ec2-50-16-125-18.compute-1.amazonaws.com [Thu, 05 Jul 2012 12:21:07 +0000]
INFO: Processing ruby_block[reload_client_config] action create
(chef-client::config line 53)
ec2-50-16-125-18.compute-1.amazonaws.com [Thu, 05 Jul 2012 12:21:07 +0000]
INFO: ruby_block[reload_client_config] called
ec2-50-16-125-18.compute-1.amazonaws.com [Thu, 05 Jul 2012 12:21:07 +0000]
INFO: template[/etc/init.d/chef-client] sending restart action to
service[chef-client] (delayed)
ec2-50-16-125-18.compute-1.amazonaws.com [Thu, 05 Jul 2012 12:21:07 +0000]
INFO: Processing service[chef-client] action restart (chef-client::service line
76)
...KEEPS ON HANGING UNTIL I FORCEFULLY (Ctrl + C) stop it...

Its so weird and I couldn't figure out why this happens?
How you guys handle the elastic ip scenerio?

!DSPAM:4ff59409196631440839711!

3

Hi, thanks for the reply.

Out of the 2 options you've suggested, I want to opt for screen/tmux style. Well I can install the screen/tmux at the bootstrapping time but didn't understand on how to run chef-client under that.

Here is my bootstrap file ubuntu11.10.erb · GitHub

Can you modify/updated it with what you are suggesting?

Thanks

@millisami
~ Sachin Sagar Rai
Ruby on Rails Developer
http://tfm.com.np
http://nepalonrails.tumblr.com
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)

On Friday, July 6, 2012 at 6:14 AM, Scott M. Likens wrote:

Hi,

Unfortunately if you are not running chef-solo under (screen and/or tmux) your SSH connection will get interrupted once the new IP address is attached to the instance.

The only real way to make this better is to either do one of the following:

Run chef-client under screen/tmux on the server itself. Likely modifying the bootstrap so it installs screen (and or tmux) and then runs chef-client under that.
Provision the servers first and then run knife bootstrap once the Elastic IP address is attached.
ec2-run-instances ami-.... then follow up with ec2-associate-addresses to the instance.

I hope that helps.
Scott

On 7/5/12 6:04 AM, Millisami wrote:

Hi Chefellas! I fire up a ec2 instance on AWS as follows: knife ec2 server create -E staging -N cf-worker -x ubuntu -G cf-worker -i ~/.ssh/deployer.pem -r 'role[resque_worker]' ..... ec2-50-16-125-18.compute-1.amazonaws.com (http://ec2-50-16-125-18.compute-1.amazonaws.com) update-initramfs: Generating /boot/initrd.img-3.2.0-23-virtual ec2-50-16-125-18.compute-1.amazonaws.com (http://ec2-50-16-125-18.compute-1.amazonaws.com) [Thu, 05 Jul 2012 12:10:15 +0000] INFO: *** Chef 10.12.0 *** ec2-50-16-125-18.compute-1.amazonaws.com (http://ec2-50-16-125-18.compute-1.amazonaws.com) [Thu, 05 Jul 2012 12:10:18 +0000] INFO: Client key /etc/chef/client.pem is not present - registering ec2-50-16-125-18.compute-1.amazonaws.com (http://ec2-50-16-125-18.compute-1.amazonaws.com) [Thu, 05 Jul 2012 12:10:27 +0000] INFO: Setting the run_list to ["role[resque_worker]"] from JSON ec2-50-16-125-18.compute-1.amazonaws.com (http://ec2-50-16-125-18.compute-1.amazonaws.com) [Thu, 05 Jul 2012 12:10:29 +0000]
INFO: Run List is [role[resque_worker]] ec2-50-16-125-18.compute-1.amazonaws.com (http://ec2-50-16-125-18.compute-1.amazonaws.com) [Thu, 05 Jul 2012 12:10:29 +0000] INFO: Run List expands to [chef-client::delete_validation, chef-client::config, chef-client::service, apt, git, build-essential, timezone, openssh, postfix, postfix::aliases, unattended_upgrades, ntp, base::install_ruby, vim, sudo, base, base::sshd, aws, base::iptables, users::sysadmins, cf_users, cf_worker::default, cf_worker::deploy, cf_worker::resque, cf_worker::associate_eip] ..... The last recipe cf_worker::associate_eip in the run_list is as follows: if node.has_key?("cloud") include_recipe "aws" aws = data_bag_item("aws", "main") elastic_ips = data_bag_item("aws", "elastic_ips") eip = elastic_ips['cf_worker'][node.chef_environment] aws_elastic_ip "eip_cf_worker" do aws_access_key aws['aws_access_key_id'] aws_secret_access_key aws['aws_secret_access_key'] ip eip action :associate end end If I exclude this last reci
pe, everything goes smoothly and well and the chef-handler completes successfully. But If I include it, doesn't matter the order, whether I put it at first or middle or last, the moment it associates the Elastic Ip, it keeps on hanging. ..... ec2-50-16-125-18.compute-1.amazonaws.com (http://ec2-50-16-125-18.compute-1.amazonaws.com) [Thu, 05 Jul 2012 12:20:52 +0000] INFO: Processing service[god-monitor] action restart (cf_worker::resque line 29) ec2-50-16-125-18.compute-1.amazonaws.com (http://ec2-50-16-125-18.compute-1.amazonaws.com) [Thu, 05 Jul 2012 12:20:57 +0000] INFO: service[god-monitor] restarted ec2-50-16-125-18.compute-1.amazonaws.com (http://ec2-50-16-125-18.compute-1.amazonaws.com) [Thu, 05 Jul 2012 12:20:57 +0000] INFO: Processing service[god-monitor] action start (cf_worker::resque line 29) ec2-50-16-125-18.compute-1.amazonaws.com (http://ec2-50-16-125-18.compute-1.amazonaws.com) [Thu, 05 Jul 2012 12:21:03 +0000] INFO: Processing aws_elastic_ip[eip_cf_worker] action asso
ciate (cf_worker::associate_eip line 10) ec2-50-16-125-18.compute-1.amazonaws.com (http://ec2-50-16-125-18.compute-1.amazonaws.com) [Thu, 05 Jul 2012 12:21:03 +0000] INFO: New RightAws::Ec2 using shared connections mode ec2-50-16-125-18.compute-1.amazonaws.com (http://ec2-50-16-125-18.compute-1.amazonaws.com) [Thu, 05 Jul 2012 12:21:03 +0000] INFO: Opening new HTTPS connection to us-east-1.ec2.amazonaws.com:443 (http://us-east-1.ec2.amazonaws.com:443) ec2-50-16-125-18.compute-1.amazonaws.com (http://ec2-50-16-125-18.compute-1.amazonaws.com) [Thu, 05 Jul 2012 12:21:07 +0000] INFO: Attaching Elastic IP 23.23.85.126 to the instance ec2-50-16-125-18.compute-1.amazonaws.com (http://ec2-50-16-125-18.compute-1.amazonaws.com) [Thu, 05 Jul 2012 12:21:07 +0000] INFO: template[/etc/chef/client.rb] sending create action to ruby_block[reload_client_config] (delayed) ec2-50-16-125-18.compute-1.amazonaws.com (http://ec2-50-16-125-18.compute-1.amazonaws.com) [Thu, 05 Jul 2012 12:21:07 +0000] INFO: P
rocessing ruby_block[reload_client_config] action create (chef-client::config line 53) ec2-50-16-125-18.compute-1.amazonaws.com (http://ec2-50-16-125-18.compute-1.amazonaws.com) [Thu, 05 Jul 2012 12:21:07 +0000] INFO: ruby_block[reload_client_config] called ec2-50-16-125-18.compute-1.amazonaws.com (http://ec2-50-16-125-18.compute-1.amazonaws.com) [Thu, 05 Jul 2012 12:21:07 +0000] INFO: template[/etc/init.d/chef-client] sending restart action to service[chef-client] (delayed) ec2-50-16-125-18.compute-1.amazonaws.com (http://ec2-50-16-125-18.compute-1.amazonaws.com) [Thu, 05 Jul 2012 12:21:07 +0000] INFO: Processing service[chef-client] action restart (chef-client::service line 76) ...KEEPS ON HANGING UNTIL I FORCEFULLY (Ctrl + C) stop it... Its so weird and I couldn't figure out why this happens? How you guys handle the elastic ip scenerio? !DSPAM:4ff59409196631440839711!

4

Anybody, any pointers?

@millisami
~ Sachin Sagar Rai
Ruby on Rails Developer
http://tfm.com.np
http://nepalonrails.tumblr.com
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)

On Friday, July 6, 2012 at 11:57 AM, Sachin Sagar Rai wrote:

Hi, thanks for the reply.

Out of the 2 options you've suggested, I want to opt for screen/tmux style. Well I can install the screen/tmux at the bootstrapping time but didn't understand on how to run chef-client under that.

Here is my bootstrap file ubuntu11.10.erb · GitHub

Can you modify/updated it with what you are suggesting?

Thanks

@millisami
~ Sachin Sagar Rai
Ruby on Rails Developer
http://tfm.com.np
http://nepalonrails.tumblr.com
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)

On Friday, July 6, 2012 at 6:14 AM, Scott M. Likens wrote:

Hi,

Unfortunately if you are not running chef-solo under (screen and/or tmux) your SSH connection will get interrupted once the new IP address is attached to the instance.

The only real way to make this better is to either do one of the following:

Run chef-client under screen/tmux on the server itself. Likely modifying the bootstrap so it installs screen (and or tmux) and then runs chef-client under that.
Provision the servers first and then run knife bootstrap once the Elastic IP address is attached.
ec2-run-instances ami-.... then follow up with ec2-associate-addresses to the instance.

I hope that helps.
Scott

On 7/5/12 6:04 AM, Millisami wrote:

Hi Chefellas! I fire up a ec2 instance on AWS as follows: knife ec2 server create -E staging -N cf-worker -x ubuntu -G cf-worker -i ~/.ssh/deployer.pem -r 'role[resque_worker]' ..... ec2-50-16-125-18.compute-1.amazonaws.com (http://ec2-50-16-125-18.compute-1.amazonaws.com) update-initramfs: Generating /boot/initrd.img-3.2.0-23-virtual ec2-50-16-125-18.compute-1.amazonaws.com (http://ec2-50-16-125-18.compute-1.amazonaws.com) [Thu, 05 Jul 2012 12:10:15 +0000] INFO: *** Chef 10.12.0 *** ec2-50-16-125-18.compute-1.amazonaws.com (http://ec2-50-16-125-18.compute-1.amazonaws.com) [Thu, 05 Jul 2012 12:10:18 +0000] INFO: Client key /etc/chef/client.pem is not present - registering ec2-50-16-125-18.compute-1.amazonaws.com (http://ec2-50-16-125-18.compute-1.amazonaws.com) [Thu, 05 Jul 2012 12:10:27 +0000] INFO: Setting the run_list to ["role[resque_worker]"] from JSON ec2-50-16-125-18.compute-1.amazonaws.com (http://ec2-50-16-125-18.compute-1.amazonaws.com) [Thu, 05 Jul 2012 12:10:29 +000
0] INFO: Run List is [role[resque_worker]] ec2-50-16-125-18.compute-1.amazonaws.com (http://ec2-50-16-125-18.compute-1.amazonaws.com) [Thu, 05 Jul 2012 12:10:29 +0000] INFO: Run List expands to [chef-client::delete_validation, chef-client::config, chef-client::service, apt, git, build-essential, timezone, openssh, postfix, postfix::aliases, unattended_upgrades, ntp, base::install_ruby, vim, sudo, base, base::sshd, aws, base::iptables, users::sysadmins, cf_users, cf_worker::default, cf_worker::deploy, cf_worker::resque, cf_worker::associate_eip] ..... The last recipe cf_worker::associate_eip in the run_list is as follows: if node.has_key?("cloud") include_recipe "aws" aws = data_bag_item("aws", "main") elastic_ips = data_bag_item("aws", "elastic_ips") eip = elastic_ips['cf_worker'][node.chef_environment] aws_elastic_ip "eip_cf_worker" do aws_access_key aws['aws_access_key_id'] aws_secret_access_key aws['aws_secret_access_key'] ip eip action :associate end end If I exclude this last re
cipe, everything goes smoothly and well and the chef-handler completes successfully. But If I include it, doesn't matter the order, whether I put it at first or middle or last, the moment it associates the Elastic Ip, it keeps on hanging. ..... ec2-50-16-125-18.compute-1.amazonaws.com (http://ec2-50-16-125-18.compute-1.amazonaws.com) [Thu, 05 Jul 2012 12:20:52 +0000] INFO: Processing service[god-monitor] action restart (cf_worker::resque line 29) ec2-50-16-125-18.compute-1.amazonaws.com (http://ec2-50-16-125-18.compute-1.amazonaws.com) [Thu, 05 Jul 2012 12:20:57 +0000] INFO: service[god-monitor] restarted ec2-50-16-125-18.compute-1.amazonaws.com (http://ec2-50-16-125-18.compute-1.amazonaws.com) [Thu, 05 Jul 2012 12:20:57 +0000] INFO: Processing service[god-monitor] action start (cf_worker::resque line 29) ec2-50-16-125-18.compute-1.amazonaws.com (http://ec2-50-16-125-18.compute-1.amazonaws.com) [Thu, 05 Jul 2012 12:21:03 +0000] INFO: Processing aws_elastic_ip[eip_cf_worker] action as
sociate (cf_worker::associate_eip line 10) ec2-50-16-125-18.compute-1.amazonaws.com (http://ec2-50-16-125-18.compute-1.amazonaws.com) [Thu, 05 Jul 2012 12:21:03 +0000] INFO: New RightAws::Ec2 using shared connections mode ec2-50-16-125-18.compute-1.amazonaws.com (http://ec2-50-16-125-18.compute-1.amazonaws.com) [Thu, 05 Jul 2012 12:21:03 +0000] INFO: Opening new HTTPS connection to us-east-1.ec2.amazonaws.com:443 (http://us-east-1.ec2.amazonaws.com:443) ec2-50-16-125-18.compute-1.amazonaws.com (http://ec2-50-16-125-18.compute-1.amazonaws.com) [Thu, 05 Jul 2012 12:21:07 +0000] INFO: Attaching Elastic IP 23.23.85.126 to the instance ec2-50-16-125-18.compute-1.amazonaws.com (http://ec2-50-16-125-18.compute-1.amazonaws.com) [Thu, 05 Jul 2012 12:21:07 +0000] INFO: template[/etc/chef/client.rb] sending create action to ruby_block[reload_client_config] (delayed) ec2-50-16-125-18.compute-1.amazonaws.com (http://ec2-50-16-125-18.compute-1.amazonaws.com) [Thu, 05 Jul 2012 12:21:07 +0000] INFO:
Processing ruby_block[reload_client_config] action create (chef-client::config line 53) ec2-50-16-125-18.compute-1.amazonaws.com (http://ec2-50-16-125-18.compute-1.amazonaws.com) [Thu, 05 Jul 2012 12:21:07 +0000] INFO: ruby_block[reload_client_config] called ec2-50-16-125-18.compute-1.amazonaws.com (http://ec2-50-16-125-18.compute-1.amazonaws.com) [Thu, 05 Jul 2012 12:21:07 +0000] INFO: template[/etc/init.d/chef-client] sending restart action to service[chef-client] (delayed) ec2-50-16-125-18.compute-1.amazonaws.com (http://ec2-50-16-125-18.compute-1.amazonaws.com) [Thu, 05 Jul 2012 12:21:07 +0000] INFO: Processing service[chef-client] action restart (chef-client::service line 76) ...KEEPS ON HANGING UNTIL I FORCEFULLY (Ctrl + C) stop it... Its so weird and I couldn't figure out why this happens? How you guys handle the elastic ip scenerio? !DSPAM:4ff59409196631440839711!

5

On Fri, Jul 6, 2012 at 8:27 PM, Sachin Sagar Rai millisami@gmail.com wrote:

Anybody, any pointers?

I use a custom bootstrap that runs chef-client --once --daemonize. I
ended up doing this because my recipe to launch new MySQL slaves
currently requires copying ~150GB of data from a single EBS volume
created from a snapshot onto a newly created EBS RAID set, which can
take a few hours and my SSH connections were timing out. But, this
seems like it would work in this scenario as well.

I call knife ec2 server create from a wrapper script which sets
ROLES and ENVIRONMENT which are used in this template. That's a bit
hacky and there's probably a better way to do this, but it's working
for me for now.

The downside to this is that you lose the in-line output from
chef-client, but it's easy to ssh into the machine and tail
/var/log/chef.log to watch it's progress.

6

@JD, thanks for the reply n ur bootstrap gist

I looked at it. But my question about the ssh hanging when elastic_ip is attached is not answered?

Or am I getting it wrong?

@millisami
~ Sachin Sagar Rai
Ruby on Rails Developer
http://tfm.com.np
http://nepalonrails.tumblr.com
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)

On Saturday, July 7, 2012 at 1:45 AM, JD Harrington wrote:

On Fri, Jul 6, 2012 at 8:27 PM, Sachin Sagar Rai <millisami@gmail.com (mailto:millisami@gmail.com)> wrote:

Anybody, any pointers?

I use a custom bootstrap that runs chef-client --once --daemonize. I
ended up doing this because my recipe to launch new MySQL slaves
currently requires copying ~150GB of data from a single EBS volume
created from a snapshot onto a newly created EBS RAID set, which can
take a few hours and my SSH connections were timing out. But, this
seems like it would work in this scenario as well.

I call knife ec2 server create from a wrapper script which sets
ROLES and ENVIRONMENT which are used in this template. That's a bit
hacky and there's probably a better way to do this, but it's working
for me for now.

The downside to this is that you lose the in-line output from
chef-client, but it's easy to ssh into the machine and tail
/var/log/chef.log to watch it's progress.

My custom Chef bootstrap for Amazon Linux · GitHub

7

On Sat, Jul 7, 2012 at 3:16 PM, Sachin Sagar Rai millisami@gmail.com wrote:

@JD, thanks for the reply n ur bootstrap gist

I looked at it. But my question about the ssh hanging when elastic_ip is
attached is not answered?

Or am I getting it wrong?

Running chef-client with the --daemonize option means it is not
attached to your TTY. ssh hangs because binding an elastic IP actually
changes the public IP of the instance you're binding it to, rather
than adding the elastic IP in addition to the already allocated public
IP. By running chef-client daemonized (or, as suggested earlier, in
screen/tmux), you're detaching chef-client from the TTY of your ssh
connection so that the run won't be interrupted.

8

@JD, Thanks for the explanation.

I'll give it a try n see it.

@millisami
~ Sachin Sagar Rai
Ruby on Rails Developer
http://tfm.com.np
http://nepalonrails.tumblr.com
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)

On Saturday, July 7, 2012 at 10:11 PM, JD Harrington wrote:

On Sat, Jul 7, 2012 at 3:16 PM, Sachin Sagar Rai <millisami@gmail.com (mailto:millisami@gmail.com)> wrote:

@JD, thanks for the reply n ur bootstrap gist

I looked at it. But my question about the ssh hanging when elastic_ip is
attached is not answered?

Or am I getting it wrong?

Running chef-client with the --daemonize option means it is not
attached to your TTY. ssh hangs because binding an elastic IP actually
changes the public IP of the instance you're binding it to, rather
than adding the elastic IP in addition to the already allocated public
IP. By running chef-client daemonized (or, as suggested earlier, in
screen/tmux), you're detaching chef-client from the TTY of your ssh
connection so that the run won't be interrupted.

9

@JD,

Well I setup the bootstrap template as yours and now when I am going to launch the instance, what would be the knife command to fire up the ec2 instance as per your modified bootstrap template?

Is it:

knife ec2 server create -E staging -x ubuntu -N logstash-server -V -i ./keys/ec2-server.pem -r 'role[logstash_server]'

But looking at the bootstrap template, it requires the ENV['ROLES'] and ENV['ENVIRONMENT'] env vars?

How should I pass those env attrs? Is it just setting while running the command as: (removing the -E and -r values)

ROLES=role[logstash_server] ENVIRONMENT=staging knife ec2 server create -x ubuntu -N logstash-server -V -i ./keys/ec2-server.pem

This has become totally confusing for me now!

@millisami
~ Sachin Sagar Rai
Ruby on Rails Developer
http://tfm.com.np
http://nepalonrails.tumblr.com
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)

On Saturday, July 7, 2012 at 10:36 PM, Sachin Sagar Rai wrote:

@JD, Thanks for the explanation.

I'll give it a try n see it.

@millisami
~ Sachin Sagar Rai
Ruby on Rails Developer
http://tfm.com.np
http://nepalonrails.tumblr.com
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)

On Saturday, July 7, 2012 at 10:11 PM, JD Harrington wrote:

On Sat, Jul 7, 2012 at 3:16 PM, Sachin Sagar Rai <millisami@gmail.com (mailto:millisami@gmail.com)> wrote:

@JD, thanks for the reply n ur bootstrap gist

I looked at it. But my question about the ssh hanging when elastic_ip is
attached is not answered?

Or am I getting it wrong?

Running chef-client with the --daemonize option means it is not
attached to your TTY. ssh hangs because binding an elastic IP actually
changes the public IP of the instance you're binding it to, rather
than adding the elastic IP in addition to the already allocated public
IP. By running chef-client daemonized (or, as suggested earlier, in
screen/tmux), you're detaching chef-client from the TTY of your ssh
connection so that the run won't be interrupted.

10

On Thu, Jul 5, 2012 at 8:29 PM, Scott M. Likens scott@likens.us wrote:

Unfortunately if you are not running chef-solo under (screen and/or tmux)
your SSH connection will get interrupted once the new IP address is attached
to the instance.

The only real way to make this better is to either do one of the following:

Run chef-client under screen/tmux on the server itself. Likely modifying
the bootstrap so it installs screen (and or tmux) and then runs chef-client
under that.
Provision the servers first and then run knife bootstrap once the Elastic IP
address is attached.

ec2-run-instances ami-.... then follow up with ec2-associate-addresses to
the instance.

It was proposed on KNIFE_EC2-67 [1] to add an option to knife-ec2 to
associate an address before beginning the bootstrap. Generally we try
to avoid adding ec2-tools features to knife-ec2 that aren't chef
specific, but it is simple and useful, so we weren't really sure where
to go with this at code review.

The bigger question is, are we lacking a way to run cookbooks in a
such a way that will trigger chef to finish and then restart itself?
One other example would be updating chef using chef, then continuing
along. Thoughts?

--
Bryan McLellan | opscode | technical program manager, open source
(c) 206.607.7108 | (t) @btmspox | (b) http://blog.loftninjas.org

[1] http://tickets.opscode.com/browse/KNIFE_EC2-67

11

Bryan,

On 12 July 2012 09:26, Bryan McLellan btm@opscode.com wrote:

On Thu, Jul 5, 2012 at 8:29 PM, Scott M. Likens scott@likens.us wrote:

Unfortunately if you are not running chef-solo under (screen and/or tmux)
your SSH connection will get interrupted once the new IP address is attached
to the instance.

The only real way to make this better is to either do one of the following:

Run chef-client under screen/tmux on the server itself. Likely modifying
the bootstrap so it installs screen (and or tmux) and then runs chef-client
under that.
Provision the servers first and then run knife bootstrap once the Elastic IP
address is attached.

ec2-run-instances ami-.... then follow up with ec2-associate-addresses to
the instance.

It was proposed on KNIFE_EC2-67 [1] to add an option to knife-ec2 to
associate an address before beginning the bootstrap. Generally we try
to avoid adding ec2-tools features to knife-ec2 that aren't chef
specific, but it is simple and useful, so we weren't really sure where
to go with this at code review.

The bigger question is, are we lacking a way to run cookbooks in a
such a way that will trigger chef to finish and then restart itself?
One other example would be updating chef using chef, then continuing
along. Thoughts?

This seems simple enough to be affective. Perhaps the new forkable
client + configurable retries would assist?

--AJ

--
Bryan McLellan | opscode | technical program manager, open source
(c) 206.607.7108 | (t) @btmspox | (b) http://blog.loftninjas.org

[1] http://tickets.opscode.com/browse/KNIFE_EC2-67

12

Hi everyone,

I dug into knife-ec2 this afternoon and was looking for a way to
associate an Elastic IP to a node created via knife-ec2. That
eventually led me to KNIFE_EC2-67, now known as KNIFE-151. It looks
like the leading pull request associated with this ticket is:

Added options to allocation and assign EIPs after launch. by jimcroft · Pull Request #88 · chef/knife-ec2 · GitHub

Are we at a stage where we can mark the ticket as "Fix Provided" so
that it can make it to the next step in the code review process?

Thanks,

--
Hector

On Wed, Jul 11, 2012 at 5:36 PM, AJ Christensen aj@junglist.gen.nz wrote:

Bryan,

On 12 July 2012 09:26, Bryan McLellan btm@opscode.com wrote:

On Thu, Jul 5, 2012 at 8:29 PM, Scott M. Likens scott@likens.us wrote:

Unfortunately if you are not running chef-solo under (screen and/or tmux)
your SSH connection will get interrupted once the new IP address is attached
to the instance.

The only real way to make this better is to either do one of the following:

Run chef-client under screen/tmux on the server itself. Likely modifying
the bootstrap so it installs screen (and or tmux) and then runs chef-client
under that.
Provision the servers first and then run knife bootstrap once the Elastic IP
address is attached.

ec2-run-instances ami-.... then follow up with ec2-associate-addresses to
the instance.

It was proposed on KNIFE_EC2-67 [1] to add an option to knife-ec2 to
associate an address before beginning the bootstrap. Generally we try
to avoid adding ec2-tools features to knife-ec2 that aren't chef
specific, but it is simple and useful, so we weren't really sure where
to go with this at code review.

The bigger question is, are we lacking a way to run cookbooks in a
such a way that will trigger chef to finish and then restart itself?
One other example would be updating chef using chef, then continuing
along. Thoughts?

This seems simple enough to be affective. Perhaps the new forkable
client + configurable retries would assist?

--AJ

--
Bryan McLellan | opscode | technical program manager, open source
(c) 206.607.7108 | (t) @btmspox | (b) http://blog.loftninjas.org

[1] http://tickets.opscode.com/browse/KNIFE_EC2-67