Failing to configure a client via user-data on Amazon EC2

Chef Infra (archive)
1

I’m failing to configure a client via user-data on Amazon EC2. The 2 biggest
issues I see in the log are “Cannot load environment prod” (i haven’t setup an
env yet, should i?) and “Connection refused connecting to localhost:4000” which
isn’t the server and not what’s set in the client.rb.

Here’s my client.rb:
log_level :info
log_location "/var/log/chef/client.log"
ssl_verify_mode :verify_none
validation_client_name "chef-validator"
validation_key "/etc/chef/validation.pem"
client_key "/etc/chef/client.pem"
chef_server_url "http://chef.pri.mydomain.com:4000"
file_cache_path "/var/cache/chef"
file_backup_path "/var/backups/chef"
pid_file "/var/run/chef/client.pid"
node_name "i-e5912c84"
Chef::Log::Formatter.show_time = true

Here’s my client.log

Logfile created on Thu Jul 28 15:22:42 +0000 2011 by logger.rb/22285

[Thu, 28 Jul 2011 15:22:42 +0000] INFO: Daemonizing…
[Thu, 28 Jul 2011 15:22:42 +0000] INFO: Forked, in 6051. Priveleges: 0 0
[Thu, 28 Jul 2011 15:22:44 +0000] INFO: *** Chef 0.10.2 ***
[Thu, 28 Jul 2011 15:22:45 +0000] INFO: Client key /etc/chef/client.pem is not
present - registering
[Thu, 28 Jul 2011 15:22:47 +0000] INFO: HTTP Request Returned 404 Not Found:
Cannot load node i-e5912c84
[Thu, 28 Jul 2011 15:22:47 +0000] INFO: Setting the run_list to
[“role[ubuntu-11_04-prod-worker-sched-mail]”] from JSON
[Thu, 28 Jul 2011 15:22:47 +0000] INFO: Run List is
[role[ubuntu-11_04-prod-worker-sched-mail]]
[Thu, 28 Jul 2011 15:22:47 +0000] INFO: Run List expands to
[ubuntu-11_04-prod-worker-sched-mail]
[Thu, 28 Jul 2011 15:22:47 +0000] INFO: Starting Chef Run for i-e5912c84
[Thu, 28 Jul 2011 15:22:47 +0000] INFO: HTTP Request Returned 404 Not Found:
Cannot load environment prod
[Thu, 28 Jul 2011 15:22:47 +0000] ERROR: Running exception handlers
[Thu, 28 Jul 2011 15:22:47 +0000] FATAL: Saving node information to
/var/cache/chef/failed-run-data.json
[Thu, 28 Jul 2011 15:22:47 +0000] ERROR: Exception handlers complete
[Thu, 28 Jul 2011 15:22:47 +0000] FATAL: Stacktrace dumped to
/var/cache/chef/chef-stacktrace.out
[Thu, 28 Jul 2011 15:22:47 +0000] FATAL: Net::HTTPServerException: 404 “Not
Found”
[Thu, 28 Jul 2011 15:22:50 +0000] INFO: *** Chef 0.10.2 ***
[Thu, 28 Jul 2011 15:22:51 +0000] ERROR: Connection refused connecting to
localhost:4000 for /nodes/ip-10-196-89-221.ec2.internal, retry 1/5
[Thu, 28 Jul 2011 15:22:56 +0000] ERROR: Connection refused connecting to
localhost:4000 for /nodes/ip-10-196-89-221.ec2.internal, retry 2/5
[Thu, 28 Jul 2011 15:23:01 +0000] ERROR: Connection refused connecting to
localhost:4000 for /nodes/ip-10-196-89-221.ec2.internal, retry 3/5
[Thu, 28 Jul 2011 15:23:06 +0000] ERROR: Connection refused connecting to
localhost:4000 for /nodes/ip-10-196-89-221.ec2.internal, retry 4/5
[Thu, 28 Jul 2011 15:23:11 +0000] ERROR: Connection refused connecting to
localhost:4000 for /nodes/ip-10-196-89-221.ec2.internal, retry 5/5
[Thu, 28 Jul 2011 15:23:16 +0000] ERROR: Errno::ECONNREFUSED: Connection
refused - Connection refused connecting to localhost:4000 for
/nodes/ip-10-196-89-221.ec2.internal, giving up
[Thu, 28 Jul 2011 15:23:16 +0000] FATAL: Stacktrace dumped to
/var/chef/cache/chef-stacktrace.out
[Thu, 28 Jul 2011 15:23:16 +0000] ERROR: Sleeping for 1800 seconds before
trying again

2

On Thursday, July 28, 2011 at 8:56 AM, clif@texicans.us wrote:

I'm failing to configure a client via user-data on Amazon EC2. The 2 biggest
issues I see in the log are "Cannot load environment prod" (i haven't setup an
env yet, should i?) and "Connection refused connecting to localhost:4000" which
isn't the server and not what's set in the client.rb.
You shouldn't need to worry about environments until you're ready to use them.

Here's my client.rb:
log_level :info
log_location "/var/log/chef/client.log"
ssl_verify_mode :verify_none
validation_client_name "chef-validator"
validation_key "/etc/chef/validation.pem"
client_key "/etc/chef/client.pem"
chef_server_url "http://chef.pri.mydomain.com:4000"
file_cache_path "/var/cache/chef"
file_backup_path "/var/backups/chef"
pid_file "/var/run/chef/client.pid"
node_name "i-e5912c84"
Chef::Log::Formatter.show_time = true

Here's my client.log

Logfile created on Thu Jul 28 15:22:42 +0000 2011 by logger.rb/22285

[Thu, 28 Jul 2011 15:22:42 +0000] INFO: Daemonizing..
[Thu, 28 Jul 2011 15:22:42 +0000] INFO: Forked, in 6051. Priveleges: 0 0
[Thu, 28 Jul 2011 15:22:44 +0000] INFO: *** Chef 0.10.2 ***
[Thu, 28 Jul 2011 15:22:45 +0000] INFO: Client key /etc/chef/client.pem is not
present - registering
[Thu, 28 Jul 2011 15:22:47 +0000] INFO: HTTP Request Returned 404 Not Found:
Cannot load node i-e5912c84
[Thu, 28 Jul 2011 15:22:47 +0000] INFO: Setting the run_list to
["role[ubuntu-11_04-prod-worker-sched-mail]"] from JSON
[Thu, 28 Jul 2011 15:22:47 +0000] INFO: Run List is
[role[ubuntu-11_04-prod-worker-sched-mail]]
[Thu, 28 Jul 2011 15:22:47 +0000] INFO: Run List expands to
[ubuntu-11_04-prod-worker-sched-mail]
[Thu, 28 Jul 2011 15:22:47 +0000] INFO: Starting Chef Run for i-e5912c84
[Thu, 28 Jul 2011 15:22:47 +0000] INFO: HTTP Request Returned 404 Not Found:
Cannot load environment prod
[Thu, 28 Jul 2011 15:22:47 +0000] ERROR: Running exception handlers
[Thu, 28 Jul 2011 15:22:47 +0000] FATAL: Saving node information to
/var/cache/chef/failed-run-data.json
[Thu, 28 Jul 2011 15:22:47 +0000] ERROR: Exception handlers complete
[Thu, 28 Jul 2011 15:22:47 +0000] FATAL: Stacktrace dumped to
/var/cache/chef/chef-stacktrace.out
[Thu, 28 Jul 2011 15:22:47 +0000] FATAL: Net::HTTPServerException: 404 "Not
Found"
[Thu, 28 Jul 2011 15:22:50 +0000] INFO: *** Chef 0.10.2 ***
[Thu, 28 Jul 2011 15:22:51 +0000] ERROR: Connection refused connecting to
localhost:4000 for /nodes/ip-10-196-89-221.ec2.internal, retry 1/5
[Thu, 28 Jul 2011 15:22:56 +0000] ERROR: Connection refused connecting to
localhost:4000 for /nodes/ip-10-196-89-221.ec2.internal, retry 2/5
[Thu, 28 Jul 2011 15:23:01 +0000] ERROR: Connection refused connecting to
localhost:4000 for /nodes/ip-10-196-89-221.ec2.internal, retry 3/5
[Thu, 28 Jul 2011 15:23:06 +0000] ERROR: Connection refused connecting to
localhost:4000 for /nodes/ip-10-196-89-221.ec2.internal, retry 4/5
[Thu, 28 Jul 2011 15:23:11 +0000] ERROR: Connection refused connecting to
localhost:4000 for /nodes/ip-10-196-89-221.ec2.internal, retry 5/5
[Thu, 28 Jul 2011 15:23:16 +0000] ERROR: Errno::ECONNREFUSED: Connection
refused - Connection refused connecting to localhost:4000 for
/nodes/ip-10-196-89-221.ec2.internal, giving up
[Thu, 28 Jul 2011 15:23:16 +0000] FATAL: Stacktrace dumped to
/var/chef/cache/chef-stacktrace.out
[Thu, 28 Jul 2011 15:23:16 +0000] ERROR: Sleeping for 1800 seconds before
trying again
It appears that a different configuration is being used between these two runs. Could anything be changing the client.rb file? Also, how are you invoking chef-client, i.e., what's the command line? And how are you installing chef? Is it baked in to your AMI?

Thanks,
Dan

3

On Jul 28, 2011, at 11:30 AM, Daniel DeLeo wrote:

On Thursday, July 28, 2011 at 8:56 AM, clif@texicans.us wrote:

I'm failing to configure a client via user-data on Amazon EC2. The 2 biggest
issues I see in the log are "Cannot load environment prod" (i haven't setup an
env yet, should i?) and "Connection refused connecting to localhost:4000" which
isn't the server and not what's set in the client.rb.
You shouldn't need to worry about environments until you're ready to use them.

Here's my client.rb:
log_level :info
log_location "/var/log/chef/client.log"
ssl_verify_mode :verify_none
validation_client_name "chef-validator"
validation_key "/etc/chef/validation.pem"
client_key "/etc/chef/client.pem"
chef_server_url "http://chef.pri.mydomain.com:4000"
file_cache_path "/var/cache/chef"
file_backup_path "/var/backups/chef"
pid_file "/var/run/chef/client.pid"
node_name "i-e5912c84"
Chef::Log::Formatter.show_time = true

Here's my client.log

Logfile created on Thu Jul 28 15:22:42 +0000 2011 by logger.rb/22285

[Thu, 28 Jul 2011 15:22:42 +0000] INFO: Daemonizing..
[Thu, 28 Jul 2011 15:22:42 +0000] INFO: Forked, in 6051. Priveleges: 0 0
[Thu, 28 Jul 2011 15:22:44 +0000] INFO: *** Chef 0.10.2 ***
[Thu, 28 Jul 2011 15:22:45 +0000] INFO: Client key /etc/chef/client.pem is not
present - registering
[Thu, 28 Jul 2011 15:22:47 +0000] INFO: HTTP Request Returned 404 Not Found:
Cannot load node i-e5912c84
[Thu, 28 Jul 2011 15:22:47 +0000] INFO: Setting the run_list to
["role[ubuntu-11_04-prod-worker-sched-mail]"] from JSON
[Thu, 28 Jul 2011 15:22:47 +0000] INFO: Run List is
[role[ubuntu-11_04-prod-worker-sched-mail]]
[Thu, 28 Jul 2011 15:22:47 +0000] INFO: Run List expands to
[ubuntu-11_04-prod-worker-sched-mail]
[Thu, 28 Jul 2011 15:22:47 +0000] INFO: Starting Chef Run for i-e5912c84
[Thu, 28 Jul 2011 15:22:47 +0000] INFO: HTTP Request Returned 404 Not Found:
Cannot load environment prod
[Thu, 28 Jul 2011 15:22:47 +0000] ERROR: Running exception handlers
[Thu, 28 Jul 2011 15:22:47 +0000] FATAL: Saving node information to
/var/cache/chef/failed-run-data.json
[Thu, 28 Jul 2011 15:22:47 +0000] ERROR: Exception handlers complete
[Thu, 28 Jul 2011 15:22:47 +0000] FATAL: Stacktrace dumped to
/var/cache/chef/chef-stacktrace.out
[Thu, 28 Jul 2011 15:22:47 +0000] FATAL: Net::HTTPServerException: 404 "Not
Found"
[Thu, 28 Jul 2011 15:22:50 +0000] INFO: *** Chef 0.10.2 ***
[Thu, 28 Jul 2011 15:22:51 +0000] ERROR: Connection refused connecting to
localhost:4000 for /nodes/ip-10-196-89-221.ec2.internal, retry 1/5
[Thu, 28 Jul 2011 15:22:56 +0000] ERROR: Connection refused connecting to
localhost:4000 for /nodes/ip-10-196-89-221.ec2.internal, retry 2/5
[Thu, 28 Jul 2011 15:23:01 +0000] ERROR: Connection refused connecting to
localhost:4000 for /nodes/ip-10-196-89-221.ec2.internal, retry 3/5
[Thu, 28 Jul 2011 15:23:06 +0000] ERROR: Connection refused connecting to
localhost:4000 for /nodes/ip-10-196-89-221.ec2.internal, retry 4/5
[Thu, 28 Jul 2011 15:23:11 +0000] ERROR: Connection refused connecting to
localhost:4000 for /nodes/ip-10-196-89-221.ec2.internal, retry 5/5
[Thu, 28 Jul 2011 15:23:16 +0000] ERROR: Errno::ECONNREFUSED: Connection
refused - Connection refused connecting to localhost:4000 for
/nodes/ip-10-196-89-221.ec2.internal, giving up
[Thu, 28 Jul 2011 15:23:16 +0000] FATAL: Stacktrace dumped to
/var/chef/cache/chef-stacktrace.out
[Thu, 28 Jul 2011 15:23:16 +0000] ERROR: Sleeping for 1800 seconds before
trying again
It appears that a different configuration is being used between these two runs. Could anything be changing the client.rb file? Also, how are you invoking chef-client, i.e., what's the command line? And how are you installing chef? Is it baked in to your AMI?

Thanks,
Dan

I don't think anything is changing the file:

ls -al /etc/chef/client.rb

-rw-r----- 1 root root 484 2011-07-28 16:40 /etc/chef/client.rb

head /var/log/chef/client.log

Logfile created on Thu Jul 28 16:40:44 +0000 2011 by logger.rb/22285

I'm wanting to use a base Ubuntu AMI. Because of this I'm using the following script which is passed to the system upon initial boot up (via Amazon EC2's user-data):

#!/bin/bash
###################################################

Setup our logfile

exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
###################################################

Setup the OpsCode apt repo

echo "deb http://apt.opscode.com/ lsb_release -cs-0.10 main" >> /etc/apt/sources.list.d/opscode.list
mkdir -p /etc/apt/trusted.gpg.d
gpg --keyserver keys.gnupg.net --recv-keys 83EF826A
gpg --export packages@opscode.com | tee /etc/apt/trusted.gpg.d/opscode-keyring.gpg > /dev/null
apt-get update
apt-get install opscode-keyring # permanent upgradeable keyring
###################################################
APT_GET="env DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get -q"
#$APT_GET -y remove ruby1.8*
$APT_GET -y install ruby ruby-dev libruby
$APT_GET -y install build-essential
$APT_GET -y install chef
#ln -sf gem1.9.1 /usr/bin/gem
#gem install --no-rdoc --no-ri chef
#ln -sf ruby1.9.1 /usr/bin/ruby
mkdir -p /var/log/chef
mkdir -p /var/backups/chef
mkdir -p /var/run/chef
mkdir -p /var/cache/chef
mkdir -p /var/lib/chef
#mkdir /etc/chef
#ln -s /var/lib/gems/1.9.1/bin/chef-client /usr/bin/chef-client
cat - >/etc/chef/bootstrap.json <<EOF
{
"run_list": [
"role[ubuntu-11_04-prod-worker-sched-mail]"
],
"default_attributes": {
},
"override_attributes": {
}
}
EOF
cat - >/etc/chef/client.rb <<EOF
log_level :info
log_location "/var/log/chef/client.log"
ssl_verify_mode :verify_none
validation_client_name "chef-validator"
validation_key "/etc/chef/validation.pem"
client_key "/etc/chef/client.pem"
chef_server_url "http://chef.pri.mydomain.com:4000"
file_cache_path "/var/cache/chef"
file_backup_path "/var/backups/chef"
pid_file "/var/run/chef/client.pid"
node_name "curl http://169.254.169.254/latest/meta-data/instance-id"
Chef::Log::Formatter.show_time = true
EOF
cat - >/etc/chef/validation.pem <<EOF
-----BEGIN RSA PRIVATE KEY-----

-----END RSA PRIVATE KEY-----
EOF
/usr/bin/chef-client -j /etc/chef/bootstrap.json

4

I think I'm all good. The issue was I was using http://tinyurl.com/3k6uwqg as a guide but rather than install chef from gems I'd like to use apt. However I was doing the apt-get at the same point as the script was previously doing the gem. Being that the apt-get starts up chef before the script populated chef's config...

On Jul 28, 2011, at 11:30 AM, Daniel DeLeo wrote:

On Thursday, July 28, 2011 at 8:56 AM, clif@texicans.us wrote:

I'm failing to configure a client via user-data on Amazon EC2. The 2 biggest
issues I see in the log are "Cannot load environment prod" (i haven't setup an
env yet, should i?) and "Connection refused connecting to localhost:4000" which
isn't the server and not what's set in the client.rb.
You shouldn't need to worry about environments until you're ready to use them.

Here's my client.rb:
log_level :info
log_location "/var/log/chef/client.log"
ssl_verify_mode :verify_none
validation_client_name "chef-validator"
validation_key "/etc/chef/validation.pem"
client_key "/etc/chef/client.pem"
chef_server_url "http://chef.pri.mydomain.com:4000"
file_cache_path "/var/cache/chef"
file_backup_path "/var/backups/chef"
pid_file "/var/run/chef/client.pid"
node_name "i-e5912c84"
Chef::Log::Formatter.show_time = true

Here's my client.log

Logfile created on Thu Jul 28 15:22:42 +0000 2011 by logger.rb/22285

[Thu, 28 Jul 2011 15:22:42 +0000] INFO: Daemonizing..
[Thu, 28 Jul 2011 15:22:42 +0000] INFO: Forked, in 6051. Priveleges: 0 0
[Thu, 28 Jul 2011 15:22:44 +0000] INFO: *** Chef 0.10.2 ***
[Thu, 28 Jul 2011 15:22:45 +0000] INFO: Client key /etc/chef/client.pem is not
present - registering
[Thu, 28 Jul 2011 15:22:47 +0000] INFO: HTTP Request Returned 404 Not Found:
Cannot load node i-e5912c84
[Thu, 28 Jul 2011 15:22:47 +0000] INFO: Setting the run_list to
["role[ubuntu-11_04-prod-worker-sched-mail]"] from JSON
[Thu, 28 Jul 2011 15:22:47 +0000] INFO: Run List is
[role[ubuntu-11_04-prod-worker-sched-mail]]
[Thu, 28 Jul 2011 15:22:47 +0000] INFO: Run List expands to
[ubuntu-11_04-prod-worker-sched-mail]
[Thu, 28 Jul 2011 15:22:47 +0000] INFO: Starting Chef Run for i-e5912c84
[Thu, 28 Jul 2011 15:22:47 +0000] INFO: HTTP Request Returned 404 Not Found:
Cannot load environment prod
[Thu, 28 Jul 2011 15:22:47 +0000] ERROR: Running exception handlers
[Thu, 28 Jul 2011 15:22:47 +0000] FATAL: Saving node information to
/var/cache/chef/failed-run-data.json
[Thu, 28 Jul 2011 15:22:47 +0000] ERROR: Exception handlers complete
[Thu, 28 Jul 2011 15:22:47 +0000] FATAL: Stacktrace dumped to
/var/cache/chef/chef-stacktrace.out
[Thu, 28 Jul 2011 15:22:47 +0000] FATAL: Net::HTTPServerException: 404 "Not
Found"
[Thu, 28 Jul 2011 15:22:50 +0000] INFO: *** Chef 0.10.2 ***
[Thu, 28 Jul 2011 15:22:51 +0000] ERROR: Connection refused connecting to
localhost:4000 for /nodes/ip-10-196-89-221.ec2.internal, retry 1/5
[Thu, 28 Jul 2011 15:22:56 +0000] ERROR: Connection refused connecting to
localhost:4000 for /nodes/ip-10-196-89-221.ec2.internal, retry 2/5
[Thu, 28 Jul 2011 15:23:01 +0000] ERROR: Connection refused connecting to
localhost:4000 for /nodes/ip-10-196-89-221.ec2.internal, retry 3/5
[Thu, 28 Jul 2011 15:23:06 +0000] ERROR: Connection refused connecting to
localhost:4000 for /nodes/ip-10-196-89-221.ec2.internal, retry 4/5
[Thu, 28 Jul 2011 15:23:11 +0000] ERROR: Connection refused connecting to
localhost:4000 for /nodes/ip-10-196-89-221.ec2.internal, retry 5/5
[Thu, 28 Jul 2011 15:23:16 +0000] ERROR: Errno::ECONNREFUSED: Connection
refused - Connection refused connecting to localhost:4000 for
/nodes/ip-10-196-89-221.ec2.internal, giving up
[Thu, 28 Jul 2011 15:23:16 +0000] FATAL: Stacktrace dumped to
/var/chef/cache/chef-stacktrace.out
[Thu, 28 Jul 2011 15:23:16 +0000] ERROR: Sleeping for 1800 seconds before
trying again
It appears that a different configuration is being used between these two runs. Could anything be changing the client.rb file? Also, how are you invoking chef-client, i.e., what's the command line? And how are you installing chef? Is it baked in to your AMI?

Thanks,
Dan