Odd behavior with Chef Server

Hello,

First time on the list…

I have a vanilla chef-server deployed on Ubuntu 11.04 on a m1.small EC2 instance. I just followed the opscode ubuntu apt-get install instructions.

My chef-server EC2 instance is associated with an elasticIP, and chef.mydomain.com is a CNAME to the public DNS of the elastic IP. I’m pretty new to both EC2 and Chef, so I’m not sure if this is an EC2 security group issue, or a chef-server issue.

Everything worked at first, but now I’m getting errors; when deleting nodes or clients from the chef-server, I get errors every time, but they are successfully deleted:

aabramson$ knife client list
aabramson
chef-validator
chef-webui
ip-10-90-186-175.ec2.internal
retest
ubuntu
yetanothertest
aabramson$ knife client delete yetanothertest
Do you really want to delete yetanothertest? (Y/N) y
ERROR: Server returned error for http://chef.mydomain.com:4000/clients/yetanothertest, retrying 1/5 in 3s
ERROR: The object you are looking for could not be found
Response: Cannot load client yetanothertest
aabramson$ knife client delete ip-10-90-186-175.ec2.internal
Do you really want to delete ip-10-90-186-175.ec2.internal? (Y/N) y
ERROR: Server returned error for http://chef.mydomain.com:4000/clients/ip-10-90-186-175.ec2.internal, retrying 1/5 in 3s
ERROR: The object you are looking for could not be found
Response: Cannot load client ip-10-90-186-175.ec2.internal
aabramson$ knife client list
aabramson
chef-validator
chef-webui
retest
ubuntu
aabramson$

And now when I try to launch/bootstrap new instances on EC2, the bootstrap fails:

aabramson$ knife ec2 server create -I ami-06ad526f -f t1.micro -S mykeypair -G default,management,www -Z us-east-1a -N newinstance -x ubuntu
.
.
.
ec2-67-202-33-226.compute-1.amazonaws.com Successfully installed chef-0.10.0
ec2-67-202-33-226.compute-1.amazonaws.com 15 gems installed
ec2-67-202-33-226.compute-1.amazonaws.com [Thu, 21 Jul 2011 16:49:03 +0000] INFO: *** Chef 0.10.0 ***
ec2-67-202-33-226.compute-1.amazonaws.com [Thu, 21 Jul 2011 16:49:03 +0000] INFO: Client key /etc/chef/client.pem is not present - registering
ec2-67-202-33-226.compute-1.amazonaws.com [Thu, 21 Jul 2011 16:49:08 +0000] INFO: HTTP Request Returned 500 Internal Server Error: Connection failed - user: chef
ec2-67-202-33-226.compute-1.amazonaws.com [Thu, 21 Jul 2011 16:49:08 +0000] ERROR: Server returned error for http://chef.mydomain.com:4000/clients, retrying 1/5 in 3s
ec2-67-202-33-226.compute-1.amazonaws.com [Thu, 21 Jul 2011 16:49:11 +0000] INFO: HTTP Request Returned 409 Conflict: Client already exists
ec2-67-202-33-226.compute-1.amazonaws.com [Thu, 21 Jul 2011 16:49:11 +0000] INFO: HTTP Request Returned 403 Forbidden: You are not allowed to take this action.
ec2-67-202-33-226.compute-1.amazonaws.com [Thu, 21 Jul 2011 16:49:11 +0000] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
ec2-67-202-33-226.compute-1.amazonaws.com [Thu, 21 Jul 2011 16:49:11 +0000] FATAL: Net::HTTPServerException: 403 “Forbidden”

Instance ID: i-2b87554a
Flavor: t1.micro
Image: ami-06ad526f
Availability Zone: us-east-1a
Security Groups:
Public DNS Name: ec2-67-202-33-226.compute-1.amazonaws.com
Public IP Address: 67.202.33.226
Private DNS Name: domU-12-31-38-04-E1-8F.compute-1.internal
SSH Key: wfg-ec2keypair
Private IP Address: 10.220.226.125
Root Device Type: ebs
Root Volume ID: vol-b9585ad2
Root Device Name: /dev/sda1
Root Device Delete on Terminate: true
Environment: _default
Run List:
abramson$ knife client list
aabramson
chef-validator
chef-webui
newinstance
retest
ubuntu
aabramson$ ssh ubuntu@ec2-67-202-33-226.compute-1.amazonaws.com
Welcome to Ubuntu 11.04 (GNU/Linux 2.6.38-8-virtual i686)
.

ubuntu@domU-12-31-38-04-E1-8F:~ sudo chef-client [Thu, 21 Jul 2011 16:51:33 +0000] INFO: *** Chef 0.10.0 *** [Thu, 21 Jul 2011 16:51:33 +0000] INFO: Client key /etc/chef/client.pem is not present - registering [Thu, 21 Jul 2011 16:51:33 +0000] INFO: HTTP Request Returned 409 Conflict: Client already exists [Thu, 21 Jul 2011 16:51:33 +0000] INFO: HTTP Request Returned 403 Forbidden: You are not allowed to take this action. [Thu, 21 Jul 2011 16:51:33 +0000] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out [Thu, 21 Jul 2011 16:51:33 +0000] FATAL: Net::HTTPServerException: 403 "Forbidden" ubuntu@domU-12-31-38-04-E1-8F:~

Well, you could run some test commands with knife on the server itself
connecting to localhost. That should show if the problem is an
external networking issue or not. If not, can you pull the server logs
and see what it thinks is happening?

KC

On Thu, Jul 21, 2011 at 10:15 AM, Aaron Abramson
aabramson@wi-figuys.com wrote:

Hello,
First time on the list...
I have a vanilla chef-server deployed on Ubuntu 11.04 on a m1.small EC2
instance. I just followed the opscode ubuntu apt-get install instructions.
My chef-server EC2 instance is associated with an elasticIP,
and chef.mydomain.com is a CNAME to the public DNS of the elastic IP. I'm
pretty new to both EC2 and Chef, so I'm not sure if this is an EC2 security
group issue, or a chef-server issue.
Everything worked at first, but now I'm getting errors; when deleting nodes
or clients from the chef-server, I get errors every time, but they are
successfully deleted:

aabramson$ knife client list
aabramson
chef-validator
chef-webui
ip-10-90-186-175.ec2.internal
retest
ubuntu
yetanothertest
aabramson$ knife client delete yetanothertest
Do you really want to delete yetanothertest? (Y/N) y
ERROR: Server returned error
for http://chef.mydomain.com:4000/clients/yetanothertest, retrying 1/5 in 3s
ERROR: The object you are looking for could not be found
Response: Cannot load client yetanothertest
aabramson$ knife client delete ip-10-90-186-175.ec2.internal
Do you really want to delete ip-10-90-186-175.ec2.internal? (Y/N) y
ERROR: Server returned error
for http://chef.mydomain.com:4000/clients/ip-10-90-186-175.ec2.internal,
retrying 1/5 in 3s
ERROR: The object you are looking for could not be found
Response: Cannot load client ip-10-90-186-175.ec2.internal
aabramson$ knife client list
aabramson
chef-validator
chef-webui
retest
ubuntu
aabramson$

And now when I try to launch/bootstrap new instances on EC2, the bootstrap
fails:
aabramson$ knife ec2 server create -I ami-06ad526f -f t1.micro -S mykeypair
-G default,management,www -Z us-east-1a -N newinstance -x ubuntu
.
.
.
ec2-67-202-33-226.compute-1.amazonaws.com Successfully installed chef-0.10.0
ec2-67-202-33-226.compute-1.amazonaws.com 15 gems installed
ec2-67-202-33-226.compute-1.amazonaws.com [Thu, 21 Jul 2011 16:49:03 +0000]
INFO: *** Chef 0.10.0 ***
ec2-67-202-33-226.compute-1.amazonaws.com [Thu, 21 Jul 2011 16:49:03 +0000]
INFO: Client key /etc/chef/client.pem is not present - registering
ec2-67-202-33-226.compute-1.amazonaws.com [Thu, 21 Jul 2011 16:49:08 +0000]
INFO: HTTP Request Returned 500 Internal Server Error: Connection failed -
user: chef
ec2-67-202-33-226.compute-1.amazonaws.com [Thu, 21 Jul 2011 16:49:08 +0000]
ERROR: Server returned error for http://chef.mydomain.com:4000/clients,
retrying 1/5 in 3s
ec2-67-202-33-226.compute-1.amazonaws.com [Thu, 21 Jul 2011 16:49:11 +0000]
INFO: HTTP Request Returned 409 Conflict: Client already exists
ec2-67-202-33-226.compute-1.amazonaws.com [Thu, 21 Jul 2011 16:49:11 +0000]
INFO: HTTP Request Returned 403 Forbidden: You are not allowed to take this
action.
ec2-67-202-33-226.compute-1.amazonaws.com [Thu, 21 Jul 2011 16:49:11 +0000]
FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
ec2-67-202-33-226.compute-1.amazonaws.com [Thu, 21 Jul 2011 16:49:11 +0000]
FATAL: Net::HTTPServerException: 403 "Forbidden"
Instance ID: i-2b87554a
Flavor: t1.micro
Image: ami-06ad526f
Availability Zone: us-east-1a
Security Groups:
Public DNS Name: ec2-67-202-33-226.compute-1.amazonaws.com
Public IP Address: 67.202.33.226
Private DNS Name: domU-12-31-38-04-E1-8F.compute-1.internal
SSH Key: wfg-ec2keypair
Private IP Address: 10.220.226.125
Root Device Type: ebs
Root Volume ID: vol-b9585ad2
Root Device Name: /dev/sda1
Root Device Delete on Terminate: true
Environment: _default
Run List:
abramson$ knife client list
aabramson
chef-validator
chef-webui
newinstance
retest
ubuntu
aabramson$ ssh ubuntu@ec2-67-202-33-226.compute-1.amazonaws.com
Welcome to Ubuntu 11.04 (GNU/Linux 2.6.38-8-virtual i686)
.
..
ubuntu@domU-12-31-38-04-E1-8F:~$ sudo chef-client
[Thu, 21 Jul 2011 16:51:33 +0000] INFO: *** Chef 0.10.0 ***
[Thu, 21 Jul 2011 16:51:33 +0000] INFO: Client key /etc/chef/client.pem is
not present - registering
[Thu, 21 Jul 2011 16:51:33 +0000] INFO: HTTP Request Returned 409 Conflict:
Client already exists
[Thu, 21 Jul 2011 16:51:33 +0000] INFO: HTTP Request Returned 403 Forbidden:
You are not allowed to take this action.
[Thu, 21 Jul 2011 16:51:33 +0000] FATAL: Stacktrace dumped to
/var/chef/cache/chef-stacktrace.out
[Thu, 21 Jul 2011 16:51:33 +0000] FATAL: Net::HTTPServerException: 403
"Forbidden"
ubuntu@domU-12-31-38-04-E1-8F:~$