Problem accessing databag from node


#1

I’m having problems with a 403 error when trying a new cookbook and I think
I’ve tracked it down to databag access.

Chef Server 10.0
chef-client 10.4

The first several lines of the stacktrace:

Generated at 2011-12-08 04:12:07 +0000
Net::HTTPServerException: 403 “Forbidden”
/usr/lib/ruby/1.9.1/net/http.rb:2303:in error!' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:237:inblock in
api_request’
/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:288:in
retriable_rest_request' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:218:inapi_request’
/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:130:in put_rest' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/data_bag_item.rb:227:insave’
/var/chef/cache/cookbooks/ejabberd/recipes/default.rb:45:in from_file' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/mixin/from_file.rb:30:ininstance_eval’
/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/mixin/from_file.rb:30:in
from_file' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/cookbook_version.rb:578:inload_recipe’

The relevant part of that recipe is:

Find cluster master, or create item and become master

if search(:ejabberd, “id:#{deployid}”).count == 0
masternode = "#{node[:ipaddress]}"
h = {}
h[deployid] = {“id” => deployid, “master” => masternode, “members” =>
[masternode]}

Create new data bag item for cluster

databag_item = Chef::DataBagItem.new
databag_item.data_bag(“ejabberd”)
databag_item.raw_data = h[deployid]
databag_item.save
else

This is for a first chef run on a new node. If I run shef, switch to
recipe context, and run ‘search(:ejabberd, “id:deployment_000010182”)’ I
get back:

chef:recipe > search(:ejabberd, “id:deployment_000010182”)
[Thu, 08 Dec 2011 04:51:05 +0000] ERROR: Connection refused connecting to
localhost:4000 for /search/ejabberd, retry 1/5
[Thu, 08 Dec 2011 04:51:10 +0000] ERROR: Connection refused connecting to
localhost:4000 for /search/ejabberd, retry 2/5
[Thu, 08 Dec 2011 04:51:15 +0000] ERROR: Connection refused connecting to
localhost:4000 for /search/ejabberd, retry 3/5
[Thu, 08 Dec 2011 04:51:20 +0000] ERROR: Connection refused connecting to
localhost:4000 for /search/ejabberd, retry 4/5
[Thu, 08 Dec 2011 04:51:25 +0000] ERROR: Connection refused connecting to
localhost:4000 for /search/ejabberd, retry 5/5
Errno::ECONNREFUSED: Connection refused - Connection refused connecting to
localhost:4000 for /search/ejabberd, giving up

but my /etc/chef/client.rb has the proper url:port for my chef server. If
I run the same command from my local box I get back:

chef:recipe > search(:ejabberd, “id:deployment_000010182”)
=> []

which is what I expect. Any ideas where I should be looking?

Thanks,

MG


#2

Hi,

We had this exception when the chef-solr service died on the chef
server. Figure out what killed itc and restart was our approach.

  • This was a result of some process updating the owner/permissions on
    /var/log/chef and /var/run/chef so that solr failed during startup.

On Thu, Dec 8, 2011 at 4:02 PM, Michael Glenney mike.glenney@gmail.com wrote:

I’m having problems with a 403 error when trying a new cookbook and I think
I’ve tracked it down to databag access.

Chef Server 10.0
chef-client 10.4

The first several lines of the stacktrace:

Generated at 2011-12-08 04:12:07 +0000
Net::HTTPServerException: 403 “Forbidden”
/usr/lib/ruby/1.9.1/net/http.rb:2303:in error!' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:237:inblock in
api_request’
/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:288:in
retriable_rest_request' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:218:inapi_request’
/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:130:in put_rest' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/data_bag_item.rb:227:insave’
/var/chef/cache/cookbooks/ejabberd/recipes/default.rb:45:in from_file' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/mixin/from_file.rb:30:ininstance_eval’
/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/mixin/from_file.rb:30:in
from_file' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/cookbook_version.rb:578:inload_recipe’

The relevant part of that recipe is:

Find cluster master, or create item and become master

if search(:ejabberd, “id:#{deployid}”).count == 0
masternode = "#{node[:ipaddress]}"
h = {}
h[deployid] = {“id” => deployid, “master” => masternode, “members” =>
[masternode]}

Create new data bag item for cluster

databag_item = Chef::DataBagItem.new
databag_item.data_bag(“ejabberd”)
databag_item.raw_data = h[deployid]
databag_item.save
else

This is for a first chef run on a new node. If I run shef, switch to recipe
context, and run ‘search(:ejabberd, “id:deployment_000010182”)’ I get back:

chef:recipe > search(:ejabberd, “id:deployment_000010182”)
[Thu, 08 Dec 2011 04:51:05 +0000] ERROR: Connection refused connecting to
localhost:4000 for /search/ejabberd, retry 1/5
[Thu, 08 Dec 2011 04:51:10 +0000] ERROR: Connection refused connecting to
localhost:4000 for /search/ejabberd, retry 2/5
[Thu, 08 Dec 2011 04:51:15 +0000] ERROR: Connection refused connecting to
localhost:4000 for /search/ejabberd, retry 3/5
[Thu, 08 Dec 2011 04:51:20 +0000] ERROR: Connection refused connecting to
localhost:4000 for /search/ejabberd, retry 4/5
[Thu, 08 Dec 2011 04:51:25 +0000] ERROR: Connection refused connecting to
localhost:4000 for /search/ejabberd, retry 5/5
Errno::ECONNREFUSED: Connection refused - Connection refused connecting to
localhost:4000 for /search/ejabberd, giving up

but my /etc/chef/client.rb has the proper url:port for my chef server. If I
run the same command from my local box I get back:

chef:recipe > search(:ejabberd, “id:deployment_000010182”)
=> []

which is what I expect. Any ideas where I should be looking?

Thanks,

MG


Cheers,

Peter Donald


#3

I figured out the issue. First of all, the 403 was because I was trying to
write to a databag within a recipe and didn’t read the disclaimer in the
wiki that said I’d have to give the node api client admin privileges to be
able to do that. Of course we don’t want to do that so I’ll have to come
up with another way to solve the problem.

BTW, I wouldn’t mind seeing chef have the capability to give a nodes
permission to write to a particular databag without giving the node admin
rights in case anyone is accepting Christmas wishes.

For the shef localhost:4000 error, that was just because, when running shef
from a client node, I forgot I need to launch shef with ‘shef -c
/etc/chef/client.rb’

MG

On Wed, Dec 7, 2011 at 10:09 PM, Peter Donald peter@realityforge.orgwrote:

Hi,

We had this exception when the chef-solr service died on the chef
server. Figure out what killed itc and restart was our approach.

  • This was a result of some process updating the owner/permissions on
    /var/log/chef and /var/run/chef so that solr failed during startup.

On Thu, Dec 8, 2011 at 4:02 PM, Michael Glenney mike.glenney@gmail.com
wrote:

I’m having problems with a 403 error when trying a new cookbook and I
think
I’ve tracked it down to databag access.

Chef Server 10.0
chef-client 10.4

The first several lines of the stacktrace:

Generated at 2011-12-08 04:12:07 +0000
Net::HTTPServerException: 403 “Forbidden”
/usr/lib/ruby/1.9.1/net/http.rb:2303:in error!' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:237:inblock
in
api_request’
/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:288:in
retriable_rest_request' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:218:inapi_request’
/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:130:in
`put_rest’

/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/data_bag_item.rb:227:in

save' /var/chef/cache/cookbooks/ejabberd/recipes/default.rb:45:infrom_file’

/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/mixin/from_file.rb:30:in

`instance_eval’

/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/mixin/from_file.rb:30:in

`from_file’

/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/cookbook_version.rb:578:in

`load_recipe’

The relevant part of that recipe is:

Find cluster master, or create item and become master

if search(:ejabberd, “id:#{deployid}”).count == 0
masternode = "#{node[:ipaddress]}"
h = {}
h[deployid] = {“id” => deployid, “master” => masternode, “members” =>
[masternode]}

Create new data bag item for cluster

databag_item = Chef::DataBagItem.new
databag_item.data_bag(“ejabberd”)
databag_item.raw_data = h[deployid]
databag_item.save
else

This is for a first chef run on a new node. If I run shef, switch to
recipe
context, and run ‘search(:ejabberd, “id:deployment_000010182”)’ I get
back:

chef:recipe > search(:ejabberd, “id:deployment_000010182”)
[Thu, 08 Dec 2011 04:51:05 +0000] ERROR: Connection refused connecting to
localhost:4000 for /search/ejabberd, retry 1/5
[Thu, 08 Dec 2011 04:51:10 +0000] ERROR: Connection refused connecting to
localhost:4000 for /search/ejabberd, retry 2/5
[Thu, 08 Dec 2011 04:51:15 +0000] ERROR: Connection refused connecting to
localhost:4000 for /search/ejabberd, retry 3/5
[Thu, 08 Dec 2011 04:51:20 +0000] ERROR: Connection refused connecting to
localhost:4000 for /search/ejabberd, retry 4/5
[Thu, 08 Dec 2011 04:51:25 +0000] ERROR: Connection refused connecting to
localhost:4000 for /search/ejabberd, retry 5/5
Errno::ECONNREFUSED: Connection refused - Connection refused connecting
to
localhost:4000 for /search/ejabberd, giving up

but my /etc/chef/client.rb has the proper url:port for my chef server.
If I
run the same command from my local box I get back:

chef:recipe > search(:ejabberd, “id:deployment_000010182”)
=> []

which is what I expect. Any ideas where I should be looking?

Thanks,

MG


Cheers,

Peter Donald


#4

What’s a use-case for a node rewriting a data bag item during a converge,
that can’t be solved by some other method?

Note that it would have to scale to 100,000 nodes talking to the same
server and converging every 5 minutes, just as it would have to scale to 2
nodes talking to the same server and converging only when you SSH in to run
the chef-client.

  • Jay Feldblum

On Fri, Dec 9, 2011 at 12:48 PM, Michael Glenney mike.glenney@gmail.comwrote:

I figured out the issue. First of all, the 403 was because I was trying
to write to a databag within a recipe and didn’t read the disclaimer in the
wiki that said I’d have to give the node api client admin privileges to be
able to do that. Of course we don’t want to do that so I’ll have to come
up with another way to solve the problem.

BTW, I wouldn’t mind seeing chef have the capability to give a nodes
permission to write to a particular databag without giving the node admin
rights in case anyone is accepting Christmas wishes.

For the shef localhost:4000 error, that was just because, when running
shef from a client node, I forgot I need to launch shef with ‘shef -c
/etc/chef/client.rb’

MG

On Wed, Dec 7, 2011 at 10:09 PM, Peter Donald peter@realityforge.orgwrote:

Hi,

We had this exception when the chef-solr service died on the chef
server. Figure out what killed itc and restart was our approach.

  • This was a result of some process updating the owner/permissions on
    /var/log/chef and /var/run/chef so that solr failed during startup.

On Thu, Dec 8, 2011 at 4:02 PM, Michael Glenney mike.glenney@gmail.com
wrote:

I’m having problems with a 403 error when trying a new cookbook and I
think
I’ve tracked it down to databag access.

Chef Server 10.0
chef-client 10.4

The first several lines of the stacktrace:

Generated at 2011-12-08 04:12:07 +0000
Net::HTTPServerException: 403 “Forbidden”
/usr/lib/ruby/1.9.1/net/http.rb:2303:in error!' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:237:inblock in
api_request’
/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:288:in
retriable_rest_request' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:218:inapi_request’
/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:130:in
`put_rest’

/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/data_bag_item.rb:227:in

save' /var/chef/cache/cookbooks/ejabberd/recipes/default.rb:45:infrom_file’

/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/mixin/from_file.rb:30:in

`instance_eval’

/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/mixin/from_file.rb:30:in

`from_file’

/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/cookbook_version.rb:578:in

`load_recipe’

The relevant part of that recipe is:

Find cluster master, or create item and become master

if search(:ejabberd, “id:#{deployid}”).count == 0
masternode = "#{node[:ipaddress]}"
h = {}
h[deployid] = {“id” => deployid, “master” => masternode, “members” =>
[masternode]}

Create new data bag item for cluster

databag_item = Chef::DataBagItem.new
databag_item.data_bag(“ejabberd”)
databag_item.raw_data = h[deployid]
databag_item.save
else

This is for a first chef run on a new node. If I run shef, switch to
recipe
context, and run ‘search(:ejabberd, “id:deployment_000010182”)’ I get
back:

chef:recipe > search(:ejabberd, “id:deployment_000010182”)
[Thu, 08 Dec 2011 04:51:05 +0000] ERROR: Connection refused connecting
to
localhost:4000 for /search/ejabberd, retry 1/5
[Thu, 08 Dec 2011 04:51:10 +0000] ERROR: Connection refused connecting
to
localhost:4000 for /search/ejabberd, retry 2/5
[Thu, 08 Dec 2011 04:51:15 +0000] ERROR: Connection refused connecting
to
localhost:4000 for /search/ejabberd, retry 3/5
[Thu, 08 Dec 2011 04:51:20 +0000] ERROR: Connection refused connecting
to
localhost:4000 for /search/ejabberd, retry 4/5
[Thu, 08 Dec 2011 04:51:25 +0000] ERROR: Connection refused connecting
to
localhost:4000 for /search/ejabberd, retry 5/5
Errno::ECONNREFUSED: Connection refused - Connection refused connecting
to
localhost:4000 for /search/ejabberd, giving up

but my /etc/chef/client.rb has the proper url:port for my chef server.
If I
run the same command from my local box I get back:

chef:recipe > search(:ejabberd, “id:deployment_000010182”)
=> []

which is what I expect. Any ideas where I should be looking?

Thanks,

MG


Cheers,

Peter Donald


#5

One simple case is a node writing a lock for other nodes to block on. This would be for cluster buildup. Or other orchestration where you might need tighter timing than what search buys you. This already exists in dominodes cook: https://github.com/websterclay/chef-dominodes

Another case might be simple info exposure for other services. Again assuming you don’t have, need or want to use search for whatever reason.

This ability already exists on platform just not on OSS server.

On Dec 9, 2011, at 9:55 AM, Jay Feldblum y_feldblum@yahoo.com wrote:

What’s a use-case for a node rewriting a data bag item during a converge, that can’t be solved by some other method?

Note that it would have to scale to 100,000 nodes talking to the same server and converging every 5 minutes, just as it would have to scale to 2 nodes talking to the same server and converging only when you SSH in to run the chef-client.

  • Jay Feldblum

On Fri, Dec 9, 2011 at 12:48 PM, Michael Glenney mike.glenney@gmail.com wrote:
I figured out the issue. First of all, the 403 was because I was trying to write to a databag within a recipe and didn’t read the disclaimer in the wiki that said I’d have to give the node api client admin privileges to be able to do that. Of course we don’t want to do that so I’ll have to come up with another way to solve the problem.

BTW, I wouldn’t mind seeing chef have the capability to give a nodes permission to write to a particular databag without giving the node admin rights in case anyone is accepting Christmas wishes.

For the shef localhost:4000 error, that was just because, when running shef from a client node, I forgot I need to launch shef with ‘shef -c /etc/chef/client.rb’

MG

On Wed, Dec 7, 2011 at 10:09 PM, Peter Donald peter@realityforge.org wrote:
Hi,

We had this exception when the chef-solr service died on the chef
server. Figure out what killed itc and restart was our approach.

  • This was a result of some process updating the owner/permissions on
    /var/log/chef and /var/run/chef so that solr failed during startup.

On Thu, Dec 8, 2011 at 4:02 PM, Michael Glenney mike.glenney@gmail.com wrote:

I’m having problems with a 403 error when trying a new cookbook and I think
I’ve tracked it down to databag access.

Chef Server 10.0
chef-client 10.4

The first several lines of the stacktrace:

Generated at 2011-12-08 04:12:07 +0000
Net::HTTPServerException: 403 “Forbidden”
/usr/lib/ruby/1.9.1/net/http.rb:2303:in error!' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:237:inblock in
api_request’
/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:288:in
retriable_rest_request' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:218:inapi_request’
/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:130:in put_rest' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/data_bag_item.rb:227:insave’
/var/chef/cache/cookbooks/ejabberd/recipes/default.rb:45:in from_file' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/mixin/from_file.rb:30:ininstance_eval’
/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/mixin/from_file.rb:30:in
from_file' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/cookbook_version.rb:578:inload_recipe’

The relevant part of that recipe is:

Find cluster master, or create item and become master

if search(:ejabberd, “id:#{deployid}”).count == 0
masternode = "#{node[:ipaddress]}"
h = {}
h[deployid] = {“id” => deployid, “master” => masternode, “members” =>
[masternode]}

Create new data bag item for cluster

databag_item = Chef::DataBagItem.new
databag_item.data_bag(“ejabberd”)
databag_item.raw_data = h[deployid]
databag_item.save
else

This is for a first chef run on a new node. If I run shef, switch to recipe
context, and run ‘search(:ejabberd, “id:deployment_000010182”)’ I get back:

chef:recipe > search(:ejabberd, “id:deployment_000010182”)
[Thu, 08 Dec 2011 04:51:05 +0000] ERROR: Connection refused connecting to
localhost:4000 for /search/ejabberd, retry 1/5
[Thu, 08 Dec 2011 04:51:10 +0000] ERROR: Connection refused connecting to
localhost:4000 for /search/ejabberd, retry 2/5
[Thu, 08 Dec 2011 04:51:15 +0000] ERROR: Connection refused connecting to
localhost:4000 for /search/ejabberd, retry 3/5
[Thu, 08 Dec 2011 04:51:20 +0000] ERROR: Connection refused connecting to
localhost:4000 for /search/ejabberd, retry 4/5
[Thu, 08 Dec 2011 04:51:25 +0000] ERROR: Connection refused connecting to
localhost:4000 for /search/ejabberd, retry 5/5
Errno::ECONNREFUSED: Connection refused - Connection refused connecting to
localhost:4000 for /search/ejabberd, giving up

but my /etc/chef/client.rb has the proper url:port for my chef server. If I
run the same command from my local box I get back:

chef:recipe > search(:ejabberd, “id:deployment_000010182”)
=> []

which is what I expect. Any ideas where I should be looking?

Thanks,

MG


Cheers,

Peter Donald


#6

Cross-node synchronization of chef-client runs or of specific resources in
chef-client runs would be nice.

But do data bags provide a primitive sufficient for synchronization? I
doubt it. Perhaps such a primitive might be a useful feature for a future
version of chef-server-api.

Some people seem to be using tools like zookeeper or noah, which are built
for this type of job.

  • Jay

On Fri, Dec 9, 2011 at 3:36 PM, Jesse Nelson spheromak@gmail.com wrote:

One simple case is a node writing a lock for other nodes to block on. This
would be for cluster buildup. Or other orchestration where you might need
tighter timing than what search buys you. This already exists in
dominodes cook: https://github.com/websterclay/chef-dominodes

Another case might be simple info exposure for other services. Again
assuming you don’t have, need or want to use search for whatever reason.

This ability already exists on platform just not on OSS server.

On Dec 9, 2011, at 9:55 AM, Jay Feldblum y_feldblum@yahoo.com wrote:

What’s a use-case for a node rewriting a data bag item during a converge,
that can’t be solved by some other method?

Note that it would have to scale to 100,000 nodes talking to the same
server and converging every 5 minutes, just as it would have to scale to 2
nodes talking to the same server and converging only when you SSH in to run
the chef-client.

  • Jay Feldblum

On Fri, Dec 9, 2011 at 12:48 PM, Michael Glenney mike.glenney@gmail.comwrote:

I figured out the issue. First of all, the 403 was because I was trying
to write to a databag within a recipe and didn’t read the disclaimer in the
wiki that said I’d have to give the node api client admin privileges to be
able to do that. Of course we don’t want to do that so I’ll have to come
up with another way to solve the problem.

BTW, I wouldn’t mind seeing chef have the capability to give a nodes
permission to write to a particular databag without giving the node admin
rights in case anyone is accepting Christmas wishes.

For the shef localhost:4000 error, that was just because, when running
shef from a client node, I forgot I need to launch shef with ‘shef -c
/etc/chef/client.rb’

MG

On Wed, Dec 7, 2011 at 10:09 PM, Peter Donald peter@realityforge.orgwrote:

Hi,

We had this exception when the chef-solr service died on the chef
server. Figure out what killed itc and restart was our approach.

  • This was a result of some process updating the owner/permissions on
    /var/log/chef and /var/run/chef so that solr failed during startup.

On Thu, Dec 8, 2011 at 4:02 PM, Michael Glenney mike.glenney@gmail.com
wrote:

I’m having problems with a 403 error when trying a new cookbook and I
think
I’ve tracked it down to databag access.

Chef Server 10.0
chef-client 10.4

The first several lines of the stacktrace:

Generated at 2011-12-08 04:12:07 +0000
Net::HTTPServerException: 403 “Forbidden”
/usr/lib/ruby/1.9.1/net/http.rb:2303:in error!' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:237:inblock in
api_request’
/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:288:in
retriable_rest_request' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:218:inapi_request’
/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:130:in
`put_rest’

/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/data_bag_item.rb:227:in

save' /var/chef/cache/cookbooks/ejabberd/recipes/default.rb:45:infrom_file’

/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/mixin/from_file.rb:30:in

`instance_eval’

/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/mixin/from_file.rb:30:in

`from_file’

/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/cookbook_version.rb:578:in

`load_recipe’

The relevant part of that recipe is:

Find cluster master, or create item and become master

if search(:ejabberd, “id:#{deployid}”).count == 0
masternode = "#{node[:ipaddress]}"
h = {}
h[deployid] = {“id” => deployid, “master” => masternode, “members” =>
[masternode]}

Create new data bag item for cluster

databag_item = Chef::DataBagItem.new
databag_item.data_bag(“ejabberd”)
databag_item.raw_data = h[deployid]
databag_item.save
else

This is for a first chef run on a new node. If I run shef, switch to
recipe
context, and run ‘search(:ejabberd, “id:deployment_000010182”)’ I get
back:

chef:recipe > search(:ejabberd, “id:deployment_000010182”)
[Thu, 08 Dec 2011 04:51:05 +0000] ERROR: Connection refused connecting
to
localhost:4000 for /search/ejabberd, retry 1/5
[Thu, 08 Dec 2011 04:51:10 +0000] ERROR: Connection refused connecting
to
localhost:4000 for /search/ejabberd, retry 2/5
[Thu, 08 Dec 2011 04:51:15 +0000] ERROR: Connection refused connecting
to
localhost:4000 for /search/ejabberd, retry 3/5
[Thu, 08 Dec 2011 04:51:20 +0000] ERROR: Connection refused connecting
to
localhost:4000 for /search/ejabberd, retry 4/5
[Thu, 08 Dec 2011 04:51:25 +0000] ERROR: Connection refused connecting
to
localhost:4000 for /search/ejabberd, retry 5/5
Errno::ECONNREFUSED: Connection refused - Connection refused
connecting to
localhost:4000 for /search/ejabberd, giving up

but my /etc/chef/client.rb has the proper url:port for my chef
server. If I
run the same command from my local box I get back:

chef:recipe > search(:ejabberd, “id:deployment_000010182”)
=> []

which is what I expect. Any ideas where I should be looking?

Thanks,

MG


Cheers,

Peter Donald


#7

The use case here is clustering. The first node that comes up would
search the databage for an item named the deployment id. If it doesn’t
find an item with that name it would create the item and within that item
state that it is the master. The next node to come up with that deployment
id would find that item and extract the master node info.

A coworker suggested I use chef’s search capabilities instead. Search chef
for a node with that deployment id. If not found set an attribute claiming
itself as the master. Subsequent nodes would find that with search and
would extract what they need from the search results. This is the method
I’m going to use now but I initially rejected his idea (and am not happy
that were having to go this route) because I have to rely on solr properly
indexing my node before the next node comes up. I don’t like having to
rely on timing when it comes to automation. It’s sloppy in my eyes. With
the databag solution, I “know” the data is there because my call to write
it is successful.

A 3rd solution I came up with is to use Amazon SQS but I’d be happier with
a pure chef solution.

MG

On Fri, Dec 9, 2011 at 10:55 AM, Jay Feldblum y_feldblum@yahoo.com wrote:

What’s a use-case for a node rewriting a data bag item during a converge,
that can’t be solved by some other method?

Note that it would have to scale to 100,000 nodes talking to the same
server and converging every 5 minutes, just as it would have to scale to 2
nodes talking to the same server and converging only when you SSH in to run
the chef-client.

  • Jay Feldblum

On Fri, Dec 9, 2011 at 12:48 PM, Michael Glenney mike.glenney@gmail.comwrote:

I figured out the issue. First of all, the 403 was because I was trying
to write to a databag within a recipe and didn’t read the disclaimer in the
wiki that said I’d have to give the node api client admin privileges to be
able to do that. Of course we don’t want to do that so I’ll have to come
up with another way to solve the problem.

BTW, I wouldn’t mind seeing chef have the capability to give a nodes
permission to write to a particular databag without giving the node admin
rights in case anyone is accepting Christmas wishes.

For the shef localhost:4000 error, that was just because, when running
shef from a client node, I forgot I need to launch shef with ‘shef -c
/etc/chef/client.rb’

MG

On Wed, Dec 7, 2011 at 10:09 PM, Peter Donald peter@realityforge.orgwrote:

Hi,

We had this exception when the chef-solr service died on the chef
server. Figure out what killed itc and restart was our approach.

  • This was a result of some process updating the owner/permissions on
    /var/log/chef and /var/run/chef so that solr failed during startup.

On Thu, Dec 8, 2011 at 4:02 PM, Michael Glenney mike.glenney@gmail.com
wrote:

I’m having problems with a 403 error when trying a new cookbook and I
think
I’ve tracked it down to databag access.

Chef Server 10.0
chef-client 10.4

The first several lines of the stacktrace:

Generated at 2011-12-08 04:12:07 +0000
Net::HTTPServerException: 403 “Forbidden”
/usr/lib/ruby/1.9.1/net/http.rb:2303:in error!' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:237:inblock in
api_request’
/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:288:in
retriable_rest_request' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:218:inapi_request’
/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:130:in
`put_rest’

/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/data_bag_item.rb:227:in

save' /var/chef/cache/cookbooks/ejabberd/recipes/default.rb:45:infrom_file’

/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/mixin/from_file.rb:30:in

`instance_eval’

/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/mixin/from_file.rb:30:in

`from_file’

/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/cookbook_version.rb:578:in

`load_recipe’

The relevant part of that recipe is:

Find cluster master, or create item and become master

if search(:ejabberd, “id:#{deployid}”).count == 0
masternode = "#{node[:ipaddress]}"
h = {}
h[deployid] = {“id” => deployid, “master” => masternode, “members” =>
[masternode]}

Create new data bag item for cluster

databag_item = Chef::DataBagItem.new
databag_item.data_bag(“ejabberd”)
databag_item.raw_data = h[deployid]
databag_item.save
else

This is for a first chef run on a new node. If I run shef, switch to
recipe
context, and run ‘search(:ejabberd, “id:deployment_000010182”)’ I get
back:

chef:recipe > search(:ejabberd, “id:deployment_000010182”)
[Thu, 08 Dec 2011 04:51:05 +0000] ERROR: Connection refused connecting
to
localhost:4000 for /search/ejabberd, retry 1/5
[Thu, 08 Dec 2011 04:51:10 +0000] ERROR: Connection refused connecting
to
localhost:4000 for /search/ejabberd, retry 2/5
[Thu, 08 Dec 2011 04:51:15 +0000] ERROR: Connection refused connecting
to
localhost:4000 for /search/ejabberd, retry 3/5
[Thu, 08 Dec 2011 04:51:20 +0000] ERROR: Connection refused connecting
to
localhost:4000 for /search/ejabberd, retry 4/5
[Thu, 08 Dec 2011 04:51:25 +0000] ERROR: Connection refused connecting
to
localhost:4000 for /search/ejabberd, retry 5/5
Errno::ECONNREFUSED: Connection refused - Connection refused
connecting to
localhost:4000 for /search/ejabberd, giving up

but my /etc/chef/client.rb has the proper url:port for my chef
server. If I
run the same command from my local box I get back:

chef:recipe > search(:ejabberd, “id:deployment_000010182”)
=> []

which is what I expect. Any ideas where I should be looking?

Thanks,

MG


Cheers,

Peter Donald


#8

Chef is a distributed system. Using search would be a good approach, so
long as your code takes that into account. If you go that route, you don’t
rely on timing; rather, you code for the fact that timing may be “off.” I
don’t think that’s sloppy - I think it’s part of what it means for nodes to
"converge" over time, over multiple runs, to a final “finished” state.

For example, to bring up a cluster, here’s what you can do. Add a
normal-level attribute on a first node indicating it is the seed node
(don’t add that attribute to any other node). Then during a convergence,
each node will only create the cluster if it is the seed node, and will
only add itself to the cluster if some other node is the seed node (found
by search) and that node is reachable and healthy. The seed node might not
be the primary node coordinating the cluster, but any node should be able
to find the primary node by asking the seed node what the primary is. If
the original seed node fails at a later point and needs to be removed from
the cluster, just delete that node object from the chef-server and make a
different node the new seed node (by setting that specific normal-level
attribute) before bringing up any new nodes into the cluster. Note that
normal-level attributes may be set via knife or via the webui, and that
they persist across chef-client runs. You can use the node tags for the
purpose of the seed attribute, since node tags are normal-level attributes
and are easily edited via the CLI (knife tag create NODE TAG) without
requiring an editor. There is certainly room for extension and variation -
for example, to eliminate the SPOF where there can only be one seed node.

  • Jay Feldblum

On Fri, Dec 9, 2011 at 6:39 PM, Michael Glenney mike.glenney@gmail.comwrote:

The use case here is clustering. The first node that comes up would
search the databage for an item named the deployment id. If it doesn’t
find an item with that name it would create the item and within that item
state that it is the master. The next node to come up with that deployment
id would find that item and extract the master node info.

A coworker suggested I use chef’s search capabilities instead. Search
chef for a node with that deployment id. If not found set an attribute
claiming itself as the master. Subsequent nodes would find that with
search and would extract what they need from the search results. This is
the method I’m going to use now but I initially rejected his idea (and am
not happy that were having to go this route) because I have to rely on solr
properly indexing my node before the next node comes up. I don’t like
having to rely on timing when it comes to automation. It’s sloppy in my
eyes. With the databag solution, I “know” the data is there because my
call to write it is successful.

A 3rd solution I came up with is to use Amazon SQS but I’d be happier with
a pure chef solution.

MG

On Fri, Dec 9, 2011 at 10:55 AM, Jay Feldblum y_feldblum@yahoo.comwrote:

What’s a use-case for a node rewriting a data bag item during a converge,
that can’t be solved by some other method?

Note that it would have to scale to 100,000 nodes talking to the same
server and converging every 5 minutes, just as it would have to scale to 2
nodes talking to the same server and converging only when you SSH in to run
the chef-client.

  • Jay Feldblum

On Fri, Dec 9, 2011 at 12:48 PM, Michael Glenney mike.glenney@gmail.comwrote:

I figured out the issue. First of all, the 403 was because I was trying
to write to a databag within a recipe and didn’t read the disclaimer in the
wiki that said I’d have to give the node api client admin privileges to be
able to do that. Of course we don’t want to do that so I’ll have to come
up with another way to solve the problem.

BTW, I wouldn’t mind seeing chef have the capability to give a nodes
permission to write to a particular databag without giving the node admin
rights in case anyone is accepting Christmas wishes.

For the shef localhost:4000 error, that was just because, when running
shef from a client node, I forgot I need to launch shef with ‘shef -c
/etc/chef/client.rb’

MG

On Wed, Dec 7, 2011 at 10:09 PM, Peter Donald peter@realityforge.orgwrote:

Hi,

We had this exception when the chef-solr service died on the chef
server. Figure out what killed itc and restart was our approach.

  • This was a result of some process updating the owner/permissions on
    /var/log/chef and /var/run/chef so that solr failed during startup.

On Thu, Dec 8, 2011 at 4:02 PM, Michael Glenney mike.glenney@gmail.com
wrote:

I’m having problems with a 403 error when trying a new cookbook and I
think
I’ve tracked it down to databag access.

Chef Server 10.0
chef-client 10.4

The first several lines of the stacktrace:

Generated at 2011-12-08 04:12:07 +0000
Net::HTTPServerException: 403 “Forbidden”
/usr/lib/ruby/1.9.1/net/http.rb:2303:in error!' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:237:inblock in
api_request’
/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:288:in
retriable_rest_request' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:218:inapi_request’
/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:130:in
`put_rest’

/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/data_bag_item.rb:227:in

save' /var/chef/cache/cookbooks/ejabberd/recipes/default.rb:45:infrom_file’

/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/mixin/from_file.rb:30:in

`instance_eval’

/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/mixin/from_file.rb:30:in

`from_file’

/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/cookbook_version.rb:578:in

`load_recipe’

The relevant part of that recipe is:

Find cluster master, or create item and become master

if search(:ejabberd, “id:#{deployid}”).count == 0
masternode = "#{node[:ipaddress]}"
h = {}
h[deployid] = {“id” => deployid, “master” => masternode, “members”
=>
[masternode]}

Create new data bag item for cluster

databag_item = Chef::DataBagItem.new
databag_item.data_bag(“ejabberd”)
databag_item.raw_data = h[deployid]
databag_item.save
else

This is for a first chef run on a new node. If I run shef, switch to
recipe
context, and run ‘search(:ejabberd, “id:deployment_000010182”)’ I get
back:

chef:recipe > search(:ejabberd, “id:deployment_000010182”)
[Thu, 08 Dec 2011 04:51:05 +0000] ERROR: Connection refused
connecting to
localhost:4000 for /search/ejabberd, retry 1/5
[Thu, 08 Dec 2011 04:51:10 +0000] ERROR: Connection refused
connecting to
localhost:4000 for /search/ejabberd, retry 2/5
[Thu, 08 Dec 2011 04:51:15 +0000] ERROR: Connection refused
connecting to
localhost:4000 for /search/ejabberd, retry 3/5
[Thu, 08 Dec 2011 04:51:20 +0000] ERROR: Connection refused
connecting to
localhost:4000 for /search/ejabberd, retry 4/5
[Thu, 08 Dec 2011 04:51:25 +0000] ERROR: Connection refused
connecting to
localhost:4000 for /search/ejabberd, retry 5/5
Errno::ECONNREFUSED: Connection refused - Connection refused
connecting to
localhost:4000 for /search/ejabberd, giving up

but my /etc/chef/client.rb has the proper url:port for my chef
server. If I
run the same command from my local box I get back:

chef:recipe > search(:ejabberd, “id:deployment_000010182”)
=> []

which is what I expect. Any ideas where I should be looking?

Thanks,

MG


Cheers,

Peter Donald


#9

Biggest issue I’ve had with search is that a fail state node will not update its node attributes. Causing search to not reflect reality. I like the idea of an infra wide mutex.

Also for cluster master election you might want to look at fujin’s pylon. Which intends to solve that exact problem. https://github.com/fujin/pylon

On Dec 9, 2011, at 3:39 PM, Michael Glenney mike.glenney@gmail.com wrote:

The use case here is clustering. The first node that comes up would search the databage for an item named the deployment id. If it doesn’t find an item with that name it would create the item and within that item state that it is the master. The next node to come up with that deployment id would find that item and extract the master node info.

A coworker suggested I use chef’s search capabilities instead. Search chef for a node with that deployment id. If not found set an attribute claiming itself as the master. Subsequent nodes would find that with search and would extract what they need from the search results. This is the method I’m going to use now but I initially rejected his idea (and am not happy that were having to go this route) because I have to rely on solr properly indexing my node before the next node comes up. I don’t like having to rely on timing when it comes to automation. It’s sloppy in my eyes. With the databag solution, I “know” the data is there because my call to write it is successful.

A 3rd solution I came up with is to use Amazon SQS but I’d be happier with a pure chef solution.

MG

On Fri, Dec 9, 2011 at 10:55 AM, Jay Feldblum y_feldblum@yahoo.com wrote:
What’s a use-case for a node rewriting a data bag item during a converge, that can’t be solved by some other method?

Note that it would have to scale to 100,000 nodes talking to the same server and converging every 5 minutes, just as it would have to scale to 2 nodes talking to the same server and converging only when you SSH in to run the chef-client.

  • Jay Feldblum

On Fri, Dec 9, 2011 at 12:48 PM, Michael Glenney mike.glenney@gmail.com wrote:
I figured out the issue. First of all, the 403 was because I was trying to write to a databag within a recipe and didn’t read the disclaimer in the wiki that said I’d have to give the node api client admin privileges to be able to do that. Of course we don’t want to do that so I’ll have to come up with another way to solve the problem.

BTW, I wouldn’t mind seeing chef have the capability to give a nodes permission to write to a particular databag without giving the node admin rights in case anyone is accepting Christmas wishes.

For the shef localhost:4000 error, that was just because, when running shef from a client node, I forgot I need to launch shef with ‘shef -c /etc/chef/client.rb’

MG

On Wed, Dec 7, 2011 at 10:09 PM, Peter Donald peter@realityforge.org wrote:
Hi,

We had this exception when the chef-solr service died on the chef
server. Figure out what killed itc and restart was our approach.

  • This was a result of some process updating the owner/permissions on
    /var/log/chef and /var/run/chef so that solr failed during startup.

On Thu, Dec 8, 2011 at 4:02 PM, Michael Glenney mike.glenney@gmail.com wrote:

I’m having problems with a 403 error when trying a new cookbook and I think
I’ve tracked it down to databag access.

Chef Server 10.0
chef-client 10.4

The first several lines of the stacktrace:

Generated at 2011-12-08 04:12:07 +0000
Net::HTTPServerException: 403 “Forbidden”
/usr/lib/ruby/1.9.1/net/http.rb:2303:in error!' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:237:inblock in
api_request’
/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:288:in
retriable_rest_request' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:218:inapi_request’
/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:130:in put_rest' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/data_bag_item.rb:227:insave’
/var/chef/cache/cookbooks/ejabberd/recipes/default.rb:45:in from_file' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/mixin/from_file.rb:30:ininstance_eval’
/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/mixin/from_file.rb:30:in
from_file' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/cookbook_version.rb:578:inload_recipe’

The relevant part of that recipe is:

Find cluster master, or create item and become master

if search(:ejabberd, “id:#{deployid}”).count == 0
masternode = "#{node[:ipaddress]}"
h = {}
h[deployid] = {“id” => deployid, “master” => masternode, “members” =>
[masternode]}

Create new data bag item for cluster

databag_item = Chef::DataBagItem.new
databag_item.data_bag(“ejabberd”)
databag_item.raw_data = h[deployid]
databag_item.save
else

This is for a first chef run on a new node. If I run shef, switch to recipe
context, and run ‘search(:ejabberd, “id:deployment_000010182”)’ I get back:

chef:recipe > search(:ejabberd, “id:deployment_000010182”)
[Thu, 08 Dec 2011 04:51:05 +0000] ERROR: Connection refused connecting to
localhost:4000 for /search/ejabberd, retry 1/5
[Thu, 08 Dec 2011 04:51:10 +0000] ERROR: Connection refused connecting to
localhost:4000 for /search/ejabberd, retry 2/5
[Thu, 08 Dec 2011 04:51:15 +0000] ERROR: Connection refused connecting to
localhost:4000 for /search/ejabberd, retry 3/5
[Thu, 08 Dec 2011 04:51:20 +0000] ERROR: Connection refused connecting to
localhost:4000 for /search/ejabberd, retry 4/5
[Thu, 08 Dec 2011 04:51:25 +0000] ERROR: Connection refused connecting to
localhost:4000 for /search/ejabberd, retry 5/5
Errno::ECONNREFUSED: Connection refused - Connection refused connecting to
localhost:4000 for /search/ejabberd, giving up

but my /etc/chef/client.rb has the proper url:port for my chef server. If I
run the same command from my local box I get back:

chef:recipe > search(:ejabberd, “id:deployment_000010182”)
=> []

which is what I expect. Any ideas where I should be looking?

Thanks,

MG


Cheers,

Peter Donald


#10

Pylon is probably a few works of solid code away from being usable,
but it does perform a master election successfully over multicast.
Getting it integrated with Chef is on my list of things to do…

If you use search for this kind of stuff, it’s pretty easy to throw
out bad nodes. I do a few searches for roles:foo, role:foo, then fall
back to node.run_list.include? foo…

Adam has been talking about some locking primitives for Chef that
would probably make this all a little easier and hopefully without a
central server or anything.

–AJ

On 10 December 2011 17:23, Jesse Nelson spheromak@gmail.com wrote:

Biggest issue I’ve had with search is that a fail state node will not update
its node attributes. Causing search to not reflect reality. I like the idea
of an infra wide mutex.

Also for cluster master election you might want to look at fujin’s pylon.
Which intends to solve that exact problem. https://github.com/fujin/pylon

On Dec 9, 2011, at 3:39 PM, Michael Glenney mike.glenney@gmail.com wrote:

The use case here is clustering. The first node that comes up would search
the databage for an item named the deployment id. If it doesn’t find an
item with that name it would create the item and within that item state that
it is the master. The next node to come up with that deployment id would
find that item and extract the master node info.

A coworker suggested I use chef’s search capabilities instead. Search chef
for a node with that deployment id. If not found set an attribute claiming
itself as the master. Subsequent nodes would find that with search and
would extract what they need from the search results. This is the method
I’m going to use now but I initially rejected his idea (and am not happy
that were having to go this route) because I have to rely on solr properly
indexing my node before the next node comes up. I don’t like having to rely
on timing when it comes to automation. It’s sloppy in my eyes. With the
databag solution, I “know” the data is there because my call to write it is
successful.

A 3rd solution I came up with is to use Amazon SQS but I’d be happier with a
pure chef solution.

MG

On Fri, Dec 9, 2011 at 10:55 AM, Jay Feldblum y_feldblum@yahoo.com wrote:

What’s a use-case for a node rewriting a data bag item during a converge,
that can’t be solved by some other method?

Note that it would have to scale to 100,000 nodes talking to the same
server and converging every 5 minutes, just as it would have to scale to 2
nodes talking to the same server and converging only when you SSH in to run
the chef-client.

  • Jay Feldblum

On Fri, Dec 9, 2011 at 12:48 PM, Michael Glenney mike.glenney@gmail.com
wrote:

I figured out the issue. First of all, the 403 was because I was trying
to write to a databag within a recipe and didn’t read the disclaimer in the
wiki that said I’d have to give the node api client admin privileges to be
able to do that. Of course we don’t want to do that so I’ll have to come up
with another way to solve the problem.

BTW, I wouldn’t mind seeing chef have the capability to give a nodes
permission to write to a particular databag without giving the node admin
rights in case anyone is accepting Christmas wishes.

For the shef localhost:4000 error, that was just because, when running
shef from a client node, I forgot I need to launch shef with ‘shef -c
/etc/chef/client.rb’

MG

On Wed, Dec 7, 2011 at 10:09 PM, Peter Donald peter@realityforge.org
wrote:

Hi,

We had this exception when the chef-solr service died on the chef
server. Figure out what killed itc and restart was our approach.

  • This was a result of some process updating the owner/permissions on
    /var/log/chef and /var/run/chef so that solr failed during startup.

On Thu, Dec 8, 2011 at 4:02 PM, Michael Glenney mike.glenney@gmail.com
wrote:

I’m having problems with a 403 error when trying a new cookbook and I
think
I’ve tracked it down to databag access.

Chef Server 10.0
chef-client 10.4

The first several lines of the stacktrace:

Generated at 2011-12-08 04:12:07 +0000
Net::HTTPServerException: 403 “Forbidden”
/usr/lib/ruby/1.9.1/net/http.rb:2303:in error!' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:237:inblock in
api_request’
/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:288:in
retriable_rest_request' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:218:inapi_request’
/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:130:in
`put_rest’

/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/data_bag_item.rb:227:in
save' /var/chef/cache/cookbooks/ejabberd/recipes/default.rb:45:infrom_file’

/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/mixin/from_file.rb:30:in
`instance_eval’

/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/mixin/from_file.rb:30:in
`from_file’

/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/cookbook_version.rb:578:in
`load_recipe’

The relevant part of that recipe is:

Find cluster master, or create item and become master

if search(:ejabberd, “id:#{deployid}”).count == 0
masternode = "#{node[:ipaddress]}"
h = {}
h[deployid] = {“id” => deployid, “master” => masternode, “members”
=>
[masternode]}

Create new data bag item for cluster

databag_item = Chef::DataBagItem.new
databag_item.data_bag(“ejabberd”)
databag_item.raw_data = h[deployid]
databag_item.save
else

This is for a first chef run on a new node. If I run shef, switch to
recipe
context, and run ‘search(:ejabberd, “id:deployment_000010182”)’ I get
back:

chef:recipe > search(:ejabberd, “id:deployment_000010182”)
[Thu, 08 Dec 2011 04:51:05 +0000] ERROR: Connection refused connecting
to
localhost:4000 for /search/ejabberd, retry 1/5
[Thu, 08 Dec 2011 04:51:10 +0000] ERROR: Connection refused connecting
to
localhost:4000 for /search/ejabberd, retry 2/5
[Thu, 08 Dec 2011 04:51:15 +0000] ERROR: Connection refused connecting
to
localhost:4000 for /search/ejabberd, retry 3/5
[Thu, 08 Dec 2011 04:51:20 +0000] ERROR: Connection refused connecting
to
localhost:4000 for /search/ejabberd, retry 4/5
[Thu, 08 Dec 2011 04:51:25 +0000] ERROR: Connection refused connecting
to
localhost:4000 for /search/ejabberd, retry 5/5
Errno::ECONNREFUSED: Connection refused - Connection refused
connecting to
localhost:4000 for /search/ejabberd, giving up

but my /etc/chef/client.rb has the proper url:port for my chef
server. If I
run the same command from my local box I get back:

chef:recipe > search(:ejabberd, “id:deployment_000010182”)
=> []

which is what I expect. Any ideas where I should be looking?

Thanks,

MG


Cheers,

Peter Donald