# Problem accessing databag from node

**URL:** <https://discourse.chef.io/t/problem-accessing-databag-from-node/2314>\
**Category:** Chef Infra (archive)\
**Created:** [December 8, 2011, 5:02am UTC](https://discourse.chef.io/t/problem-accessing-databag-from-node/2314 "2011-12-08T05:02:30Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Michael\_Glenney](https://avatars.discourse-cdn.com/v4/letter/m/a9adbd/32.png) [@Michael\_Glenney](https://discourse.chef.io/u/Michael_Glenney)\
**Post date:** [December 8, 2011, 5:02am UTC](https://discourse.chef.io/t/problem-accessing-databag-from-node/2314/1 "2011-12-08T05:02:30Z")

</div>

I’m having problems with a 403 error when trying a new cookbook and I think  
I’ve tracked it down to databag access.

Chef Server 10.0  
chef-client 10.4

The first several lines of the stacktrace:

Generated at 2011-12-08 04:12:07 +0000  
Net::HTTPServerException: 403 “Forbidden”  
/usr/lib/ruby/1.9.1/net/http.rb:2303:in `error!' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:237:in`block in  
api\_request’  
/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:288:in  
`retriable_rest_request' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:218:in`api\_request’  
/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:130:in `put_rest' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/data_bag_item.rb:227:in`save’  
/var/chef/cache/cookbooks/ejabberd/recipes/default.rb:45:in `from_file' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/mixin/from_file.rb:30:in`instance\_eval’  
/usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/mixin/from\_file.rb:30:in  
`from_file' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/cookbook_version.rb:578:in`load\_recipe’

The relevant part of that recipe is:

# Find cluster master, or create item and become master

if search(:ejabberd, “id:#{deployid}”).count == 0  
masternode = "#{node[:ipaddress]}"  
h = {}  
h[deployid] = {“id” =\> deployid, “master” =\> masternode, “members” =\>  
[masternode]}

# Create new data bag item for cluster

databag\_item = Chef::DataBagItem.new  
databag\_item.data\_bag(“ejabberd”)  
databag\_item.raw\_data = h[deployid]  
databag\_item.save  
else

This is for a first chef run on a new node. If I run shef, switch to  
recipe context, and run ‘search(:ejabberd, “id:deployment\_000010182”)’ I  
get back:

chef:recipe \> search(:ejabberd, “id:deployment\_000010182”)  
[Thu, 08 Dec 2011 04:51:05 +0000] ERROR: Connection refused connecting to  
localhost:4000 for /search/ejabberd, retry 1/5  
[Thu, 08 Dec 2011 04:51:10 +0000] ERROR: Connection refused connecting to  
localhost:4000 for /search/ejabberd, retry 2/5  
[Thu, 08 Dec 2011 04:51:15 +0000] ERROR: Connection refused connecting to  
localhost:4000 for /search/ejabberd, retry 3/5  
[Thu, 08 Dec 2011 04:51:20 +0000] ERROR: Connection refused connecting to  
localhost:4000 for /search/ejabberd, retry 4/5  
[Thu, 08 Dec 2011 04:51:25 +0000] ERROR: Connection refused connecting to  
localhost:4000 for /search/ejabberd, retry 5/5  
Errno::ECONNREFUSED: Connection refused - Connection refused connecting to  
localhost:4000 for /search/ejabberd, giving up

but my /etc/chef/client.rb has the proper url:port for my chef server. If  
I run the same command from my local box I get back:

chef:recipe \> search(:ejabberd, “id:deployment\_000010182”)  
=\> []

which is what I expect. Any ideas where I should be looking?

Thanks,

MG

---

<div class="post-metadata">

**Author:** ![realityforge](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/realityforge/32/395_2.png) [@realityforge](https://discourse.chef.io/u/realityforge)\
**Post date:** [December 8, 2011, 5:09am UTC](https://discourse.chef.io/t/problem-accessing-databag-from-node/2314/2 "2011-12-08T05:09:42Z")

</div>

Hi,

We had this exception when the chef-solr service died on the chef  
server. Figure out what killed itc and restart was our approach.

- This was a result of some process updating the owner/permissions on  
/var/log/chef and /var/run/chef so that solr failed during startup.

On Thu, Dec 8, 2011 at 4:02 PM, Michael Glenney [mike.glenney@gmail.com](mailto:mike.glenney@gmail.com) wrote:

> I'm having problems with a 403 error when trying a new cookbook and I think  
> I've tracked it down to databag access.
> 
> Chef Server 10.0  
> chef-client 10.4
> 
> The first several lines of the stacktrace:
> 
> Generated at 2011-12-08 04:12:07 +0000  
> Net::HTTPServerException: 403 "Forbidden"  
> /usr/lib/ruby/1.9.1/net/http.rb:2303:in `error!' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:237:in `block in  
> api\_request'  
> /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:288:in  
> `retriable_rest_request' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:218:in `api\_request'  
> /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:130:in `put_rest' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/data_bag_item.rb:227:in `save'  
> /var/chef/cache/cookbooks/ejabberd/recipes/default.rb:45:in `from_file' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/mixin/from_file.rb:30:in `instance\_eval'  
> /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/mixin/from\_file.rb:30:in  
> `from_file' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/cookbook_version.rb:578:in `load\_recipe'
> 
> The relevant part of that recipe is:
> 
> # Find cluster master, or create item and become master
> 
> if search(:ejabberd, "id:#{deployid}").count == 0  
> masternode = "#{node[:ipaddress]}"  
> h = {}  
> h[deployid] = {"id" =\> deployid, "master" =\> masternode, "members" =\>  
> [masternode]}
> 
> # Create new data bag item for cluster
> 
> databag\_item = Chef::DataBagItem.new  
> databag\_item.data\_bag("ejabberd")  
> databag\_item.raw\_data = h[deployid]  
> databag\_item.save  
> else
> 
> This is for a first chef run on a new node. If I run shef, switch to recipe  
> context, and run 'search(:ejabberd, "id:deployment\_000010182")' I get back:
> 
> chef:recipe \> search(:ejabberd, "id:deployment\_000010182")  
> [Thu, 08 Dec 2011 04:51:05 +0000] ERROR: Connection refused connecting to  
> localhost:4000 for /search/ejabberd, retry 1/5  
> [Thu, 08 Dec 2011 04:51:10 +0000] ERROR: Connection refused connecting to  
> localhost:4000 for /search/ejabberd, retry 2/5  
> [Thu, 08 Dec 2011 04:51:15 +0000] ERROR: Connection refused connecting to  
> localhost:4000 for /search/ejabberd, retry 3/5  
> [Thu, 08 Dec 2011 04:51:20 +0000] ERROR: Connection refused connecting to  
> localhost:4000 for /search/ejabberd, retry 4/5  
> [Thu, 08 Dec 2011 04:51:25 +0000] ERROR: Connection refused connecting to  
> localhost:4000 for /search/ejabberd, retry 5/5  
> Errno::ECONNREFUSED: Connection refused - Connection refused connecting to  
> localhost:4000 for /search/ejabberd, giving up
> 
> but my /etc/chef/client.rb has the proper url:port for my chef server. If I  
> run the same command from my local box I get back:
> 
> chef:recipe \> search(:ejabberd, "id:deployment\_000010182")  
> =\>
> 
> which is what I expect. Any ideas where I should be looking?
> 
> Thanks,
> 
> MG

--  
Cheers,

Peter Donald

---

<div class="post-metadata">

**Author:** ![Michael\_Glenney](https://avatars.discourse-cdn.com/v4/letter/m/a9adbd/32.png) [@Michael\_Glenney](https://discourse.chef.io/u/Michael_Glenney)\
**Post date:** [December 9, 2011, 5:48pm UTC](https://discourse.chef.io/t/problem-accessing-databag-from-node/2314/3 "2011-12-09T17:48:12Z")

</div>

I figured out the issue. First of all, the 403 was because I was trying to  
write to a databag within a recipe and didn't read the disclaimer in the  
wiki that said I'd have to give the node api client admin privileges to be  
able to do that. Of course we don't want to do that so I'll have to come  
up with another way to solve the problem.

BTW, I wouldn't mind seeing chef have the capability to give a nodes  
permission to write to a particular databag without giving the node admin  
rights in case anyone is accepting Christmas wishes.

For the shef localhost:4000 error, that was just because, when running shef  
from a client node, I forgot I need to launch shef with 'shef -c  
/etc/chef/client.rb'

MG

On Wed, Dec 7, 2011 at 10:09 PM, Peter Donald [peter@realityforge.org](mailto:peter@realityforge.org)wrote:

> Hi,
> 
> We had this exception when the chef-solr service died on the chef  
> server. Figure out what killed itc and restart was our approach.
> 
> - This was a result of some process updating the owner/permissions on  
> /var/log/chef and /var/run/chef so that solr failed during startup.
> 
> On Thu, Dec 8, 2011 at 4:02 PM, Michael Glenney [mike.glenney@gmail.com](mailto:mike.glenney@gmail.com)  
> wrote:
> 
> > I'm having problems with a 403 error when trying a new cookbook and I  
> > think  
> > I've tracked it down to databag access.
> > 
> > Chef Server 10.0  
> > chef-client 10.4
> > 
> > The first several lines of the stacktrace:
> > 
> > Generated at 2011-12-08 04:12:07 +0000  
> > Net::HTTPServerException: 403 "Forbidden"  
> > /usr/lib/ruby/1.9.1/net/http.rb:2303:in `error!' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:237:in `block  
> > in  
> > api\_request'  
> > /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:288:in  
> > `retriable_rest_request' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:218:in `api\_request'  
> > /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:130:in  
> > `put\_rest'
> 
> /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/data\_bag\_item.rb:227:in
> 
> > `save' /var/chef/cache/cookbooks/ejabberd/recipes/default.rb:45:in `from\_file'
> 
> /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/mixin/from\_file.rb:30:in
> 
> > `instance\_eval'
> 
> /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/mixin/from\_file.rb:30:in
> 
> > `from\_file'
> 
> /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/cookbook\_version.rb:578:in
> 
> > `load\_recipe'
> > 
> > The relevant part of that recipe is:
> > 
> > # Find cluster master, or create item and become master
> > 
> > if search(:ejabberd, "id:#{deployid}").count == 0  
> > masternode = "#{node[:ipaddress]}"  
> > h = {}  
> > h[deployid] = {"id" =\> deployid, "master" =\> masternode, "members" =\>  
> > [masternode]}
> > 
> > # Create new data bag item for cluster
> > 
> > databag\_item = Chef::DataBagItem.new  
> > databag\_item.data\_bag("ejabberd")  
> > databag\_item.raw\_data = h[deployid]  
> > databag\_item.save  
> > else
> > 
> > This is for a first chef run on a new node. If I run shef, switch to  
> > recipe  
> > context, and run 'search(:ejabberd, "id:deployment\_000010182")' I get  
> > back:
> > 
> > chef:recipe \> search(:ejabberd, "id:deployment\_000010182")  
> > [Thu, 08 Dec 2011 04:51:05 +0000] ERROR: Connection refused connecting to  
> > localhost:4000 for /search/ejabberd, retry 1/5  
> > [Thu, 08 Dec 2011 04:51:10 +0000] ERROR: Connection refused connecting to  
> > localhost:4000 for /search/ejabberd, retry 2/5  
> > [Thu, 08 Dec 2011 04:51:15 +0000] ERROR: Connection refused connecting to  
> > localhost:4000 for /search/ejabberd, retry 3/5  
> > [Thu, 08 Dec 2011 04:51:20 +0000] ERROR: Connection refused connecting to  
> > localhost:4000 for /search/ejabberd, retry 4/5  
> > [Thu, 08 Dec 2011 04:51:25 +0000] ERROR: Connection refused connecting to  
> > localhost:4000 for /search/ejabberd, retry 5/5  
> > Errno::ECONNREFUSED: Connection refused - Connection refused connecting  
> > to  
> > localhost:4000 for /search/ejabberd, giving up
> > 
> > but my /etc/chef/client.rb has the proper url:port for my chef server.  
> > If I  
> > run the same command from my local box I get back:
> > 
> > chef:recipe \> search(:ejabberd, "id:deployment\_000010182")  
> > =\>
> > 
> > which is what I expect. Any ideas where I should be looking?
> > 
> > Thanks,
> > 
> > MG
> 
> --  
> Cheers,
> 
> Peter Donald

---

<div class="post-metadata">

**Author:** ![Jay\_Feldblum](https://avatars.discourse-cdn.com/v4/letter/j/9fc29f/32.png) [@Jay\_Feldblum](https://discourse.chef.io/u/Jay_Feldblum)\
**Post date:** [December 9, 2011, 5:55pm UTC](https://discourse.chef.io/t/problem-accessing-databag-from-node/2314/4 "2011-12-09T17:55:30Z")

</div>

What's a use-case for a node rewriting a data bag item during a converge,  
that can't be solved by some other method?

Note that it would have to scale to 100,000 nodes talking to the same  
server and converging every 5 minutes, just as it would have to scale to 2  
nodes talking to the same server and converging only when you SSH in to run  
the chef-client.

- Jay Feldblum

On Fri, Dec 9, 2011 at 12:48 PM, Michael Glenney [mike.glenney@gmail.com](mailto:mike.glenney@gmail.com)wrote:

> I figured out the issue. First of all, the 403 was because I was trying  
> to write to a databag within a recipe and didn't read the disclaimer in the  
> wiki that said I'd have to give the node api client admin privileges to be  
> able to do that. Of course we don't want to do that so I'll have to come  
> up with another way to solve the problem.
> 
> BTW, I wouldn't mind seeing chef have the capability to give a nodes  
> permission to write to a particular databag without giving the node admin  
> rights in case anyone is accepting Christmas wishes.
> 
> For the shef localhost:4000 error, that was just because, when running  
> shef from a client node, I forgot I need to launch shef with 'shef -c  
> /etc/chef/client.rb'
> 
> MG
> 
> On Wed, Dec 7, 2011 at 10:09 PM, Peter Donald [peter@realityforge.org](mailto:peter@realityforge.org)wrote:
> 
> > Hi,
> > 
> > We had this exception when the chef-solr service died on the chef  
> > server. Figure out what killed itc and restart was our approach.
> > 
> > - This was a result of some process updating the owner/permissions on  
> > /var/log/chef and /var/run/chef so that solr failed during startup.
> > 
> > On Thu, Dec 8, 2011 at 4:02 PM, Michael Glenney [mike.glenney@gmail.com](mailto:mike.glenney@gmail.com)  
> > wrote:
> > 
> > > I'm having problems with a 403 error when trying a new cookbook and I  
> > > think  
> > > I've tracked it down to databag access.
> > > 
> > > Chef Server 10.0  
> > > chef-client 10.4
> > > 
> > > The first several lines of the stacktrace:
> > > 
> > > Generated at 2011-12-08 04:12:07 +0000  
> > > Net::HTTPServerException: 403 "Forbidden"  
> > > /usr/lib/ruby/1.9.1/net/http.rb:2303:in `error!' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:237:in `block in  
> > > api\_request'  
> > > /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:288:in  
> > > `retriable_rest_request' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:218:in `api\_request'  
> > > /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:130:in  
> > > `put\_rest'
> > 
> > /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/data\_bag\_item.rb:227:in
> > 
> > > `save' /var/chef/cache/cookbooks/ejabberd/recipes/default.rb:45:in `from\_file'
> > 
> > /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/mixin/from\_file.rb:30:in
> > 
> > > `instance\_eval'
> > 
> > /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/mixin/from\_file.rb:30:in
> > 
> > > `from\_file'
> > 
> > /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/cookbook\_version.rb:578:in
> > 
> > > `load\_recipe'
> > > 
> > > The relevant part of that recipe is:
> > > 
> > > # Find cluster master, or create item and become master
> > > 
> > > if search(:ejabberd, "id:#{deployid}").count == 0  
> > > masternode = "#{node[:ipaddress]}"  
> > > h = {}  
> > > h[deployid] = {"id" =\> deployid, "master" =\> masternode, "members" =\>  
> > > [masternode]}
> > > 
> > > # Create new data bag item for cluster
> > > 
> > > databag\_item = Chef::DataBagItem.new  
> > > databag\_item.data\_bag("ejabberd")  
> > > databag\_item.raw\_data = h[deployid]  
> > > databag\_item.save  
> > > else
> > > 
> > > This is for a first chef run on a new node. If I run shef, switch to  
> > > recipe  
> > > context, and run 'search(:ejabberd, "id:deployment\_000010182")' I get  
> > > back:
> > > 
> > > chef:recipe \> search(:ejabberd, "id:deployment\_000010182")  
> > > [Thu, 08 Dec 2011 04:51:05 +0000] ERROR: Connection refused connecting  
> > > to  
> > > localhost:4000 for /search/ejabberd, retry 1/5  
> > > [Thu, 08 Dec 2011 04:51:10 +0000] ERROR: Connection refused connecting  
> > > to  
> > > localhost:4000 for /search/ejabberd, retry 2/5  
> > > [Thu, 08 Dec 2011 04:51:15 +0000] ERROR: Connection refused connecting  
> > > to  
> > > localhost:4000 for /search/ejabberd, retry 3/5  
> > > [Thu, 08 Dec 2011 04:51:20 +0000] ERROR: Connection refused connecting  
> > > to  
> > > localhost:4000 for /search/ejabberd, retry 4/5  
> > > [Thu, 08 Dec 2011 04:51:25 +0000] ERROR: Connection refused connecting  
> > > to  
> > > localhost:4000 for /search/ejabberd, retry 5/5  
> > > Errno::ECONNREFUSED: Connection refused - Connection refused connecting  
> > > to  
> > > localhost:4000 for /search/ejabberd, giving up
> > > 
> > > but my /etc/chef/client.rb has the proper url:port for my chef server.  
> > > If I  
> > > run the same command from my local box I get back:
> > > 
> > > chef:recipe \> search(:ejabberd, "id:deployment\_000010182")  
> > > =\>
> > > 
> > > which is what I expect. Any ideas where I should be looking?
> > > 
> > > Thanks,
> > > 
> > > MG
> > 
> > --  
> > Cheers,
> > 
> > Peter Donald

---

<div class="post-metadata">

**Author:** ![Jesse\_Nelson](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/jesse_nelson/32/262_2.png) [@Jesse\_Nelson](https://discourse.chef.io/u/Jesse_Nelson)\
**Post date:** [December 9, 2011, 8:36pm UTC](https://discourse.chef.io/t/problem-accessing-databag-from-node/2314/5 "2011-12-09T20:36:34Z")

</div>

One simple case is a node writing a lock for other nodes to block on. This would be for cluster buildup. Or other orchestration where you might need tighter timing than what search buys you. This already exists in dominodes cook: [GitHub - websterclay/chef-dominodes: A Chef resource for mutual exclusion of blocks of recipe code. Useful for cross-cluster rolling restarts.](https://github.com/websterclay/chef-dominodes)

Another case might be simple info exposure for other services. Again assuming you don't have, need or want to use search for whatever reason.

This ability already exists on platform just not on OSS server.

On Dec 9, 2011, at 9:55 AM, Jay Feldblum [y\_feldblum@yahoo.com](mailto:y_feldblum@yahoo.com) wrote:

> What's a use-case for a node rewriting a data bag item during a converge, that can't be solved by some other method?
> 
> Note that it would have to scale to 100,000 nodes talking to the same server and converging every 5 minutes, just as it would have to scale to 2 nodes talking to the same server and converging only when you SSH in to run the chef-client.
> 
> - Jay Feldblum
> 
> On Fri, Dec 9, 2011 at 12:48 PM, Michael Glenney [mike.glenney@gmail.com](mailto:mike.glenney@gmail.com) wrote:  
> I figured out the issue. First of all, the 403 was because I was trying to write to a databag within a recipe and didn't read the disclaimer in the wiki that said I'd have to give the node api client admin privileges to be able to do that. Of course we don't want to do that so I'll have to come up with another way to solve the problem.
> 
> BTW, I wouldn't mind seeing chef have the capability to give a nodes permission to write to a particular databag without giving the node admin rights in case anyone is accepting Christmas wishes.
> 
> For the shef localhost:4000 error, that was just because, when running shef from a client node, I forgot I need to launch shef with 'shef -c /etc/chef/client.rb'
> 
> MG
> 
> On Wed, Dec 7, 2011 at 10:09 PM, Peter Donald [peter@realityforge.org](mailto:peter@realityforge.org) wrote:  
> Hi,
> 
> We had this exception when the chef-solr service died on the chef  
> server. Figure out what killed itc and restart was our approach.
> 
> - This was a result of some process updating the owner/permissions on  
> /var/log/chef and /var/run/chef so that solr failed during startup.
> 
> On Thu, Dec 8, 2011 at 4:02 PM, Michael Glenney [mike.glenney@gmail.com](mailto:mike.glenney@gmail.com) wrote:
> 
> > I'm having problems with a 403 error when trying a new cookbook and I think  
> > I've tracked it down to databag access.
> > 
> > Chef Server 10.0  
> > chef-client 10.4
> > 
> > The first several lines of the stacktrace:
> > 
> > Generated at 2011-12-08 04:12:07 +0000  
> > Net::HTTPServerException: 403 "Forbidden"  
> > /usr/lib/ruby/1.9.1/net/http.rb:2303:in `error!' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:237:in `block in  
> > api\_request'  
> > /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:288:in  
> > `retriable_rest_request' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:218:in `api\_request'  
> > /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:130:in `put_rest' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/data_bag_item.rb:227:in `save'  
> > /var/chef/cache/cookbooks/ejabberd/recipes/default.rb:45:in `from_file' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/mixin/from_file.rb:30:in `instance\_eval'  
> > /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/mixin/from\_file.rb:30:in  
> > `from_file' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/cookbook_version.rb:578:in `load\_recipe'
> > 
> > The relevant part of that recipe is:
> > 
> > # Find cluster master, or create item and become master
> > 
> > if search(:ejabberd, "id:#{deployid}").count == 0  
> > masternode = "#{node[:ipaddress]}"  
> > h = {}  
> > h[deployid] = {"id" =\> deployid, "master" =\> masternode, "members" =\>  
> > [masternode]}
> > 
> > # Create new data bag item for cluster
> > 
> > databag\_item = Chef::DataBagItem.new  
> > databag\_item.data\_bag("ejabberd")  
> > databag\_item.raw\_data = h[deployid]  
> > databag\_item.save  
> > else
> > 
> > This is for a first chef run on a new node. If I run shef, switch to recipe  
> > context, and run 'search(:ejabberd, "id:deployment\_000010182")' I get back:
> > 
> > chef:recipe \> search(:ejabberd, "id:deployment\_000010182")  
> > [Thu, 08 Dec 2011 04:51:05 +0000] ERROR: Connection refused connecting to  
> > localhost:4000 for /search/ejabberd, retry 1/5  
> > [Thu, 08 Dec 2011 04:51:10 +0000] ERROR: Connection refused connecting to  
> > localhost:4000 for /search/ejabberd, retry 2/5  
> > [Thu, 08 Dec 2011 04:51:15 +0000] ERROR: Connection refused connecting to  
> > localhost:4000 for /search/ejabberd, retry 3/5  
> > [Thu, 08 Dec 2011 04:51:20 +0000] ERROR: Connection refused connecting to  
> > localhost:4000 for /search/ejabberd, retry 4/5  
> > [Thu, 08 Dec 2011 04:51:25 +0000] ERROR: Connection refused connecting to  
> > localhost:4000 for /search/ejabberd, retry 5/5  
> > Errno::ECONNREFUSED: Connection refused - Connection refused connecting to  
> > localhost:4000 for /search/ejabberd, giving up
> > 
> > but my /etc/chef/client.rb has the proper url:port for my chef server. If I  
> > run the same command from my local box I get back:
> > 
> > chef:recipe \> search(:ejabberd, "id:deployment\_000010182")  
> > =\>
> > 
> > which is what I expect. Any ideas where I should be looking?
> > 
> > Thanks,
> > 
> > MG
> 
> --  
> Cheers,
> 
> Peter Donald

---

<div class="post-metadata">

**Author:** ![Jay\_Feldblum](https://avatars.discourse-cdn.com/v4/letter/j/9fc29f/32.png) [@Jay\_Feldblum](https://discourse.chef.io/u/Jay_Feldblum)\
**Post date:** [December 9, 2011, 8:53pm UTC](https://discourse.chef.io/t/problem-accessing-databag-from-node/2314/6 "2011-12-09T20:53:34Z")

</div>

Cross-node synchronization of chef-client runs or of specific resources in  
chef-client runs would be nice.

But do data bags provide a primitive sufficient for synchronization? I  
doubt it. Perhaps such a primitive might be a useful feature for a future  
version of chef-server-api.

Some people seem to be using tools like zookeeper or noah, which are built  
for this type of job.

- Jay

On Fri, Dec 9, 2011 at 3:36 PM, Jesse Nelson [spheromak@gmail.com](mailto:spheromak@gmail.com) wrote:

> One simple case is a node writing a lock for other nodes to block on. This  
> would be for cluster buildup. Or other orchestration where you might need  
> tighter timing than what search buys you. This already exists in  
> dominodes cook: [GitHub - websterclay/chef-dominodes: A Chef resource for mutual exclusion of blocks of recipe code. Useful for cross-cluster rolling restarts.](https://github.com/websterclay/chef-dominodes)
> 
> Another case might be simple info exposure for other services. Again  
> assuming you don't have, need or want to use search for whatever reason.
> 
> This ability already exists on platform just not on OSS server.
> 
> On Dec 9, 2011, at 9:55 AM, Jay Feldblum [y\_feldblum@yahoo.com](mailto:y_feldblum@yahoo.com) wrote:
> 
> What's a use-case for a node rewriting a data bag item during a converge,  
> that can't be solved by some other method?
> 
> Note that it would have to scale to 100,000 nodes talking to the same  
> server and converging every 5 minutes, just as it would have to scale to 2  
> nodes talking to the same server and converging only when you SSH in to run  
> the chef-client.
> 
> - Jay Feldblum
> 
> On Fri, Dec 9, 2011 at 12:48 PM, Michael Glenney [mike.glenney@gmail.com](mailto:mike.glenney@gmail.com)wrote:
> 
> > I figured out the issue. First of all, the 403 was because I was trying  
> > to write to a databag within a recipe and didn't read the disclaimer in the  
> > wiki that said I'd have to give the node api client admin privileges to be  
> > able to do that. Of course we don't want to do that so I'll have to come  
> > up with another way to solve the problem.
> > 
> > BTW, I wouldn't mind seeing chef have the capability to give a nodes  
> > permission to write to a particular databag without giving the node admin  
> > rights in case anyone is accepting Christmas wishes.
> > 
> > For the shef localhost:4000 error, that was just because, when running  
> > shef from a client node, I forgot I need to launch shef with 'shef -c  
> > /etc/chef/client.rb'
> > 
> > MG
> > 
> > On Wed, Dec 7, 2011 at 10:09 PM, Peter Donald [peter@realityforge.org](mailto:peter@realityforge.org)wrote:
> > 
> > > Hi,
> > > 
> > > We had this exception when the chef-solr service died on the chef  
> > > server. Figure out what killed itc and restart was our approach.
> > > 
> > > - This was a result of some process updating the owner/permissions on  
> > > /var/log/chef and /var/run/chef so that solr failed during startup.
> > > 
> > > On Thu, Dec 8, 2011 at 4:02 PM, Michael Glenney [mike.glenney@gmail.com](mailto:mike.glenney@gmail.com)  
> > > wrote:
> > > 
> > > > I'm having problems with a 403 error when trying a new cookbook and I  
> > > > think  
> > > > I've tracked it down to databag access.
> > > > 
> > > > Chef Server 10.0  
> > > > chef-client 10.4
> > > > 
> > > > The first several lines of the stacktrace:
> > > > 
> > > > Generated at 2011-12-08 04:12:07 +0000  
> > > > Net::HTTPServerException: 403 "Forbidden"  
> > > > /usr/lib/ruby/1.9.1/net/http.rb:2303:in `error!' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:237:in `block in  
> > > > api\_request'  
> > > > /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:288:in  
> > > > `retriable_rest_request' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:218:in `api\_request'  
> > > > /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:130:in  
> > > > `put\_rest'
> > > 
> > > /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/data\_bag\_item.rb:227:in
> > > 
> > > > `save' /var/chef/cache/cookbooks/ejabberd/recipes/default.rb:45:in `from\_file'
> > > 
> > > /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/mixin/from\_file.rb:30:in
> > > 
> > > > `instance\_eval'
> > > 
> > > /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/mixin/from\_file.rb:30:in
> > > 
> > > > `from\_file'
> > > 
> > > /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/cookbook\_version.rb:578:in
> > > 
> > > > `load\_recipe'
> > > > 
> > > > The relevant part of that recipe is:
> > > > 
> > > > # Find cluster master, or create item and become master
> > > > 
> > > > if search(:ejabberd, "id:#{deployid}").count == 0  
> > > > masternode = "#{node[:ipaddress]}"  
> > > > h = {}  
> > > > h[deployid] = {"id" =\> deployid, "master" =\> masternode, "members" =\>  
> > > > [masternode]}
> > > > 
> > > > # Create new data bag item for cluster
> > > > 
> > > > databag\_item = Chef::DataBagItem.new  
> > > > databag\_item.data\_bag("ejabberd")  
> > > > databag\_item.raw\_data = h[deployid]  
> > > > databag\_item.save  
> > > > else
> > > > 
> > > > This is for a first chef run on a new node. If I run shef, switch to  
> > > > recipe  
> > > > context, and run 'search(:ejabberd, "id:deployment\_000010182")' I get  
> > > > back:
> > > > 
> > > > chef:recipe \> search(:ejabberd, "id:deployment\_000010182")  
> > > > [Thu, 08 Dec 2011 04:51:05 +0000] ERROR: Connection refused connecting  
> > > > to  
> > > > localhost:4000 for /search/ejabberd, retry 1/5  
> > > > [Thu, 08 Dec 2011 04:51:10 +0000] ERROR: Connection refused connecting  
> > > > to  
> > > > localhost:4000 for /search/ejabberd, retry 2/5  
> > > > [Thu, 08 Dec 2011 04:51:15 +0000] ERROR: Connection refused connecting  
> > > > to  
> > > > localhost:4000 for /search/ejabberd, retry 3/5  
> > > > [Thu, 08 Dec 2011 04:51:20 +0000] ERROR: Connection refused connecting  
> > > > to  
> > > > localhost:4000 for /search/ejabberd, retry 4/5  
> > > > [Thu, 08 Dec 2011 04:51:25 +0000] ERROR: Connection refused connecting  
> > > > to  
> > > > localhost:4000 for /search/ejabberd, retry 5/5  
> > > > Errno::ECONNREFUSED: Connection refused - Connection refused  
> > > > connecting to  
> > > > localhost:4000 for /search/ejabberd, giving up
> > > > 
> > > > but my /etc/chef/client.rb has the proper url:port for my chef  
> > > > server. If I  
> > > > run the same command from my local box I get back:
> > > > 
> > > > chef:recipe \> search(:ejabberd, "id:deployment\_000010182")  
> > > > =\>
> > > > 
> > > > which is what I expect. Any ideas where I should be looking?
> > > > 
> > > > Thanks,
> > > > 
> > > > MG
> > > 
> > > --  
> > > Cheers,
> > > 
> > > Peter Donald

---

<div class="post-metadata">

**Author:** ![Michael\_Glenney](https://avatars.discourse-cdn.com/v4/letter/m/a9adbd/32.png) [@Michael\_Glenney](https://discourse.chef.io/u/Michael_Glenney)\
**Post date:** [December 9, 2011, 11:39pm UTC](https://discourse.chef.io/t/problem-accessing-databag-from-node/2314/7 "2011-12-09T23:39:47Z")

</div>

The use case here is clustering. The first node that comes up would  
search the databage for an item named the deployment id. If it doesn't  
find an item with that name it would create the item and within that item  
state that it is the master. The next node to come up with that deployment  
id would find that item and extract the master node info.

A coworker suggested I use chef's search capabilities instead. Search chef  
for a node with that deployment id. If not found set an attribute claiming  
itself as the master. Subsequent nodes would find that with search and  
would extract what they need from the search results. This is the method  
I'm going to use now but I initially rejected his idea (and am not happy  
that were having to go this route) because I have to rely on solr properly  
indexing my node before the next node comes up. I don't like having to  
rely on timing when it comes to automation. It's sloppy in my eyes. With  
the databag solution, I "know" the data is there because my call to write  
it is successful.

A 3rd solution I came up with is to use Amazon SQS but I'd be happier with  
a pure chef solution.

MG

On Fri, Dec 9, 2011 at 10:55 AM, Jay Feldblum [y\_feldblum@yahoo.com](mailto:y_feldblum@yahoo.com) wrote:

> What's a use-case for a node rewriting a data bag item during a converge,  
> that can't be solved by some other method?
> 
> Note that it would have to scale to 100,000 nodes talking to the same  
> server and converging every 5 minutes, just as it would have to scale to 2  
> nodes talking to the same server and converging only when you SSH in to run  
> the chef-client.
> 
> - Jay Feldblum
> 
> On Fri, Dec 9, 2011 at 12:48 PM, Michael Glenney [mike.glenney@gmail.com](mailto:mike.glenney@gmail.com)wrote:
> 
> > I figured out the issue. First of all, the 403 was because I was trying  
> > to write to a databag within a recipe and didn't read the disclaimer in the  
> > wiki that said I'd have to give the node api client admin privileges to be  
> > able to do that. Of course we don't want to do that so I'll have to come  
> > up with another way to solve the problem.
> > 
> > BTW, I wouldn't mind seeing chef have the capability to give a nodes  
> > permission to write to a particular databag without giving the node admin  
> > rights in case anyone is accepting Christmas wishes.
> > 
> > For the shef localhost:4000 error, that was just because, when running  
> > shef from a client node, I forgot I need to launch shef with 'shef -c  
> > /etc/chef/client.rb'
> > 
> > MG
> > 
> > On Wed, Dec 7, 2011 at 10:09 PM, Peter Donald [peter@realityforge.org](mailto:peter@realityforge.org)wrote:
> > 
> > > Hi,
> > > 
> > > We had this exception when the chef-solr service died on the chef  
> > > server. Figure out what killed itc and restart was our approach.
> > > 
> > > - This was a result of some process updating the owner/permissions on  
> > > /var/log/chef and /var/run/chef so that solr failed during startup.
> > > 
> > > On Thu, Dec 8, 2011 at 4:02 PM, Michael Glenney [mike.glenney@gmail.com](mailto:mike.glenney@gmail.com)  
> > > wrote:
> > > 
> > > > I'm having problems with a 403 error when trying a new cookbook and I  
> > > > think  
> > > > I've tracked it down to databag access.
> > > > 
> > > > Chef Server 10.0  
> > > > chef-client 10.4
> > > > 
> > > > The first several lines of the stacktrace:
> > > > 
> > > > Generated at 2011-12-08 04:12:07 +0000  
> > > > Net::HTTPServerException: 403 "Forbidden"  
> > > > /usr/lib/ruby/1.9.1/net/http.rb:2303:in `error!' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:237:in `block in  
> > > > api\_request'  
> > > > /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:288:in  
> > > > `retriable_rest_request' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:218:in `api\_request'  
> > > > /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:130:in  
> > > > `put\_rest'
> > > 
> > > /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/data\_bag\_item.rb:227:in
> > > 
> > > > `save' /var/chef/cache/cookbooks/ejabberd/recipes/default.rb:45:in `from\_file'
> > > 
> > > /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/mixin/from\_file.rb:30:in
> > > 
> > > > `instance\_eval'
> > > 
> > > /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/mixin/from\_file.rb:30:in
> > > 
> > > > `from\_file'
> > > 
> > > /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/cookbook\_version.rb:578:in
> > > 
> > > > `load\_recipe'
> > > > 
> > > > The relevant part of that recipe is:
> > > > 
> > > > # Find cluster master, or create item and become master
> > > > 
> > > > if search(:ejabberd, "id:#{deployid}").count == 0  
> > > > masternode = "#{node[:ipaddress]}"  
> > > > h = {}  
> > > > h[deployid] = {"id" =\> deployid, "master" =\> masternode, "members" =\>  
> > > > [masternode]}
> > > > 
> > > > # Create new data bag item for cluster
> > > > 
> > > > databag\_item = Chef::DataBagItem.new  
> > > > databag\_item.data\_bag("ejabberd")  
> > > > databag\_item.raw\_data = h[deployid]  
> > > > databag\_item.save  
> > > > else
> > > > 
> > > > This is for a first chef run on a new node. If I run shef, switch to  
> > > > recipe  
> > > > context, and run 'search(:ejabberd, "id:deployment\_000010182")' I get  
> > > > back:
> > > > 
> > > > chef:recipe \> search(:ejabberd, "id:deployment\_000010182")  
> > > > [Thu, 08 Dec 2011 04:51:05 +0000] ERROR: Connection refused connecting  
> > > > to  
> > > > localhost:4000 for /search/ejabberd, retry 1/5  
> > > > [Thu, 08 Dec 2011 04:51:10 +0000] ERROR: Connection refused connecting  
> > > > to  
> > > > localhost:4000 for /search/ejabberd, retry 2/5  
> > > > [Thu, 08 Dec 2011 04:51:15 +0000] ERROR: Connection refused connecting  
> > > > to  
> > > > localhost:4000 for /search/ejabberd, retry 3/5  
> > > > [Thu, 08 Dec 2011 04:51:20 +0000] ERROR: Connection refused connecting  
> > > > to  
> > > > localhost:4000 for /search/ejabberd, retry 4/5  
> > > > [Thu, 08 Dec 2011 04:51:25 +0000] ERROR: Connection refused connecting  
> > > > to  
> > > > localhost:4000 for /search/ejabberd, retry 5/5  
> > > > Errno::ECONNREFUSED: Connection refused - Connection refused  
> > > > connecting to  
> > > > localhost:4000 for /search/ejabberd, giving up
> > > > 
> > > > but my /etc/chef/client.rb has the proper url:port for my chef  
> > > > server. If I  
> > > > run the same command from my local box I get back:
> > > > 
> > > > chef:recipe \> search(:ejabberd, "id:deployment\_000010182")  
> > > > =\>
> > > > 
> > > > which is what I expect. Any ideas where I should be looking?
> > > > 
> > > > Thanks,
> > > > 
> > > > MG
> > > 
> > > --  
> > > Cheers,
> > > 
> > > Peter Donald

---

<div class="post-metadata">

**Author:** ![Jay\_Feldblum](https://avatars.discourse-cdn.com/v4/letter/j/9fc29f/32.png) [@Jay\_Feldblum](https://discourse.chef.io/u/Jay_Feldblum)\
**Post date:** [December 10, 2011, 12:18am UTC](https://discourse.chef.io/t/problem-accessing-databag-from-node/2314/8 "2011-12-10T00:18:28Z")

</div>

Chef is a distributed system. Using search would be a good approach, so  
long as your code takes that into account. If you go that route, you don't  
rely on timing; rather, you code for the fact that timing may be "off." I  
don't think that's sloppy - I think it's part of what it means for nodes to  
"converge" over time, over multiple runs, to a final "finished" state.

For example, to bring up a cluster, here's what you can do. Add a  
normal-level attribute on a first node indicating it is the seed node  
(don't add that attribute to any other node). Then during a convergence,  
each node will only create the cluster if it is the seed node, and will  
only add itself to the cluster if some other node is the seed node (found  
by search) and that node is reachable and healthy. The seed node might not  
be the primary node coordinating the cluster, but any node should be able  
to find the primary node by asking the seed node what the primary is. If  
the original seed node fails at a later point and needs to be removed from  
the cluster, just delete that node object from the chef-server and make a  
different node the new seed node (by setting that specific normal-level  
attribute) before bringing up any new nodes into the cluster. Note that  
normal-level attributes may be set via knife or via the webui, and that  
they persist across chef-client runs. You can use the node tags for the  
purpose of the seed attribute, since node tags are normal-level attributes  
and are easily edited via the CLI (`knife tag create NODE TAG`) without  
requiring an editor. There is certainly room for extension and variation -  
for example, to eliminate the SPOF where there can only be one seed node.

- Jay Feldblum

On Fri, Dec 9, 2011 at 6:39 PM, Michael Glenney [mike.glenney@gmail.com](mailto:mike.glenney@gmail.com)wrote:

> The use case here is clustering. The first node that comes up would  
> search the databage for an item named the deployment id. If it doesn't  
> find an item with that name it would create the item and within that item  
> state that it is the master. The next node to come up with that deployment  
> id would find that item and extract the master node info.
> 
> A coworker suggested I use chef's search capabilities instead. Search  
> chef for a node with that deployment id. If not found set an attribute  
> claiming itself as the master. Subsequent nodes would find that with  
> search and would extract what they need from the search results. This is  
> the method I'm going to use now but I initially rejected his idea (and am  
> not happy that were having to go this route) because I have to rely on solr  
> properly indexing my node before the next node comes up. I don't like  
> having to rely on timing when it comes to automation. It's sloppy in my  
> eyes. With the databag solution, I "know" the data is there because my  
> call to write it is successful.
> 
> A 3rd solution I came up with is to use Amazon SQS but I'd be happier with  
> a pure chef solution.
> 
> MG
> 
> On Fri, Dec 9, 2011 at 10:55 AM, Jay Feldblum [y\_feldblum@yahoo.com](mailto:y_feldblum@yahoo.com)wrote:
> 
> > What's a use-case for a node rewriting a data bag item during a converge,  
> > that can't be solved by some other method?
> > 
> > Note that it would have to scale to 100,000 nodes talking to the same  
> > server and converging every 5 minutes, just as it would have to scale to 2  
> > nodes talking to the same server and converging only when you SSH in to run  
> > the chef-client.
> > 
> > - Jay Feldblum
> > 
> > On Fri, Dec 9, 2011 at 12:48 PM, Michael Glenney [mike.glenney@gmail.com](mailto:mike.glenney@gmail.com)wrote:
> > 
> > > I figured out the issue. First of all, the 403 was because I was trying  
> > > to write to a databag within a recipe and didn't read the disclaimer in the  
> > > wiki that said I'd have to give the node api client admin privileges to be  
> > > able to do that. Of course we don't want to do that so I'll have to come  
> > > up with another way to solve the problem.
> > > 
> > > BTW, I wouldn't mind seeing chef have the capability to give a nodes  
> > > permission to write to a particular databag without giving the node admin  
> > > rights in case anyone is accepting Christmas wishes.
> > > 
> > > For the shef localhost:4000 error, that was just because, when running  
> > > shef from a client node, I forgot I need to launch shef with 'shef -c  
> > > /etc/chef/client.rb'
> > > 
> > > MG
> > > 
> > > On Wed, Dec 7, 2011 at 10:09 PM, Peter Donald [peter@realityforge.org](mailto:peter@realityforge.org)wrote:
> > > 
> > > > Hi,
> > > > 
> > > > We had this exception when the chef-solr service died on the chef  
> > > > server. Figure out what killed itc and restart was our approach.
> > > > 
> > > > - This was a result of some process updating the owner/permissions on  
> > > > /var/log/chef and /var/run/chef so that solr failed during startup.
> > > > 
> > > > On Thu, Dec 8, 2011 at 4:02 PM, Michael Glenney [mike.glenney@gmail.com](mailto:mike.glenney@gmail.com)  
> > > > wrote:
> > > > 
> > > > > I'm having problems with a 403 error when trying a new cookbook and I  
> > > > > think  
> > > > > I've tracked it down to databag access.
> > > > > 
> > > > > Chef Server 10.0  
> > > > > chef-client 10.4
> > > > > 
> > > > > The first several lines of the stacktrace:
> > > > > 
> > > > > Generated at 2011-12-08 04:12:07 +0000  
> > > > > Net::HTTPServerException: 403 "Forbidden"  
> > > > > /usr/lib/ruby/1.9.1/net/http.rb:2303:in `error!' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:237:in `block in  
> > > > > api\_request'  
> > > > > /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:288:in  
> > > > > `retriable_rest_request' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:218:in `api\_request'  
> > > > > /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:130:in  
> > > > > `put\_rest'
> > > > 
> > > > /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/data\_bag\_item.rb:227:in
> > > > 
> > > > > `save' /var/chef/cache/cookbooks/ejabberd/recipes/default.rb:45:in `from\_file'
> > > > 
> > > > /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/mixin/from\_file.rb:30:in
> > > > 
> > > > > `instance\_eval'
> > > > 
> > > > /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/mixin/from\_file.rb:30:in
> > > > 
> > > > > `from\_file'
> > > > 
> > > > /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/cookbook\_version.rb:578:in
> > > > 
> > > > > `load\_recipe'
> > > > > 
> > > > > The relevant part of that recipe is:
> > > > > 
> > > > > # Find cluster master, or create item and become master
> > > > > 
> > > > > if search(:ejabberd, "id:#{deployid}").count == 0  
> > > > > masternode = "#{node[:ipaddress]}"  
> > > > > h = {}  
> > > > > h[deployid] = {"id" =\> deployid, "master" =\> masternode, "members"  
> > > > > =\>  
> > > > > [masternode]}
> > > > > 
> > > > > # Create new data bag item for cluster
> > > > > 
> > > > > databag\_item = Chef::DataBagItem.new  
> > > > > databag\_item.data\_bag("ejabberd")  
> > > > > databag\_item.raw\_data = h[deployid]  
> > > > > databag\_item.save  
> > > > > else
> > > > > 
> > > > > This is for a first chef run on a new node. If I run shef, switch to  
> > > > > recipe  
> > > > > context, and run 'search(:ejabberd, "id:deployment\_000010182")' I get  
> > > > > back:
> > > > > 
> > > > > chef:recipe \> search(:ejabberd, "id:deployment\_000010182")  
> > > > > [Thu, 08 Dec 2011 04:51:05 +0000] ERROR: Connection refused  
> > > > > connecting to  
> > > > > localhost:4000 for /search/ejabberd, retry 1/5  
> > > > > [Thu, 08 Dec 2011 04:51:10 +0000] ERROR: Connection refused  
> > > > > connecting to  
> > > > > localhost:4000 for /search/ejabberd, retry 2/5  
> > > > > [Thu, 08 Dec 2011 04:51:15 +0000] ERROR: Connection refused  
> > > > > connecting to  
> > > > > localhost:4000 for /search/ejabberd, retry 3/5  
> > > > > [Thu, 08 Dec 2011 04:51:20 +0000] ERROR: Connection refused  
> > > > > connecting to  
> > > > > localhost:4000 for /search/ejabberd, retry 4/5  
> > > > > [Thu, 08 Dec 2011 04:51:25 +0000] ERROR: Connection refused  
> > > > > connecting to  
> > > > > localhost:4000 for /search/ejabberd, retry 5/5  
> > > > > Errno::ECONNREFUSED: Connection refused - Connection refused  
> > > > > connecting to  
> > > > > localhost:4000 for /search/ejabberd, giving up
> > > > > 
> > > > > but my /etc/chef/client.rb has the proper url:port for my chef  
> > > > > server. If I  
> > > > > run the same command from my local box I get back:
> > > > > 
> > > > > chef:recipe \> search(:ejabberd, "id:deployment\_000010182")  
> > > > > =\>
> > > > > 
> > > > > which is what I expect. Any ideas where I should be looking?
> > > > > 
> > > > > Thanks,
> > > > > 
> > > > > MG
> > > > 
> > > > --  
> > > > Cheers,
> > > > 
> > > > Peter Donald

---

<div class="post-metadata">

**Author:** ![Jesse\_Nelson](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/jesse_nelson/32/262_2.png) [@Jesse\_Nelson](https://discourse.chef.io/u/Jesse_Nelson)\
**Post date:** [December 10, 2011, 4:23am UTC](https://discourse.chef.io/t/problem-accessing-databag-from-node/2314/9 "2011-12-10T04:23:40Z")

</div>

Biggest issue I've had with search is that a fail state node will not update its node attributes. Causing search to not reflect reality. I like the idea of an infra wide mutex.

Also for cluster master election you might want to look at fujin's pylon. Which intends to solve that exact problem. [GitHub - fujin/pylon: coordination helpers for distributed celluloid for use with systems automation frameworks e.g. Chef](https://github.com/fujin/pylon)

On Dec 9, 2011, at 3:39 PM, Michael Glenney [mike.glenney@gmail.com](mailto:mike.glenney@gmail.com) wrote:

> The use case here is clustering. The first node that comes up would search the databage for an item named the deployment id. If it doesn't find an item with that name it would create the item and within that item state that it is the master. The next node to come up with that deployment id would find that item and extract the master node info.
> 
> A coworker suggested I use chef's search capabilities instead. Search chef for a node with that deployment id. If not found set an attribute claiming itself as the master. Subsequent nodes would find that with search and would extract what they need from the search results. This is the method I'm going to use now but I initially rejected his idea (and am not happy that were having to go this route) because I have to rely on solr properly indexing my node before the next node comes up. I don't like having to rely on timing when it comes to automation. It's sloppy in my eyes. With the databag solution, I "know" the data is there because my call to write it is successful.
> 
> A 3rd solution I came up with is to use Amazon SQS but I'd be happier with a pure chef solution.
> 
> MG
> 
> On Fri, Dec 9, 2011 at 10:55 AM, Jay Feldblum [y\_feldblum@yahoo.com](mailto:y_feldblum@yahoo.com) wrote:  
> What's a use-case for a node rewriting a data bag item during a converge, that can't be solved by some other method?
> 
> Note that it would have to scale to 100,000 nodes talking to the same server and converging every 5 minutes, just as it would have to scale to 2 nodes talking to the same server and converging only when you SSH in to run the chef-client.
> 
> - Jay Feldblum
> 
> On Fri, Dec 9, 2011 at 12:48 PM, Michael Glenney [mike.glenney@gmail.com](mailto:mike.glenney@gmail.com) wrote:  
> I figured out the issue. First of all, the 403 was because I was trying to write to a databag within a recipe and didn't read the disclaimer in the wiki that said I'd have to give the node api client admin privileges to be able to do that. Of course we don't want to do that so I'll have to come up with another way to solve the problem.
> 
> BTW, I wouldn't mind seeing chef have the capability to give a nodes permission to write to a particular databag without giving the node admin rights in case anyone is accepting Christmas wishes.
> 
> For the shef localhost:4000 error, that was just because, when running shef from a client node, I forgot I need to launch shef with 'shef -c /etc/chef/client.rb'
> 
> MG
> 
> On Wed, Dec 7, 2011 at 10:09 PM, Peter Donald [peter@realityforge.org](mailto:peter@realityforge.org) wrote:  
> Hi,
> 
> We had this exception when the chef-solr service died on the chef  
> server. Figure out what killed itc and restart was our approach.
> 
> - This was a result of some process updating the owner/permissions on  
> /var/log/chef and /var/run/chef so that solr failed during startup.
> 
> On Thu, Dec 8, 2011 at 4:02 PM, Michael Glenney [mike.glenney@gmail.com](mailto:mike.glenney@gmail.com) wrote:
> 
> > I'm having problems with a 403 error when trying a new cookbook and I think  
> > I've tracked it down to databag access.
> > 
> > Chef Server 10.0  
> > chef-client 10.4
> > 
> > The first several lines of the stacktrace:
> > 
> > Generated at 2011-12-08 04:12:07 +0000  
> > Net::HTTPServerException: 403 "Forbidden"  
> > /usr/lib/ruby/1.9.1/net/http.rb:2303:in `error!' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:237:in `block in  
> > api\_request'  
> > /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:288:in  
> > `retriable_rest_request' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:218:in `api\_request'  
> > /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:130:in `put_rest' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/data_bag_item.rb:227:in `save'  
> > /var/chef/cache/cookbooks/ejabberd/recipes/default.rb:45:in `from_file' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/mixin/from_file.rb:30:in `instance\_eval'  
> > /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/mixin/from\_file.rb:30:in  
> > `from_file' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/cookbook_version.rb:578:in `load\_recipe'
> > 
> > The relevant part of that recipe is:
> > 
> > # Find cluster master, or create item and become master
> > 
> > if search(:ejabberd, "id:#{deployid}").count == 0  
> > masternode = "#{node[:ipaddress]}"  
> > h = {}  
> > h[deployid] = {"id" =\> deployid, "master" =\> masternode, "members" =\>  
> > [masternode]}
> > 
> > # Create new data bag item for cluster
> > 
> > databag\_item = Chef::DataBagItem.new  
> > databag\_item.data\_bag("ejabberd")  
> > databag\_item.raw\_data = h[deployid]  
> > databag\_item.save  
> > else
> > 
> > This is for a first chef run on a new node. If I run shef, switch to recipe  
> > context, and run 'search(:ejabberd, "id:deployment\_000010182")' I get back:
> > 
> > chef:recipe \> search(:ejabberd, "id:deployment\_000010182")  
> > [Thu, 08 Dec 2011 04:51:05 +0000] ERROR: Connection refused connecting to  
> > localhost:4000 for /search/ejabberd, retry 1/5  
> > [Thu, 08 Dec 2011 04:51:10 +0000] ERROR: Connection refused connecting to  
> > localhost:4000 for /search/ejabberd, retry 2/5  
> > [Thu, 08 Dec 2011 04:51:15 +0000] ERROR: Connection refused connecting to  
> > localhost:4000 for /search/ejabberd, retry 3/5  
> > [Thu, 08 Dec 2011 04:51:20 +0000] ERROR: Connection refused connecting to  
> > localhost:4000 for /search/ejabberd, retry 4/5  
> > [Thu, 08 Dec 2011 04:51:25 +0000] ERROR: Connection refused connecting to  
> > localhost:4000 for /search/ejabberd, retry 5/5  
> > Errno::ECONNREFUSED: Connection refused - Connection refused connecting to  
> > localhost:4000 for /search/ejabberd, giving up
> > 
> > but my /etc/chef/client.rb has the proper url:port for my chef server. If I  
> > run the same command from my local box I get back:
> > 
> > chef:recipe \> search(:ejabberd, "id:deployment\_000010182")  
> > =\>
> > 
> > which is what I expect. Any ideas where I should be looking?
> > 
> > Thanks,
> > 
> > MG
> 
> --  
> Cheers,
> 
> Peter Donald

---

<div class="post-metadata">

**Author:** ![AJ\_Christensen](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/aj_christensen/32/525_2.png) [@AJ\_Christensen](https://discourse.chef.io/u/AJ_Christensen)\
**Post date:** [December 10, 2011, 6:03am UTC](https://discourse.chef.io/t/problem-accessing-databag-from-node/2314/10 "2011-12-10T06:03:34Z")

</div>

Pylon is probably a few works of solid code away from being usable,  
but it does perform a master election successfully over multicast.  
Getting it integrated with Chef is on my list of things to do..

If you use search for this kind of stuff, it's pretty easy to throw  
out bad nodes. I do a few searches for roles:foo, role:foo, then fall  
back to node.run\_list.include? foo..

Adam has been talking about some locking primitives for Chef that  
would probably make this all a little easier and hopefully without a  
central server or anything.

--AJ

On 10 December 2011 17:23, Jesse Nelson [spheromak@gmail.com](mailto:spheromak@gmail.com) wrote:

> Biggest issue I've had with search is that a fail state node will not update  
> its node attributes. Causing search to not reflect reality. I like the idea  
> of an infra wide mutex.
> 
> Also for cluster master election you might want to look at fujin's pylon.  
> Which intends to solve that exact problem. [GitHub - fujin/pylon: coordination helpers for distributed celluloid for use with systems automation frameworks e.g. Chef](https://github.com/fujin/pylon)
> 
> On Dec 9, 2011, at 3:39 PM, Michael Glenney [mike.glenney@gmail.com](mailto:mike.glenney@gmail.com) wrote:
> 
> The use case here is clustering. The first node that comes up would search  
> the databage for an item named the deployment id. If it doesn't find an  
> item with that name it would create the item and within that item state that  
> it is the master. The next node to come up with that deployment id would  
> find that item and extract the master node info.
> 
> A coworker suggested I use chef's search capabilities instead. Search chef  
> for a node with that deployment id. If not found set an attribute claiming  
> itself as the master. Subsequent nodes would find that with search and  
> would extract what they need from the search results. This is the method  
> I'm going to use now but I initially rejected his idea (and am not happy  
> that were having to go this route) because I have to rely on solr properly  
> indexing my node before the next node comes up. I don't like having to rely  
> on timing when it comes to automation. It's sloppy in my eyes. With the  
> databag solution, I "know" the data is there because my call to write it is  
> successful.
> 
> A 3rd solution I came up with is to use Amazon SQS but I'd be happier with a  
> pure chef solution.
> 
> MG
> 
> On Fri, Dec 9, 2011 at 10:55 AM, Jay Feldblum [y\_feldblum@yahoo.com](mailto:y_feldblum@yahoo.com) wrote:
> 
> > What's a use-case for a node rewriting a data bag item during a converge,  
> > that can't be solved by some other method?
> > 
> > Note that it would have to scale to 100,000 nodes talking to the same  
> > server and converging every 5 minutes, just as it would have to scale to 2  
> > nodes talking to the same server and converging only when you SSH in to run  
> > the chef-client.
> > 
> > - Jay Feldblum
> > 
> > On Fri, Dec 9, 2011 at 12:48 PM, Michael Glenney [mike.glenney@gmail.com](mailto:mike.glenney@gmail.com)  
> > wrote:
> > 
> > > I figured out the issue. First of all, the 403 was because I was trying  
> > > to write to a databag within a recipe and didn't read the disclaimer in the  
> > > wiki that said I'd have to give the node api client admin privileges to be  
> > > able to do that. Of course we don't want to do that so I'll have to come up  
> > > with another way to solve the problem.
> > > 
> > > BTW, I wouldn't mind seeing chef have the capability to give a nodes  
> > > permission to write to a particular databag without giving the node admin  
> > > rights in case anyone is accepting Christmas wishes.
> > > 
> > > For the shef localhost:4000 error, that was just because, when running  
> > > shef from a client node, I forgot I need to launch shef with 'shef -c  
> > > /etc/chef/client.rb'
> > > 
> > > MG
> > > 
> > > On Wed, Dec 7, 2011 at 10:09 PM, Peter Donald [peter@realityforge.org](mailto:peter@realityforge.org)  
> > > wrote:
> > > 
> > > > Hi,
> > > > 
> > > > We had this exception when the chef-solr service died on the chef  
> > > > server. Figure out what killed itc and restart was our approach.
> > > > 
> > > > - This was a result of some process updating the owner/permissions on  
> > > > /var/log/chef and /var/run/chef so that solr failed during startup.
> > > > 
> > > > On Thu, Dec 8, 2011 at 4:02 PM, Michael Glenney [mike.glenney@gmail.com](mailto:mike.glenney@gmail.com)  
> > > > wrote:
> > > > 
> > > > > I'm having problems with a 403 error when trying a new cookbook and I  
> > > > > think  
> > > > > I've tracked it down to databag access.
> > > > > 
> > > > > Chef Server 10.0  
> > > > > chef-client 10.4
> > > > > 
> > > > > The first several lines of the stacktrace:
> > > > > 
> > > > > Generated at 2011-12-08 04:12:07 +0000  
> > > > > Net::HTTPServerException: 403 "Forbidden"  
> > > > > /usr/lib/ruby/1.9.1/net/http.rb:2303:in `error!' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:237:in `block in  
> > > > > api\_request'  
> > > > > /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:288:in  
> > > > > `retriable_rest_request' /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:218:in `api\_request'  
> > > > > /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/rest.rb:130:in  
> > > > > `put\_rest'
> > > > > 
> > > > > /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/data\_bag\_item.rb:227:in  
> > > > > `save' /var/chef/cache/cookbooks/ejabberd/recipes/default.rb:45:in `from\_file'
> > > > > 
> > > > > /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/mixin/from\_file.rb:30:in  
> > > > > `instance\_eval'
> > > > > 
> > > > > /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/mixin/from\_file.rb:30:in  
> > > > > `from\_file'
> > > > > 
> > > > > /usr/lib/ruby/gems/1.9.1/gems/chef-0.10.4/lib/chef/cookbook\_version.rb:578:in  
> > > > > `load\_recipe'
> > > > > 
> > > > > The relevant part of that recipe is:
> > > > > 
> > > > > # Find cluster master, or create item and become master
> > > > > 
> > > > > if search(:ejabberd, "id:#{deployid}").count == 0  
> > > > > masternode = "#{node[:ipaddress]}"  
> > > > > h = {}  
> > > > > h[deployid] = {"id" =\> deployid, "master" =\> masternode, "members"  
> > > > > =\>  
> > > > > [masternode]}
> > > > > 
> > > > > # Create new data bag item for cluster
> > > > > 
> > > > > databag\_item = Chef::DataBagItem.new  
> > > > > databag\_item.data\_bag("ejabberd")  
> > > > > databag\_item.raw\_data = h[deployid]  
> > > > > databag\_item.save  
> > > > > else
> > > > > 
> > > > > This is for a first chef run on a new node. If I run shef, switch to  
> > > > > recipe  
> > > > > context, and run 'search(:ejabberd, "id:deployment\_000010182")' I get  
> > > > > back:
> > > > > 
> > > > > chef:recipe \> search(:ejabberd, "id:deployment\_000010182")  
> > > > > [Thu, 08 Dec 2011 04:51:05 +0000] ERROR: Connection refused connecting  
> > > > > to  
> > > > > localhost:4000 for /search/ejabberd, retry 1/5  
> > > > > [Thu, 08 Dec 2011 04:51:10 +0000] ERROR: Connection refused connecting  
> > > > > to  
> > > > > localhost:4000 for /search/ejabberd, retry 2/5  
> > > > > [Thu, 08 Dec 2011 04:51:15 +0000] ERROR: Connection refused connecting  
> > > > > to  
> > > > > localhost:4000 for /search/ejabberd, retry 3/5  
> > > > > [Thu, 08 Dec 2011 04:51:20 +0000] ERROR: Connection refused connecting  
> > > > > to  
> > > > > localhost:4000 for /search/ejabberd, retry 4/5  
> > > > > [Thu, 08 Dec 2011 04:51:25 +0000] ERROR: Connection refused connecting  
> > > > > to  
> > > > > localhost:4000 for /search/ejabberd, retry 5/5  
> > > > > Errno::ECONNREFUSED: Connection refused - Connection refused  
> > > > > connecting to  
> > > > > localhost:4000 for /search/ejabberd, giving up
> > > > > 
> > > > > but my /etc/chef/client.rb has the proper url:port for my chef  
> > > > > server. If I  
> > > > > run the same command from my local box I get back:
> > > > > 
> > > > > chef:recipe \> search(:ejabberd, "id:deployment\_000010182")  
> > > > > =\>
> > > > > 
> > > > > which is what I expect. Any ideas where I should be looking?
> > > > > 
> > > > > Thanks,
> > > > > 
> > > > > MG
> > > > 
> > > > --  
> > > > Cheers,
> > > > 
> > > > Peter Donald