# Effeciency of data bags?

**URL:** <https://discourse.chef.io/t/effeciency-of-data-bags/1649>\
**Category:** Chef Infra (archive)\
**Created:** [December 8, 2010, 7:40pm UTC](https://discourse.chef.io/t/effeciency-of-data-bags/1649 "2010-12-08T19:40:54Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mason\_Turner](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/mason_turner/32/817_2.png) [@Mason\_Turner](https://discourse.chef.io/u/Mason_Turner)\
**Post date:** [December 8, 2010, 7:40pm UTC](https://discourse.chef.io/t/effeciency-of-data-bags/1649/1 "2010-12-08T19:40:54Z")

</div>

I have a recipe that builds an rdist distfile by searching for nodes  
that match a particular attribute. These queries are taking about 3  
seconds, and i have less then 20 nodes in my chef DB. I’m sure there  
are some ways to optimize my chef server, but I am also considering  
pre-building the data in a databag. Does chef-client use a “last  
modified” field to prevent lookups from the databag if nothing has  
changed?

---

<div class="post-metadata">

**Author:** ![Seth\_Falcon\_01](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/seth_falcon_01/32/429_2.png) [@Seth\_Falcon\_01](https://discourse.chef.io/u/Seth_Falcon_01)\
**Post date:** [December 8, 2010, 11:54pm UTC](https://discourse.chef.io/t/effeciency-of-data-bags/1649/2 "2010-12-08T23:54:03Z")

</div>

Hi,

On Wed, Dec 8, 2010 at 11:40 AM, Mason Turner [opsmason@gmail.com](mailto:opsmason@gmail.com) wrote:

> I have a recipe that builds an rdist distfile by searching for nodes  
> that match a particular attribute. These queries are taking about 3  
> seconds, and i have less then 20 nodes in my chef DB. I'm sure there  
> are some ways to optimize my chef server, but I am also considering  
> pre-building the data in a databag. Does chef-client use a "last  
> modified" field to prevent lookups from the databag if nothing has  
> changed?

I don't think that chef-client currently does any conditional GETs.  
And in fact, a common way of accessing data bags is via search :-\

My general advice is to stick with search and to investigate tuning if  
the performance isn't satisfactory.

How many queries are you doing in your recipe? Three second response  
time isn't so great, but in the context of a chef-client run might not  
be so bad unless you are running many such queries. Perhaps you can  
pull across a larger query -- like all nodes if you only have 20 and  
do all of the querying locally. Then you'd only need a single query.

- seth

---

<div class="post-metadata">

**Author:** ![Joshua\_Timberman](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/joshua_timberman/32/424_2.png) [@Joshua\_Timberman](https://discourse.chef.io/u/Joshua_Timberman)\
**Post date:** [December 9, 2010, 12:51am UTC](https://discourse.chef.io/t/effeciency-of-data-bags/1649/3 "2010-12-09T00:51:51Z")

</div>

-----BEGIN PGP SIGNED MESSAGE-----  
Hash: SHA1

Hi Mason!

On Dec 8, 2010, at 12:40 PM, Mason Turner wrote:

> I have a recipe that builds an rdist distfile by searching for nodes  
> that match a particular attribute. These queries are taking about 3  
> seconds, and i have less then 20 nodes in my chef DB. I'm sure there  
> are some ways to optimize my chef server, but I am also considering  
> pre-building the data in a databag. Does chef-client use a "last  
> modified" field to prevent lookups from the databag if nothing has  
> changed?

A couple hints for data bag use:

1. You can load a data bag directly rather than searching.

Then you can access mything in your recipe.

1. You can save the data bag to the node's run\_state. This is an internal holding area where Chef keeps track of the recipes and templates that it has seen, is available across all recipes in the node's run list, and is not persisted as a node attribute at the end of the run.

In one recipe:

```
node.run_state["mything"] = data_bag_item("bagname", "itemname")

```

In another:

```
mything = node.run_state["mything"]

```

We do this in the application cookbook[0](http://cookbooks.opscode.com/cookbooks/application), for example, to store the application data to use across multiple recipes.

1. (bonus) If your search query would be the same in multiple recipes (ie, "_:_" or similar), you can store the results in the run\_state, too.

* * *

Opscode, Inc  
Joshua Timberman, Technical Evangelist  
IRC, Skype, Twitter, Github: jtimberman

-----BEGIN PGP SIGNATURE-----  
Version: GnuPG v1.4.10 (Darwin)

iEYEARECAAYFAk0AKCcACgkQO97WSdVpzT2H3wCfXCIxzM+A4YmXOBWiAtR5euBR  
l04An0sY11txqh+nlW/nBlGeQagsqLjM  
=9vaq  
-----END PGP SIGNATURE-----

---

<div class="post-metadata">

**Author:** ![Mason\_Turner](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/mason_turner/32/817_2.png) [@Mason\_Turner](https://discourse.chef.io/u/Mason_Turner)\
**Post date:** [December 9, 2010, 3:39am UTC](https://discourse.chef.io/t/effeciency-of-data-bags/1649/4 "2010-12-09T03:39:24Z")

</div>

(apologies for the top post)

Thanks. Right now I have my chef server running off a Xen VM with a single CPU and 4GB of RAM. I'm safely moving from "playing around with chef" to some serious production support, so I'll move everything to a beefier VM. Hopefully that'll speed up the queries.

My recipe is making two data\_bag\_item calls (against two different bags) and one search call. The search call is very specific (node,"custom\_attribute:#{unsigned\_int\_key}") that is searching over about 12 nodes to return two. It doesn't seem like they should take too long. I'll benchmark before and after changing VM images.

The run\_state looks very promising, though. Thanks for the tip.

-- Mason Turner (mobile)

On Dec 8, 2010, at 7:51 PM, Joshua Timberman [joshua@opscode.com](mailto:joshua@opscode.com) wrote:

> -----BEGIN PGP SIGNED MESSAGE-----  
> Hash: SHA1
> 
> Hi Mason!
> 
> On Dec 8, 2010, at 12:40 PM, Mason Turner wrote:
> 
> > I have a recipe that builds an rdist distfile by searching for nodes  
> > that match a particular attribute. These queries are taking about 3  
> > seconds, and i have less then 20 nodes in my chef DB. I'm sure there  
> > are some ways to optimize my chef server, but I am also considering  
> > pre-building the data in a databag. Does chef-client use a "last  
> > modified" field to prevent lookups from the databag if nothing has  
> > changed?
> 
> A couple hints for data bag use:
> 
> 1. You can load a data bag directly rather than searching.
> 
> Then you can access mything in your recipe.
> 
> 1. You can save the data bag to the node's run\_state. This is an internal holding area where Chef keeps track of the recipes and templates that it has seen, is available across all recipes in the node's run list, and is not persisted as a node attribute at the end of the run.
> 
> In one recipe:
> 
> node.run\_state["mything"] = data\_bag\_item("bagname", "itemname")
> 
> In another:
> 
> mything = node.run\_state["mything"]
> 
> We do this in the application cookbook[0](http://cookbooks.opscode.com/cookbooks/application), for example, to store the application data to use across multiple recipes.
> 
> 1. (bonus) If your search query would be the same in multiple recipes (ie, "_:_" or similar), you can store the results in the run\_state, too.
> 
> * * *
> 
> Opscode, Inc  
> Joshua Timberman, Technical Evangelist  
> IRC, Skype, Twitter, Github: jtimberman
> 
> -----BEGIN PGP SIGNATURE-----  
> Version: GnuPG v1.4.10 (Darwin)
> 
> iEYEARECAAYFAk0AKCcACgkQO97WSdVpzT2H3wCfXCIxzM+A4YmXOBWiAtR5euBR  
> l04An0sY11txqh+nlW/nBlGeQagsqLjM  
> =9vaq  
> -----END PGP SIGNATURE-----