New Community Cookbook: MongoDB

Chef Infra (archive)
1

Hi!

this week we are going to release a couple of Chef technologies
developed at edelight GmbH.

The first one is a new MongoDB cookbook, mostly implemented by my
colleague Markus Korn. The code can be found at our Github repository:
https://github.com/edelight/cookbooks/tree/master/mongodb

As there were no MongoDB cookbooks out there, we decided to invest in
a really capable one that could manage MongoDB instances in a variety
of configurations.

Instances can be started in a recipe by using the "mongodb_instance"
definition, and the cookbook itself provides the following recipes:

  • default.rb (single node instance)
  • configserver
  • mongos
  • replicaset
  • shard.rb

We have put a tutorial on the wiki describing the setup of a full
sharded and replicated cluster:
https://github.com/edelight/cookbooks/wiki/MongoDB%3A-Replication%2BSharding

There are some things that could still be improved, but we are pretty
happy with it as-is.

Cheers!
Miquel Torres

PS: We are entering the Opscode Chef Cookbook Contest with this
cookbook btw. :wink:

2

On Sep 6, 2011, at 9:44 AM, Miquel Torres wrote:

We have put a tutorial on the wiki describing the setup of a full
sharded and replicated cluster:
https://github.com/edelight/cookbooks/wiki/MongoDB%3A-Replication%2BSharding

There are some things that could still be improved, but we are pretty
happy with it as-is.

We've looked at the tutorial page that you've got, but we're trying to modify this cookbook to work on CentOS 5.6, and we're running into some problems.

In cookbooks/mongodb/definitions/mongodb.rb, starting at line 140, we have this code:

replicaset

if !replicaset_name.nil?
rs_nodes = search(
:node,
"mongodb_cluster_name:#{replicaset['mongodb']['cluster_name']} AND
recipes:mongodb\:\:replicaset AND
mongodb_shard_name:#{replicaset['mongodb']['shard_name']} AND
chef_environment:#{replicaset.chef_environment}"
)

However, this search is returning null for us. We confirmed this by adding the following logging statement:

Chef::Log.info("mongodb:rs_nodes:#{rs_nodes}")

The result is a chef-client run that looks like this:
173-203-195-99.static.cloud-ips.com [Thu, 15 Sep 2011 23:43:43 +0000] WARN: We are not using a configfile, as the daemons can be configured via commandline
173-203-195-99.static.cloud-ips.com [Thu, 15 Sep 2011 23:43:43 +0000] WARN: We are not using a configfile, as the daemons can be configured via commandline
173-203-195-99.static.cloud-ips.com [Thu, 15 Sep 2011 23:43:44 +0000] INFO: mongodb:rs_nodes:
173-203-195-99.static.cloud-ips.com [Thu, 15 Sep 2011 23:43:44 +0000] INFO: Processing gem_package[mongo] action install (mongodb::default line 34)

We're having trouble figuring out why this search would be returning null, and our forays with Shef and all of our other testing so far have not been very enlightening.

I tried posting this as a comment at http://community.opscode.com/cookbooks/mongodb, but all the formatting was destroyed in that message, and there doesn't seem to be any way that I can delete a comment that I had made.

I don't suppose you have any hints or comments? Thanks!

--
Brad Knowles bknowles@ihiji.com

3

Hi Brad,
first of all, thanks for using the mongodb cookbook, and your feedback.

On Fri, Sep 16, 2011 at 2:30 AM, Brad Knowles bknowles@ihiji.com wrote:
[...]

In cookbooks/mongodb/definitions/mongodb.rb, starting at line 140, we have this code:

replicaset

if !replicaset_name.nil?
rs_nodes = search(
:node,
"mongodb_cluster_name:#{replicaset['mongodb']['cluster_name']} AND
recipes:mongodb\:\:replicaset AND
mongodb_shard_name:#{replicaset['mongodb']['shard_name']} AND
chef_environment:#{replicaset.chef_environment}"
)

However, this search is returning null for us. We confirmed this by adding the following logging statement:

Chef::Log.info("mongodb:rs_nodes:#{rs_nodes}")

There could be multible reasons for this search returning nil, can you
please check if

  • node['mongodb']['cluster_name'] is defined and has the same value
    for all the nodes in the replicaset (e.g you have one role which
    defines this attribute in 'default_attributes' and this role is in
    every nodes run_list)

  • in every replicaset member there is either directly or indeirectly
    the 'mongodb::replicaset' recipe in the run_list (maybe by also adding
    this recipe to the role )

  • If you are only using a replicaset (without sharding) you should
    not really care about mongodb's shard_name attribute, it should be set
    to '_default' on all mongodb related nodes

  • and last but not least you could check if chef_envionment is set correctly

If I had to debug this further I would remove the the various AND
conditions from the search statement one-by-one and see if rs_nodes
has somemembers then. This way it is easy to find the
culprit.

The result is a chef-client run that looks like this:
[...]
I don't suppose you have any hints or comments? Thanks!

I hope I could help you with my hints, please come back if you need
some more help with the mongodb cookbook.

Markus

4

On Sep 16, 2011, at 1:13 AM, markus korn wrote:

first of all, thanks for using the mongodb cookbook, and your feedback.

Thank you for all the hard work you've put into this cookbook! There are other cookbooks for MongoDB that we've seen, and while they are simpler and easier to use, they are ... simpler -- they don't do sharding, they don't do replicasets, etc.... You've clearly done a lot of hard work here.

We're just trying to figure out how to adapt this to also work on CentOS 5.6, and sometimes we get tripped up by being less than completely familiar with Chef and we don't necessarily recognize when something is an Ubuntu-ism that's in the cookbook, an Ubuntu-ism that's in Chef, or maybe just our lack of fully understanding how things work.

  • node['mongodb']['cluster_name'] is defined and has the same value
    for all the nodes in the replicaset (e.g you have one role which
    defines this attribute in 'default_attributes' and this role is in
    every nodes run_list)

Right now, we're trying to get the first node spun up. Once we can get a machine running under Chef with an empty MongoDB starting up properly (ready for replicas and shards), then we were planning on tearing it down and firing up a group of four nodes in a two-shard/one-replicaset-per-shard configuration and seeing how that works.

I can confirm that on the one node we've been working on so far, this value is defined and is what we had configured it to be.

  • in every replicaset member there is either directly or indeirectly
    the 'mongodb::replicaset' recipe in the run_list (maybe by also adding
    this recipe to the role )

I'm not seeing this in the run_list at the point where we display that information, but I don't know if that's being added at a later time. I will take a look.

  • If you are only using a replicaset (without sharding) you should
    not really care about mongodb's shard_name attribute, it should be set
    to '_default' on all mongodb related nodes

We're not sharded yet, but our production database will be, so we want to plan for that from the beginning.

  • and last but not least you could check if chef_envionment is set correctly

It seems to be set to "_default", but I don't know if that's correct.

If I had to debug this further I would remove the the various AND
conditions from the search statement one-by-one and see if rs_nodes
has somemembers then. This way it is easy to find the
culprit.

I'll start decomposing that next.

I hope I could help you with my hints, please come back if you need
some more help with the mongodb cookbook.

Indeed, this has been helpful. I will let you know what I find out. Thanks again!

--
Brad Knowles bknowles@ihiji.com

5

On Sep 16, 2011, at 9:30 AM, Brad Knowles wrote:

  • in every replicaset member there is either directly or indeirectly
    the 'mongodb::replicaset' recipe in the run_list (maybe by also adding
    this recipe to the role )

I'm not seeing this in the run_list at the point where we display that information, but I don't know if that's being added at a later time. I will take a look.

I took a deeper look, and I can confirm that 'mongodb::replicaset' is being added to the run_list.

If I had to debug this further I would remove the the various AND
conditions from the search statement one-by-one and see if rs_nodes
has somemembers then. This way it is easy to find the
culprit.

I'll start decomposing that next.

In de-composing the various AND conditions, we ran across an issue with the line that reads:

         recipes:mongodb\\:\\:replicaset AND \

I was wondering why this particular syntax was used (with the double-backslashes), instead of a syntax more like this:

         recipes[mongodb::replicaset] AND \

Or why that block is enclosed with double quote characters, where I understand that single-quote characters would remove the requirement for the backslash escape mechanism?

Thanks again!

--
Brad Knowles bknowles@ihiji.com
Junior Apprentice Prep Cook Trainee

6

On Fri, Sep 16, 2011 at 6:45 PM, Brad Knowles bknowles@ihiji.com wrote:

On Sep 16, 2011, at 9:30 AM, Brad Knowles wrote:

[...]

In de-composing the various AND conditions, we ran across an issue with the line that reads:

        recipes:mongodb\\:\\:replicaset AND \

I was wondering why this particular syntax was used (with the double-backslashes), instead of a syntax more like this:

        recipes[mongodb::replicaset] AND \

The query syntax for searches in the expanded run_list is documented
in the chef wiki at [0].
It should be
search(:node, 'recipes:apache2::mod_proxy_ajp')
And this translates to
irb(main):006:0* 'recipes:apache2::mod_proxy_ajp'
=> "recipes:apache2\:\:mod_proxy_ajp"
So the query itself should be alright. But, as mentioned in the wiki,
you need at least chef 0.9.8 for this feature.
It is possible you are using an older version of chef?

Greetings,
have a nice weekend,
Markus

[0] http://wiki.opscode.com/display/chef/Search#Search-FindNodeswithaRecipeintheExpandedRunList

7

On Sep 16, 2011, at 9:30 AM, Brad Knowles wrote:

I hope I could help you with my hints, please come back if you need
some more help with the mongodb cookbook.

Indeed, this has been helpful. I will let you know what I find out. Thanks again!

I'm not sure if I've gotten past the previous problem, or if I have somehow regressed, but I ran into another problem that's got me puzzled. This time, it's keeping the code from even compiling, so I can't be sure if the previous problem is resolved or not.

I've added some debugging statements, so the line numbers have changed, but this code block is largely still the same:

181     ruby_block "config_replicaset" do
182       block do
183         Chef::Log.info("mongodb:config_replicaset:#{replicaset} #{replicaset_name} #{rs_nodes}")
184         if not replicaset.nil?
185           MongoDB.configure_replicaset(replicaset, replicaset_name, rs_nodes)
186         else
187           Chef::Log.info("replicaset nil!")
188         end
189       end
190       action :nothing
191     end

Can you tell me under what circumstances that you might think that the entire replicaset itself could be nil? By the time we've gotten down to this code and extracted the replicaset name and the rs_nodes, it seems that the entire replicaset itself couldn't possibly be nil. Because of this test against nil, we are seeing the following problem trying to compile this ruby_block:

FATAL: NoMethodError: ruby_block[config_replicaset] (mongodb::shard line 181) had an error: undefined method `length' for nil:NilClass

As you can see, line 181 is the very first line of this ruby_block. I added lines 183, 186, and 187. But the test of "!replicaset.nil?" was already there, and looking at your current code at https://github.com/edelight/cookbooks/blob/master/mongodb/definitions/mongodb.rb I see the same test.

--
Brad Knowles bknowles@ihiji.com
Junior Apprentice Prep Cook Trainee

8

On Sep 16, 2011, at 2:35 PM, markus korn wrote:

So the query itself should be alright. But, as mentioned in the wiki,
you need at least chef 0.9.8 for this feature.
It is possible you are using an older version of chef?

I'm using the latest 0.10.4-3 version from the omnibus installer.

The only thing that I can think of is that maybe the node hasn't been saved after the run_list has been expanded, so that the expanded run_list isn't available everywhere, at least not reliably? I know they did a forced node.save in the mysql server recipe for Chef types other than solo (cookbooks/mysql/recipes/server.rb, line 186), but I don't know if that kind of thing has been an issue elsewhere.

--
Brad Knowles bknowles@ihiji.com
Junior Apprentice Prep Cook Trainee

9

Hi Brad,

On Fri, Sep 16, 2011 at 9:44 PM, Brad Knowles bknowles@ihiji.com wrote:

On Sep 16, 2011, at 9:30 AM, Brad Knowles wrote:
[...]

FATAL: NoMethodError: ruby_block[config_replicaset] (mongodb::shard line 181) had an error: undefined method `length' for nil:NilClass

This is still the same issue, the search query returns an empty
result, and thus rs_nodes is nil. Unfortunatly this error message does
not help, the error is in fact in the mongodb.rb library file [0].

For me it is a bit hard to debug your issue further, without knowing
which changes you made to the cookbook, and how your configuration
looks like.
If you don't mind, let's take further debugging off-list and let's
return with our results :wink:

So, can you please send me information about your setup, and changes
you've made to the cookbook to thekorn at gmx dot de and I'll
have a more detailed look.

Greetings,
Markus

[0] https://github.com/edelight/cookbooks/blob/master/mongodb/libraries/mongodb.rb#L31

10

On Sep 19, 2011, at 6:03 AM, markus korn wrote:

This is still the same issue, the search query returns an empty
result, and thus rs_nodes is nil. Unfortunatly this error message does
not help, the error is in fact in the mongodb.rb library file [0].

For me it is a bit hard to debug your issue further, without knowing
which changes you made to the cookbook, and how your configuration
looks like.
If you don't mind, let's take further debugging off-list and let's
return with our results :wink:

Fair enough.

So, can you please send me information about your setup, and changes
you've made to the cookbook to thekorn at gmx dot de and I'll
have a more detailed look.

Will do. Thanks again!

--
Brad Knowles bknowles@ihiji.com
Junior Apprentice Prep Cook Trainee

11

PS: We are entering the Opscode Chef Cookbook Contest with this
cookbook btw. :wink:

Looks like an excellent entry, thanks!

Thanks,
Matt Ray
Senior Technical Evangelist | Opscode Inc.
matt@opscode.com | (512) 731-2218
Twitter, IRC, GitHub: mattray

12

Thanks. We think it really highlights the autodiscovery power of Chef.

You can add or remove replicaset nodes to a shard, or even complete
shards and config servers, and the whole cluster will reconfigure
itself. To do that automatically would have been totally impossible
without search. Awesome feature.

2011/9/6 Matt Ray matt@opscode.com:

PS: We are entering the Opscode Chef Cookbook Contest with this
cookbook btw. :wink:

Looks like an excellent entry, thanks!

Thanks,
Matt Ray
Senior Technical Evangelist | Opscode Inc.
matt@opscode.com | (512) 731-2218
Twitter, IRC, GitHub: mattray