Mongodb replicaset failure: Could not connect to database: 'localhost:27017'

[Liam_Kirsher]

Hi,

I'm having a little issue with getting mongo replicasets working.
Partly, I'm needing a little more understanding of Chef's execution
order -- what happens when, notifies, etc..
The problem is that the replicaset is never initiated, although the
parameters look correct. The apparent reason is that /when/ Chef tries
to access the database it is not running.
However, the database is running and accessible at localhost:27017 at
the end of the run.

I thought the issue might be related to this:

Fix timing bug in Mongodb replicaset configuration

so I added that to the mongo cookbook I'm using.

LiamK/Ops

(That's my fork of the Ops version, but the only thing I changed was to
disable upstart and use the standard init.d script.)
Unfortunately, that didn't help.

So, at this point I'm just looking for a good place to "sleep" to give
the mongo process time to restart, so it's running when the replicaset
config command runs.
Or, better, to sleep & test to see if it's running and accepting
connections. Should I just put that in the config_replicaset ruby_block?

Thanks,
Liam

Here is the log (with a few debugging statements thrown in):

ec2-54-218-157-24.us-west-2.compute.amazonaws.com * service[mongodb]
action enable
ec2-54-218-157-24.us-west-2.compute.amazonaws.com (up to date)
ec2-54-218-157-24.us-west-2.compute.amazonaws.com
ec2-54-218-157-24.us-west-2.compute.amazonaws.com * service[mongodb]
action start
ec2-54-218-157-24.us-west-2.compute.amazonaws.com (up to date)
ec2-54-218-157-24.us-west-2.compute.amazonaws.com
ec2-54-218-157-24.us-west-2.compute.amazonaws.com *
ruby_block[config_replicaset] action nothing
ec2-54-218-157-24.us-west-2.compute.amazonaws.com (up to date)
ec2-54-218-157-24.us-west-2.compute.amazonaws.com
ec2-54-218-157-24.us-west-2.compute.amazonaws.com Recipe: bti_mongo::default
ec2-54-218-157-24.us-west-2.compute.amazonaws.com
ec2-54-218-157-24.us-west-2.compute.amazonaws.com
ec2-54-218-157-24.us-west-2.compute.amazonaws.com * chef_gem[mongo]
action install
ec2-54-218-157-24.us-west-2.compute.amazonaws.com (up to date)
ec2-54-218-157-24.us-west-2.compute.amazonaws.com
ec2-54-218-157-24.us-west-2.compute.amazonaws.com * chef_gem[bson_ext]
action install
ec2-54-218-157-24.us-west-2.compute.amazonaws.com (up to date)
ec2-54-218-157-24.us-west-2.compute.amazonaws.com
ec2-54-218-157-24.us-west-2.compute.amazonaws.com Recipe: ntp::default
ec2-54-218-157-24.us-west-2.compute.amazonaws.com
ec2-54-218-157-24.us-west-2.compute.amazonaws.com * service[ntp]
action restart
ec2-54-218-157-24.us-west-2.compute.amazonaws.com
ec2-54-218-157-24.us-west-2.compute.amazonaws.com - restart service
service[ntp]
ec2-54-218-157-24.us-west-2.compute.amazonaws.com
ec2-54-218-157-24.us-west-2.compute.amazonaws.com
ec2-54-218-157-24.us-west-2.compute.amazonaws.com
ec2-54-218-157-24.us-west-2.compute.amazonaws.com Recipe:
mongodb::replicaset
ec2-54-218-157-24.us-west-2.compute.amazonaws.com
ec2-54-218-157-24.us-west-2.compute.amazonaws.com * service[mongodb]
action restart
ec2-54-218-157-24.us-west-2.compute.amazonaws.com
ec2-54-218-157-24.us-west-2.compute.amazonaws.com - restart service
service[mongodb]
ec2-54-218-157-24.us-west-2.compute.amazonaws.com
ec2-54-218-157-24.us-west-2.compute.amazonaws.com
ec2-54-218-157-24.us-west-2.compute.amazonaws.com
ec2-54-218-157-24.us-west-2.compute.amazonaws.com *
ruby_block[config_replicaset] action create
ec2-54-218-157-24.us-west-2.compute.amazonaws.com "ruby_block
replicaset: node[i-a62c2193]"
ec2-54-218-157-24.us-west-2.compute.amazonaws.com "ruby_block
replicaset.nil?: false"
ec2-54-218-157-24.us-west-2.compute.amazonaws.com "ruby_block >>
ebs_snaps: false"
ec2-54-218-157-24.us-west-2.compute.amazonaws.com "ruby_block >>
configuring: node[i-a62c2193] rs_default [node[i-2ee2e01b],
node[i-2edbd91b]]"
ec2-54-218-157-24.us-west-2.compute.amazonaws.com
[2013-07-01T07:25:03+00:00] WARN: Could not connect to database:
'localhost:27017'
ec2-54-218-157-24.us-west-2.compute.amazonaws.com
ec2-54-218-157-24.us-west-2.compute.amazonaws.com - execute the ruby
block config_replicaset
ec2-54-218-157-24.us-west-2.compute.amazonaws.com
ec2-54-218-157-24.us-west-2.compute.amazonaws.com
ec2-54-218-157-24.us-west-2.compute.amazonaws.com
ec2-54-218-157-24.us-west-2.compute.amazonaws.com Chef Client finished,
26 resources updated

--
Liam Kirsher
PGP: http://liam.numenet.com/pgp/

[Liam_Kirsher]

I ended up adding some code to the library that sleeps until the
connection is established and that does the trick.
I'll see if I can contribute it back to the authors.

On 07/01/2013 12:21 PM, Liam Kirsher wrote:

Hi,

I'm having a little issue with getting mongo replicasets working.
Partly, I'm needing a little more understanding of Chef's execution
order -- what happens when, notifies, etc..
The problem is that the replicaset is never initiated, although the
parameters look correct. The apparent reason is that /when/ Chef
tries to access the database it is not running.
However, the database is running and accessible at localhost:27017 at
the end of the run.

I thought the issue might be related to this:
Fix timing bug in Mongodb replicaset configuration · mongodb/mongo@695c026 · GitHub
so I added that to the mongo cookbook I'm using.
Ops/chef/cookbooks/mongodb at master · LiamK/Ops · GitHub
(That's my fork of the Ops version, but the only thing I changed was
to disable upstart and use the standard init.d script.)
Unfortunately, that didn't help.

So, at this point I'm just looking for a good place to "sleep" to give
the mongo process time to restart, so it's running when the replicaset
config command runs.
Or, better, to sleep & test to see if it's running and accepting
connections. Should I just put that in the config_replicaset ruby_block?

Thanks,
Liam

Here is the log (with a few debugging statements thrown in):

ec2-54-218-157-24.us-west-2.compute.amazonaws.com * service[mongodb]
action enable
ec2-54-218-157-24.us-west-2.compute.amazonaws.com (up to date)
ec2-54-218-157-24.us-west-2.compute.amazonaws.com
ec2-54-218-157-24.us-west-2.compute.amazonaws.com * service[mongodb]
action start
ec2-54-218-157-24.us-west-2.compute.amazonaws.com (up to date)
ec2-54-218-157-24.us-west-2.compute.amazonaws.com
ec2-54-218-157-24.us-west-2.compute.amazonaws.com *
ruby_block[config_replicaset] action nothing
ec2-54-218-157-24.us-west-2.compute.amazonaws.com (up to date)
ec2-54-218-157-24.us-west-2.compute.amazonaws.com
ec2-54-218-157-24.us-west-2.compute.amazonaws.com Recipe:
bti_mongo::default
ec2-54-218-157-24.us-west-2.compute.amazonaws.com
ec2-54-218-157-24.us-west-2.compute.amazonaws.com
ec2-54-218-157-24.us-west-2.compute.amazonaws.com * chef_gem[mongo]
action install
ec2-54-218-157-24.us-west-2.compute.amazonaws.com (up to date)
ec2-54-218-157-24.us-west-2.compute.amazonaws.com
ec2-54-218-157-24.us-west-2.compute.amazonaws.com *
chef_gem[bson_ext] action install
ec2-54-218-157-24.us-west-2.compute.amazonaws.com (up to date)
ec2-54-218-157-24.us-west-2.compute.amazonaws.com
ec2-54-218-157-24.us-west-2.compute.amazonaws.com Recipe: ntp::default
ec2-54-218-157-24.us-west-2.compute.amazonaws.com
ec2-54-218-157-24.us-west-2.compute.amazonaws.com * service[ntp]
action restart
ec2-54-218-157-24.us-west-2.compute.amazonaws.com
ec2-54-218-157-24.us-west-2.compute.amazonaws.com - restart
service service[ntp]
ec2-54-218-157-24.us-west-2.compute.amazonaws.com
ec2-54-218-157-24.us-west-2.compute.amazonaws.com
ec2-54-218-157-24.us-west-2.compute.amazonaws.com
ec2-54-218-157-24.us-west-2.compute.amazonaws.com Recipe:
mongodb::replicaset
ec2-54-218-157-24.us-west-2.compute.amazonaws.com
ec2-54-218-157-24.us-west-2.compute.amazonaws.com * service[mongodb]
action restart
ec2-54-218-157-24.us-west-2.compute.amazonaws.com
ec2-54-218-157-24.us-west-2.compute.amazonaws.com - restart
service service[mongodb]
ec2-54-218-157-24.us-west-2.compute.amazonaws.com
ec2-54-218-157-24.us-west-2.compute.amazonaws.com
ec2-54-218-157-24.us-west-2.compute.amazonaws.com
ec2-54-218-157-24.us-west-2.compute.amazonaws.com *
ruby_block[config_replicaset] action create
ec2-54-218-157-24.us-west-2.compute.amazonaws.com "ruby_block
replicaset: node[i-a62c2193]"
ec2-54-218-157-24.us-west-2.compute.amazonaws.com "ruby_block
replicaset.nil?: false"
ec2-54-218-157-24.us-west-2.compute.amazonaws.com "ruby_block >>
ebs_snaps: false"
ec2-54-218-157-24.us-west-2.compute.amazonaws.com "ruby_block >>
configuring: node[i-a62c2193] rs_default [node[i-2ee2e01b],
node[i-2edbd91b]]"
ec2-54-218-157-24.us-west-2.compute.amazonaws.com
[2013-07-01T07:25:03+00:00] WARN: Could not connect to database:
'localhost:27017'
ec2-54-218-157-24.us-west-2.compute.amazonaws.com
ec2-54-218-157-24.us-west-2.compute.amazonaws.com - execute the
ruby block config_replicaset
ec2-54-218-157-24.us-west-2.compute.amazonaws.com
ec2-54-218-157-24.us-west-2.compute.amazonaws.com
ec2-54-218-157-24.us-west-2.compute.amazonaws.com
ec2-54-218-157-24.us-west-2.compute.amazonaws.com Chef Client
finished, 26 resources updated

--
Liam Kirsher
PGP: http://liam.numenet.com/pgp/

--
Liam Kirsher
PGP: http://liam.numenet.com/pgp/