Couchdb randomly dying

Has anyone noticed couchdb randomly dying? The logs aren’t helpful at all. I’ll continue to look into it… just thought someone may have encountered this problem before.

BTW, I’m running on CentOS 5.3…

[me@chef001 ~]# rpm -qa | grep chef
rubygem-chef-0.7.10-1.el5
rubygem-chef-server-0.7.10-1.el5
rubygem-chef-server-slice-0.7.10-1.el5

[me@chef001 ~]# couchdb -V
couchdb - Apache CouchDB 0.9.1

Hello,

On 29/12/2009, at 11:51 AM, m2 wrote:

Has anyone noticed couchdb randomly dying? The logs aren't helpful at all. I'll continue to look into it.. just thought someone may have encountered this problem before.

The logs certainly are helpful, just plentiful. It's a massive tuple, and would help with diagnosis.

BTW, I'm running on CentOS 5.3..

[me@chef001 ~]# rpm -qa | grep chef
rubygem-chef-0.7.10-1.el5
rubygem-chef-server-0.7.10-1.el5
rubygem-chef-server-slice-0.7.10-1.el5

[me@chef001 ~]# couchdb -V
couchdb - Apache CouchDB 0.9.1

If possible, I'd suggest upgrading to CouchDB 0.10, although 0.9.1 was substantially more stable than 0.8.x

--
Arjuna Christensen, Software Development Engineer
Opscode, Inc.
E: aj@opscode.com

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

m2 wrote:

Has anyone noticed couchdb randomly dying? The logs aren't helpful at all. I'll continue to look into it.. just thought someone may have encountered this problem before.

BTW, I'm running on CentOS 5.3..

[me@chef001 ~]# rpm -qa | grep chef
rubygem-chef-0.7.10-1.el5
rubygem-chef-server-0.7.10-1.el5
rubygem-chef-server-slice-0.7.10-1.el5

[me@chef001 ~]# couchdb -V
couchdb - Apache CouchDB 0.9.1

Can you confirm that you are compacting couch?

Adam


Opscode, Inc.
Adam Jacob, CTO
T: (206) 508-7449 E: adam@opscode.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJLOTnLAAoJEK+ksDjf4JoC+0UH/0sEl4wYb53eJH7sYY5LRxHR
LSaZzUMWcy32WCZuRIJIkt4jO17Rym0jT82LOvfTEye/XSII5nxxt1qC3hDb2khK
Y7B5ZZ45HRcPhswutisRUlzLG6+Wd5napT9/iNRqT8BKO2AhNFej1o1KUGgt7xRI
1ozbjWGUZQxS8CZr+W0oCi6Qrun9YFb5MYnQ8EaSBduhf9xVas9+61s2smUG2dC1
HrdEJzC2tVIl4dHovpmrRi5Fw0yJyv2kLmtnbyrB8STFU9cg6T/o4/Ru9ShyvynD
OE5Sy5W+Fns9KRCeUI329aw+93GoTu/kzBvwHp+wfA6a/Ur5pwhjSBn36I07vPo=
=BoMi
-----END PGP SIGNATURE-----

Thanks for the feedback. BTW, didn't mean to slam couchdb logging.. just
that I'm not familiar with reading them yet.

So, I did add a recipe to compact couchdb. After you metnioned that, I
noticed it was growing excessively large. File size went from ~2GB to 2MB
after compacting!

I'll continue to monitor the couch instance and report back if it continues
to die. Thanks for the tips!

On Mon, Dec 28, 2009 at 3:05 PM, Adam Jacob adam@opscode.com wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

m2 wrote:

Has anyone noticed couchdb randomly dying? The logs aren't helpful at
all. I'll continue to look into it.. just thought someone may have
encountered this problem before.

BTW, I'm running on CentOS 5.3..

[me@chef001 ~]# rpm -qa | grep chef
rubygem-chef-0.7.10-1.el5
rubygem-chef-server-0.7.10-1.el5
rubygem-chef-server-slice-0.7.10-1.el5

[me@chef001 ~]# couchdb -V
couchdb - Apache CouchDB 0.9.1

Can you confirm that you are compacting couch?

Adam


Opscode, Inc.
Adam Jacob, CTO
T: (206) 508-7449 E: adam@opscode.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJLOTnLAAoJEK+ksDjf4JoC+0UH/0sEl4wYb53eJH7sYY5LRxHR
LSaZzUMWcy32WCZuRIJIkt4jO17Rym0jT82LOvfTEye/XSII5nxxt1qC3hDb2khK
Y7B5ZZ45HRcPhswutisRUlzLG6+Wd5napT9/iNRqT8BKO2AhNFej1o1KUGgt7xRI
1ozbjWGUZQxS8CZr+W0oCi6Qrun9YFb5MYnQ8EaSBduhf9xVas9+61s2smUG2dC1
HrdEJzC2tVIl4dHovpmrRi5Fw0yJyv2kLmtnbyrB8STFU9cg6T/o4/Ru9ShyvynD
OE5Sy5W+Fns9KRCeUI329aw+93GoTu/kzBvwHp+wfA6a/Ur5pwhjSBn36I07vPo=
=BoMi
-----END PGP SIGNATURE-----

--
Matt Matson
m2matson@gmail.com
415.254.2777

AIM: flagsout

Hi Matt,

If you add to the Chef Server's runlist chef::server running the latest cookbooks from GitHub - chef-boneyard/cookbooks: DEPRECATED: This repository has been split up into separate repositories by cookbook under the "opscode-cookbooks" organization. it will do a compaction if your chef.couch is > 100megs on each client run; which will greatly help in avoiding said situation.

Additionally there is a bug in apache's system [COUCHDB-275] couch crashes erlang vm under heavy load - ASF JIRA ... basically you would need to update erlang in order to fix it... or you run couchdb 0.8, and that with the bugged erlang can cause 20gb couch files that won't compact.

But basically if you use the chef::server recipe it will help ensure your couchdb won't grow too big

Scott

On Dec 30, 2009, at 2:30 PM, Matt Matson wrote:

Thanks for the feedback. BTW, didn't mean to slam couchdb logging.. just that I'm not familiar with reading them yet.

So, I did add a recipe to compact couchdb. After you metnioned that, I noticed it was growing excessively large. File size went from ~2GB to 2MB after compacting!

I'll continue to monitor the couch instance and report back if it continues to die. Thanks for the tips!

On Mon, Dec 28, 2009 at 3:05 PM, Adam Jacob adam@opscode.com wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

m2 wrote:

Has anyone noticed couchdb randomly dying? The logs aren't helpful at all. I'll continue to look into it.. just thought someone may have encountered this problem before.

BTW, I'm running on CentOS 5.3..

[me@chef001 ~]# rpm -qa | grep chef
rubygem-chef-0.7.10-1.el5
rubygem-chef-server-0.7.10-1.el5
rubygem-chef-server-slice-0.7.10-1.el5

[me@chef001 ~]# couchdb -V
couchdb - Apache CouchDB 0.9.1

Can you confirm that you are compacting couch?

Adam


Opscode, Inc.
Adam Jacob, CTO
T: (206) 508-7449 E: adam@opscode.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJLOTnLAAoJEK+ksDjf4JoC+0UH/0sEl4wYb53eJH7sYY5LRxHR
LSaZzUMWcy32WCZuRIJIkt4jO17Rym0jT82LOvfTEye/XSII5nxxt1qC3hDb2khK
Y7B5ZZ45HRcPhswutisRUlzLG6+Wd5napT9/iNRqT8BKO2AhNFej1o1KUGgt7xRI
1ozbjWGUZQxS8CZr+W0oCi6Qrun9YFb5MYnQ8EaSBduhf9xVas9+61s2smUG2dC1
HrdEJzC2tVIl4dHovpmrRi5Fw0yJyv2kLmtnbyrB8STFU9cg6T/o4/Ru9ShyvynD
OE5Sy5W+Fns9KRCeUI329aw+93GoTu/kzBvwHp+wfA6a/Ur5pwhjSBn36I07vPo=
=BoMi
-----END PGP SIGNATURE-----

--
Matt Matson
m2matson@gmail.com
415.254.2777

AIM: flagsout
!DSPAM:4b3bd49e114941804284693!