Couchdb is huge

Chef Infra (archive)
1

Hi,

The couchdb instance on chef-server is getting huge. I have to restart
it sometimes because its memory footprint keeps growing but the file
itself is becoming humongous:
[gilles@chefserver couchdb]$ ll -h
total 90G
-rw-r–r-- 1 couchdb couchdb 90G Sep 8 17:18 chef.couch
-rw-r–r-- 1 couchdb couchdb 3.7K Sep 8 17:18 couchdb.stderr
-rw-r–r-- 1 couchdb couchdb 9.7K Sep 8 17:18 couchdb.stdout

I don’t have an exceptional setup, all opscode cookbooks + 15 of my
own, one of them contains a 35Mo file (yes I know, not ideal, I chose
the easy way). Still far from 90Go.

Any clue?

–Gilles

2

sounds like your not running the chef server recipe as it compacts the db for your. If there is a reason your not running it the easies way is to create a new recipe with these lines in it and run on the server, otherwise you can curl a url like

http://wiki.apache.org/couchdb/Compaction

example:

jmiller@srv-203-03:~$ curl -H "Content-Type: application/json" -X POST http://localhost:5984/chef/_compact
{"ok":true}
jmiller@srv-203-03:~$

code for compact in recipe from chef-server recipe

http_request "compact chef couchDB" do
action :post
url "#{Chef::Config[:couchdb_url]}/chef/_compact"
only_if do
begin
open("#{Chef::Config[:couchdb_url]}/chef")
JSON::parse(open("#{Chef::Config[:couchdb_url]}/chef").read)["disk_size"] > 100_000_000
rescue OpenURI::HTTPError
nil
end
end
end

%w(nodes roles registrations clients data_bags data_bag_items users).each do |view|
http_request "compact chef couchDB view #{view}" do
action :post
url "#{Chef::Config[:couchdb_url]}/chef/_compact/#{view}"
only_if do
begin
open("#{Chef::Config[:couchdb_url]}/chef/_design/#{view}/_info")
JSON::parse(open("#{Chef::Config[:couchdb_url]}/chef/_design/#{view}/_info").read)["view_index"]["disk_size"] > 100_000_000
rescue OpenURI::HTTPError
nil
end
end
end
end

On Sep 8, 2010, at 10:19 AM, Gilles Devaux wrote:

Hi,

The couchdb instance on chef-server is getting huge. I have to restart
it sometimes because its memory footprint keeps growing but the file
itself is becoming humongous:
[gilles@chefserver couchdb]$ ll -h
total 90G
-rw-r--r-- 1 couchdb couchdb 90G Sep 8 17:18 chef.couch
-rw-r--r-- 1 couchdb couchdb 3.7K Sep 8 17:18 couchdb.stderr
-rw-r--r-- 1 couchdb couchdb 9.7K Sep 8 17:18 couchdb.stdout

I don't have an exceptional setup, all opscode cookbooks + 15 of my
own, one of them contains a 35Mo file (yes I know, not ideal, I chose
the easy way). Still far from 90Go.

Any clue?

--Gilles

3

Yeah, installing chef server was learning chef 0.8X, the result is a
bit of a mess, as are my first recipes...

Thanks for the info on compacting couchdb, that did the trick.

On Wed, Sep 8, 2010 at 10:41 AM, Joshua Miller jassinpain@gmail.com wrote:

sounds like your not running the chef server recipe as it compacts the db for your. If there is a reason your not running it the easies way is to create a new recipe with these lines in it and run on the server, otherwise you can curl a url like

http://wiki.apache.org/couchdb/Compaction

example:

jmiller@srv-203-03:~$ curl -H "Content-Type: application/json" -X POST http://localhost:5984/chef/_compact
{"ok":true}
jmiller@srv-203-03:~$

code for compact in recipe from chef-server recipe

http_request "compact chef couchDB" do
action :post
url "#{Chef::Config[:couchdb_url]}/chef/_compact"
only_if do
begin
open("#{Chef::Config[:couchdb_url]}/chef")
JSON::parse(open("#{Chef::Config[:couchdb_url]}/chef").read)["disk_size"] > 100_000_000
rescue OpenURI::HTTPError
nil
end
end
end

%w(nodes roles registrations clients data_bags data_bag_items users).each do |view|
http_request "compact chef couchDB view #{view}" do
action :post
url "#{Chef::Config[:couchdb_url]}/chef/_compact/#{view}"
only_if do
begin
open("#{Chef::Config[:couchdb_url]}/chef/_design/#{view}/_info")
JSON::parse(open("#{Chef::Config[:couchdb_url]}/chef/_design/#{view}/_info").read)["view_index"]["disk_size"] > 100_000_000
rescue OpenURI::HTTPError
nil
end
end
end
end

On Sep 8, 2010, at 10:19 AM, Gilles Devaux wrote:

Hi,

The couchdb instance on chef-server is getting huge. I have to restart
it sometimes because its memory footprint keeps growing but the file
itself is becoming humongous:
[gilles@chefserver couchdb]$ ll -h
total 90G
-rw-r--r-- 1 couchdb couchdb 90G Sep 8 17:18 chef.couch
-rw-r--r-- 1 couchdb couchdb 3.7K Sep 8 17:18 couchdb.stderr
-rw-r--r-- 1 couchdb couchdb 9.7K Sep 8 17:18 couchdb.stdout

I don't have an exceptional setup, all opscode cookbooks + 15 of my
own, one of them contains a 35Mo file (yes I know, not ideal, I chose
the easy way). Still far from 90Go.

Any clue?

--Gilles