Morning All,
I am running a chef 10.18.2 from the opscode repository on Ubuntu 12.04, and I
am managing 15 nodes, all running Ubuntu 12.04.
My issue is that something changed and the memory usage has exploded, and what
fit nicely in a VM with 2GB of RAM now crashes due to OOM multiple times per
day.
My install differs from stock very little, I did have to increase the
maxFieldLength of the solr config to 100,000 due to searches failing to return
results with both 1,000 and 10,000.
Each crash is a bit different, but the worst offenders are usual, in order:
- chef-server
- chef-expander (Can eat up to 60% of memory before there is none left)
- chef-server-webui (when I visit the status page)
Is 4GB of RAM simply too little for 15 nodes? IMHO it seems a bit excessive for
the size of the deployment.
I am not sure where to go to diagnose the issue, I prefer Chef’s workflow and
would like to fix this so I can continue utilizing the system.
Brian
On Tue, Jan 29, 2013 at 8:10 AM, arjesins@gmail.com wrote:
Is 4GB of RAM simply too little for 15 nodes? IMHO it seems a bit excessive for
the size of the deployment.
Not sure about 15 nodes, but from my experience, the memory
consumption isn't that much different between 100 nodes and 1000
nodes.
Brian,
So in your experience it seems to plateaus at some point (in your cahse
100nodes)? What size machine are you using, and is everything running on
one box or have you deployed it as shards?
Thanks,
Brian
On Tue, Jan 29, 2013 at 8:14 AM, Brian Akins brian@akins.org wrote:
On Tue, Jan 29, 2013 at 8:10 AM, arjesins@gmail.com wrote:
Is 4GB of RAM simply too little for 15 nodes? IMHO it seems a bit
excessive for
the size of the deployment.
Not sure about 15 nodes, but from my experience, the memory
consumption isn't that much different between 100 nodes and 1000
nodes.
On Tue, Jan 29, 2013 at 8:21 AM, Brian Malinconico arjesins@gmail.com wrote:
So in your experience it seems to plateaus at some point (in your cahse
100nodes)?
100 nodes was a nice round number - not sure what the real number was,
but yes memory plateaus.
What size machine are you using, and is everything running on one
box or have you deployed it as shards?
When we were running open source chef-server, we used an 8GB box. We
ran the api service under rainbows, not unicorn, and this help with
the overall memory usage as well. Not 100% sure if that is "supported"
but we did it for about a year and it worked fine.
We use private chef now and run several thousand nodes on each server
(one per physical datacenter). We are using larger boxes now, however.