Chef-client run on large number of nodes

Chef Infra (archive)
1

Hi

I have my own chef server installed and I am going to add some 1000 of nodes to my server.
I wonder if chef-client run on all the nodes after a specified period of time repeatedly it may result in lower performance of chef server.
Can I automate the chef-client run only after I update & save the run-list on server. I mean only when I will update the run-list my chef server should notify the nodes to run chef-client at its end.

Thanks & Regards,
Upma Gupta (WT01 - Integrated Cloud Group)
Wipro Technology
Bangalore, India

Please do not print this email unless it is absolutely necessary.

The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments.

WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email.

2

Hi.

I mentioned some benchmark conclusions here that might help:
chef - [chef] Re: Re: Sizing a Chef Server. The 500
nodes were converged concurrently without splay (although in normal
operation as mentioned elsewhere it is a good idea to configure it).

You talk about performance degradation over time. The Chef server does not
store history. A non-exclusive list of things that grow are:

  • general logging, until they are rotated out according to your schedule
  • postgres' various transactional logs until they too expire

However, these are a part of normal operations, and properly managed need
not impact performance.

You also ask about running the chef-client only after updating the
run_list. To be clear, Chef currently does not support a "push" or "poke"
model without some outside influence. The onus is on the client to contact
the server. All of Andy's suggestions are good glue with which to invoke
the client across many boxes in order coordinate running it with some other
event, like updating the run_list.

On 1 October 2013 12:50, upma.gupta@wipro.com wrote:

Hi****

I have my own chef server installed and I am going to add some 1000 of
nodes to my server.****

I wonder if chef-client run on all the nodes after a specified period of
time repeatedly it may result in lower performance of chef server.****

Can I automate the chef-client run only after I update & save the run-list
on server. I mean only when I will update the run-list my chef server
should notify the nodes to run chef-client at its end.****

Thanks & Regards,****

Upma Gupta (WT01 - Integrated Cloud Group)****

Wipro Technology****

Bangalore, India****

  • Please do not print this email unless it is absolutely necessary. *****

The information contained in this electronic message and any attachments
to this message are intended for the exclusive use of the addressee(s) and
may contain proprietary, confidential or privileged information. If you are
not the intended recipient, you should not disseminate, distribute or copy
this e-mail. Please notify the sender immediately and destroy all copies of
this message and any attachments.

WARNING: Computer viruses can be transmitted via email. The recipient
should check this email and any attachments for the presence of viruses.
The company accepts no liability for any damage caused by any virus
transmitted by this email.

www.wipro.com

3

There's a splay option you can set in order to offset the run times per node.

if you're running chef 11 it's been proven that it can handle thousands of nodes on a single chef server with no performance inpacts.

—
Sent from Mailbox for iPhone

On Tue, Oct 1, 2013 at 7:52 AM, null upma.gupta@wipro.com wrote:

Hi
I have my own chef server installed and I am going to add some 1000 of nodes to my server.
I wonder if chef-client run on all the nodes after a specified period of time repeatedly it may result in lower performance of chef server.
Can I automate the chef-client run only after I update & save the run-list on server. I mean only when I will update the run-list my chef server should notify the nodes to run chef-client at its end.
Thanks & Regards,
Upma Gupta (WT01 - Integrated Cloud Group)
Wipro Technology
Bangalore, India
Please do not print this email unless it is absolutely necessary.
The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments.
WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email.
www.wipro.com

4

Thanks Elvin,

Elvin I want some trigger mechanism from my chef server so that when I update the run_list It should automatically run the chef-client on the respective nodes using some local script.

From: Elvin Abordo [mailto:elvin159@gmail.com]
Sent: Tuesday, October 01, 2013 5:29 PM
To: chef@lists.opscode.com
Cc: chef@lists.opscode.com
Subject: [chef] Re: chef-client run on large number of nodes

There's a splay option you can set in order to offset the run times per node.

if you're running chef 11 it's been proven that it can handle thousands of nodes on a single chef server with no performance inpacts.
—
Sent from Mailboxhttps://www.dropbox.com/mailbox for iPhone

On Tue, Oct 1, 2013 at 7:52 AM, upma.gupta@wipro.commailto:upma.gupta@wipro.com <upma.gupta@wipro.commailto:upma.gupta@wipro.com> wrote:
Hi

I have my own chef server installed and I am going to add some 1000 of nodes to my server.
I wonder if chef-client run on all the nodes after a specified period of time repeatedly it may result in lower performance of chef server.
Can I automate the chef-client run only after I update & save the run-list on server. I mean only when I will update the run-list my chef server should notify the nodes to run chef-client at its end.

Thanks & Regards,
Upma Gupta (WT01 - Integrated Cloud Group)
Wipro Technology
Bangalore, India

Please do not print this email unless it is absolutely necessary.

The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments.

WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email.

www.wipro.comhttp://www.wipro.com

Please do not print this email unless it is absolutely necessary.

The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments.

WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email.

5

On Tue, Oct 1, 2013 at 1:19 PM, upma.gupta@wipro.com wrote:

Elvin I want some trigger mechanism from my chef server so that when I
update the run_list It should automatically run the chef-client on the
respective nodes using some local script.

You chef server might struggle converging 1000s node at a time!!!

There's apparently a new feature coming "soon" to Chef that will allow
you to do this but you could use this:

$ knife ssh name:* chef-client

or this:

https://github.com/substack/fleet

or this:

http://puppetlabs.com/mcollective

or this:

http://docs.fabfile.org/en/1.8/

--
Andy Gale

http://twitter.com/andygale