Rebuilding ephemeral chef servers


#1

Ohai Chefs!

I’m investigating building an ephemeral chef server that could be rebuilt in a cloud environment. We already have a solution for re-populating all of our cookbooks, environment files, and roles; however the node data and client keys are problematic, since chef-client always tries to use client.pem if it exists.

I’ve written a small utility to check whether the client key is valid, by fetching /nodes/ once per minute, and deleting client.pem if authentication fails. Would this put excessive load on the Chef Server to have this many requests? Are there alternate solutions folks have used to solve this problem?

Thanks!
Andrew


#2

Our approach was to wrap chef-client in a script, run-chef-client, that gets invoke from cron. If chef-client fails with “Unauthorized”, it will delete client.pem, re-register, and then re-run chef-client.

Another clever thing run-chef-client does is to force a time sync when authorization fails due to time skew.

Joe

On Aug 28, 2014, at 7:24 AM, Andrew Brown anbrown@blackberry.com wrote:

Ohai Chefs!

I’m investigating building an ephemeral chef server that could be rebuilt in a cloud environment. We already have a solution for re-populating all of our cookbooks, environment files, and roles; however the node data and client keys are problematic, since chef-client always tries to use client.pem if it exists.

I’ve written a small utility to check whether the client key is valid, by fetching /nodes/ once per minute, and deleting client.pem if authentication fails. Would this put excessive load on the Chef Server to have this many requests? Are there alternate solutions folks have used to solve this problem?

Thanks!
Andrew


#3

Hi Andrew,

From my experience, whenever you delete client.pem, chef-client will use your validation key to have a new one generated. Since this is the case, instead of needing to check on a per-minute basis that the key is valid, I say that it’s probably okay to delete the key and have a new one generated for you when it’s needed. This assumes that your validation key always remains valid; it wasn’t clear that this is the case from your email.

Thanks,
Ameir


#4

Thanks Ameir and Joe.

Joe - we’re running chef-client as a daemon, so checking return codes of chef-client is problematic. :slight_smile:
Ameir - that’s interesting. Is there any extra load on the chef server by continually inserting new client keys? How to synchronize with the interval/splay options for chef-client?

Cheers,
Andrew

On 2014-08-28, 10:42 AM, “Joe Nuspl” <nuspl@nvwls.commailto:nuspl@nvwls.com> wrote:

Our approach was to wrap chef-client in a script, run-chef-client, that gets invoke from cron. If chef-client fails with “Unauthorized”, it will delete client.pem, re-register, and then re-run chef-client.

Another clever thing run-chef-client does is to force a time sync when authorization fails due to time skew.

Joe

On Aug 28, 2014, at 7:24 AM, Andrew Brown <anbrown@blackberry.commailto:anbrown@blackberry.com> wrote:

Ohai Chefs!

I’m investigating building an ephemeral chef server that could be rebuilt in a cloud environment. We already have a solution for re-populating all of our cookbooks, environment files, and roles; however the node data and client keys are problematic, since chef-client always tries to use client.pem if it exists.

I’ve written a small utility to check whether the client key is valid, by fetching /nodes/ once per minute, and deleting client.pem if authentication fails. Would this put excessive load on the Chef Server to have this many requests? Are there alternate solutions folks have used to solve this problem?

Thanks!
Andrew