I’ve seen references to this topic before, but without great solutions.
Suppose I have a farm of 100 hosts, all of which get provisioned around the same time with Chef (via a startup script or the like). Now, suppose I need just one of those servers to have a cron job installed to do some basic housekeeping. What’s the best way to approach this? Let’s assume that these hosts are on AWS in an autoscaling group, so they’re all treated identically.
I know I could easily tag a host with knife, which is a manual step. I could also do a search within a recipe to search for a host being tagged, and claim the tag if it’s free, but since node data isn’t pushed to the server until a Chef run is complete, (unless that’s changed?) everyone could claim the tag.
Is there any way to implement a global lock, of sorts? e.g., I could do a search in a recipe for a tagged host, and if no results turn up, the recipe could set a real-time lock (held by only one node) that allows the tag to be set. Ideally (optionally?), the other nodes would wait until the lock is released before progressing.
This could also be applicable to fresh master/slave MySQL setups, Elasticsearch, and the like.
Unfortunately the Chef API doesn’t really have anything good enough to offer true CP leader elections. The best solution depends on how bad it would be for the job to execute on two hosts sometimes. If it is wasted time but not actually bad (i.e. the job is idempotent internally) then a solution entirely inside Chef using node tags and the search index might be okay. You would have to very carefully manage what happens when two nodes try to claim the tag though, since the search index can lag by quite a bit sometimes (think ~60 seconds). This would provide “at least once” behavior. If instead you need “at most once”, you would need a CP data store like Consul or ZooKeeper. You could either query the system from Chef, or build it into the cron job that every machine tries to run it, but only the first in each cycle will get the lock and the others will no-op. I know PagerDuty has used this strategy successfully, but I’m not sure if there are any open source implementations. You could also look at Chronos if you are eyeing bigger frameworks, but that would be overkill for one cron job by a lot.
why not just create a role which has the original run list + recipe involving the cron job and call it housekeeping. that way the special case is captured as well in chef.
You can use an external strongly consistent data store like zookeeper or consul or etcd for distributed locks. I have used etcd for locks, but for a completely different purpose. The new chef handler DSL has an example of exactly what you mentioned.
We provision/maintain a lot of distributed systems (zk, cass, consul, serf) as well as mysql clusters (multi-master/master-slave), and our learning is to orchestrate them from outside rather making the individual recipe/resources smarter. i.e. do cluster provisioning where the leaders are upfront decided by the provisioners. This eases the error handling as well.
When you try to automate such a task, you really have to deal with two separate questions. One is how to select the “special animal” among all the other ones.
The second one is how to deal with the special animal going away for any reason. For instance, in an AWS autoscaling group, what happens when AWS selects the machine you selected as a housekeeper for removal during low utilization? The corresponding node object may still be stored in your chef server until you remove it somehow.
For this reason, it’s often best not to use chef for this purpose.
I would probably consider using a separate instance outside your autoscaling group to handle such tasks - a “controller” of sorts, with a very different role. Making the instances in an autoscaling group deliberately different from each other seems like a square peg-round hole.