I’ve seen references to this topic before, but without great solutions.
Suppose I have a farm of 100 hosts, all of which get provisioned around the same time with Chef (via a startup script or the like). Now, suppose I need just one of those servers to have a cron job installed to do some basic housekeeping. What’s the best way to approach this? Let’s assume that these hosts are on AWS in an autoscaling group, so they’re all treated identically.
I know I could easily tag a host with knife, which is a manual step. I could also do a search within a recipe to search for a host being tagged, and claim the tag if it’s free, but since node data isn’t pushed to the server until a Chef run is complete, (unless that’s changed?) everyone could claim the tag.
Is there any way to implement a global lock, of sorts? e.g., I could do a search in a recipe for a tagged host, and if no results turn up, the recipe could set a real-time lock (held by only one node) that allows the tag to be set. Ideally (optionally?), the other nodes would wait until the lock is released before progressing.
This could also be applicable to fresh master/slave MySQL setups, Elasticsearch, and the like.
Any thoughts would be well-appreciated.