I have a question about possible race conditions during resource allocation. Let me give a simple (but somewhat silly) example. Imagine I have two types of nodes: web-server nodes and database nodes. As far as Chef is concerned, new nodes can come alive at any random time (in reality, they come alive when an auto-scaler determines that more resources are needed in the deployment). I want to make sure that each web-server, once configured and running, is connected to exactly one database. I can accomplish this by storing an attribute on the web-server nodes called “using-database”. When a new web-server comes alive, it searches the chef-server db for the list of web-server and database nodes, and it looks for a database node that is not being used by any other web-server. Then it sets that database node in its “using-database” attribute.
This works fine, but you get the usual race condition problem with shared read-write resources. If two web-servers come alive at the same time, they will search for an available database at the same time, and possibly select the same one, and save it to their “using-database” attributes. The one-web-server-per-database constraint is then violated.
Has anyone else come across this problem? Does this situation go against the intent of a Chef system? Or is there support for it already, or some other Chef best-practice?
Thanks in advance.