Hi,
I have a set of servers that need run chef-client at 30 min. interval.
Any of them have to execute an operation at any given time but can’t have
more then one at the same time.
I’m looking to make a lock(mutex) that is saved onto the chef server by
a recipe.
My question is how can I save it ?
thanks!
-silviu
Hey Silviu,
It is 'possible' to do this with Chef, though I can't guarantee Chef will
do it well. This sort of coordination is not what Chef excels at, and if
it is really important I would recommend using something better at
coordination, like Zookeeper. That said, here are some possible Chef
solutions:
Use a library file to make an API call to some endpoint that distributes
your locks, this can be something hacked up in Chef server, or a different
service (like Zookeeper or Chubby). Have the result from this library call
block the recipe if you can't get the lock.
Node Attributes can work, but you need to be careful. You can set a
particular attribute when you 'have the lock' and then always search for
that attribute on any nodes before you do you dangerous operation. There is
no built in locking mechanism, so you will need to wrap it very carefully.
I would not recommend it.
Databags are another option. These have many of the same caveats as
using Node attributes as it is hard to guarantee and atomic update. The
most likely failure scenario is two nodes read that no one has the lock and
both try to add it. Since databags always overwrite with the latest data,
they will both 'write' the lock and proceed to cheerfully destroy data.
To do this sort of thing properly, you will want a system than can do an
atomic compare-and-update, which is not what Chef was designed for in Node
attributes and Databags. There may be some other subsystem that can
perform this, but I cannot think of it right now.
Andrew
On Thu, Nov 14, 2013 at 9:14 PM, silviudicu@gmail.com wrote:
Hi,
I have a set of servers that need run chef-client at 30 min. interval.
Any of them have to execute an operation at any given time but can't have
more then one at the same time.I'm looking to make a lock(mutex) that is saved onto the chef server by
a recipe.My question is how can I save it ?
- use a node attribute and all other will do a search for it
- make use of a databag(need to distribute keys to be a client on nodes)
...thanks!
-silviu
andrew@yipit.com writes:
It is 'possible' to do this with Chef, though I can't guarantee Chef will
do it well. This sort of coordination is not what Chef excels at, and if
it is really important I would recommend using something better at
coordination, like Zookeeper. That said, here are some possible Chef
solutions:
+1 on the caveats. Chef Server doesn't provide orchestration primitives
at this time.
It is something that we discuss adding and it always helps to examples
of what would be useful.
- Databags are another option. These have many of the same caveats as
using Node attributes as it is hard to guarantee and atomic update. The
most likely failure scenario is two nodes read that no one has the lock and
both try to add it. Since databags always overwrite with the latest data,
they will both 'write' the lock and proceed to cheerfully destroy
data.
While it really is not a good idea, if you are going go down this route,
I'd recommend using the presence/absence of a data bag (not data bag
item).
api.post("data", {:name => "lock1"})
This will only succeed if there is not already such a data bag (returns
409 if a bag named lock1 already exists). Top-level data bags are about
as light weight as you can get and get you closest to the atomic CAS
thing.
--
Seth Falcon | Development Lead | Opscode | @sfalcon
Hi,
Can you elaborate on 'this can be something hacked up in Chef server'
From a node/client that is not admin on the open source chef you are limited to node attributes - unless there is a different way to do it.
The strategy check if exists is fine (I imagined the same).
thanks!
On Nov 15, 2013, at 11:48 AM, Seth Falcon seth@opscode.com wrote:
andrew@yipit.com writes:
It is 'possible' to do this with Chef, though I can't guarantee Chef will
do it well. This sort of coordination is not what Chef excels at, and if
it is really important I would recommend using something better at
coordination, like Zookeeper. That said, here are some possible Chef
solutions:+1 on the caveats. Chef Server doesn't provide orchestration primitives
at this time.It is something that we discuss adding and it always helps to examples
of what would be useful.
- Databags are another option. These have many of the same caveats as
using Node attributes as it is hard to guarantee and atomic update. The
most likely failure scenario is two nodes read that no one has the lock and
both try to add it. Since databags always overwrite with the latest data,
they will both 'write' the lock and proceed to cheerfully destroy
data.While it really is not a good idea, if you are going go down this route,
I'd recommend using the presence/absence of a data bag (not data bag
item).api.post("data", {:name => "lock1"})
This will only succeed if there is not already such a data bag (returns
409 if a bag named lock1 already exists). Top-level data bags are about
as light weight as you can get and get you closest to the atomic CAS
thing.
- seth
--
Seth Falcon | Development Lead | Opscode | @sfalcon
Hi,
silviudicu@gmail.com writes:
Can you elaborate on 'this can be something hacked up in Chef server'
From a node/client that is not admin on the open source chef you are
limited to node attributes - unless there is a different way to do it.
I'm assuming you're asking about my reply, but not clear on what you'd
like more detail on.
Hi,
Sorry - I didn't include the original text.
"
So what I inquired is what else could be hacked up into the chef server except what was covered, node attributes and data bags.
thanks!
On Nov 17, 2013, at 11:59 AM, Seth Falcon seth@opscode.com wrote:
Hi,
silviudicu@gmail.com writes:
Can you elaborate on 'this can be something hacked up in Chef server'
From a node/client that is not admin on the open source chef you are
limited to node attributes - unless there is a different way to do it.I'm assuming you're asking about my reply, but not clear on what you'd
like more detail on.
- seth