Identifying if a node is being bootstrapped


#1

Hey all,

Came across a situation tonight that I wanted to get feedback on. I’ve
got some standard patterns I use for dealing with TRUE one-time run
recipes.

A concrete example was from my previous employer was something like this:

  • spin up node of role “fooapp” with additional recipe “fooapp::bootstrap”
  • at the end of the run, the “fooapp::bootstrap” recipe removes itself
    from the runlist for that node like so

ruby_block “remove_bootstrap_recipe” do
block do
Chef::Log.info(“Removing fooapp boostrap from runlist”)
node.run_list.remove(“recipe[fooapp::bootstrap]”) if
node.run_list.include?(“recipe[fooapp::bootstrap]”)
end
action :create
end

This works fine but now it was always cumbersome to tell folks
"initially the runlist is THIS but afterwards the runlist will be
THAT". I’ve been thinking about ways to make this more data-driven.
The thought I was having is relying on something that I previously
maligned - bootstrapping nodes not saving until the complete run is
done. I’m thinking now I can leverage that functionality to include a
::bootstrap recipe from any cookbook.

Can anyone see any downsides in this? The general idea is something
like (in fooapp::default):

   existence = search("node", "name:#{node.name}")

   # Do some stuff that should always run/check

   # At the end of fooapp::default
   include_recipe "fooapp::bootstrap" if existence.size == 0

Does anyone see a problem with this? Is there a convenience method
that can tell me if I’m bootstrapping that I’m just not aware of?

Again, the end goal is to NOT need to specify the commensurate
"bootstrap" recipe for a given cookbook on setup.

Right now I’m actually dealing with a private cloud setup where I
don’t have access to a userdata construct so I need to do some similar
to that, reload ohai and continue with the bootstrap. My concern is
accidentally calling node.save and having things blow up. I want this
to be as simple and durable as possible.

Thoughts?


#2

John, I probably don’t understand your needs in full, but this is what I do:

  1. knife bootstrap with a special bootstrap role
  2. immediately when done, set the node’s TRUE role(s)

In other words, use knife bootstrap for bootstrapping! I happen to use Trac’s Cloud plugin for this orchestration (I’m the author) which is just a simple wrapper of pychef and boto, but I’m there are many other easy ways of changing a node’s role and attributes.

  • Rob

On Mar 9, 2012, at 1:04 AM, John E. Vincent (lusis) wrote:

Hey all,

Came across a situation tonight that I wanted to get feedback on. I’ve
got some standard patterns I use for dealing with TRUE one-time run
recipes.

A concrete example was from my previous employer was something like this:

  • spin up node of role “fooapp” with additional recipe “fooapp::bootstrap”
  • at the end of the run, the “fooapp::bootstrap” recipe removes itself
    from the runlist for that node like so

ruby_block “remove_bootstrap_recipe” do
block do
Chef::Log.info(“Removing fooapp boostrap from runlist”)
node.run_list.remove(“recipe[fooapp::bootstrap]”) if
node.run_list.include?(“recipe[fooapp::bootstrap]”)
end
action :create
end

This works fine but now it was always cumbersome to tell folks
"initially the runlist is THIS but afterwards the runlist will be
THAT". I’ve been thinking about ways to make this more data-driven.
The thought I was having is relying on something that I previously
maligned - bootstrapping nodes not saving until the complete run is
done. I’m thinking now I can leverage that functionality to include a
::bootstrap recipe from any cookbook.

Can anyone see any downsides in this? The general idea is something
like (in fooapp::default):

  existence = search("node", "name:#{node.name}")

  # Do some stuff that should always run/check

  # At the end of fooapp::default
  include_recipe "fooapp::bootstrap" if existence.size == 0

Does anyone see a problem with this? Is there a convenience method
that can tell me if I’m bootstrapping that I’m just not aware of?

Again, the end goal is to NOT need to specify the commensurate
"bootstrap" recipe for a given cookbook on setup.

Right now I’m actually dealing with a private cloud setup where I
don’t have access to a userdata construct so I need to do some similar
to that, reload ohai and continue with the bootstrap. My concern is
accidentally calling node.save and having things blow up. I want this
to be as simple and durable as possible.

Thoughts?


#3

On Fri, Mar 9, 2012 at 1:04 AM, John E. Vincent (lusis)
lusis.org+chef-list@gmail.com wrote:

Came across a situation tonight that I wanted to get feedback on. I’ve
got some standard patterns I use for dealing with TRUE one-time run
recipes.

Thoughts?

Maybe you are making this too complicated and hard on yourself?

Why is the work done by the resources in the recipe not idempotent?
If that is complicated, why not wrap the whole recipe in a wrapper
that prevents it from happening again?

unless File.exists(’/opt/app/.done’)

do stuff

done with stuff

file "/opt/app/.done"
end

If you do a lot of on every system, maybe a ‘bootstrap’ role that is
in the beginning of a run list for all nodes that contains the
bootstrap recipes?

Bryan


#4

On Fri, Mar 9, 2012 at 12:55 PM, Bryan McLellan btm@loftninjas.org wrote:

On Fri, Mar 9, 2012 at 1:04 AM, John E. Vincent (lusis)
lusis.org+chef-list@gmail.com wrote:

Came across a situation tonight that I wanted to get feedback on. I’ve
got some standard patterns I use for dealing with TRUE one-time run
recipes.

Thoughts?

Maybe you are making this too complicated and hard on yourself?

Why is the work done by the resources in the recipe not idempotent?
If that is complicated, why not wrap the whole recipe in a wrapper
that prevents it from happening again?

unless File.exists(’/opt/app/.done’)

do stuff

done with stuff

file "/opt/app/.done"
end

If you do a lot of on every system, maybe a ‘bootstrap’ role that is
in the beginning of a run list for all nodes that contains the
bootstrap recipes?

Bryan

That’s pretty much exactly what I had to do.

As to the idempotent question, I’ll give a bit of background as to why
it needed to be done that way:

(please note this applied to my previous company)

We had our application code stored in S3. We did not use the
application cookbook because it didn’t fit our model.

So when we provisioned one of our components, the “bootstrap” process was:

  • Download tarball of latest released code from S3
  • extract tarball
  • Move some files around because the tarball didn’t actually reflect
    the final on-disk representation (long story as to why that wasn’t
    modifiable at the time)
  • run some commands that HAD to be run on the local system and
    couldn’t be packaged (think compiling of sorts)
  • Start jetty

Each and every one of those steps ONLY needed to be run to bootstrap a
new node of that type of application. Deploys to that app server from
that point on were done via jenkins. Having it run again meant wiping
out any newer app code we pushed.

The way I worked around this (though it was HIGHLY fragile and scared
me from day one was:

  • move onetime operations into a bootstrap recipe
  • move the steps of moving files around into a template shell script
    that got called
  • Shell script created a dotfile akin to what you describe that I
    could use in a not_if guard
  • use notifications to trigger the compilation-alike behaviors
  • start the server
  • remove the bootstrap recipe from the runlist

Again, while it was fragile it worked 100% of the time. The problem
was around training and teaching others. It would have been nice if I
could simply said ‘bootstrap the node with this role’ but instead it
was ‘bootstrap the node with this role and this recipe’. I’m a big fan
of composition in roles and I personally prefer a server to have a
single role (even if it’s made up of 20 other roles). Yes it’s a bit
of extra hierarchy but it’s easier to for people to grok (I think)
instead of a top-level run_list that’s got roles, recipes and more
roles.

As I said, I’m not at that company anymore but I was running into a
similar need at enStratus. AWS does a lot of things for you that you
take for granted. I needed to do some cloud-init type work as part of
the initial bootstrap of our private cloud nodes. Now I just (as of
about an hour ago) discovered that Cloudstack indeed has meta-data and
user-data support so my original needs are not the same.

The question still stands (and I think there’s a valid use case for
it). A good example has already come up several times - changing the
hostname of a system. You only need to do it once, reload ohai, and
continue on with the run. It’s not something you need to do more than
once and it’s not something you WANT to do every single run.

Hope that clears up the issue/question?


#5

Bryan I like your method of either making the various tasks
idempotent, or to just put them all in an unless File.exists block…
maybe even use /etc/chef/.[-].bootstrapped so
they’re all in the same place.

I used to need to manage the network and hostname fields, but for now
I have knife vsphere handling that… but if I run into something else
that’s only a run-once, and idempotent is hard, I’ll use this method.

Thanks!

On Fri, Mar 9, 2012 at 13:41, John E. Vincent (lusis)
lusis.org+chef-list@gmail.com wrote:

On Fri, Mar 9, 2012 at 12:55 PM, Bryan McLellan btm@loftninjas.org wrote:

On Fri, Mar 9, 2012 at 1:04 AM, John E. Vincent (lusis)
lusis.org+chef-list@gmail.com wrote:

Came across a situation tonight that I wanted to get feedback on. I’ve
got some standard patterns I use for dealing with TRUE one-time run
recipes.

Thoughts?

Maybe you are making this too complicated and hard on yourself?

Why is the work done by the resources in the recipe not idempotent?
If that is complicated, why not wrap the whole recipe in a wrapper
that prevents it from happening again?

unless File.exists(’/opt/app/.done’)

do stuff

done with stuff

file "/opt/app/.done"
end

If you do a lot of on every system, maybe a ‘bootstrap’ role that is
in the beginning of a run list for all nodes that contains the
bootstrap recipes?

Bryan


#6

On Mar 9, 2012, at 12:41 PM, John E. Vincent (lusis) wrote:

The question still stands (and I think there’s a valid use case for
it). A good example has already come up several times - changing the
hostname of a system. You only need to do it once, reload ohai, and
continue on with the run. It’s not something you need to do more than
once and it’s not something you WANT to do every single run.

With hostname, you can keep that recipe in the run_list. Check to see that the hostname is of the format you want with an only_if/not_if guard, but the rest can stay there without any risk of harming anything.

We do this today, and it is precisely for the hostname issue.


Brad Knowles bknowles@ihiji.com
SAGE Level IV, Chef Level 0.0.1