How to parameterize cluster setup


#1

Chef folks,

I am trying to figure out the best way to accomplish something in Chef, and am
hoping someone can give me some advice. I’m new to Chef, so forgive me if
this is naïve.

I need to set up machine clusters in which some machines will run Varnish,
others will run Tomcat, and still others will run Mongo. I need to set up
several clusters like this – several for development, several for QA, and at
least one for production.

For a development cluster, there’s not much point in using a lot of machines.
For QA and production clusters we will have several machines of each kind. I
need to be able to vary the numbers of each kind easily. For development, we
will use IP addresses as the lines of communication from Varnish to Tomcat,
and Tomcat to Mongo. For QA and production, these lines of communication will
be specified as URLs.

The kinds of clusters are quite similar, so I’d like to write one piece of
code to set up a generic cluster, and then define the cluster’s
characteristics in a JSON file. If I were simply writing this in ordinary
Ruby, I’d write the script, and then supply the JSON file that defines the
cluster as a command line argument.

Here is my question: How should I plan to do this in Chef? Seems like one way
would be to write a Ruby script that reads a JSON cluster definition file like
that described above, then calls knife several times for each node in the
cluster. I think that for each cluster node my script would do a knife
bootstrap, followed by a knife run_list add. I would still need to pass a bit
of configuration information to each node, namely what to use as the line of
communication to the downstream machines. I think I could write this
configuration information into a data bag having a standard name, and my nodes
could pick up the values from the data bag.

I guess this would work, but I have the feeling I’m not doing this in the
Chef way. Is there a better approach?

Thanks in advance,

Jeff


#2

Hi Jeff,

Although I haven’t used it personally, spiceweasel looks like it addresses
this need:

On Wed, Sep 28, 2011 at 3:05 PM, jeff.stroomer@disney.com wrote:

Chef folks,

I am trying to figure out the best way to accomplish something in Chef, and
am
hoping someone can give me some advice. I’m new to Chef, so forgive me if
this is naïve.

I need to set up machine clusters in which some machines will run Varnish,
others will run Tomcat, and still others will run Mongo. I need to set up
several clusters like this – several for development, several for QA, and
at
least one for production.

For a development cluster, there’s not much point in using a lot of
machines.
For QA and production clusters we will have several machines of each kind.
I
need to be able to vary the numbers of each kind easily. For development,
we
will use IP addresses as the lines of communication from Varnish to
Tomcat,
and Tomcat to Mongo. For QA and production, these lines of communication
will
be specified as URLs.

The kinds of clusters are quite similar, so I’d like to write one piece of
code to set up a generic cluster, and then define the cluster’s
characteristics in a JSON file. If I were simply writing this in
ordinary
Ruby, I’d write the script, and then supply the JSON file that defines the
cluster as a command line argument.

Here is my question: How should I plan to do this in Chef? Seems like one
way
would be to write a Ruby script that reads a JSON cluster definition file
like
that described above, then calls knife several times for each node in the
cluster. I think that for each cluster node my script would do a knife
bootstrap, followed by a knife run_list add. I would still need to pass a
bit
of configuration information to each node, namely what to use as the line
of
communication to the downstream machines. I think I could write this
configuration information into a data bag having a standard name, and my
nodes
could pick up the values from the data bag.

I guess this would work, but I have the feeling I’m not doing this in the
Chef way. Is there a better approach?

Thanks in advance,

Jeff


#3

On Sep 28, 2011, at 6:13 PM, Three Tee wrote:

Although I haven’t used it personally, spiceweasel looks like it addresses this need:

https://github.com/mattray/spiceweasel

I use spiceweasel on a daily basis (thanks, Matt!!! ;-). Spiceweasel will tell you what chef commands you need to run in order to re-create your infrastructure according to the infrastructure.yml file that you build based on the kind of infrastructure that you want to create.

However, that doesn’t necessarily help you orchestrate things in the correct order – you still need to take some care with ordering. And ultimately, you still have to describe everything that you’re going to be doing, and put that into the infrastructure.yml file.


Brad Knowles bknowles@ihiji.com
SAGE Level IV, Chef Level 0.0.1