I run a smallish shop of about 50 Linux and OSX boxes, all of varying vintage. All are one-offs, and all are SPOFs. There are very few apps/configurations in common - MTA, SSH, some backup systems, etc.
I tried Puppet ca. 2008, and ran into 2 problems, which I’d like to avoid with Chef:
Where to start? I began learning by putting my .emacs and .bashrc everywhere. While that was cute, it quickly became more trouble than it was worth, because I didn’t automate anything of high risk, value or volume.
How do I get legacy systems into the fold? I have every version of Ubuntu and OSX since 2007, and therefore my Ruby will be archaic.
I don’t have the luxury of tearing down most systems to start anew, and we don’t (yet) have a good testing infrastructure or protocol.
FWIW, Vagrant was my gateway drug to Chef, so I have hope that I can at least use that to prototype/test services for cybernetic replacement.
What advice do you have for somebody trying to automate a shop like this?
On Thu, Sep 20, 2012 at 09:52:33PM -0400, Christopher DeMarco wrote:
Where to start? I began learning by putting my .emacs and .bashrc
everywhere. While that was cute, it quickly became more trouble than it was
worth, because I didn't automate anything of high risk, value or volume.
You start by picking one system, and pulling one config at a time until
everything you are about on that system is controlled by Chef. Then you add
more systems.
How do I get legacy systems into the fold? I have every version of
Ubuntu and OSX since 2007, and therefore my Ruby will be archaic.
Doesn't matter, you'll use the omnibus build which includes it's own Ruby.
And with 'whyrun', you can get a pretty good idea of what's going to happen.
Yup, it's hard to have reproducable systems until you have them in config
management, and it's hard to deploy config management without reproducable
systems... but you just have to start somewhere and keep chipping away at it
until you can easily rebuild every system you have exactly as it stands today,
ancient versions and all.
On Sep 20, 2012, at 10:10 PM, Phil Dibowitz phil@ipom.com wrote:
On Thu, Sep 20, 2012 at 09:52:33PM -0400, Christopher DeMarco wrote:
Where to start? I began learning by putting my .emacs and .bashrc
everywhere. While that was cute, it quickly became more trouble than it was
worth, because I didn't automate anything of high risk, value or volume.
You start by picking one system, and pulling one config at a time until
everything you are about on that system is controlled by Chef. Then you add
more systems.
"Bite the bullet" he sez. Humph. I was hoping to avoid hard work XD
Seriously though, I was just mulling over a different project and wondered if it'd be a good Trojan Horse:
I have to install a UPS agent on every system, with a hard deadline of a utility cut in mid-October. Last time I had a situation like this, I heavily hacked my backup agent's installer to detect/handle every variant in my mutant horde.
Any reason not to use that project as my Chef beachhead?
On Thu, Sep 20, 2012 at 10:19:09PM -0400, Christopher DeMarco wrote:
"Bite the bullet" he sez. Humph. I was hoping to avoid hard work XD
Seriously though, I was just mulling over a different project and wondered
if it'd be a good Trojan Horse:
I have to install a UPS agent on every system, with a hard deadline of a
utility cut in mid-October. Last time I had a situation like this, I
heavily hacked my backup agent's installer to detect/handle every variant in
my mutant horde.
Any reason not to use that project as my Chef beachhead?
On Thu, Sep 20, 2012 at 9:52 PM, Christopher DeMarco cdemarco@gmail.com wrote:
I run a smallish shop of about 50 Linux and OSX boxes, all of varying
vintage. All are one-offs, and all are SPOFs. There are very few
apps/configurations in common - MTA, SSH, some backup systems, etc.
I tried Puppet ca. 2008, and ran into 2 problems, which I'd like to avoid
with Chef:
Where to start? I began learning by putting my .emacs and .bashrc
everywhere. While that was cute, it quickly became more trouble than it was
worth, because I didn't automate anything of high risk, value or volume.
How do I get legacy systems into the fold? I have every version of
Ubuntu and OSX since 2007, and therefore my Ruby will be archaic.
I don't have the luxury of tearing down most systems to start anew, and we
don't (yet) have a good testing infrastructure or protocol.
FWIW, Vagrant was my gateway drug to Chef, so I have hope that I can at
least use that to prototype/test services for cybernetic replacement.
What advice do you have for somebody trying to automate a shop like this?
Thanks in advance, and I'll see you at LISA!
--cmd
That may seem like a joke but it's not. Converting legacy systems is a
sliding puzzle game and it's how I've handled it at the last N orgs.
Don't try to bring the systems into the fold except for the most basic
of things. The only thing you need is a free server somewhere. Just
one. If you want to keep your sanity, pick a system (preferably the
most simple) and rebuild it in Chef. Do this over and over and over
and over again on your test system until it works and comes out
exactly like it should. Don't stress about doing things the "right"
way. Hell, use a single cookbook entirely made up of execute resources
if that's what it takes.
Once you know that system is EXACTLY the same as the other (or that
the differences are not in any critical path, swap it out. After a
small validation period and burn in wipe the old one, pull in the next
complex system and do the same thing. Eventually you will have all of
your systems migrated to Chef. Start refactoring the cookbooks to use
more elegant resource types (i.e. switch from an execute resource
calling apt-get install to a package resource). Red - Green -
Refactor.
It's painful but you can probably start seeing results in a few weeks.
The latter systems will start to convert faster as you get more
comfortable with the process.
On Thu, Sep 20, 2012 at 09:52:33PM -0400, Christopher DeMarco wrote:
Where to start? I began learning by putting my .emacs and .bashrc
everywhere. While that was cute, it quickly became more trouble than it was
worth, because I didn't automate anything of high risk, value or volume.
You start by picking one system, and pulling one config at a time until
everything you are about on that system is controlled by Chef. Then you add
more systems.
I've typically found it works better to deploy configuration management
everywhere (doing nothing), and get it integrated on all new builds,
then start with the trivial and uniform things like ntp and dns and get
them working. Then figure out what the largest common "style" of app
deployment is and start to focus on more comprehensively doing full
stacks once you've got a solid base role across all the servers.
I would recommend that you not radically change things around, unless it
is done uniformly. I've done a post mortem on one failed attempt at
deploying configuration management across a large, heterogenous
environment that failed because the servers that were under config
management looked nothing like the other servers and after the CM
deployment failed after fully converting a few hundred servers it was
eventually ripped out completely after the employee left the company and
the project stalled and the servers that had been converted were the
'weird' ones that nobody knew how to properly manage.
That is one advantage of tackling things in vertical slices instead of
fully converting horizontal silos -- if you decide the implement some
new hotness in {ntp, dns, smtp, whatever} then you fully deploy it
across the board and get it actually finished before getting ADD and
moving on to something else.
Anyway, just getting time set right across the board can eliminate the
tickets, pages and outages caused by incorrectly set clocks.
On Thu, Sep 20, 2012 at 6:52 PM, Christopher DeMarco cdemarco@gmail.com wrote:
I run a smallish shop of about 50 Linux and OSX boxes, all of varying
vintage. All are one-offs, and all are SPOFs. There are very few
apps/configurations in common - MTA, SSH, some backup systems, etc.
I tried Puppet ca. 2008, and ran into 2 problems, which I'd like to avoid
with Chef:
Where to start? I began learning by putting my .emacs and .bashrc
everywhere. While that was cute, it quickly became more trouble than it was
worth, because I didn't automate anything of high risk, value or volume.
How do I get legacy systems into the fold? I have every version of
Ubuntu and OSX since 2007, and therefore my Ruby will be archaic.
You can't use the OS ruby dependably to run Chef. If there's an omnibus version
for your OS that's great, but building the Chef client and it's
dependencies from
source is reasonably straightforward. Build it in /opt/chef and follow
the omnibus file layout pattern.
The Chef server is a lot more complex and will be changing in the near
future. I would only
run that on a supported platform and definitely use "chef to install
chef-server". But given the
snowflake nature of your setup, you might be better off starting with
chef-solo.
Since it sounds like you have nothing now, anything in this direction
will be an improvement.
Just to put out an alternate perspective, I would start with something
simple ( like /etc/motd )
and deploy it as widely as possible. That will give you some hint of
the pitfalls involved. If you
focus on just one OS, you can get into some bad habits that will make
code reuse much harder.
It really depends on the time scale and goals. There's a lot to
running chef besides writing chef
cookbooks; you need to figure out a workflow and repository setup that
will work for your situation.