Any experiences building on top of Chef?


#1

Ohai,

I would be interested in hearing how people are building tools on top of
Chef. Basically I’m thinking an orchestration layer with simple tasks such
as “spin up a new qa environment with latest build”, “shutdown all dev
environments” etc.

We have all the low level components done with Chef. And while everything
works using e.g. knife commands, it would be nice to shield e.g. QA from
having to install chef, clone repo, remember long command lines etc.

So, are there any good tools for these scenarios? If you’ve built your own,
how did you approach it it? Using knife, the Chef gem or the Chef REST Api?

/Jeppe


#2

On Jun 11, 2013, at 6:32 AM, Jeppe Nejsum Madsen jeppe@ingolfs.dk wrote:

I would be interested in hearing how people are building tools on top of Chef. Basically I’m thinking an orchestration layer with simple tasks such as “spin up a new qa environment with latest build”, “shutdown all dev environments” etc.

There are plenty of places that have done this sort of thing using tools like RunDeck, ZooKeeper, Noah, etc… I’d be surprised if you can’t find some examples with some simple Google searches based on names like that combined with keywords like opscode and chef.

So, are there any good tools for these scenarios? If you’ve built your own, how did you approach it it? Using knife, the Chef gem or the Chef REST Api?

I think a lot of people tend to roll their own using simple scripting tools plus knife commands.


Brad Knowles brad@shub-internet.org
LinkedIn Profile: http://tinyurl.com/y8kpxu


#3

3 years before i had moved from puppet to chef specifically for this reason
(albeit there are other reasons, but this was the main one). Extending
chef is much easier. There are quiet a few projects which is built on top
of chef . Infochimp’s ironfan, Dell’s crowbar are examples .

Chef servers REST interface is pretty decent . Low level domain objects
related apis are also good, Though they are not perfectly idiomatic (like
Chef::Node.list returns a hash) , but there are wrappers (like ridley)
which provide a more idiomatic layer on top of chef. this means you can
build any webapp easily chef aware.
knife is another good component that lets you glue together existing
command line tools and build your own command line tools on top of it.
knife also has a rich set of cloud specific extension that you can reuse.

Over the last three years I have integrated chef with various webapps
(mostly rails, but java bindings are also straight forward, and you can see
examples on github, theres an android app also), as well as used the knife
interface for building command line tools for deployment strategies (like
blue/green, canary release) that can be used from workstations as well as
CI server.

So, in short depending upon your use case u’ll be using any or all of those
approaches, but all of them are widely explored and exploited :slight_smile: .
Specifically for reducing the setup efforts , you can use either bundler /
Gemfile and have example knife.rb on your wiki (this will be easier to do)
which your QA’s can refer. Or you can actually chefify the workstation. I
have done this for linux , but i know there are ample people who uses chef
to automate their own laptops…

best
ranjib

On Tue, Jun 11, 2013 at 5:32 AM, Jeppe Nejsum Madsen jeppe@ingolfs.dkwrote:

Ohai,

I would be interested in hearing how people are building tools on top of
Chef. Basically I’m thinking an orchestration layer with simple tasks such
as “spin up a new qa environment with latest build”, “shutdown all dev
environments” etc.

We have all the low level components done with Chef. And while everything
works using e.g. knife commands, it would be nice to shield e.g. QA from
having to install chef, clone repo, remember long command lines etc.

So, are there any good tools for these scenarios? If you’ve built your
own, how did you approach it it? Using knife, the Chef gem or the Chef REST
Api?

/Jeppe


#4

Hey - this is what we built at EA: http://vimeo.com/57861199 - the “tools
and dashes” part is the tail-end of the talk, around 20 minutes in if you
want to skip ahead. If you’ve any questions about any component I can
certainly answer in general terms.

S.

On Tue, Jun 11, 2013 at 1:32 PM, Jeppe Nejsum Madsen jeppe@ingolfs.dkwrote:

Ohai,

I would be interested in hearing how people are building tools on top of
Chef. Basically I’m thinking an orchestration layer with simple tasks such
as “spin up a new qa environment with latest build”, “shutdown all dev
environments” etc.

We have all the low level components done with Chef. And while everything
works using e.g. knife commands, it would be nice to shield e.g. QA from
having to install chef, clone repo, remember long command lines etc.

So, are there any good tools for these scenarios? If you’ve built your
own, how did you approach it it? Using knife, the Chef gem or the Chef REST
Api?

/Jeppe


#5

On Jun 11, 2013, at 5:32 AM, Jeppe Nejsum Madsen wrote:

Ohai,

I would be interested in hearing how people are building tools on top of Chef. Basically I’m thinking an orchestration layer with simple tasks such as “spin up a new qa environment with latest build”, “shutdown all dev environments” etc.

We have all the low level components done with Chef. And while everything works using e.g. knife commands, it would be nice to shield e.g. QA from having to install chef, clone repo, remember long command lines etc.

So, are there any good tools for these scenarios? If you’ve built your own, how did you approach it it? Using knife, the Chef gem or the Chef REST Api?

http://pychef.readthedocs.org/en/latest/fabric.html if you are a fan of Fabric (hint: you should be).

–Noah


#6

I recommend moving away from knife for orchestration. It is a great
command line tool, but is not intended for heavy automation.

Use the REST Api, either through Chef or Ridley.
If possible, make it so you don’t have to SSH in to bootstrap nodes (we are
on EC2 and use User Data w/ cloud-init).

Our automation is Python with Boto, and a dash of PyChef. Our primary
development language is python so it makes it easier for me to work with
other developers and forces cleaner interfaces between the infrastructure
and Chef.

In our case we use Boto to set up cloud formation templates instead of
configuring Zookeeper. Either is very workable once you don’t need to SSH
for bootstrapping.

On Tue, Jun 11, 2013 at 12:36 PM, Noah Kantrowitz noah@coderanger.netwrote:

On Jun 11, 2013, at 5:32 AM, Jeppe Nejsum Madsen wrote:

Ohai,

I would be interested in hearing how people are building tools on top of
Chef. Basically I’m thinking an orchestration layer with simple tasks such
as “spin up a new qa environment with latest build”, “shutdown all dev
environments” etc.

We have all the low level components done with Chef. And while
everything works using e.g. knife commands, it would be nice to shield e.g.
QA from having to install chef, clone repo, remember long command lines etc.

So, are there any good tools for these scenarios? If you’ve built your
own, how did you approach it it? Using knife, the Chef gem or the Chef REST
Api?

http://pychef.readthedocs.org/en/latest/fabric.html if you are a fan of
Fabric (hint: you should be).

–Noah


#7

knife is great for adding, modifying and deleting data in your chef server. But it’s way too general purpose for our node workflow. We want to make it idiot proof to ensure instance sizes, cloud-init user data, chef roles and hostnames all are consistent.

So we’re similar developing tooling and workflow around cloud-init bootstrapping. We’re implementing it as a knife plugin, but we’re trying to get it so we only use our plugin’s actions for actual node workflow. We’re planning on extracting this into a standalone set of thor scripts once we’ve got all the use cases locked down.

Andrew: I’d be very interested in seeing your specific implementation, especially how you’re bootstrapping secrets, and what metadata you keep where.

~j

On 2013-06-11, at 15:52, Andrew Gross andrew@yipit.com wrote:

I recommend moving away from knife for orchestration. It is a great command line tool, but is not intended for heavy automation.

Use the REST Api, either through Chef or Ridley.
If possible, make it so you don’t have to SSH in to bootstrap nodes (we are on EC2 and use User Data w/ cloud-init).

Our automation is Python with Boto, and a dash of PyChef. Our primary development language is python so it makes it easier for me to work with other developers and forces cleaner interfaces between the infrastructure and Chef.

In our case we use Boto to set up cloud formation templates instead of configuring Zookeeper. Either is very workable once you don’t need to SSH for bootstrapping.

On Tue, Jun 11, 2013 at 12:36 PM, Noah Kantrowitz noah@coderanger.net wrote:

On Jun 11, 2013, at 5:32 AM, Jeppe Nejsum Madsen wrote:

Ohai,

I would be interested in hearing how people are building tools on top of Chef. Basically I’m thinking an orchestration layer with simple tasks such as “spin up a new qa environment with latest build”, “shutdown all dev environments” etc.

We have all the low level components done with Chef. And while everything works using e.g. knife commands, it would be nice to shield e.g. QA from having to install chef, clone repo, remember long command lines etc.

So, are there any good tools for these scenarios? If you’ve built your own, how did you approach it it? Using knife, the Chef gem or the Chef REST Api?

http://pychef.readthedocs.org/en/latest/fabric.html if you are a fan of Fabric (hint: you should be).

–Noah


#8

Most of the secret bootstrapping is done via IAM roles in EC2. Instead of
having keys on the machine to authenticate your API requests, the machine
just has a role assigned to it so that any requests coming from that
machine have access per the role definition.

Netflix gave us the idea after we questioned how they had “No keys on the
machine, only in memory"
https://groups.google.com/forum/?fromgroups#!topicsearchin/asgardusers/authorname$3A
"Andrew$20Gross”/asgardusers/n-1V5mmkvW4

Most metadata is kept in databags, though we are still transitioning a lot
out of our monolithic app. At the moment we “launch” from a custom AMI we
created that has a base_client.pem key for initial registration, though we
may transition that to download S3 in the future.

We are currently moving away from individual instance creation to having
everything launched in a “Launch Configuration”, wrapped in an autoscaling
group. Cloud Formation forces a bunch of this, but even for services where
we do not use CFN we find it is a good practice to keep up standardization.
The main differentiator between Launch Configs is the user data, which
contains the lightweight role for the machine.

On Tue, Jun 11, 2013 at 7:40 PM, Joseph Holsten joseph@josephholsten.comwrote:

knife is great for adding, modifying and deleting data in your chef
server. But it’s way too general purpose for our node workflow. We want to
make it idiot proof to ensure instance sizes, cloud-init user data, chef
roles and hostnames all are consistent.

So we’re similar developing tooling and workflow around cloud-init
bootstrapping. We’re implementing it as a knife plugin, but we’re trying to
get it so we only use our plugin’s actions for actual node workflow. We’re
planning on extracting this into a standalone set of thor scripts once
we’ve got all the use cases locked down.

Andrew: I’d be very interested in seeing your specific implementation,
especially how you’re bootstrapping secrets, and what metadata you keep
where.

~j

On 2013-06-11, at 15:52, Andrew Gross andrew@yipit.com wrote:

I recommend moving away from knife for orchestration. It is a great
command line tool, but is not intended for heavy automation.

Use the REST Api, either through Chef or Ridley.
If possible, make it so you don’t have to SSH in to bootstrap nodes (we
are on EC2 and use User Data w/ cloud-init).

Our automation is Python with Boto, and a dash of PyChef. Our primary
development language is python so it makes it easier for me to work with
other developers and forces cleaner interfaces between the infrastructure
and Chef.

In our case we use Boto to set up cloud formation templates instead of
configuring Zookeeper. Either is very workable once you don’t need to SSH
for bootstrapping.

On Tue, Jun 11, 2013 at 12:36 PM, Noah Kantrowitz noah@coderanger.net
wrote:

On Jun 11, 2013, at 5:32 AM, Jeppe Nejsum Madsen wrote:

Ohai,

I would be interested in hearing how people are building tools on top
of Chef. Basically I’m thinking an orchestration layer with simple tasks
such as “spin up a new qa environment with latest build”, “shutdown all dev
environments” etc.

We have all the low level components done with Chef. And while
everything works using e.g. knife commands, it would be nice to shield e.g.
QA from having to install chef, clone repo, remember long command lines etc.

So, are there any good tools for these scenarios? If you’ve built your
own, how did you approach it it? Using knife, the Chef gem or the Chef REST
Api?

http://pychef.readthedocs.org/en/latest/fabric.html if you are a fan of
Fabric (hint: you should be).

–Noah


#9

if you want to interact with chef via command line, knife is your best bet,

knife has nothing to do with orchestration, its a chef aware , mixlib-cli
based component (which means its primary goal is to build command line
tools), which also provides following additional features:

  1. re-usability with other knife components. For example you can build a
    knife server provision -C cloud , and use knife-linode, knife-google or
    knife-ec2 based on the -C parameter
  2. it has a ui component which can do colored output, and provides a
    presenter. For example inside knife if you do ui.outpur(object) it will
    automatically print readable colored output by default, but can also print
    json if you supply -Fj
  3. Chef awareness… it has a baked in rest method (with and without auth),
    helpers for doing crud operations on chef’s domain objects (like apiclient,
    node, role etc).

that said, knife has some pain (primarily propagating from mixlib-cli), but
their workarounds are pretty well documented.
As per your first argument on orchestration, first of all each organization
has their own requirement of orchestration, with varying level of
complexity, and if they can be offloaded to command line mechanisms, i dont
see any pain in building knife plugins and using them for orchestration.
For example, if you decide to remove a node from a load balancer, do a
deployment and add it back again, and you want to trigger it from jenkins,
you can pretty much do this with knife plugins, and trigger them from
jenkins.

What we should be concerned about, is not to build thick plugins, or
putting way to much logic in them. If you can refactor them into separate
classes you get the ability to reuse them across knife and other
components. But this is true with any library from re-usability standpoint.
For example most of the knife-cloud plugins uses fog underneath and are
pretty thin.

on the contrary i’ll not try to go ployglot unless i have to. Re-usability
will be restricted to individual components once you go ployglot. Also you
have to maintain the language/framework specific idioms & toolchain. This
is fine if you have people who are good at multiple languages, but often
times its not the case. If i can use Boto then why not take the
func/fabric/salt route (which has a 0mq based orchestration built, and you
can fall back to ssh also because you have func/fabric ). They are pretty
decent, and also i get to reuse my code across these components. I would
only recommend going polyglot if the underlying tool or framework forces me
to do so. Like we built our jenkins - chef integration in java, because
jenkins java binding are first class.

At the end of the day I want to come of with a solution that comprises of
smallest , readable, testable code, that can be extended intuitively

comments ?