Chef with Docker and Chroot

Chef Infra (archive)
1

Hi,

While tinkering today with Chef Solo and Docker, there’s a thought I really
like to get others feedback on.

Of the many facets of using both Chef and Docker there’s one I’d like to
focus on in this thread.

Let’s take for example a web server like Apache or Nginx.

Option A)
Use a well supported community chef cookbook to install and configure
Apache/Nginx

Option B)
Use the chef docker cookbook to pull down and install an apache/nginxcontainer
and install a start up script for the container.

So of all the pro’s and con’s to these two installation options the one I’d
big advantage I get
using Option B instead of Option A is my web server is “chrooted” out of
the box.

So if there’s some exploit for Apache/Nginx that comes out (and my web
server has unrestricted access on the web)
I’ll get a good security buffer out of the box - the Docker isolated
container (chroot).

In a previous job, long before Chef existed, I used to to manual apache
installs and chroot them.

Recently I’ve stopped doing that by default, however; with the docker
install method I’d get that feature back again.

Any thoughts on this? Agree? Disagree? Am I missing something or thinking
about this incorrectly?

Thanks!

2

If you take the chef root, you can create variations of a container from a
single base container, dynamically. Like containers in dev and prod might
run the same app, but with different configs or with different versions.
With docker thats not straight forward. Also, its difficult to reuse
dockerfiles, thus their composability is limited.

but all these things can be nulled depending upon your workflow. Docker
provides an awesome workflow using union mounts and container (and few more
parts). If you can maintain a very low dev to production infrastructure
parity, if you can run immutable hosts (something that coreos offers),
and/or if you are on baremetal, docker can shine. But still its not clear
how the last mile will be completed (who will install docker? , if not
chef :smiley: ). Or even in that scale, compute wont be your only problem.. etc

Using raw chef. ruby lxc binding[1] is also present (GA will be coupled
with lxc 1.0 public release), which mean you can spawn raw containers using
ruby (hence chef cookbooks are also easy). Chef cookbooks for managing
containers are already there[2],[3]. Also, since ohai 7 release now its
possible to run chef client inside a container without installing ....
something like

inside host recipe

container 'apache' do
run_list 'recipe[apache]' # invokes chef client inside container (though
its installed only inside the host, not in the container)
end

this will be fairly easy to build (and im using this approach for the time
being).

In short, if you are exploiting the full docker workflow features, go for
it. If chrooting is only what you want (or even raw container), stick to
chef. Things on raw lxc and chef is only going to get awesome!

[1]GitHub - lxc/ruby-lxc: ruby bindings for liblxc
[2]GitHub - hw-cookbooks/lxc: Linux Containers via Chef
[3]GitHub - ranjib/lxc-cookbook: Chef cookbook for managing lxc

On Fri, Jan 24, 2014 at 1:38 AM, Rudi ooly.me@gmail.com wrote:

Hi,

While tinkering today with Chef Solo and Docker, there's a thought I
really like to get others feedback on.

Of the many facets of using both Chef and Docker there's one I'd like to
focus on in this thread.

Let's take for example a web server like Apache or Nginx.

Option A)
Use a well supported community chef cookbook to install and configure
Apache/Nginx

Option B)
Use the chef docker cookbook to pull down and install an apache/nginxcontainer
and install a start up script for the container.

So of all the pro's and con's to these two installation options the one
I'd big advantage I get
using Option B instead of Option A is my web server is "chrooted" out of
the box.

So if there's some exploit for Apache/Nginx that comes out (and my web
server has unrestricted access on the web)
I'll get a good security buffer out of the box - the Docker isolated
container (chroot).

In a previous job, long before Chef existed, I used to to manual apache
installs and chroot them.

Recently I've stopped doing that by default, however; with the docker
install method I'd get that feature back again.

Any thoughts on this? Agree? Disagree? Am I missing something or thinking
about this incorrectly?

Thanks!

3

Ranjib,

Very interesting, thanks for taking the time to tap all that out.

I've been following Docker for months but have only really jumped in this
past week or so.

Chef + lxc - I've not looked at at all to date. A quick google leads to
cookbooks for me to follow up on.

who will install docker? , if not chef

Yeah, exactly, this is what led me to this point.

Also, since ohai 7 release now its possible to run chef client inside a
container without installing

Wow .. I had no idea. I need to find out where I should follow to keep on
top of release details like this one.

In short, if you are exploiting the full docker workflow features, go for
it

Well, my workstation is Ubuntu, so the first thing I use Docker for is so I
don't need to fire up Virtual Box.

I can run MongoDB/Rethink instances and rig up clusters much easier.

I use a Ruby mine and chrome and databases so all that is less resource
hungry without Virtualbox running.

For things like web services I'm exploring mounting different config files
into the docker container (dev, prod etc).

However, I don't want to be reinventing wheels, the community cookbooks are
there and should be used I think.

Things on raw lxc and chef is only going to get awesome!

This is a great tip and something I've already put on my list to level up
on.

If chrooting is only what you want

I'm not sure that's all I want.

What I want is to be doing best practice, right now that's just trying to
figure out how docker might fit into Chef best practice.

Thanks again for your input.

On Fri, Jan 24, 2014 at 6:08 PM, Ranjib Dey dey.ranjib@gmail.com wrote:

If you take the chef root, you can create variations of a container from a
single base container, dynamically. Like containers in dev and prod might
run the same app, but with different configs or with different versions.
With docker thats not straight forward. Also, its difficult to reuse
dockerfiles, thus their composability is limited.

but all these things can be nulled depending upon your workflow. Docker
provides an awesome workflow using union mounts and container (and few more
parts). If you can maintain a very low dev to production infrastructure
parity, if you can run immutable hosts (something that coreos offers),
and/or if you are on baremetal, docker can shine. But still its not clear
how the last mile will be completed (who will install docker? , if not
chef :smiley: ). Or even in that scale, compute wont be your only problem.. etc

Using raw chef. ruby lxc binding[1] is also present (GA will be coupled
with lxc 1.0 public release), which mean you can spawn raw containers using
ruby (hence chef cookbooks are also easy). Chef cookbooks for managing
containers are already there[2],[3]. Also, since ohai 7 release now its
possible to run chef client inside a container without installing ....
something like

inside host recipe

container 'apache' do
run_list 'recipe[apache]' # invokes chef client inside container
(though its installed only inside the host, not in the container)
end

this will be fairly easy to build (and im using this approach for the time
being).

In short, if you are exploiting the full docker workflow features, go for
it. If chrooting is only what you want (or even raw container), stick to
chef. Things on raw lxc and chef is only going to get awesome!

[1]GitHub - lxc/ruby-lxc: ruby bindings for liblxc
[2]GitHub - hw-cookbooks/lxc: Linux Containers via Chef
[3]GitHub - ranjib/lxc-cookbook: Chef cookbook for managing lxc

On Fri, Jan 24, 2014 at 1:38 AM, Rudi ooly.me@gmail.com wrote:

Hi,

While tinkering today with Chef Solo and Docker, there's a thought I
really like to get others feedback on.

Of the many facets of using both Chef and Docker there's one I'd like to
focus on in this thread.

Let's take for example a web server like Apache or Nginx.

Option A)
Use a well supported community chef cookbook to install and configure
Apache/Nginx

Option B)
Use the chef docker cookbook to pull down and install an apache/nginxcontainer
and install a start up script for the container.

So of all the pro's and con's to these two installation options the one
I'd big advantage I get
using Option B instead of Option A is my web server is "chrooted" out of
the box.

So if there's some exploit for Apache/Nginx that comes out (and my web
server has unrestricted access on the web)
I'll get a good security buffer out of the box - the Docker isolated
container (chroot).

In a previous job, long before Chef existed, I used to to manual apache
installs and chroot them.

Recently I've stopped doing that by default, however; with the docker
install method I'd get that feature back again.

Any thoughts on this? Agree? Disagree? Am I missing something or thinking
about this incorrectly?

Thanks!

4

In my “playing” with chef and docker, I’ve been just treating docker as a
"package manager." I name/tag docker images like
<product/deployment>-. I also assume that the config variations
are controlled via environment variables to docker (ala 12-factor).
So, in chef, I docker pull the image and setup a runit service that wraps
docker run.
I don’t have generic nginx/apache images, rather I create images that
provide a “service” and name them as such. So, “thatsite-proxy” may be an
image based on a generic nginx image, but has a config for "thatsite-proxy"
and uses environment variables for variations. nginx is actually not a
great example as it’s mostly “code” I’m deploying - sinatra apps, go stuff,
node, etc - that are created by a build system that spits out docker images.

This is all just me playing in my spare time and probably falls apart in
the “real world.”

5

If vagrant is part of your toolchain you should consider vagrant-lxc as
well:

Vagrant + + Chef is a great combo, where X might be lxc, virtualbox,
aws, rackspace, etc...

HTH, Torben

On Fri, Jan 24, 2014 at 11:56 AM, Rudi ooly.me@gmail.com wrote:

Ranjib,

Very interesting, thanks for taking the time to tap all that out.

I've been following Docker for months but have only really jumped in this
past week or so.

Chef + lxc - I've not looked at at all to date. A quick google leads to
cookbooks for me to follow up on.

who will install docker? , if not chef

Yeah, exactly, this is what led me to this point.

Also, since ohai 7 release now its possible to run chef client inside a
container without installing

Wow .. I had no idea. I need to find out where I should follow to keep on
top of release details like this one.

In short, if you are exploiting the full docker workflow features, go
for it

Well, my workstation is Ubuntu, so the first thing I use Docker for is so
I don't need to fire up Virtual Box.

I can run MongoDB/Rethink instances and rig up clusters much easier.

I use a Ruby mine and chrome and databases so all that is less resource
hungry without Virtualbox running.

For things like web services I'm exploring mounting different config files
into the docker container (dev, prod etc).

However, I don't want to be reinventing wheels, the community cookbooks
are there and should be used I think.

Things on raw lxc and chef is only going to get awesome!

This is a great tip and something I've already put on my list to level up
on.

If chrooting is only what you want

I'm not sure that's all I want.

What I want is to be doing best practice, right now that's just trying to
figure out how docker might fit into Chef best practice.

Thanks again for your input.

On Fri, Jan 24, 2014 at 6:08 PM, Ranjib Dey dey.ranjib@gmail.com wrote:

If you take the chef root, you can create variations of a container from
a single base container, dynamically. Like containers in dev and prod might
run the same app, but with different configs or with different versions.
With docker thats not straight forward. Also, its difficult to reuse
dockerfiles, thus their composability is limited.

but all these things can be nulled depending upon your workflow. Docker
provides an awesome workflow using union mounts and container (and few more
parts). If you can maintain a very low dev to production infrastructure
parity, if you can run immutable hosts (something that coreos offers),
and/or if you are on baremetal, docker can shine. But still its not clear
how the last mile will be completed (who will install docker? , if not
chef :smiley: ). Or even in that scale, compute wont be your only problem.. etc

Using raw chef. ruby lxc binding[1] is also present (GA will be coupled
with lxc 1.0 public release), which mean you can spawn raw containers using
ruby (hence chef cookbooks are also easy). Chef cookbooks for managing
containers are already there[2],[3]. Also, since ohai 7 release now its
possible to run chef client inside a container without installing ....
something like

inside host recipe

container 'apache' do
run_list 'recipe[apache]' # invokes chef client inside container
(though its installed only inside the host, not in the container)
end

this will be fairly easy to build (and im using this approach for the
time being).

In short, if you are exploiting the full docker workflow features, go for
it. If chrooting is only what you want (or even raw container), stick to
chef. Things on raw lxc and chef is only going to get awesome!

[1]GitHub - lxc/ruby-lxc: ruby bindings for liblxc
[2]GitHub - hw-cookbooks/lxc: Linux Containers via Chef
[3]GitHub - ranjib/lxc-cookbook: Chef cookbook for managing lxc

On Fri, Jan 24, 2014 at 1:38 AM, Rudi ooly.me@gmail.com wrote:

Hi,

While tinkering today with Chef Solo and Docker, there's a thought I
really like to get others feedback on.

Of the many facets of using both Chef and Docker there's one I'd like to
focus on in this thread.

Let's take for example a web server like Apache or Nginx.

Option A)
Use a well supported community chef cookbook to install and configure
Apache/Nginx

Option B)
Use the chef docker cookbook to pull down and install an apache/nginxcontainer
and install a start up script for the container.

So of all the pro's and con's to these two installation options the one
I'd big advantage I get
using Option B instead of Option A is my web server is "chrooted" out
of the box.

So if there's some exploit for Apache/Nginx that comes out (and my web
server has unrestricted access on the web)
I'll get a good security buffer out of the box - the Docker isolated
container (chroot).

In a previous job, long before Chef existed, I used to to manual apache
installs and chroot them.

Recently I've stopped doing that by default, however; with the docker
install method I'd get that feature back again.

Any thoughts on this? Agree? Disagree? Am I missing something or
thinking about this incorrectly?

Thanks!