Ideas for using Chef across VPN

Chef Infra (archive)
1

Hello chef-list,

I administer several geographically separated workstations which are
hidden behind NATs. Currently we’re working through OpenVPN with a
server on a VPS which is used as web hosting and mail server, which I
want to configure with Chef too.

I have some ideas about using chef-solo to bootstrap chef-client and
OpenVPN client on the workstations but I’m interesting in comments from
more experienced chef users.

Is it a feasible idea? How best to manage openvpn keys?

Another question what bothers me, is where to place the chef-server. We
don’t have a separate host for it and seems to me it’s better not to
combine server with a client doing something useful. Would it be a good
solution to have it as a guest os in VirtualBox on one of workstations?
Is it required that chef-server is online 24/7 or it can be activated
only then something in configuration changed?

As a newbie I have many more questions, but these are most urgent.
Thanks for attention.

Best wishes,
Dmitry

2

It seems a good point to start from.

Its certainly possible to run a chef-server on some workstation which
has good connectivity and is always on. It is important that the
computer has enough spare memory for chef-server stack. I figure thats
about 1-2 ruby processes (for merb), plus java and solr. Something
like 200-400Mb RAM. Hosting inside a Virtual Machine will add a bit
more RAM to those basic requirements.

Of course the search index will be offline whenever chef-server is
offline. If the search index matters, it depends for what reasons the
search index is being used by the client nodes. That can come later
on, and it doesnt really matter to begin with.

For the VPN, that seems better and more flexible than SSH tunnel. Some
VPN providers like LogMeIn Hamachi are free to try out for
non-commercial use, and will provide you a virtual VPN gateway server
for your VPN hub. You are going to need to write some cookbook to
setup the VPN software locally on your nodes. It should be well worth
the time / effort. Such a recipe can be included for the chef-solo
bootstrap run.

Be sure to go to chef irc channels for help / advice.

On Sun, May 16, 2010 at 12:25 PM, Dmitry V'yal akamaus@gmail.com wrote:

Hello chef-list,

I administer several geographically separated workstations which are hidden
behind NATs. Currently we're working through OpenVPN with a server on a VPS
which is used as web hosting and mail server, which I want to configure with
Chef too.

I have some ideas about using chef-solo to bootstrap chef-client and OpenVPN
client on the workstations but I'm interesting in comments from more
experienced chef users.

Is it a feasible idea? How best to manage openvpn keys?

Another question what bothers me, is where to place the chef-server. We
don't have a separate host for it and seems to me it's better not to combine
server with a client doing something useful. Would it be a good solution to
have it as a guest os in VirtualBox on one of workstations? Is it required
that chef-server is online 24/7 or it can be activated only then something
in configuration changed?

As a newbie I have many more questions, but these are most urgent. Thanks
for attention.

Best wishes,
Dmitry

3

On 05/16/2010 05:33 PM, dreamcat four wrote:

It seems a good point to start from.

Its certainly possible to run a chef-server on some workstation which
has good connectivity and is always on. It is important that the
computer has enough spare memory for chef-server stack. I figure thats
about 1-2 ruby processes (for merb), plus java and solr. Something
like 200-400Mb RAM. Hosting inside a Virtual Machine will add a bit
more RAM to those basic requirements.

Of course the search index will be offline whenever chef-server is
offline. If the search index matters, it depends for what reasons the
search index is being used by the client nodes. That can come later
on, and it doesnt really matter to begin with.

Can you please give an example of what it may be useful for? I'm still
struggling with all the concepts and don't have a finished picture in my
head.

For the VPN, that seems better and more flexible than SSH tunnel. Some
VPN providers like LogMeIn Hamachi are free to try out for
non-commercial use, and will provide you a virtual VPN gateway server
for your VPN hub. You are going to need to write some cookbook to
setup the VPN software locally on your nodes. It should be well worth
the time / effort. Such a recipe can be included for the chef-solo
bootstrap run.

My plan was to bootstrap openvpn server along with the chef-client on
the VPS and openvpn clients on the other hosts. After that I would be
able to do the remaining configuration through chef-server.

Be sure to go to chef irc channels for help / advice.

I'm already there. It's a very nice place with many helpful people I
must say. The only problem is what probably most of them are inactive
around midday gmt+4 then I'm cooking )

4

On Sun, May 16, 2010 at 5:36 PM, Dmitry V'yal akamaus@gmail.com wrote:

On 05/16/2010 05:33 PM, dreamcat four wrote:

It seems a good point to start from.

Its certainly possible to run a chef-server on some workstation which
has good connectivity and is always on. It is important that the
computer has enough spare memory for chef-server stack. I figure thats
about 1-2 ruby processes (for merb), plus java and solr. Something
like 200-400Mb RAM. Hosting inside a Virtual Machine will add a bit
more RAM to those basic requirements.

Of course the search index will be offline whenever chef-server is
offline. If the search index matters, it depends for what reasons the
search index is being used by the client nodes. That can come later
on, and it doesnt really matter to begin with.

Can you please give an example of what it may be useful for? I'm still
struggling with all the concepts and don't have a finished picture in my
head.

Search index will hold dynamic information about your nodes. It is
useful for allowing nodes to sense each other and auto-configure.

For example a load balancer node might need to know the list of other
nodes to which it is sending requests to. A search query in a chef
recipe for the load balancer role can update the configuration file of
the load balancer software. The chef index / search query will return
in ruby the list of active nodes which are up and active, and
configured to the web server role.

When one of the nodes with the web server role becomes offline, it is
removed from the index on the chef server. Then the next time the load
balancer updates, it is removed from the load balancer. Same for
adding new nodes.

Adam has a screencast showing this example. Its somewhere in the
archives for the Opscode blog.

For the VPN, that seems better and more flexible than SSH tunnel. Some
VPN providers like LogMeIn Hamachi are free to try out for
non-commercial use, and will provide you a virtual VPN gateway server
for your VPN hub. You are going to need to write some cookbook to
setup the VPN software locally on your nodes. It should be well worth
the time / effort. Such a recipe can be included for the chef-solo
bootstrap run.

My plan was to bootstrap openvpn server along with the chef-client on the
VPS and openvpn clients on the other hosts. After that I would be able to do
the remaining configuration through chef-server.

Yeah, most people use OpenVPN. Although today I found 2 alternatives
to OpenVPN. These are perhaps useful if OpenVPN does not work through
your Firewall.

http://www.neorouter.com
http://www.neorouter.com/wiki/index.php/NeoRouterWiki:ServerSetup#Install_NeoRouter_server_for_Linux

http://files.hamachi.cc/linux/
http://networkshamachi.com/default.asp?page=mac-os-x-hamachi

Be sure to go to chef irc channels for help / advice.

I'm already there. It's a very nice place with many helpful people I must
say. The only problem is what probably most of them are inactive around
midday gmt+4 then I'm cooking )

5

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello!

On May 16, 2010, at 11:58 AM, dreamcat four wrote:

When one of the nodes with the web server role becomes offline, it is
removed from the index on the chef server. Then the next time the load
balancer updates, it is removed from the load balancer. Same for
adding new nodes.

Do note that Chef doesn't automatically remove nodes if they're
offline. You'll need some mechanism to trigger a node deletion,
whether that is manually running:

knife node delete NODE

Or if its handled automatically somewhere.

Opscode, Inc
Joshua Timberman, Senior Solutions Engineer
C: 720.334.RUBY E: joshua@opscode.com

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (Darwin)

iEYEARECAAYFAkvxSsAACgkQO97WSdVpzT1KuwCdH6AjZeJSIVy8OWqyYE0z6RXQ
xYcAn3IEHhu/OGOr1Alf7Y0OfS8SM9iH
=qOYj
-----END PGP SIGNATURE-----

6

On Sun, May 16, 2010 at 4:25 AM, Dmitry V'yal akamaus@gmail.com wrote:

I have some ideas about using chef-solo to bootstrap chef-client and OpenVPN
client on the workstations but I'm interesting in comments from more
experienced chef users.

Is it a feasible idea? How best to manage openvpn keys?

I've done this with EC2 nodes that needed to communicate with other
servers in a physical datacenter.

I manually created the keypairs and added them to a central chef
repository. When a new node is built, a number of prerequisite tasks
are completed related to the EC2 instance, then the openvpn cookbook
is copied out to the node. Chef-solo runs, and brings up a vpn
connection. Then chef-client runs to register with a chef-server on
the other side of the VPN link.

It took quite a bit of tinkering to get this functional, mostly
because of OpenVPN being quirky, but once I did its easy to appreciate
configuration management when it can bring up an openvpn link on a
fresh EC2 node in an instant.

Bryan

7

Hello Bryan and others,

On Sun, May 16, 2010 at 4:25 AM, Dmitry V'yalakamaus@gmail.com wrote:

I have some ideas about using chef-solo to bootstrap chef-client and OpenVPN
client on the workstations but I'm interesting in comments from more
experienced chef users.

Is it a feasible idea? How best to manage openvpn keys?

I've done this with EC2 nodes that needed to communicate with other
servers in a physical datacenter.

I manually created the keypairs and added them to a central chef
repository. When a new node is built, a number of prerequisite tasks
are completed related to the EC2 instance, then the openvpn cookbook
is copied out to the node. Chef-solo runs, and brings up a vpn
connection. Then chef-client runs to register with a chef-server on
the other side of the VPN link.

Looks like my task is almost the same. The biggest problem for me is
distributing the generated keys. You said you stored them in central
chef repository, how did a freshly created node get it's key from there?

Currently I'm writing a receipt which unpacks the key from zip archive
generated by rake client task from openvpn cookbook from opscode. I
'm almost done, but this scheme looks quite inelegant for me.

It took quite a bit of tinkering to get this functional, mostly
because of OpenVPN being quirky, but once I did its easy to appreciate
configuration management when it can bring up an openvpn link on a
fresh EC2 node in an instant.

would you mind sharing this?

Best wishes,
Dmitry