Chef Vault Writeup

Chef Infra (archive)
1

Ohai Chefs,

I wrote up a blog post yesterday about my experience with Nordstrom’s
Chef Vault.

TL;DR - Chef-vault is awesome, and I think it is the best way to use
encrypted data bags w/ a Chef Server. Here’s how to do a few common
tasks with it.

http://jtimberman.housepub.org/blog/2013/09/10/managing-secrets-with-chef-vault/

Enjoy!
Joshua

2

Grady, we can use the office # for Opscode: 206-508-4799. Thanks.

On Wed, Sep 11, 2013 at 8:09 AM, Joshua Timberman joshua@opscode.comwrote:

Ohai Chefs,

I wrote up a blog post yesterday about my experience with Nordstrom's
Chef Vault.

TL;DR - Chef-vault is awesome, and I think it is the best way to use
encrypted data bags w/ a Chef Server. Here's how to do a few common
tasks with it.

http://jtimberman.housepub.org/blog/2013/09/10/managing-secrets-with-chef-vault/

Enjoy!
Joshua

3

++ for Vault!

-pete

On Sep 11, 2013, at 8:09 AM, Joshua Timberman joshua@opscode.com wrote:

Ohai Chefs,

I wrote up a blog post yesterday about my experience with Nordstrom's
Chef Vault.

TL;DR - Chef-vault is awesome, and I think it is the best way to use
encrypted data bags w/ a Chef Server. Here's how to do a few common
tasks with it.

http://jtimberman.housepub.org/blog/2013/09/10/managing-secrets-with-chef-vault/

Enjoy!
Joshua

4

Hi Joshua -

This is really cool. Thanks for sharing this discovery.

Jordan

On Wed, Sep 11, 2013 at 8:09 AM, Joshua Timberman joshua@opscode.comwrote:

Ohai Chefs,

I wrote up a blog post yesterday about my experience with Nordstrom's
Chef Vault.

TL;DR - Chef-vault is awesome, and I think it is the best way to use
encrypted data bags w/ a Chef Server. Here's how to do a few common
tasks with it.

http://jtimberman.housepub.org/blog/2013/09/10/managing-secrets-with-chef-vault/

Enjoy!
Joshua

5

I appreciate the concept behind Chef Vault and I think a vault like storage
for secrets like certificates and passwords is a much needed addition to
Chef infrastructure.

When I was evaluating Vault recently, I had concerns about how much real
security it provides. In reading the implementation, it appears that this
is only security by obfuscation and not truly more secure than regular
encrypted data bags as you still must use the encrypted_data_bag_secret to
unlock the key for the node you're operating on... you still have access to
all of the secrets for all of the nodes, just not in a direct way. If
someone were to compromise a node and retrieve the
encrypted_data_bag_secret, you have just added 1 hoop to jump through for
the attacker rather than adding more security. Maybe that'll save you, but
this depends on the attacker not understanding how Vault works... not
something I'd like to depend on.

Have you found this to be the case? My evaluation was cursory and probably
not fully informed.

I think that, for something like Vault to work, I would need a server side
component sitting in front of the encrypted data bags to handle access
rights.

I really want something like Vault, but I want to provide a personal secret
key to unlock the vault at run time so only the secrets I have access to
get unlocked rather than giving everyone access to everything, having an
additional layer keyed off node names as Vault does.

The solution, perhaps, would be that if I have access to perform a deploy,
I provide my key and the deploy uses my access rights temporarily (holding
my secret in memory during the chef run) to unlock the vault of secrets.
Think torpedo launch keys: I put the key in and kick off the automated
systems to fire the torpedo... or I put my private key in and kick off the
automated systems to deploy my software.

For those running chef in daemon mode, perhaps when the chef-client daemon
starts up, it can accept a key to hold in memory, which would be better
than storing it in a file, though still perhaps not 100% secure. There are
more concerns to figure out for daemon mode, hence this is all still
theoretical.

Having to leave the key on the node (as we do currently) means that once a
single node is compromised, the entirety of things encrypted with
/etc/chef/encrypted_data_bag_secret are easily compromised.

Currently, encrypted data bags only give security if your chef server
itself is compromised but they make people feel like they are being more
secure by using them... the general attack vector, however, is through
nodes exposed to the public internet that have flawed applications running,
not through the Chef server which is probably internal network access only.
All of these nodes contain the encrypted_data_bag_secret.

As a result: encrypted data bags are not often any more secure than regular
data bags for the general case and may inadvertently harm security due to
the aforementioned feeling of security that encrypting data bags gives and
so some active secret storage system would be very valuable... just
something more substantial than Vault.

Thoughts?

Josiah

On Thu, Sep 12, 2013 at 1:13 PM, Jordan Dea-Mattson <
jdm-lists@dea-mattson.com> wrote:

Hi Joshua -

This is really cool. Thanks for sharing this discovery.

Jordan

On Wed, Sep 11, 2013 at 8:09 AM, Joshua Timberman joshua@opscode.comwrote:

Ohai Chefs,

I wrote up a blog post yesterday about my experience with Nordstrom's
Chef Vault.

TL;DR - Chef-vault is awesome, and I think it is the best way to use
encrypted data bags w/ a Chef Server. Here's how to do a few common
tasks with it.

http://jtimberman.housepub.org/blog/2013/09/10/managing-secrets-with-chef-vault/

Enjoy!
Joshua

6

The main benefit I see to encrypted data bags is that you can safely
check them in to source control, and only the people who have the keys
can read them.

What we're thinking of doing (since we're mostly using chef in AWS) is
putting the secrets in a S3 bucket and using IAM roles to provide access
to specific secrets (we generally have a different data bag secret for
every Chef role that has secrets) within the bucket. Additionally, since
the role credentials are exposed through the metadata service, and
therefore accessible to any user with a shell account, we're going to
add an iptables rule to the OUTPUT chain to only allow access to the
metadata service for a specific local group. The data bag secrets would
then be retrieved on demand during a chef run. All of this would
actually be wrapped up in a library, of course.

But it still feels a bit like theatre, since any services that need the
encrypted data are going to be reading it out of a file anyway, so we're
back to protecting secrets with "only" filesystem protections. The only
thing our solution really solves is automating the data bag secret
distribution in as secure a manner (that I can think of:) as reasonably
possible.

Greg

On 09/12/13 22:13, Josiah Kiehl wrote:

I appreciate the concept behind Chef Vault and I think a vault like
storage for secrets like certificates and passwords is a much needed
addition to Chef infrastructure.

When I was evaluating Vault recently, I had concerns about how much
real security it provides. In reading the implementation, it appears
that this is only security by obfuscation and not truly more secure
than regular encrypted data bags as you still must use the
encrypted_data_bag_secret to unlock the key for the node you're
operating on... you still have access to all of the secrets for all of
the nodes, just not in a direct way. If someone were to compromise a
node and retrieve the encrypted_data_bag_secret, you have just added 1
hoop to jump through for the attacker rather than adding more
security. Maybe that'll save you, but this depends on the attacker not
understanding how Vault works... not something I'd like to depend on.

Have you found this to be the case? My evaluation was cursory and
probably not fully informed.

I think that, for something like Vault to work, I would need a server
side component sitting in front of the encrypted data bags to handle
access rights.

I really want something like Vault, but I want to provide a personal
secret key to unlock the vault at run time so only the secrets I have
access to get unlocked rather than giving everyone access to
everything, having an additional layer keyed off node names as Vault
does.

The solution, perhaps, would be that if I have access to perform a
deploy, I provide my key and the deploy uses my access rights
temporarily (holding my secret in memory during the chef run) to
unlock the vault of secrets. Think torpedo launch keys: I put the key
in and kick off the automated systems to fire the torpedo... or I put
my private key in and kick off the automated systems to deploy my
software.

For those running chef in daemon mode, perhaps when the chef-client
daemon starts up, it can accept a key to hold in memory, which would
be better than storing it in a file, though still perhaps not 100%
secure. There are more concerns to figure out for daemon mode, hence
this is all still theoretical.

Having to leave the key on the node (as we do currently) means that
once a single node is compromised, the entirety of things encrypted
with /etc/chef/encrypted_data_bag_secret are easily compromised.

Currently, encrypted data bags only give security if your chef server
itself is compromised but they make people feel like they are being
more secure by using them... the general attack vector, however, is
through nodes exposed to the public internet that have flawed
applications running, not through the Chef server which is probably
internal network access only. All of these nodes contain the
encrypted_data_bag_secret.

As a result: encrypted data bags are not often any more secure than
regular data bags for the general case and may inadvertently harm
security due to the aforementioned feeling of security that encrypting
data bags gives and so some active secret storage system would be very
valuable... just something more substantial than Vault.

Thoughts?

Josiah

On Thu, Sep 12, 2013 at 1:13 PM, Jordan Dea-Mattson
<jdm-lists@dea-mattson.com mailto:jdm-lists@dea-mattson.com> wrote:

Hi Joshua -

This is really cool. Thanks for sharing this discovery.

Jordan


On Wed, Sep 11, 2013 at 8:09 AM, Joshua Timberman
<joshua@opscode.com <mailto:joshua@opscode.com>> wrote:

    Ohai Chefs,

    I wrote up a blog post yesterday about my experience with
    Nordstrom's
    Chef Vault.

    TL;DR - Chef-vault is awesome, and I think it is the best way
    to use
    encrypted data bags w/ a Chef Server. Here's how to do a few
    common
    tasks with it.

    http://jtimberman.housepub.org/blog/2013/09/10/managing-secrets-with-chef-vault/

    Enjoy!
    Joshua
7

Hey there,

A few comments on Chef Vault, inline...

bluepojo@gmail.com writes:

When I was evaluating Vault recently, I had concerns about how much real
security it provides. In reading the implementation, it appears that this
is only security by obfuscation and not truly more secure than regular
encrypted data bags as you still must use the encrypted_data_bag_secret to
unlock the key for the node you're operating on... you still have access to
all of the secrets for all of the nodes, just not in a direct way.

Chef Vault helps you to solve the key distribution problem by taking
advantage of the RSA key pair assigned to each user and client on a Chef
Server. Encrypted data bags are used to implement it.

Here's (my understanding of) what happens when you create a Chef Vault
item (encrypt a data bag item using Chef Vault):

  1. You tell Chef Vault:

    • the name of a vault (maps to a data bag)
    • the name of an item (maps to a data bag item)
    • secret item data
    • the clients and users that should be able to access the secret

  2. Chef Vault generates a random shared key and uses the encrypted data
    bag support in Chef to encrypt the item data with this key.

  3. Chef Vault uses the Chef API to obtain the public RSA keys of the
    users and clients that should have access to item. It then encrypts
    the shared key generated in #1 for each of the public keys. It stores
    the user/client name to encrypted shared key in an $ITEM_keys data
    bag item in the $VAULT data bag.

Here's what I think this gives you:

  1. Compared to vanilla encrypted data bags, you don't have to arrange
    for a shared seceret to be distributed to clients (nodes) and users
    in your infrastructure.

  2. If an attacker gains root access to a node, the attacker can decrypt
    items that have been encrypted for that nodes client public
    key. So an attacker would not have access to all of the secrets for
    all of the nodes.

And some things it does not give you:

  1. Nodes store the client private key on disk. Using this key to decrypt
    secrets means compromise of the node will compromise any secrets
    which can be accessed via that key. So no torpedo launch key
    feature. In most cases, the secret data is used on a node in such a
    way that root access to the node would mean compromise of that secret
    regardless of how it was originally obtained. Avoiding that is hard.

  2. If a node is compromised, you will need to do more than remove the
    vault entry that encrypts the shared secret for the compromised key
    pair. In fact, if an attacker is able to obtain old copies of the
    encrypted data bag, changing the shared secret is not enough. You
    will need to change your underlying secret data -- e.g. if the item
    stored db passwords, you'll need to change the db password.

Having to leave the key on the node (as we do currently) means that once a
single node is compromised, the entirety of things encrypted with
/etc/chef/encrypted_data_bag_secret are easily compromised.

Encrypted data bags in general do not have to use that file and
different items can use different shared secrets. For chef vault, only
those items that the node had access to will be compromised.

Currently, encrypted data bags only give security if your chef server
itself is compromised but they make people feel like they are being more
secure by using them... the general attack vector, however, is through
nodes exposed to the public internet that have flawed applications running,
not through the Chef server which is probably internal network access only.
All of these nodes contain the encrypted_data_bag_secret.

As a result: encrypted data bags are not often any more secure than regular
data bags for the general case and may inadvertently harm security due to
the aforementioned feeling of security that encrypting data bags gives and
so some active secret storage system would be very valuable... just
something more substantial than Vault.

I see where you're coming from, but I disagree a bit. Protecting secrets
stored in the Chef Server has value. Even without something like chef
vault, it is possible to use different secrets for different items and
thus limit the impact of an attack.

  • seth

--
Seth Falcon | Development Lead | Opscode | @sfalcon

8

This discussion focuses mostly on distributing secrets to nodes; chef-vault
also allows to share secrets between users, encrypted by their
private Chef API key. This can be quite convenient means to store secrets
such as access keys to other APIs, confidential source data or shared
secrets for plain old encrypted data bags, and so on - data that should be
accessible to admins rather than directly to nodes. Chef-vault since
version 2.0 supports key rotation - exchanging the secret - which lets us
securely remove a user or node from the access list, and prevent them
from seeing updated content.

Shameless Plug: I've just written a usable backend for git-annex (
http://git-annex.branchable.com/) that stores files in a chef-vault data
bag, encrypted for all the users with admin=true - it would also be
possible to support various access levels for various items, for now it's
just a simple secret sharing mechanism. This way, once a teammate is
registered with chef-server, they can just git annex get all the keys
they need straight into the chef-repo, which is quite convenient. Project
is at GitHub - 3ofcoins/knife-annex: Knife plugin implementing a git-annex backend in chef-vault

-- M

On 17 September 2013 01:09, Seth Falcon seth@opscode.com wrote:

Hey there,

A few comments on Chef Vault, inline...

bluepojo@gmail.com writes:

When I was evaluating Vault recently, I had concerns about how much real
security it provides. In reading the implementation, it appears that this
is only security by obfuscation and not truly more secure than regular
encrypted data bags as you still must use the encrypted_data_bag_secret
to
unlock the key for the node you're operating on... you still have access
to
all of the secrets for all of the nodes, just not in a direct way.

Chef Vault helps you to solve the key distribution problem by taking
advantage of the RSA key pair assigned to each user and client on a Chef
Server. Encrypted data bags are used to implement it.

Here's (my understanding of) what happens when you create a Chef Vault
item (encrypt a data bag item using Chef Vault):

  1. You tell Chef Vault:

    • the name of a vault (maps to a data bag)
    • the name of an item (maps to a data bag item)
    • secret item data
    • the clients and users that should be able to access the secret

  2. Chef Vault generates a random shared key and uses the encrypted data
    bag support in Chef to encrypt the item data with this key.

  3. Chef Vault uses the Chef API to obtain the public RSA keys of the
    users and clients that should have access to item. It then encrypts
    the shared key generated in #1 for each of the public keys. It stores
    the user/client name to encrypted shared key in an $ITEM_keys data
    bag item in the $VAULT data bag.

Here's what I think this gives you:

  1. Compared to vanilla encrypted data bags, you don't have to arrange
    for a shared seceret to be distributed to clients (nodes) and users
    in your infrastructure.

  2. If an attacker gains root access to a node, the attacker can decrypt
    items that have been encrypted for that nodes client public
    key. So an attacker would not have access to all of the secrets for
    all of the nodes.

And some things it does not give you:

  1. Nodes store the client private key on disk. Using this key to decrypt
    secrets means compromise of the node will compromise any secrets
    which can be accessed via that key. So no torpedo launch key
    feature. In most cases, the secret data is used on a node in such a
    way that root access to the node would mean compromise of that secret
    regardless of how it was originally obtained. Avoiding that is hard.

  2. If a node is compromised, you will need to do more than remove the
    vault entry that encrypts the shared secret for the compromised key
    pair. In fact, if an attacker is able to obtain old copies of the
    encrypted data bag, changing the shared secret is not enough. You
    will need to change your underlying secret data -- e.g. if the item
    stored db passwords, you'll need to change the db password.

Having to leave the key on the node (as we do currently) means that once
a
single node is compromised, the entirety of things encrypted with
/etc/chef/encrypted_data_bag_secret are easily compromised.

Encrypted data bags in general do not have to use that file and
different items can use different shared secrets. For chef vault, only
those items that the node had access to will be compromised.

Currently, encrypted data bags only give security if your chef server
itself is compromised but they make people feel like they are being
more
secure by using them... the general attack vector, however, is through
nodes exposed to the public internet that have flawed applications
running,
not through the Chef server which is probably internal network access
only.
All of these nodes contain the encrypted_data_bag_secret.

As a result: encrypted data bags are not often any more secure than
regular
data bags for the general case and may inadvertently harm security due to
the aforementioned feeling of security that encrypting data bags gives
and
so some active secret storage system would be very valuable... just
something more substantial than Vault.

I see where you're coming from, but I disagree a bit. Protecting secrets
stored in the Chef Server has value. Even without something like chef
vault, it is possible to use different secrets for different items and
thus limit the impact of an attack.

  • seth

--
Seth Falcon | Development Lead | Opscode | @sfalcon

9

Good point to bring up! On a similar note we found after using chef-vault internally in chef runs that it also had this nice side effect too. We now use it as a keepass replacement. Keeping passwords to systems/APIs in chef-vault items and storing them in source control, so you get versioning like you would in keepass. Keeping them in the chef-repo is also nice because now they are already there if you ever need them in a client run, and if not, then they're in a location that easily accessible by all members of the team.

Kevin

From: Maciej Pasternacki <maciej@3ofcoins.netmailto:maciej@3ofcoins.net>
Reply-To: "chef@lists.opscode.commailto:chef@lists.opscode.com" <chef@lists.opscode.commailto:chef@lists.opscode.com>
Date: Tuesday, October 1, 2013 6:29 AM
To: "chef@lists.opscode.commailto:chef@lists.opscode.com" <chef@lists.opscode.commailto:chef@lists.opscode.com>
Subject: [chef] Re: Re: Re: Re: Chef Vault Writeup

This discussion focuses mostly on distributing secrets to nodes; chef-vault also allows to share secrets between users, encrypted by their private Chef API key. This can be quite convenient means to store secrets such as access keys to other APIs, confidential source data or shared secrets for plain old encrypted data bags, and so on - data that should be accessible to admins rather than directly to nodes. Chef-vault since version 2.0 supports key rotation - exchanging the secret - which lets us securely remove a user or node from the access list, and prevent them from seeing updated content.

Shameless Plug: I've just written a usable backend for git-annex (http://git-annex.branchable.com/) that stores files in a chef-vault data bag, encrypted for all the users with admin=true - it would also be possible to support various access levels for various items, for now it's just a simple secret sharing mechanism. This way, once a teammate is registered with chef-server, they can just git annex get all the keys they need straight into the chef-repo, which is quite convenient. Project is at GitHub - 3ofcoins/knife-annex: Knife plugin implementing a git-annex backend in chef-vault

-- M

On 17 September 2013 01:09, Seth Falcon <seth@opscode.commailto:seth@opscode.com> wrote:
Hey there,

A few comments on Chef Vault, inline...

bluepojo@gmail.commailto:bluepojo@gmail.com writes:

When I was evaluating Vault recently, I had concerns about how much real
security it provides. In reading the implementation, it appears that this
is only security by obfuscation and not truly more secure than regular
encrypted data bags as you still must use the encrypted_data_bag_secret to
unlock the key for the node you're operating on... you still have access to
all of the secrets for all of the nodes, just not in a direct way.

Chef Vault helps you to solve the key distribution problem by taking
advantage of the RSA key pair assigned to each user and client on a Chef
Server. Encrypted data bags are used to implement it.

Here's (my understanding of) what happens when you create a Chef Vault
item (encrypt a data bag item using Chef Vault):

  1. You tell Chef Vault:

    • the name of a vault (maps to a data bag)
    • the name of an item (maps to a data bag item)
    • secret item data
    • the clients and users that should be able to access the secret

  2. Chef Vault generates a random shared key and uses the encrypted data
    bag support in Chef to encrypt the item data with this key.

  3. Chef Vault uses the Chef API to obtain the public RSA keys of the
    users and clients that should have access to item. It then encrypts
    the shared key generated in #1 for each of the public keys. It stores
    the user/client name to encrypted shared key in an $ITEM_keys data
    bag item in the $VAULT data bag.

Here's what I think this gives you:

  1. Compared to vanilla encrypted data bags, you don't have to arrange
    for a shared seceret to be distributed to clients (nodes) and users
    in your infrastructure.

  2. If an attacker gains root access to a node, the attacker can decrypt
    items that have been encrypted for that nodes client public
    key. So an attacker would not have access to all of the secrets for
    all of the nodes.

And some things it does not give you:

  1. Nodes store the client private key on disk. Using this key to decrypt
    secrets means compromise of the node will compromise any secrets
    which can be accessed via that key. So no torpedo launch key
    feature. In most cases, the secret data is used on a node in such a
    way that root access to the node would mean compromise of that secret
    regardless of how it was originally obtained. Avoiding that is hard.

  2. If a node is compromised, you will need to do more than remove the
    vault entry that encrypts the shared secret for the compromised key
    pair. In fact, if an attacker is able to obtain old copies of the
    encrypted data bag, changing the shared secret is not enough. You
    will need to change your underlying secret data -- e.g. if the item
    stored db passwords, you'll need to change the db password.

Having to leave the key on the node (as we do currently) means that once a
single node is compromised, the entirety of things encrypted with
/etc/chef/encrypted_data_bag_secret are easily compromised.

Encrypted data bags in general do not have to use that file and
different items can use different shared secrets. For chef vault, only
those items that the node had access to will be compromised.

Currently, encrypted data bags only give security if your chef server
itself is compromised but they make people feel like they are being more
secure by using them... the general attack vector, however, is through
nodes exposed to the public internet that have flawed applications running,
not through the Chef server which is probably internal network access only.
All of these nodes contain the encrypted_data_bag_secret.

As a result: encrypted data bags are not often any more secure than regular
data bags for the general case and may inadvertently harm security due to
the aforementioned feeling of security that encrypting data bags gives and
so some active secret storage system would be very valuable... just
something more substantial than Vault.

I see where you're coming from, but I disagree a bit. Protecting secrets
stored in the Chef Server has value. Even without something like chef
vault, it is possible to use different secrets for different items and
thus limit the impact of an attack.

  • seth

--
Seth Falcon | Development Lead | Opscode | @sfalcon