I'm finally catching up with my backlog of Food Fight episodes and the one
on secrets got me thinking a bit and I wrote up my thoughts here.
The more I think about it, the more I think encrypted data bags aren't the
solution.
- Booker C. Bense
For what it's worth, I completely agree with this article, and have been
trying to think about ways to leverage these existing keys specifically for
this purpose. I wish they were used more.
On Mon, Sep 16, 2013 at 2:54 PM, Booker Bense bbense@gmail.com wrote:
I'm finally catching up with my backlog of Food Fight episodes and the one
on secrets got me thinking a bit and I wrote up my thoughts here.Curse the Darkness: Chef Encrypted Data Bags Are A Code Smell.
The more I think about it, the more I think encrypted data bags aren't the
solution.
- Booker C. Bense
On Sep 16, 2013, at 1:54 PM, Booker Bense bbense@gmail.com wrote:
Curse the Darkness: Chef Encrypted Data Bags Are A Code Smell.
The more I think about it, the more I think encrypted data bags aren't the solution.
The problem that was intended to be solved by encrypted data bags is where you share the Chef Server infrastructure with one or more other parties, and where you do not trust that infrastructure. Therefore, you encrypt your data bag content before uploading it to the Chef Server, and on the other end you decrypt it after you download the data bag content from the Chef Server. This is done with symmetric encryption keys.
In other words, they're solving the problem of not trusting a Hosted Chef environment.
Encrypted data bags were never intended to do anything else. Anyone who uses them for anything else is just setting themselves up for future pain and problems. Anyone who recommends that anyone use them for anything else is being foolish and reckless.
I'm not convinced that Chef Vault is anything of an improvement in this space, except perhaps for the issue of how to distribute a shared symmetric encryption key. I'm still trying to figure out how I feel about that.
Meanwhile, if we could completely eliminate the shared symmetric encryption key and use asymmetric public key cryptography instead, I think that would go a long ways towards solving at least some of the problems.
I know that Chef Vault tries to do this to a degree, but I am not convinced that they have covered all or even most of the holes that need to be addressed.
--
Brad Knowles brad@shub-internet.org
LinkedIn Profile: http://tinyurl.com/y8kpxu
On Sep 16, 2013, at 12:49 PM, Brad Knowles brad@shub-internet.org wrote:
On Sep 16, 2013, at 1:54 PM, Booker Bense bbense@gmail.com wrote:
Curse the Darkness: Chef Encrypted Data Bags Are A Code Smell.
The more I think about it, the more I think encrypted data bags aren't the solution.
The problem that was intended to be solved by encrypted data bags is where you share the Chef Server infrastructure with one or more other parties, and where you do not trust that infrastructure. Therefore, you encrypt your data bag content before uploading it to the Chef Server, and on the other end you decrypt it after you download the data bag content from the Chef Server. This is done with symmetric encryption keys.
In other words, they're solving the problem of not trusting a Hosted Chef environment.
Encrypted data bags were never intended to do anything else. Anyone who uses them for anything else is just setting themselves up for future pain and problems. Anyone who recommends that anyone use them for anything else is being foolish and reckless.
I'm not convinced that Chef Vault is anything of an improvement in this space, except perhaps for the issue of how to distribute a shared symmetric encryption key. I'm still trying to figure out how I feel about that.
Meanwhile, if we could completely eliminate the shared symmetric encryption key and use asymmetric public key cryptography instead, I think that would go a long ways towards solving at least some of the problems.
I know that Chef Vault tries to do this to a degree, but I am not convinced that they have covered all or even most of the holes that need to be addressed.
--
Brad Knowles brad@shub-internet.org
LinkedIn Profile: http://tinyurl.com/y8kpxu
This would be an excellent discussion topic for the Community Summit.
-Pete
@peter *
https://wiki.opscode.com/display/chef/Suggestions+for+2013+Community+Summit*
On Mon, Sep 16, 2013 at 1:03 PM, Peter Loron peterl@standingwave.orgwrote:
On Sep 16, 2013, at 12:49 PM, Brad Knowles brad@shub-internet.org wrote:
On Sep 16, 2013, at 1:54 PM, Booker Bense bbense@gmail.com wrote:
Curse the Darkness: Chef Encrypted Data Bags Are A Code Smell.
The more I think about it, the more I think encrypted data bags aren't
the solution.The problem that was intended to be solved by encrypted data bags is
where you share the Chef Server infrastructure with one or more other
parties, and where you do not trust that infrastructure. Therefore, you
encrypt your data bag content before uploading it to the Chef Server, and
on the other end you decrypt it after you download the data bag content
from the Chef Server. This is done with symmetric encryption keys.In other words, they're solving the problem of not trusting a Hosted
Chef environment.Encrypted data bags were never intended to do anything else. Anyone who
uses them for anything else is just setting themselves up for future pain
and problems. Anyone who recommends that anyone use them for anything else
is being foolish and reckless.I'm not convinced that Chef Vault is anything of an improvement in this
space, except perhaps for the issue of how to distribute a shared symmetric
encryption key. I'm still trying to figure out how I feel about that.Meanwhile, if we could completely eliminate the shared symmetric
encryption key and use asymmetric public key cryptography instead, I think
that would go a long ways towards solving at least some of the problems.I know that Chef Vault tries to do this to a degree, but I am not
convinced that they have covered all or even most of the holes that need to
be addressed.--
Brad Knowles brad@shub-internet.org
LinkedIn Profile: http://tinyurl.com/y8kpxuThis would be an excellent discussion topic for the Community Summit.
-Pete
--
Daniel DeLeo
On Monday, September 16, 2013 at 12:49 PM, Brad Knowles wrote:
On Sep 16, 2013, at 1:54 PM, Booker Bense <bbense@gmail.com (mailto:bbense@gmail.com)> wrote:
Curse the Darkness: Chef Encrypted Data Bags Are A Code Smell.
The more I think about it, the more I think encrypted data bags aren't the solution.
The problem that was intended to be solved by encrypted data bags is where you share the Chef Server infrastructure with one or more other parties, and where you do not trust that infrastructure. Therefore, you encrypt your data bag content before uploading it to the Chef Server, and on the other end you decrypt it after you download the data bag content from the Chef Server. This is done with symmetric encryption keys.
In other words, they're solving the problem of not trusting a Hosted Chef environment.
Encrypted data bags were never intended to do anything else. Anyone who uses them for anything else is just setting themselves up for future pain and problems. Anyone who recommends that anyone use them for anything else is being foolish and reckless.
I'm not convinced that Chef Vault is anything of an improvement in this space, except perhaps for the issue of how to distribute a shared symmetric encryption key. I'm still trying to figure out how I feel about that.
Meanwhile, if we could completely eliminate the shared symmetric encryption key and use asymmetric public key cryptography instead, I think that would go a long ways towards solving at least some of the problems.
I know that Chef Vault tries to do this to a degree, but I am not convinced that they have covered all or even most of the holes that need to be addressed.
--
Brad Knowles <brad@shub-internet.org (mailto:brad@shub-internet.org)>
I disagree. By controlling which machines you distribute which secrets to, you can limit exposure of secrets if a particular machine is compromised. We've certainly found this useful even for infrastructure we manage with a Chef server completely under our control.
That said, more elegant and flexible solutions are possible, and at Opscode we're not opposed to anyone developing them as a standalone project or as a feature patch to Chef.
--
Daniel DeLeo
i dont think encrypted data bag in itself is a code smell. Depending upon
the context they may be. The fact that we need to store secrets in raw text
files is smell (in one extreme). Tools (like s3cmd, knife , aws command
line tools etc) that expect un encrypted secrets can be run against
ephemeral configuration files. But thats another extreme. Between them
there are myriad of options, encrypted data bag is one of them. if you plan
to take snapshot of your entire chef infra, you'll need to backup and store
the databags too, and if they hold secrets its better to store them
encrypted in the restore/backup tapes. Albeit chef-valult is a better
option, but it requires more house keeping. You'll be storing multiple
encrypted copies of same data, each corresponding to one client. these
things comes with great deal of network traffic cost as well.
So yes, they can be smell, but it depends,
On Mon, Sep 16, 2013 at 11:54 AM, Booker Bense bbense@gmail.com wrote:
I'm finally catching up with my backlog of Food Fight episodes and the one
on secrets got me thinking a bit and I wrote up my thoughts here.Curse the Darkness: Chef Encrypted Data Bags Are A Code Smell.
The more I think about it, the more I think encrypted data bags aren't the
solution.
- Booker C. Bense
So yes, they can be smell, but it depends,
This statement can apply to pretty much anything, ever.
If your use case doesn't map to the tool you're using, that's fine. Find
something that works for you.
Nobody is here telling you that you must use one thing or another - rather
most people here are sharing "what works for me" approaches, thus
delivering some of the best ideas that get discussed openly, and better
ideas come from them, sometimes.
Definitely a summit topic to discuss. And don't talk about my feet
smelling, either.
-M
On Mon, Sep 16, 2013 at 4:53 PM, Ranjib Dey dey.ranjib@gmail.com wrote:
i dont think encrypted data bag in itself is a code smell. Depending upon
the context they may be. The fact that we need to store secrets in raw text
files is smell (in one extreme). Tools (like s3cmd, knife , aws command
line tools etc) that expect un encrypted secrets can be run against
ephemeral configuration files. But thats another extreme. Between them
there are myriad of options, encrypted data bag is one of them. if you plan
to take snapshot of your entire chef infra, you'll need to backup and store
the databags too, and if they hold secrets its better to store them
encrypted in the restore/backup tapes. Albeit chef-valult is a better
option, but it requires more house keeping. You'll be storing multiple
encrypted copies of same data, each corresponding to one client. these
things comes with great deal of network traffic cost as well.
So yes, they can be smell, but it depends,On Mon, Sep 16, 2013 at 11:54 AM, Booker Bense bbense@gmail.com wrote:
I'm finally catching up with my backlog of Food Fight episodes and the
one on secrets got me thinking a bit and I wrote up my thoughts here.Curse the Darkness: Chef Encrypted Data Bags Are A Code Smell.
The more I think about it, the more I think encrypted data bags aren't
the solution.
- Booker C. Bense
On Mon, Sep 16, 2013 at 12:49 PM, Brad Knowles brad@shub-internet.orgwrote:
On Sep 16, 2013, at 1:54 PM, Booker Bense bbense@gmail.com wrote:
Curse the Darkness: Chef Encrypted Data Bags Are A Code Smell.
The more I think about it, the more I think encrypted data bags aren't
the solution.The problem that was intended to be solved by encrypted data bags is where
you share the Chef Server infrastructure with one or more other parties,
and where you do not trust that infrastructure. Therefore, you encrypt
your data bag content before uploading it to the Chef Server, and on the
other end you decrypt it after you download the data bag content from the
Chef Server. This is done with symmetric encryption keys.In other words, they're solving the problem of not trusting a Hosted Chef
environment.
If you don't trust Hosted Chef to keep your keys, why are you trusting it
to keep code you run as root on your system? There is some value to EDB in
the hosted chef scenerio. It certainly helps with the problem of leaking
data via backups, code repos, etc... even outside the hosted chef scenerio.
But EDB's don't solve the problem of putting a secret on the machine in the
first place. They just change which secret needs to be installed. If you've
got a process for installing the key for the EDB, why not just use that to
install your secret and avoid the exposure of Hosted Chef altogether?
- Booker C. Bense
no not really. % of raw ruby code inside a recipe but outside resources is
a smell, irrespective of context. less you have better it is. we have
several such smells (another would be searching 'recipes:foo'). what i
meant is this is not bad, in fact it can be a blessing at times.
remember if you take the encrypted data bag route, you would be able to run
chef zero as it is (and now chef-zero integration is in master), which does
not support client certs.
On Mon, Sep 16, 2013 at 2:02 PM, Mike miketheman@gmail.com wrote:
So yes, they can be smell, but it depends,
This statement can apply to pretty much anything, ever.If your use case doesn't map to the tool you're using, that's fine. Find
something that works for you.
Nobody is here telling you that you must use one thing or another - rather
most people here are sharing "what works for me" approaches, thus
delivering some of the best ideas that get discussed openly, and better
ideas come from them, sometimes.Definitely a summit topic to discuss. And don't talk about my feet
smelling, either.-M
On Mon, Sep 16, 2013 at 4:53 PM, Ranjib Dey dey.ranjib@gmail.com wrote:
i dont think encrypted data bag in itself is a code smell. Depending upon
the context they may be. The fact that we need to store secrets in raw text
files is smell (in one extreme). Tools (like s3cmd, knife , aws command
line tools etc) that expect un encrypted secrets can be run against
ephemeral configuration files. But thats another extreme. Between them
there are myriad of options, encrypted data bag is one of them. if you plan
to take snapshot of your entire chef infra, you'll need to backup and store
the databags too, and if they hold secrets its better to store them
encrypted in the restore/backup tapes. Albeit chef-valult is a better
option, but it requires more house keeping. You'll be storing multiple
encrypted copies of same data, each corresponding to one client. these
things comes with great deal of network traffic cost as well.
So yes, they can be smell, but it depends,On Mon, Sep 16, 2013 at 11:54 AM, Booker Bense bbense@gmail.com wrote:
I'm finally catching up with my backlog of Food Fight episodes and the
one on secrets got me thinking a bit and I wrote up my thoughts here.Curse the Darkness: Chef Encrypted Data Bags Are A Code Smell.
The more I think about it, the more I think encrypted data bags aren't
the solution.
- Booker C. Bense
Encrypted data bags were never intended to do anything else. Anyone who
uses them for anything else is just setting themselves up for future pain
and problems. Anyone who recommends that anyone use them for anything else
is being foolish and reckless.
Encrypted databags provide protection against two kinds of access:
I also disagree, especially with your assertion of what and "how many"
things EDB is protecting against.
I've certainly been in the situation of sharing a common Chef code-base
amongst many groups where secrets needed to be siloed amongst consumers,
and kept from the administrators of the source control system too. We
shouldn't assume there is one operations group that is the keeper of all of
the keys, because in most large organisations that isn't the case.
Sam Pointer
Lead Consultant
www.opsunit.com
On 16 September 2013 22:31, Ranjib Dey dey.ranjib@gmail.com wrote:
no not really. % of raw ruby code inside a recipe but outside resources
is a smell, irrespective of context. less you have better it is. we have
several such smells (another would be searching 'recipes:foo'). what i
meant is this is not bad, in fact it can be a blessing at times.remember if you take the encrypted data bag route, you would be able to
run chef zero as it is (and now chef-zero integration is in master), which
does not support client certs.On Mon, Sep 16, 2013 at 2:02 PM, Mike miketheman@gmail.com wrote:
So yes, they can be smell, but it depends,
This statement can apply to pretty much anything, ever.If your use case doesn't map to the tool you're using, that's fine.
Find something that works for you.
Nobody is here telling you that you must use one thing or another -
rather most people here are sharing "what works for me" approaches, thus
delivering some of the best ideas that get discussed openly, and better
ideas come from them, sometimes.Definitely a summit topic to discuss. And don't talk about my feet
smelling, either.-M
On Mon, Sep 16, 2013 at 4:53 PM, Ranjib Dey dey.ranjib@gmail.com wrote:
i dont think encrypted data bag in itself is a code smell. Depending
upon the context they may be. The fact that we need to store secrets in raw
text files is smell (in one extreme). Tools (like s3cmd, knife , aws
command line tools etc) that expect un encrypted secrets can be run against
ephemeral configuration files. But thats another extreme. Between them
there are myriad of options, encrypted data bag is one of them. if you plan
to take snapshot of your entire chef infra, you'll need to backup and store
the databags too, and if they hold secrets its better to store them
encrypted in the restore/backup tapes. Albeit chef-valult is a better
option, but it requires more house keeping. You'll be storing multiple
encrypted copies of same data, each corresponding to one client. these
things comes with great deal of network traffic cost as well.
So yes, they can be smell, but it depends,On Mon, Sep 16, 2013 at 11:54 AM, Booker Bense bbense@gmail.com wrote:
I'm finally catching up with my backlog of Food Fight episodes and the
one on secrets got me thinking a bit and I wrote up my thoughts here.Curse the Darkness: Chef Encrypted Data Bags Are A Code Smell.
The more I think about it, the more I think encrypted data bags aren't
the solution.
- Booker C. Bense
On Tue, Sep 17, 2013 at 4:41 AM, Sam Pointer sam.pointer@opsunit.comwrote:
Encrypted data bags were never intended to do anything else. Anyone who
uses them for anything else is just setting themselves up for future pain
and problems. Anyone who recommends that anyone use them for anything else
is being foolish and reckless.Encrypted databags provide protection against two kinds of access:
I also disagree, especially with your assertion of what and "how many"
things EDB is protecting against.I've certainly been in the situation of sharing a common Chef code-base
amongst many groups where secrets needed to be siloed amongst consumers,
and kept from the administrators of the source control system too. We
shouldn't assume there is one operations group that is the keeper of all of
the keys, because in most large organisations that isn't the case.
Well, I guess I didn't do a very good job of explain the concepts of
Implicit and Explicit access, because they cover all the cases you list
above.
Explicit access is using your chef client key to access data in the chef
data store via the chef API.
Implicit access is every other kind of access to the data store via code
repo's, database backups, root access on the server, etc...
My argument is not that EDB's cannot implement both kinds of access
control, but that they do it in a way that has bad long term
consequences[1] and still leaves you with the problem of installing a
secret on the machine. All they really do is establish an SEP
field[2] around the problem.
The reason distributing secrets in an automated way is such a hard problem
is that encryption cannot create trust, it can only extend it. Ultimately,
you have to pick something to trust and then use that trust to install a
secret. The reason this is so difficult in the general case is that every
organization has different levels of risk acceptance and items of trust.
Chef already has a item of trust in the key pair each client must have to
use the system. Rather than creating a whole new ecosystem to manage the
ACL's and keys of EDB's ( which is what Chef Vault attempts), it seems to
me to make more sense to try and build something on the existing trust
item. You already have a process for installing the chef key pair client.
- Booker C. Bense
[1]- Especially in large organizations that have significant audit trail
requirements.
I tend to agree with Brad here about the proper role of encrypted databags.
Having been somewhat involved with them early on, I figured I'd relink some
old blog posts on the subject:
Really encrypted databags are nothing more than obfuscation. The reality
check section on the first link is my perspective. Everyone's shit has more
risky attack vectors than just chef databags. Additionally they don't even
begin to deal with advanced issues like Authorization or Authentication
(though Chef-Vault handles some of that). Your API creds or validation pem
are a bigger attack vector than one encrypted value.
With the API creds I can upload a new cookbook or databag that creates a
user for me on one of your systems and I'll have access. Then I don't even
CARE about the encrypted creds. I'll just find the file on the system where
you're writing the decrypted values.
On Mon, Sep 16, 2013 at 3:49 PM, Brad Knowles brad@shub-internet.orgwrote:
On Sep 16, 2013, at 1:54 PM, Booker Bense bbense@gmail.com wrote:
Curse the Darkness: Chef Encrypted Data Bags Are A Code Smell.
The more I think about it, the more I think encrypted data bags aren't
the solution.The problem that was intended to be solved by encrypted data bags is where
you share the Chef Server infrastructure with one or more other parties,
and where you do not trust that infrastructure. Therefore, you encrypt
your data bag content before uploading it to the Chef Server, and on the
other end you decrypt it after you download the data bag content from the
Chef Server. This is done with symmetric encryption keys.In other words, they're solving the problem of not trusting a Hosted Chef
environment.Encrypted data bags were never intended to do anything else. Anyone who
uses them for anything else is just setting themselves up for future pain
and problems. Anyone who recommends that anyone use them for anything else
is being foolish and reckless.I'm not convinced that Chef Vault is anything of an improvement in this
space, except perhaps for the issue of how to distribute a shared symmetric
encryption key. I'm still trying to figure out how I feel about that.Meanwhile, if we could completely eliminate the shared symmetric
encryption key and use asymmetric public key cryptography instead, I think
that would go a long ways towards solving at least some of the problems.I know that Chef Vault tries to do this to a degree, but I am not
convinced that they have covered all or even most of the holes that need to
be addressed.--
Brad Knowles brad@shub-internet.org
LinkedIn Profile: http://tinyurl.com/y8kpxu
bbense@gmail.com writes:
Chef already has a item of trust in the key pair each client must have to
use the system. Rather than creating a whole new ecosystem to manage the
ACL's and keys of EDB's ( which is what Chef Vault attempts), it seems to
me to make more sense to try and build something on the existing trust
item. You already have a process for installing the chef key pair
client.
I'm confused. Isn't that what Chef Vault is doing? It uses the existing
key pairs in the Chef system to provide access to a shared secret by
encrypting the secret for each public key that needs access to it.
- seth
--
Seth Falcon | Development Lead | Opscode | @sfalcon
Here's some of the early work I did that I suspect chef-vault is based on.
It may give a simpler view into what's going on underneath.
#1 create a symmetric key that is never saved to disk for use with an EDB
#2 encrypt that symmetric key to the public_key of each node (client) that
needs access ( I distribute using a unencrypted data bag for that)
#3 EDB the contents you want known only to those nodes with the symmetric
key
#4 throw away the symmetric key
Nodes/Clients search for the data bags from #2 that contain an entry for
themselves, decrypt the symmetric key, and open the EDB.
It would be cleaner if we could encrypt directly to all the nodes (clients)
at once using some type of PKI.
It's very likely possible, but I haven't enough time on it.
On Tue, Sep 17, 2013 at 10:27 PM, Seth Falcon seth@opscode.com wrote:
bbense@gmail.com writes:
Chef already has a item of trust in the key pair each client must have to
use the system. Rather than creating a whole new ecosystem to manage the
ACL's and keys of EDB's ( which is what Chef Vault attempts), it seems to
me to make more sense to try and build something on the existing trust
item. You already have a process for installing the chef key pair
client.I'm confused. Isn't that what Chef Vault is doing? It uses the existing
key pairs in the Chef system to provide access to a shared secret by
encrypting the secret for each public key that needs access to it.
- seth
--
Seth Falcon | Development Lead | Opscode | @sfalcon
On Tue, Sep 17, 2013 at 1:27 PM, Seth Falcon seth@opscode.com wrote:
bbense@gmail.com writes:
Chef already has a item of trust in the key pair each client must have to
use the system. Rather than creating a whole new ecosystem to manage the
ACL's and keys of EDB's ( which is what Chef Vault attempts), it seems to
me to make more sense to try and build something on the existing trust
item. You already have a process for installing the chef key pair
client.I'm confused. Isn't that what Chef Vault is doing? It uses the existing
key pairs in the Chef system to provide access to a shared secret by
encrypting the secret for each public key that needs access to it.
If I'm reading the documentation correctly, Chef Vault is using public keys
to manage access
to a shared secret for an EDB. If you are going to use EDB's in their
current state, this
is probably the most reasonable approach possible.
But why use an EDB at all? Why not just encrypt the item with the scheme
you are using
to encrypt the access to the item? These are largely rhetorical questions.
I completely understand
why chef vault is built the way it is.
We ought to be asking ourselves why Chef Vault is required at all.
Implementing ACL's and access control on the client side of a protocol
ought to be ringing alarm bells.
Here's a simple example:
If I do not rotate keys after every ACL change, then I have only made ACL
changes for those clients that play nice. Bad clients
that stash the secret can still access the data items.
I think the interface to knife that Chef Vault provides covers most of my
objections to EDB's. However, before we declare victory and say the problem
is solved we should think a bit about the larger problem. I realize this
whole area is probably sensitive as ACL's are a "pro feature" of Enterprise
Chef.
- Booker C. Bense
bbense@gmail.com writes:
If I'm reading the documentation correctly, Chef Vault is using public
keys to manage access to a shared secret for an EDB.
Yes, that's correct.
But why use an EDB at all? Why not just encrypt the item with the
scheme you are using to encrypt the access to the item?
The size of a message that can be encrypted with RSA is limited by the
key size. So to make a general solution and to follow how (as far as I
know) RSA is generally used, you use a symmetric cipher for the message
and RSA to encrypt the key for the symmetric cipher. Hence chef vault
using EDBs.
Here's a simple example:
If I do not rotate keys after every ACL change, then I have only made ACL
changes for those clients that play nice. Bad clients
that stash the secret can still access the data items.
Absolutely something to understand and pay attention to.
- seth
--
Seth Falcon | Development Lead | Opscode | @sfalcon