What ways have people used to maintain database secrets? I’m thinking
specifically of the mysql root password which is just an attribute in the
mysql cookbook, and passwords for production databases?
I don’t want to be checking passwords into Git.
What strategies have you successfully used?
-aob
How about letting mysql cookbook generate root password and store it in the
Chef node object, then use knife-backup gem to backup node objects to
secure location?
On Sun, Sep 21, 2014 at 9:34 PM, Angus Buchanan angus.o.buchanan@gmail.com
wrote:
What ways have people used to maintain database secrets? I'm thinking
specifically of the mysql root password which is just an attribute in the
mysql cookbook, and passwords for production databases?I don't want to be checking passwords into Git.
What strategies have you successfully used?
-aob
--
Best regards, Dmitriy V.
Hi,
we store encryption secret file within separate git repos (one different
repo for each environment cookbook it refers to) with very restricted
access despite to each environment cookbook chef repo that contains
encrypted data bag items only.
In this way we don'let cookbooks to generate random passwords out of our
control, but we decide our passwords rules and change policies.
Regards,
Marco
Il 22/set/2014 06:34 "Angus Buchanan" angus.o.buchanan@gmail.com ha
scritto:
What ways have people used to maintain database secrets? I'm thinking
specifically of the mysql root password which is just an attribute in the
mysql cookbook, and passwords for production databases?I don't want to be checking passwords into Git.
What strategies have you successfully used?
-aob
this is a Hard Problem.
good article by Noah here: Secrets Management and Chef – Noah Kantrowitz
I like the look of GitHub - oleiade/trousseau: File based encrypted key-value store but haven't
really built out integration with Chef.
Storing secrets in Consul or etcd is high on my to-evaluate list as well!
cheers,
aj
On Mon, Sep 22, 2014 at 4:34 PM, Angus Buchanan
angus.o.buchanan@gmail.com wrote:
What ways have people used to maintain database secrets? I'm thinking
specifically of the mysql root password which is just an attribute in the
mysql cookbook, and passwords for production databases?I don't want to be checking passwords into Git.
What strategies have you successfully used?
-aob
Here's what we do (the text below is adapted from our documentation):
We use chef's built-in encrypted databags. This solution is less
secure than some of the alternatives, because you need a shared key,
this key is needed by you to encrypt the databag and by the instances
to decrypt the databag. However it is pretty simple, the main problem
is where to store this shared key. We didn't want this key on every
server (it would have meant the key has to be in our bootstrap or
cloudformation template). Therefore we chose to download the key from
S3 when we need it.
The S3 solution is not ideal but with the help of the AWS policy we
deny access to the bucket where the shared keys are, except for an IAM
role which is used by our instances. Therefore if you know where is
the key, you can get it as long as you do it through a server with the
correct IAM profile and by using the tmp credentials.
The standard process following opscode instructions is to do a knife data bag from file with the shared key as an argument, however if we
want to do it in our continuous deployment it is a problem because
In order to fix that we created a rake task which will, take a json
formatted databag, download the shared key via an ssh connection to an
instance + an S3 download, encrypt the databag, create a file in the
chef-repo with the encrypted databag. Our CI process will then deploy
the encrypted databag like any other databag.
If you need to update a databag you need to decrypt it, then encrypt
it again. The take task will do that for you.
Chef will have to download the key from S3, then load the databag and
decrypt it. We created a cookbook which helps to load encrypted
databag.
On Mon, Sep 22, 2014 at 12:11 AM, AJ Christensen
aj@junglistheavy.industries wrote:
this is a Hard Problem.
good article by Noah here: Secrets Management and Chef – Noah Kantrowitz
I like the look of GitHub - oleiade/trousseau: File based encrypted key-value store but haven't
really built out integration with Chef.Storing secrets in Consul or etcd is high on my to-evaluate list as well!
cheers,
aj
On Mon, Sep 22, 2014 at 4:34 PM, Angus Buchanan
angus.o.buchanan@gmail.com wrote:What ways have people used to maintain database secrets? I'm thinking
specifically of the mysql root password which is just an attribute in the
mysql cookbook, and passwords for production databases?I don't want to be checking passwords into Git.
What strategies have you successfully used?
-aob
--
Brian Pitts
Web Operations Engineer
Storing secrets are a hard problem, since you're always pushing your
peas around your plate.
You're looking to manage them "out of band" of Chef, (or at least out
of band of Chef's SCM).
Gather your secrets during recipe compilation, then pass them into the
mysql_service resource in your recipe.
Chef Vault example:
vault_ssl = ChefVault::Item.load('secrets', 'www.widgetco.biz.pem')
my_secret = vault_ssl['www.widgetco.biz.pem']
mysql_service 'default' do
server_root_password vault_ssl['www.widgetco.biz.pem']
action :create
end
On Mon, Sep 22, 2014 at 12:34 AM, Angus Buchanan
angus.o.buchanan@gmail.com wrote:
What ways have people used to maintain database secrets? I'm thinking
specifically of the mysql root password which is just an attribute in the
mysql cookbook, and passwords for production databases?I don't want to be checking passwords into Git.
What strategies have you successfully used?
-aob
that was supposed to read:
server_root_password my_secret
On Mon, Sep 22, 2014 at 10:58 AM, Sean OMeara someara@opscode.com wrote:
Storing secrets are a hard problem, since you're always pushing your
peas around your plate.You're looking to manage them "out of band" of Chef, (or at least out
of band of Chef's SCM).
- You can bootstrap them onto the machines with jumpstart/whatever
- Use something like Chef Vault to store them on Chef Server
- Use an external database or directory service
Gather your secrets during recipe compilation, then pass them into the
mysql_service resource in your recipe.Chef Vault example:
vault_ssl = ChefVault::Item.load('secrets', 'www.widgetco.biz.pem')
my_secret = vault_ssl['www.widgetco.biz.pem']mysql_service 'default' do
server_root_password vault_ssl['www.widgetco.biz.pem']
action :create
endOn Mon, Sep 22, 2014 at 12:34 AM, Angus Buchanan
angus.o.buchanan@gmail.com wrote:What ways have people used to maintain database secrets? I'm thinking
specifically of the mysql root password which is just an attribute in the
mysql cookbook, and passwords for production databases?I don't want to be checking passwords into Git.
What strategies have you successfully used?
-aob
Others have already commented with my post on this topic, but just for the record I wanted to make this clearer:
Under no circumstances store private data in node attributes. All nodes can see all other node attributes by default. Unless you want every node in your network to have effective root on your DB server, this is not viable.
--Noah
On Sep 21, 2014, at 9:56 PM, DV vindimy@gmail.com wrote:
How about letting mysql cookbook generate root password and store it in the Chef node object, then use knife-backup gem to backup node objects to secure location?
On Sun, Sep 21, 2014 at 9:34 PM, Angus Buchanan angus.o.buchanan@gmail.com wrote:
What ways have people used to maintain database secrets? I'm thinking specifically of the mysql root password which is just an attribute in the mysql cookbook, and passwords for production databases?I don't want to be checking passwords into Git.
What strategies have you successfully used?
-aob
--
Best regards, Dmitriy V.
Although this is a thing:
http://onddo.github.io/chef-encrypted-attributes/
TL;DR: Symmetric encryption of node attribute values with the Chef client key. Only the key of the node that wrote the attribute can decrypt the value.
(It does a bunch more too).
--Charles
On September 22, 2014 at 10:17:00 AM, Noah Kantrowitz (noah@coderanger.net) wrote:
Others have already commented with my post on this topic, but just for the record I wanted to make this clearer:
Under no circumstances store private data in node attributes. All nodes can see all other node attributes by default. Unless you want every node in your network to have effective root on your DB server, this is not viable.
--Noah
On Sep 21, 2014, at 9:56 PM, DV vindimy@gmail.com wrote:
How about letting mysql cookbook generate root password and store it in the Chef node object, then use knife-backup gem to backup node objects to secure location?
On Sun, Sep 21, 2014 at 9:34 PM, Angus Buchanan angus.o.buchanan@gmail.com wrote:
What ways have people used to maintain database secrets? I'm thinking specifically of the mysql root password which is just an attribute in the mysql cookbook, and passwords for production databases?I don't want to be checking passwords into Git.
What strategies have you successfully used?
-aob
--
Best regards, Dmitriy V.
Can't quite understand exactly what your implementation is. However,
everyone should be aware that by default the clients group is allowed
read access to all the cookbooks -- so any node on your network can read
any cookbook. Without managing the default Chef ACLs you cannot publish
secrets in cookbooks/recipes and expect them to stay secret from
attackers who control one of your nodes at all. There's also still the
search loophole where any Chef ACL can be defeated if you can get search
to return a result (although cookbooks are not indexed). Obviously if
you try to leverage using cookbooks to push out encrypted data bag
secrets in order to use encrypted data bags, then attackers can read the
cookbooks to get the encrypted data bag secrets in order to decrypt the
data bags.
So,
#1 don't store unencrypted secrets in cookbooks/recipes
#2 don't store unencrypted secrets in data bags
#3 don't store unencrypted secrets in node data ('normal' attrs, roles,
environments, etc)
#4 the decryption keys for encrypted databags are unencrypted secrets
and rules #1, #2, and #3 apply.
On 9/21/14, 10:09 PM, Marco Betti wrote:
Hi,
we store encryption secret file within separate git repos (one
different repo for each environment cookbook it refers to) with very
restricted access despite to each environment cookbook chef repo that
contains encrypted data bag items only.
In this way we don'let cookbooks to generate random passwords out of
our control, but we decide our passwords rules and change policies.Regards,
MarcoIl 22/set/2014 06:34 "Angus Buchanan" <angus.o.buchanan@gmail.com
mailto:angus.o.buchanan@gmail.com> ha scritto:What ways have people used to maintain database secrets? I'm thinking specifically of the mysql root password which is just an attribute in the mysql cookbook, and passwords for production databases? I don't want to be checking passwords into Git. What strategies have you successfully used? -aob
I have thought a lot about this problem and it really requires using a
stateful orchestration server to store the secerets and arbitrate access to
them. Why the orchestration server? Because the orchestration server can
verify the identity of your instances, encrypt the secrets using instance
specific creds, and can remove access for instances that have been
destroyed. Sadly I am not aware of an open source tool that currently does
this.
On Sep 22, 2014 9:00 PM, "Lamont Granquist" lamont@opscode.com wrote:
Can't quite understand exactly what your implementation is. However,
everyone should be aware that by default the clients group is allowed read
access to all the cookbooks -- so any node on your network can read any
cookbook. Without managing the default Chef ACLs you cannot publish
secrets in cookbooks/recipes and expect them to stay secret from attackers
who control one of your nodes at all. There's also still the search
loophole where any Chef ACL can be defeated if you can get search to return
a result (although cookbooks are not indexed). Obviously if you try to
leverage using cookbooks to push out encrypted data bag secrets in order to
use encrypted data bags, then attackers can read the cookbooks to get the
encrypted data bag secrets in order to decrypt the data bags.So,
#1 don't store unencrypted secrets in cookbooks/recipes
#2 don't store unencrypted secrets in data bags
#3 don't store unencrypted secrets in node data ('normal' attrs, roles,
environments, etc)
#4 the decryption keys for encrypted databags are unencrypted secrets and
rules #1, #2, and #3 apply.On 9/21/14, 10:09 PM, Marco Betti wrote:
Hi,
we store encryption secret file within separate git repos (one different
repo for each environment cookbook it refers to) with very restricted
access despite to each environment cookbook chef repo that contains
encrypted data bag items only.
In this way we don'let cookbooks to generate random passwords out of our
control, but we decide our passwords rules and change policies.Regards,
Marco
Il 22/set/2014 06:34 "Angus Buchanan" angus.o.buchanan@gmail.com ha
scritto:What ways have people used to maintain database secrets? I'm thinking
specifically of the mysql root password which is just an attribute in the
mysql cookbook, and passwords for production databases?I don't want to be checking passwords into Git.
What strategies have you successfully used?
-aob
On Mon, 22 Sep 2014 11:27:44 -0700
Charles Johnson charles@opscode.com wrote:
Although this is a thing:
Yo can see an example of how to use this with MySQL in the
PostfixAdmin cookbook:
Although not by default, you can also allow other nodes to read them if
you want.
--
Xabier de Zuazo
Another way to go about this would be to ACL search results. If you had
a data bag that only one node could read and you couldn't bypass that
with search then that would help. Or you could introduce new objects
that weren't indexed and searchable and only the node could retrieve
(e.g. a node-like object that was not indexed and was by default created
with private permissions so that only admins and the node client key
could write to it). We need to break up the node object anyway since
having the run_list and normal attrs stored with the rest of the node
attrs is a bit terrible (CHEF-4978).
Hosted would still need encryption, but if you trust the Chef Server you
should be able to fix this problem by rubbing ACLs on it.
On 9/22/14, 12:26 PM, Bryan Berry wrote:
I have thought a lot about this problem and it really requires using a
stateful orchestration server to store the secerets and arbitrate
access to them. Why the orchestration server? Because the
orchestration server can verify the identity of your instances,
encrypt the secrets using instance specific creds, and can remove
access for instances that have been destroyed. Sadly I am not aware of
an open source tool that currently does this.On Sep 22, 2014 9:00 PM, "Lamont Granquist" <lamont@opscode.com
mailto:lamont@opscode.com> wrote:Can't quite understand exactly what your implementation is. However, everyone should be aware that by default the clients group is allowed read access to all the cookbooks -- so any node on your network can read any cookbook. Without managing the default Chef ACLs you cannot publish secrets in cookbooks/recipes and expect them to stay secret from attackers who control one of your nodes at all. There's also still the search loophole where any Chef ACL can be defeated if you can get search to return a result (although cookbooks are not indexed). Obviously if you try to leverage using cookbooks to push out encrypted data bag secrets in order to use encrypted data bags, then attackers can read the cookbooks to get the encrypted data bag secrets in order to decrypt the data bags. So, #1 don't store unencrypted secrets in cookbooks/recipes #2 don't store unencrypted secrets in data bags #3 don't store unencrypted secrets in node data ('normal' attrs, roles, environments, etc) #4 the decryption keys for encrypted databags are unencrypted secrets and rules #1, #2, and #3 apply. On 9/21/14, 10:09 PM, Marco Betti wrote:Hi, we store encryption secret file within separate git repos (one different repo for each environment cookbook it refers to) with very restricted access despite to each environment cookbook chef repo that contains encrypted data bag items only. In this way we don'let cookbooks to generate random passwords out of our control, but we decide our passwords rules and change policies. Regards, Marco Il 22/set/2014 06:34 "Angus Buchanan" <angus.o.buchanan@gmail.com <mailto:angus.o.buchanan@gmail.com>> ha scritto: What ways have people used to maintain database secrets? I'm thinking specifically of the mysql root password which is just an attribute in the mysql cookbook, and passwords for production databases? I don't want to be checking passwords into Git. What strategies have you successfully used? -aob
On 22 September 2014 21:22, Xabier de Zuazo xabier@onddo.com wrote:
On Mon, 22 Sep 2014 11:27:44 -0700
Charles Johnson charles@opscode.com wrote:Although this is a thing:
Yo can see an example of how to use this with MySQL in the
PostfixAdmin cookbook:https://github.com/onddo/postfixadmin-cookbook/blob/master/recipes/mysql.rb
Although not by default, you can also allow other nodes to read them if
you want.
That's pretty cool! I haven't seen the basic chef-vault idea applied
to node attributes before. That's nice.
S.