New Authentication in Chef 0.8

Hi,

While reading the new Authentication page on Chef wiki [1] it seemed to me that it’s very similar to authentication with SSL client certificates.
Are you somehow reinventing the wheel or is there some obvious benefit that I’m not aware of?

Best Regards

Miguel Cabeça

[1[ http://wiki.opscode.com/display/chef/Authentication

Not reinventing, using directly in the signature process. The SSH key
metaphor seems to resonate for most people.

If you look at the EC2 API auth, this is very similar, but with public/
private keys rather than a shared secret.

Sent from my iPhone

On Feb 10, 2010, at 7:30 AM, Miguel Cabeça cabeca@ist.utl.pt wrote:

Hi,

While reading the new Authentication page on Chef wiki [1] it seemed
to me that it's very similar to authentication with SSL client
certificates.
Are you somehow reinventing the wheel or is there some obvious
benefit that I'm not aware of?

Best Regards

Miguel Cabeça

[1[ http://wiki.opscode.com/display/chef/Authentication

Now that I'm not on my iPhone. What we're doing is creating a digital
signature with RSA keys, which signs each request.

Adam

On Wed, Feb 10, 2010 at 8:24 AM, Adam Jacob adam@opscode.com wrote:

Not reinventing, using directly in the signature process. The SSH key
metaphor seems to resonate for most people.

If you look at the EC2 API auth, this is very similar, but with
public/private keys rather than a shared secret.

Sent from my iPhone

On Feb 10, 2010, at 7:30 AM, Miguel Cabeça cabeca@ist.utl.pt wrote:

Hi,

While reading the new Authentication page on Chef wiki [1] it seemed to me
that it's very similar to authentication with SSL client certificates.
Are you somehow reinventing the wheel or is there some obvious benefit
that I'm not aware of?

Best Regards

Miguel Cabeça

[1[ http://wiki.opscode.com/display/chef/Authentication

--
Opscode, Inc.
Adam Jacob, CTO
T: (206) 508-7449 E: adam@opscode.com

Hi,

Now that I'm not on my iPhone. What we're doing is creating a digital
signature with RSA keys, which signs each request.

That's precisely what presenting a client certificate does in establishing an SSL connection. Part of the SSL handshaking involves the client signing something with his private key for the other side to verify with the client public key (present in the certificate offered to the other side). After the handshake, the secure channel guarantees that both sides of the communication are authenticated to each other. The request could be sent without further signing or any other special requirements.

It seems to me that you've implemented something already offered for free with the traditional SSL handshake. And the validation key is a substitution of the CA-siging-the-certificate-to-be-presented-by-the-client stuff.

What am I seeing wrong here?

Best Regards

Miguel Cabeça

Adam

On Wed, Feb 10, 2010 at 8:24 AM, Adam Jacob adam@opscode.com wrote:

Not reinventing, using directly in the signature process. The SSH key
metaphor seems to resonate for most people.

If you look at the EC2 API auth, this is very similar, but with
public/private keys rather than a shared secret.

Sent from my iPhone

On Feb 10, 2010, at 7:30 AM, Miguel Cabeça cabeca@ist.utl.pt wrote:

Hi,

While reading the new Authentication page on Chef wiki [1] it seemed to me
that it's very similar to authentication with SSL client certificates.
Are you somehow reinventing the wheel or is there some obvious benefit
that I'm not aware of?

Best Regards

Miguel Cabeça

[1[ http://wiki.opscode.com/display/chef/Authentication

--
Opscode, Inc.
Adam Jacob, CTO
T: (206) 508-7449 E: adam@opscode.com

If i remember correctly, ssl uses the handshake in order to establish
a secret session key for the entirety of the HTTP session. Session key
encrypts the network traffic at the level of the TCP connection.

I think Adam is right not to copy verbatim the ssl system, because its
used for a different purpose. The way the hash is passed (unencrypted)
within an http x-header is clearly a different mechanism to the way
SSL encrypts its information. For example you might choose to encrypt
all chef traffic over SSL in addition to the chef authentication
mechanism. (if i've understood it right?)

Most importantly, the chef authentication mechanism identifies the
originator of the api call by its client ID and allows chef to
determine the level of client privileges allowed. If the client ID
isn't registered in couchdb and known to the chef-server, then the api
request is rejected.

The explanation given in the wiki seems clear enough. If you also have
existing knowledge about SSL Certificates side of things, then it can
be more confusing to explain things in terms of SSL / referring back
to SSL. Saying that chef calls out to the openssl library is fair
enough though. Because, well. Its good to be familiar with the openssl
library.

:slight_smile:

On Wed, Feb 10, 2010 at 9:59 PM, Miguel Cabeça cabeca@ist.utl.pt wrote:

It seems to me that you've implemented something already offered for free with the traditional SSL handshake. And the validation key is a substitution of the CA-siging-the-certificate-to-be-presented-by-the-client stuff.

What am I seeing wrong here?

Best Regards

Miguel Cabeça

On Wed, Feb 10, 2010 at 1:59 PM, Miguel Cabeça cabeca@ist.utl.pt wrote:

That's precisely what presenting a client certificate does in establishing an SSL connection. Part of the SSL handshaking involves the client signing something with his private key for the other side to verify with the client public key (present in the certificate offered to the other side). After the handshake, the secure channel guarantees that both sides of the communication are authenticated to each other. The request could be sent without further signing or any other special requirements.

It seems to me that you've implemented something already offered for free with the traditional SSL handshake. And the validation key is a substitution of the CA-siging-the-certificate-to-be-presented-by-the-client stuff.

What am I seeing wrong here?

dreamcat four does a fine job in his reply, but I'm going to reiterate it.

The difference here is problem domain, applicability to the task, and
security profile. What we are doing is digitally signing the
payload and shape of each request - confirming that it was, in
fact, authored by someone in control of the given client key. This is
different from (but uses a similar mechanism to) SSL/TLS, but it's
happening at a different layer. SSL/TLS says that, on the protocol
level, you have secure channel between two sides (who may or may not
mutually authenticate one another.) That's great, and you should do
that.

The new authentication mechanism within Chef is saying that, at the
application level, we are certain that the request as drafted and
signed by the origin
is the request that we receive on the other
side. We are digitally signing the content and shape of each request,
giving us another layer of useful security - against things like
replay attacks, etc.

Additionally, this mechanism can be re-used outside of the transport
layer: if you want to authenticate a different kind of payload, that's
easily possible here.

We're not re-inventing the wheel - we're simply saying that there are
two layers of security here: transport layer security and application
layer security. Chef provides for the application layer security, and
for most people, SSL/TLS handles transport layer security.

So rather than thinking about this as "rebuilding ssl", it's actually
"implementing per-request digital signatures".

If what we did was to use SSL certificates as our authentication
layer, it assumes that all traffic must be traversing SSL. Our
answer is that all traffic must be digitally signed with a trusted
private key - and it may also traverse an encrypted (and perhaps
authenticated) transport layer. Imagine wrapping Chef requests in a
new protocol (like AMQP) and you'll start to see the idea - as long as
the request is signed, we can authenticate it at the application
level.

It's not 'this or SSL', it's 'this and SSL'.

Adam

--
Opscode, Inc.
Adam Jacob, CTO
T: (206) 508-7449 E: adam@opscode.com

Guys,

The difference here is problem domain, applicability to the task, and
security profile. What we are doing is digitally signing the
payload and shape of each request - confirming that it was, in
fact, authored by someone in control of the given client key. This is
different from (but uses a similar mechanism to) SSL/TLS, but it's
happening at a different layer. SSL/TLS says that, on the protocol
level, you have secure channel between two sides (who may or may not
mutually authenticate one another.) That's great, and you should do
that.

I fully understand this. When you say you are signing the request (payload and shape) you are in fact authenticating the source of the request.
When you establish a TLS connection with client certificates you are creating an encrypted AND authenticated channel. Everything that traverses that channel can be assumed to be an authenticated payload.
So, in both cases you have authentication of the payload, albeit on different levels (transport level vs application level)

The new authentication mechanism within Chef is saying that, at the
application level, we are certain that the request as drafted and
signed by the origin
is the request that we receive on the other
side. We are digitally signing the content and shape of each request,
giving us another layer of useful security - against things like
replay attacks, etc.

What for? If you already have an authenticated TLS connection, signing the payload inside that channel gives you nothing more but complexity and overhead. Everything is taken care of by the security layer, man-in-the-middle attacks, replay attacks, etc. If you do not have an authenticated TLS connection, signing the payload gives you the necessary authentication of the client, and you have to implement additional protections against the more common attacks.

We're not re-inventing the wheel - we're simply saying that there are
two layers of security here: transport layer security and application
layer security. Chef provides for the application layer security, and
for most people, SSL/TLS handles transport layer security.

What I'm trying to say here is that using payload signing INSIDE an authenticated TLS channel gives you nothing additionally in terms of authentication. The TLS channel can give you encryption, but can also give you optional authentication.
This last feature of TLS could have been used. Instead you are suggesting the use of the encryption feature of TLS (ignoring the authentication feature) and implementing authentication yourself of the payload. Am I making any sense?

If what we did was to use SSL certificates as our authentication
layer, it assumes that all traffic must be traversing SSL. Our
answer is that all traffic must be digitally signed with a trusted
private key - and it may also traverse an encrypted (and perhaps
authenticated) transport layer. Imagine wrapping Chef requests in a
new protocol (like AMQP) and you'll start to see the idea - as long as
the request is signed, we can authenticate it at the application
level.

Or you could use TLS to protect AMQP[1], or use TLS to protect SMTP, or use TLS to protect any other protocol you choose besides HTTP to transmit Chef requests.

Most importantly, the chef authentication mechanism identifies the
originator of the api call by its client ID and allows chef to
determine the level of client privileges allowed. If the client ID
isn't registered in couchdb and known to the chef-server, then the api
request is rejected.

Using a client certificate with TLS the client could be identified by the CN of that certificate, like any browser that verifies the CN of the certificate presented by the server to match the hostname being accessed. It's the same process.

Don't get me wrong on this, I'll use this authentication method when 0.8 is released, but I think It was an unnecessary development effort.

Just my 2¢

Best Regards

Miguel Cabeça

[1] http://www.amqp.org/confluence/display/AMQP/AMQP1.0+SIG

On Thu, Feb 11, 2010 at 4:21 PM, Miguel Cabeça cabeca@ist.utl.pt wrote:

Using a client certificate with TLS the client could be identified by the CN of that certificate, like any browser that verifies the CN of the certificate presented by the server to match the hostname being accessed. It's the same process.

Don't get me wrong on this, I'll use this authentication method when 0.8 is released, but I think It was an unnecessary development effort.

Just my 2¢

Best Regards

Miguel Cabeça

Hmm, you sound like you know TLS very well. Bear in mind however that
the Mixlib-Authentication gem (which is what chef 0.8 now uses) is
just 2 pages of Ruby in its entirety. I can see no evidence that the
TLS based solution you are suggesting would not be substantially more
complex, given that Opscode's implementation is already so incredibly
neat and concise.

You'd might be hard pressed to find others who would agree that TLS is
a good mechanism to use here.

dreamcat4
dreamcat4@gmail.com

On Thu, Feb 11, 2010 at 8:21 AM, Miguel Cabeça cabeca@ist.utl.pt wrote:

I fully understand this. When you say you are signing the request (payload and shape) you are in fact authenticating the source of the request.
When you establish a TLS connection with client certificates you are creating an encrypted AND authenticated channel. Everything that traverses that channel can be assumed to be an authenticated payload.
So, in both cases you have authentication of the payload, albeit on different levels (transport level vs application level)

Sure - what happens to the payload after it has left the TLS
connection? To use SMTP as an example, if you want to allow arbitrary
recipients of an email to verify it was sent by a particular
individual, you do not trust the transport layer security that allowed
the message to be injected. You digitally sign the message itself,
and allow any recipient to verify the message with the signers public
key.

It's not crazy to imagine a world where requests are accepted via
HTTPS, and then get relayed to their final destination via AMQP. By
relying on TLS, you make it very difficult for that final endpoint to
verify that the message it received is the one sent by the original
endpoint.

The source of our disagreement is that I don't think we should force
decisions about the transport layer in order to gain a perceived
benefit of simplicity in the application layer. By signing each
request, we gain the ability to, at any link in the chain, verify that
the request has not been tampered with. We gain a huge amount of
application architecture flexibility, and it's applicable to a much
wider range of problems.

What for? If you already have an authenticated TLS connection, signing the payload inside that channel gives you nothing more but complexity and overhead. Everything is taken care of by the security layer, man-in-the-middle attacks, replay attacks, etc. If you do not have an authenticated TLS connection, signing the payload gives you the necessary authentication of the client, and you have to implement additional protections against the more common attacks.

Again, you're taking assumptions about the transport layer and forcing
them on the application architecture. If you want to ensure that each
Chef client uses an authenticated TLS connection to talk to the Chef
Server, you should absolutely do that.

I get that, for your use case, you don't see any practical benefit
to not just using a client-authenticated TLS handshake for
authentication. I, on the other hand, see a bunch of use cases where
there is practical benefit to not using client-authenticated TLS
handshakes for authentication.

What I'm trying to say here is that using payload signing INSIDE an authenticated TLS channel gives you nothing additionally in terms of authentication. The TLS channel can give you encryption, but can also give you optional authentication.
This last feature of TLS could have been used. Instead you are suggesting the use of the encryption feature of TLS (ignoring the authentication feature) and implementing authentication yourself of the payload. Am I making any sense?

Yes - and my response is that, if we did what you are saying, we would
be constraining the way requests can be received and verified by the
system to those that rely on point-to-point TLS connections. While
that is true today in many cases, it is already not true in many Chef
deployments, and may not be true in many future ones.

Or you could use TLS to protect AMQP[1], or use TLS to protect SMTP, or use TLS to protect any other protocol you choose besides HTTP to transmit Chef requests.

You're assuming a point-to-point, one client to single host
architecture. You can build an application that relies on transport
layer authentication for security, and by doing so you add the
constraint that it must have all communication occur over that
transport layer. By digitally signing each request, we gain the
flexibility of being able to re-use that functionality across any
transport layer, to serialize and distribute requests in any way we
see fit, and to continually ensure that the payload is exactly as it
was created by the signing client
, regardless of intervening
transport layers.

Imagine a scenario where you start building orchestration
functionality around Chef, where each client is gossiping about it's
current status ("I'm at step one! I'm at step two! I'm the
master!"). In a world where we only had TLS for client
authentication, each of those messages would need to be either sent
via a specific point-to-point connection with every other member of
the quorum, or coordinated through a central server.

In a world where each client signs their own payloads, you have lots
of options. The clients themselves can gossip about their public
keys, they can sign the payload of each gossip request, and they can
do it via multi-cast.

That's just one example - there are many more.

Don't get me wrong on this, I'll use this authentication method when 0.8 is released, but I think It was an unnecessary development effort.

I agree that it might not have been necessary for the use case you
have foremost in your mind (which is a single chef client
communicating with a remote chef server over HTTPS,) but I think it
was absolutely necessary for a huge number of other use cases.

Adam

--
Opscode, Inc.
Adam Jacob, CTO
T: (206) 508-7449 E: adam@opscode.com

It's not crazy to imagine a world where requests are accepted via
HTTPS, and then get relayed to their final destination via AMQP. By
relying on TLS, you make it very difficult for that final endpoint to
verify that the message it received is the one sent by the original
endpoint.

The source of our disagreement is that I don't think we should force
decisions about the transport layer in order to gain a perceived
benefit of simplicity in the application layer. By signing each
request, we gain the ability to, at any link in the chain, verify that
the request has not been tampered with. We gain a huge amount of
application architecture flexibility, and it's applicable to a much
wider range of problems.

You're assuming a point-to-point, one client to single host
architecture. You can build an application that relies on transport
layer authentication for security, and by doing so you add the
constraint that it must have all communication occur over that
transport layer. By digitally signing each request, we gain the
flexibility of being able to re-use that functionality across any
transport layer, to serialize and distribute requests in any way we
see fit, and to continually ensure that the payload is exactly as it
was created by the signing client
, regardless of intervening
transport layers.

Imagine a scenario where you start building orchestration
functionality around Chef, where each client is gossiping about it's
current status ("I'm at step one! I'm at step two! I'm the
master!"). In a world where we only had TLS for client
authentication, each of those messages would need to be either sent
via a specific point-to-point connection with every other member of
the quorum, or coordinated through a central server.

In a world where each client signs their own payloads, you have lots
of options. The clients themselves can gossip about their public
keys, they can sign the payload of each gossip request, and they can
do it via multi-cast.

I agree that it might not have been necessary for the use case you
have foremost in your mind (which is a single chef client
communicating with a remote chef server over HTTPS,) but I think it
was absolutely necessary for a huge number of other use cases.

Thank you for the time you spent explaining this to me. I finally got the benefit of this approach. If you'd only mentioned "payload relaying" and "end-to-end authentication through multiple hop communication" earlier :slight_smile:

Best Regards

Miguel Cabeça

On Thu, Feb 11, 2010 at 10:44 AM, Miguel Cabeça cabeca@ist.utl.pt wrote:

Thank you for the time you spent explaining this to me. I finally got the benefit of this approach. If you'd only mentioned "payload relaying" and "end-to-end authentication through multiple hop communication" earlier :slight_smile:

Thank you for the time you spent challenging the decision - the fact
that you care enough about Chef (and it's future) to weigh in is one
of the many reasons you're a hugely valuable member of our bourgeoning
community. Please do it again next time you see us potentially making
a mistake.

And, for the record, I'm adding a section to the wiki that uses the
words "payload relaying" and "end-to-end authentication through
multiple hop communication" right now. :slight_smile:

Adam

--
Opscode, Inc.
Adam Jacob, CTO
T: (206) 508-7449 E: adam@opscode.com