Error downloading cookbooks in Chef 11


#1

Ohai!

I’m getting random failures when using the REST api to download
cookbooks in Chef 11.0.6.

I am getting the url of each resource, and performing a GET request to
download its contents, but sometimes (and without a reproducible
pattern), I get a 403 response saying that the signature is not the
expected one.

The same code is working fine against Chef 0.10.8 and Enterprise
(Hosted) Chef, but in Chef 11 the failures appear in different
resources each time: sometimes a resource is downloaded, sometimes
downloading the same resource results in a 403 error.

I’ve taken a look at the chef-server logs (with chef-server-ctl tail),
but I haven’t seen any error or details that could help figuring out
what could be going on.

Is there any known issue about this?

Any help would be much appreciated.

Thanks,

Ignasi


#2

I just added some more cookbooks to my Hosted Chef account, and I get
the same failure.

Is there anything that has changed in the authentication checks
between Chef 0.10 and Chef 11 and the latest Hosted Chef version? Chef
0.10 keeps working fine.

Thank you,

Ignasi

On 26 August 2013 16:10, Ignasi ignasi.barrera@gmail.com wrote:

Ohai!

I’m getting random failures when using the REST api to download
cookbooks in Chef 11.0.6.

I am getting the url of each resource, and performing a GET request to
download its contents, but sometimes (and without a reproducible
pattern), I get a 403 response saying that the signature is not the
expected one.

The same code is working fine against Chef 0.10.8 and Enterprise
(Hosted) Chef, but in Chef 11 the failures appear in different
resources each time: sometimes a resource is downloaded, sometimes
downloading the same resource results in a 403 error.

I’ve taken a look at the chef-server logs (with chef-server-ctl tail),
but I haven’t seen any error or details that could help figuring out
what could be going on.

Is there any known issue about this?

Any help would be much appreciated.

Thanks,

Ignasi


#3

Hello.

Could you provide debug-log of chef-client?

26.08.2013, в 23:52, Ignasi ignasi.barrera@gmail.com написал(а):

I just added some more cookbooks to my Hosted Chef account, and I get
the same failure.

Is there anything that has changed in the authentication checks
between Chef 0.10 and Chef 11 and the latest Hosted Chef version? Chef
0.10 keeps working fine.

Thank you,

Ignasi

On 26 August 2013 16:10, Ignasi ignasi.barrera@gmail.com wrote:

Ohai!

I’m getting random failures when using the REST api to download
cookbooks in Chef 11.0.6.

I am getting the url of each resource, and performing a GET request to
download its contents, but sometimes (and without a reproducible
pattern), I get a 403 response saying that the signature is not the
expected one.

The same code is working fine against Chef 0.10.8 and Enterprise
(Hosted) Chef, but in Chef 11 the failures appear in different
resources each time: sometimes a resource is downloaded, sometimes
downloading the same resource results in a 403 error.

I’ve taken a look at the chef-server logs (with chef-server-ctl tail),
but I haven’t seen any error or details that could help figuring out
what could be going on.

Is there any known issue about this?

Any help would be much appreciated.

Thanks,

Ignasi


#4

Thanks for the response,

I am not using chef-client, I’m using directly the REST api from Java.

I am the maintainer of jclouds-chef [1], a Java library to access
Chef, and just started testing it against Enterprise Chef and Chef 11.
It used to work with Chef 10 and Hosted Chef, and I am just trying to
figure out if there is something that has changed in the signature
verification that is making it fail now with Chef 11 and Enterprise
Chef.

The signature generation is generic [2] and the same code is used to
sign every request, so I don’t understand why I only get failures when
trying to download the contents of a file (and why it only fails in
Chef 11 and Hosted Chef).

I can provide the logs of the generated requests and the obtained
responses, if it helps.

Thanks for your help, it is very much appreciated,

Ignasi

[1] https://github.com/jclouds/jclouds-chef
[2] https://github.com/jclouds/jclouds-chef/blob/master/core/src/main/java/org/jclouds/chef/filters/SignedHeaderAuth.java

On 26 August 2013 16:56, Антон Баранов anton@atalanta-systems.com wrote:

Hello.

Could you provide debug-log of chef-client?

26.08.2013, в 23:52, Ignasi ignasi.barrera@gmail.com написал(а):

I just added some more cookbooks to my Hosted Chef account, and I get
the same failure.

Is there anything that has changed in the authentication checks
between Chef 0.10 and Chef 11 and the latest Hosted Chef version? Chef
0.10 keeps working fine.

Thank you,

Ignasi

On 26 August 2013 16:10, Ignasi ignasi.barrera@gmail.com wrote:

Ohai!

I’m getting random failures when using the REST api to download
cookbooks in Chef 11.0.6.

I am getting the url of each resource, and performing a GET request to
download its contents, but sometimes (and without a reproducible
pattern), I get a 403 response saying that the signature is not the
expected one.

The same code is working fine against Chef 0.10.8 and Enterprise
(Hosted) Chef, but in Chef 11 the failures appear in different
resources each time: sometimes a resource is downloaded, sometimes
downloading the same resource results in a 403 error.

I’ve taken a look at the chef-server logs (with chef-server-ctl tail),
but I haven’t seen any error or details that could help figuring out
what could be going on.

Is there any known issue about this?

Any help would be much appreciated.

Thanks,

Ignasi


#5

Hi Ignasi,

ignasi.barrera@gmail.com writes:

I am the maintainer of jclouds-chef [1], a Java library to access
Chef, and just started testing it against Enterprise Chef and Chef 11.
It used to work with Chef 10 and Hosted Chef, and I am just trying to
figure out if there is something that has changed in the signature
verification that is making it fail now with Chef 11 and Enterprise
Chef.

The signature generation is generic [2] and the same code is used to
sign every request, so I don’t understand why I only get failures when
trying to download the contents of a file (and why it only fails in
Chef 11 and Hosted Chef).

There are a number of significant changes introduced in Chef 11 and more
recently deployed into Hosted Enterprise Chef (HEC).

Here’s my understanding of the problem you’re seeing, please let me know
if I’m off:

  1. You can use jclouds-chef to make some successful API requests against
    Open Source Chef 11 Server (OSC) and against your org in Hosted
    Enterprise Chef.

  2. You are seeing intermittent 403 errors when doing GETs of cookbook
    content.

I can provide the logs of the generated requests and the obtained
responses, if it helps.

It would help to have examples of the type of URL you are attempting to
access (you can put in placeholders to maintain privacy if you will be
posting this to the list).

Similarly, associated responses would be helpful.

Cookbook content, by which I mean the files in a cookbook stored by
checksum, are handled outside of the core server in both OSC 11 and HEC.

OSC 11 uses a newly added component called “bookshelf” to serve the
cookbook content. It provides a think S3 API.

HEC stores cookbook content in AWS S3.

In both cases, when fetching a cookbook, the client receives a cookbook
version object containing pre-signed URLs that it can use to fetch the
cookbook content out of either bookshelf or S3. Although it is OK to
sign those requests like other API requests, both bookshelf and S3 use
S3-style authentication. If these are the requests that are failing,
inspecting the response body might give us a clue of where to look.

  • seth


Seth Falcon | Development Lead | Opscode | @sfalcon


#6

Hi Seth,

That is exactly the case. The failures occur when requesting those files.

I’ve been investigating a bit more and I’ve found a pattern for the
failure. I’ve seen that the signature validation only fails when the
URL of the resource has URL encoded characters in the "Signature"
query parameter. For example, the following two URLs are failing:

In Enterprise Chef:
https://s3-external-1.amazonaws.com:443/opscode-platform-production-data/organization-/checksum-f111ba97dc5160e26eee6473dc8e86d5?AWSAccessKeyId=&Expires=1377590794&Signature=QPnT5XLRam%2Br8p09hXra5N3jd6A%3D

In Chef 11:
https://ubuntu64-1104.defaultdomain:443/bookshelf/organization-00000000000000000000000000000000/checksum-818a38d41bd95ebec58b6d430eabd7cb?AWSAccessKeyId=&Expires=1377588824&Signature=31OUx%2BUBdZSyzzA9xpHlFIOxjHc%3D

Both have the ‘%2B’ character in the signature. I’ve also seen URLs
with a ‘%20’ that are also failing. However, if the signature does not
have those URL encoded characters, it just works fine (the padding
’%3D’ character is not causing any failure). Here is an example URL
that is working:

https://ubuntu64-1104.defaultdomain:443/bookshelf/organization-00000000000000000000000000000000/checksum-a7daae249eca97adcd4b3617c745940d?AWSAccessKeyId=&Expires=1377589454&Signature=Mp7TFuAyObhtsjgE7wLSv3Sm6WU%3D

Is there something special that should be done to treat those encoded
characters when signing? I’ve been taking a look at the knife and mix
lib-authentication source code (a knife cookbook download works fine)
but still haven’t found anything relevant.

Many thanks for your help,

Ignasi

On 27 August 2013 00:18, Seth Falcon seth@opscode.com wrote:

Hi Ignasi,

ignasi.barrera@gmail.com writes:

I am the maintainer of jclouds-chef [1], a Java library to access
Chef, and just started testing it against Enterprise Chef and Chef 11.
It used to work with Chef 10 and Hosted Chef, and I am just trying to
figure out if there is something that has changed in the signature
verification that is making it fail now with Chef 11 and Enterprise
Chef.

The signature generation is generic [2] and the same code is used to
sign every request, so I don’t understand why I only get failures when
trying to download the contents of a file (and why it only fails in
Chef 11 and Hosted Chef).

There are a number of significant changes introduced in Chef 11 and more
recently deployed into Hosted Enterprise Chef (HEC).

Here’s my understanding of the problem you’re seeing, please let me know
if I’m off:

  1. You can use jclouds-chef to make some successful API requests against
    Open Source Chef 11 Server (OSC) and against your org in Hosted
    Enterprise Chef.

  2. You are seeing intermittent 403 errors when doing GETs of cookbook
    content.

I can provide the logs of the generated requests and the obtained
responses, if it helps.

It would help to have examples of the type of URL you are attempting to
access (you can put in placeholders to maintain privacy if you will be
posting this to the list).

Similarly, associated responses would be helpful.

Cookbook content, by which I mean the files in a cookbook stored by
checksum, are handled outside of the core server in both OSC 11 and HEC.

OSC 11 uses a newly added component called “bookshelf” to serve the
cookbook content. It provides a think S3 API.

HEC stores cookbook content in AWS S3.

In both cases, when fetching a cookbook, the client receives a cookbook
version object containing pre-signed URLs that it can use to fetch the
cookbook content out of either bookshelf or S3. Although it is OK to
sign those requests like other API requests, both bookshelf and S3 use
S3-style authentication. If these are the requests that are failing,
inspecting the response body might give us a clue of where to look.

  • seth


Seth Falcon | Development Lead | Opscode | @sfalcon


#7

Hi again,

Finally I’ve just been able to fix the issue. If I do not encode the
characters when performing the GET request, it just works.

Thank you all for the help!

ignasi

On 27 August 2013 09:32, Ignasi ignasi.barrera@gmail.com wrote:

Hi Seth,

That is exactly the case. The failures occur when requesting those files.

I’ve been investigating a bit more and I’ve found a pattern for the
failure. I’ve seen that the signature validation only fails when the
URL of the resource has URL encoded characters in the "Signature"
query parameter. For example, the following two URLs are failing:

In Enterprise Chef:
https://s3-external-1.amazonaws.com:443/opscode-platform-production-data/organization-/checksum-f111ba97dc5160e26eee6473dc8e86d5?AWSAccessKeyId=&Expires=1377590794&Signature=QPnT5XLRam%2Br8p09hXra5N3jd6A%3D

In Chef 11:
https://ubuntu64-1104.defaultdomain:443/bookshelf/organization-00000000000000000000000000000000/checksum-818a38d41bd95ebec58b6d430eabd7cb?AWSAccessKeyId=&Expires=1377588824&Signature=31OUx%2BUBdZSyzzA9xpHlFIOxjHc%3D

Both have the ‘%2B’ character in the signature. I’ve also seen URLs
with a ‘%20’ that are also failing. However, if the signature does not
have those URL encoded characters, it just works fine (the padding
’%3D’ character is not causing any failure). Here is an example URL
that is working:

https://ubuntu64-1104.defaultdomain:443/bookshelf/organization-00000000000000000000000000000000/checksum-a7daae249eca97adcd4b3617c745940d?AWSAccessKeyId=&Expires=1377589454&Signature=Mp7TFuAyObhtsjgE7wLSv3Sm6WU%3D

Is there something special that should be done to treat those encoded
characters when signing? I’ve been taking a look at the knife and mix
lib-authentication source code (a knife cookbook download works fine)
but still haven’t found anything relevant.

Many thanks for your help,

Ignasi

On 27 August 2013 00:18, Seth Falcon seth@opscode.com wrote:

Hi Ignasi,

ignasi.barrera@gmail.com writes:

I am the maintainer of jclouds-chef [1], a Java library to access
Chef, and just started testing it against Enterprise Chef and Chef 11.
It used to work with Chef 10 and Hosted Chef, and I am just trying to
figure out if there is something that has changed in the signature
verification that is making it fail now with Chef 11 and Enterprise
Chef.

The signature generation is generic [2] and the same code is used to
sign every request, so I don’t understand why I only get failures when
trying to download the contents of a file (and why it only fails in
Chef 11 and Hosted Chef).

There are a number of significant changes introduced in Chef 11 and more
recently deployed into Hosted Enterprise Chef (HEC).

Here’s my understanding of the problem you’re seeing, please let me know
if I’m off:

  1. You can use jclouds-chef to make some successful API requests against
    Open Source Chef 11 Server (OSC) and against your org in Hosted
    Enterprise Chef.

  2. You are seeing intermittent 403 errors when doing GETs of cookbook
    content.

I can provide the logs of the generated requests and the obtained
responses, if it helps.

It would help to have examples of the type of URL you are attempting to
access (you can put in placeholders to maintain privacy if you will be
posting this to the list).

Similarly, associated responses would be helpful.

Cookbook content, by which I mean the files in a cookbook stored by
checksum, are handled outside of the core server in both OSC 11 and HEC.

OSC 11 uses a newly added component called “bookshelf” to serve the
cookbook content. It provides a think S3 API.

HEC stores cookbook content in AWS S3.

In both cases, when fetching a cookbook, the client receives a cookbook
version object containing pre-signed URLs that it can use to fetch the
cookbook content out of either bookshelf or S3. Although it is OK to
sign those requests like other API requests, both bookshelf and S3 use
S3-style authentication. If these are the requests that are failing,
inspecting the response body might give us a clue of where to look.

  • seth


Seth Falcon | Development Lead | Opscode | @sfalcon


#8

Hi,

ignasi.barrera@gmail.com writes:

Finally I’ve just been able to fix the issue. If I do not encode the
characters when performing the GET request, it just works.

Glad to hear that you got things working. Was the code double encoding
before? That would explain why removing the encoding fixed things.

Cheers,

  • seth


Seth Falcon | Development Lead | Opscode | @sfalcon


#9

That’s exactly what happened :slight_smile:

Thx again!

On 28 August 2013 18:03, Seth Falcon seth@opscode.com wrote:

Hi,

ignasi.barrera@gmail.com writes:

Finally I’ve just been able to fix the issue. If I do not encode the
characters when performing the GET request, it just works.

Glad to hear that you got things working. Was the code double encoding
before? That would explain why removing the encoding fixed things.

Cheers,

  • seth


Seth Falcon | Development Lead | Opscode | @sfalcon