Seeing a weird 403 Error while running recipe

Chef Infra (archive)
1

On the client, when the recipe runs, I get this error:

  • Net::HTTPServerException occurred in chef run:
    template[/etc/sysconfig/rsyslog] (server::default line 14) had an error:
    Net::HTTPServerException: 403 “Forbidden”

On the server side, I see this:

10.50.69.179 - - [04/Dec/2014:15:19:45 -0500] “GET
/bookshelf/organization-09088a3388934e9296242442edfbd52a/checksum-8d5acc7b2c778715c2263d11b0ab60ed?AWSAccessKeyId=249876d41aeba662b849413527a1370f906b5527&Expires=1417724019&Signature=6vDiSNhkV2WVAsrWEFYWyNPQTV8%3D
HTTP/1.1” 403 “0.002” 206 “-” “Chef Client/11.12.8 (ruby-1.9.3-p484;
ohai-7.0.4; x86_64-linux; +http://opscode.com)” “127.0.0.1:4321” "403"
“0.002” “11.12.8” “algorithm=sha1;version=1.0;” "macine1"
“2014-12-04T20:19:45Z” “2jmj7l5rSw0yVb/vlWAYkK/YBwk=” 1200

The second run seems to be fine. Anyone see anythign similar? How did you
resolve?

We are using ent server.
TIA

Jennifer Fountain
DevOPS

2

Hi,

The url you see getting a 403 is a signed bookshelf link that is only
valid for a set number of seconds. In this case, your link is valid
until the unix time 1417724019 which is

Thu, 04 Dec 2014 20:13:39 -0000

or

Thu, 04 Dec 2014 15:13:39 -0500

which is about 6 minutes before this request was made
(04/Dec/2014:15:19:45 -050). By default, certain files are downloaded
lazily during your chef run. Thus, if your chef run takes a long
time, the links for the files you got at the beginning of the run can
become invalid. To fix this you have a few options:

  1. Increase the time to live on the links. If you control your
    server, you can place the following in /etc/chef-server/chef-server.rb
    (/etc/opscode/chef-server.rb on 12):

    erchef['s3_url_ttl'] = 3600

and run

chef-server-ctl reconfigure

The value is in seconds, so you can set it to whatever you feel is
appropriate.

  1. Turn of lazy-loading in chef-client. By downloading all of the
    files at the beginning of the run, you often avoid this. To turn of
    lazy loading, put the following in /etc/chef/client.rb on your
    chef-client's:

    no_lazy_load true

  2. Find what parts of your chef-client run are slow, and make then
    faster. This is sometimes impossible :slight_smile:

I typically go with (1).

Cheers,

Steven

On Thu, Dec 4, 2014 at 9:05 PM, Jennifer Fountain jfountain@meetme.com wrote:

On the client, when the recipe runs, I get this error:

  • Net::HTTPServerException occurred in chef run:
    template[/etc/sysconfig/rsyslog] (server::default line 14) had an error:
    Net::HTTPServerException: 403 "Forbidden"

On the server side, I see this:

10.50.69.179 - - [04/Dec/2014:15:19:45 -0500] "GET
/bookshelf/organization-09088a3388934e9296242442edfbd52a/checksum-8d5acc7b2c778715c2263d11b0ab60ed?AWSAccessKeyId=249876d41aeba662b849413527a1370f906b5527&Expires=1417724019&Signature=6vDiSNhkV2WVAsrWEFYWyNPQTV8%3D
HTTP/1.1" 403 "0.002" 206 "-" "Chef Client/11.12.8 (ruby-1.9.3-p484;
ohai-7.0.4; x86_64-linux; +http://opscode.com)" "127.0.0.1:4321" "403"
"0.002" "11.12.8" "algorithm=sha1;version=1.0;" "macine1"
"2014-12-04T20:19:45Z" "2jmj7l5rSw0yVb/vlWAYkK/YBwk=" 1200

The second run seems to be fine. Anyone see anythign similar? How did you
resolve?

We are using ent server.

TIA

Jennifer Fountain
DevOPS

3

Oh Thank you so much!

On Thu, Dec 4, 2014 at 6:39 PM, Steven Danna steve@opscode.com wrote:

Hi,

The url you see getting a 403 is a signed bookshelf link that is only
valid for a set number of seconds. In this case, your link is valid
until the unix time 1417724019 which is

Thu, 04 Dec 2014 20:13:39 -0000

or

Thu, 04 Dec 2014 15:13:39 -0500

which is about 6 minutes before this request was made
(04/Dec/2014:15:19:45 -050). By default, certain files are downloaded
lazily during your chef run. Thus, if your chef run takes a long
time, the links for the files you got at the beginning of the run can
become invalid. To fix this you have a few options:

  1. Increase the time to live on the links. If you control your
    server, you can place the following in /etc/chef-server/chef-server.rb
    (/etc/opscode/chef-server.rb on 12):

    erchef['s3_url_ttl'] = 3600

and run

chef-server-ctl reconfigure

The value is in seconds, so you can set it to whatever you feel is
appropriate.

  1. Turn of lazy-loading in chef-client. By downloading all of the
    files at the beginning of the run, you often avoid this. To turn of
    lazy loading, put the following in /etc/chef/client.rb on your
    chef-client's:

    no_lazy_load true

  2. Find what parts of your chef-client run are slow, and make then
    faster. This is sometimes impossible :slight_smile:

I typically go with (1).

Cheers,

Steven

On Thu, Dec 4, 2014 at 9:05 PM, Jennifer Fountain jfountain@meetme.com
wrote:

On the client, when the recipe runs, I get this error:

  • Net::HTTPServerException occurred in chef run:
    template[/etc/sysconfig/rsyslog] (server::default line 14) had an error:
    Net::HTTPServerException: 403 "Forbidden"

On the server side, I see this:

10.50.69.179 - - [04/Dec/2014:15:19:45 -0500] "GET

/bookshelf/organization-09088a3388934e9296242442edfbd52a/checksum-8d5acc7b2c778715c2263d11b0ab60ed?AWSAccessKeyId=249876d41aeba662b849413527a1370f906b5527&Expires=1417724019&Signature=6vDiSNhkV2WVAsrWEFYWyNPQTV8%3D

HTTP/1.1" 403 "0.002" 206 "-" "Chef Client/11.12.8 (ruby-1.9.3-p484;
ohai-7.0.4; x86_64-linux; +http://opscode.com)" "127.0.0.1:4321" "403"
"0.002" "11.12.8" "algorithm=sha1;version=1.0;" "macine1"
"2014-12-04T20:19:45Z" "2jmj7l5rSw0yVb/vlWAYkK/YBwk=" 1200

The second run seems to be fine. Anyone see anythign similar? How did
you
resolve?

We are using ent server.

TIA

Jennifer Fountain
DevOPS

--

Jennifer Fountain
DevOPS