# Large cookbook uploads fail with "unauthorized" after data upload

**URL:** <https://discourse.chef.io/t/large-cookbook-uploads-fail-with-unauthorized-after-data-upload/2597>\
**Category:** Chef Infra (archive)\
**Created:** [March 24, 2012, 12:54am UTC](https://discourse.chef.io/t/large-cookbook-uploads-fail-with-unauthorized-after-data-upload/2597 "2012-03-24T00:54:41Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![jasonjwwilliams](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/jasonjwwilliams/32/101_2.png) [@jasonjwwilliams](https://discourse.chef.io/u/jasonjwwilliams)\
**Post date:** [March 24, 2012, 12:54am UTC](https://discourse.chef.io/t/large-cookbook-uploads-fail-with-unauthorized-after-data-upload/2597/1 "2012-03-24T00:54:41Z")

</div>

Is there a way to get a cookbook with several very large files to  
upload via knife without this happening:

Uploading testcookbook [0.5.0]  
\<-----Pauses here for about 30 mins while the cookbook uploads 90MB  
ERROR: Failed to upload #Chef::CookbookVersion:0x10178fdf0 : 401 Unauthorized  
{“error”:[“Failed to authenticate. Please synchronize the clock on  
your client”]}  
/Library/Ruby/Gems/1.8/gems/rest-client-1.6.1/lib/restclient/abstract\_response.rb:48:in  
`return!': 401 Unauthorized (RestClient::Request::Unauthorized) from /Library/Ruby/Gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:220:in`process\_result’  
from /Library/Ruby/Gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:169:in  
`transmit' from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:543:in`start’  
from /Library/Ruby/Gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:166:in  
`transmit' from /Library/Ruby/Gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:60:in`execute’  
from /Library/Ruby/Gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:31:in  
`execute' from /Library/Ruby/Gems/1.8/gems/rest-client-1.6.1/lib/restclient/resource.rb:72:in`put’  
from /Library/Ruby/Gems/1.8/gems/chef-0.10.8/lib/chef/cookbook\_uploader.rb:134:in  
`uploader_function_for' from /Library/Ruby/Gems/1.8/gems/chef-0.10.8/lib/chef/cookbook_uploader.rb:25:in`call’  
from /Library/Ruby/Gems/1.8/gems/chef-0.10.8/lib/chef/cookbook\_uploader.rb:25:in  
`setup_worker_threads' from /Library/Ruby/Gems/1.8/gems/chef-0.10.8/lib/chef/cookbook_uploader.rb:24:in`loop’  
from /Library/Ruby/Gems/1.8/gems/chef-0.10.8/lib/chef/cookbook\_uploader.rb:24:in  
`setup_worker_threads' from /Library/Ruby/Gems/1.8/gems/chef-0.10.8/lib/chef/cookbook_uploader.rb:23:in`initialize’  
from /Library/Ruby/Gems/1.8/gems/chef-0.10.8/lib/chef/cookbook\_uploader.rb:23:in  
`new' from /Library/Ruby/Gems/1.8/gems/chef-0.10.8/lib/chef/cookbook_uploader.rb:23:in`setup\_worker\_threads’  
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/custom\_require.rb:31:in  
`map' from /Library/Ruby/Gems/1.8/gems/chef-0.10.8/lib/chef/cookbook_uploader.rb:22:in`each’  
from /Library/Ruby/Gems/1.8/gems/chef-0.10.8/lib/chef/cookbook\_uploader.rb:22:in  
`map' from /Library/Ruby/Gems/1.8/gems/chef-0.10.8/lib/chef/cookbook_uploader.rb:22:in`setup\_worker\_threads’  
from /Library/Ruby/Gems/1.8/gems/chef-0.10.8/lib/chef/cookbook\_uploader.rb:69:in  
`upload_cookbook' from /Library/Ruby/Gems/1.8/gems/chef-0.10.8/lib/chef/knife/cookbook_upload.rb:170:in`upload’  
from /Library/Ruby/Gems/1.8/gems/chef-0.10.8/lib/chef/knife/cookbook\_upload.rb:103:in  
`run' from /Library/Ruby/Gems/1.8/gems/chef-0.10.8/lib/chef/knife/cookbook_upload.rb:94:in`each’  
from /Library/Ruby/Gems/1.8/gems/chef-0.10.8/lib/chef/knife/cookbook\_upload.rb:94:in  
`run' from /Library/Ruby/Gems/1.8/gems/chef-0.10.8/lib/chef/knife.rb:391:in`run\_with\_pretty\_exceptions’  
from /Library/Ruby/Gems/1.8/gems/chef-0.10.8/lib/chef/knife.rb:166:in `run' from /Library/Ruby/Gems/1.8/gems/chef-0.10.8/lib/chef/application/knife.rb:128:in`run’  
from /Library/Ruby/Gems/1.8/gems/chef-0.10.8/bin/knife:25  
from /usr/bin/knife:19:in `load’  
from /usr/bin/knife:19

---

<div class="post-metadata">

**Author:** ![Joshua\_Timberman](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/joshua_timberman/32/424_2.png) [@Joshua\_Timberman](https://discourse.chef.io/u/Joshua_Timberman)\
**Post date:** [March 24, 2012, 2:01am UTC](https://discourse.chef.io/t/large-cookbook-uploads-fail-with-unauthorized-after-data-upload/2597/2 "2012-03-24T02:01:28Z")

</div>

Ohai,

On Fri, Mar 23, 2012 at 6:54 PM, Jason J. W. Williams  
[jasonjwwilliams@gmail.com](mailto:jasonjwwilliams@gmail.com) wrote:

> Is there a way to get a cookbook with several very large files to  
> upload via knife without this happening:
> 
> Uploading testcookbook [0.5.0]  
> \<-----Pauses here for about 30 mins while the cookbook uploads 90MB  
> ERROR: Failed to upload #Chef::CookbookVersion:0x10178fdf0 : 401 Unauthorized  
> {"error":["Failed to authenticate. Please synchronize the clock on  
> your client"]}  
> /Library/Ruby/Gems/1.8/gems/rest-client-1.6.1/lib/restclient/abstract\_response.rb:48:in  
> `return!': 401 Unauthorized (RestClient::Request::Unauthorized)

Why are you putting several very large files in Chef? Surely there are  
more efficient ways to manage these files? It is likely that such  
files would take a long time to download when required by the client  
and likewise time out.

--  
Opscode, Inc  
Joshua Timberman, Technical Program Manager  
IRC, Skype, Twitter, Github: jtimberman

---

<div class="post-metadata">

**Author:** ![Joshua\_Miller](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/joshua_miller/32/212_2.png) [@Joshua\_Miller](https://discourse.chef.io/u/Joshua_Miller)\
**Post date:** [March 24, 2012, 2:48pm UTC](https://discourse.chef.io/t/large-cookbook-uploads-fail-with-unauthorized-after-data-upload/2597/3 "2012-03-24T14:48:06Z")

</div>

I think timberman was trying to direct you to something like this. For anything over a one or two meg I find it works much better.

[http://wiki.opscode.com/display/chef/Resources#Resources-RemoteFile](http://wiki.opscode.com/display/chef/Resources#Resources-RemoteFile)

Transfer a file only if the remote source has changed (uses http\_request resource)  
remote\_file "/tmp/couch.png" do  
source "[http://couchdb.apache.org/img/sketch.png](http://couchdb.apache.org/img/sketch.png)"  
action :nothing  
end

http\_request "HEAD #{[http://couchdb.apache.org/img/sketch.png](http://couchdb.apache.org/img/sketch.png)}" do  
message ""  
url [http://couchdb.apache.org/img/sketch.png](http://couchdb.apache.org/img/sketch.png)  
action :head  
if File.exists?("/tmp/couch.png")  
headers "If-Modified-Since" =\> File.mtime("/tmp/couch.png").httpdate  
end  
notifies :create, resources(:remote\_file =\> "/tmp/couch.png"), :immediately  
end

Joshua

On Mar 23, 2012, at 7:01 PM, Joshua Timberman [joshua@opscode.com](mailto:joshua@opscode.com) wrote:

> Ohai,
> 
> On Fri, Mar 23, 2012 at 6:54 PM, Jason J. W. Williams  
> [jasonjwwilliams@gmail.com](mailto:jasonjwwilliams@gmail.com) wrote:
> 
> > Is there a way to get a cookbook with several very large files to  
> > upload via knife without this happening:
> > 
> > Uploading testcookbook [0.5.0]  
> > \<-----Pauses here for about 30 mins while the cookbook uploads 90MB  
> > ERROR: Failed to upload #Chef::CookbookVersion:0x10178fdf0 : 401 Unauthorized  
> > {"error":["Failed to authenticate. Please synchronize the clock on  
> > your client"]}  
> > /Library/Ruby/Gems/1.8/gems/rest-client-1.6.1/lib/restclient/abstract\_response.rb:48:in  
> > `return!': 401 Unauthorized (RestClient::Request::Unauthorized)
> 
> Why are you putting several very large files in Chef? Surely there are  
> more efficient ways to manage these files? It is likely that such  
> files would take a long time to download when required by the client  
> and likewise time out.
> 
> --  
> Opscode, Inc  
> Joshua Timberman, Technical Program Manager  
> IRC, Skype, Twitter, Github: jtimberman

---

<div class="post-metadata">

**Author:** ![Geoff\_Meakin](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/geoff_meakin/32/718_2.png) [@Geoff\_Meakin](https://discourse.chef.io/u/Geoff_Meakin)\
**Post date:** [March 24, 2012, 9:13pm UTC](https://discourse.chef.io/t/large-cookbook-uploads-fail-with-unauthorized-after-data-upload/2597/4 "2012-03-24T21:13:43Z")

</div>

Hi all, I am on debian

doing:

package “foo” do  
action :install  
end

works fine in most cases.

Some packages however (I read somewhere there are 8 of them?) seem to be critical/core packages and trying to uninstall them mean that the normal way of installing under the covers:  
apt-get -y

just doesnt work. upstart is such a package as it uninstalls sysvinit.

Is there a standard way of automatically installing this using chef?

If I had to do a commandline version I would do this:

yes “Yes, do as I say!” | apt-get -y --force-yes upstart

But then this deviates somewhat from the nicely distribution-agnostic package chef dsl

Thanks

---

<div class="post-metadata">

**Author:** ![Joshua\_Miller](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/joshua_miller/32/212_2.png) [@Joshua\_Miller](https://discourse.chef.io/u/Joshua_Miller)\
**Post date:** [March 24, 2012, 9:33pm UTC](https://discourse.chef.io/t/large-cookbook-uploads-fail-with-unauthorized-after-data-upload/2597/5 "2012-03-24T21:33:31Z")

</div>

You could wrap the "options" attribute in a case statement?

[http://wiki.opscode.com/display/chef/Resources#Resources-Package](http://wiki.opscode.com/display/chef/Resources#Resources-Package)

Install Package with options  
package "debian-archive-keyring" do  
action :install  
options "--force-yes"  
end

On Mar 24, 2012, at 2:13 PM, Geoff Meakin [geoffmeakin@aciddevelopments.co.uk](mailto:geoffmeakin@aciddevelopments.co.uk) wrote:

> Hi all, I am on debian
> 
> doing:
> 
> package "foo" do  
> action :install  
> end
> 
> works fine in most cases.
> 
> Some packages however (I read somewhere there are 8 of them?) seem to be critical/core packages and trying to uninstall them mean that the normal way of installing under the covers:  
> apt-get -y
> 
> just doesnt work. upstart is such a package as it uninstalls sysvinit.
> 
> Is there a standard way of automatically installing this using chef?
> 
> If I had to do a commandline version I would do this:
> 
> yes "Yes, do as I say!" | apt-get -y --force-yes upstart
> 
> But then this deviates somewhat from the nicely distribution-agnostic package chef dsl
> 
> Thanks

---

<div class="post-metadata">

**Author:** ![Geoff\_Meakin](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/geoff_meakin/32/718_2.png) [@Geoff\_Meakin](https://discourse.chef.io/u/Geoff_Meakin)\
**Post date:** [March 25, 2012, 9:34am UTC](https://discourse.chef.io/t/large-cookbook-uploads-fail-with-unauthorized-after-data-upload/2597/6 "2012-03-25T09:34:11Z")

</div>

It still demands interactivity from STDIN that the particular phrase is typed: "Yes, do as I say!" (It's a weird apt thing for packages marked essential)

I guess I'll have to use a bash resource instead, I just wondered if package could do it to make it distro independent.

Cheers  
Geoff

On 24 Mar 2012, at 21:33, Joshua Miller wrote:

> You could wrap the "options" attribute in a case statement?
> 
> [http://wiki.opscode.com/display/chef/Resources#Resources-Package](http://wiki.opscode.com/display/chef/Resources#Resources-Package)
> 
> Install Package with options  
> package "debian-archive-keyring" do  
> action :install  
> options "--force-yes"  
> end
> 
> On Mar 24, 2012, at 2:13 PM, Geoff Meakin [geoffmeakin@aciddevelopments.co.uk](mailto:geoffmeakin@aciddevelopments.co.uk) wrote:
> 
> > Hi all, I am on debian
> > 
> > doing:
> > 
> > package "foo" do  
> > action :install  
> > end
> > 
> > works fine in most cases.
> > 
> > Some packages however (I read somewhere there are 8 of them?) seem to be critical/core packages and trying to uninstall them mean that the normal way of installing under the covers:  
> > apt-get -y
> > 
> > just doesnt work. upstart is such a package as it uninstalls sysvinit.
> > 
> > Is there a standard way of automatically installing this using chef?
> > 
> > If I had to do a commandline version I would do this:
> > 
> > yes "Yes, do as I say!" | apt-get -y --force-yes upstart
> > 
> > But then this deviates somewhat from the nicely distribution-agnostic package chef dsl
> > 
> > Thanks

---

<div class="post-metadata">

**Author:** ![Joshua\_Miller](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/joshua_miller/32/212_2.png) [@Joshua\_Miller](https://discourse.chef.io/u/Joshua_Miller)\
**Post date:** [March 25, 2012, 1:49pm UTC](https://discourse.chef.io/t/large-cookbook-uploads-fail-with-unauthorized-after-data-upload/2597/7 "2012-03-25T13:49:17Z")

</div>

The -y option should take care of that I believe.

> **[apt-get(8) - Linux man page](https://linux.die.net/man/8/apt-get)**
>
> apt-get is the command-line tool for handling packages, and may be considered the user's "back-end" to other tools using the APT library.

> > options "--force-yes -Y"

On Mar 25, 2012, at 2:34 AM, Geoff Meakin [geoffmeakin@aciddevelopments.co.uk](mailto:geoffmeakin@aciddevelopments.co.uk) wrote:

> It still demands interactivity from STDIN that the particular phrase is typed: "Yes, do as I say!" (It's a weird apt thing for packages marked essential)
> 
> I guess I'll have to use a bash resource instead, I just wondered if package could do it to make it distro independent.
> 
> Cheers  
> Geoff
> 
> On 24 Mar 2012, at 21:33, Joshua Miller wrote:
> 
> > You could wrap the "options" attribute in a case statement?
> > 
> > [http://wiki.opscode.com/display/chef/Resources#Resources-Package](http://wiki.opscode.com/display/chef/Resources#Resources-Package)
> > 
> > Install Package with options  
> > package "debian-archive-keyring" do  
> > action :install  
> > options "--force-yes"  
> > end
> > 
> > On Mar 24, 2012, at 2:13 PM, Geoff Meakin [geoffmeakin@aciddevelopments.co.uk](mailto:geoffmeakin@aciddevelopments.co.uk) wrote:
> > 
> > > Hi all, I am on debian
> > > 
> > > doing:
> > > 
> > > package "foo" do  
> > > action :install  
> > > end
> > > 
> > > works fine in most cases.
> > > 
> > > Some packages however (I read somewhere there are 8 of them?) seem to be critical/core packages and trying to uninstall them mean that the normal way of installing under the covers:  
> > > apt-get -y
> > > 
> > > just doesnt work. upstart is such a package as it uninstalls sysvinit.
> > > 
> > > Is there a standard way of automatically installing this using chef?
> > > 
> > > If I had to do a commandline version I would do this:
> > > 
> > > yes "Yes, do as I say!" | apt-get -y --force-yes upstart
> > > 
> > > But then this deviates somewhat from the nicely distribution-agnostic package chef dsl
> > > 
> > > Thanks

---

<div class="post-metadata">

**Author:** ![Geoff\_Meakin](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/geoff_meakin/32/718_2.png) [@Geoff\_Meakin](https://discourse.chef.io/u/Geoff_Meakin)\
**Post date:** [March 25, 2012, 2:38pm UTC](https://discourse.chef.io/t/large-cookbook-uploads-fail-with-unauthorized-after-data-upload/2597/8 "2012-03-25T14:38:10Z")

</div>

Hi,

From the page you just sent --\>  
-y, --yes, --assume-yes  
Automatic yes to prompts. Assume "yes" as answer to all prompts and run non-interactively. If an undesirable situation, such as changing a held package or removing an essential package, occurs then apt-get will abort.

As you can see apt-get aborts and doesnt do the work on an "essential" package.

Cheers  
Geoff

On 25 Mar 2012, at 14:49, Joshua Miller wrote:

> The -y option should take care of that I believe.
> 
> [http://linux.die.net/man/8/apt-get](http://linux.die.net/man/8/apt-get)
> 
> > > options "--force-yes -Y"
> 
> On Mar 25, 2012, at 2:34 AM, Geoff Meakin [geoffmeakin@aciddevelopments.co.uk](mailto:geoffmeakin@aciddevelopments.co.uk) wrote:
> 
> > It still demands interactivity from STDIN that the particular phrase is typed: "Yes, do as I say!" (It's a weird apt thing for packages marked essential)
> > 
> > I guess I'll have to use a bash resource instead, I just wondered if package could do it to make it distro independent.
> > 
> > Cheers  
> > Geoff
> > 
> > On 24 Mar 2012, at 21:33, Joshua Miller wrote:
> > 
> > > You could wrap the "options" attribute in a case statement?
> > > 
> > > [http://wiki.opscode.com/display/chef/Resources#Resources-Package](http://wiki.opscode.com/display/chef/Resources#Resources-Package)
> > > 
> > > Install Package with options  
> > > package "debian-archive-keyring" do  
> > > action :install  
> > > options "--force-yes"  
> > > end
> > > 
> > > On Mar 24, 2012, at 2:13 PM, Geoff Meakin [geoffmeakin@aciddevelopments.co.uk](mailto:geoffmeakin@aciddevelopments.co.uk) wrote:
> > > 
> > > > Hi all, I am on debian
> > > > 
> > > > doing:
> > > > 
> > > > package "foo" do  
> > > > action :install  
> > > > end
> > > > 
> > > > works fine in most cases.
> > > > 
> > > > Some packages however (I read somewhere there are 8 of them?) seem to be critical/core packages and trying to uninstall them mean that the normal way of installing under the covers:  
> > > > apt-get -y
> > > > 
> > > > just doesnt work. upstart is such a package as it uninstalls sysvinit.
> > > > 
> > > > Is there a standard way of automatically installing this using chef?
> > > > 
> > > > If I had to do a commandline version I would do this:
> > > > 
> > > > yes "Yes, do as I say!" | apt-get -y --force-yes upstart
> > > > 
> > > > But then this deviates somewhat from the nicely distribution-agnostic package chef dsl
> > > > 
> > > > Thanks

---

<div class="post-metadata">

**Author:** ![jasonjwwilliams](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/jasonjwwilliams/32/101_2.png) [@jasonjwwilliams](https://discourse.chef.io/u/jasonjwwilliams)\
**Post date:** [March 26, 2012, 9:26pm UTC](https://discourse.chef.io/t/large-cookbook-uploads-fail-with-unauthorized-after-data-upload/2597/9 "2012-03-26T21:26:42Z")

</div>

> Why are you putting several very large files in Chef? Surely there are  
> more efficient ways to manage these files? It is likely that such  
> files would take a long time to download when required by the client  
> and likewise time out.

Because if we're going to have to run our own mirrors of the files,  
we'd just as well have it in Chef rather than manage an internal  
mirror that's only used by Chef anyway.

-J

---

<div class="post-metadata">

**Author:** ![jasonjwwilliams](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/jasonjwwilliams/32/101_2.png) [@jasonjwwilliams](https://discourse.chef.io/u/jasonjwwilliams)\
**Post date:** [March 26, 2012, 9:27pm UTC](https://discourse.chef.io/t/large-cookbook-uploads-fail-with-unauthorized-after-data-upload/2597/10 "2012-03-26T21:27:43Z")

</div>

On Sat, Mar 24, 2012 at 7:48 AM, Joshua Miller [jassinpain@gmail.com](mailto:jassinpain@gmail.com) wrote:

> I think timberman was trying to direct you to something like this. For  
> anything over a one or two meg I find it works much better.
> 
> [http://wiki.opscode.com/display/chef/Resources#Resources-RemoteFile](http://wiki.opscode.com/display/chef/Resources#Resources-RemoteFile)

Yeah, we use that already quite a lot for packages we don't need to mirror  
ourselves. These files need to be mirrored by us.

-J

---

<div class="post-metadata">

**Author:** ![AJ\_Christensen](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/aj_christensen/32/525_2.png) [@AJ\_Christensen](https://discourse.chef.io/u/AJ_Christensen)\
**Post date:** [March 26, 2012, 10:28pm UTC](https://discourse.chef.io/t/large-cookbook-uploads-fail-with-unauthorized-after-data-upload/2597/11 "2012-03-26T22:28:21Z")

</div>

On 27 March 2012 10:26, Jason J. W. Williams [jasonjwwilliams@gmail.com](mailto:jasonjwwilliams@gmail.com) wrote:

> > Why are you putting several very large files in Chef? Surely there are  
> > more efficient ways to manage these files? It is likely that such  
> > files would take a long time to download when required by the client  
> > and likewise time out.
> 
> Because if we're going to have to run our own mirrors of the files,  
> we'd just as well have it in Chef rather than manage an internal  
> mirror that's only used by Chef anyway.

This is a false assertion.

The chef-server file server is not just a 'mirror of files'. You  
should explicitly avoid loading and streaming large files with the  
Signed Header authentication, as many cycles will be used to hash HTTP  
headers and body (in my experience).

In your position, I would consider distributing these large files via  
BitTorrent or HTTP, potentially from CDN-backed Object Storage such as  
S3 depending on geographic proximity.

Warm Regards,

--AJ

> -J

---

<div class="post-metadata">

**Author:** ![jasonjwwilliams](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/jasonjwwilliams/32/101_2.png) [@jasonjwwilliams](https://discourse.chef.io/u/jasonjwwilliams)\
**Post date:** [March 26, 2012, 10:39pm UTC](https://discourse.chef.io/t/large-cookbook-uploads-fail-with-unauthorized-after-data-upload/2597/12 "2012-03-26T22:39:55Z")

</div>

> In your position, I would consider distributing these large files via  
> BitTorrent or HTTP, potentially from CDN-backed Object Storage such as  
> S3 depending on geographic proximity.

Thanks AJ. Since we don't have a choice given the current behavior  
that's what we're converting to for this.

-J