How to configure apt to not hit the repo every time?


#1

Hi, I’m developing a cookbook using Berkshelf and Vagrant latest.

There is a place in the recipe that runs apt-get update frequently and
other dependent cookbooks too like the following:

[2013-05-18T11:21:48+00:00] INFO: Processing execute[apt-get update] action
run (mysql::ruby line 23)

Its fine on the production node, but in the vagrant provision way, every
time I run, it will hit the apt-get update which is a bit slow coz it hits
via the internet.

How can I configure Vagrant to just hit only once in a single vagrant
provision command?
Or do I have to do change something else?

@millisami
~Sachin Sagar Rai
Ruby on Rails Developer
http://tfm.com.np
http://nepalonrails.com


#2

On May 18, 2013, at 6:27 AM, millisami r millisami@gmail.com wrote:

Hi, I’m developing a cookbook using Berkshelf and Vagrant latest.

There is a place in the recipe that runs apt-get update frequently and other dependent cookbooks too like the following:

[2013-05-18T11:21:48+00:00] INFO: Processing execute[apt-get update] action run (mysql::ruby line 23)

Its fine on the production node, but in the vagrant provision way, every time I run, it will hit the apt-get update which is a bit slow coz it hits via the internet.

How can I configure Vagrant to just hit only once in a single vagrant provision command?
Or do I have to do change something else?

You might look into the resolver cookbook to set up your VM DNS. I found that by default the VM passes everything to VirtualBox and, only when that timed out, to the host machine’s DNS gateway. My build times dropped 25% when I put resolver in as the first recipe in the run list.

Hope this helps,
Mark

@millisami
~Sachin Sagar Rai
Ruby on Rails Developer
http://tfm.com.np
http://nepalonrails.com


#3

Or run vagrant with --no-provisioner and if you need chef to do something
just run the chef scripts. Or is it something else you are looking for?

On Saturday, May 18, 2013, Mark Nichols wrote:

On May 18, 2013, at 6:27 AM, millisami r <millisami@gmail.com<javascript:_e({}, ‘cvml’, ‘millisami@gmail.com’);>>
wrote:

Hi, I’m developing a cookbook using Berkshelf and Vagrant latest.

There is a place in the recipe that runs apt-get update frequently and
other dependent cookbooks too like the following:

[2013-05-18T11:21:48+00:00] INFO: Processing execute[apt-get update]
action run (mysql::ruby line 23)

Its fine on the production node, but in the vagrant provision way, every
time I run, it will hit the apt-get update which is a bit slow coz it hits
via the internet.

How can I configure Vagrant to just hit only once in a single vagrant
provision command?
Or do I have to do change something else?

You might look into the resolver cookbook to set up your VM DNS. I found
that by default the VM passes everything to VirtualBox and, only when that
timed out, to the host machine’s DNS gateway. My build times dropped 25%
when I put resolver in as the first recipe in the run list.

Hope this helps,
Mark

@millisami
~Sachin Sagar Rai
Ruby on Rails Developer
http://tfm.com.np
http://nepalonrails.com

Jens Skott
Tel: +46-8-5142 4396
Schibsted Centralen IT


#4

To answer a more general and generic question, which is not specific
to Vagrant or Berkshelf: “how do I prevent apt-get update from running
too often?”, use this code in the recipe, in place of the line
calling “apt-get update”:

if node[‘platform_family’] == "debian"
if node[‘platform_version’].to_f >= 12
x = execute “apt-get update” do
action :nothing
not_if do
::File.exists?(’/var/lib/apt/periodic/update-success-stamp’) &&
::File.mtime(’/var/lib/apt/periodic/update-success-stamp’) >
Time.now - 864007
end
end
else
x = execute “apt-get update” do
action :nothing
command "touch /tmp/apt-get-update ; /usr/bin/apt-get update"
not_if do
::File.exists?(’/tmp/apt-get-update’) &&
::File.mtime(’/tmp/apt-get-update’) > Time.now - 86400
7
end
end
end
x.run_action(:run)
end

On Sat, May 18, 2013 at 2:27 PM, millisami r millisami@gmail.com wrote:

Hi, I’m developing a cookbook using Berkshelf and Vagrant latest.

There is a place in the recipe that runs apt-get update frequently and other
dependent cookbooks too like the following:

[2013-05-18T11:21:48+00:00] INFO: Processing execute[apt-get update] action
run (mysql::ruby line 23)

Its fine on the production node, but in the vagrant provision way, every
time I run, it will hit the apt-get update which is a bit slow coz it hits
via the internet.

How can I configure Vagrant to just hit only once in a single vagrant
provision command?
Or do I have to do change something else?

@millisami
~Sachin Sagar Rai
Ruby on Rails Developer
http://tfm.com.np
http://nepalonrails.com


#5

Or - use apt-cacher ng to proxy all apt calls and make it super fast:

I will often spin up a small Vagrant box and push all of my apt
requests through it - especially when I’m building from scratch over
and over.

Works pretty well

Apparently if you’re using the apt cookbook - you can set a cacher ip
address and have it pull from there pretty easily:

I haven’t tried that yet - was doing it manually before.

On Sat, May 18, 2013 at 9:02 AM, Sam Darwin samuel.d.darwin@gmail.com wrote:

To answer a more general and generic question, which is not specific
to Vagrant or Berkshelf: “how do I prevent apt-get update from running
too often?”, use this code in the recipe, in place of the line
calling “apt-get update”:

if node[‘platform_family’] == "debian"
if node[‘platform_version’].to_f >= 12
x = execute “apt-get update” do
action :nothing
not_if do
::File.exists?(’/var/lib/apt/periodic/update-success-stamp’) &&
::File.mtime(’/var/lib/apt/periodic/update-success-stamp’) >
Time.now - 864007
end
end
else
x = execute “apt-get update” do
action :nothing
command "touch /tmp/apt-get-update ; /usr/bin/apt-get update"
not_if do
::File.exists?(’/tmp/apt-get-update’) &&
::File.mtime(’/tmp/apt-get-update’) > Time.now - 86400
7
end
end
end
x.run_action(:run)
end

On Sat, May 18, 2013 at 2:27 PM, millisami r millisami@gmail.com wrote:

Hi, I’m developing a cookbook using Berkshelf and Vagrant latest.

There is a place in the recipe that runs apt-get update frequently and other
dependent cookbooks too like the following:

[2013-05-18T11:21:48+00:00] INFO: Processing execute[apt-get update] action
run (mysql::ruby line 23)

Its fine on the production node, but in the vagrant provision way, every
time I run, it will hit the apt-get update which is a bit slow coz it hits
via the internet.

How can I configure Vagrant to just hit only once in a single vagrant
provision command?
Or do I have to do change something else?

@millisami
~Sachin Sagar Rai
Ruby on Rails Developer
http://tfm.com.np
http://nepalonrails.com


#6

It was fixed just recently.

It is now relying on build-essential mechanism


Max Stepanov

On Sat, May 18, 2013 at 8:00 PM, Darron Froese darron@froese.org wrote:

Or - use apt-cacher ng to proxy all apt calls and make it super fast:

https://gist.github.com/fnichol/1747868

I will often spin up a small Vagrant box and push all of my apt
requests through it - especially when I’m building from scratch over
and over.

Works pretty well

Apparently if you’re using the apt cookbook - you can set a cacher ip
address and have it pull from there pretty easily:

https://github.com/opscode-cookbooks/apt/blob/master/recipes/cacher-client.rb

I haven’t tried that yet - was doing it manually before.

On Sat, May 18, 2013 at 9:02 AM, Sam Darwin samuel.d.darwin@gmail.com
wrote:

To answer a more general and generic question, which is not specific
to Vagrant or Berkshelf: “how do I prevent apt-get update from running
too often?”, use this code in the recipe, in place of the line
calling “apt-get update”:

if node[‘platform_family’] == "debian"
if node[‘platform_version’].to_f >= 12
x = execute “apt-get update” do
action :nothing
not_if do
::File.exists?(’/var/lib/apt/periodic/update-success-stamp’) &&
::File.mtime(’/var/lib/apt/periodic/update-success-stamp’) >
Time.now - 864007
end
end
else
x = execute “apt-get update” do
action :nothing
command "touch /tmp/apt-get-update ; /usr/bin/apt-get update"
not_if do
::File.exists?(’/tmp/apt-get-update’) &&
::File.mtime(’/tmp/apt-get-update’) > Time.now - 86400
7
end
end
end
x.run_action(:run)
end

On Sat, May 18, 2013 at 2:27 PM, millisami r millisami@gmail.com
wrote:

Hi, I’m developing a cookbook using Berkshelf and Vagrant latest.

There is a place in the recipe that runs apt-get update frequently and
other

dependent cookbooks too like the following:

[2013-05-18T11:21:48+00:00] INFO: Processing execute[apt-get update]
action

run (mysql::ruby line 23)

Its fine on the production node, but in the vagrant provision way, every
time I run, it will hit the apt-get update which is a bit slow coz it
hits

via the internet.

How can I configure Vagrant to just hit only once in a single vagrant
provision command?
Or do I have to do change something else?

@millisami
~Sachin Sagar Rai
Ruby on Rails Developer
http://tfm.com.np
http://nepalonrails.com


#7

@sam, it works, but that is just for the example I cited.
There are other dependent cookbooks, for eg mysql cookbook, nginx, etc.
I won’t be adding those lines in those cookbooks, right?

On Sunday, May 19, 2013, Max Stepanov wrote:

It was fixed just recently.

https://github.com/opscode-cookbooks/mysql/commit/ad39e46ab1a56db4c54a2cb7bd64172cc28ea46a

It is now relying on build-essential mechanism

https://github.com/opscode-cookbooks/build-essential/blob/02ef6753d09fd5c99ebd70aba327764c1315e35f/recipes/default.rb#L27-L39


Max Stepanov

On Sat, May 18, 2013 at 8:00 PM, Darron Froese <darron@froese.org<javascript:_e({}, ‘cvml’, ‘darron@froese.org’);>

wrote:

Or - use apt-cacher ng to proxy all apt calls and make it super fast:

https://gist.github.com/fnichol/1747868

I will often spin up a small Vagrant box and push all of my apt
requests through it - especially when I’m building from scratch over
and over.

Works pretty well

Apparently if you’re using the apt cookbook - you can set a cacher ip
address and have it pull from there pretty easily:

https://github.com/opscode-cookbooks/apt/blob/master/recipes/cacher-client.rb

I haven’t tried that yet - was doing it manually before.

On Sat, May 18, 2013 at 9:02 AM, Sam Darwin <samuel.d.darwin@gmail.com<javascript:_e({}, ‘cvml’, ‘samuel.d.darwin@gmail.com’);>>
wrote:

To answer a more general and generic question, which is not specific
to Vagrant or Berkshelf: “how do I prevent apt-get update from running
too often?”, use this code in the recipe, in place of the line
calling “apt-get update”:

if node[‘platform_family’] == "debian"
if node[‘platform_version’].to_f >= 12
x = execute “apt-get update” do
action :nothing
not_if do
::File.exists?(’/var/lib/apt/periodic/update-success-stamp’) &&
::File.mtime(’/var/lib/apt/periodic/update-success-stamp’) >
Time.now - 864007
end
end
else
x = execute “apt-get update” do
action :nothing
command "touch /tmp/apt-get-update ; /usr/bin/apt-get update"
not_if do
::File.exists?(’/tmp/apt-get-update’) &&
::File.mtime(’/tmp/apt-get-update’) > Time.now - 86400
7
end
end
end
x.run_action(:run)
end

On Sat, May 18, 2013 at 2:27 PM, millisami r <millisami@gmail.com<javascript:_e({}, ‘cvml’, ‘millisami@gmail.com’);>>
wrote:

Hi, I’m developing a cookbook using Berkshelf and Vagrant latest.

There is a place in the recipe that runs apt-get update frequently and
other

dependent cookbooks too like the following:

[2013-05-18T11:21:48+00:00] INFO: Processing execute[apt-get update]
action

run (mysql::ruby line 23)

Its fine on the production node, but in the vagrant provision way,
every

time I run, it will hit the apt-get update which is a bit slow coz it
hits

via the internet.

How can I configure Vagrant to just hit only once in a single vagrant
provision command?
Or do I have to do change something else?

@millisami
~Sachin Sagar Rai
Ruby on Rails Developer
http://tfm.com.np
http://nepalonrails.com

@millisami
~Sachin Sagar Rai
Ruby on Rails Developer
http://tfm.com.np
http://nepalonrails.com


#8

Millisami,

Cookbooks shouldn’t run “apt-get update” on every single chef-client
run, and probably most aren’t. You can add that code into your
cookbooks manually, even site cookbooks, yes. You can contact the
official cookbook authors and ask them to add code like that …

On Sun, May 19, 2013 at 12:37 PM, millisami r millisami@gmail.com wrote:

@sam, it works, but that is just for the example I cited.
There are other dependent cookbooks, for eg mysql cookbook, nginx, etc.
I won’t be adding those lines in those cookbooks, right?

On Sunday, May 19, 2013, Max Stepanov wrote:

It was fixed just recently.

https://github.com/opscode-cookbooks/mysql/commit/ad39e46ab1a56db4c54a2cb7bd64172cc28ea46a

It is now relying on build-essential mechanism

https://github.com/opscode-cookbooks/build-essential/blob/02ef6753d09fd5c99ebd70aba327764c1315e35f/recipes/default.rb#L27-L39


Max Stepanov

On Sat, May 18, 2013 at 8:00 PM, Darron Froese darron@froese.org wrote:

Or - use apt-cacher ng to proxy all apt calls and make it super fast:

https://gist.github.com/fnichol/1747868

I will often spin up a small Vagrant box and push all of my apt
requests through it - especially when I’m building from scratch over
and over.

Works pretty well

Apparently if you’re using the apt cookbook - you can set a cacher ip
address and have it pull from there pretty easily:

https://github.com/opscode-cookbooks/apt/blob/master/recipes/cacher-client.rb

I haven’t tried that yet - was doing it manually before.

On Sat, May 18, 2013 at 9:02 AM, Sam Darwin samuel.d.darwin@gmail.com
wrote:

To answer a more general and generic question, which is not specific
to Vagrant or Berkshelf: “how do I prevent apt-get update from running
too often?”, use this code in the recipe, in place of the line
calling “apt-get update”:

if node[‘platform_family’] == "debian"
if node[‘platform_version’].to_f >= 12
x = execute “apt-get update” do
action :nothing
not_if do
::File.exists?(’/var/lib/apt/periodic/update-success-stamp’) &&
::File.mtime(’/var/lib/apt/periodic/update-success-stamp’) >
Time.now - 864007
end
end
else
x = execute “apt-get update” do
action :nothing
command "touch /tmp/apt-get-update ; /usr/bin/apt-get update"
not_if do
::File.exists?(’/tmp/apt-get-update’) &&
::File.mtime(’/tmp/apt-get-update’) > Time.now - 86400
7
end
end
end
x.run_action(:run)
end

On Sat, May 18, 2013 at 2:27 PM, millisami r millisami@gmail.com
wrote:

Hi, I’m developing a cookbook using Berkshelf and Vagrant latest.

There is a place in the recipe that runs apt-get update frequently and
other
dependent cookbooks too like the following:

[2013-05-18T11:21:48+00:00] INFO: Processing execute[apt-get update]
action
run (mysql::ruby line 23)

Its fine on the production node, but in the vagrant provision way,
every
time I run, it will hit the apt-get update which is a bit slow coz it
hits
via the internet.

How can I configure Vagrant to just hit only once in a single vagrant
provision command?
Or do I have to do change something else?

@millisami
~Sachin Sagar Rai
Ruby on Rails Developer
http://tfm.com.np
http://nepalonrails.com

@millisami
~Sachin Sagar Rai
Ruby on Rails Developer
http://tfm.com.np
http://nepalonrails.com


#9

You’ll probably make mirror owners grumpy if you do, they probably won’t
appreciate being hit by a slew of servers every 30 minutes (unless
you’re running your own mirror/repository).

Paul

On 5/19/2013 10:19 PM, Sam Darwin wrote:

Millisami,

Cookbooks shouldn’t run “apt-get update” on every single chef-client
run, and probably most aren’t. You can add that code into your
cookbooks manually, even site cookbooks, yes. You can contact the
official cookbook authors and ask them to add code like that …

On Sun, May 19, 2013 at 12:37 PM, millisami r millisami@gmail.com wrote:

@sam, it works, but that is just for the example I cited.
There are other dependent cookbooks, for eg mysql cookbook, nginx, etc.
I won’t be adding those lines in those cookbooks, right?

On Sunday, May 19, 2013, Max Stepanov wrote:

It was fixed just recently.

https://github.com/opscode-cookbooks/mysql/commit/ad39e46ab1a56db4c54a2cb7bd64172cc28ea46a

It is now relying on build-essential mechanism

https://github.com/opscode-cookbooks/build-essential/blob/02ef6753d09fd5c99ebd70aba327764c1315e35f/recipes/default.rb#L27-L39


Max Stepanov

On Sat, May 18, 2013 at 8:00 PM, Darron Froese darron@froese.org wrote:

Or - use apt-cacher ng to proxy all apt calls and make it super fast:

https://gist.github.com/fnichol/1747868

I will often spin up a small Vagrant box and push all of my apt
requests through it - especially when I’m building from scratch over
and over.

Works pretty well

Apparently if you’re using the apt cookbook - you can set a cacher ip
address and have it pull from there pretty easily:

https://github.com/opscode-cookbooks/apt/blob/master/recipes/cacher-client.rb

I haven’t tried that yet - was doing it manually before.

On Sat, May 18, 2013 at 9:02 AM, Sam Darwin samuel.d.darwin@gmail.com
wrote:

To answer a more general and generic question, which is not specific
to Vagrant or Berkshelf: “how do I prevent apt-get update from running
too often?”, use this code in the recipe, in place of the line
calling “apt-get update”:

if node[‘platform_family’] == "debian"
if node[‘platform_version’].to_f >= 12
x = execute “apt-get update” do
action :nothing
not_if do
::File.exists?(’/var/lib/apt/periodic/update-success-stamp’) &&
::File.mtime(’/var/lib/apt/periodic/update-success-stamp’) >
Time.now - 864007
end
end
else
x = execute “apt-get update” do
action :nothing
command "touch /tmp/apt-get-update ; /usr/bin/apt-get update"
not_if do
::File.exists?(’/tmp/apt-get-update’) &&
::File.mtime(’/tmp/apt-get-update’) > Time.now - 86400
7
end
end
end
x.run_action(:run)
end

On Sat, May 18, 2013 at 2:27 PM, millisami r millisami@gmail.com
wrote:

Hi, I’m developing a cookbook using Berkshelf and Vagrant latest.

There is a place in the recipe that runs apt-get update frequently and
other
dependent cookbooks too like the following:

[2013-05-18T11:21:48+00:00] INFO: Processing execute[apt-get update]
action
run (mysql::ruby line 23)

Its fine on the production node, but in the vagrant provision way,
every
time I run, it will hit the apt-get update which is a bit slow coz it
hits
via the internet.

How can I configure Vagrant to just hit only once in a single vagrant
provision command?
Or do I have to do change something else?

@millisami
~Sachin Sagar Rai
Ruby on Rails Developer
http://tfm.com.np
http://nepalonrails.com

@millisami
~Sachin Sagar Rai
Ruby on Rails Developer
http://tfm.com.np
http://nepalonrails.com


#10

Right, cookbooks should be using something along the lines of Package, and
if you aren’t able to resolve this automatically in a chef run, then
including a call to an apt recipe to update the repo information in the run
should resolve the issue automatically if run-time order is resolved
correctly.


~~ StormeRider ~~

“Every world needs its heroes […] They inspire us to be better than we
are. And they protect from the darkness that’s just around the corner.”

(from Smallville Season 6x1: “Zod”)

On why I hate the phrase “that’s so lame”… http://bit.ly/Ps3uSS

On Mon, May 20, 2013 at 1:19 AM, Sam Darwin samuel.d.darwin@gmail.comwrote:

Millisami,

Cookbooks shouldn’t run “apt-get update” on every single chef-client
run, and probably most aren’t. You can add that code into your
cookbooks manually, even site cookbooks, yes. You can contact the
official cookbook authors and ask them to add code like that …

On Sun, May 19, 2013 at 12:37 PM, millisami r millisami@gmail.com wrote:

@sam, it works, but that is just for the example I cited.
There are other dependent cookbooks, for eg mysql cookbook, nginx, etc.
I won’t be adding those lines in those cookbooks, right?

On Sunday, May 19, 2013, Max Stepanov wrote:

It was fixed just recently.

https://github.com/opscode-cookbooks/mysql/commit/ad39e46ab1a56db4c54a2cb7bd64172cc28ea46a

It is now relying on build-essential mechanism

https://github.com/opscode-cookbooks/build-essential/blob/02ef6753d09fd5c99ebd70aba327764c1315e35f/recipes/default.rb#L27-L39


Max Stepanov

On Sat, May 18, 2013 at 8:00 PM, Darron Froese darron@froese.org
wrote:

Or - use apt-cacher ng to proxy all apt calls and make it super fast:

https://gist.github.com/fnichol/1747868

I will often spin up a small Vagrant box and push all of my apt
requests through it - especially when I’m building from scratch over
and over.

Works pretty well

Apparently if you’re using the apt cookbook - you can set a cacher ip
address and have it pull from there pretty easily:

https://github.com/opscode-cookbooks/apt/blob/master/recipes/cacher-client.rb

I haven’t tried that yet - was doing it manually before.

On Sat, May 18, 2013 at 9:02 AM, Sam Darwin <samuel.d.darwin@gmail.com

wrote:

To answer a more general and generic question, which is not specific
to Vagrant or Berkshelf: "how do I prevent apt-get update from
running

too often?", use this code in the recipe, in place of the line
calling “apt-get update”:

if node[‘platform_family’] == "debian"
if node[‘platform_version’].to_f >= 12
x = execute “apt-get update” do
action :nothing
not_if do
::File.exists?(’/var/lib/apt/periodic/update-success-stamp’)
&&

    ::File.mtime('/var/lib/apt/periodic/update-success-stamp') >

Time.now - 864007
end
end
else
x = execute “apt-get update” do
action :nothing
command "touch /tmp/apt-get-update ; /usr/bin/apt-get update"
not_if do
::File.exists?(’/tmp/apt-get-update’) &&
::File.mtime(’/tmp/apt-get-update’) > Time.now - 86400
7
end
end
end
x.run_action(:run)
end

On Sat, May 18, 2013 at 2:27 PM, millisami r millisami@gmail.com
wrote:

Hi, I’m developing a cookbook using Berkshelf and Vagrant latest.

There is a place in the recipe that runs apt-get update frequently
and

other
dependent cookbooks too like the following:

[2013-05-18T11:21:48+00:00] INFO: Processing execute[apt-get update]
action
run (mysql::ruby line 23)

Its fine on the production node, but in the vagrant provision way,
every
time I run, it will hit the apt-get update which is a bit slow coz
it

hits
via the internet.

How can I configure Vagrant to just hit only once in a single
vagrant

provision command?
Or do I have to do change something else?

@millisami
~Sachin Sagar Rai
Ruby on Rails Developer
http://tfm.com.np
http://nepalonrails.com

@millisami
~Sachin Sagar Rai
Ruby on Rails Developer
http://tfm.com.np
http://nepalonrails.com