# Apt-get update strategy

**URL:** <https://discourse.chef.io/t/apt-get-update-strategy/1754>\
**Category:** Chef Infra (archive)\
**Created:** [March 25, 2011, 9:45am UTC](https://discourse.chef.io/t/apt-get-update-strategy/1754 "2011-03-25T09:45:48Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Luke\_Biddell](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/luke_biddell/32/663_2.png) [@Luke\_Biddell](https://discourse.chef.io/u/Luke_Biddell)\
**Post date:** [March 25, 2011, 9:45am UTC](https://discourse.chef.io/t/apt-get-update-strategy/1754/1 "2011-03-25T09:45:48Z")

</div>

A recent failure in the apache cassandra apt repo got me thinking  
about the way I’ve assembled my cookbooks. I have the apt cookbook  
first in my run and do an update. The repo being down gave me the  
classic “apt-get update returned 100” error within chef.

The failure meant that none of my recipes ran against the node (and  
updated our application war file), despite the fact that the node had  
previously been converged and all the required apt packages were  
already installed.

My chef run needs to be resilient to those kinds of failures as once a  
node is initially converged and all apt packages installed, apt  
doesn’t need to do an update (I don’t do package :upgrade at the  
moment).

So what I’d ideally like is to be able to trigger an apt-get update on  
the first package which requires installing. If no packages require  
installing, no apt-get update is performed. The fact an update has  
been performed needs to be recorded as we don’t want to do it for  
every package that’s installed as it will kill performance. Once is  
enough per chef run unless we add/remove a sources.list.d entry (which  
I already handle using :notifies).

Opinions welcome, am I looking and an LWRP?

---

<div class="post-metadata">

**Author:** ![Michael\_Hale](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/michael_hale/32/834_2.png) [@Michael\_Hale](https://discourse.chef.io/u/Michael_Hale)\
**Post date:** [March 25, 2011, 1:18pm UTC](https://discourse.chef.io/t/apt-get-update-strategy/1754/2 "2011-03-25T13:18:42Z")

</div>

Maybe a feature could be added to the package resource/providers to  
handle updating the package caches once per run?

On Fri, Mar 25, 2011 at 5:45 AM, Luke Biddell [luke.biddell@gmail.com](mailto:luke.biddell@gmail.com) wrote:

> A recent failure in the apache cassandra apt repo got me thinking  
> about the way I've assembled my cookbooks. I have the apt cookbook  
> first in my run and do an update. The repo being down gave me the  
> classic "apt-get update returned 100" error within chef.
> 
> The failure meant that none of my recipes ran against the node (and  
> updated our application war file), despite the fact that the node had  
> previously been converged and all the required apt packages were  
> already installed.
> 
> My chef run needs to be resilient to those kinds of failures as once a  
> node is initially converged and all apt packages installed, apt  
> doesn't need to do an update (I don't do package :upgrade at the  
> moment).
> 
> So what I'd ideally like is to be able to trigger an apt-get update on  
> the first package which requires installing. If no packages require  
> installing, no apt-get update is performed. The fact an update has  
> been performed needs to be recorded as we don't want to do it for  
> every package that's installed as it will kill performance. Once is  
> enough per chef run unless we add/remove a sources.list.d entry (which  
> I already handle using :notifies).
> 
> Opinions welcome, am I looking and an LWRP?

---

<div class="post-metadata">

**Author:** ![Rob\_Guttman](https://avatars.discourse-cdn.com/v4/letter/r/a9adbd/32.png) [@Rob\_Guttman](https://discourse.chef.io/u/Rob_Guttman)\
**Post date:** [March 25, 2011, 1:35pm UTC](https://discourse.chef.io/t/apt-get-update-strategy/1754/3 "2011-03-25T13:35:12Z")

</div>

Michael, my current workaround is to effectively run "apt-get update" as a  
cron job once per day and also upon a host (re)boot. Our longer-term  
workaround idea is to mirror the apt repos locally which would solve this  
and other apt problems (e.g., updates or dropped support of package versions  
we depend upon).

These workarounds are not ideal and may not be appropriate for everyone.

- Rob

On Fri, Mar 25, 2011 at 9:18 AM, Michael Hale [mikehale@gmail.com](mailto:mikehale@gmail.com) wrote:

> Maybe a feature could be added to the package resource/providers to  
> handle updating the package caches once per run?
> 
> On Fri, Mar 25, 2011 at 5:45 AM, Luke Biddell [luke.biddell@gmail.com](mailto:luke.biddell@gmail.com)  
> wrote:
> 
> > A recent failure in the apache cassandra apt repo got me thinking  
> > about the way I've assembled my cookbooks. I have the apt cookbook  
> > first in my run and do an update. The repo being down gave me the  
> > classic "apt-get update returned 100" error within chef.
> > 
> > The failure meant that none of my recipes ran against the node (and  
> > updated our application war file), despite the fact that the node had  
> > previously been converged and all the required apt packages were  
> > already installed.
> > 
> > My chef run needs to be resilient to those kinds of failures as once a  
> > node is initially converged and all apt packages installed, apt  
> > doesn't need to do an update (I don't do package :upgrade at the  
> > moment).
> > 
> > So what I'd ideally like is to be able to trigger an apt-get update on  
> > the first package which requires installing. If no packages require  
> > installing, no apt-get update is performed. The fact an update has  
> > been performed needs to be recorded as we don't want to do it for  
> > every package that's installed as it will kill performance. Once is  
> > enough per chef run unless we add/remove a sources.list.d entry (which  
> > I already handle using :notifies).
> > 
> > Opinions welcome, am I looking and an LWRP?

---

<div class="post-metadata">

**Author:** ![Matt\_Ray\_OPSCODE](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/matt_ray_opscode/32/459_2.png) [@Matt\_Ray\_OPSCODE](https://discourse.chef.io/u/Matt_Ray_OPSCODE)\
**Post date:** [March 25, 2011, 2:52pm UTC](https://discourse.chef.io/t/apt-get-update-strategy/1754/4 "2011-03-25T14:52:32Z")

</div>

While this doesn't solve the original problem exactly, you may want to  
take a look at the apt cookbook and the recent apt-cacher updates.  
It's super simple to set up, just have 1 server apply the apt::cacher  
and everyone (including the server) use apt::cacher-client, no  
tweaking needed. This will start proxying and caching your apt  
downloads and minimize your exposure to remote repos being  
unavailable, especially if you modify the expiration times on the  
server.

Thanks,  
Matt Ray  
Technical Evangelist | Opscode, Inc  
E: [matt@opscode.com](mailto:matt@opscode.com) T: (512) 731-2218  
Twitter, Github: mattray

On Fri, Mar 25, 2011 at 8:35 AM, Rob Guttman [robguttman@gmail.com](mailto:robguttman@gmail.com) wrote:

> Michael, my current workaround is to effectively run "apt-get update" as a  
> cron job once per day and also upon a host (re)boot. Our longer-term  
> workaround idea is to mirror the apt repos locally which would solve this  
> and other apt problems (e.g., updates or dropped support of package versions  
> we depend upon).
> 
> These workarounds are not ideal and may not be appropriate for everyone.
> 
> - Rob
> 
> On Fri, Mar 25, 2011 at 9:18 AM, Michael Hale [mikehale@gmail.com](mailto:mikehale@gmail.com) wrote:
> 
> > Maybe a feature could be added to the package resource/providers to  
> > handle updating the package caches once per run?
> > 
> > On Fri, Mar 25, 2011 at 5:45 AM, Luke Biddell [luke.biddell@gmail.com](mailto:luke.biddell@gmail.com)  
> > wrote:
> > 
> > > A recent failure in the apache cassandra apt repo got me thinking  
> > > about the way I've assembled my cookbooks. I have the apt cookbook  
> > > first in my run and do an update. The repo being down gave me the  
> > > classic "apt-get update returned 100" error within chef.
> > > 
> > > The failure meant that none of my recipes ran against the node (and  
> > > updated our application war file), despite the fact that the node had  
> > > previously been converged and all the required apt packages were  
> > > already installed.
> > > 
> > > My chef run needs to be resilient to those kinds of failures as once a  
> > > node is initially converged and all apt packages installed, apt  
> > > doesn't need to do an update (I don't do package :upgrade at the  
> > > moment).
> > > 
> > > So what I'd ideally like is to be able to trigger an apt-get update on  
> > > the first package which requires installing. If no packages require  
> > > installing, no apt-get update is performed. The fact an update has  
> > > been performed needs to be recorded as we don't want to do it for  
> > > every package that's installed as it will kill performance. Once is  
> > > enough per chef run unless we add/remove a sources.list.d entry (which  
> > > I already handle using :notifies).
> > > 
> > > Opinions welcome, am I looking and an LWRP?

---

<div class="post-metadata">

**Author:** ![Bryan\_McLellan](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/bryan_mclellan/32/81_2.png) [@Bryan\_McLellan](https://discourse.chef.io/u/Bryan_McLellan)\
**Post date:** [March 27, 2011, 1:46pm UTC](https://discourse.chef.io/t/apt-get-update-strategy/1754/5 "2011-03-27T13:46:34Z")

</div>

On Fri, Mar 25, 2011 at 2:45 AM, Luke Biddell [luke.biddell@gmail.com](mailto:luke.biddell@gmail.com) wrote:

> So what I'd ideally like is to be able to trigger an apt-get update on  
> the first package which requires installing. If no packages require  
> installing, no apt-get update is performed. The fact an update has  
> been performed needs to be recorded as we don't want to do it for  
> every package that's installed as it will kill performance. Once is  
> enough per chef run unless we add/remove a sources.list.d entry (which  
> I already handle using :notifies).

There are a number of strategies. Here's another I used to do.

Only trigger an apt-get update when a repo or key is added, otherwise  
rely on Ubuntu to run a daily apt-get update but run it ourselves if  
we need to. Note that I was silently rescuing failures as well.

# Run apt-get update to create the stamp file

execute "apt-get-update" do  
ignore\_failure true  
epic\_fail true  
command "apt-get update"  
not\_if do File.exists?('/var/lib/apt/periodic/update-success-stamp') end  
end

# provides /var/lib/apt/periodic/update-success-stamp on apt-get update

package "update-notifier-common" do  
ignore\_failure true  
notifies :run, resources(:execute =\> "apt-get-update"), :immediately  
end

execute "apt-get-update-periodic" do  
ignore\_failure true  
epic\_fail true  
command "apt-get update"  
only\_if do  
File.exists?('/var/lib/apt/periodic/update-success-stamp') &&  
File.mtime('/var/lib/apt/periodic/update-success-stamp') \< Time.now - 86400  
end  
end

---

<div class="post-metadata">

**Author:** ![Luke\_Biddell](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/luke_biddell/32/663_2.png) [@Luke\_Biddell](https://discourse.chef.io/u/Luke_Biddell)\
**Post date:** [March 28, 2011, 9:54am UTC](https://discourse.chef.io/t/apt-get-update-strategy/1754/6 "2011-03-28T09:54:46Z")

</div>

Thanks for all the suggestions chaps. I'll certainly check them all out.

I had a scribble on Friday (that's what Friday's are for?) and hacked  
up my own provider.

I've added it to our existing apt cookbook. It's intended as a drop-in  
replacement for package. If you do an install it only does an update  
if the package isn't installed. Ie, the first package to be installed  
will trigger the only update for that run. If you do a package upgrade  
it makes sure an upgrade is done once only. If all packages are  
converged, no update is done.

The flag to indicate if an update has been done is stored on the node  
and has to be reset at the start of each run. Is there a better way of  
setting transient attributes for the run?

I've posted the code here in case it's any use to anyone else. I'm  
going to commit it in dev here and see what comes out in the wash. I'm  
sure I've broken/abused something.

* * *

- apt/resources/pkg.rb

actions :install, :upgrade, :remove, :purge  
attribute :name, :kind\_of =\> String

* * *

- apt/providers/pkg.rb

action :install do  
if(!system("dpkg-query -W -f='${Status}' #{@new\_resource.name} \> /dev/null"))  
process\_package :install  
end  
end

action :upgrade do  
process\_package :upgrade  
end

action :remove do  
package @new\_resource.name do  
action :remove  
end  
end

action :purge do  
package @new\_resource.name do  
action :purge  
end  
end

private

def process\_package (mode)  
Chef::Log.debug("#{mode} of package #{@new\_resource.name} requested")  
if(!node[:apt\_update\_performed\_this\_chef\_run])  
Chef::Log.debug("apt-get update required for this run, performing")  
execute "apt-get update"  
node[:apt\_update\_performed\_this\_chef\_run] = true  
else  
Chef::Log.debug("apt-get update already performed for this run")  
end  
package @new\_resource.name do  
action mode  
end  
@new\_resource.updated\_by\_last\_action(true)  
end

On 27 March 2011 14:46, Bryan McLellan [btm@loftninjas.org](mailto:btm@loftninjas.org) wrote:

> On Fri, Mar 25, 2011 at 2:45 AM, Luke Biddell [luke.biddell@gmail.com](mailto:luke.biddell@gmail.com) wrote:
> 
> > So what I'd ideally like is to be able to trigger an apt-get update on  
> > the first package which requires installing. If no packages require  
> > installing, no apt-get update is performed. The fact an update has  
> > been performed needs to be recorded as we don't want to do it for  
> > every package that's installed as it will kill performance. Once is  
> > enough per chef run unless we add/remove a sources.list.d entry (which  
> > I already handle using :notifies).
> 
> There are a number of strategies. Here's another I used to do.
> 
> Only trigger an apt-get update when a repo or key is added, otherwise  
> rely on Ubuntu to run a daily apt-get update but run it ourselves if  
> we need to. Note that I was silently rescuing failures as well.
> 
> # Run apt-get update to create the stamp file
> 
> execute "apt-get-update" do  
> ignore\_failure true  
> epic\_fail true  
> command "apt-get update"  
> not\_if do File.exists?('/var/lib/apt/periodic/update-success-stamp') end  
> end
> 
> # provides /var/lib/apt/periodic/update-success-stamp on apt-get update
> 
> package "update-notifier-common" do  
> ignore\_failure true  
> notifies :run, resources(:execute =\> "apt-get-update"), :immediately  
> end
> 
> execute "apt-get-update-periodic" do  
> ignore\_failure true  
> epic\_fail true  
> command "apt-get update"  
> only\_if do  
> File.exists?('/var/lib/apt/periodic/update-success-stamp') &&  
> File.mtime('/var/lib/apt/periodic/update-success-stamp') \< Time.now - 86400  
> end  
> end

---

<div class="post-metadata">

**Author:** ![Mason\_Turner](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/mason_turner/32/817_2.png) [@Mason\_Turner](https://discourse.chef.io/u/Mason_Turner)\
**Post date:** [March 28, 2011, 11:17am UTC](https://discourse.chef.io/t/apt-get-update-strategy/1754/7 "2011-03-28T11:17:56Z")

</div>

On Mar 28, 2011, at 5:54 AM, Luke Biddell [luke.biddell@gmail.com](mailto:luke.biddell@gmail.com) wrote:

> The flag to indicate if an update has been done is stored on the node  
> and has to be reset at the start of each run. Is there a better way of  
> setting transient attributes for the run?

Take a look at node.run\_state for transient storage. I stow my searches there for reuse across a few recipes.

-- Mason Turner (mobile)

---

<div class="post-metadata">

**Author:** ![Luke\_Biddell](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/luke_biddell/32/663_2.png) [@Luke\_Biddell](https://discourse.chef.io/u/Luke_Biddell)\
**Post date:** [March 28, 2011, 12:43pm UTC](https://discourse.chef.io/t/apt-get-update-strategy/1754/8 "2011-03-28T12:43:45Z")

</div>

Perfect, just what I was looking for. Thanks.

On 28 March 2011 12:17, Mason Turner [opsmason@gmail.com](mailto:opsmason@gmail.com) wrote:

> On Mar 28, 2011, at 5:54 AM, Luke Biddell [luke.biddell@gmail.com](mailto:luke.biddell@gmail.com) wrote:
> 
> The flag to indicate if an update has been done is stored on the node  
> and has to be reset at the start of each run. Is there a better way of  
> setting transient attributes for the run?
> 
> Take a look at node.run\_state for transient storage. I stow my searches  
> there for reuse across a few recipes.
> 
> -- Mason Turner (mobile)

---

<div class="post-metadata">

**Author:** ![Michael\_Hale](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/michael_hale/32/834_2.png) [@Michael\_Hale](https://discourse.chef.io/u/Michael_Hale)\
**Post date:** [March 28, 2011, 1:29pm UTC](https://discourse.chef.io/t/apt-get-update-strategy/1754/9 "2011-03-28T13:29:03Z")

</div>

Luke,

I'm thinking you would want someway to apt-get update again if you add  
or remove a repository regardless of whether or not you have already  
updated in a given chef run.

On Mon, Mar 28, 2011 at 8:43 AM, Luke Biddell [luke.biddell@gmail.com](mailto:luke.biddell@gmail.com) wrote:

> Perfect, just what I was looking for. Thanks.
> 
> On 28 March 2011 12:17, Mason Turner [opsmason@gmail.com](mailto:opsmason@gmail.com) wrote:
> 
> > On Mar 28, 2011, at 5:54 AM, Luke Biddell [luke.biddell@gmail.com](mailto:luke.biddell@gmail.com) wrote:
> > 
> > The flag to indicate if an update has been done is stored on the node  
> > and has to be reset at the start of each run. Is there a better way of  
> > setting transient attributes for the run?
> > 
> > Take a look at node.run\_state for transient storage. I stow my searches  
> > there for reuse across a few recipes.
> > 
> > -- Mason Turner (mobile)

---

<div class="post-metadata">

**Author:** ![Michael\_Hale](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/michael_hale/32/834_2.png) [@Michael\_Hale](https://discourse.chef.io/u/Michael_Hale)\
**Post date:** [March 28, 2011, 4:11pm UTC](https://discourse.chef.io/t/apt-get-update-strategy/1754/10 "2011-03-28T16:11:39Z")

</div>

Maybe you could add something like this to your provider:

update\_success\_stamp\_mtime =  
File.mtime("/var/lib/apt/periodic/update-success-stamp")  
Dir["/etc/apt/\*\*/\*.list"].any?{|list\_file| File.mtime(list\_file) \>  
update\_success\_stamp\_mtime }

On Mon, Mar 28, 2011 at 9:28 AM, mikehale [mikehale@gmail.com](mailto:mikehale@gmail.com) wrote:

> Luke,
> 
> I'm thinking you would want someway to apt-get update again if you add or remove a repository regardless of whether or not you have already updated in a given chef run.
> 
> On Mon, Mar 28, 2011 at 8:43 AM, Luke Biddell [luke.biddell@gmail.com](mailto:luke.biddell@gmail.com) wrote:
> 
> > Perfect, just what I was looking for. Thanks.
> > 
> > On 28 March 2011 12:17, Mason Turner [opsmason@gmail.com](mailto:opsmason@gmail.com) wrote:
> > 
> > > On Mar 28, 2011, at 5:54 AM, Luke Biddell [luke.biddell@gmail.com](mailto:luke.biddell@gmail.com) wrote:
> > > 
> > > The flag to indicate if an update has been done is stored on the node  
> > > and has to be reset at the start of each run. Is there a better way of  
> > > setting transient attributes for the run?
> > > 
> > > Take a look at node.run\_state for transient storage. I stow my searches  
> > > there for reuse across a few recipes.
> > > 
> > > -- Mason Turner (mobile)

---

<div class="post-metadata">

**Author:** ![Luke\_Biddell](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/luke_biddell/32/663_2.png) [@Luke\_Biddell](https://discourse.chef.io/u/Luke_Biddell)\
**Post date:** [March 28, 2011, 9:43pm UTC](https://discourse.chef.io/t/apt-get-update-strategy/1754/11 "2011-03-28T21:43:41Z")

</div>

Absolutely, the apt-get update execute resource is not guarded in any  
way. Whenever we modify the content of /etc/apt/sources.list.d/ we use  
a notifies to trigger an apt-get update.

We only guard on package :install (within my custom LWRP) so on a  
fully converged node we don't apt-get update at all.

On 28 March 2011 14:29, Michael Hale [mikehale@gmail.com](mailto:mikehale@gmail.com) wrote:

> Luke,
> 
> I'm thinking you would want someway to apt-get update again if you add  
> or remove a repository regardless of whether or not you have already  
> updated in a given chef run.
> 
> On Mon, Mar 28, 2011 at 8:43 AM, Luke Biddell [luke.biddell@gmail.com](mailto:luke.biddell@gmail.com) wrote:
> 
> > Perfect, just what I was looking for. Thanks.
> > 
> > On 28 March 2011 12:17, Mason Turner [opsmason@gmail.com](mailto:opsmason@gmail.com) wrote:
> > 
> > > On Mar 28, 2011, at 5:54 AM, Luke Biddell [luke.biddell@gmail.com](mailto:luke.biddell@gmail.com) wrote:
> > > 
> > > The flag to indicate if an update has been done is stored on the node  
> > > and has to be reset at the start of each run. Is there a better way of  
> > > setting transient attributes for the run?
> > > 
> > > Take a look at node.run\_state for transient storage. I stow my searches  
> > > there for reuse across a few recipes.
> > > 
> > > -- Mason Turner (mobile)