apt_package_hold or preventing version critical packages from being upgraded


#1

Hey there,

in case someone is interested…

We have a lot of packages which are version critical like Postgres or Erlang, and package upgrades/security fixes with apt-get upgrade/dist-upgrade would lead to service restarts or incompatibilites to other parts in case a never version is available, what needs to be avoided in production… On debian based systems this can be circumvented by doing a

echo ‘packagename hold’ | dpkg --set-selections

which will set a package on hold and exclude it from upgrading when doing a apt-get upgrade/dist-upgrade

e.g.

root@ip-10-39-44-97:~ # echo ‘esl-erlang hold’ | dpkg --set-selections
root@ip-10-39-44-97:~ # dpkg --get-selections |grep esl-erlang
esl-erlang hold
root@ip-10-39-44-97:~ # apt-cache policy esl-erlang
esl-erlang:
Installed: 1:15.b.2-1~debian~squeeze
Candidate: 1:15.b.3-1~debian~squeeze
Version table:
1:15.b.3-1~debian~squeeze 0
500 http://binaries.erlang-solutions.com/debian/ squeeze/contrib amd64 Packages
*** 1:15.b.2-1~debian~squeeze 0
500 http://binaries.erlang-solutions.com/debian/ squeeze/contrib amd64 Packages
100 /var/lib/dpkg/status
1:15.b.1-1~debian~squeeze 0
500 http://binaries.erlang-solutions.com/debian/ squeeze/contrib amd64 Packages
root@ip-10-39-44-97:~ # apt-get dist-upgrade
Reading package lists… Done
Building dependency tree
Reading state information… Done
Calculating upgrade… Done
The following packages have been kept back:
esl-erlang
The following packages will be upgraded:
base-files debian-archive-keyring dpkg dpkg-dev libc-bin libc-dev-bin libc6 libc6-dev libdpkg-perl libexpat1 linux-libc-dev locales
12 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.
Need to get 17.6 MB of archives.
After this operation, 20.5 kB of additional disk space will be used.
Do you want to continue [Y/n]?

qed…

Because it seam’s like I am the only one on earth which needs such a feature (because it’s not already chef and no one cares…), I wrote a resource ‘apt_package_hold’ [1] to do this with chef, like

apt_package_hold “esl-erlang” do
version node[:erlang][:version]
action [:install, :hold]
end

Of course that only works for debian based systems, don’t know if there is a similar mechanism on other platforms.

Greets
Holger Amann
Sauspiel GmbH, Berlin

[1] https://github.com/sauspiel/chef_cookbooks/blob/master/apt/libraries/apt_package_hold.rb


#2

I’m interested! any chance you can file a Jira and send a pull request to
the apt cookbook?

I’m not that proficient with apt so let me ask you something.
If I hold a package and then try to install another one that conflicts with
that one, would this stop that too? That is even more important to me.

Case in point: MongoDB packages from 10gen conflict with each other.
Trying to install two different versions would result in a ping-pong of
uninstall-install-uninstall-install ad libitum… not what I want! :smiley:

On Thu, Dec 6, 2012 at 5:35 PM, Holger Amann holger@sauspiel.de wrote:

Hey there,

in case someone is interested…

We have a lot of packages which are version critical like Postgres or
Erlang, and package upgrades/security fixes with apt-get
upgrade/dist-upgrade would lead to service restarts or incompatibilites to
other parts in case a never version is available, what needs to be avoided
in production… On debian based systems this can be circumvented by doing a

echo ‘packagename hold’ | dpkg --set-selections

which will set a package on hold and exclude it from upgrading when doing
a apt-get upgrade/dist-upgrade

e.g.

root@ip-10-39-44-97:~ # echo ‘esl-erlang hold’ | dpkg --set-selections
root@ip-10-39-44-97:~ # dpkg --get-selections |grep esl-erlang

esl-erlang hold
root@ip-10-39-44-97:~ # apt-cache policy esl-erlang
esl-erlang:
Installed: 1:15.b.2-1~debian~squeeze
Candidate: 1:15.b.3-1~debian~squeeze
Version table:
1:15.b.3-1~debian~squeeze 0
500 http://binaries.erlang-solutions.com/debian/ squeeze/contrib
amd64 Packages
*** 1:15.b.2-1~debian~squeeze 0
500 http://binaries.erlang-solutions.com/debian/ squeeze/contrib
amd64 Packages
100 /var/lib/dpkg/status
1:15.b.1-1~debian~squeeze 0
500 http://binaries.erlang-solutions.com/debian/ squeeze/contrib
amd64 Packages
root@ip-10-39-44-97:~ # apt-get dist-upgrade
Reading package lists… Done
Building dependency tree
Reading state information… Done
Calculating upgrade… Done
The following packages have been kept back:
esl-erlang
The following packages will be upgraded:
base-files debian-archive-keyring dpkg dpkg-dev libc-bin libc-dev-bin
libc6 libc6-dev libdpkg-perl libexpat1 linux-libc-dev locales
12 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.
Need to get 17.6 MB of archives.
After this operation, 20.5 kB of additional disk space will be used.
Do you want to continue [Y/n]?

qed…

Because it seam’s like I am the only one on earth which needs such a
feature (because it’s not already chef and no one cares…), I wrote a
resource ‘apt_package_hold’ [1] to do this with chef, like

apt_package_hold “esl-erlang” do
version node[:erlang][:version]
action [:install, :hold]
end

Of course that only works for debian based systems, don’t know if there is
a similar mechanism on other platforms.

Greets
Holger Amann
Sauspiel GmbH, Berlin

[1]
https://github.com/sauspiel/chef_cookbooks/blob/master/apt/libraries/apt_package_hold.rb


#3

On Dec 19, 2012, at 4:05 AM, Andrea Campi andrea.campi@zephirworks.com quoted Holger Amann holger@sauspiel.de:

We have a lot of packages which are version critical like Postgres or Erlang, and package upgrades/security fixes with apt-get upgrade/dist-upgrade would lead to service restarts or incompatibilites to other parts in case a never version is available, what needs to be avoided in production… On debian based systems this can be circumvented by doing a

echo ‘packagename hold’ | dpkg --set-selections

I’m confused. Why not just lock the version number you want in the cookbook? Then version the cookbooks in your repository. If you want different versions of the same cookbook on different machines, you can control that through environments.

That way, Chef would never try to upgrade a package in the first place.


Brad Knowles brad@shub-internet.org
LinkedIn Profile: http://tinyurl.com/y8kpxu


#4

Am 19.12.2012 um 16:44 schrieb Brad Knowles brad@shub-internet.org:

That way, Chef would never try to upgrade a package in the first place.

Yeah, chef is not the problem in that case. But sometimes there are more or less tons of system packages which are to be upgraded due to bug or security fixes. If you’re (or a staff member) doing ‘apt-get upgarde’ and you’re seeing about 80 packages to be upgraded, how do you manage that? Do you check every single package if it’s managed by chef and maybe ‘locked’? If you don’t care and/or you’re upgrading a version critical package by a mistake (and let apt-get upgrade do it’s work), at least your next chef run will fail and you have to fix your chef role/cookbook/node version attribute for one or more packages, or downgrade the package again. And in the worst case it will upgrade and restart your database!
What’s your way?

Holger Amann
Sauspiel GmbH
Reichenberger Str. 113a
10999 Berlin

holger@sauspiel.de

Tel. +49 (0) 30 61651135
Fax +49 (0) 30 61651138

Handelsregister Berlin: HRB 131781
Steuernummer: 29/014/05394
Geschäftsführung: A. Reissner, M. Kavalar


#5

On Dec 19, 2012, at 8:22 AM, Holger Amann holger@sauspiel.de wrote:

That way, Chef would never try to upgrade a package in the first place.

Yeah, chef is not the problem in that case. But sometimes there are more or less tons of system packages which are to be upgraded due to bug or security fixes. If you’re (or a staff member) doing ‘apt-get upgarde’ and you’re seeing about 80 packages to be upgraded, how do you manage that?

Don’t allow anyone to do that manually. All package management should be done exclusively through Chef. In fact, all systems management of all types should be done exclusively through Chef.

If there is ever an emergency need to have an exception to this rule, and that same emergency happens more than once, you should think about updating your Chef recipes to be able to handle that emergency so that you don’t have to do that manually anymore. Or, at the very least, you should be able to manually kick off the appropriate Chef process.


Brad Knowles brad@shub-internet.org
LinkedIn Profile: http://tinyurl.com/y8kpxu


#6

I also tend to set up my own copy of the repository in this case. If everyone is using that repo and I change files under a controlled process you don’t tend to see that happen.

This is, however, why I am against cookbooks adding repositories as a rule. I would rather have to specify yum::epel in the run list than have a cookbook add it for me.

Jeffrey Hulten
Principal Consultant at Automated Labs, LLC
jeffh@automatedlabs.com 206-853-5216
Skype: jeffhulten

On Dec 19, 2012, at 8:58 AM, Brad Knowles wrote:

On Dec 19, 2012, at 8:22 AM, Holger Amann holger@sauspiel.de wrote:

That way, Chef would never try to upgrade a package in the first place.

Yeah, chef is not the problem in that case. But sometimes there are more or less tons of system packages which are to be upgraded due to bug or security fixes. If you’re (or a staff member) doing ‘apt-get upgarde’ and you’re seeing about 80 packages to be upgraded, how do you manage that?

Don’t allow anyone to do that manually. All package management should be done exclusively through Chef. In fact, all systems management of all types should be done exclusively through Chef.

If there is ever an emergency need to have an exception to this rule, and that same emergency happens more than once, you should think about updating your Chef recipes to be able to handle that emergency so that you don’t have to do that manually anymore. Or, at the very least, you should be able to manually kick off the appropriate Chef process.


Brad Knowles brad@shub-internet.org
LinkedIn Profile: http://tinyurl.com/y8kpxu


#7

Am 19.12.2012 um 17:58 schrieb Brad Knowles brad@shub-internet.org:

Don’t allow anyone to do that manually. All package management should be done exclusively through Chef. In fact, all systems management of all types should be done exclusively through Chef.

If there is ever an emergency need to have an exception to this rule, and that same emergency happens more than once, you should think about updating your Chef recipes to be able to handle that emergency so that you don’t have to do that manually anymore. Or, at the very least, you should be able to manually kick off the appropriate Chef process.

Ok, in theory, but in practice that sounds totally impossible to me. As an example - you’re setting up a naked OS, and after bootstrapping you’re about to install Postgres/Apache/someotherlargeservice which itself will install hundred of libraries as dependencies. If there are updates for one or more of those dependencies, how do you want to do it with chef instead of doing the upgrade step manually? Am I missing some magic chef functionality/cookbook which is able to do that?


#8

Not sure if that’s what Brad had in mind, but you always have the option to
include in your recipe:

execute “apt-get upgrade -y” do
action :run
end

On Fri, Dec 21, 2012 at 6:01 PM, Holger Amann holger@sauspiel.de wrote:

Am 19.12.2012 um 17:58 schrieb Brad Knowles brad@shub-internet.org:

Don’t allow anyone to do that manually. All package management should be
done exclusively through Chef. In fact, all systems management of all
types should be done exclusively through Chef.

If there is ever an emergency need to have an exception to this rule, and
that same emergency happens more than once, you should think about updating
your Chef recipes to be able to handle that emergency so that you don’t
have to do that manually anymore. Or, at the very least, you should be
able to manually kick off the appropriate Chef process.

Ok, in theory, but in practice that sounds totally impossible to me. As an
example - you’re setting up a naked OS, and after bootstrapping you’re
about to install Postgres/Apache/someotherlargeservice which itself will
install hundred of libraries as dependencies. If there are updates for one
or more of those dependencies, how do you want to do it with chef instead
of doing the upgrade step manually? Am I missing some magic chef
functionality/cookbook which is able to do that?


Loic ANTOINE-GOMBEAUD
IT contact & DevOps Engineer

Plinga GmbH | Saarbrücker Straße 20/21 | 10405 Berlin | Germany
E-Mail: loic.gombeaud@plinga.com | skype: loic.plinga
Cell: +49 (0) 160 922 86753

Geschäftsführer: Johannes Kreibohm, Florian Schmidt-Amelung
Eingetragen beim Amtsgericht Charlottenburg, HRB 119994


#9

On Dec 21, 2012, at 9:01 AM, Holger Amann holger@sauspiel.de wrote:

Ok, in theory, but in practice that sounds totally impossible to me. As an example - you’re setting up a naked OS, and after bootstrapping you’re about to install Postgres/Apache/someotherlargeservice which itself will install hundred of libraries as dependencies. If there are updates for one or more of those dependencies, how do you want to do it with chef instead of doing the upgrade step manually? Am I missing some magic chef functionality/cookbook which is able to do that?

What I have done at previous sites where I’ve managed things, I specified the specific version of the high level components I needed inside of versioned cookbooks, and then my environments were locked down to specific versions of each cookbook. I didn’t try to specify specific versions of all sub-components that might be installed as a result of a dependency of a higher level component.

That said, you find out about problems in this area the same way – things break. When they break, you could either tell apt not to try to upgrade a specific component, or you could explicitly configure the installation of that component through Chef – and lock it down to a particular version.

Either way, the outcome is the same. “It’s turtles, all the way down”.


Brad Knowles brad@shub-internet.org
LinkedIn Profile: http://tinyurl.com/y8kpxu