Transient dpkg lock issues?

Chef Infra (archive)
1

Hi, all –

I’m seeing a bit of transient weirdness:

I’m running chef-client in a VMware guest (Ubuntu 9.04), having it
call out to a chef-server running on my laptop.

Occasionally, some apt package installation (though not the same one
every time) will fail because the dpkg lock is still open, presumably
from the preceding apt action.

Here’s an example of how it looks in the logs:

[Thu, 06 Aug 2009 16:09:39 +0000] INFO: Installing package[apache2]
version 2.2.11-2ubuntu2
[Thu, 06 Aug 2009 16:09:48 +0000] INFO: Upgrading package[apache2-
prefork-dev] version from uninstalled to 2.2.11-2ubuntu2
/var/lib/gems/1.8/gems/chef-0.7.4/lib/chef/mixin/command.rb:157:in
run_command': apt-get -q -y install apache2-prefork- dev=2.2.11-2ubuntu2 returned 100, expected 0 (Chef::Exceptions::Exec) ---- Begin output of apt-get -q -y install apache2-prefork- dev=2.2.11-2ubuntu2 ---- STDOUT: STDERR: E: Could not get lock /var/lib/dpkg/lock - open (11 Resource temporarily unavailable) E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?---- End output of apt-get -q -y install apache2-prefork-dev=2.2.11-2ubuntu2 ---- from /var/lib/gems/1.8/gems/chef-0.7.4/lib/chef/mixin/command.rb: 133:inchdir’
from /var/lib/gems/1.8/gems/chef-0.7.4/lib/chef/mixin/command.rb:
133:in run_command' from /var/lib/gems/1.8/gems/chef-0.7.4/lib/chef/provider/package/ apt.rb:68:ininstall_package’
from /var/lib/gems/1.8/gems/chef-0.7.4/lib/chef/provider/package/
apt.rb:73:in upgrade_package' from /var/lib/gems/1.8/gems/chef-0.7.4/lib/chef/provider/package.rb: 70:inaction_upgrade’
from /var/lib/gems/1.8/gems/chef-0.7.4/lib/chef/runner.rb:87:in send' from /var/lib/gems/1.8/gems/chef-0.7.4/lib/chef/runner.rb:87:inconverge’
from /var/lib/gems/1.8/gems/chef-0.7.4/lib/chef/runner.rb:85:in each' from /var/lib/gems/1.8/gems/chef-0.7.4/lib/chef/runner.rb:85:inconverge’
from /var/lib/gems/1.8/gems/chef-0.7.4/lib/chef/
resource_collection.rb:58:in each' from /var/lib/gems/1.8/gems/chef-0.7.4/lib/chef/ resource_collection.rb:57:ineach’
from /var/lib/gems/1.8/gems/chef-0.7.4/lib/chef/runner.rb:63:in
converge' from /var/lib/gems/1.8/gems/chef-0.7.4/lib/chef/client.rb:373:inconverge’
from /var/lib/gems/1.8/gems/chef-0.7.4/lib/chef/client.rb:81:in run' from /var/lib/gems/1.8/gems/chef-0.7.4/lib/chef/application/ client.rb:164:inrun_application’
from /var/lib/gems/1.8/gems/chef-0.7.4/lib/chef/application/
client.rb:162:in loop' from /var/lib/gems/1.8/gems/chef-0.7.4/lib/chef/application/ client.rb:162:inrun_application’
from /var/lib/gems/1.8/gems/chef-0.7.4/lib/chef/application.rb:57:in
run' from /var/lib/gems/1.8/gems/chef-0.7.4/bin/chef-client:26 from /var/lib/gems/1.8/bin/chef-client:19:inload’
from /var/lib/gems/1.8/bin/chef-client:19

It’s a bit maddening, because it’s not always the same package that
fails, and not even every run that some package fails – just often
enough to make me want to close my eyes and hope it will go away!

Any thoughts?

I’ve been reading through Chef::Mixin::Command#popen4, but I’ve
reached my present brain-melting threshold.

Thanks in advance for any leads, – Matthew

2

Is the system setup to automatically run apt-get update?

On Thu, Aug 6, 2009 at 7:16 AM, Matthew Toddmatthew.todd@gmail.com wrote:

Hi, all --

I'm seeing a bit of transient weirdness:

I'm running chef-client in a VMware guest (Ubuntu 9.04), having it call out
to a chef-server running on my laptop.

Occasionally, some apt package installation (though not the same one every
time) will fail because the dpkg lock is still open, presumably from the
preceding apt action.

Here's an example of how it looks in the logs:

[Thu, 06 Aug 2009 16:09:39 +0000] INFO: Installing package[apache2] version
2.2.11-2ubuntu2
[Thu, 06 Aug 2009 16:09:48 +0000] INFO: Upgrading
package[apache2-prefork-dev] version from uninstalled to 2.2.11-2ubuntu2
/var/lib/gems/1.8/gems/chef-0.7.4/lib/chef/mixin/command.rb:157:in
run_command': apt-get -q -y install apache2-prefork-dev=2.2.11-2ubuntu2 returned 100, expected 0 (Chef::Exceptions::Exec) ---- Begin output of apt-get -q -y install apache2-prefork-dev=2.2.11-2ubuntu2 ---- STDOUT: STDERR: E: Could not get lock /var/lib/dpkg/lock - open (11 Resource temporarily unavailable) E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?---- End output of apt-get -q -y install apache2-prefork-dev=2.2.11-2ubuntu2 ---- from /var/lib/gems/1.8/gems/chef-0.7.4/lib/chef/mixin/command.rb:133:in chdir'
from
/var/lib/gems/1.8/gems/chef-0.7.4/lib/chef/mixin/command.rb:133:in
run_command' from /var/lib/gems/1.8/gems/chef-0.7.4/lib/chef/provider/package/apt.rb:68:in install_package'
from
/var/lib/gems/1.8/gems/chef-0.7.4/lib/chef/provider/package/apt.rb:73:in
upgrade_package' from /var/lib/gems/1.8/gems/chef-0.7.4/lib/chef/provider/package.rb:70:in action_upgrade'
from /var/lib/gems/1.8/gems/chef-0.7.4/lib/chef/runner.rb:87:in
send' from /var/lib/gems/1.8/gems/chef-0.7.4/lib/chef/runner.rb:87:in converge'
from /var/lib/gems/1.8/gems/chef-0.7.4/lib/chef/runner.rb:85:in
each' from /var/lib/gems/1.8/gems/chef-0.7.4/lib/chef/runner.rb:85:in converge'
from
/var/lib/gems/1.8/gems/chef-0.7.4/lib/chef/resource_collection.rb:58:in
each' from /var/lib/gems/1.8/gems/chef-0.7.4/lib/chef/resource_collection.rb:57:in each'
from /var/lib/gems/1.8/gems/chef-0.7.4/lib/chef/runner.rb:63:in
converge' from /var/lib/gems/1.8/gems/chef-0.7.4/lib/chef/client.rb:373:in converge'
from /var/lib/gems/1.8/gems/chef-0.7.4/lib/chef/client.rb:81:in run' from /var/lib/gems/1.8/gems/chef-0.7.4/lib/chef/application/client.rb:164:in run_application'
from
/var/lib/gems/1.8/gems/chef-0.7.4/lib/chef/application/client.rb:162:in
loop' from /var/lib/gems/1.8/gems/chef-0.7.4/lib/chef/application/client.rb:162:in run_application'
from /var/lib/gems/1.8/gems/chef-0.7.4/lib/chef/application.rb:57:in
run' from /var/lib/gems/1.8/gems/chef-0.7.4/bin/chef-client:26 from /var/lib/gems/1.8/bin/chef-client:19:in load'
from /var/lib/gems/1.8/bin/chef-client:19

It's a bit maddening, because it's not always the same package that fails,
and not even every run that some package fails -- just often enough to make
me want to close my eyes and hope it will go away!

Any thoughts?

I've been reading through Chef::Mixin::Command#popen4, but I've reached my
present brain-melting threshold.

Thanks in advance for any leads, -- Matthew

--
Joe Van Dyk
http://fixieconsulting.com

3

On Thu, Aug 6, 2009 at 7:16 AM, Matthew Toddmatthew.todd@gmail.com wrote:

E: Unable to lock the administration directory (/var/lib/dpkg/), is another
process using it?

That's apt that's failing, and Chef is just letting you know.

Any change you are running apt elsewhere?

Is this a workstation? Left synaptic open?

Chef shouldn't be doing anything in parallel, so it shouldn't be
stepping on itself via the apt provider.

4

On Aug 6, 2009, at 10:53 PM, Joe Van Dyk wrote:

Is the system setup to automatically run apt-get update?

On Aug 6, 2009, at 11:07 PM, Bryan McLellan wrote:

Any chance you are running apt elsewhere?

Thanks Joe, thanks, Bryan!

You guys provided just the lateral-thinking stimulus I needed:

Short version:
I had 2 chef clients running at the same time!

Long version:
Rather than go through a separate "just install chef-client"
bootstrapping phase, I thought I'd bring up my system by manually
running chef-client the first time, with the appropriate config file
pointing to my chef-server. (This was motivated by wanting to avoid
duplicating my cookbooks: I'm using a slightly modified template for /
etc/chef/client.rb that supports http auth, so I couldn't use the
stock bootstrap-latest.tar.gz.)

This is a bad idea.

Halfway through the convergence phase of my manual chef-client run,
runit sprang into being with its own ongoing chef-client run, which
then raced from time to time for the dpkg lock with the remaining
half of my manual convergence phase. D'oh!

So, I'm backing off this strategy and making my own bootstrapping
tarball.

Cheers! Long live Chef!

-- Matthew

As an aside, dropping the following monkey-patch into one of my
cookbooks' libraries was a huge help in tracking down the problem:

class Chef::Provider::package::Apt < Chef::Provider::Package
def install_package(name, version)
currently_running_processes = ps axf
run_command(
:command => "apt-get -q -y#{expand_options
(@new_resource.options)} install #{name}=#{version}",
:environment => { "DEBIAN_FRONTEND" => "noninteractive" }
)
rescue Chef::Exceptions::Exec
Chef::Log.error("Running processes were:\n#
{currently_running_processes}")
raise
end
end