I’m running chef on aws ubuntu’s and a few of my ubuntu instances crashed.
When looking at the log I found that the chef-client process which runs as
init.d script, stumbles upon what seems to be a kernel bug.
Full details are here: https://gist.github.com/3885828
And here’s the gist of the gist is:
[119832.732086] BUG: unable to handle kernel NULL pointer dereference at
[119832.732111] IP:  pick_next_task_fair+0xa7/0x1a0
[119832.732124] PGD 1cee6f067 PUD 1cd5e7067 PMD 0
[119832.732132] Oops: 0000 [#1] SMP
[119832.732137] last sysfs file:
[119832.732145] CPU 1
[119832.732147] Modules linked in: acpiphp raid456 async_pq async_xor xor
async_memcpy async_raid6_recov raid10 raid6_pq async_tx raid1 raid0 multipath
[119832.732177] Pid: 7896, comm: chef-client Not tainted 2.6.38-8-virtual
$ uname -a
Linux hostname 2.6.38-8-virtual #42-Ubuntu SMP Mon Apr 11 04:06:34 UTC 2011
x86_64 x86_64 x86_64 GNU/Linux
Had anyone seem that?
Been googling that but nothing of note came up…
I see a few ways out…
- upgrade ubuntu and hope I don’t see this again. (it doesn’t happen every
day, but of the month I’d been using chef it happened four times to three
different hosts, out of 20 hosts)
- Instead of using init script, use a cron or other method to run chef-client
on the hour.
- OK, move to centos or something else desperate…
I’d be happy to get to the bottom of it and see where the bug really comes from
and whether it was fixed in newer versions of the kernel but I shamefully admit
that so far I’d failed to get to the bottom of this bug.
what would you do?