Trouble installing chef 11 on RHEL 6 (AWS Linux)


#1

ok. I’m gonna try it. I’ve been running an ubuntu chef 10.x server for a long
time. Now I’m gonna try to go for chef 11 on AWS Linux.

AWS Linux 2012.09
chef-server-11.0.8-1.el6.x86_64.rpm

When I run “chef-server-ctl reconfigure”, it never gets past:

ruby_block[supervise_rabbitmq_sleep] action run

Anyone have advice? Lots of detail below. Thanks!

kallen

These are the processes in the process table.

_ /bin/bash /usr/bin/chef-server-ctl reconfigure
_ /opt/chef-server/embedded/bin/ruby /opt/chef-server/embedded/bin/omnibus-ctl chef-server /opt/chef-server/embedded/service/omnibus-ctl reconfigure
_ /opt/chef-server/embedded/bin/ruby /opt/chef-server/embedded/bin/chef-solo -c /opt/chef-server/embedded/cookbooks/solo.rb -j /opt/chef-server/embedded/cookbooks/dna.json

The chef-solo pid above is “spinning” stat’ing file
/opt/chef-server/sv/rabbitmq/supervise/ok which doesn’t exist:

strace -f -p 10991

Process 10991 attached with 2 threads - interrupt to quit
[pid 10991] restart_syscall(<… resuming interrupted call …> <unfinished …>
[pid 11144] select(4, [3], NULL, NULL, NULL <unfinished …>
[pid 10991] <… restart_syscall resumed> ) = -1 ETIMEDOUT (Connection timed out)
[pid 10991] futex(0x1b37618, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 10991] clock_gettime(CLOCK_MONOTONIC, {18412, 939895146}) = 0
[pid 10991] time(NULL) = 1371882393
[pid 10991] stat("/opt/chef-server/sv/rabbitmq/supervise/ok", 0x7fff06bc7ae0) = -1 ENOENT (No such file or directory)
[pid 10991] time(NULL) = 1371882393

fwiw:

lsof -p 10991 | tail -6

chef-solo 10991 root 1u CHR 136,1 0t0 4 /dev/pts/1
chef-solo 10991 root 2u CHR 136,1 0t0 4 /dev/pts/1
chef-solo 10991 root 3r FIFO 0,8 0t0 35742 pipe
chef-solo 10991 root 4w FIFO 0,8 0t0 35742 pipe
chef-solo 10991 root 5uW REG 202,1 0 294048 /opt/chef-server/embedded/cookbooks/cache/chef-client-running.pid
chef-solo 10991 root 7u REG 202,1 0 25439 /tmp/chef-tempfile20130622-10991-1flwkn0

ls -l /opt/chef-server/sv/rabbitmq/

total 8
drwxr-xr-x 3 root root 4096 Jun 22 06:19 log
-rwxr-xr-x 1 root root 159 Jun 22 06:19 run

ohai:

chef-server-ctl status

warning: rabbitmq: unable to open supervise/ok: file does not exist

There are no logs in /var/log/chef-server or /opt/chef-server/embedded/logs

The output of the reconfigure, snipped for brevity:

  • directory[/opt/chef-server/sv/rabbitmq/log/main] action create

    • create new directory /opt/chef-server/sv/rabbitmq/log/main
    • change mode from ‘’ to ‘0755’
    • change owner from ‘’ to ‘root’
    • change group from ‘’ to ‘root’
  • template[/opt/chef-server/sv/rabbitmq/run] action create

    • create template[/opt/chef-server/sv/rabbitmq/run]
      — /tmp/chef-tempfile20130622-10991-1xg4am0 2013-06-22 06:19:45.063728595 +0000
      +++ /tmp/chef-rendered-template20130622-10991-1aceze7 2013-06-22 06:19:45.063728595 +0000
      @@ -0,0 +1,4 @@
      +#!/bin/sh
      +exec 2>&1
      +exec chpst -P -u chef_server -U chef_server /usr/bin/env HOME=/var/opt/chef-server/rabbitmq /opt/chef-server/embedded/bin/rabbitmq-server
      +
  • template[/opt/chef-server/sv/rabbitmq/log/run] action create

    • create template[/opt/chef-server/sv/rabbitmq/log/run]
      — /tmp/chef-tempfile20130622-10991-1flwkn0 2013-06-22 06:19:45.159727152 +0000
      +++ /tmp/chef-rendered-template20130622-10991-4vvaii 2013-06-22 06:19:45.159727152 +0000
      @@ -0,0 +1,2 @@
      +#!/bin/sh
      +exec svlogd -tt /var/log/chef-server/rabbitmq
  • file[/opt/chef-server/sv/rabbitmq/down] action delete (up to date)

  • link[/opt/chef-server/init/rabbitmq] action create

    • create symlink at /opt/chef-server/init/rabbitmq to /opt/chef-server/embedded/bin/sv
  • link[/opt/chef-server/service/rabbitmq] action create

    • create symlink at /opt/chef-server/service/rabbitmq to /opt/chef-server/sv/rabbitmq
  • ruby_block[supervise_rabbitmq_sleep] action run <— hang


#2

On Jun 22, 2013, at 12:32 AM, kallen@groknaut.net wrote:

ok. I’m gonna try it. I’ve been running an ubuntu chef 10.x server for a long
time. Now I’m gonna try to go for chef 11 on AWS Linux.

AWS Linux 2012.09
chef-server-11.0.8-1.el6.x86_64.rpm

When I run “chef-server-ctl reconfigure”, it never gets past:

ruby_block[supervise_rabbitmq_sleep] action run

Anyone have advice? Lots of detail below. Thanks!

Hmm. We’re installing Private Chef 11.4 on RHEL 6.4 (in a vmWare environment), and that works just fine. It looks like the latest version of Open Source chef 11 is 11.0.8, which you appear to be installing. Have you tried installing that on RHEL 6.4 outside of AWS, maybe on a VM on your local machine?

I’m just wondering if maybe AWS is the problem, or if the problem actually lies with Chef 11 on RHEL 6.


Brad Knowles brad@shub-internet.org
LinkedIn Profile: http://tinyurl.com/y8kpxu


#3

On Sat, 22 Jun 2013, Brad Knowles wrote:

On Jun 22, 2013, at 12:32 AM, kallen@groknaut.net wrote:

ok. I’m gonna try it. I’ve been running an ubuntu chef 10.x server for a long
time. Now I’m gonna try to go for chef 11 on AWS Linux.

AWS Linux 2012.09
chef-server-11.0.8-1.el6.x86_64.rpm

When I run “chef-server-ctl reconfigure”, it never gets past:

ruby_block[supervise_rabbitmq_sleep] action run

Anyone have advice? Lots of detail below. Thanks!

Hmm. We’re installing Private Chef 11.4 on RHEL 6.4 (in a vmWare environment), and that works just fine. It looks like the latest version of Open Source chef 11 is 11.0.8, which you appear to be installing. Have you tried installing that on RHEL 6.4 outside of AWS, maybe on a VM on your local machine?

I’m just wondering if maybe AWS is the problem, or if the problem actually lies with Chef 11 on RHEL 6.

I haven’t yet tried. I’ll try to make some time early next week if I can.
Gonna try to take the weekend off :>

I did try chef-server 11.0.6 and 11.0.4, both failed with same problem, but
the failure looks different.

kallen

fwiw:

- create template[/opt/chef-server/sv/rabbitmq/run]
    --- /tmp/chef-tempfile20130622-11272-5qid93     2013-06-22 06:57:25.514398549 +0000
    +++ /tmp/chef-rendered-template20130622-11272-jidgro    2013-06-22 06:57:25.514398549 +0000
    @@ -0,0 +1,4 @@
    +#!/bin/sh
    +exec 2>&1
    +exec chpst -P -u chef_server -U chef_server /usr/bin/env HOME=/var/opt/chef-server/rabbitmq /opt/chef-server

/embedded/bin/rabbitmq-server
+

  • template[/opt/chef-server/sv/rabbitmq/log/run] action create

    • create template[/opt/chef-server/sv/rabbitmq/log/run]
      — /tmp/chef-tempfile20130622-11272-1mqheue 2013-06-22 06:57:25.606397191 +0000
      +++ /tmp/chef-rendered-template20130622-11272-1qb8tp1 2013-06-22 06:57:25.602397250 +0000
      @@ -0,0 +1,2 @@
      +#!/bin/sh
      +exec svlogd -tt /var/log/chef-server/rabbitmq
  • file[/opt/chef-server/sv/rabbitmq/down] action delete (up to date)

  • link[/opt/chef-server/init/rabbitmq] action create

    • create symlink at /opt/chef-server/init/rabbitmq to /opt/chef-server/embedded/bin/sv
  • link[/opt/chef-server/service/rabbitmq] action create

    • create symlink at /opt/chef-server/service/rabbitmq to /opt/chef-server/sv/rabbitmq
  • ruby_block[supervise_rabbitmq_sleep] action run

    • execute the ruby block supervise_rabbitmq_sleep
  • service[rabbitmq] action nothing (up to date)

  • execute[/opt/chef-server/bin/chef-server-ctl start rabbitmq] action run
    ================================================================================
    Error executing action run on resource ‘execute[/opt/chef-server/bin/chef-server-ctl start rabbitmq]’
    ================================================================================

Mixlib::ShellOut::ShellCommandFailed

Expected process to exit with [0], but received ‘1’
---- Begin output of /opt/chef-server/bin/chef-server-ctl start rabbitmq ----
STDOUT: warning: rabbitmq: unable to open supervise/ok: file does not exist
STDERR:
---- End output of /opt/chef-server/bin/chef-server-ctl start rabbitmq ----
Ran /opt/chef-server/bin/chef-server-ctl start rabbitmq returned 1

Resource Declaration:

In /opt/chef-server/embedded/cookbooks/chef-server/recipes/rabbitmq.rb

76: execute “/opt/chef-server/bin/chef-server-ctl start rabbitmq” do
77: retries 20
78: end
79:

Compiled Resource:

Declared in /opt/chef-server/embedded/cookbooks/chef-server/recipes/rabbitmq.rb:76:in `from_file’

execute("/opt/chef-server/bin/chef-server-ctl start rabbitmq") do
action "run"
retries 0
retry_delay 2
command "/opt/chef-server/bin/chef-server-ctl start rabbitmq"
backup 5
returns 0
cookbook_name :"chef-server"
recipe_name "rabbitmq"
end

[2013-06-22T06:58:19+00:00] ERROR: Running exception handlers
[2013-06-22T06:58:19+00:00] ERROR: Exception handlers complete
Chef Client failed. 10 resources updated
[2013-06-22T06:58:19+00:00] FATAL: Stacktrace dumped to /opt/chef-server/embedded/cookbooks/cache/chef-stacktrace.out
[2013-06-22T06:58:19+00:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: execute[/opt/chef-server/bin/chef-server-ctl start rabbitmq] (chef-server::rabbitmq line 76) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received ‘1’
---- Begin output of /opt/chef-server/bin/chef-server-ctl start rabbitmq ----
STDOUT: warning: rabbitmq: unable to open supervise/ok: file does not exist
STDERR:
---- End output of /opt/chef-server/bin/chef-server-ctl start rabbitmq ----
Ran /opt/chef-server/bin/chef-server-ctl start rabbitmq returned 1


#4

Do you have qpid running?

Adam

On 6/22/13 12:11 AM, "kallen@groknaut.net" kallen@groknaut.net wrote:

On Sat, 22 Jun 2013, Brad Knowles wrote:

On Jun 22, 2013, at 12:32 AM, kallen@groknaut.net wrote:

ok. I’m gonna try it. I’ve been running an ubuntu chef 10.x server
for a long
time. Now I’m gonna try to go for chef 11 on AWS Linux.

AWS Linux 2012.09
chef-server-11.0.8-1.el6.x86_64.rpm

When I run “chef-server-ctl reconfigure”, it never gets past:

ruby_block[supervise_rabbitmq_sleep] action run

Anyone have advice? Lots of detail below. Thanks!

Hmm. We’re installing Private Chef 11.4 on RHEL 6.4 (in a vmWare
environment), and that works just fine. It looks like the latest
version of Open Source chef 11 is 11.0.8, which you appear to be
installing. Have you tried installing that on RHEL 6.4 outside of AWS,
maybe on a VM on your local machine?

I’m just wondering if maybe AWS is the problem, or if the problem
actually lies with Chef 11 on RHEL 6.

I haven’t yet tried. I’ll try to make some time early next week if I can.
Gonna try to take the weekend off :>

I did try chef-server 11.0.6 and 11.0.4, both failed with same problem,
but
the failure looks different.

kallen

fwiw:

  • create template[/opt/chef-server/sv/rabbitmq/run]
    — /tmp/chef-tempfile20130622-11272-5qid93 2013-06-22
    06:57:25.514398549 +0000
    +++ /tmp/chef-rendered-template20130622-11272-jidgro
    2013-06-22 06:57:25.514398549 +0000
    @@ -0,0 +1,4 @@
    +#!/bin/sh
    +exec 2>&1
    +exec chpst -P -u chef_server -U chef_server /usr/bin/env
    HOME=/var/opt/chef-server/rabbitmq /opt/chef-server
    /embedded/bin/rabbitmq-server
    +
  • template[/opt/chef-server/sv/rabbitmq/log/run] action create

    • create template[/opt/chef-server/sv/rabbitmq/log/run]
      — /tmp/chef-tempfile20130622-11272-1mqheue 2013-06-22
      06:57:25.606397191 +0000
      +++ /tmp/chef-rendered-template20130622-11272-1qb8tp1
      2013-06-22 06:57:25.602397250 +0000
      @@ -0,0 +1,2 @@
      +#!/bin/sh
      +exec svlogd -tt /var/log/chef-server/rabbitmq
  • file[/opt/chef-server/sv/rabbitmq/down] action delete (up to date)

  • link[/opt/chef-server/init/rabbitmq] action create

    • create symlink at /opt/chef-server/init/rabbitmq to
      /opt/chef-server/embedded/bin/sv
  • link[/opt/chef-server/service/rabbitmq] action create

    • create symlink at /opt/chef-server/service/rabbitmq to
      /opt/chef-server/sv/rabbitmq
  • ruby_block[supervise_rabbitmq_sleep] action run

    • execute the ruby block supervise_rabbitmq_sleep
  • service[rabbitmq] action nothing (up to date)

  • execute[/opt/chef-server/bin/chef-server-ctl start rabbitmq] action
    run
    ==========================================================================
    ======
    Error executing action run on resource
    ’execute[/opt/chef-server/bin/chef-server-ctl start rabbitmq]’
    ==========================================================================
    ======

Mixlib::ShellOut::ShellCommandFailed

Expected process to exit with [0], but received ‘1’
---- Begin output of /opt/chef-server/bin/chef-server-ctl start rabbitmq

STDOUT: warning: rabbitmq: unable to open supervise/ok: file does not
exist
STDERR:
---- End output of /opt/chef-server/bin/chef-server-ctl start rabbitmq

Ran /opt/chef-server/bin/chef-server-ctl start rabbitmq returned 1

Resource Declaration:

In /opt/chef-server/embedded/cookbooks/chef-server/recipes/rabbitmq.rb

76: execute “/opt/chef-server/bin/chef-server-ctl start rabbitmq” do
77: retries 20
78: end
79:

Compiled Resource:

Declared in

/opt/chef-server/embedded/cookbooks/chef-server/recipes/rabbitmq.rb:76:in
`from_file’

execute("/opt/chef-server/bin/chef-server-ctl start rabbitmq") do
action "run"
retries 0
retry_delay 2
command "/opt/chef-server/bin/chef-server-ctl start rabbitmq"
backup 5
returns 0
cookbook_name :"chef-server"
recipe_name "rabbitmq"
end

[2013-06-22T06:58:19+00:00] ERROR: Running exception handlers
[2013-06-22T06:58:19+00:00] ERROR: Exception handlers complete
Chef Client failed. 10 resources updated
[2013-06-22T06:58:19+00:00] FATAL: Stacktrace dumped to
/opt/chef-server/embedded/cookbooks/cache/chef-stacktrace.out
[2013-06-22T06:58:19+00:00] FATAL: Mixlib::ShellOut::ShellCommandFailed:
execute[/opt/chef-server/bin/chef-server-ctl start rabbitmq]
(chef-server::rabbitmq line 76) had an error:
Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0],
but received ‘1’
---- Begin output of /opt/chef-server/bin/chef-server-ctl start rabbitmq

STDOUT: warning: rabbitmq: unable to open supervise/ok: file does not
exist
STDERR:
---- End output of /opt/chef-server/bin/chef-server-ctl start rabbitmq

Ran /opt/chef-server/bin/chef-server-ctl start rabbitmq returned 1


#5

On Sun, 23 Jun 2013, Adam Jacob wrote:

Do you have qpid running?

I wish I did. Alas, no.

On 6/22/13 12:11 AM, "kallen@groknaut.net" kallen@groknaut.net wrote:

On Sat, 22 Jun 2013, Brad Knowles wrote:

On Jun 22, 2013, at 12:32 AM, kallen@groknaut.net wrote:

ok. I’m gonna try it. I’ve been running an ubuntu chef 10.x server
for a long
time. Now I’m gonna try to go for chef 11 on AWS Linux.

AWS Linux 2012.09
chef-server-11.0.8-1.el6.x86_64.rpm

When I run “chef-server-ctl reconfigure”, it never gets past:

ruby_block[supervise_rabbitmq_sleep] action run

Anyone have advice? Lots of detail below. Thanks!

Hmm. We’re installing Private Chef 11.4 on RHEL 6.4 (in a vmWare
environment), and that works just fine. It looks like the latest
version of Open Source chef 11 is 11.0.8, which you appear to be
installing. Have you tried installing that on RHEL 6.4 outside of AWS,
maybe on a VM on your local machine?

I’m just wondering if maybe AWS is the problem, or if the problem
actually lies with Chef 11 on RHEL 6.

I haven’t yet tried. I’ll try to make some time early next week if I can.
Gonna try to take the weekend off :>

I did try chef-server 11.0.6 and 11.0.4, both failed with same problem,
but
the failure looks different.

kallen

fwiw:

  • create template[/opt/chef-server/sv/rabbitmq/run]
    — /tmp/chef-tempfile20130622-11272-5qid93 2013-06-22
    06:57:25.514398549 +0000
    +++ /tmp/chef-rendered-template20130622-11272-jidgro
    2013-06-22 06:57:25.514398549 +0000
    @@ -0,0 +1,4 @@
    +#!/bin/sh
    +exec 2>&1
    +exec chpst -P -u chef_server -U chef_server /usr/bin/env
    HOME=/var/opt/chef-server/rabbitmq /opt/chef-server
    /embedded/bin/rabbitmq-server
    +
  • template[/opt/chef-server/sv/rabbitmq/log/run] action create

    • create template[/opt/chef-server/sv/rabbitmq/log/run]
      — /tmp/chef-tempfile20130622-11272-1mqheue 2013-06-22
      06:57:25.606397191 +0000
      +++ /tmp/chef-rendered-template20130622-11272-1qb8tp1
      2013-06-22 06:57:25.602397250 +0000
      @@ -0,0 +1,2 @@
      +#!/bin/sh
      +exec svlogd -tt /var/log/chef-server/rabbitmq
  • file[/opt/chef-server/sv/rabbitmq/down] action delete (up to date)

  • link[/opt/chef-server/init/rabbitmq] action create

    • create symlink at /opt/chef-server/init/rabbitmq to
      /opt/chef-server/embedded/bin/sv
  • link[/opt/chef-server/service/rabbitmq] action create

    • create symlink at /opt/chef-server/service/rabbitmq to
      /opt/chef-server/sv/rabbitmq
  • ruby_block[supervise_rabbitmq_sleep] action run

    • execute the ruby block supervise_rabbitmq_sleep
  • service[rabbitmq] action nothing (up to date)

  • execute[/opt/chef-server/bin/chef-server-ctl start rabbitmq] action
    run
    ==========================================================================
    ======
    Error executing action run on resource
    ’execute[/opt/chef-server/bin/chef-server-ctl start rabbitmq]’
    ==========================================================================
    ======

Mixlib::ShellOut::ShellCommandFailed

Expected process to exit with [0], but received ‘1’
---- Begin output of /opt/chef-server/bin/chef-server-ctl start rabbitmq

STDOUT: warning: rabbitmq: unable to open supervise/ok: file does not
exist
STDERR:
---- End output of /opt/chef-server/bin/chef-server-ctl start rabbitmq

Ran /opt/chef-server/bin/chef-server-ctl start rabbitmq returned 1

Resource Declaration:

In /opt/chef-server/embedded/cookbooks/chef-server/recipes/rabbitmq.rb

76: execute “/opt/chef-server/bin/chef-server-ctl start rabbitmq” do
77: retries 20
78: end
79:

Compiled Resource:

Declared in

/opt/chef-server/embedded/cookbooks/chef-server/recipes/rabbitmq.rb:76:in
`from_file’

execute("/opt/chef-server/bin/chef-server-ctl start rabbitmq") do
action "run"
retries 0
retry_delay 2
command "/opt/chef-server/bin/chef-server-ctl start rabbitmq"
backup 5
returns 0
cookbook_name :"chef-server"
recipe_name "rabbitmq"
end

[2013-06-22T06:58:19+00:00] ERROR: Running exception handlers
[2013-06-22T06:58:19+00:00] ERROR: Exception handlers complete
Chef Client failed. 10 resources updated
[2013-06-22T06:58:19+00:00] FATAL: Stacktrace dumped to
/opt/chef-server/embedded/cookbooks/cache/chef-stacktrace.out
[2013-06-22T06:58:19+00:00] FATAL: Mixlib::ShellOut::ShellCommandFailed:
execute[/opt/chef-server/bin/chef-server-ctl start rabbitmq]
(chef-server::rabbitmq line 76) had an error:
Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0],
but received ‘1’
---- Begin output of /opt/chef-server/bin/chef-server-ctl start rabbitmq

STDOUT: warning: rabbitmq: unable to open supervise/ok: file does not
exist
STDERR:
---- End output of /opt/chef-server/bin/chef-server-ctl start rabbitmq

Ran /opt/chef-server/bin/chef-server-ctl start rabbitmq returned 1


#6

On Sat, Jun 22, 2013 at 2:32 AM, kallen@groknaut.net wrote:

ok. I’m gonna try it. I’ve been running an ubuntu chef 10.x server for a
long
time. Now I’m gonna try to go for chef 11 on AWS Linux.

AWS Linux 2012.09
chef-server-11.0.8-1.el6.x86_64.rpm

See this ticket:
http://tickets.opscode.com/browse/CHEF-3838

I ran into this a while back on Amazon Linux as well. The only fully
supported platforms for chef-server are currently Ubuntu and RHEL.