How to stop all processes related to chef


#1

hello,
I newly entered chef a few weeks ago. therefore I’m really confused about
chef administration.
and there is a question about it.
Once I stopped chef server by running chef-server-ctl stop, most of
processes had started by chef, however there were still some processes of
chef like the followings

root 857 1 0 Mar10 ? 00:13:17 runsvdir -P
/opt/opscode/service log: ary failure?runsv postgresql: fatal: unable to
lock supervise/lock: temporary failure?runsv redis_lb: fatal: unable to
lock supervise/lock: temporary failure?runsv opscode-erchef: fatal: unable
to lock supervise/lock: temporary failure?runsv opscode-expander-reindexer:
fatal: unable to lock supervise/lock: temporary failure?runsv nginx: fatal:
unable to lock supervise/lock: temporary failure?
root 18489 1 0 Mar10 ? 00:00:00 runsv opscode-erchef
opscode 18492 18489 0 Mar10 ? 00:00:06 svlogd -tt
/var/log/opscode/opscode-erchef
root 20649 1 0 Mar10 ? 00:00:00 runsv opscode-chef-mover
opscode 20650 20649 0 Mar10 ? 00:00:00 svlogd -tt
/var/log/opscode/opscode-chef-mover
opscode 20651 20649 0 Mar10 ? 00:00:09
/opt/opscode/embedded/service/opscode-chef-mover/erts-5.10.4/bin/beam.smp
-K true -A 5 – -root /opt/opscode/embedded/service/opscode-chef-mover
-progname mover – -home /var/opt/opscode/opscode-chef-mover – -noshell
-boot
/opt/opscode/embedded/service/opscode-chef-mover/releases/2.2.20/mover
-embedded -config
/opt/opscode/embedded/service/opscode-chef-mover/etc/sys.config -name
mover@127.0.0.1 -setcookie mover – runit
root 26356 15114 0 17:51 pts/7 00:00:00 grep chef

So how can i stop all the processes related chef, completely?

Thx


#2

Kill 'em off and see if they come back: pkill -9 -f "(runsvdir|runsv|opscode|runit)"

I would probably expect chef-server-ctl stop to work in this case.
May be worth filing a bug.

cheers,

–aj

On Wed, Mar 11, 2015 at 9:52 PM, 범희대 heedae@solbox.com wrote:

hello,
I newly entered chef a few weeks ago. therefore I’m really confused about
chef administration.
and there is a question about it.
Once I stopped chef server by running chef-server-ctl stop, most of
processes had started by chef, however there were still some processes of
chef like the followings

root 857 1 0 Mar10 ? 00:13:17 runsvdir -P
/opt/opscode/service log: ary failure?runsv postgresql: fatal: unable to
lock supervise/lock: temporary failure?runsv redis_lb: fatal: unable to lock
supervise/lock: temporary failure?runsv opscode-erchef: fatal: unable to
lock supervise/lock: temporary failure?runsv opscode-expander-reindexer:
fatal: unable to lock supervise/lock: temporary failure?runsv nginx: fatal:
unable to lock supervise/lock: temporary failure?
root 18489 1 0 Mar10 ? 00:00:00 runsv opscode-erchef
opscode 18492 18489 0 Mar10 ? 00:00:06 svlogd -tt
/var/log/opscode/opscode-erchef
root 20649 1 0 Mar10 ? 00:00:00 runsv opscode-chef-mover
opscode 20650 20649 0 Mar10 ? 00:00:00 svlogd -tt
/var/log/opscode/opscode-chef-mover
opscode 20651 20649 0 Mar10 ? 00:00:09
/opt/opscode/embedded/service/opscode-chef-mover/erts-5.10.4/bin/beam.smp -K
true -A 5 – -root /opt/opscode/embedded/service/opscode-chef-mover
-progname mover – -home /var/opt/opscode/opscode-chef-mover – -noshell
-boot /opt/opscode/embedded/service/opscode-chef-mover/releases/2.2.20/mover
-embedded -config
/opt/opscode/embedded/service/opscode-chef-mover/etc/sys.config -name
mover@127.0.0.1 -setcookie mover – runit
root 26356 15114 0 17:51 pts/7 00:00:00 grep chef

So how can i stop all the processes related chef, completely?

Thx


#3

Did you run that command as root/sudo? I would expect a failure if not, but maybe it fails silently.


Michael F. Weinberg | Director of Operations
http://heavywaterops.com | @heavywaterops

On Mar 11, 2015, at 12:09 PM, AJ Christensen aj@junglistheavy.industries wrote:

Kill 'em off and see if they come back: pkill -9 -f "(runsvdir|runsv|opscode|runit)"

I would probably expect chef-server-ctl stop to work in this case.
May be worth filing a bug.

cheers,

–aj

On Wed, Mar 11, 2015 at 9:52 PM, 범희대 heedae@solbox.com wrote:
hello,
I newly entered chef a few weeks ago. therefore I’m really confused about
chef administration.
and there is a question about it.
Once I stopped chef server by running chef-server-ctl stop, most of
processes had started by chef, however there were still some processes of
chef like the followings

root 857 1 0 Mar10 ? 00:13:17 runsvdir -P
/opt/opscode/service log: ary failure?runsv postgresql: fatal: unable to
lock supervise/lock: temporary failure?runsv redis_lb: fatal: unable to lock
supervise/lock: temporary failure?runsv opscode-erchef: fatal: unable to
lock supervise/lock: temporary failure?runsv opscode-expander-reindexer:
fatal: unable to lock supervise/lock: temporary failure?runsv nginx: fatal:
unable to lock supervise/lock: temporary failure?
root 18489 1 0 Mar10 ? 00:00:00 runsv opscode-erchef
opscode 18492 18489 0 Mar10 ? 00:00:06 svlogd -tt
/var/log/opscode/opscode-erchef
root 20649 1 0 Mar10 ? 00:00:00 runsv opscode-chef-mover
opscode 20650 20649 0 Mar10 ? 00:00:00 svlogd -tt
/var/log/opscode/opscode-chef-mover
opscode 20651 20649 0 Mar10 ? 00:00:09
/opt/opscode/embedded/service/opscode-chef-mover/erts-5.10.4/bin/beam.smp -K
true -A 5 – -root /opt/opscode/embedded/service/opscode-chef-mover
-progname mover – -home /var/opt/opscode/opscode-chef-mover – -noshell
-boot /opt/opscode/embedded/service/opscode-chef-mover/releases/2.2.20/mover
-embedded -config
/opt/opscode/embedded/service/opscode-chef-mover/etc/sys.config -name
mover@127.0.0.1 -setcookie mover – runit
root 26356 15114 0 17:51 pts/7 00:00:00 grep chef

So how can i stop all the processes related chef, completely?

Thx


#4

chef-server-ctl stop leaves the runsvdir, runsv, svlogd, etc runit
processes running by design. i suspect the mover process is there for
supporting upgrades as well which needs to be running when the other
services are down.

On 3/11/15 12:09 PM, AJ Christensen wrote:

Kill 'em off and see if they come back: pkill -9 -f "(runsvdir|runsv|opscode|runit)"

I would probably expect chef-server-ctl stop to work in this case.
May be worth filing a bug.

cheers,

–aj

On Wed, Mar 11, 2015 at 9:52 PM, 범희대 heedae@solbox.com wrote:

hello,
I newly entered chef a few weeks ago. therefore I’m really confused about
chef administration.
and there is a question about it.
Once I stopped chef server by running chef-server-ctl stop, most of
processes had started by chef, however there were still some processes of
chef like the followings

root 857 1 0 Mar10 ? 00:13:17 runsvdir -P
/opt/opscode/service log: ary failure?runsv postgresql: fatal: unable to
lock supervise/lock: temporary failure?runsv redis_lb: fatal: unable to lock
supervise/lock: temporary failure?runsv opscode-erchef: fatal: unable to
lock supervise/lock: temporary failure?runsv opscode-expander-reindexer:
fatal: unable to lock supervise/lock: temporary failure?runsv nginx: fatal:
unable to lock supervise/lock: temporary failure?
root 18489 1 0 Mar10 ? 00:00:00 runsv opscode-erchef
opscode 18492 18489 0 Mar10 ? 00:00:06 svlogd -tt
/var/log/opscode/opscode-erchef
root 20649 1 0 Mar10 ? 00:00:00 runsv opscode-chef-mover
opscode 20650 20649 0 Mar10 ? 00:00:00 svlogd -tt
/var/log/opscode/opscode-chef-mover
opscode 20651 20649 0 Mar10 ? 00:00:09
/opt/opscode/embedded/service/opscode-chef-mover/erts-5.10.4/bin/beam.smp -K
true -A 5 – -root /opt/opscode/embedded/service/opscode-chef-mover
-progname mover – -home /var/opt/opscode/opscode-chef-mover – -noshell
-boot /opt/opscode/embedded/service/opscode-chef-mover/releases/2.2.20/mover
-embedded -config
/opt/opscode/embedded/service/opscode-chef-mover/etc/sys.config -name
mover@127.0.0.1 -setcookie mover – runit
root 26356 15114 0 17:51 pts/7 00:00:00 grep chef

So how can i stop all the processes related chef, completely?

Thx


#5

@AJ : I already tried it but they are still alive. T.T
@Michale : yes, surely i did.
@Lamont Granquist : thanks, helpful!

Then, could u tell me which ailing list I could get the solution from, else
than this?

Thx.

2015-03-12 7:59 GMT+09:00 Lamont Granquist lamont@chef.io:

chef-server-ctl stop leaves the runsvdir, runsv, svlogd, etc runit
processes running by design. i suspect the mover process is there for
supporting upgrades as well which needs to be running when the other
services are down.

On 3/11/15 12:09 PM, AJ Christensen wrote:

Kill 'em off and see if they come back: pkill -9 -f "(runsvdir|runsv|opscode|runit)"

I would probably expect chef-server-ctl stop to work in this case.
May be worth filing a bug.

cheers,

–aj

On Wed, Mar 11, 2015 at 9:52 PM, 범희대 heedae@solbox.com wrote:

hello,
I newly entered chef a few weeks ago. therefore I’m really confused about
chef administration.
and there is a question about it.
Once I stopped chef server by running chef-server-ctl stop, most of
processes had started by chef, however there were still some processes of
chef like the followings

root 857 1 0 Mar10 ? 00:13:17 runsvdir -P
/opt/opscode/service log: ary failure?runsv postgresql: fatal: unable to
lock supervise/lock: temporary failure?runsv redis_lb: fatal: unable to
lock
supervise/lock: temporary failure?runsv opscode-erchef: fatal: unable to
lock supervise/lock: temporary failure?runsv opscode-expander-reindexer:
fatal: unable to lock supervise/lock: temporary failure?runsv nginx:
fatal:
unable to lock supervise/lock: temporary failure?
root 18489 1 0 Mar10 ? 00:00:00 runsv opscode-erchef
opscode 18492 18489 0 Mar10 ? 00:00:06 svlogd -tt
/var/log/opscode/opscode-erchef
root 20649 1 0 Mar10 ? 00:00:00 runsv opscode-chef-mover
opscode 20650 20649 0 Mar10 ? 00:00:00 svlogd -tt
/var/log/opscode/opscode-chef-mover
opscode 20651 20649 0 Mar10 ? 00:00:09
/opt/opscode/embedded/service/opscode-chef-mover/erts-5.10.4/bin/beam.smp
-K
true -A 5 – -root /opt/opscode/embedded/service/opscode-chef-mover
-progname mover – -home /var/opt/opscode/opscode-chef-mover – -noshell
-boot /opt/opscode/embedded/service/opscode-chef-mover/releases/2.
2.20/mover
-embedded -config
/opt/opscode/embedded/service/opscode-chef-mover/etc/sys.config -name
mover@127.0.0.1 -setcookie mover – runit
root 26356 15114 0 17:51 pts/7 00:00:00 grep chef

So how can i stop all the processes related chef, completely?

Thx


#6

Hi,

The steps will depend on what platform you are on. The problem you
are likely running into is this:

  • On SysV-based system the current installer places an entry in
    inittab with respawn set

https://github.com/opscode-cookbooks/enterprise-chef-common/blob/master/recipes/runit_sysvinit.rb#L12

  • On Upstart-based system they set Restart=always:

In both cases, the systems init system will respawn runsvdir (the
top-level supervisor for runit that we use to manage servicies), which
in tern will restart all the services.

For SysV systems (RHEL 5), you probably want to do something like this

chef-server-ctl stop

modify inittab to remove the respawn entry

init q

For upstart it is a little more complicated because upstart will send
upstart the wrong signal by default and not all versions of upstart
allow you to control the kill signal. I would try something like
this:

chef-server-ctl stop
initctl stop chef-server-runsvdir

a command like AJs to clean up anything that got left behind

because of the broken upstart script

I hope this helps.

Sincerely,

Steven

On Fri, Mar 13, 2015 at 8:45 AM, 범희대 heedae@solbox.com wrote:

@AJ : I already tried it but they are still alive. T.T
@Michale : yes, surely i did.
@Lamont Granquist : thanks, helpful!

Then, could u tell me which ailing list I could get the solution from, else
than this?

Thx.

2015-03-12 7:59 GMT+09:00 Lamont Granquist lamont@chef.io:

chef-server-ctl stop leaves the runsvdir, runsv, svlogd, etc runit
processes running by design. i suspect the mover process is there for
supporting upgrades as well which needs to be running when the other
services are down.

On 3/11/15 12:09 PM, AJ Christensen wrote:

Kill 'em off and see if they come back: pkill -9 -f "(runsvdir|runsv|opscode|runit)"

I would probably expect chef-server-ctl stop to work in this case.
May be worth filing a bug.

cheers,

–aj

On Wed, Mar 11, 2015 at 9:52 PM, 범희대 heedae@solbox.com wrote:

hello,
I newly entered chef a few weeks ago. therefore I’m really confused
about
chef administration.
and there is a question about it.
Once I stopped chef server by running chef-server-ctl stop, most of
processes had started by chef, however there were still some processes
of
chef like the followings

root 857 1 0 Mar10 ? 00:13:17 runsvdir -P
/opt/opscode/service log: ary failure?runsv postgresql: fatal: unable to
lock supervise/lock: temporary failure?runsv redis_lb: fatal: unable to
lock
supervise/lock: temporary failure?runsv opscode-erchef: fatal: unable to
lock supervise/lock: temporary failure?runsv opscode-expander-reindexer:
fatal: unable to lock supervise/lock: temporary failure?runsv nginx:
fatal:
unable to lock supervise/lock: temporary failure?
root 18489 1 0 Mar10 ? 00:00:00 runsv opscode-erchef
opscode 18492 18489 0 Mar10 ? 00:00:06 svlogd -tt
/var/log/opscode/opscode-erchef
root 20649 1 0 Mar10 ? 00:00:00 runsv opscode-chef-mover
opscode 20650 20649 0 Mar10 ? 00:00:00 svlogd -tt
/var/log/opscode/opscode-chef-mover
opscode 20651 20649 0 Mar10 ? 00:00:09

/opt/opscode/embedded/service/opscode-chef-mover/erts-5.10.4/bin/beam.smp -K
true -A 5 – -root /opt/opscode/embedded/service/opscode-chef-mover
-progname mover – -home /var/opt/opscode/opscode-chef-mover – -noshell
-boot
/opt/opscode/embedded/service/opscode-chef-mover/releases/2.2.20/mover
-embedded -config
/opt/opscode/embedded/service/opscode-chef-mover/etc/sys.config -name
mover@127.0.0.1 -setcookie mover – runit
root 26356 15114 0 17:51 pts/7 00:00:00 grep chef

So how can i stop all the processes related chef, completely?

Thx


#7

Hi,

Quick follow up. I accidentally linked to the systemd config. The
upstart config is here;

On Fri, Mar 13, 2015 at 12:43 PM, Steven Danna steve@chef.io wrote:

Hi,

The steps will depend on what platform you are on. The problem you
are likely running into is this:

  • On SysV-based system the current installer places an entry in
    inittab with respawn set

https://github.com/opscode-cookbooks/enterprise-chef-common/blob/master/recipes/runit_sysvinit.rb#L12

  • On Upstart-based system they set Restart=always:

https://github.com/opscode-cookbooks/enterprise-chef-common/blob/master/templates/default/runsvdir-start.service.erb#L6

In both cases, the systems init system will respawn runsvdir (the
top-level supervisor for runit that we use to manage servicies), which
in tern will restart all the services.

For SysV systems (RHEL 5), you probably want to do something like this

chef-server-ctl stop

modify inittab to remove the respawn entry

init q

For upstart it is a little more complicated because upstart will send
upstart the wrong signal by default and not all versions of upstart
allow you to control the kill signal. I would try something like
this:

chef-server-ctl stop
initctl stop chef-server-runsvdir

a command like AJs to clean up anything that got left behind

because of the broken upstart script

I hope this helps.

Sincerely,

Steven

On Fri, Mar 13, 2015 at 8:45 AM, 범희대 heedae@solbox.com wrote:

@AJ : I already tried it but they are still alive. T.T
@Michale : yes, surely i did.
@Lamont Granquist : thanks, helpful!

Then, could u tell me which ailing list I could get the solution from, else
than this?

Thx.

2015-03-12 7:59 GMT+09:00 Lamont Granquist lamont@chef.io:

chef-server-ctl stop leaves the runsvdir, runsv, svlogd, etc runit
processes running by design. i suspect the mover process is there for
supporting upgrades as well which needs to be running when the other
services are down.

On 3/11/15 12:09 PM, AJ Christensen wrote:

Kill 'em off and see if they come back: pkill -9 -f "(runsvdir|runsv|opscode|runit)"

I would probably expect chef-server-ctl stop to work in this case.
May be worth filing a bug.

cheers,

–aj

On Wed, Mar 11, 2015 at 9:52 PM, 범희대 heedae@solbox.com wrote:

hello,
I newly entered chef a few weeks ago. therefore I’m really confused
about
chef administration.
and there is a question about it.
Once I stopped chef server by running chef-server-ctl stop, most of
processes had started by chef, however there were still some processes
of
chef like the followings

root 857 1 0 Mar10 ? 00:13:17 runsvdir -P
/opt/opscode/service log: ary failure?runsv postgresql: fatal: unable to
lock supervise/lock: temporary failure?runsv redis_lb: fatal: unable to
lock
supervise/lock: temporary failure?runsv opscode-erchef: fatal: unable to
lock supervise/lock: temporary failure?runsv opscode-expander-reindexer:
fatal: unable to lock supervise/lock: temporary failure?runsv nginx:
fatal:
unable to lock supervise/lock: temporary failure?
root 18489 1 0 Mar10 ? 00:00:00 runsv opscode-erchef
opscode 18492 18489 0 Mar10 ? 00:00:06 svlogd -tt
/var/log/opscode/opscode-erchef
root 20649 1 0 Mar10 ? 00:00:00 runsv opscode-chef-mover
opscode 20650 20649 0 Mar10 ? 00:00:00 svlogd -tt
/var/log/opscode/opscode-chef-mover
opscode 20651 20649 0 Mar10 ? 00:00:09

/opt/opscode/embedded/service/opscode-chef-mover/erts-5.10.4/bin/beam.smp -K
true -A 5 – -root /opt/opscode/embedded/service/opscode-chef-mover
-progname mover – -home /var/opt/opscode/opscode-chef-mover – -noshell
-boot
/opt/opscode/embedded/service/opscode-chef-mover/releases/2.2.20/mover
-embedded -config
/opt/opscode/embedded/service/opscode-chef-mover/etc/sys.config -name
mover@127.0.0.1 -setcookie mover – runit
root 26356 15114 0 17:51 pts/7 00:00:00 grep chef

So how can i stop all the processes related chef, completely?

Thx