Habitat and SELinux

Hello,

StackOverflow Crosspost

I’ve just gotten started with Habitat, after running into this issue: https://forums.facto rio.com/viewtopic.php?f=49&t=54619 with a game server on an older glibc. (Actually, compiling glibc was easy, getting the game server to properly use it was more of a hassle…)

Enter Habitat!

After a bit of [trying]: https://github.com/maraaaa/habitat-factorio/commits/master it works! And OMG! Running Factorio on “any” OS seems to be a breeze! Until…

Trying to run hab as a service on my home machine running SELinux… (Is it strange that my home machine is hardened more than the server running services for our Corp?)

SO seemed like a more appropriate location to post this question being that it seems to be an SELinux problem and not a Habitat problem… But after a couple weeks, “tons” of views (it’s my first SO post… and it’s gotten more than one a day, that is a ton for me! :smile: ), maybe it’s time to check the habitat forums! (I just learned about them a week ago actually…) It seems [no one else]: https://discourse.chef.io/search?q=selinux has had the same problems… so maybe a good topic for the forum!

Anyway, to summarize:

  • Followed docs and created a systemd unit file
  • Able to start habitat when SELinux is disabled (not a solution!)
  • Able to start habitat “manually” with hab sup run
  • GoogleFU seems to indicate the error has to do with being linked from /usr/bin ???
  • Am very rusty on SELinux, so really looking for some guidance before just wildly creating modules… (Anyone else got SELinux troubles?)

Happy to post [the errors here from the SO forum]: https://stackoverflow.com/questions/49837987/running-habitat-as-a-service-with-selinux)… but it would literally just be a copy paste of that post to here, which didn’t really seem valuable…

Anyway, even if the guidance is “go talk to the systemd folks” that will be a better lead than generated from SO! :grinning:

Thanks!

This forum actively asks new users to break helpful links… so, sorry… guess you’ll have to remove the spaces yourself…

@maraaaa I can’t believe you’d want to harden your home machine. MADNESS! :smiley:

Just a clarifying question: If you run hab sup run outside of a systemd context , do you still get the selinux errors?

Based on the output it definitely seems like it doesn’t like whatever is in the init process execing something else. @eeyun might have some thoughts on this as well

@maraaaa I can’t believe you’d want to harden your home machine. MADNESS! :smiley:

I know madness!

Just a clarifying question: If you run hab sup run outside of a systemd context , do you still get the selinux errors?

Negative. Runs just fine. In fact, I've toyed with adding a /bin/hab sup run to my /etc/rc.local just to see if it would work... But I hardly ever reboot my machine... lol

Thanks for the quick reply!

Hmm indeed I have a few gut feelings I could trace around. I’ll take a look at this in a little bit here. Right now, getting ready for the commute through Copenhagen and on thumbs only!

Pretty sure we checked the journald logs… and this is only coming because of the 0.56 release (congrats BTW, few… shall we say learning curves… but always awesome to see releases like this!)

Jun 08 00:09:18 server.local audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=habitat comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jun 08 00:09:18 server.local audit[24670]: AVC avc:  denied  { execute } for  pid=24670 comm="(hab)" name="hab" dev="dm-0" ino=1451340 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:default_t:s0 tclass=file permissive=0
Jun 08 00:09:18 server.local systemd[24670]: habitat.service: Failed at step EXEC spawning /bin/hab: Permission denied
-- Subject: Process /bin/hab could not be executed
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- The process /bin/hab could not be executed and failed.
-- 
-- The error number returned by this process is 13.
Jun 08 00:09:18 server.local systemd[1]: habitat.service: Main process exited, code=exited, status=203/EXEC
Jun 08 00:09:18 server.local systemd[1]: habitat.service: Unit entered failed state.
Jun 08 00:09:18 server.local audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=habitat comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
Jun 08 00:09:18 server.local systemd[1]: habitat.service: Failed with result 'exit-code'.
Jun 08 00:09:21 server.local dbus-daemon[970]: [system] Activating service name='org.fedoraproject.Setroubleshootd' requested by ':1.131' (uid=0 pid=941 comm="/usr/sbin/sedispatch " label="system_u:system_r:audisp_t:s0") (using servicehelper)
Jun 08 00:09:21 server.local dbus-daemon[970]: [system] Successfully activated service 'org.fedoraproject.Setroubleshootd'
Jun 08 00:09:22 server.local setroubleshoot[24677]: failed to retrieve rpm info for /hab/pkgs/core/hab/0.56.0/20180530234036/bin/hab
Jun 08 00:09:29 server.local setroubleshoot[24677]: Plugin Exception catchall_labels
Jun 08 00:09:29 server.local setroubleshoot[24677]: SELinux is preventing (hab) from execute access on the file /hab/pkgs/core/hab/0.56.0/20180530234036/bin/hab. For complete SELinux messages run: sealert -l fd88f2f5-adb6-4c9c-bca2-f12866c34836
Jun 08 00:09:29 server.local python3[24677]: SELinux is preventing (hab) from execute access on the file /hab/pkgs/core/hab/0.56.0/20180530234036/bin/hab.
                                          
                                          *****  Plugin catchall (100. confidence) suggests   **************************
                                          
                                          If you believe that (hab) should be allowed execute access on the hab file by default.
                                          Then you should report this as a bug.
                                          You can generate a local policy module to allow this access.
                                          Do
                                          allow this access for now by executing:
                                          # ausearch -c '(hab)' --raw | audit2allow -M my-hab
                                          # semodule -X 300 -i my-hab.pp

(could also be SElinux or auditd updates… but I’m going to say it’s the 0.56 update that uncovered this info :slight_smile: )

I think though… that this selinux module will only work until the next time habitat is updated since the path to the binary will change…?

Well not necessarily, it kind of depends how you’re handling your install. For now we have precompiled binaries that can be curled down and placed anywhere on $PATH. The hab binary itself doesn’t quite care where it exists. However the supervisor also gets pulled down and it’s install path will change on every release.

going to piggyback on this because I have a pretty locked down server with selinux. It appears I can’t start the process. Is there some thing I have to set in selinux to allow me to run hab services?

# RUST_LOG=debug RUST_BACKTRACE=1 hab svc load habitat/builder-datastore --channel "stable" --force
DEBUG 2018-08-24T05:04:38Z: habitat_common::ui: UI { shell: Shell { input: InputStream { isatty: true }, out: OutputStream { coloring: Auto, isatty: true, is_colored(): true, supports_color(): true }, err: OutputStream { coloring: Auto, isatty: true, is_colored(): true, supports_color(): true } } }
DEBUG 2018-08-24T05:04:38Z: hab: clap cli args: ["hab", "svc", "load", "habitat/builder-datastore", "--channel", "stable", "--force"]
DEBUG 2018-08-24T05:04:38Z: hab: remaining cli args: []
DEBUG 2018-08-24T05:04:38Z: hab::config: No CLI config found, loading defaults
DEBUG 2018-08-24T05:04:38Z: tokio::reactor: loop process - 1 events, 0.000s
DEBUG 2018-08-24T05:04:38Z: tokio::reactor: dropping I/O source: 0
✗✗✗
✗✗✗ Connection refused (os error 111)
✗✗✗

@bdangit is your supervisor running? hab svc load omits that connection refused error when I've forgotten to start the supervisor. In the event it's started, continue on reading...

@maraaaa @bdangit can you post the output of the following please:

( ausearch -c 'hab-sup' --raw; ausearch -c 'hab' -c 'hab-launch' --raw; ausearch -c 'hab' --raw )

Message begins:

Ok, so I got a new laptop today! You know the deal, friend of a friend got a new laptop didn't need the old one... of course I can make use of it... which, I mean...

So anyway, first thing I do is blow away windows and put linux on it. On a whim I threw FC28 on it. (we can discuss at length why I chose Fedora in a PM :wink: )

In setting it up, I install habitat and am trying to run it. My experience is similar to the OP. I'm able to hab sup run but when I try to run it as a systemd service, all sorts of AVC denials. (I'll see if I can't come up with a good report... the way hab tries to restart services generated a lot of the same message...! Screenshot%20from%202018-08-26%2003-32-46|690x315

Probably flawed... but I basically ran this:

until systemctl status hab; do 
  systemctl start hab; 
  ( 
    ausearch -c 'hab-sup' --raw; 
    ausearch -c 'hab' -c 'hab-launch' --raw; 
    ausearch -c 'hab' --raw 
  ) | audit2allow -M hab-svc-load; 
  wc -l hab-svc-load.te; 
  semodule -X 300 -i hab-svc-load.pp; 
done

# actually, I just did it as a oneliner, I'm not sure if that ^ will run...
# until systemctl status hab; do systemctl start hab; ( ausearch -c 'hab-sup' --raw; ausearch -c 'hab' -c 'hab-launch' --raw; ausearch -c 'hab' --raw ) | audit2allow -M hab-svc-load; wc -l hab-svc-load.te; semodule -X 300 -i hab-svc-load.pp; done

And came up with this:

# cat my-hab.te 

module my-hab 1.0;

require {
        type tmp_t;
        type init_t;
        type default_t;
        type http_port_t;
        class sock_file { create write };
        class file { create execute execute_no_trans map open read rename setattr unlink write };
        class lnk_file read;
        class dir { create rename reparent rmdir setattr };
        class tcp_socket name_connect;
}

#============= init_t ==============

#!!!! This avc is allowed in the current policy
allow init_t default_t:dir { create rename reparent rmdir setattr };

#!!!! This avc is allowed in the current policy
allow init_t default_t:file { create execute execute_no_trans map open read rename setattr unlink write };

#!!!! This avc is allowed in the current policy
allow init_t default_t:lnk_file read;

#!!!! This avc is allowed in the current policy
allow init_t http_port_t:tcp_socket name_connect;

#!!!! This avc is allowed in the current policy
allow init_t tmp_t:sock_file { create write };

Then I tried a hab svc load and received another error, another iteration of search came up with the below diff.

# diff *.te
2c2
< module hab-svc-load 1.0;
---
> module my-hab 1.0;
5d4
<       type init_t;
6a6
>       type init_t;
10d9
<       class process setpgid;
30d28
< allow init_t self:process setpgid;

I am not an SELinux expert, so I think finding one and running this type enforcement by them would be a good idea before anyone just goes running it...

But I think this should work for anyone:

wget https://gist.github.com/qubitrenegade/0ae199d6369637ba9eb0116ac551ab98 -o my-hab.te
checkmodule -M -m -o my-hab.mod my-hab.te
semodule_package -o my-hab.pp -m my-hab.mod
semodule -X 300 -i hab-svc-load.pp

Now, after doing this, I still can't run core/docker (because file not found?) and trying to run core/consul gives me a DIFFERENT SELinux error (this time for the consul process).

So I think there's something around the /hab filesystem that we could clean up? Maybe a SELinux policy that says "any 'hab' program can do whatever in /hab" but without, you know, invalidating the entire purpose of SELinux? lol.

I think I get what's going on? SELinux is trying to say "/bin/hab is linked to some random binary, we can't trust that"... I'm not so sure on how we say "hey SELinux, it's k"...

(@elliott-davis psssst as this is portable and might leave my home, I'm pro SELinux too! :smile_cat:)

Oh, and! I run CentOS at work and have not run into this...

Thanks! I ran hab sup start > svc.log 2>&1 & and then was able to run hab svc load ...

Each command ran and nothing is coming out on stdout. I am currently able to start some stuff with hab-sup and hab with no AVC denials, yet (and yes I'm running SELINUX permissive mode). Of course, I'm not running into other start up issues that I'm not quite able to understand just yet.