# Cannot re-enter studio on Ubuntu 18.04: dev/proc/etc filesystems not unmounting

**URL:** <https://discourse.chef.io/t/cannot-re-enter-studio-on-ubuntu-18-04-dev-proc-etc-filesystems-not-unmounting/18238>\
**Category:** Habitat Help\
**Created:** [January 17, 2019, 6:36pm UTC](https://discourse.chef.io/t/cannot-re-enter-studio-on-ubuntu-18-04-dev-proc-etc-filesystems-not-unmounting/18238 "2019-01-17T18:36:12Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![irvingpop\_1](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/irvingpop_1/32/2763_2.png) [@irvingpop\_1](https://discourse.chef.io/u/irvingpop_1)\
**Post date:** [January 17, 2019, 6:36pm UTC](https://discourse.chef.io/t/cannot-re-enter-studio-on-ubuntu-18-04-dev-proc-etc-filesystems-not-unmounting/18238/1 "2019-01-17T18:36:12Z")

</div>

Hi friends, I’ve got a new dev workstation (physical machine, fresh Ubuntu 18.04 install) and am seeing some whacky problems that I don’t understand.

1. I create a studio in `/home/irving/src` - this works fine, I see the typical and expected (?) filesystems get mounted

```auto
devpts on /hab/studios/home--irving--src/dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000)
proc on /hab/studios/home--irving--src/proc type proc (rw,relatime)
sysfs on /hab/studios/home--irving--src/sys type sysfs (rw,relatime)
tmpfs on /hab/studios/home--irving--src/run type tmpfs (rw,relatime)
tmpfs on /hab/studios/home--irving--src/var/run/docker.sock type tmpfs (rw,nosuid,noexec,relatime,size=3283068k,mode=755)
/dev/nvme0n1p3 on /hab/studios/home--irving--src/hab/cache/artifacts type xfs (rw,relatime,attr2,inode64,noquota)
/dev/nvme0n1p3 on /hab/studios/home--irving--src/src type xfs (rw,relatime,attr2,inode64,noquota)

```

1. I exit the studio, all filesystems still stick around

2. I re-enter the studio, and get an error very early:

```auto
$ RUST_LOG=debug hab studio enter
[2019-01-17T18:15:25Z DEBUG habitat_common::ui] UI { shell: Shell { input: InputStream { isatty: true }, out: OutputStream { coloring: Auto, isatty: true, is_colored(): true, supports_color(): true }, err: OutputStream { coloring: Auto, isatty: true, is_colored(): true, supports_color(): true } } }
[2019-01-17T18:15:25Z DEBUG hab::command::studio::enter] Setting HAB_CACHE_KEY_PATH=/home/irving/.hab/cache/keys
[2019-01-17T18:15:25Z DEBUG hab::command::studio::enter] Setting ARTIFACT_PATH=/home/irving/.hab/cache/artifacts
[2019-01-17T18:15:25Z DEBUG habitat_core::os::process::imp] Calling execvp(): ("/usr/bin/sudo") ["-p", "[sudo hab-studio] password for %u: ", "-E", "hab", "studio", "enter"]
[2019-01-17T18:15:25Z DEBUG habitat_common::ui] UI { shell: Shell { input: InputStream { isatty: true }, out: OutputStream { coloring: Auto, isatty: true, is_colored(): true, supports_color(): true }, err: OutputStream { coloring: Auto, isatty: true, is_colored(): true, supports_color(): true } } }
[2019-01-17T18:15:25Z DEBUG habitat_core::os::process::imp] Calling execvp(): ("/hab/pkgs/core/hab-studio/0.73.0/20190115010614/bin/hab-studio") ["enter"]
   hab-studio: Creating Studio at /hab/studios/home--irving--src (default)
mount: mounting proc on /hab/studios/home--irving--src/proc failed: Resource busy

```

1. I attempt to clean up the mounted filesystems (seeing 15-16 of EACH one!), but dev refuses to unmount

```auto
$ grep hab /proc/mounts | awk '{print $2}' | sort | uniq | xargs -n1 sudo umount -f
umount: /hab/studios/home--irving--src/dev: target is busy.

```

1. One time I did a `hab studio rm` and it ended up wiping out some/all of my `/dev/` filesystem and temporarily hosing the box until I rebooted - this scared the crap out of me so I’m nervous about doing that again . 🙂

Has anyone seen this before? what kinds of things can I do to troubleshoot this?

---

<div class="post-metadata">

**Author:** ![irvingpop\_1](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/irvingpop_1/32/2763_2.png) [@irvingpop\_1](https://discourse.chef.io/u/irvingpop_1)\
**Post date:** [January 17, 2019, 7:04pm UTC](https://discourse.chef.io/t/cannot-re-enter-studio-on-ubuntu-18-04-dev-proc-etc-filesystems-not-unmounting/18238/2 "2019-01-17T19:04:50Z")

</div>

Providing a tiny bit more information, I found the VERBOSE flag and it appears that `cleanup_studio` is silently failing:

```auto
1][default:/src:0]# exit
logout
+ cleanup_studio
+ kill_launcher
...
+ umount_fs -v /hab/studios/home--irving--src3/run
+ eval _mount_point=$2
+ _mount_point=/hab/studios/home--irving--src3/run
+ is_fs_mounted /hab/studios/home--irving--src3/run
+ _mount_point=/hab/studios/home--irving--src3/run
+ /hab/pkgs/core/hab-studio/0.73.0/20190115010614/libexec/busybox mount
+ /hab/pkgs/core/hab-studio/0.73.0/20190115010614/libexec/busybox grep -q on /hab/studios/home--irving--src3/run type
+ return 0
+ [-z]
+ umount_fs -v /hab/studios/home--irving--src3/sys
+ eval _mount_point=$2
+ _mount_point=/hab/studios/home--irving--src3/sys
+ is_fs_mounted /hab/studios/home--irving--src3/sys
+ _mount_point=/hab/studios/home--irving--src3/sys
+ /hab/pkgs/core/hab-studio/0.73.0/20190115010614/libexec/busybox mount
+ /hab/pkgs/core/hab-studio/0.73.0/20190115010614/libexec/busybox grep -q on /hab/studios/home--irving--src3/sys type
+ return 0
+ umount_fs -v /hab/studios/home--irving--src3/proc
+ eval _mount_point=$2
+ _mount_point=/hab/studios/home--irving--src3/proc
+ is_fs_mounted /hab/studios/home--irving--src3/proc
+ _mount_point=/hab/studios/home--irving--src3/proc
+ /hab/pkgs/core/hab-studio/0.73.0/20190115010614/libexec/busybox mount
+ /hab/pkgs/core/hab-studio/0.73.0/20190115010614/libexec/busybox grep -q on /hab/studios/home--irving--src3/proc type
+ return 0
+ umount_fs -v /hab/studios/home--irving--src3/dev/pts
+ eval _mount_point=$2
+ _mount_point=/hab/studios/home--irving--src3/dev/pts
+ is_fs_mounted /hab/studios/home--irving--src3/dev/pts
+ _mount_point=/hab/studios/home--irving--src3/dev/pts
+ /hab/pkgs/core/hab-studio/0.73.0/20190115010614/libexec/busybox mount
+ /hab/pkgs/core/hab-studio/0.73.0/20190115010614/libexec/busybox grep -q on /hab/studios/home--irving--src3/dev/pts type
+ return 0
+ umount_fs -v -l /hab/studios/home--irving--src3/dev
+ eval _mount_point=$3
+ _mount_point=/hab/studios/home--irving--src3/dev
+ is_fs_mounted /hab/studios/home--irving--src3/dev
+ _mount_point=/hab/studios/home--irving--src3/dev
+ /hab/pkgs/core/hab-studio/0.73.0/20190115010614/libexec/busybox mount
+ /hab/pkgs/core/hab-studio/0.73.0/20190115010614/libexec/busybox grep -q on /hab/studios/home--irving--src3/dev type
+ return 0
+ umount_fs -v -l /hab/studios/home--irving--src3/var/run/docker.sock
+ eval _mount_point=$3
+ _mount_point=/hab/studios/home--irving--src3/var/run/docker.sock
+ is_fs_mounted /hab/studios/home--irving--src3/var/run/docker.sock
+ _mount_point=/hab/studios/home--irving--src3/var/run/docker.sock
+ /hab/pkgs/core/hab-studio/0.73.0/20190115010614/libexec/busybox mount
+ /hab/pkgs/core/hab-studio/0.73.0/20190115010614/libexec/busybox grep -q on /hab/studios/home--irving--src3/var/run/docker.sock type
+ return 0

```

but they are clearly still mounted.

---

<div class="post-metadata">

**Author:** ![stocksy](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/stocksy/32/2635_2.png) [@stocksy](https://discourse.chef.io/u/stocksy)\
**Post date:** [May 22, 2019, 10:23am UTC](https://discourse.chef.io/t/cannot-re-enter-studio-on-ubuntu-18-04-dev-proc-etc-filesystems-not-unmounting/18238/3 "2019-05-22T10:23:23Z")

</div>

I’m also seeing this with the [chefes/buildkite](https://hub.docker.com/r/chefes/buildkite) image.

I have a script for CI that calls `hab pkg build foo` then enters a studio to run tests against the newly built package.  
The script works fine on my Macbook but when I run it inside [chefes/buildkite](https://hub.docker.com/r/chefes/buildkite) it fails with `mount: mounting proc on /hab/studios/workdir/proc failed: Resource busy`

As a weak workaround, I got my script working by having the 2nd usage of `hab` use the slim studio (`hab studio -q -r /opt/slim run`).  
This is weak because the slim studio also cannot be re-entered. And I’m unsure what limitations I’m putting on myself by having the slim studio.

---

<div class="post-metadata">

**Author:** ![stocksy](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/stocksy/32/2635_2.png) [@stocksy](https://discourse.chef.io/u/stocksy)\
**Post date:** [May 22, 2019, 12:31pm UTC](https://discourse.chef.io/t/cannot-re-enter-studio-on-ubuntu-18-04-dev-proc-etc-filesystems-not-unmounting/18238/4 "2019-05-22T12:31:09Z")

</div>

I filed an issue, with my case as reproduction steps [https://github.com/habitat-sh/habitat/issues/6577](https://github.com/habitat-sh/habitat/issues/6577)

---

<div class="post-metadata">

**Author:** ![chris1](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/chris1/32/2143_2.png) [@chris1](https://discourse.chef.io/u/chris1)\
**Post date:** [May 22, 2019, 9:53pm UTC](https://discourse.chef.io/t/cannot-re-enter-studio-on-ubuntu-18-04-dev-proc-etc-filesystems-not-unmounting/18238/5 "2019-05-22T21:53:50Z")

</div>

I’ve seen this a ton too in the last few months and haven’t pinned down what causes it, but it’s a big pain to clean up manually by `umount`ing each and every mount depth-first

---

<div class="post-metadata">

**Author:** ![smacfarlane](https://avatars.discourse-cdn.com/v4/letter/s/e19adc/32.png) [@smacfarlane](https://discourse.chef.io/u/smacfarlane)\
**Post date:** [May 31, 2019, 7:35pm UTC](https://discourse.chef.io/t/cannot-re-enter-studio-on-ubuntu-18-04-dev-proc-etc-filesystems-not-unmounting/18238/6 "2019-05-31T19:35:50Z")

</div>

@irvingpop @chris I’m fairly confident that this is a musl bug exposed when you have a single /proc/mount entry with \> 1024 characters. See [https://github.com/habitat-sh/habitat/issues/6591](https://github.com/habitat-sh/habitat/issues/6591) for more details.

Can either of you confirm that the systems you saw this on had a mount entry with \>1024 characters? overlayfs with a docker image comprised of multiple layers seems to be the most common trigger, but I’ve got a repro using a loopback mount to a heavily nested set of sub directories.

---

<div class="post-metadata">

**Author:** ![chris1](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/chris1/32/2143_2.png) [@chris1](https://discourse.chef.io/u/chris1)\
**Post date:** [May 31, 2019, 8:53pm UTC](https://discourse.chef.io/t/cannot-re-enter-studio-on-ubuntu-18-04-dev-proc-etc-filesystems-not-unmounting/18238/7 "2019-05-31T20:53:05Z")

</div>

@smacfarlane I’ll check next time I see this, but the docker-driven long mount entry cause is totally plausible where I’ve seen it

---

<div class="post-metadata">

**Author:** ![irvingpop\_1](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/irvingpop_1/32/2763_2.png) [@irvingpop\_1](https://discourse.chef.io/u/irvingpop_1)\
**Post date:** [May 31, 2019, 9:16pm UTC](https://discourse.chef.io/t/cannot-re-enter-studio-on-ubuntu-18-04-dev-proc-etc-filesystems-not-unmounting/18238/8 "2019-05-31T21:16:26Z")

</div>

can confirm, I’ve got mounts \> 1024 line length (cause docker)

```
# mount | wc --max-line-length
1169
```