Resource 'mount' and chef 12.5.1

Chef Infra (archive)
1

Hello,

I have nodes machines running this mount point:

[root@pompeia16 ~]# mount | grep cassandra
/dev/md0 on /cassandra-data type ext4 (rw)

With this recipe:

mount '/cassandra-data' do
  device 'cassandra-data'
  device_type :label
  fstype 'ext4'
  options 'rw'
end

But is not working anymore with Chef 12.5.1:

pompeia16.datac.com Chef: 12.5.1  - NOK
pompeia6.datac.com  Chef: 12.3.0 - OK
pompeia4.datac.com  Chef: 12.3.0 - OK
pompeia10.datac.com Chef: 12.4.3 - OK
pompeia5.datac.com  Chef: 12.3.0 - OK
pompeia9.datac.com  Chef: 12.4.3 - OK
pompeia8.datac.com  Chef: 12.4.1 - OK
pompeia2.datac.com  Chef: 12.3.0 - OK
pompeia11.datac.com Chef: 12.4.3 - OK
pompeia7.datac.com  Chef: 12.4.1 - OK
pompeia13.datac.com Chef: 12.4.3 - OK
pompeia12.datac.com Chef: 12.4.3 - OK

The error is:

  * mount[/cassandra-data] action mount
    
    ================================================================================
    Error executing action `mount` on resource 'mount[/cassandra-data]'
    ================================================================================
    
    Mixlib::ShellOut::ShellCommandFailed
    ------------------------------------
    Expected process to exit with [0], but received '1'
    ---- Begin output of umount /cassandra-data ----
    STDOUT: 
    STDERR: umount: /cassandra-data: device is busy.
            (In some cases useful info about processes that use
             the device is found by lsof(8) or fuser(1))
    ---- End output of umount /cassandra-data ----
    Ran umount /cassandra-data returned 1
    
    Resource Declaration:
    ---------------------
    # In /var/cache/chef/cookbooks/cassandra/recipes/pompeia.rb
    
     42: mount '/cassandra-data' do
     43:   device 'cassandra-data'
     44:   device_type :label
     45:   fstype 'ext4'
     46:   options 'rw'
     47: end
     48: 
    
    Compiled Resource:
    ------------------
    # Declared in /var/cache/chef/cookbooks/cassandra/recipes/pompeia.rb:42:in `from_file'
    
    mount("/cassandra-data") do
      action [:mount]
      supports {:remount=>false}
      retries 0
      retry_delay 2
      default_guard_interpreter :default
      mount_point "/cassandra-data"
      device "cassandra-data"
      device_type :label
      fsck_device "-"
      fstype "ext4"
      options "rw"
      dump 0
      pass 2
      declared_type :mount
      cookbook_name "cassandra"
      recipe_name "pompeia"
    end
    

Running handlers:
Running handlers complete
Chef Client failed. 1 resources updated in 17 seconds

Thanks!

2

I would assume this is unrelated to Chef 12.5.1, as it seems to be the unmount command in your actual system that is failing. It doesnā€™t look like chef/provider/mount/mount.rb has actually changed since June, which encompassed multiple recent releases. Is it possible that something else (i.e. not the 'cassandra-data' device) is currently mounted to /cassandra-data and thereā€™s still something using it?

3

Did you both file this on Github and on Discourse manually or did this sync automatically somehow?

4

Hello @coderanger,

I did this manually, sorry :smile:

5

Please donā€™t do that in the future.

6

Hello @martinb3

Yeah, the cassandra-data is mounted, is used and even so they didnā€™t fail in older versions

[root@pompeia2 ~]# lsof -n | grep /cassandra-data | wc -l
723
[root@pompeia2 ~]# chef-client && echo $?
...
[2015-10-14T02:29:50+00:00] INFO: Report handlers complete
Chef Client finished, 8/466 resources updated in 41.761216867 seconds
0

[root@pompeia2 ~]# knife  -v
Chef: 12.3.0

Iā€™m not sure why the mount resource is trying to unmount the device. They should just make sure heā€™s mounted :smile:

7

I think you need to define your resource like so:

mount '/cassandra-data' do
  device '/dev/md0'
  device_type :label
  fstype 'ext4'
  options 'rw'
end

The device specified in your original resource seems to point to an arbitrary string, something you likely donā€™t want.
See https://docs.chef.io/resource_mount.html#examples for further examples of the mount resource.

EDIT: Upon review, I see youā€™re using the :label option, something I missed earlier. I guess adding debug logging to this would like help.

8

Hello @Mike,

Strangeā€¦ forcing the ā€˜deviceā€™, the error is:

    Chef::Exceptions::Mount
    -----------------------
    Device /dev/md0 does not exist

But exists, and Iā€™m using mdadm resource to make sure:

/dev/md0        158G   17G  141G  11% /cassandra-data
  • Recipe:
mdadm "/dev/md0" do
  action :create
  devices [ "/dev/xvdb1", "/dev/xvdc1" ]
  level 0
end
mount '/cassandra-data' do
  device 'cassandra-data'
  device_type :label
  fstype 'ext4'
  options 'rw'
  device '/dev/md0'
  action   [:mount, :enable]
end

If I use not_if instead, it works:

  not_if 'mount | grep /cassandra-data'

But I still thinking this is some bug on knife 12.5.1 'cause works in 12.2.1 when I downgrade :smile:

9

So you've told it to mount by label (device_type :label) but then you've given it a device name (device '/dev/md0') so it is probably trying to find a device with that label and mount it in place of the current mount.

Either change to device_type :device or specify a label as the device.

Unless you really do have a label of /dev/md0 in the device of course...

10

Am I correct in asserting that if that gaurd fixed your problem, then your problem was that that resource was already mounted?

11

Hello @tomhughes,

This worked with knife 12.3.0, but not with 12.5.1 :frowning:

  * mount[/cassandra-data] action mount
    
    ================================================================================
    Error executing action `mount` on resource 'mount[/cassandra-data]'
    ================================================================================
    
    Mixlib::ShellOut::ShellCommandFailed
    ------------------------------------
    Expected process to exit with [0], but received '1'
    ---- Begin output of umount /cassandra-data ----
    STDOUT: 
    STDERR: umount: /cassandra-data: device is busy.
            (In some cases useful info about processes that use
             the device is found by lsof(8) or fuser(1))
    ---- End output of umount /cassandra-data ----
    Ran umount /cassandra-data returned 1
    
    Resource Declaration:
    ---------------------
    # In /var/cache/chef/cookbooks/cassandra/recipes/pompeia.rb
    
     42: mount '/cassandra-data' do
     43:   device 'cassandra-data'
     44:   device_type :device
     45:   fstype 'ext4'
     46:   options 'rw'
     47:   device '/dev/md0'
     48:   action   [:mount, :enable]
     49: end
    
    Compiled Resource:
    ------------------
    # Declared in /var/cache/chef/cookbooks/cassandra/recipes/pompeia.rb:42:in `from_file'
    
    mount("/cassandra-data") do
      action [:mount, :enable]
      supports {:remount=>false}
      retries 0
      retry_delay 2
      default_guard_interpreter :default
      mount_point "/cassandra-data"
      device "/dev/md0"
      device_type :device
      fsck_device "-"
      fstype "ext4"
      options "rw"
      dump 0
      pass 2
      declared_type :mount
      cookbook_name "cassandra"
      recipe_name "pompeia"
    end

Thanks!

12

Hello @chahn1138,

Yeah, the resource is already mounted, and Iā€™m just want make sure that they still mounted always :smile:

13

The code that changed is here:

https://github.com/chef/chef/pull/3792

James Belchamber wrote that back in Aug 2014, but it was only finally merged in 12.5.1

With the new code it does ā€œmount now remouts if options changeā€ which means that if your options are different from the mounted options, chef detects the difference and remounts the filesystem. That fits with the declarative semantics of resources and my analogy to other resources like when you have file with action :create and the modes differ, we still fix the modes (the names we give the actions are, of course, not declarative which produces some confusion, but that ship sailed a long time ago).

So most likely you need to look at /proc/mounts and make sure that your options really match the mounted options of the filesystem now. Weā€™re probably trying to fix it so that the /only/ option on the filesystem is ā€˜rwā€™ and you most likely have other options that show up in /proc/mounts is my guess.

1 Like