Resource 'mount' and chef 12.5.1

Tiago_Cruz · October 14, 2015, 1:29am

Hello,

I have nodes machines running this mount point:

[root@pompeia16 ~]# mount | grep cassandra
/dev/md0 on /cassandra-data type ext4 (rw)

With this recipe:

mount '/cassandra-data' do
  device 'cassandra-data'
  device_type :label
  fstype 'ext4'
  options 'rw'
end

But is not working anymore with Chef 12.5.1:

pompeia16.datac.com Chef: 12.5.1  - NOK
pompeia6.datac.com  Chef: 12.3.0 - OK
pompeia4.datac.com  Chef: 12.3.0 - OK
pompeia10.datac.com Chef: 12.4.3 - OK
pompeia5.datac.com  Chef: 12.3.0 - OK
pompeia9.datac.com  Chef: 12.4.3 - OK
pompeia8.datac.com  Chef: 12.4.1 - OK
pompeia2.datac.com  Chef: 12.3.0 - OK
pompeia11.datac.com Chef: 12.4.3 - OK
pompeia7.datac.com  Chef: 12.4.1 - OK
pompeia13.datac.com Chef: 12.4.3 - OK
pompeia12.datac.com Chef: 12.4.3 - OK

The error is:

  * mount[/cassandra-data] action mount
    
    ================================================================================
    Error executing action `mount` on resource 'mount[/cassandra-data]'
    ================================================================================
    
    Mixlib::ShellOut::ShellCommandFailed
    ------------------------------------
    Expected process to exit with [0], but received '1'
    ---- Begin output of umount /cassandra-data ----
    STDOUT: 
    STDERR: umount: /cassandra-data: device is busy.
            (In some cases useful info about processes that use
             the device is found by lsof(8) or fuser(1))
    ---- End output of umount /cassandra-data ----
    Ran umount /cassandra-data returned 1
    
    Resource Declaration:
    ---------------------
    # In /var/cache/chef/cookbooks/cassandra/recipes/pompeia.rb
    
     42: mount '/cassandra-data' do
     43:   device 'cassandra-data'
     44:   device_type :label
     45:   fstype 'ext4'
     46:   options 'rw'
     47: end
     48: 
    
    Compiled Resource:
    ------------------
    # Declared in /var/cache/chef/cookbooks/cassandra/recipes/pompeia.rb:42:in `from_file'
    
    mount("/cassandra-data") do
      action [:mount]
      supports {:remount=>false}
      retries 0
      retry_delay 2
      default_guard_interpreter :default
      mount_point "/cassandra-data"
      device "cassandra-data"
      device_type :label
      fsck_device "-"
      fstype "ext4"
      options "rw"
      dump 0
      pass 2
      declared_type :mount
      cookbook_name "cassandra"
      recipe_name "pompeia"
    end
    

Running handlers:
Running handlers complete
Chef Client failed. 1 resources updated in 17 seconds

Thanks!

martinb3 · October 14, 2015, 1:41am

I would assume this is unrelated to Chef 12.5.1, as it seems to be the unmount command in your actual system that is failing. It doesn’t look like chef/provider/mount/mount.rb has actually changed since June, which encompassed multiple recent releases. Is it possible that something else (i.e. not the 'cassandra-data' device) is currently mounted to /cassandra-data and there’s still something using it?

coderanger · October 14, 2015, 1:47am

Did you both file this on Github and on Discourse manually or did this sync automatically somehow?

Tiago_Cruz · October 14, 2015, 2:27am

Hello @coderanger,

I did this manually, sorry

coderanger · October 14, 2015, 2:28am

Please don’t do that in the future.

Tiago_Cruz · October 14, 2015, 2:36am

Hello @martinb3

Yeah, the cassandra-data is mounted, is used and even so they didn’t fail in older versions

[root@pompeia2 ~]# lsof -n | grep /cassandra-data | wc -l
723
[root@pompeia2 ~]# chef-client && echo $?
...
[2015-10-14T02:29:50+00:00] INFO: Report handlers complete
Chef Client finished, 8/466 resources updated in 41.761216867 seconds
0

[root@pompeia2 ~]# knife  -v
Chef: 12.3.0

I’m not sure why the mount resource is trying to unmount the device. They should just make sure he’s mounted

Mike · October 14, 2015, 5:31am

I think you need to define your resource like so:

mount '/cassandra-data' do
  device '/dev/md0'
  device_type :label
  fstype 'ext4'
  options 'rw'
end

The device specified in your original resource seems to point to an arbitrary string, something you likely don’t want.
See https://docs.chef.io/resource_mount.html#examples for further examples of the mount resource.

EDIT: Upon review, I see you’re using the :label option, something I missed earlier. I guess adding debug logging to this would like help.

Tiago_Cruz · October 14, 2015, 1:30pm

Hello @Mike,

Strange… forcing the ‘device’, the error is:

    Chef::Exceptions::Mount
    -----------------------
    Device /dev/md0 does not exist

But exists, and I’m using mdadm resource to make sure:

/dev/md0        158G   17G  141G  11% /cassandra-data

Recipe:

mdadm "/dev/md0" do
  action :create
  devices [ "/dev/xvdb1", "/dev/xvdc1" ]
  level 0
end
mount '/cassandra-data' do
  device 'cassandra-data'
  device_type :label
  fstype 'ext4'
  options 'rw'
  device '/dev/md0'
  action   [:mount, :enable]
end

If I use not_if instead, it works:

  not_if 'mount | grep /cassandra-data'

But I still thinking this is some bug on knife 12.5.1 'cause works in 12.2.1 when I downgrade

tomhughes · October 14, 2015, 1:39pm

So you've told it to mount by label (device_type :label) but then you've given it a device name (device '/dev/md0') so it is probably trying to find a device with that label and mount it in place of the current mount.

Either change to device_type :device or specify a label as the device.

Unless you really do have a label of /dev/md0 in the device of course...

chahn1138 · October 14, 2015, 1:40pm

Am I correct in asserting that if that gaurd fixed your problem, then your problem was that that resource was already mounted?

Tiago_Cruz · October 14, 2015, 1:47pm

Hello @tomhughes,

This worked with knife 12.3.0, but not with 12.5.1

  * mount[/cassandra-data] action mount
    
    ================================================================================
    Error executing action `mount` on resource 'mount[/cassandra-data]'
    ================================================================================
    
    Mixlib::ShellOut::ShellCommandFailed
    ------------------------------------
    Expected process to exit with [0], but received '1'
    ---- Begin output of umount /cassandra-data ----
    STDOUT: 
    STDERR: umount: /cassandra-data: device is busy.
            (In some cases useful info about processes that use
             the device is found by lsof(8) or fuser(1))
    ---- End output of umount /cassandra-data ----
    Ran umount /cassandra-data returned 1
    
    Resource Declaration:
    ---------------------
    # In /var/cache/chef/cookbooks/cassandra/recipes/pompeia.rb
    
     42: mount '/cassandra-data' do
     43:   device 'cassandra-data'
     44:   device_type :device
     45:   fstype 'ext4'
     46:   options 'rw'
     47:   device '/dev/md0'
     48:   action   [:mount, :enable]
     49: end
    
    Compiled Resource:
    ------------------
    # Declared in /var/cache/chef/cookbooks/cassandra/recipes/pompeia.rb:42:in `from_file'
    
    mount("/cassandra-data") do
      action [:mount, :enable]
      supports {:remount=>false}
      retries 0
      retry_delay 2
      default_guard_interpreter :default
      mount_point "/cassandra-data"
      device "/dev/md0"
      device_type :device
      fsck_device "-"
      fstype "ext4"
      options "rw"
      dump 0
      pass 2
      declared_type :mount
      cookbook_name "cassandra"
      recipe_name "pompeia"
    end

Thanks!

Tiago_Cruz · October 14, 2015, 1:49pm

Hello @chahn1138,

Yeah, the resource is already mounted, and I’m just want make sure that they still mounted always

Lamont_Granquist · October 16, 2015, 5:00pm

The code that changed is here:

https://github.com/chef/chef/pull/3792

James Belchamber wrote that back in Aug 2014, but it was only finally merged in 12.5.1

With the new code it does “mount now remouts if options change” which means that if your options are different from the mounted options, chef detects the difference and remounts the filesystem. That fits with the declarative semantics of resources and my analogy to other resources like when you have file with action :create and the modes differ, we still fix the modes (the names we give the actions are, of course, not declarative which produces some confusion, but that ship sailed a long time ago).

So most likely you need to look at /proc/mounts and make sure that your options really match the mounted options of the filesystem now. We’re probably trying to fix it so that the /only/ option on the filesystem is ‘rw’ and you most likely have other options that show up in /proc/mounts is my guess.

Topic		Replies	Views
Problem mounting Linux share on Windows Chef Infra (archive)	0	556	March 24, 2016
Problems using resources Chef Infra (archive)	1	267	October 17, 2011
Chef-server 14.7.1: chef-server-ctl upgrade failed Chef Infra (archive)	0	488	July 22, 2021
NFS mount not working Windows Server Chef Infra (archive)	5	1102	August 15, 2018
Ruby syntax error Chef Infra (archive)	2	411	July 28, 2018

Resource 'mount' and chef 12.5.1

Related Topics