aws_ebs_raid fails on ec2 node, but only when it's rebooted


#1

Hi!

Something strange it’s happening with one mongoDB node that needs and
EBS raid with provisioned IOPS, using Chef 10.26.0

First, node creates EBS raid type 10 with two disks without problems,
and then use it to hosts mongoDB databases.

Then, I reboot or stops the ec2 node.

When the node finished the boot process, EBS raid devices are
attached to the machine, but when I run chef-client, it fails with
message:

FATAL: ArgumentError: ruby_block[wait_mongodb_data_raid]
(/var/chef/cache/cookbooks/aws/providers/ebs_raid.rb line 182) had an
error: ArgumentError: You tried to set a nested key, where the parent
is not a hash-like object:
aws/raid//var/lib/mongodb/device_map/sdi1/sdi2/sdi1

Node’s info seems right to me:

"aws": {
  "raid": {
    "/var/lib/mongodb": {
      "raid_dev": "md0",
      "device_map": {
        "sdi1": "vol-6bba1647",
        "sdi2": "vol-bbbb1797"
      }
    }
  },
  "ebs_volume": {
    "sdi1": {
      "volume_id": "vol-6bba1647"
    },
    "sdi2": {
      "volume_id": "vol-bbbb1797"
    }
  }
}

So, I tried with different aws cookbook versions: 0.101.3 0.101.2
0.100. All fails in the same steps.

Any suggestions?


#2

I suspect that something in ohai overwriting the existing array, but
only once the ebs volume has mounted.

Are you looking at the failed run log, or the node saved on the Chef
server? Node info only saves on a successful run.

Using ‘chef-shell -z’ (or ‘shef -z’, if you’re 10.X) will drop you
into an irb shell right after everything loads for a chef run. From
there, you can switch to recipe mode, and start digging into
node[‘attr’] values as they would be presented to your recipe code.

The other option is dropping Chef::Log.info statements into the aws
cookbook. That’s got a longer debug loop time, but gives you a bit
more power; you don’t have to approximate what the cookbook is doing,
you can just take it straight off the line.

Cheers,


Nathaniel Eliot
T9 Productions

On Wed, Jun 26, 2013 at 6:06 AM, Jorge Bianquetti jbianquetti@gmail.com wrote:

Hi!

Something strange it’s happening with one mongoDB node that needs and
EBS raid with provisioned IOPS, using Chef 10.26.0

First, node creates EBS raid type 10 with two disks without problems,
and then use it to hosts mongoDB databases.

Then, I reboot or stops the ec2 node.

When the node finished the boot process, EBS raid devices are
attached to the machine, but when I run chef-client, it fails with
message:

FATAL: ArgumentError: ruby_block[wait_mongodb_data_raid]
(/var/chef/cache/cookbooks/aws/providers/ebs_raid.rb line 182) had an
error: ArgumentError: You tried to set a nested key, where the parent
is not a hash-like object:
aws/raid//var/lib/mongodb/device_map/sdi1/sdi2/sdi1

Node’s info seems right to me:

"aws": {
  "raid": {
    "/var/lib/mongodb": {
      "raid_dev": "md0",
      "device_map": {
        "sdi1": "vol-6bba1647",
        "sdi2": "vol-bbbb1797"
      }
    }
  },
  "ebs_volume": {
    "sdi1": {
      "volume_id": "vol-6bba1647"
    },
    "sdi2": {
      "volume_id": "vol-bbbb1797"
    }
  }
}

So, I tried with different aws cookbook versions: 0.101.3 0.101.2
0.100. All fails in the same steps.

Any suggestions?


#3

On Wednesday, June 26, 2013 at 4:06 AM, Jorge Bianquetti wrote:

Hi!

Something strange it’s happening with one mongoDB node that needs and
EBS raid with provisioned IOPS, using Chef 10.26.0

First, node creates EBS raid type 10 with two disks without problems,
and then use it to hosts mongoDB databases.

Then, I reboot or stops the ec2 node.

When the node finished the boot process, EBS raid devices are
attached to the machine, but when I run chef-client, it fails with
message:

FATAL: ArgumentError: ruby_block[wait_mongodb_data_raid]
(/var/chef/cache/cookbooks/aws/providers/ebs_raid.rb line 182) had an
error: ArgumentError: You tried to set a nested key, where the parent
is not a hash-like object:
aws/raid//var/lib/mongodb/device_map/sdi1/sdi2/sdi1

Node’s info seems right to me:

“aws”: {
“raid”: {
"/var/lib/mongodb": {
“raid_dev”: “md0”,
“device_map”: {
“sdi1”: “vol-6bba1647”,
“sdi2”: “vol-bbbb1797”
}
}
},
“ebs_volume”: {
“sdi1”: {
“volume_id”: “vol-6bba1647”
},
“sdi2”: {
“volume_id”: “vol-bbbb1797”
}
}
}

So, I tried with different aws cookbook versions: 0.101.3 0.101.2
0.100. All fails in the same steps.

Any suggestions?
What does the ruby_block look like? Also, if you can try it with Chef 11, the attribute system is reworked so you should not see such inscrutable errors.


Daniel DeLeo