Using ruby_block

Chef Infra (archive)
1

Ohai! chefs!

My Recipe has to import from S3, extract and seed the data from the mysqldump.

To do this, I want it using ruby and the aws-sdk gem.
AFAIK, the ruby code in the recipe file is evaluated at compil time and the ones that are inside ruby_block is evaluated at convergence time.
So if I do require 'aws', it won’t since its not installed. When I add it inside the ruby_block, it errors out.
So to fix it, I used the chef_gem 'aws-sdk', but this will run at the compile time which is fine.
But to install it, it depends on packages like lib-xml2 to be installed too at the compile time.

What I want is, what is a better way to do this using the ruby_block way?

…
if app[‘db_import’]

gem_package ‘aws-sdk’

ruby_block “import_backed-up_files_from_S3” do
block do
# Getting the files from S3 backup
require ‘aws’
…

[2013-01-26T09:12:25+00:00] INFO: Processing gem_package[aws-sdk] action install (empowerme::extract_files_from_s3_backup line 6)
[2013-01-26T09:12:25+00:00] DEBUG: gem_package[aws-sdk] detected omnibus installation in /opt/chef/embedded/bin
[2013-01-26T09:12:25+00:00] DEBUG: gem_package[aws-sdk] searching for ‘gem’ binary in path: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/chef/embedded/bin:/opt/chef/embedded/bin
[2013-01-26T09:12:25+00:00] DEBUG: gem_package[aws-sdk] using gem ‘/usr/bin/gem’
[2013-01-26T09:12:26+00:00] DEBUG: gem_package[aws-sdk] no installed version found for aws-sdk (>= 0)
RubyGems Environment:

  • RUBYGEMS VERSION: 1.8.23
  • RUBY VERSION: 1.9.3 (2012-11-10 patchlevel 327) [i686-linux]
  • INSTALLATION DIRECTORY: /var/lib/gems/1.9.1
  • RUBY EXECUTABLE: /usr/bin/ruby1.9.1
  • EXECUTABLE DIRECTORY: /usr/local/bin
  • RUBYGEMS PLATFORMS:
    • ruby
    • x86-linux
  • GEM PATHS:
    • /var/lib/gems/1.9.1
    • /root/.gem/ruby/1.9.1
  • GEM CONFIGURATION:
    • :update_sources => true
    • :verbose => true
    • :benchmark => false
    • :backtrace => false
    • :bulk_threshold => 1000
  • REMOTE SOURCES:
    • http://rubygems.org/
      [2013-01-26T09:12:27+00:00] DEBUG: found gem aws-sdk version 1.8.0 for platform ruby from http://rubygems.org/
      When you HTTParty, you must party hard!
      Building native extensions. This could take a while…
      Successfully installed uuidtools-2.1.3
      Successfully installed multi_xml-0.5.2
      Successfully installed httparty-0.10.0
      Successfully installed json-1.7.6
      Successfully installed aws-sdk-1.8.0
      5 gems installed
      [2013-01-26T09:12:42+00:00] DEBUG: gem_package[aws-sdk] resetting gem environment to default
      [2013-01-26T09:12:42+00:00] INFO: Processing ruby_block[import_backed-up_files_from_S3] action create (empowerme::extract_files_from_s3_backup line 8)

================================================================================
Error executing action create on resource ‘ruby_block[import_backed-up_files_from_S3]’

LoadError

cannot load such file – aws

Cookbook Trace:

/var/chef/cache/cookbooks/empowerme/recipes/extract_files_from_s3_backup.rb:11:in `block (2 levels) in from_file’

Resource Declaration:

In /var/chef/cache/cookbooks/empowerme/recipes/extract_files_from_s3_backup.rb

8: ruby_block “import_backed-up_files_from_S3” do
9: block do
10: # Getting the files from S3 backup
11: require 'aws’
12: AWS.config(access_key_id: node[‘aws’][‘access_key_id’], secret_access_key: node[‘aws’][‘secret_access_key’])
13: s3 = AWS::S3.new
14:
15: bucket = s3.buckets[node[‘empowerme’][‘s3_bucket’]]
16:
17: file_to_download = bucket.objects.map{|ob| ob.key}.sort.last # e.g. “empowerme/2012.10.14.20.40.02/empowerme.tar"
18:
19: Chef::Log.info(”\n\nBackup Import from S3\n---------- \n #{file_to_download}\n-------------\n")
20:
21: obj = bucket.objects[file_to_download]
22:
23: local_file = Pathname.new("/tmp/#{node[‘empowerme’][‘backup_model_name’]}.tar")
24:
25: File.open(local_file, “w”) do |f|
26: obj.read do |chunk|
27: f.write chunk
28: end
29: end
30:
31: execute “extracting downloaded file” do
32: cwd local_file.dirname.to_s
33: command "tar -xf ./#{local_file.basename.to_path}"
34: user "root"
35: end
36:
37: # execute “change permission on #{app[‘deploy_to’]}” do
38: # command "chmod g+rw #{app[‘deploy_to’]}; chown -R #{app[‘owner’]}:#{app[‘group’]} #{app[‘deploy_to’]}"
39: # end
40:
41: # execute “change permission on #{node[‘empowerme’][‘ebs_mount_dir’]}” do
42: # command "chmod g+rw #{node[‘empowerme’][‘ebs_mount_dir’]} && chown -R #{app[‘owner’]}:#{app[‘group’]} #{node[‘empowerme’][‘ebs_mount_dir’]}"
43: # end
44: end
45: action :create
46: end
47: end

Compiled Resource:

Declared in /var/chef/cache/cookbooks/empowerme/recipes/extract_files_from_s3_backup.rb:8:in `from_file’

ruby_block(“import_backed-up_files_from_S3”) do
action [:create]
retries 0
retry_delay 2
block_name "import_backed-up_files_from_S3"
cookbook_name "empowerme"
recipe_name "extract_files_from_s3_backup"
block #Proc:0x85c7300@/var/chef/cache/cookbooks/empowerme/recipes/extract_files_from_s3_backup.rb:9
end

[2013-01-26T09:12:43+00:00] DEBUG: Re-raising exception: LoadError - ruby_block[import_backed-up_files_from_S3] (empowerme::extract_files_from_s3_backup line 8) had an error: LoadError: cannot load such file – aws
/opt/chef/embedded/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in require' /opt/chef/embedded/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:inrequire’
/var/chef/cache/cookbooks/empowerme/recipes/extract_files_from_s3_backup.rb:11:in block (2 levels) in from_file' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/provider/ruby_block.rb:33:incall’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/provider/ruby_block.rb:33:in block in action_create' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/mixin/why_run.rb:52:incall’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/mixin/why_run.rb:52:in add_action' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/provider.rb:152:inconverge_by’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/provider/ruby_block.rb:32:in action_create' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/provider.rb:115:inrun_action’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/resource.rb:597:in run_action' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/runner.rb:49:inrun_action’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/runner.rb:81:in block (2 levels) in converge' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/runner.rb:81:ineach’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/runner.rb:81:in block in converge' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/resource_collection.rb:94:inblock in execute_each_resource’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/resource_collection/stepable_iterator.rb:116:in call' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/resource_collection/stepable_iterator.rb:116:incall_iterator_block’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/resource_collection/stepable_iterator.rb:85:in step' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/resource_collection/stepable_iterator.rb:104:initerate’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/resource_collection/stepable_iterator.rb:55:in each_with_index' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/resource_collection.rb:92:inexecute_each_resource’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/runner.rb:80:in converge' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/client.rb:378:inconverge’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/client.rb:420:in do_run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/client.rb:176:inrun’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/application.rb:140:in run_chef_client' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/application/client.rb:274:inblock in run_application’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/application/client.rb:267:in loop' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/application/client.rb:267:inrun_application’
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/application.rb:72:in run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/bin/chef-client:26:in<top (required)>’
/usr/bin/chef-client:23:in load' /usr/bin/chef-client:23:in‘
[2013-01-26T09:12:43+00:00] ERROR: Running exception handlers
[2013-01-26T09:12:43+00:00] ERROR: Report handler Chef::Handler::ErrorReport raised #<Encoding::UndefinedConversionError: “\xE2” from ASCII-8BIT to UTF-8>
[2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/resource.rb:500:in encode' [2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/resource.rb:500:into_json’
[2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/resource.rb:500:in to_json' [2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/1.9.1/json/common.rb:278:ingenerate’
[2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/1.9.1/json/common.rb:278:in pretty_generate' [2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/json_compat.rb:49:into_json_pretty’
[2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/handler/error_report.rb:27:in report' [2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/handler.rb:226:inrun_report_unsafe’
[2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/handler.rb:214:in run_report_safely' [2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/handler.rb:118:inblock in run_exception_handlers’
[2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/handler.rb:117:in each' [2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/handler.rb:117:inrun_exception_handlers’
[2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/handler.rb:127:in block in <class:Handler>' [2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/client.rb:118:incall’
[2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/client.rb:118:in block in run_failed' [2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/client.rb:117:ineach’
[2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/client.rb:117:in run_failed' [2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/client.rb:436:inrescue in do_run’
[2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/client.rb:444:in do_run' [2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/client.rb:176:inrun’
[2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/application.rb:140:in run_chef_client' [2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/application/client.rb:274:inblock in run_application’
[2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/application/client.rb:267:in loop' [2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/application/client.rb:267:inrun_application’
[2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/lib/chef/application.rb:72:in run' [2013-01-26T09:12:43+00:00] ERROR: /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-10.18.1/bin/chef-client:26:in<top (required)>’
[2013-01-26T09:12:43+00:00] ERROR: /usr/bin/chef-client:23:in load' [2013-01-26T09:12:43+00:00] ERROR: /usr/bin/chef-client:23:in'
[2013-01-26T09:12:43+00:00] ERROR: Exception handlers complete
[2013-01-26T09:12:43+00:00] DEBUG: Server doesn’t support resource history, skipping resource report.
[2013-01-26T09:12:43+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2013-01-26T09:12:43+00:00] DEBUG: LoadError: ruby_block[import_backed-up_files_from_S3] (empowerme::extract_files_from_s3_backup line 8) had an error: LoadError: cannot load such file – aws

@millisami
~ Sachin Sagar Rai
Ruby on Rails Developer
http://tfm.com.np
http://nepalonrails.com (http://nepalonrails.tumblr.com)
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)

2

On Saturday, January 26, 2013 at 3:56 AM, Sachin Sagar Rai wrote:

Ohai! chefs!

My Recipe has to import from S3, extract and seed the data from the mysqldump.

To do this, I want it using ruby and the aws-sdk gem.
AFAIK, the ruby code in the recipe file is evaluated at compil time and the ones that are inside ruby_block is evaluated at convergence time.
So if I do require 'aws', it won't since its not installed. When I add it inside the ruby_block, it errors out.
So to fix it, I used the chef_gem 'aws-sdk', but this will run at the compile time which is fine.
But to install it, it depends on packages like lib-xml2 to be installed too at the compile time.

You can either continue the arms race of moving resources to compile time, which means you probably need to install build-essentials, libxml2-dev, etc. at compile time, too, or you can split your recipe into smaller bits and put each piece at an appropriate place in the run_list, which might look like this:

  1. build-essential
  2. aws::install_gem (would install libxml2-dev or you could put that in a different recipe before this one if you want)
  3. recipe that does things with aws gem be anywhere after this in your run_list

--
Daniel DeLeo

3

Well @Daniel, I've been doing that way already and using the chef_gem resource and ordered execution of recipes.

I was trying to get feedback whether I'm going in the wrong direction.
Thanks

@millisami
~ Sachin Sagar Rai
Ruby on Rails Developer
http://tfm.com.np
http://nepalonrails.com (http://nepalonrails.tumblr.com)
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)

On Sunday, January 27, 2013 at 1:27 AM, Daniel DeLeo wrote:

On Saturday, January 26, 2013 at 3:56 AM, Sachin Sagar Rai wrote:

Ohai! chefs!

My Recipe has to import from S3, extract and seed the data from the mysqldump.

To do this, I want it using ruby and the aws-sdk gem.
AFAIK, the ruby code in the recipe file is evaluated at compil time and the ones that are inside ruby_block is evaluated at convergence time.
So if I do require 'aws', it won't since its not installed. When I add it inside the ruby_block, it errors out.
So to fix it, I used the chef_gem 'aws-sdk', but this will run at the compile time which is fine.
But to install it, it depends on packages like lib-xml2 to be installed too at the compile time.

You can either continue the arms race of moving resources to compile time, which means you probably need to install build-essentials, libxml2-dev, etc. at compile time, too, or you can split your recipe into smaller bits and put each piece at an appropriate place in the run_list, which might look like this:

  1. build-essential
  2. aws::install_gem (would install libxml2-dev or you could put that in a different recipe before this one if you want)
  3. recipe that does things with aws gem be anywhere after this in your run_list

--
Daniel DeLeo

4

On Saturday, January 26, 2013 at 11:47 AM, Sachin Sagar Rai wrote:

Well @Daniel, I've been doing that way already and using the chef_gem resource and ordered execution of recipes.

I was trying to get feedback whether I'm going in the wrong direction.
Thanks

@millisami
~ Sachin Sagar Rai
There's a lively discussion going on in chef-dev about this topic. You can read the current archives here:

http://lists.opscode.com/sympa/arc/chef-dev/2013-01/msg00053.html

…if you're not subscribed to that list.

My summary is that people feel that compile-time resources should be avoided where possible, but there are some cases where it's the only currently available way to load extensions into Chef.

HTH,

--
Daniel DeLeo