Error executing action `converge` on resource 'machine[mychefserver]'

Hi,

I am getting below error, while trying to create chef cluster using command: rake setup:cluster --trace

  • double-checked the configuration steps
  • checked the AWS parameters
  • also tried a few steps on provisioning node, to make sure that connectivity or permissions issues are not there

Converging 33 resources
Recipe: delivery-cluster::_settings

  • link[/opt/chefproject/delivery-cluster/.chef/…/.chef/delivery-cluster-data] action create (up to date)
    Recipe: delivery-cluster::setup_chef_server

  • machine[mychefserver] action converge

    ================================================================================
    Error executing action converge on resource ‘machine[mychefserver]’

    SocketError

    getaddrinfo: Name or service not known

    Resource Declaration:

    In /opt/chefproject/delivery-cluster/.chef/local-mode-cache/cache/cookbooks/delivery-cluster/recipes/setup_chef_server.rb

    31: machine chef_server_hostname do
    32: provisioning.specific_machine_options(‘chef-server’).each do |option|
    33: add_machine_options(option)
    34: end
    35: # Transfer any trusted certs
    36: Dir.glob("#{Chef::Config[:trusted_certs_dir]}/*").each do |cert_path|
    37: file ::File.join(’/etc/chef/trusted_certs’, ::File.basename(cert_path)), cert_path
    38: end
    39: action :converge
    40: end
    41:

    Compiled Resource:

    Declared in /opt/chefproject/delivery-cluster/.chef/local-mode-cache/cache/cookbooks/delivery-cluster/recipes/setup_chef_server.rb:31:in `from_file’

    machine(“mychefserver”) do
    action [:converge]
    retries 0
    retry_delay 2
    default_guard_interpreter :default
    chef_server {:chef_server_url=>“chefzero://localhost:8890”, :options=>{:client_name=>“delivery”, :signing_key_filename=>nil, :api_version=>“0”}}
    driver "aws"
    machine_options #<Cheffish::MergedConfig:0x000000082917d0 @configs=[{:bootstrap_options=>{:instance_type=>“c3.xlarge”}}, {:convergence_options=>{:bootstrap_proxy=>nil, :chef_config=>nil, :chef_version=>nil}, :bootstrap_options=>{:instance_type=>nil, :key_name=>“LCCASTACK”, :security_group_ids=>“sg-c8d3c1ac”, :subnet_id=>“subnet-6862d31f”}, :ssh_username=>“ubuntu”, :image_id=>“ami-3d50120d”, :use_private_ip_for_ssh=>true}], @merge_arrays={}>
    declared_type :machine
    cookbook_name "delivery-cluster"
    recipe_name "setup_chef_server"
    end

Running handlers:
[2016-01-05T20:54:44+00:00] ERROR: Running exception handlers
Running handlers complete
[2016-01-05T20:54:44+00:00] ERROR: Exception handlers complete
Chef Client failed. 0 resources updated in 11 seconds
[2016-01-05T20:54:44+00:00] FATAL: Stacktrace dumped to /opt/chefproject/delivery-cluster/.chef/local-mode-cache/cache/chef-stacktrace.out
[2016-01-05T20:54:44+00:00] ERROR: machine[mychefserver] (delivery-cluster::setup_chef_server line 31) had an error: SocketError: getaddrinfo: Name or service not known
[2016-01-05T20:54:45+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
rake aborted!
Failed executing ChefZero run
/opt/chefproject/delivery-cluster/Rakefile:109:in chef_zero' /opt/chefproject/delivery-cluster/Rakefile:351:inblock (2 levels) in <top (required)>’
/home/ubuntu/.chefdk/gem/ruby/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:240:in call' /home/ubuntu/.chefdk/gem/ruby/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:240:inblock in execute’
/home/ubuntu/.chefdk/gem/ruby/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:235:in each' /home/ubuntu/.chefdk/gem/ruby/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:235:inexecute’
/home/ubuntu/.chefdk/gem/ruby/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:179:in block in invoke_with_call_chain' /opt/chefdk/embedded/lib/ruby/2.1.0/monitor.rb:211:inmon_synchronize’
/home/ubuntu/.chefdk/gem/ruby/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:172:in invoke_with_call_chain' /home/ubuntu/.chefdk/gem/ruby/2.1.0/gems/rake-10.4.2/lib/rake/task.rb:165:ininvoke’
/home/ubuntu/.chefdk/gem/ruby/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:150:in invoke_task' /home/ubuntu/.chefdk/gem/ruby/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:106:inblock (2 levels) in top_level’
/home/ubuntu/.chefdk/gem/ruby/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:106:in each' /home/ubuntu/.chefdk/gem/ruby/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:106:inblock in top_level’
/home/ubuntu/.chefdk/gem/ruby/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:115:in run_with_threads' /home/ubuntu/.chefdk/gem/ruby/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:100:intop_level’
/home/ubuntu/.chefdk/gem/ruby/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:78:in block in run' /home/ubuntu/.chefdk/gem/ruby/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:176:instandard_exception_handling’
/home/ubuntu/.chefdk/gem/ruby/2.1.0/gems/rake-10.4.2/lib/rake/application.rb:75:in run' /home/ubuntu/.chefdk/gem/ruby/2.1.0/gems/rake-10.4.2/bin/rake:33:in<top (required)>’
/opt/chefdk/embedded/bin/rake:23:in load' /opt/chefdk/embedded/bin/rake:23:in'
Tasks: TOP => setup:cluster

Hello,

I would like you to help me sharing your environment file you are using to configure the cluster. This is a json file that should live under: delivery-cluster/environments

I believe the problem here is that the machine that will be the Chef Server is not able to resolve an specific host from the internet, I have seen this before when chef-provisioning tries to install chef if the server does not have it already.

Does the environment has internet access?

Hi Salim,

Thank you for the help. I am attaching the JSON file.

I can connect to the internet from my provisioning node. I am using the standard VPC in AWS. The provisioning node is part of the VPC.

Thanks,
Srikar

Hi,

I also ran the below command on the provisioning node and got the correct output:

ruby -r aws-sdk-v1 -e ‘puts “Number of instances: #{AWS.ec2.instances.count}”’

The output is:
Number of instances: 15

Thanks,
Srikar

Hi Afiune,

I also opened a support ticket for this issue 3 days back. I uploaded the stracetrace, ENV json and other log files as well.

https://getchef.zendesk.com/hc/en-us/requests/7741

Is it possible for someone to help us with this issue?

Thanks much,

-Srikar

Hello Srikar,

I don’t see any JSON file attached and the link that you sent with the ticket is not working for me. (I am checking internally)

Could you please validate you have attached the file?

Don’t know how to upload a JSON file to the forum. It only support image upload

{
“name”: “CHEFDEMO”,
“description”: “Delivery Cluster Environment”,
“json_class”: “Chef::Environment”,
“chef_type”: “environment”,
“override_attributes”: {
“delivery-cluster”: {
“id”: “CHEFDEMO”,
“driver”: “aws”,
“aws”: {
“key_name”: “LCCASTACK”,
“ssh_username”: “ubuntu”,
“image_id”: “ami-3d50120d”,
“subnet_id”: “subnet-6862d31f”,
“security_group_ids”: “sg-c8d3c1ac”,
“use_private_ip_for_ssh”: true
},
“chef-server”: {
“organization”: “CHEFDEMO”,
“existing”: false,
“flavor”: “c3.xlarge”
},
“delivery”: {
“version”: “latest”,
“enterprise”: “CHEFDEMO”,
“license_file”: “/home/ubuntu/delivery.license”,
“flavor”: “c3.xlarge”
},
“supermarket”: {
“flavor”: “c3.xlarge”
},
“builders”: {
“count”: “3”,
“flavor”: “c3.large”
}
}
}
}

Perfect! This works… I can see it somehow :slightly_smiling:

It looks good, so the next think I want you to do is.

  1. Login to the machine that will be the chef-server.
  2. Show me the /etc/hosts
  3. Run: ping chef.io

And if you can do the same in the provisioning node that would be awesome. :smile:

I don’t have a ‘Chef Server’ yet. I was thinking the delivery-cluster will create the server, delivery, build-nodes.

This is from the provisioning node:

root@ip-10-0-109-38:~# cat /etc/hosts
127.0.0.1 localhost

The following lines are desirable for IPv6 capable hosts

::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
root@ip-10-0-109-38:~# ping chef.io
PING chef.io (52.84.21.187) 56(84) bytes of data.
64 bytes from 52.84.21.187: icmp_seq=1 ttl=56 time=6.11 ms
64 bytes from 52.84.21.187: icmp_seq=2 ttl=56 time=6.16 ms
64 bytes from 52.84.21.187: icmp_seq=3 ttl=56 time=6.13 ms
64 bytes from 52.84.21.187: icmp_seq=4 ttl=56 time=6.11 ms
64 bytes from 52.84.21.187: icmp_seq=5 ttl=56 time=6.22 ms

Could you validate in the AWS console that you do not have a machine called: chef-server-CHEFDEMO?

If it does exist, that is the server I want you to login and show me the things I’ve asked. If not then lets try to run chef-provisioning in debug mode.

To do that, from the provisioning node, go inside the delivery-cluster directory and run:

$ chef exec chef-client -z -o delivery-cluster::setup -E CHEFDEMO -l debug

Hi Afiune,

No. chef-server-CHEFDEMO is not created. I ran the command and have very big output file with log. How can I send that file to you?

My suspicion is: either the program has issues connecting to chefzero:8890 or localhost:8890

— END RESPONSE —
[2016-01-07T21:40:05+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::ValidateContentLength#handle_response
[2016-01-07T21:40:05+00:00] DEBUG: HTTP server did not include a Content-Length header in response, cannot identify truncated downloads.
[2016-01-07T21:40:05+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::RemoteRequestID#handle_response
[2016-01-07T21:40:05+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::Authenticator#handle_response
[2016-01-07T21:40:05+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::Decompressor#handle_response
[2016-01-07T21:40:05+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::CookieManager#handle_response
[2016-01-07T21:40:05+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::JSONToModelOutput#handle_response
[2016-01-07T21:40:05+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::JSONInput#handle_response
[2016-01-07T21:40:05+00:00] INFO: Run List is [recipe[delivery-cluster::setup]]
[2016-01-07T21:40:05+00:00] INFO: Run List expands to [delivery-cluster::setup]
[2016-01-07T21:40:05+00:00] INFO: Starting Chef Run for delivery
[2016-01-07T21:40:05+00:00] INFO: Running start handlers
[2016-01-07T21:40:05+00:00] INFO: Start handlers complete.
[2016-01-07T21:40:05+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::JSONInput#handle_request
[2016-01-07T21:40:05+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::JSONToModelOutput#handle_request
[2016-01-07T21:40:05+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::CookieManager#handle_request
[2016-01-07T21:40:05+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::Decompressor#handle_request
[2016-01-07T21:40:05+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::Authenticator#handle_request
[2016-01-07T21:40:05+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::RemoteRequestID#handle_request
[2016-01-07T21:40:05+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::ValidateContentLength#handle_request
[2016-01-07T21:40:05+00:00] DEBUG: #<ChefZero::RestRequest:0x00000001b6d720 @env={“SCRIPT_NAME”=>"", “SERVER_NAME”=>“localhost”, “REQUEST_METHOD”=>“POST”, “PATH_INFO”=>"/reports/nodes/delivery/runs", “QUERY_STRING”=>nil, “SERVER_PORT”=>8890, “HTTP_HOST”=>“localhost:8890”, “rack.url_scheme”=>“chefzero”, “rack.input”=>#StringIO:0x00000001b6db08}, @rest_base_prefix=[“organizations”, “chef”]>
[2016-01-07T21:40:05+00:00] DEBUG: {“action”:“start”,“run_id”:“a34f9889-34c5-4cf5-86c8-84d7e9a9d911”,“start_time”:“2016-01-07 21:40:05 +0000”}
[2016-01-07T21:40:05+00:00] DEBUG:
— RESPONSE (404) —
{
“error”: [
"Object not found: "
]
}

— END RESPONSE —
[2016-01-07T21:40:05+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::ValidateContentLength#handle_response
[2016-01-07T21:40:05+00:00] DEBUG: HTTP server did not include a Content-Length header in response, cannot identify truncated downloads.
[2016-01-07T21:40:05+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::RemoteRequestID#handle_response
[2016-01-07T21:40:05+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::Authenticator#handle_response
[2016-01-07T21:40:05+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::Decompressor#handle_response
[2016-01-07T21:40:05+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::CookieManager#handle_response
[2016-01-07T21:40:05+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::JSONToModelOutput#handle_response

And then I see this error further down:

/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application.rb:60:in run' /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/bin/chef-client:26:in<top (required)>’
/opt/chefdk/bin/chef-client:54:in load' /opt/chefdk/bin/chef-client:54:in'
[2016-01-07T21:40:12+00:00] DEBUG:
— RESPONSE (404) —
{
“error”: [
“Object not found: chefzero://localhost:8890/nodes/chef-server-CHEFDEMO”
]
}

— END RESPONSE —
[2016-01-07T21:40:12+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::RemoteRequestID#handle_response
[2016-01-07T21:40:12+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::Authenticator#handle_response
[2016-01-07T21:40:12+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::Decompressor#handle_response
[2016-01-07T21:40:12+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::CookieManager#handle_response
[2016-01-07T21:40:12+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::JSONOutput#handle_response
[2016-01-07T21:40:12+00:00] DEBUG: Chef::HTTP calling Chef::HTTP::JSONInput#handle_response
[2016-01-07T21:40:12+00:00] INFO: HTTP Request Returned 404 Not Found: Object not found: chefzero://localhost:8890/nodes/chef-server-CHEFDEMO
[2016-01-07T21:40:16+00:00] INFO: [AWS EC2 200 2.255944 3 retries] describe_security_groups(:filters=>[{:name=>“group-id”,:values=>[“sg-c8d3c1ac”]}]) SocketError getaddrinfo: Name or service not known

================================================================================
Error executing action `converge` on resource 'machine[chef-server-CHEFDEMO]'
================================================================================

SocketError
-----------
getaddrinfo: Name or service not known

Resource Declaration:
---------------------
# In /opt/chefproject/delivery-cluster/.chef/local-mode-cache/cache/cookbooks/delivery-cluster/recipes/setup_chef_server.rb

 31: machine chef_server_hostname do
 32:   provisioning.specific_machine_options('chef-server').each do |option|
 33:     add_machine_options(option)
 34:   end
 35:   # Transfer any trusted certs
 36:   Dir.glob("#{Chef::Config[:trusted_certs_dir]}/*").each do |cert_path|
 37:     file ::File.join('/etc/chef/trusted_certs', ::File.basename(cert_path)), cert_path
 38:   end
 39:   action :converge
 40: end
 41: 

Thanks,
Srikar

Hi Afiune,

I uploaded the file to S3. Please download it from here:

https://s3-us-west-2.amazonaws.com/srikar/CHEFDEMO/output.txt

Thanks,
Srikar
408 386 5174