ERROR: No socketless chef-zero server


#1

I’m getting the following using chef-client 12.3 with chef-zero 4.2.1:

ERROR: No socketless chef-zero server on given port 8889

I have a centos VM running under Vagrant, on a private network. I then run
chef-zero on the same private network:

chef-zero -H 10.0.5.1

And use knife to bootstrap. The run fails when trying to create the client
with the following stacktrace:

ChefZero::ServerNotFound: No socketless chef-zero server on given port 8889
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-zero-4.2.1/lib/chef_zero/socketless_server_map.rb:78:in
request' /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-zero-4.2.1/lib/chef_zero/socketless_server_map.rb:33:inrequest’
/opt/chef/embedded/apps/chef/lib/chef/http/socketless_chef_zero_client.rb:153:in
`request’

I tried running chef-client with both --no-listen and --listen, but got the
same error in both cases.

Any ideas?

Thanks,
Christine Draper


#2

Can you paste your knife bootstrap command? and also your knife.rb? Some
obvious things to check which you probably might have already verified…

Did you try running the chef-zero run with log_level debug?

Are you using any other name for the host other than localhost?

Your /etc/hosts does have localhost bound to 127.0.0.1

This is unlikely, but is it possible you attempted too many runs of your
chef-zero run?

On Sat, May 2, 2015 at 5:41 PM, Christine Draper <
christine_draper@thirdwaveinsights.com> wrote:

I’m getting the following using chef-client 12.3 with chef-zero 4.2.1:

ERROR: No socketless chef-zero server on given port 8889

I have a centos VM running under Vagrant, on a private network. I then run
chef-zero on the same private network:

chef-zero -H 10.0.5.1

And use knife to bootstrap. The run fails when trying to create the client
with the following stacktrace:

ChefZero::ServerNotFound: No socketless chef-zero server on given port 8889
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-zero-4.2.1/lib/chef_zero/socketless_server_map.rb:78:in
request' /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-zero-4.2.1/lib/chef_zero/socketless_server_map.rb:33:inrequest’
/opt/chef/embedded/apps/chef/lib/chef/http/socketless_chef_zero_client.rb:153:in
`request’

I tried running chef-client with both --no-listen and --listen, but got
the same error in both cases.

Any ideas?

Thanks,
Christine Draper


#3

On Saturday, May 2, 2015 at 2:41 PM, Christine Draper wrote:

I’m getting the following using chef-client 12.3 with chef-zero 4.2.1:

ERROR: No socketless chef-zero server on given port 8889
First, a bit of background on the socketless Chef Zero stuff. In any version of Chef with local mode, when you run chef-client -z or knife -z, it runs some code that asks Chef Zero to start, and then asks Chef Zero what URL to use to connect to it. It then sets Chef::Config[:chef_server_url] to the given URL. Before Chef 12.3, this URL would be http://localhost:8889 (usually). In Chef 12.3, socketless mode was introduced. Chef Zero still runs in a background thread and listens for connections on localhost:8889 by default (unless you use the —no-listen option), but internally chef/knife will ask Chef Zero for a “socktless mode URL”, which will be something like “chefzero://localhost:8889”. When Chef/knife makes an HTTP request, it checks the URL scheme, and if it’s “chefzero” then it follows a different code path to make the “request” just by calling ruby code and not actually using HTTP.

I have a centos VM running under Vagrant, on a private network. I then run chef-zero on the same private network:

chef-zero -H 10.0.5.1

And use knife to bootstrap. The run fails when trying to create the client with the following stacktrace:

ChefZero::ServerNotFound: No socketless chef-zero server on given port 8889
What are the exact arguments you’re using with knife bootstrap ? Based on the behavior you’re seeing, it does seem like you’re running knife bootstrap with local mode enabled. But then I am confused about how this would have worked before, since the chef_server_url for knife would have to have been http://localhost:8889. But that doesn’t make sense because then how would chef-client on the bootstrap machine talk to the other chef-zero you’re spinning up?

/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-zero-4.2.1/lib/chef_zero/socketless_server_map.rb:78:in request' /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-zero-4.2.1/lib/chef_zero/socketless_server_map.rb:33:inrequest’
/opt/chef/embedded/apps/chef/lib/chef/http/socketless_chef_zero_client.rb:153:in `request’

I tried running chef-client with both --no-listen and --listen, but got the same error in both cases.

Any ideas?

Thanks,
Christine Draper

Since the socketless operation is a new feature, it’s definitely possible there’s a bug here, but I’ll need to know more about your configuration to see how things are supposed to be working.

Thanks,


Daniel DeLeo


#4

Daniel,

I don’t think there’s a bug. Just user error. I finally figured it out.

The way I typically run is I spin up one or more VMs on a private network
in VirtualBox with IPs like 10.0.5.x, and then I run chef-zero on that
private network e.g. using chef-zero -H10.0.5.1

I then run knife bootstrap 10.0.5.x --sudo -xuser -Ppassword

So I am NOT using local mode and do need sockets.

What happened was I accidentally put a server URL of https://10.0.5.1:8899
rather than http://10.0.5.1:8899

I then got into SSL validation problems, which of course I couldnt solve
using ‘knife ssh fetch’ because chef-zero doesn’t do certificates. I
wrongly latched onto ‘chef-zero.enabled true’ as the answer, not realizing
that it overrode my server URL and ran in local mode. Which of course meant
the bootstrap didnt work (or at least, the chef client run as part of the
bootstrap didnt). As an aside: is there a way to do bootstrap from knife in
local mode?

Anyway,when I fixed the server URL to “http://10.0.5.1:8899” and removed
’chef-zero.enabled true’, its all fine again.

Regards,
Christine

On Mon, May 4, 2015 at 12:36 PM, Daniel DeLeo dan@kallistec.com wrote:

On Saturday, May 2, 2015 at 2:41 PM, Christine Draper wrote:

I’m getting the following using chef-client 12.3 with chef-zero 4.2.1:

ERROR: No socketless chef-zero server on given port 8889
First, a bit of background on the socketless Chef Zero stuff. In any
version of Chef with local mode, when you run chef-client -z or knife -z, it runs some code that asks Chef Zero to start, and then asks Chef
Zero what URL to use to connect to it. It then sets
Chef::Config[:chef_server_url] to the given URL. Before Chef 12.3, this
URL would be http://localhost:8889 (usually). In Chef 12.3, socketless
mode was introduced. Chef Zero still runs in a background thread and
listens for connections on localhost:8889 by default (unless you use the
—no-listen option), but internally chef/knife will ask Chef Zero for a
“socktless mode URL”, which will be something like
“chefzero://localhost:8889”. When Chef/knife makes an HTTP request, it
checks the URL scheme, and if it’s “chefzero” then it follows a different
code path to make the “request” just by calling ruby code and not actually
using HTTP.

I have a centos VM running under Vagrant, on a private network. I then
run chef-zero on the same private network:

chef-zero -H 10.0.5.1

And use knife to bootstrap. The run fails when trying to create the
client with the following stacktrace:

ChefZero::ServerNotFound: No socketless chef-zero server on given port
8889
What are the exact arguments you’re using with knife bootstrap ? Based
on the behavior you’re seeing, it does seem like you’re running knife bootstrap with local mode enabled. But then I am confused about how this
would have worked before, since the chef_server_url for knife would have to
have been http://localhost:8889. But that doesn’t make sense because then
how would chef-client on the bootstrap machine talk to the other chef-zero
you’re spinning up?

/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-zero-4.2.1/lib/chef_zero/socketless_server_map.rb:78:in
`request’

/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-zero-4.2.1/lib/chef_zero/socketless_server_map.rb:33:in
`request’

/opt/chef/embedded/apps/chef/lib/chef/http/socketless_chef_zero_client.rb:153:in
`request’

I tried running chef-client with both --no-listen and --listen, but got
the same error in both cases.

Any ideas?

Thanks,
Christine Draper

Since the socketless operation is a new feature, it’s definitely possible
there’s a bug here, but I’ll need to know more about your configuration to
see how things are supposed to be working.

Thanks,


Daniel DeLeo


#5

On Monday, May 4, 2015 at 12:41 PM, Christine Draper wrote:

Daniel,

I don’t think there’s a bug. Just user error. I finally figured it out.
Glad to hear.
As an aside: is there a way to do bootstrap from knife in local mode?

Chef Provisioning can create machines from local mode but knife does not have this ability.

Anyway,when I fixed the server URL to “http://10.0.5.1:8899” and removed ‘chef-zero.enabled true’, its all fine again.

Regards,
Christine


Daniel DeLeo


#6

On 7 May 2015, at 11:13 am, Daniel DeLeo dan@kallistec.com wrote:

I don’t think there’s a bug. Just user error. I finally figured it out.
Glad to hear.
As an aside: is there a way to do bootstrap from knife in local mode?
Chef Provisioning can create machines from local mode but knife does not have this ability.

I’m having the same issue - but I’m using chef-provisioning for bootstrap. It only fails when the chef zero socketless mode is tried to be used by the client on each node.

- [cheffrontendTEST1.int] update node cheffrontendTEST1.int at chefzero://localhost:8889
- [cheffrontendTEST1.int]   add normal.aws_ha_chef = {:aws_access_key_id=>”x", :aws_secret_access_key=>”y", :backend1=>{:fqdn=>”chefbackendTEST1.int", :ip_address=>"172.21.1.85"}, :backend2=>{:fqdn=>”chefbackendTEST2.int", :ip_address=>"172.21.1.86"}, :frontends=>{:fe1=>{:fqdn=>”cheffrontendTEST1.int", :ip_address=>"172.21.1.88"}, :fe2=>{:fqdn=>”cheffrontendTEST2.int", :ip_address=>"172.21.1.111"}}}
- [cheffrontendTEST1.int]   remove normal.aws_ha_chef[2015-05-18T17:04:41+10:00] INFO: Converging cheffrontendTEST1.int because the resource was updated ...

[2015-05-18T17:04:41+10:00] INFO: Port forwarded: local URL chefzero://localhost:8889 is available to 172.21.1.88 as chefzero://localhost:8889 for the duration of this SSH connection.
[2015-05-18T17:04:41+10:00] INFO: Executing sudo chef-client -l info on ubuntu@172.21.1.88
[cheffrontendTEST1.int] [2015-05-18T07:04:41+00:00] INFO: Forking chef instance to converge…
Starting Chef Client, version 12.3.0
[2015-05-18T07:04:41+00:00] INFO: *** Chef 12.3.0 ***
[2015-05-18T07:04:41+00:00] INFO: Chef-client pid: 1505

                             ================================================================================
                             Chef encountered an error attempting to load the node data for “cheffrontendTEST1.int"
                             ================================================================================
                             
                             Unexpected Error:
                             -----------------
                             ChefZero::ServerNotFound: No socketless chef-zero server on given port 8889
                             
                             [2015-05-18T07:04:42+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
                             Chef Client failed. 0 resources updated in 0.859637226 seconds
                             [2015-05-18T07:04:42+00:00] ERROR: No socketless chef-zero server on given port 8889
                             [2015-05-18T07:04:42+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

[2015-05-18T17:04:42+10:00] INFO: Completed chef-client -l info on ubuntu@172.21.1.88: exit status 1

Using chefDK 0.6.0 and client 12.3 as well.

$ chef gem list |grep chef
chef (12.3.0)
chef-dk (0.6.0)
chef-provisioning (1.1.1)
chef-provisioning-aws (1.1.1)
chef-provisioning-azure (0.3.2)
chef-provisioning-fog (0.13.2)
chef-provisioning-vagrant (0.8.3)
chef-vault (2.4.0)
chef-zero (4.2.1, 1.5.6)
cheffish (1.2)
chefspec (4.2.0)

#knife.rb
log_level :info
log_location STDOUT
node_name 'bootstrap’
client_key '/Users/ted/chef/.chef/bootstrap.pem’
validation_client_name ‘bootstrap’
#chef_server_url 'http://localhost:8889
validation_key '/Users/ted/chef/.chef/keys/bootstrap.pem’
syntax_check_cache_path '/Users/ted/chef/.chef/syntax_check_cache’
cookbook_path [ ‘~/chef/cookbooks’, ‘~/chef/berks-cookbooks’ ]
knife[:aws_access_key_id] = “x"
knife[:aws_secret_access_key] = “y"
knife[:region] = "ap-southeast-2”

It’s been driving me crazy all day.
Before ChefDK 0.6.0 and Client 12.3 I used to instantiate a local chef-zero server and it would bootstrap fine.
I’ve even tried the —no-listen option; but no luck.

Not sure if it’s a bug in chef-provisioing-aws, but I might raise an issue over with them too.

(I do have a forward slash in my secret access key - which I’ve read somewhere can be an issue, but chef-provisioning is bootstrapping OK; just client runs failing)


Ted
GPG Key: 0x2B272DFD | E564 BCC1 F601 5D1F 01CD AC10 C37D D2B0 2B27 2DFD


#7

On Monday, May 18, 2015 at 12:17 AM, Ted B wrote:

On 7 May 2015, at 11:13 am, Daniel DeLeo <dan@kallistec.com (mailto:dan@kallistec.com)> wrote:

I don’t think there’s a bug. Just user error. I finally figured it out.

Glad to hear.

As an aside: is there a way to do bootstrap from knife in local mode?

Chef Provisioning can create machines from local mode but knife does not have this ability.

I’m having the same issue - but I’m using chef-provisioning for bootstrap. It only fails when the chef zero socketless mode is tried to be used by the client on each node.

  • [cheffrontendTEST1.int] update node cheffrontendTEST1.int at chefzero://localhost:8889
  • [cheffrontendTEST1.int] add normal.aws_ha_chef = {:aws_access_key_id=>”x", :aws_secret_access_key=>”y", :backend1=>{:fqdn=>”chefbackendTEST1.int", :ip_address=>“172.21.1.85”}, :backend2=>{:fqdn=>”chefbackendTEST2.int", :ip_address=>“172.21.1.86”}, :frontends=>{:fe1=>{:fqdn=>”cheffrontendTEST1.int", :ip_address=>“172.21.1.88”}, :fe2=>{:fqdn=>”cheffrontendTEST2.int", :ip_address=>“172.21.1.111”}}}
  • [cheffrontendTEST1.int] remove normal.aws_ha_chef[2015-05-18T17:04:41+10:00] INFO: Converging cheffrontendTEST1.int because the resource was updated …
    [2015-05-18T17:04:41+10:00] INFO: Port forwarded: local URL chefzero://localhost:8889 is available to 172.21.1.88 as chefzero://localhost:8889 for the duration of this SSH connection.
    [2015-05-18T17:04:41+10:00] INFO: Executing sudo chef-client -l info on ubuntu@172.21.1.88 (mailto:ubuntu@172.21.1.88)
    [cheffrontendTEST1.int] [2015-05-18T07:04:41+00:00] INFO: Forking chef instance to converge…
    Starting Chef Client, version 12.3.0
    [2015-05-18T07:04:41+00:00] INFO: *** Chef 12.3.0 ***
    [2015-05-18T07:04:41+00:00] INFO: Chef-client pid: 1505

================================================================================
Chef encountered an error attempting to load the node data for “cheffrontendTEST1.int"

Unexpected Error:

ChefZero::ServerNotFound: No socketless chef-zero server on given port 8889

[2015-05-18T07:04:42+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
Chef Client failed. 0 resources updated in 0.859637226 seconds
[2015-05-18T07:04:42+00:00] ERROR: No socketless chef-zero server on given port 8889
[2015-05-18T07:04:42+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
[2015-05-18T17:04:42+10:00] INFO: Completed chef-client -l info on ubuntu@172.21.1.88 (mailto:ubuntu@172.21.1.88): exit status 1

Using chefDK 0.6.0 and client 12.3 as well.
This is an incompatibility between Chef 12.3 and Chef Provisioning 1.1.1. A fix has been committed for this and will be released in the next version of Chef Provisioning. We were hoping to release this with ChefDK 0.6, but unfortunately there are some interactions between Chef, knife-windows, Chef Vault, and Chef Provisioning that made the latest versions of all of these incompatible with each other. We’ll be implementing some changes to fix that also.


Daniel DeLeo


#8

On 19 May 2015, at 04:24, Daniel DeLeo dan@kallistec.com wrote:

On Monday, May 18, 2015 at 12:17 AM, Ted B wrote:
I’m having the same issue - but I’m using chef-provisioning for bootstrap. It only fails when the chef zero socketless mode is tried to be used by the client on each node.

Using chefDK 0.6.0 and client 12.3 as well.
This is an incompatibility between Chef 12.3 and Chef Provisioning 1.1.1. A fix has been committed for this and will be released in the next version of Chef Provisioning.

Is this the issue tracking this?


#9

This is the one https://github.com/chef/chef-provisioning/pull/337


Daniel DeLeo

On Monday, May 18, 2015 at 2:17 PM, Ted wrote:

On 19 May 2015, at 04:24, Daniel DeLeo <dan@kallistec.com (mailto:dan@kallistec.com)> wrote:

On Monday, May 18, 2015 at 12:17 AM, Ted B wrote:
I’m having the same issue - but I’m using chef-provisioning for bootstrap. It only fails when the chef zero socketless mode is tried to be used by the client on each node.

Using chefDK 0.6.0 and client 12.3 as well.
This is an incompatibility between Chef 12.3 and Chef Provisioning 1.1.1. A fix has been committed for this and will be released in the next version of Chef Provisioning.

Is this the issue tracking this?
https://github.com/chef/chef-provisioning/issues/330


#10

On 19 May 2015, at 7:32 am, Daniel DeLeo dan@kallistec.com wrote:

This is the one https://github.com/chef/chef-provisioning/pull/337

Ok - I closed the other issue on chef-provisioing-aws.

Confirmed it working with a gem build of chef-provisioning-1.2.0.dev.0.gem

[2015-05-19T11:35:03+10:00] INFO: Port forwarded: local URL chefzero://localhost:8889 is available to 172.21.1.88 as chefzero://localhost:8889 for the duration of this SSH connection.


Ted
GPG Key: 0x2B272DFD | E564 BCC1 F601 5D1F 01CD AC10 C37D D2B0 2B27 2DFD


#11

fwiw, specifying the local url seems to work as a workaround for the
chef-provisioning issue:

machine “blah” do
chef_server( :chef_server_url => ‘http://localhost:8889’)
run_list [‘blah’]
end

On Mon, May 18, 2015 at 8:38 PM, Ted B ted@pobox.com wrote:

On 19 May 2015, at 7:32 am, Daniel DeLeo dan@kallistec.com wrote:

This is the one https://github.com/chef/chef-provisioning/pull/337

Ok - I closed the other issue on chef-provisioing-aws.

Confirmed it working with a gem build of chef-provisioning-1.2.0.dev.0.gem

[2015-05-19T11:35:03+10:00] INFO: Port forwarded: local URL
chefzero://localhost:8889 is available to 172.21.1.88 as
chefzero://localhost:8889 for the duration of this SSH connection.


Ted
GPG Key: 0x2B272DFD | E564 BCC1 F601 5D1F 01CD AC10 C37D D2B0 2B27 2DFD


#12

Hi,

I have found another case where I get this error, not involving
chef-provisioning.

All I’m doing is using local mode to bootstrap a Vagrant VM:

knife bootstrap -z 127.0.0.1:2222 --sudo -xuser -Ppassword

This creates a client.rb on the target with:

chef_server_url “chefzero://localhost:8889”

and returns error:

ERROR: No socketless chef-zero server on given port 8889

when it tries to start the run. I’m using chefdk 0.6/chef 12.3.0

Seems such a basic case that I must be doing something wrong - any ideas?

Regards,
Christine

On Tue, May 19, 2015 at 4:25 PM, Christine Draper <
christine_draper@thirdwaveinsights.com> wrote:

fwiw, specifying the local url seems to work as a workaround for the
chef-provisioning issue:

machine “blah” do
chef_server( :chef_server_url => ‘http://localhost:8889’)
run_list [‘blah’]
end

On Mon, May 18, 2015 at 8:38 PM, Ted B ted@pobox.com wrote:

On 19 May 2015, at 7:32 am, Daniel DeLeo dan@kallistec.com wrote:

This is the one https://github.com/chef/chef-provisioning/pull/337

Ok - I closed the other issue on chef-provisioing-aws.

Confirmed it working with a gem build of chef-provisioning-1.2.0.dev.0.gem

[2015-05-19T11:35:03+10:00] INFO: Port forwarded: local URL
chefzero://localhost:8889 is available to 172.21.1.88 as
chefzero://localhost:8889 for the duration of this SSH connection.


Ted
GPG Key: 0x2B272DFD | E564 BCC1 F601 5D1F 01CD AC10 C37D D2B0 2B27 2DFD