# Moved node to new chef server, no longer converges

**URL:** <https://discourse.chef.io/t/moved-node-to-new-chef-server-no-longer-converges/5526>\
**Category:** Chef Infra (archive)\
**Created:** [July 16, 2014, 6:53pm UTC](https://discourse.chef.io/t/moved-node-to-new-chef-server-no-longer-converges/5526 "2014-07-16T18:53:50Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mark\_Nichols](https://avatars.discourse-cdn.com/v4/letter/m/eb8c5e/32.png) [@Mark\_Nichols](https://discourse.chef.io/u/Mark_Nichols)\
**Post date:** [July 16, 2014, 6:53pm UTC](https://discourse.chef.io/t/moved-node-to-new-chef-server-no-longer-converges/5526/1 "2014-07-16T18:53:50Z")

</div>

Ohai Chefs,

We are in the process of moving several nodes from older hosted Chef to Enterprise Chef servers. I’ve moved several alpha (test) nodes but they no longer converge. Here are steps I followed.

1. Noted the IP address of the node
2. Deleted the node from the old chef server
3. Via ssh visited the node and clean out the `/etc/chef` directory
4. Ran `knife bootstrap`
5. Visited the node and confirmed via `ps -ef | grep chef-client` that chef-client was running with the proper interval and splay.

After waiting the 15 minutes for the interval the node fails to converge with these messages in `/var/log/chef/client.log`

[2014-07-16T13:38:50-05:00] INFO: \*\*\* Chef 11.12.2 \*\*\*  
[2014-07-16T13:38:50-05:00] INFO: Chef-client pid: 7550  
[2014-07-16T13:38:50-05:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out  
[2014-07-16T13:38:50-05:00] ERROR: Unable to determine node name: configure node\_name or configure the system’s hostname and fqdn  
[2014-07-16T13:38:50-05:00] ERROR: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)  
[2014-07-16T13:38:50-05:00] ERROR: Sleeping for 900 seconds before trying again

And these messages in `/var/chef/cache/chef-stacktrace.out`

Generated at 2014-07-16 13:38:50 -0500  
Chef::Exceptions::CannotDetermineNodeName: Unable to determine node name: configure node\_name or configure the system’s hostname and fqdn  
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.2/lib/chef/client.rb:299:in `node_name' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.2/lib/chef/client.rb:313:in`register’  
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.2/lib/chef/client.rb:416:in `do_run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.2/lib/chef/client.rb:213:in`block in run’  
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.2/lib/chef/client.rb:207:in `fork' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.2/lib/chef/client.rb:207:in`run’  
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.2/lib/chef/application.rb:217:in `run_chef_client' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.2/lib/chef/application/client.rb:328:in`block in run\_application’  
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.2/lib/chef/application/client.rb:317:in `loop' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.2/lib/chef/application/client.rb:317:in`run\_application’  
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.2/lib/chef/application.rb:67:in `run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.2/bin/chef-client:26:in`\<top (required)\>’  
/usr/bin/chef-client:23:in `load' /usr/bin/chef-client:23:in`'  
/var/chef/cache/chef-stacktrace.out (END)

I’ve checked `hostname -f` and it produces the FQDN. Running chef-client manually on the node succeeds, it’s only the background process that’s failing. For what it’s worth, here is the output from `ps -ef | grep chef-client`

root 2662 1 0 Jul11 ? 00:00:13 /opt/chef/embedded/bin/ruby /usr/bin/chef-client -d -c /etc/chef/client.rb -L /var/log/chef/client.log -P /var/run/chef/client.pid -i 900 -s 180

I’m stumped.

What am I not seeing?

Thanks,  
Mark

---

<div class="post-metadata">

**Author:** ![dcondomitti](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/dcondomitti/32/104_2.png) [@dcondomitti](https://discourse.chef.io/u/dcondomitti)\
**Post date:** [July 16, 2014, 7:12pm UTC](https://discourse.chef.io/t/moved-node-to-new-chef-server-no-longer-converges/5526/2 "2014-07-16T19:12:06Z")

</div>

Can you increase the log level to debug in client.rb? That should show the failing ohai output. You can also try running ohai manually as well.

On Wednesday, July 16, 2014 at 11:53 AM, Mark Nichols wrote:

> Ohai Chefs,
> 
> We are in the process of moving several nodes from older hosted Chef to Enterprise Chef servers. I’ve moved several alpha (test) nodes but they no longer converge. Here are steps I followed.
> 
> 1. Noted the IP address of the node
> 2. Deleted the node from the old chef server
> 3. Via ssh visited the node and clean out the `/etc/chef` directory
> 4. Ran `knife bootstrap`
> 5. Visited the node and confirmed via `ps -ef | grep chef-client` that chef-client was running with the proper interval and splay.
> 
> After waiting the 15 minutes for the interval the node fails to converge with these messages in `/var/log/chef/client.log`
> 
> > [2014-07-16T13:38:50-05:00] INFO: \*\*\* Chef 11.12.2 \*\*\*  
> > [2014-07-16T13:38:50-05:00] INFO: Chef-client pid: 7550  
> > [2014-07-16T13:38:50-05:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out  
> > [2014-07-16T13:38:50-05:00] ERROR: Unable to determine node name: configure node\_name or configure the system's hostname and fqdn  
> > [2014-07-16T13:38:50-05:00] ERROR: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)  
> > [2014-07-16T13:38:50-05:00] ERROR: Sleeping for 900 seconds before trying again
> 
> And these messages in `/var/chef/cache/chef-stacktrace.out`
> 
> > Generated at 2014-07-16 13:38:50 -0500  
> > Chef::Exceptions::CannotDetermineNodeName: Unable to determine node name: configure node\_name or configure the system's hostname and fqdn  
> > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.2/lib/chef/client.rb:299:in `node_name' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.2/lib/chef/client.rb:313:in `register'  
> > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.2/lib/chef/client.rb:416:in `do_run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.2/lib/chef/client.rb:213:in `block in run'  
> > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.2/lib/chef/client.rb:207:in `fork' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.2/lib/chef/client.rb:207:in `run'  
> > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.2/lib/chef/application.rb:217:in `run_chef_client' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.2/lib/chef/application/client.rb:328:in `block in run\_application'  
> > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.2/lib/chef/application/client.rb:317:in `loop' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.2/lib/chef/application/client.rb:317:in `run\_application'  
> > /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.2/lib/chef/application.rb:67:in `run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.2/bin/chef-client:26:in `\<top (required)\>'  
> > /usr/bin/chef-client:23:in `load' /usr/bin/chef-client:23:in `'  
> > /var/chef/cache/chef-stacktrace.out (END)
> 
> I’ve checked `hostname -f` and it produces the FQDN. Running chef-client manually on the node succeeds, it’s only the background process that’s failing. For what it’s worth, here is the output from `ps -ef | grep chef-client`
> 
> > root 2662 1 0 Jul11 ? 00:00:13 /opt/chef/embedded/bin/ruby /usr/bin/chef-client -d -c /etc/chef/client.rb -L /var/log/chef/client.log -P /var/run/chef/client.pid -i 900 -s 180
> 
> I’m stumped.
> 
> What am I not seeing?
> 
> Thanks,  
> Mark

---

<div class="post-metadata">

**Author:** ![DV1](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/dv1/32/186_2.png) [@DV1](https://discourse.chef.io/u/DV1)\
**Post date:** [July 17, 2014, 12:21am UTC](https://discourse.chef.io/t/moved-node-to-new-chef-server-no-longer-converges/5526/3 "2014-07-17T00:21:11Z")

</div>

Very strange that manual chef-client run succeeds but chef-client daemon  
does not.

This is silly, but did you kill the old chef-client process before  
bootstrapping? Also, have you tried rebooting the host?

On Wed, Jul 16, 2014 at 11:53 AM, Mark Nichols [chef@zanshin.net](mailto:chef@zanshin.net) wrote:

> Ohai Chefs,
> 
> We are in the process of moving several nodes from older hosted Chef to  
> Enterprise Chef servers. I’ve moved several alpha (test) nodes but they no  
> longer converge. Here are steps I followed.
> 
> 1. Noted the IP address of the node
> 2. Deleted the node from the old chef server
> 3. Via ssh visited the node and clean out the `/etc/chef` directory
> 4. Ran `knife bootstrap`
> 5. Visited the node and confirmed via `ps -ef | grep chef-client` that  
> chef-client was running with the proper interval and splay.
> 
> After waiting the 15 minutes for the interval the node fails to converge  
> with these messages in `/var/log/chef/client.log`
> 
> [2014-07-16T13:38:50-05:00] INFO: \*\*\* Chef 11.12.2 \*\*\*  
> [2014-07-16T13:38:50-05:00] INFO: Chef-client pid: 7550  
> [2014-07-16T13:38:50-05:00] FATAL: Stacktrace dumped to  
> /var/chef/cache/chef-stacktrace.out  
> [2014-07-16T13:38:50-05:00] ERROR: Unable to determine node name:  
> configure node\_name or configure the system's hostname and fqdn  
> [2014-07-16T13:38:50-05:00] ERROR: Chef::Exceptions::ChildConvergeError:  
> Chef run process exited unsuccessfully (exit code 1)  
> [2014-07-16T13:38:50-05:00] ERROR: Sleeping for 900 seconds before trying  
> again
> 
> And these messages in `/var/chef/cache/chef-stacktrace.out`
> 
> Generated at 2014-07-16 13:38:50 -0500  
> Chef::Exceptions::CannotDetermineNodeName: Unable to determine node name:  
> configure node\_name or configure the system's hostname and fqdn  
> /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.2/lib/chef/client.rb:299:in  
> `node_name' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.2/lib/chef/client.rb:313:in `register'  
> /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.2/lib/chef/client.rb:416:in  
> `do_run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.2/lib/chef/client.rb:213:in `block in run'  
> /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.2/lib/chef/client.rb:207:in  
> `fork' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.2/lib/chef/client.rb:207:in `run'  
> /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.2/lib/chef/application.rb:217:in  
> `run_chef_client' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.2/lib/chef/application/client.rb:328:in `block in run\_application'  
> /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.2/lib/chef/application/client.rb:317:in  
> `loop' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.2/lib/chef/application/client.rb:317:in `run\_application'  
> /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.2/lib/chef/application.rb:67:in  
> `run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.12.2/bin/chef-client:26:in `\<top (required)\>'  
> /usr/bin/chef-client:23:in `load' /usr/bin/chef-client:23:in `'  
> /var/chef/cache/chef-stacktrace.out (END)
> 
> I’ve checked `hostname -f` and it produces the FQDN. Running chef-client  
> manually on the node succeeds, it’s only the background process that’s  
> failing. For what it’s worth, here is the output from `ps -ef | grep chef-client`
> 
> root 2662 1 0 Jul11 ? 00:00:13  
> /opt/chef/embedded/bin/ruby /usr/bin/chef-client -d -c /etc/chef/client.rb  
> -L /var/log/chef/client.log -P /var/run/chef/client.pid -i 900 -s 180
> 
> I’m stumped.
> 
> What am I not seeing?
> 
> Thanks,  
> Mark

--  
Best regards, Dmitriy V.

---

<div class="post-metadata">

**Author:** ![Mark\_Nichols](https://avatars.discourse-cdn.com/v4/letter/m/eb8c5e/32.png) [@Mark\_Nichols](https://discourse.chef.io/u/Mark_Nichols)\
**Post date:** [July 17, 2014, 2:48am UTC](https://discourse.chef.io/t/moved-node-to-new-chef-server-no-longer-converges/5526/4 "2014-07-17T02:48:08Z")

</div>

On Jul 16, 2014, at 7:21 PM, DV [vindimy@gmail.com](mailto:vindimy@gmail.com) wrote:

> Very strange that manual chef-client run succeeds but chef-client daemon does not.
> 
> This is silly, but did you kill the old chef-client process before bootstrapping? Also, have you tried rebooting the host?

I did kill the old chef-client process prior to the new bootstrap command. I haven’t tried rebooting the host. That would be, ah, complicated. These are legacy (i.e., pre-chef) servers that aren’t easily rebooted.

I’ll explore more with tcpdump and debug turned on for the chef client.

— Mark