Problem with Chef after update

Chef Infra (archive)
1

We recently updated chef and now bookshelf fail to start.

2014-03-03_17:44:39.71399 Usage: bookshelf {start|stop|restart|reboot|ping|console|console_clean|attach}
[root@chef bookshelf]# chef-server-ctl status
down: bookshelf: 0s, normally up, want up; run: log: (pid 1642) 19934080s
run: chef-expander: (pid 28588) 1389s; run: log: (pid 1644) 19934080s
run: chef-server-webui: (pid 28593) 1388s; run: log: (pid 1645) 19934080s
run: chef-solr: (pid 28693) 1387s; run: log: (pid 1643) 19934080s
run: erchef: (pid 7429) 8s; run: log: (pid 1640) 19934080s
run: nginx: (pid 28930) 1385s; run: log: (pid 1646) 19934080s
run: postgresql: (pid 29157) 1383s; run: log: (pid 1647) 19934080s
run: rabbitmq: (pid 8078) 1s; run: log: (pid 1641) 19934080s
[root@chef status]# rpm -qa | grep chef

chef-server-11.0.11-1.el6.x86_64

Any ideas? googlefu isn’t picking anything up on it :frowning:

Thanks!

2

Also, seeing this on chef client runs

[root@messages1 ~]# cat /var/chef/cache/chef-stacktrace.out
Generated at 2014-03-03 12:52:30 -0500
Net::HTTPFatalError: 500 "Internal Server Error"
/opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:2633:in error!' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/rest.rb:207:in block in raw_http_request'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/rest.rb:289:in retriable_rest_request' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/rest.rb:167:in raw_http_request'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/rest.rb:161:in api_request' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/rest.rb:128:in put'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/node.rb:489:in save' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:261:in save_updated_node'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:496:in do_run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:199:in block in run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:193:in fork' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:193:in run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application.rb:183:in run_chef_client' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:302:in block in run_application'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:294:in loop' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:294:in run_application'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application.rb:66:in run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/bin/chef-client:26:in <top (required)>'
/usr/bin/chef-client:23:in load' /usr/bin/chef-client:23:in
On Mar 3, 2014, at 12:46 PM, Jenn Fountain jfountain@meetme.com wrote:

We recently updated chef and now bookshelf fail to start.

2014-03-03_17:44:39.71399 Usage: bookshelf {start|stop|restart|reboot|ping|console|console_clean|attach}
[root@chef bookshelf]# chef-server-ctl status
down: bookshelf: 0s, normally up, want up; run: log: (pid 1642) 19934080s
run: chef-expander: (pid 28588) 1389s; run: log: (pid 1644) 19934080s
run: chef-server-webui: (pid 28593) 1388s; run: log: (pid 1645) 19934080s
run: chef-solr: (pid 28693) 1387s; run: log: (pid 1643) 19934080s
run: erchef: (pid 7429) 8s; run: log: (pid 1640) 19934080s
run: nginx: (pid 28930) 1385s; run: log: (pid 1646) 19934080s
run: postgresql: (pid 29157) 1383s; run: log: (pid 1647) 19934080s
run: rabbitmq: (pid 8078) 1s; run: log: (pid 1641) 19934080s
[root@chef status]# rpm -qa | grep chef

chef-server-11.0.11-1.el6.x86_64

Any ideas? googlefu isn’t picking anything up on it :frowning:

Thanks!

3

Hi Jenn,

Is there anything useful in the bookshelf logs? Logs should be located in
the /var/log/chef-server/bookshelf directory.

Also, which version did you upgrade from?

Thanks,
Stephen

On Mon, Mar 3, 2014 at 9:55 AM, Jenn Fountain jfountain@meetme.com wrote:

Also, seeing this on chef client runs

[root@messages1 ~]# cat /var/chef/cache/chef-stacktrace.out
Generated at 2014-03-03 12:52:30 -0500
Net::HTTPFatalError: 500 "Internal Server Error"
/opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:2633:in error!' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/rest.rb:207:in block in raw_http_request'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/rest.rb:289:in
retriable_rest_request' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/rest.rb:167:in raw_http_request'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/rest.rb:161:in
api_request' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/rest.rb:128:in put'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/node.rb:489:in
save' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:261:in save_updated_node'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:496:in
do_run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:199:in block in run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:193:in
fork' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:193:in run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application.rb:183:in
run_chef_client' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:302:in block in run_application'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:294:in
loop' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:294:in run_application'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application.rb:66:in
run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/bin/chef-client:26:in <top (required)>'
/usr/bin/chef-client:23:in load' /usr/bin/chef-client:23:in
On Mar 3, 2014, at 12:46 PM, Jenn Fountain jfountain@meetme.com wrote:

We recently updated chef and now bookshelf fail to start.
2014-03-03_17:44:39.71399 Usage: bookshelf
{start|stop|restart|reboot|ping|console|console_clean|attach}
[root@chef bookshelf]# chef-server-ctl status
down: bookshelf: 0s, normally up, want up; run: log: (pid 1642) 19934080s
run: chef-expander: (pid 28588) 1389s; run: log: (pid 1644) 19934080s
run: chef-server-webui: (pid 28593) 1388s; run: log: (pid 1645) 19934080s
run: chef-solr: (pid 28693) 1387s; run: log: (pid 1643) 19934080s
run: erchef: (pid 7429) 8s; run: log: (pid 1640) 19934080s
run: nginx: (pid 28930) 1385s; run: log: (pid 1646) 19934080s
run: postgresql: (pid 29157) 1383s; run: log: (pid 1647) 19934080s
run: rabbitmq: (pid 8078) 1s; run: log: (pid 1641) 19934080s

[root@chef status]# rpm -qa | grep chef

chef-server-11.0.11-1.el6.x86_64

Any ideas? googlefu isn't picking anything up on it :frowning:

Thanks!

--
Stephen Delano
Software Development Engineer
Opscode, Inc.
1008 Western Avenue
Suite 601
Seattle, WA 98104

4

We finally got it up and running after running the chef-configure-ctl several times. Thanks for the quick response though!

On Mar 3, 2014, at 4:33 PM, Stephen Delano stephen@opscode.com wrote:

Hi Jenn,

Is there anything useful in the bookshelf logs? Logs should be located in the /var/log/chef-server/bookshelf directory.

Also, which version did you upgrade from?

Thanks,
Stephen

On Mon, Mar 3, 2014 at 9:55 AM, Jenn Fountain jfountain@meetme.com wrote:
Also, seeing this on chef client runs

[root@messages1 ~]# cat /var/chef/cache/chef-stacktrace.out
Generated at 2014-03-03 12:52:30 -0500
Net::HTTPFatalError: 500 "Internal Server Error"
/opt/chef/embedded/lib/ruby/1.9.1/net/http.rb:2633:in error!' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/rest.rb:207:in block in raw_http_request'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/rest.rb:289:in retriable_rest_request' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/rest.rb:167:in raw_http_request'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/rest.rb:161:in api_request' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/rest.rb:128:in put'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/node.rb:489:in save' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:261:in save_updated_node'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:496:in do_run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:199:in block in run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:193:in fork' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/client.rb:193:in run'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application.rb:183:in run_chef_client' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:302:in block in run_application'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:294:in loop' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application/client.rb:294:in run_application'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/lib/chef/application.rb:66:in run' /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/chef-11.6.0/bin/chef-client:26:in <top (required)>'
/usr/bin/chef-client:23:in load' /usr/bin/chef-client:23:in
On Mar 3, 2014, at 12:46 PM, Jenn Fountain jfountain@meetme.com wrote:

We recently updated chef and now bookshelf fail to start.

2014-03-03_17:44:39.71399 Usage: bookshelf {start|stop|restart|reboot|ping|console|console_clean|attach}
[root@chef bookshelf]# chef-server-ctl status
down: bookshelf: 0s, normally up, want up; run: log: (pid 1642) 19934080s
run: chef-expander: (pid 28588) 1389s; run: log: (pid 1644) 19934080s
run: chef-server-webui: (pid 28593) 1388s; run: log: (pid 1645) 19934080s
run: chef-solr: (pid 28693) 1387s; run: log: (pid 1643) 19934080s
run: erchef: (pid 7429) 8s; run: log: (pid 1640) 19934080s
run: nginx: (pid 28930) 1385s; run: log: (pid 1646) 19934080s
run: postgresql: (pid 29157) 1383s; run: log: (pid 1647) 19934080s
run: rabbitmq: (pid 8078) 1s; run: log: (pid 1641) 19934080s
[root@chef status]# rpm -qa | grep chef

chef-server-11.0.11-1.el6.x86_64

Any ideas? googlefu isn’t picking anything up on it :frowning:

Thanks!

--
Stephen Delano
Software Development Engineer
Opscode, Inc.
1008 Western Avenue
Suite 601
Seattle, WA 98104