Chef Server Webui fails after VM restart

Chef Infra (archive)
1

After restarting/rebooting the Ubuntu Server 12.04 (x86_64) VM with the chef
server 11.0.11-1
(https://opscode-omnibus-packages.s3.amazonaws.com/ubuntu/12.04/x86_64/chef-server_11.0.11-1.ubuntu.12.04_amd64.deb)
installed, the chef-server-webui cannot start successfully anymore. All chef
related services (i.e. erchef, ngnix, …) worked as expected and the
chef-server-webui was accessible via web browser before restarting the VM.
However after rebooting the VirtualBox (4.3.10) VM (in the VHD format) the
chef-server-webui was not accessible anymore and the log (sudo chef-server-ctl
tail chef-server-webui) repeatedly shows the following error message:

################################################################## BEGIN sudo
chef-server-ctl tail chef-server-webui
##################################################################
2014-04-01_17:33:33.37368
/opt/chef-server/embedded/service/gem/ruby/1.9.1/gems/tilt-1.4.1/lib/tilt.rb:169:in
<module:Tilt>': uninitialized constant Tilt::MarkabyTemplate (NameError) 2014-04-01_17:33:33.37396 from /opt/chef-server/embedded/service/gem/ruby/1.9.1/gems/tilt-1.4.1/lib/tilt.rb:1:in<top (required)>'
2014-04-01_17:33:33.37432 from
/opt/chef-server/embedded/service/gem/ruby/1.9.1/gems/sprockets-2.2.2/lib/sprockets/eco_template.rb:1:in
<top (required)>' 2014-04-01_17:33:33.37451 from /opt/chef-server/embedded/service/gem/ruby/1.9.1/gems/sprockets-2.2.2/lib/sprockets/engines.rb:1:in<top (required)>'
2014-04-01_17:33:33.37463 from
/opt/chef-server/embedded/service/gem/ruby/1.9.1/gems/sprockets-2.2.2/lib/sprockets.rb:42:in
<module:Sprockets>' 2014-04-01_17:33:33.37488 from /opt/chef-server/embedded/service/gem/ruby/1.9.1/gems/sprockets-2.2.2/lib/sprockets.rb:3:in<top (required)>'
2014-04-01_17:33:33.37506 from
/opt/chef-server/embedded/service/gem/ruby/1.9.1/gems/actionpack-3.2.15/lib/sprockets/railtie.rb:21:in
block in <class:Railtie>' 2014-04-01_17:33:33.37521 from /opt/chef-server/embedded/service/gem/ruby/1.9.1/gems/railties-3.2.15/lib/rails/initializable.rb:30:ininstance_exec’
2014-04-01_17:33:33.37533 from
/opt/chef-server/embedded/service/gem/ruby/1.9.1/gems/railties-3.2.15/lib/rails/initializable.rb:30:in
run' 2014-04-01_17:33:33.37547 from /opt/chef-server/embedded/service/gem/ruby/1.9.1/gems/railties-3.2.15/lib/rails/initializable.rb:55:inblock in run_initializers’
2014-04-01_17:33:33.37561 from
/opt/chef-server/embedded/service/gem/ruby/1.9.1/gems/railties-3.2.15/lib/rails/initializable.rb:54:in
each' 2014-04-01_17:33:33.37575 from /opt/chef-server/embedded/service/gem/ruby/1.9.1/gems/railties-3.2.15/lib/rails/initializable.rb:54:inrun_initializers’
2014-04-01_17:33:33.37589 from
/opt/chef-server/embedded/service/gem/ruby/1.9.1/gems/railties-3.2.15/lib/rails/application.rb:136:in
initialize!' 2014-04-01_17:33:33.37606 from /opt/chef-server/embedded/service/gem/ruby/1.9.1/gems/railties-3.2.15/lib/rails/railtie/configurable.rb:30:inmethod_missing’
2014-04-01_17:33:33.37623 from
/opt/chef-server/embedded/service/chef-server-webui/config/environment.rb:5:in
<top (required)>' 2014-04-01_17:33:33.37636 from /opt/chef-server/embedded/service/chef-server-webui/config.ru:3:inrequire’
2014-04-01_17:33:33.37654 from
/opt/chef-server/embedded/service/chef-server-webui/config.ru:3:in `block in

' 2014-04-01_17:33:33.37667 from /opt/chef-server/embedded/service/gem/ruby/1.9.1/gems/rack-1.4.5/lib/rack/builder.rb:51:in `instance_eval' 2014-04-01_17:33:33.37686 from /opt/chef-server/embedded/service/gem/ruby/1.9.1/gems/rack-1.4.5/lib/rack/builder.rb:51:in `initialize' 2014-04-01_17:33:33.37697 from /opt/chef-server/embedded/service/chef-server-webui/config.ru:1:in `new' 2014-04-01_17:33:33.37711 from /opt/chef-server/embedded/service/chef-server-webui/config.ru:1:in `' 2014-04-01_17:33:33.37727 from /opt/chef-server/embedded/service/gem/ruby/1.9.1/gems/unicorn-2.0.1/lib/unicorn.rb:30:in `eval' 2014-04-01_17:33:33.37741 from /opt/chef-server/embedded/service/gem/ruby/1.9.1/gems/unicorn-2.0.1/lib/unicorn.rb:30:in `block in builder' 2014-04-01_17:33:33.37757 from /opt/chef-server/embedded/service/gem/ruby/1.9.1/gems/unicorn-2.0.1/lib/unicorn/http_server.rb:675:in `call' 2014-04-01_17:33:33.37770 from /opt/chef-server/embedded/service/gem/ruby/1.9.1/gems/unicorn-2.0.1/lib/unicorn/http_server.rb:675:in `build_app!' 2014-04-01_17:33:33.37787 from /opt/chef-server/embedded/service/gem/ruby/1.9.1/gems/unicorn-2.0.1/lib/unicorn/http_server.rb:542:in `init_worker_process' 2014-04-01_17:33:33.37805 from /opt/chef-server/embedded/service/gem/ruby/1.9.1/gems/unicorn-2.0.1/lib/unicorn/http_server.rb:557:in `worker_loop' 2014-04-01_17:33:33.37817 from /opt/chef-server/embedded/service/gem/ruby/1.9.1/gems/unicorn-2.0.1/lib/unicorn/http_server.rb:473:in `block (2 levels) in spawn_missing_workers' 2014-04-01_17:33:33.37829 from /opt/chef-server/embedded/service/gem/ruby/1.9.1/gems/unicorn-2.0.1/lib/unicorn/http_server.rb:470:in `fork' 2014-04-01_17:33:33.37844 from /opt/chef-server/embedded/service/gem/ruby/1.9.1/gems/unicorn-2.0.1/lib/unicorn/http_server.rb:470:in `block in spawn_missing_workers' 2014-04-01_17:33:33.37857 from /opt/chef-server/embedded/service/gem/ruby/1.9.1/gems/unicorn-2.0.1/lib/unicorn/http_server.rb:466:in `each' 2014-04-01_17:33:33.37871 from /opt/chef-server/embedded/service/gem/ruby/1.9.1/gems/unicorn-2.0.1/lib/unicorn/http_server.rb:466:in `spawn_missing_workers' 2014-04-01_17:33:33.37884 from /opt/chef-server/embedded/service/gem/ruby/1.9.1/gems/unicorn-2.0.1/lib/unicorn/http_server.rb:480:in `maintain_worker_count' 2014-04-01_17:33:33.37896 from /opt/chef-server/embedded/service/gem/ruby/1.9.1/gems/unicorn-2.0.1/lib/unicorn/http_server.rb:299:in `join' 2014-04-01_17:33:33.37910 from /opt/chef-server/embedded/service/gem/ruby/1.9.1/gems/unicorn-2.0.1/lib/unicorn.rb:13:in `run' 2014-04-01_17:33:33.37925 from /opt/chef-server/embedded/service/gem/ruby/1.9.1/gems/unicorn-2.0.1/bin/unicorn:120:in `' 2014-04-01_17:33:33.37929 from /opt/chef-server/embedded/service/gem/ruby/1.9.1/bin/unicorn:23:in `load' 2014-04-01_17:33:33.37931 from /opt/chef-server/embedded/service/gem/ruby/1.9.1/bin/unicorn:23:in `' 2014-04-01_17:33:33.40220 E, [2014-04-01T19:33:33.402119 #16997] ERROR -- : reaped # worker=0 2014-04-01_17:33:33.40271 I, [2014-04-01T19:33:33.402679 #16997] INFO -- : worker=0 spawning... 2014-04-01_17:33:33.40380 I, [2014-04-01T19:33:33.403694 #20585] INFO -- : worker=0 spawned pid=20585 2014-04-01_17:33:33.40410 I, [2014-04-01T19:33:33.404061 #20585] INFO -- : Refreshing Gem list ################################################################## END sudo chef-server-ctl tail chef-server-webui ####################################################################

Note that the other chef related services work as expected (e.g. “knife node
list” successfully lists all nodes managed by chef). The problem occurs
reproducible after the Chef server VM (fresh Chef installation) has been
restarted.

The following approaches to solve the problem have been tried but were not
successful:

  • Stop/Start/Restart all chef services or just the chef-server-webui (Before
    restarting the VM it was possible )
  • Adjust the number of unicorn worker threads via
    /etc/chef-server/chef-server.rb
  • Revert the only custom setting applied to chef-server.rb: nginx[‘ssl_port’]
    = 8443 (since VirtualBox cannot forward port 443 on Mac OS X hosts).
  • sudo chef-server-ctl reconfigure

What do you suggest to solve this problem? I gladly provide further information
if required.

Thanks in advance,
Joe