Chef v11 fails on Centos 6.4

Chef Infra (archive)
1

I set up a RHEL 64bit VM and got the Chef v11 rpm and installed it as root
but there were some errors.

These are two of the lines that stand out.

Error executing action run on resource 'execute[verify-system-status]'
Mixlib::ShellOut::ShellCommandFailed

The test shows a missing file.

[root@chefserver ~]# chef-server-ctl test
/opt/chef-server/embedded/service/chef-pedant/lib/pedant/config.rb:34:in
from_argv': Configuration file '/var/opt/chef-server/chef-pedant/etc/pedant_config.rb' not found! (RuntimeError) from /opt/chef-server/embedded/service/chef-pedant/lib/pedant.rb:44:insetup’
from ./bin/chef-pedant:26:in `’

I used the installation instructions from here:
http://docs.opscode.com/chef/install_server.html
And found this BUG reference
http://tickets.opscode.com/browse/CHEF-4096

Is there a fix for this?
I created an output file of the installation process for inspection if
needed.
thanks,

-lun*
*

2

On Thursday, May 2, 2013 at 4:26 PM, Lunixer wrote:

I set up a RHEL 64bit VM and got the Chef v11 rpm and installed it as root but there were some errors.

These are two of the lines that stand out.

Error executing action run on resource 'execute[verify-system-status]'
Mixlib::ShellOut::ShellCommandFailed

The test shows a missing file.

[root@chefserver ~]# chef-server-ctl test
/opt/chef-server/embedded/service/chef-pedant/lib/pedant/config.rb:34:in from_argv': Configuration file '/var/opt/chef-server/chef-pedant/etc/pedant_config.rb' not found! (RuntimeError) from /opt/chef-server/embedded/service/chef-pedant/lib/pedant.rb:44:in setup'
from ./bin/chef-pedant:26:in `'

I used the installation instructions from here:
http://docs.opscode.com/chef/install_server.html
And found this BUG reference
http://tickets.opscode.com/browse/CHEF-4096

Is there a fix for this?
I created an output file of the installation process for inspection if needed.
thanks,

-lun
That bug seems to indicate that you ran chef-server-ctl reconfigure as non-root?

--
Daniel DeLeo

3

Did everything as root.
I read the bug and they make it clear to use sudo. So, I used root instead.
At first I thought it was a Centos thing, but the bug is against RHEL 6.3.

My VM is Centos 6.4, all updates. Using the recommended ruby version.

BTW folks, what's the convention to post/reply on this list... Reply at the
top or bottom of the message for easy reading?

-lun

On Thursday, May 2, 2013, Daniel DeLeo wrote:

On Thursday, May 2, 2013 at 4:26 PM, Lunixer wrote:

I set up a RHEL 64bit VM and got the Chef v11 rpm and installed it as root
but there were some errors.

These are two of the lines that stand out.

Error executing action run on resource 'execute[verify-system-status]'
Mixlib::ShellOut::ShellCommandFailed

The test shows a missing file.

[root@chefserver ~]# chef-server-ctl test
/opt/chef-server/embedded/service/chef-pedant/lib/pedant/config.rb:34:in
from_argv': Configuration file '/var/opt/chef-server/chef-pedant/etc/pedant_config.rb' not found! (RuntimeError) from /opt/chef-server/embedded/service/chef-pedant/lib/pedant.rb:44:in setup'
from ./bin/chef-pedant:26:in `'

I used the installation instructions from here:
http://docs.opscode.com/chef/install_server.html
And found this BUG reference
http://tickets.opscode.com/browse/CHEF-4096

Is there a fix for this?
I created an output file of the installation process for inspection if
needed.
thanks,

-lun*
*

That bug seems to indicate that you ran chef-server-ctl reconfigure as
non-root?

--
Daniel DeLeo

4

On 5/2/13 4:26 PM, Lunixer wrote:

I set up a RHEL 64bit VM and got the Chef v11 rpm and installed it as
root but there were some errors.

These are two of the lines that stand out.

Error executing action run on resource 'execute[verify-system-status]'
Mixlib::ShellOut::ShellCommandFailed

The test shows a missing file.

[root@chefserver ~]# chef-server-ctl test
/opt/chef-server/embedded/service/chef-pedant/lib/pedant/config.rb:34:in
from_argv': Configuration file '/var/opt/chef-server/chef-pedant/etc/pedant_config.rb' not found! (RuntimeError) from /opt/chef-server/embedded/service/chef-pedant/lib/pedant.rb:44:in setup'
from ./bin/chef-pedant:26:in `'

This second error is likely simply a consequence of the first since that
file is created after the verify-system-status portion of the reconfigure.

Was there any other output during the chef-server-ctl reconfigure?

Sincerely,

Steven

--
Steven Danna
Systems Engineer, Opscode, Inc
GPG Key: http://stevendanna.github.com/downloads/code/public.key

5

Below is the trimmed installation output.
Basically the problem occurs at the end of the process.
The status command at the end of this email shows several PIDs running.

[....snip...]

  • ruby_block[supervise_erchef_sleep] action run

    • execute the ruby block supervise_erchef_sleep

  • service[erchef] action nothing (up to date)

  • execute[/opt/chef-server/bin/chef-server-ctl start erchef] action run

    • execute /opt/chef-server/bin/chef-server-ctl start erchef

Recipe: chef-server::bootstrap

  • execute[verify-system-status] action run
    ================================================================================
    Error executing action run on resource 'execute[verify-system-status]'
    ================================================================================

Mixlib::ShellOut::ShellCommandFailed

Expected process to exit with [0], but received '22'
---- Begin output of curl -sf http://127.0.0.1:8000/_status ----
STDOUT:
STDERR:
---- End output of curl -sf http://127.0.0.1:8000/_status ----
Ran curl -sf http://127.0.0.1:8000/_status returned 22

Resource Declaration:

In /opt/chef-server/embedded/cookbooks/chef-server/recipes/bootstrap.rb

24: execute "verify-system-status" do
25: command "curl -sf #{erchef_status_url}"
26: retries 20
27: not_if { File.exists?(bootstrap_status_file) }
28: end
29:

Compiled Resource:

Declared in

/opt/chef-server/embedded/cookbooks/chef-server/recipes/bootstrap.rb:24:in
`from_file'

execute("verify-system-status") do
action "run"
retries 0
retry_delay 2
command "curl -sf http://127.0.0.1:8000/_status"
backup 5
returns 0
cookbook_name :"chef-server"
recipe_name "bootstrap"
not_if { #code block }
end

[2013-05-02T15:16:55-07:00] ERROR: Running exception handlers
[2013-05-02T15:16:55-07:00] ERROR: Exception handlers complete
Chef Client failed. 123 resources updated
[2013-05-02T15:16:55-07:00] FATAL: Stacktrace dumped to
/opt/chef-server/embedded/cookbooks/cache/chef-stacktrace.out
[2013-05-02T15:16:55-07:00] FATAL: Mixlib::ShellOut::ShellCommandFailed:
execute[verify-system-status] (chef-server::bootstrap line 24) had an
error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with
[0], but received '22'
---- Begin output of curl -sf http://127.0.0.1:8000/_status ----
STDOUT:
STDERR:
---- End output of curl -sf http://127.0.0.1:8000/_status ----
Ran curl -sf http://127.0.0.1:8000/_status returned 22

0;root@chefserver:~[root@chefserver ~]# chef-server-ctl status
run: bookshelf: (pid 23190) 740s; run: log: (pid 23189) 740s
run: chef-expander: (pid 23092) 746s; run: log: (pid 23091) 746s
down: chef-solr: 0s, normally up, want up; run: log: (pid 23058) 747s
run: erchef: (pid 23283) 734s; run: log: (pid 23282) 734s
run: postgresql: (pid 22969) 754s; run: log: (pid 22968) 754s
run: rabbitmq: (pid 22689) 765s; run: log: (pid 22688) 765s

-lun

On Thu, May 2, 2013 at 9:22 PM, Steven Danna steve@opscode.com wrote:

On 5/2/13 4:26 PM, Lunixer wrote:

I set up a RHEL 64bit VM and got the Chef v11 rpm and installed it as
root but there were some errors.

These are two of the lines that stand out.

Error executing action run on resource 'execute[verify-system-status]'
Mixlib::ShellOut::ShellCommandFailed

The test shows a missing file.

[root@chefserver ~]# chef-server-ctl test
/opt/chef-server/embedded/service/chef-pedant/lib/pedant/config.rb:34:in
from_argv': Configuration file '/var/opt/chef-server/chef-pedant/etc/pedant_config.rb' not found! (RuntimeError) from /opt/chef-server/embedded/service/chef-pedant/lib/pedant.rb:44:in setup'
from ./bin/chef-pedant:26:in `'

This second error is likely simply a consequence of the first since that
file is created after the verify-system-status portion of the reconfigure.

Was there any other output during the chef-server-ctl reconfigure?

Sincerely,

Steven

--
Steven Danna
Systems Engineer, Opscode, Inc
GPG Key: http://stevendanna.github.com/downloads/code/public.key

6

On 5/3/13 10:15 AM, Lunixer wrote:

0;root@chefserver:~[root@chefserver ~]# chef-server-ctl status
run: bookshelf: (pid 23190) 740s; run: log: (pid 23189) 740s
run: chef-expander: (pid 23092) 746s; run: log: (pid 23091) 746s
down: chef-solr: 0s, normally up, want up; run: log: (pid 23058) 747s
run: erchef: (pid 23283) 734s; run: log: (pid 23282) 734s
run: postgresql: (pid 22969) 754s; run: log: (pid 22968) 754s
run: rabbitmq: (pid 22689) 765s; run: log: (pid 22688) 765s

It appears that solr is down. Can you try starting solr?

chef-server-ctl start chef-solr

If it doesn't start see if there is anything in the logs at

/var/log/chef-server/chef-solr/current

Cheers,

Steven

--
Steven Danna
Systems Engineer, Opscode, Inc
GPG Key: http://stevendanna.github.com/downloads/code/public.key

7

The chef-solr service starts but status still shows down.

[root@chefserver ~]# chef-server-ctl stop chef-solr
ok: down: chef-solr: 1s, normally up, want up

[root@chefserver ~]# chef-server-ctl start chef-solr
ok: run: chef-solr: (pid 26072) 0s

[root@chefserver ~]# chef-server-ctl status
run: bookshelf: (pid 1148) 2427s; run: log: (pid 1142) 2427s
run: chef-expander: (pid 1150) 2427s; run: log: (pid 1144) 2427s
down: chef-solr: 1s, normally up, want up; run: log: (pid 1143) 2427s
run: erchef: (pid 1145) 2427s; run: log: (pid 1139) 2427s
run: postgresql: (pid 13826) 1193s; run: log: (pid 1141) 2427s
run: rabbitmq: (pid 1146) 2427s; run: log: (pid 1140) 2427s

[root@chefserver ~]# ps -ef | grep solr
root 1134 1115 0 17:00 ? 00:00:01 runsv chef-solr
root 1143 1134 0 17:00 ? 00:00:00 svlogd -tt
/var/log/chef-server/chef-solr

[root@chefserver ~]# chef-server-ctl service-list
bookshelf*
chef-expander*
chef-solr*
erchef*
postgresql*
rabbitmq*

-lun

On Sat, May 4, 2013 at 9:05 PM, Steven Danna steve@opscode.com wrote:

On 5/3/13 10:15 AM, Lunixer wrote:

0;root@chefserver:~[root@chefserver ~]# chef-server-ctl status
run: bookshelf: (pid 23190) 740s; run: log: (pid 23189) 740s
run: chef-expander: (pid 23092) 746s; run: log: (pid 23091) 746s
down: chef-solr: 0s, normally up, want up; run: log: (pid 23058) 747s
run: erchef: (pid 23283) 734s; run: log: (pid 23282) 734s
run: postgresql: (pid 22969) 754s; run: log: (pid 22968) 754s
run: rabbitmq: (pid 22689) 765s; run: log: (pid 22688) 765s

It appears that solr is down. Can you try starting solr?

chef-server-ctl start chef-solr

If it doesn't start see if there is anything in the logs at

/var/log/chef-server/chef-solr/current

Cheers,

Steven

--
Steven Danna
Systems Engineer, Opscode, Inc
GPG Key: http://stevendanna.github.com/downloads/code/public.key

8

On Monday, May 6, 2013 at 5:41 PM, Lunixer wrote:

The chef-solr service starts but status still shows down.

[root@chefserver ~]# chef-server-ctl stop chef-solr
ok: down: chef-solr: 1s, normally up, want up

[root@chefserver ~]# chef-server-ctl start chef-solr
ok: run: chef-solr: (pid 26072) 0s

[root@chefserver ~]# chef-server-ctl status
run: bookshelf: (pid 1148) 2427s; run: log: (pid 1142) 2427s
run: chef-expander: (pid 1150) 2427s; run: log: (pid 1144) 2427s
down: chef-solr: 1s, normally up, want up; run: log: (pid 1143) 2427s
run: erchef: (pid 1145) 2427s; run: log: (pid 1139) 2427s
run: postgresql: (pid 13826) 1193s; run: log: (pid 1141) 2427s
run: rabbitmq: (pid 1146) 2427s; run: log: (pid 1140) 2427s

[root@chefserver ~]# ps -ef | grep solr
root 1134 1115 0 17:00 ? 00:00:01 runsv chef-solr
root 1143 1134 0 17:00 ? 00:00:00 svlogd -tt /var/log/chef-server/chef-solr

[root@chefserver ~]# chef-server-ctl service-list
bookshelf*
chef-expander*
chef-solr*
erchef*
postgresql*
rabbitmq*

-lun

solr may be in a crash loop for some reason. Out of the blue guess is that the java min memory setting is too high for your machine. Try 'chef-server-ctl tail' or just look at /var/log/chef-server/chef-solr

--
Daniel DeLeo

9

Problem fixed.

THANKS for your time and help! Much appreciated.

'chef-server-ctl tail' gave me the simple and obvious clue to solve this.
It boiled down to name resolution.

My server is a VM running on a DHCP lease. Thus, the indication below that
the host is not know. I edited the /etc/hosts file and added the hostname
and ip as issued by the server. Then ran the reconfigure once again, solr
started and I got a WebUI.
BUT, the part indicating the error on the initial post persisted.

This is what the log file said:

==> /var/log/chef-server/chef-solr/current <==
2013-05-07_03:06:02.85529 Error: Exception thrown by the agent :
java.net.MalformedURLException: Local host name unknown:
java.net.UnknownHostException: chefserver: chefserver: Name or service not
known

A test showed all is well.

chef-server-ctl test

Configuring logging...
Creating platform...
Starting Pedant Run: 2013-05-07 03:33:03 UTC
setting up rspec config for #Pedant::OpenSourcePlatform:0x0000000291ed38
Configuring RSpec for Open-Source Tests

| || || || || || | | |
| _ || _ || || || _ || _ || |
| | | || || || | | || | | || | | || |_
| || || ||__ || || || || || || |
| || | _| || | | || || |
|||| |||||||| |_______|

 _______  _______  ______   _______  __    _  _______
|       ||       ||      | |   _   ||  |  | ||       |
|    _  ||    ___||  _    ||  |_|  ||   |_| ||_     _|
|   |_| ||   |___ | | |   ||       ||       |  |   |
|    ___||    ___|| |_|   ||       ||  _    |  |   |
|   |    |   |___ |       ||   _   || | |   |  |   |
|___|    |_______||______| |__| |__||_|  |__|  |___|

                "Accuracy Over Tact"

              === Testing Environment ===
             Config File:

/var/opt/chef-server/chef-pedant/etc/pedant_config.rb
HTTP Traffic Log File:
/var/log/chef-server/chef-pedant/http-traffic.log
[...big snip...]

Would you know whether the hostname is kept somewhere in the chef server
configuration? I'd like to be able to keep using this config and only
change the hostname. Are there certificates involved. I'm new to chef, so
any hints are most helpful.

Thanks again.

-lun

On Mon, May 6, 2013 at 5:49 PM, Daniel DeLeo dan@kallistec.com wrote:

On Monday, May 6, 2013 at 5:41 PM, Lunixer wrote:

The chef-solr service starts but status still shows down.

[root@chefserver ~]# chef-server-ctl stop chef-solr
ok: down: chef-solr: 1s, normally up, want up

[root@chefserver ~]# chef-server-ctl start chef-solr
ok: run: chef-solr: (pid 26072) 0s

[root@chefserver ~]# chef-server-ctl status
run: bookshelf: (pid 1148) 2427s; run: log: (pid 1142) 2427s
run: chef-expander: (pid 1150) 2427s; run: log: (pid 1144) 2427s
down: chef-solr: 1s, normally up, want up; run: log: (pid 1143) 2427s
run: erchef: (pid 1145) 2427s; run: log: (pid 1139) 2427s
run: postgresql: (pid 13826) 1193s; run: log: (pid 1141) 2427s
run: rabbitmq: (pid 1146) 2427s; run: log: (pid 1140) 2427s

[root@chefserver ~]# ps -ef | grep solr
root 1134 1115 0 17:00 ? 00:00:01 runsv chef-solr
root 1143 1134 0 17:00 ? 00:00:00 svlogd -tt
/var/log/chef-server/chef-solr

[root@chefserver ~]# chef-server-ctl service-list
bookshelf*
chef-expander*
chef-solr*
erchef*
postgresql*
rabbitmq*

-lun

solr may be in a crash loop for some reason. Out of the blue guess is that
the java min memory setting is too high for your machine. Try
'chef-server-ctl tail' or just look at /var/log/chef-server/chef-solr

--
Daniel DeLeo

10

On Monday, May 6, 2013 at 8:45 PM, Lunixer wrote:

Would you know whether the hostname is kept somewhere in the chef server configuration? I'd like to be able to keep using this config and only change the hostname. Are there certificates involved. I'm new to chef, so any hints are most helpful.

Thanks again.

-lun

There is a master config file that is used to generate config files for the individual components:

http://docs.opscode.com/config_rb_chef_server.html

The way to go is to make your changes in the chef_server.rb file and then use chef-server-ctl reconfigure to render the changes and restart components.

--
Daniel DeLeo