Unable to create first user due to error 500

Chef Infra (archive)
1

Hi all, using Chef 12.14.0 on RHEL7.3

the reconfigure command seems to work successfully but attempts to create a user all fail no matter how I reinstall, cleanup, or restart server.

I always get the same error as below:

# chef-server-ctl user-create admin admin admin 'username@example.com' 'password'
ERROR: Server returned error 500 for https://127.0.0.1/users/, retrying 1/5 in 4s
ERROR: Server returned error 500 for https://127.0.0.1/users/, retrying 2/5 in 6s
ERROR: Server returned error 500 for https://127.0.0.1/users/, retrying 3/5 in 14s
ERROR: Server returned error 500 for https://127.0.0.1/users/, retrying 4/5 in 18s
ERROR: Server returned error 500 for https://127.0.0.1/users/, retrying 5/5 in 59s
ERROR: internal server error 
Response: internal service error
#

Here is the configuration file:

[root@servername ~]# cat /etc/opscode/chef-server.rb
bootstrap['enable']=false
nginx['ssl_certificate']="/etc/pki/tls/private/servername.pem"
nginx['ssl_certificate_key']="/etc/pki/tls/private/servername.key"
nginx['ssl_protocols'] = "TLSv1.1 TLSv1.2"
nginx['ssl_ciphers'] = "HIGH:MEDIUM:!LOW:!kEDH:!aNULL:!ADH:!eNULL:!EXP:!SSLv2:!SEED:!CAMELLIA:!PSK:!RC4"
default_orgname="Myorg"
postgresql['username']="opscode-pgsql"
oc_chef_pedant['debug_org_creation']=true

Here is the result of opscode-erchef log

# chef-server-ctl tail opscode-erchef
==> /var/log/opscode/opscode-erchef/current <==
2017-07-06_10:31:37.95054 [info] Can't connect to RabbitMQ management console to fetch max-length
2017-07-06_10:32:07.95011 [info] Can't connect to RabbitMQ management console to fetch max-length
2017-07-06_10:32:37.95027 [info] Can't connect to RabbitMQ management console to fetch max-length
2017-07-06_10:33:07.94938 [info] Can't connect to RabbitMQ management console to fetch max-length
2017-07-06_10:33:37.94959 [info] Can't connect to RabbitMQ management console to fetch max-length
2017-07-06_10:34:07.95006 [info] Can't connect to RabbitMQ management console to fetch max-length
2017-07-06_10:34:37.95039 [info] Can't connect to RabbitMQ management console to fetch max-length
2017-07-06_10:35:05.02913 [error] {<<"method=POST; path=/users/; status=500; ">>,"Internal Server Error"}
2017-07-06_10:35:07.94944 [info] Can't connect to RabbitMQ management console to fetch max-length
2017-07-06_10:35:09.05598 [error] {<<"method=POST; path=/users/; status=500; ">>,"Internal Server Error"}

==> /var/log/opscode/opscode-erchef/requests.log.siz <==
@

==> /var/log/opscode/opscode-erchef/requests.log.1 <==
...
2017-07-06T10:35:05Z erchef@127.0.0.1 method=POST; path=/users/; status=500; req_id=g3IAA2QAEGVyY2hlZkAxMjcuMC4wLjECAAAdfAAAAAIAAAAA; msg={error_finding_user_or_client,{syntax,{<<"relation \"keys_by_name\" does not exi"...>>,[{position,<<"87">>}]}}}; couchdb_groups=false; couchdb_organizations=false; couchdb_containers=false; couchdb_acls=false; 503_mode=false; couchdb_associations=false; couchdb_association_requests=false; req_time=2; rdbms_time=0; rdbms_count=1; user=pivotal; req_api_version=0;
2017-07-06T10:35:09Z erchef@127.0.0.1 method=POST; path=/users/; status=500; req_id=g3IAA2QAEGVyY2hlZkAxMjcuMC4wLjECAAAdrgAAAAIAAAAA; msg={error_finding_user_or_client,{syntax,{<<"relation \"keys_by_name\" does not exi"...>>,[{position,<<"87">>}]}}}; couchdb_groups=false; couchdb_organizations=false; couchdb_containers=false; couchdb_acls=false; 503_mode=false; couchdb_associations=false; couchdb_association_requests=false; req_time=2; rdbms_time=0; rdbms_count=1; user=pivotal; req_api_version=0;

==> /var/log/opscode/opscode-erchef/crash.log <==
2017-07-06 19:19:44 =ERROR REPORT====
{<<"method=POST; path=/users/; status=500; ">>,"Internal Server Error"}
2017-07-06 19:20:02 =ERROR REPORT====
{<<"method=POST; path=/users/; status=500; ">>,"Internal Server Error"}
2017-07-06 19:21:01 =ERROR REPORT====
{<<"method=POST; path=/users/; status=500; ">>,"Internal Server Error"}
2017-07-06 19:35:05 =ERROR REPORT====
{<<"method=POST; path=/users/; status=500; ">>,"Internal Server Error"}
2017-07-06 19:35:09 =ERROR REPORT====
{<<"method=POST; path=/users/; status=500; ">>,"Internal Server Error"}

==> /var/log/opscode/opscode-erchef/sasl-error.log <==

==> /var/log/opscode/opscode-erchef/requests.log.idx <==

==> /var/log/opscode/opscode-erchef/erchef.log <==
2017-07-06 19:31:37.949 [info] Can't connect to RabbitMQ management console to fetch max-length
2017-07-06 19:32:07.949 [info] Can't connect to RabbitMQ management console to fetch max-length
2017-07-06 19:32:37.949 [info] Can't connect to RabbitMQ management console to fetch max-length
2017-07-06 19:33:07.949 [info] Can't connect to RabbitMQ management console to fetch max-length
2017-07-06 19:33:37.949 [info] Can't connect to RabbitMQ management console to fetch max-length
2017-07-06 19:34:07.949 [info] Can't connect to RabbitMQ management console to fetch max-length
2017-07-06 19:34:37.950 [info] Can't connect to RabbitMQ management console to fetch max-length
2017-07-06 19:35:05.026 [error] {<<"method=POST; path=/users/; status=500; ">>,"Internal Server Error"}
2017-07-06 19:35:07.949 [info] Can't connect to RabbitMQ management console to fetch max-length
2017-07-06 19:35:09.053 [error] {<<"method=POST; path=/users/; status=500; ">>,"Internal Server Error"}

Any help much appreciated. No idea what’s going on really, but I know that the the test command also fails.

# chef-server-ctl test
Configuring logging...
Creating platform...
Configured URL: https://servername.example.com
Creating org pedant_testorg_servername_30767
bundler: failed to load command: ./bin/oc-chef-pedant (./bin/oc-chef-pedant)
RuntimeError: Bad error code 500 from create org: {"error":["internal service error"]}
  /opt/opscode/embedded/service/oc-chef-pedant/lib/pedant/platform.rb:337:in `block in create_org'
  /opt/opscode/embedded/service/oc-chef-pedant/lib/pedant/platform.rb:322:in `times'
  /opt/opscode/embedded/service/oc-chef-pedant/lib/pedant/platform.rb:322:in `create_org'
  /opt/opscode/embedded/service/oc-chef-pedant/lib/pedant/platform.rb:124:in `org_from_config'
  /opt/opscode/embedded/service/oc-chef-pedant/lib/pedant/platform.rb:43:in `initialize'
  /opt/opscode/embedded/service/oc-chef-pedant/lib/pedant.rb:81:in `new'
  /opt/opscode/embedded/service/oc-chef-pedant/lib/pedant.rb:81:in `create_platform'
  /opt/opscode/embedded/service/oc-chef-pedant/lib/pedant.rb:58:in `setup'
  /opt/opscode/embedded/service/oc-chef-pedant/bin/oc-chef-pedant:10:in `<top (required)>'

Looks like a similar error.
Any assistance for diagnosing would be greatly appreciated.

The same error has been coming basically since fresh installation.

Here is the output of chef-server-ctl status:

# chef-server-ctl status
run: bookshelf: (pid 17526) 322419s; run: log: (pid 17558) 322419s
run: nginx: (pid 17341) 322423s; run: log: (pid 17754) 322417s
run: oc_bifrost: (pid 17254) 322425s; run: log: (pid 17324) 322425s
run: oc_id: (pid 17331) 322424s; run: log: (pid 17337) 322424s
run: opscode-erchef: (pid 17573) 322418s; run: log: (pid 17587) 322418s
run: opscode-expander: (pid 17410) 322420s; run: log: (pid 17433) 322420s
run: opscode-solr4: (pid 17377) 322422s; run: log: (pid 17394) 322421s
run: postgresql: (pid 3813) 342914s; run: log: (pid 3823) 342913s
run: rabbitmq: (pid 3330) 342922s; run: log: (pid 3198) 342926s
run: redis_lb: (pid 17198) 322470s; run: log: (pid 17747) 322417s
2

What’s the output of chef-server-ctl status look like?

3

Added the output of chef-server-ctl status to the original post.

4

Was able to fix the error by changing the GECOS field of the user from which I had sudo -ed into root as. It had a < character in it, which caused issues during the initial install, and I think was never recovered from.

Steps to fix:

  1. Change the GECOS field of the Linux username to NOT include any < or [ special character.
  2. Reinstall Chef server from scratch (i.e. rpm -e chef-server-core then remove /var/opscode /var/log/opscode /opt/opscode and /etc/opscode, then run rpm -ivh chef-server-core.blahblah.rpm)
  3. Run chef-server-ctl reconfigure

More details found at Stack Overflow (also my post)

The original issue with the < was due to an error from sqitch which only made itself clear on the first two times attempting to run reconfigure, and which seemed to hide itself after that. It is not clear to me why the Linux user’s GECOS field was being used at all, but this seems to be an issue with sqitch . It would be helpful to have the limitation documented that a requirement for installing Chef server, however, that really caught me off guard.