I am trying to troubleshoot an issue with an on prem builder install. I have a new prod instance I am trying to get going but it is failing to auth with A2. Initially there was an error with the minio db as it wasn’t able to start because not enough disk space. I increased disks pace, uninstalled on prem builder, deleted the minio db by deleting the /hab/svc/builder-minio dir and then reinstalled the onprem builder. The minio service was then able to start.
After getting that sorted I started troubleshooting the UI error of “Unable to retrieve OAuth token” by confirming tokens, IDs, and URLS. Digging around awhile I found that the API service on this new instance doesn’t seem to be listening at all / binding to port 9636 (port is not in use), so OAuth is failing. I got this info from the error in /hab/svc/builder-api-proxy/host.error.log where upstream error of connection refused on port 9636. I can’t seem to get an error message in the logs for the api service log itself. I have tried changing the port it binds to by modifying the user.toml to another port but still no errors or port bindings. Any suggestions on how to try and work through this, like roll back to an older version like what is working in my dev instance or should I nuke this server and restart the process? Thanks in advance for any ideas.