On-prem builder api port not binding

Habitat Help
1

Hello,

I am trying to troubleshoot an issue with an on prem builder install. I have a new prod instance I am trying to get going but it is failing to auth with A2. Initially there was an error with the minio db as it wasn’t able to start because not enough disk space. I increased disks pace, uninstalled on prem builder, deleted the minio db by deleting the /hab/svc/builder-minio dir and then reinstalled the onprem builder. The minio service was then able to start.

After getting that sorted I started troubleshooting the UI error of “Unable to retrieve OAuth token” by confirming tokens, IDs, and URLS. Digging around awhile I found that the API service on this new instance doesn’t seem to be listening at all / binding to port 9636 (port is not in use), so OAuth is failing. I got this info from the error in /hab/svc/builder-api-proxy/host.error.log where upstream error of connection refused on port 9636. I can’t seem to get an error message in the logs for the api service log itself. I have tried changing the port it binds to by modifying the user.toml to another port but still no errors or port bindings. Any suggestions on how to try and work through this, like roll back to an older version like what is working in my dev instance or should I nuke this server and restart the process? Thanks in advance for any ideas.

Best,
David

2

Hello,
An update on this. I rebuilt the server and ended up in the same spot. I left out a bit of info that I think helped me solve this in a very odd way. This is a FIPS complaint environment. The way I ended up getting this to work was following a non-error message in syslog that said “hab: builder-api.default(O): Waiting for builder-github-app.pem”. I checked the code and it looked like there and noticed it was placing a dummy file there, https://github.com/habitat-sh/builder/blob/master/.expeditor/templates/studiorc#L16. I made one by making an ssl key and placing it /hab/svc/builder-api/files/builder-github-app.pem and now everything is working.

My question is what should I do to try and prevent this in the future? Should I make an issue in the git repo with this or what?

Best,
David

3

@aredeex1 thanks for bringing this our attention! :smile: We've got a PR in with a fix: accounting for other oauth providers in api run hook by jeremymv2 · Pull Request #1397 · habitat-sh/builder · GitHub

1 Like
4

Just an update that the fix has been promoted to the on-prem-stable channel.
The specific pkg is habitat/builder-api/8885/20200504153856