On-prem GitHub OAuth Issues

Habitat Help
#1

I'm setting up the on-prem Habitat Depot following the instructions detailed in the repo README here: on-prem-builder. I'm getting the following error message after configuring GitHub OAuth and could use some help troubleshooting.

Authentication Failed, Unable to retrieve OAuth token

I've verified that my client id and client secret match what is in /hab/svc/builder-api/config/config.toml and that my GitHub OAuth app URLs are filled out correctly, here is a screenshot of the config (the domain names are internal-only)

image

Any help would be greatly appreciated!

#2

@danielcbright Is there anything useful in the network console output from your browser when you get the auth error?

#3

@elliott-davis: Using Firefox Debugging, here are the request/response headers to GitHub:

Request:

Host: github.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:60.0) Gecko/20100101 Firefox/60.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://hab-depot.dbright.com/
Cookie: logged_in=yes; _octo=GH1.1.1351481130.1517324159; _ga=GA1.2.1174774092.1517324159; _gh_sess=..................................=America%2FNew_York; user_session=..................................; dotcom_user=danielcbright; _gat=1
Connection: keep-alive
Upgrade-Insecure-Requests: 1

Response:

HTTP/1.1 302 Found
Server: GitHub.com
Date: Tue, 29 May 2018 16:21:57 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Status: 302 Found
Cache-Control: no-cache
Vary: X-PJAX
Location: https://hab-depot.dbright.com/?code=.........................................
Set-Cookie: user_session=.........................................; path=/; expires=Tue, 12 Jun 2018 16:21:57 -0000; secure; HttpOnly
Set-Cookie: __Host-user_session_same_site=.........................................; path=/; expires=Tue, 12 Jun 2018 16:21:57 -0000; secure; HttpOnly; SameSite=Strict
Set-Cookie: _gh_sess=.........................................; path=/; secure; HttpOnly
X-Request-Id: b63b53c1-255f-410c-bf0e-f6df1b51c18c
X-Runtime: 0.085335
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Frame-Options: deny
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin
Expect-CT: max-age=2592000, report-uri="https://api.github.com/_private/browser/errors"
Content-Security-Policy: default-src 'none'; base-uri 'self'; block-all-mixed-content; connect-src 'self' uploads.github.com status.github.com collector.githubapp.com api.github.com www.google-analytics.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com wss://live.github.com; font-src assets-cdn.github.com; form-action 'self' github.com gist.github.com; frame-ancestors 'self'; frame-src render.githubusercontent.com; img-src 'self' data: assets-cdn.github.com identicons.github.com collector.githubapp.com github-cloud.s3.amazonaws.com *.githubusercontent.com; manifest-src 'self'; media-src 'none'; script-src assets-cdn.github.com; style-src 'unsafe-inline' assets-cdn.github.com
X-Runtime-rack: 0.096285
X-GitHub-Request-Id: CD2E:6EEE:418A116:78297D4:5B0D7E1C
#4

Were there any network responses that resulted in an error?

#5

There is a server piece that does authentication as well. Is there anything in the server logs (which you should be able to mine with journalctl) that looks like an error, specifically around the /authenticate endpoint?

#6

@raskchanky nothing is showing up in journalctl for authenticate, any other troubleshooting steps?

#7

On a possibly unrelated note, but adding here for the sake of completeness, my sl logs show this error repeatedly:

hab-sup(ER)[components/sup/src/error.rs:452:9]: Butterfly error: Cannot bind to port: Error { repr: Os { code: 98, message: "Address already in use" } }
hab-sup(MR): Supervisor Member-ID c3f669b8d9fc49b1b350ada2df761d56
hab-sup(MR): Starting gossip-listener on 0.0.0.0:9638
hab-sup(ER)[components/sup/src/error.rs:452:9]: Butterfly error: Cannot bind to port: Error { repr: Os { code: 98, message: "Address already in use" } }
hab-sup(MR): Supervisor Member-ID c3f669b8d9fc49b1b350ada2df761d56
hab-sup(MR): Starting gossip-listener on 0.0.0.0:9638
hab-sup(ER)[components/sup/src/error.rs:452:9]: Butterfly error: Cannot bind to port: Error { repr: Os { code: 98, message: "Address already in use" } }
hab-sup(MR): Supervisor Member-ID c3f669b8d9fc49b1b350ada2df761d56
hab-sup(MR): Starting gossip-listener on 0.0.0.0:9638
hab-sup(ER)[components/sup/src/error.rs:452:9]: Butterfly error: Cannot bind to port: Error { repr: Os { code: 98, message: "Address already in use" } }
hab-sup(MR): Supervisor Member-ID c3f669b8d9fc49b1b350ada2df761d56
hab-sup(MR): Starting gossip-listener on 0.0.0.0:9638
hab-sup(ER)[components/sup/src/error.rs:452:9]: Butterfly error: Cannot bind to port: Error { repr: Os { code: 98, message: "Address already in use" } }
hab-sup(MR): Supervisor Member-ID c3f669b8d9fc49b1b350ada2df761d56
hab-sup(MR): Starting gossip-listener on 0.0.0.0:9638

A netstat -anp | grep 9638 shows this output:

[root@hab-depot ~]# netstat -anp | grep 9638
tcp        0      0 0.0.0.0:9638            0.0.0.0:*               LISTEN      1416/hab-sup
udp        0      0 0.0.0.0:9638            0.0.0.0:*                           1416/hab-sup
#8

Yeah I think this is definitely part of the problem. Somehow you have multiple supervisors trying to bind to the same port. Any time you enter a studio, it tries to start a supervisor in the background, and if there’s already one running outside the studio, that will likely cause a conflict. You can enter a studio without starting a supervisor by running env HAB_STUDIO_SUP=0 hab studio enter.

#9

I’m not following why you are entering a studio on your node running on-prem builder

#10

I get hab-sup listening on both UDP and TCP at system startup, I don’t even have to enter the studio for that to occur.

#11

You can have both TCP and UDP listening on the same port at the same time without any problems. As far as I know, this is expected behavior from the supervisor. I don’t think that’s what’s causing your error.

#12

I think it’s a red herring, it’s only when using the sl command from within the studio, when I ran as @raskchanky suggested the Cannot bind to port error went away, but I’m still having the OAuth issues.

#13

When you enter a studio, you start a supervisor in the background. If you are entering a studio on the same node as the Habitat Depot is running, you’ll get a collision

#14

Totally understood, I’m past that point and back to the original problem of OAuth and GitHub not working, curious if any other troubleshooting steps I can take.

#15

I did a complete re-install (super easy :slight_smile: ), and deleted/recreated the app in GitHub, got a new Client/Priv token, and still the same issue. I’m wondering if others are having this issue with on-prem Depot or not. I’ve been looking at GitHub’s OAuth documentation and the 302 redirect may be the issue here? Not sure…

#16

This is the first issue like this that we’ve seen. We have a number of people using the on-prem depot with GitHub without issues.

It’s difficult to assist any further without more detailed information from your browser console and/or server logs. One possibility would be to right click in the network pane of the dev tools and save everything as a HAR file (not to be confused with hart file, different things) and send the HAR file to one of the Habitat core team members via Slack. That would let us see exactly what you’re seeing.

FWIW, the 302 from GitHub is correct. The way it should work is, after clicking the sign in button, you’re redirected to GitHub, which redirects back to your site with a code. The server then uses that code to authenticate your user and pull down some relevant information. The debug log that you posted from Firefox looks like that’s exactly what’s happening.

#17

ok thanks @raskchanky, I’ll do that and post the outcome here for anyone who runs into the same issue (I’m sure it’s something silly on my side now)

#18

Thanks to @echohack for taking a look at my bldr.env, it was a very simple mistake of leaving out the s in https for this line:

export APP_URL=http://hab-depot.dbright.com

Thanks for the help everyone! Hopefully this helps others :slight_smile:

2 Likes