[Err: 4] secret key mismatch

sbrars · September 25, 2019, 7:27am

Hello,

I have started using habitat studio locally on my windows 10 laptop. Did the hab setup correctly and able to enter the studio. I have created a local habitat plan replicating one of the examples available online. When I try to load it using the following command:
hab svc load origin/pkgname
I always get the following error:`
XXX
XXX [Err: 4] secret key mismatch
XXX
Enabled debug logs:
[2019-09-25T07:22:56Z DEBUG habitat_common::ui] UI { shell: Shell { input: InputStream { isatty: true }, out: OutputStream { coloring: Auto, isatty: true }, err: OutputStream { coloring: Auto, isatty: true } } }
[2019-09-25T07:22:56Z DEBUG hab] clap cli args: [“C:\hab\pkgs\core\hab-studio\0.83.0\20190712234514\bin\hab\hab.exe”, “svc”, “load”, “core/sqlserver”]
[2019-09-25T07:22:56Z DEBUG hab] remaining cli args:
[2019-09-25T07:22:56Z DEBUG hab::config] No CLI config found, loading defaults
[2019-09-25T07:22:56Z DEBUG tokio_reactor] adding I/O source: 0
[2019-09-25T07:22:56Z DEBUG tokio_reactor::background] starting background reactor
[2019-09-25T07:22:56Z DEBUG tokio_reactor::registration] scheduling Write for: 0
[2019-09-25T07:22:56Z DEBUG tokio_reactor::registration] scheduling Read for: 0
[2019-09-25T07:22:56Z DEBUG tokio_reactor::registration] scheduling Read for: 0
[2019-09-25T07:22:56Z DEBUG tokio_reactor] dropping I/O source: 0
XXX
XXX [Err: 4] secret key mismatch
XXX
I have even tried loading one of the publicly available packages but end up in same error.
Couple of things that I have already tried:

tried hab setup again
removed hab folder from user profile and it gets re-created

predominant · September 26, 2019, 7:12am

I don’t have a lot of experience with errors on the Windows client side. However there are a couple of things I would look at:

Update Habitat to the latest release. I believe there were HTTP client changes and SSL Cert handling changes that may resolve this problem
Debug your HTTP/s connections from your machine to Habitat Builder. I feel like the key mismatch is because of a failure of the HTTP client to reach the builder services correctly.

Let me know how you go.

sbrars · September 26, 2019, 7:28am

I am still new to habitat and trying to understand all components and their workflow. I am using hab 0.83.0/20190712231714. When I am trying to load a service from local hab studio, why would it try and connect to builder.
Does it need to download some dependent packages to load any service? I am working in an air gapped env, just fyi. So I probably need to resolve such issues explicitly rather than auto download.

sbrars · September 26, 2019, 8:04am

I am getting consistently issues while installing different packages with the following error while resolving dependencies and failing at different points. I am now on latest version of habitat 0.85:

PS C:> hab pkg install C:\hab\cache\artifacts\core-hab-sup-0.85.0-20190916222057-x86_64-windows.hart
» Installing C:\hab\cache\artifacts\core-hab-sup-0.85.0-20190916222057-x86_64-windows.hart
→ Verifying core/hab-sup/0.85.0/20190916222057
→ Using core/powershell/6.2.1/20190621130612
→ Verifying core/openssl/1.0.2r/20190725154702
→ Using core/zlib/1.2.11/20181108223359
→ Verifying core/libarchive/3.3.3/20190725155508
→ Using core/libsodium/1.0.13/20181108224956
→ Using core/visual-cpp-redist-2015/14.0.23026/20190128180056
→ Using core/xz/5.2.3/20181108223545
→ Verifying core/zeromq/4.3.1/20190802173722
→ Using core/bzip2/1.0.6/20190725144604
XXX
XXX Access is denied. (os error 5)
XXX

predominant · September 26, 2019, 8:36am

Can you confirm you have downloaded all dependencies? If you are airgapped and are missing any of the package dependencies, habitat will try to reach the builder service as defined in the HAB_BLDR_URL environment variable, or https://bldr.habitat.sh by default.

In addition, it needs to fetch public keys for those packages to be able to verify them before installation.

sbrars · September 27, 2019, 2:03am

So I am able to get into studio environment successfully and can see clean supervisor logs. When I try to load the locally built .hart package from studio; it complains about the same
XXX
XXX [Err: 4] secret key mismatch
XXX

Does it need to contact builder for loading my locally packages even as while loading the local package I am air-gapped. What other dependencies do I need here?

predominant · September 27, 2019, 2:17am

Is it possible you’re running into the issues described here? https://github.com/habitat-sh/habitat/issues/5169

sbrars · September 27, 2019, 2:43am

Yeah I guess so and I commented on the issue already yeasterday but still awaiting someone to respond.

christophermaier_1 · September 30, 2019, 2:14pm

@sbrar7 Do you have a Supervisor running on your host machine outside of the Studio?

When you enter the Studio, does the Supervisor in the studio start up successfully? You should be able to use the sup-log command in the studio to show the logs.

The error comes about when the hab command isn’t using the authentication token the Supervisor expects, so this is isn’t related to communication with Builder or anything like that. There may be some other issues in your environment that do involve Builder, but the “secret key mismatch” error is separate from any of that.

Matt_Wrock · September 30, 2019, 9:26pm

Another thing to look for is if you setup a Supervisor CtlGateway secret setup. You would have specified this during hab setup and if you did then this could explain why you are getting this in a local windows studio. Try running $env:HAB_CTL_SECRET=$null and then running your hab svc load command.

sbrars · October 1, 2019, 1:47am

@christophermaier, @mwrock thanks for your reply.
none of the above helped still, to answer your queries above,
When I enter the studio, yes the supervisor starts successfully and I can see the following:
Logging configuration file ‘H:\hab\studios\hab/sup\default\config\log.yml’ not found; using default logging configuration
hab-sup(MR): core/hab-sup (core/hab-sup/0.85.0/20190916222057)
hab-sup(MR): Supervisor Member-ID a7bcd344466f48749548c69350807dc3
hab-sup(MR): Starting gossip-listener on 0.0.0.0:9638
hab-sup(MR): Starting ctl-gateway on 127.0.0.1:9632
hab-sup(MR): Starting http-gateway on 0.0.0.0:9631

Also, I did set the CtlGateway secret during the setup and have now reset it explicitly to null value but no luck. Additionally, I tried running the hab setup again to say no to Ctl Gateway secret but that also didn’t help. While trying to load the service, I still get same error:
[HAB-STUDIO] Habitat:\src> hab svc load sbrar7/windows-sample-service
XXX
XXX [Err: 4] secret key mismatch
XXX

Does it matter, where my results folder should be present which contains .hart file. At present, it sits parallel to service:
[HAB-STUDIO] Habitat:\src> ls

Directory: H:\hab\studios\src

Mode LastWriteTime Length Name

d----- 26/09/2019 4:53 PM results
d----- 26/09/2019 4:50 PM windows-service-sample

Thanks

Matt_Wrock · October 1, 2019, 3:29am

It occurs to me that nullifying the environment variable will not work because the supervisor has already started. You will need to remove the ctl_secret from your hab/etc/cli.toml file before entering the studio.

sbrars · October 1, 2019, 2:27am

@mwrock awesome! that worked. Just to understand this a bit more in detail, so when should I be setting the hab ctl secret and why was that conflicting in my environment.

Matt_Wrock · October 1, 2019, 3:17am

You would set that if you have remote supervisors that run with that secret. We have a PR open that will prevent the ctl_secret from being added to a studio environment since you would not want/need a ctl_secret when interacting with a local supervisor.