Habitat constantly recompiling/restarting app?

Habitat Help
1

I updated the core/artifactory-pro plan to have a bind for psql:

url=jdbc:postgresql://{{bind.database.leader.sys.ip}}:{{bind.database.leader.cfg.port}}/artifactory

This has been running fine for a few weeks and in fact is running fine on one other node.

However, on one node Habitat seems to be continually restarting Artifactory.

Jun 28 02:28:53 artifactory-1 hab[1485]: artifactory-pro.default(SR): The group 'postgresql.default' satisfies the `database` bind
Jun 28 02:28:53 artifactory-1 hab[1485]: artifactory-pro.default(HK): Hooks compiled
Jun 28 02:28:54 artifactory-1 hab[1485]: artifactory-pro.default(SR): The group 'postgresql.default' satisfies the `database` bind
Jun 28 02:28:54 artifactory-1 hab[1485]: artifactory-pro.default(HK): Hooks compiled
Jun 28 02:28:57 artifactory-1 hab[1485]: artifactory-pro.default(SR): The group 'postgresql.default' satisfies the `database` bind
Jun 28 02:28:57 artifactory-1 hab[1485]: artifactory-pro.default(HK): Hooks compiled
Jun 28 02:28:58 artifactory-1 hab[1485]: artifactory-pro.default(SR): The group 'postgresql.default' satisfies the `database` bind
Jun 28 02:28:58 artifactory-1 hab[1485]: artifactory-pro.default(HK): Hooks compiled
Jun 28 02:28:59 artifactory-1 hab[1485]: artifactory-pro.default(SR): The group 'postgresql.default' satisfies the `database` bind
Jun 28 02:28:59 artifactory-1 hab[1485]: artifactory-pro.default(HK): Hooks compiled
Jun 28 02:29:00 artifactory-1 hab[1485]: artifactory-pro.default(SR): The group 'postgresql.default' satisfies the `database` bind
Jun 28 02:29:00 artifactory-1 hab[1485]: artifactory-pro.default(HK): Hooks compiled

## Artifactory/TomCat Startup Messages
Jun 28 02:29:34 artifactory-1 hab[1485]: artifactory-pro.default(O): Caught termination signal
Jun 28 02:29:35 artifactory-1 hab[1485]: artifactory-pro.default(O): Starting Artifactory Pro
Jun 28 02:29:35 artifactory-1 hab[1485]: artifactory-pro.default(O): JAVA_HOME: /hab/pkgs/core/jre8/8.172.0/20180419233349
Jun 28 02:29:35 artifactory-1 hab[1485]: artifactory-pro.default(O): ARTIFACTORY_HOME: /hab/svc/artifactory-pro/var
Jun 28 02:29:35 artifactory-1 hab[1485]: artifactory-pro.default(O): Found java executable in JAVA_HOME (/hab/pkgs/core/jre8/8.172.0/20180419233349)
Jun 28 02:29:35 artifactory-1 hab[1485]: artifactory-pro.default(O): Jun 28, 2018 2:29:35 AM org.apache.coyote.AbstractProtocol init
Jun 28 02:29:35 artifactory-1 hab[1485]: artifactory-pro.default(SR): The group 'postgresql.default' satisfies the `database` bind
Jun 28 02:29:35 artifactory-1 hab[1485]: artifactory-pro.default(HK): Hooks compiled

Note that I’m running the same version of .hart file on my second node and getting the same messages, but artifactory does not seem to be restarting:

Jun 28 02:32:43 artifactory-2 hab[2489]: #033[32martifactory-pro.default#033[0m(#033[1;37mSR#033[0m): The group 'postgresql.default' satisfies the `database` bind
Jun 28 02:32:43 artifactory-2 hab[2489]: #033[32martifactory-pro.default#033[0m(#033[1;37mHK#033[0m): Hooks compiled

Also, running the run script in /hab/svc/artifactory-pro/run starts and runs without issue.

I’m stumped. I even upgraded from 0.56 to 0.57. There seems to be no way to coax “why are you doing what you’re doing!!!” out of the supervisor? I’m fairly certain that the habitat supervisor is sending the kill signal, but I can’t seem to prove it…

Thanks!

2

So after a bit more mucking about, it seems that when Artifactory starts up, it encrypts the password in config/db.properties. I just happened to notice:

Jun 28 03:04:37 artifactory-1 hab[18390]: default(CF): Updated db.properties 4bccf1f397d626b42619688a89f0bef609e1ecad52fd7281b7aae588f75e8c42

Now, why:

  1. …this all of a sudden started happening today…
  2. …the password isn’t updated on my second node…
  3. …artifactory proceeds to encrypt the password after disabling password encryption…

I do not know…

What it looks like is happening:

  1. Habitat compiles config file with unencrypted password
  2. Artifactory starts up, as a final step it encrypts the password in db.properties
  3. Habitat sees that the config has changed, so it updates the config, and restarts the service. GOTO 2

It seems like the problem boils down to 2 things:

  1. How to create a config file that can be updated by the system service?
  2. How to get supervisor log messages. With everything being dumped to /var/log/messages|syslog seeing the “Updated db.properties” was hard to find and seeing what the change required a watch -n0.1 cat /hab/svc/artifactory-pro/config/db.properties (I’m sure there’s another way, but it was the easiest I could think of to watch the file change…)

Ultimately, I want the ability to bind to the master of my pgsql cluster… Maybe copying from config/ to var/etc instead of linking would work in the interim? That way the system can update the config file, but if the IP is changed in the JDBC string it’ll be copied over the existing config file and artifactory restarted…

3

@christophermaier Any thoughts here?

4

Just to follow up, I was able to solve my immediate problem of the server restarting by updating the init/run hooks to copy db.properties from {svc.pkg_cfg_dir}/ to {svc.pkg_var_dir} instead of linking, which allows the process to make the files that it wants to.

I’m not convinced this is the best solution, but seems to be the easiest to implement at the moment.

Fundamentally, I think the issue is that Habitat wants to be the source of Truth, but sometimes it can’t completely be the source of truth…

(Kinda like the reason why the line cookbook exists… Chef really wants to own/manage the whole config file, but sometimes it’s just not possible)

Ultimately, I think what I’m asking for is “Partial Configs”…

2 Likes
5

Oh man, I think partial configs could end up being a pretty dope feature. If you have some time this week @qubitrenegade would you be willing to open a feature request in habitat-sh/habitat for that?

6

Config partials would be kickass.

1 Like
7

@eeyun not last week with the holiday :stuck_out_tongue:

but yea, I think I can do that!

1 Like