Service reloading due to changing order of templated config file

Habitat Help
1

Hi,

Since Habitat version 0.79.1 we see our Prometheus service, which binds to many Prometheus exporters, being endlessly reloaded every ~10 seconds with the message Modified configuration file /hab/svc/prometheus/config/prometheus.yml. Diffing prometheus.yml between reloads shows that the content is the same, but written out in a different order.

The simplified example below demonstrates the behaviour. In the example, I load two Prometheus Exporters, then load a Prometheus service that binds to the exporters. I create a simple template file under the config directory, and examine the content through four service reloads. On the fourth reload, the order of the config file is reversed.

The behaviour is impacting us because, in our prometheus.yml template, we loop over all items in the bind object, rather than explicitly naming each of our growing list of 200+ binds.

[18][default:/src:0]# hab --version
hab 0.79.1/20190410220617

[19][default:/src:0]# hab svc load tm/node_exporter
The tm/node_exporter service was successfully loaded
[20][default:/src:0]# hab svc load tm/hab_exporter
The tm/hab_exporter service was successfully loaded
[21][default:/src:0]# hab sup status
package                                 type        desired  state  elapsed (s)  pid  group
tm/hab_exporter/0.1.0/20190508152149    standalone  up       up     8            299  hab_exporter.default
tm/node_exporter/0.17.0/20190621170541  standalone  up       up     13           283  node_exporter.default

[22][default:/src:0]# hab pkg install mfol/prometheus
» Installing mfol/prometheus
☁ Determining latest version of mfol/prometheus in the 'stable' channel
→ Using mfol/prometheus/2.3.2/20190703101825
★ Install of mfol/prometheus/2.3.2/20190703101825 complete with 0 new packages installed.

[23][default:/src:0]# cat << EOF > /hab/pkgs/mfol/prometheus/2.3.2/20190703101825/config/bind.json
> {{toJson bind}}
> EOF

[24][default:/src:0]# hab svc load mfol/prometheus --bind grafana_node_exporter:node_exporter.default --bind grafana_hab_exporter:hab_exporter.default
The mfol/prometheus service was successfully loaded

[25][default:/src:0]# grep "_exporter" /hab/svc/prometheus/config/bind.json 
  "grafana_hab_exporter": {
        "name": "hab_exporter",
      "service": "hab_exporter",
          "name": "hab_exporter",
        "service": "hab_exporter",
  "grafana_node_exporter": {
        "name": "node_exporter",
      "service": "node_exporter",
          "name": "node_exporter",
        "service": "node_exporter",

[26][default:/src:0]# hab svc unload mfol/prometheus
Unloading mfol/prometheus
[27][default:/src:0]# hab svc load mfol/prometheus --bind grafana_node_exporter:node_exporter.default --bind grafana_hab_exporter:hab_exporter.default
The mfol/prometheus service was successfully loaded
[28][default:/src:0]# grep "_exporter" /hab/svc/prometheus/config/bind.json 
  "grafana_hab_exporter": {
        "name": "hab_exporter",
      "service": "hab_exporter",
          "name": "hab_exporter",
        "service": "hab_exporter",
  "grafana_node_exporter": {
        "name": "node_exporter",
      "service": "node_exporter",
          "name": "node_exporter",
        "service": "node_exporter",

[29][default:/src:0]# hab svc unload mfol/prometheus
Unloading mfol/prometheus
[30][default:/src:0]# hab svc load mfol/prometheus --bind grafana_node_exporter:node_exporter.default --bind grafana_hab_exporter:hab_exporter.default
The mfol/prometheus service was successfully loaded
[31][default:/src:0]# grep "_exporter" /hab/svc/prometheus/config/bind.json 
  "grafana_hab_exporter": {
        "name": "hab_exporter",
      "service": "hab_exporter",
          "name": "hab_exporter",
        "service": "hab_exporter",
  "grafana_node_exporter": {
        "name": "node_exporter",
      "service": "node_exporter",
          "name": "node_exporter",
        "service": "node_exporter",

[32][default:/src:0]# hab svc unload mfol/prometheus
Unloading mfol/prometheus
[33][default:/src:0]# hab svc load mfol/prometheus --bind grafana_node_exporter:node_exporter.default --bind grafana_hab_exporter:hab_exporter.default
The mfol/prometheus service was successfully loaded
[34][default:/src:0]# grep "_exporter" /hab/svc/prometheus/config/bind.json 
  "grafana_node_exporter": {
        "name": "node_exporter",
      "service": "node_exporter",
          "name": "node_exporter",
        "service": "node_exporter",
  "grafana_hab_exporter": {
        "name": "hab_exporter",
      "service": "hab_exporter",
          "name": "hab_exporter",
        "service": "hab_exporter",
[35][default:/src:0]#

I imagine that this is expected behaviour based on the current code, but wonder if this is an edge case that may have been overlooked.

Thanks,
Oliver

2

Yeah, I think this was an unanticipated edge case, unfortunately. Would you mind filing an issue? I think the preserve_order feature you mentioned about this in Slack is probably the way to fix it.

Thanks!

3

Done, thanks Chris. https://github.com/habitat-sh/habitat/issues/6713

2 Likes