# How to debug supervisor which does not update a certain package to its latest version

**URL:** <https://discourse.chef.io/t/how-to-debug-supervisor-which-does-not-update-a-certain-package-to-its-latest-version/19409>\
**Category:** Habitat Help\
**Created:** [October 10, 2018, 4:50pm UTC](https://discourse.chef.io/t/how-to-debug-supervisor-which-does-not-update-a-certain-package-to-its-latest-version/19409 "2018-10-10T16:50:05Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Steve\_Hummingbird](https://avatars.discourse-cdn.com/v4/letter/s/b487fb/32.png) [@Steve\_Hummingbird](https://discourse.chef.io/u/Steve_Hummingbird)\
**Post date:** [October 10, 2018, 4:50pm UTC](https://discourse.chef.io/t/how-to-debug-supervisor-which-does-not-update-a-certain-package-to-its-latest-version/19409/1 "2018-10-10T16:50:06Z")

</div>

Any suggestions on how to debug or collect information about a supervisor which does not update a package to its latest version?

habitat is set up via the latest official chef habitat cookbook. The supervisor manages two services: One that updates without any issues and one that is stale at the currently installed version. An identical setup is running on another node, which updates just fine, however is a little younger than the one in question.

I fount that there is one way to update to the latest version manually:

```
hab svc unload myorg/mypkg
hab pkg install --auth mytoken myorg/mypkg (no explicit version, but it installs the latest)
hab svc load myorg/mypkg

```

invoking unload and load without install does not update to the latest version

---

<div class="post-metadata">

**Author:** ![eeyun1](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/eeyun1/32/2658_2.png) [@eeyun1](https://discourse.chef.io/u/eeyun1)\
**Post date:** [October 10, 2018, 5:37pm UTC](https://discourse.chef.io/t/how-to-debug-supervisor-which-does-not-update-a-certain-package-to-its-latest-version/19409/2 "2018-10-10T17:37:28Z")

</div>

Strange, theres a couple things that could cause this behavior, usually related to envvars or the flags passed when the service was started. I don’t know much about the habitat cookbook but I think the first thing you’re going to want to do is get the supervisor logs. With the cookbook I think the sup is (usually) being started as a systemd service so you’re going to want to use `journalctl` to see whats going on. If you could grab those logs and dump them here thats probably the easiest way to deduce whats going on.

You could also theoretically use `ps aux | grep hab` to see if one of those services wasn’t started with an upgrade strategy.

---

<div class="post-metadata">

**Author:** ![Steve\_Hummingbird](https://avatars.discourse-cdn.com/v4/letter/s/b487fb/32.png) [@Steve\_Hummingbird](https://discourse.chef.io/u/Steve_Hummingbird)\
**Post date:** [October 11, 2018, 3:11pm UTC](https://discourse.chef.io/t/how-to-debug-supervisor-which-does-not-update-a-certain-package-to-its-latest-version/19409/3 "2018-10-11T15:11:53Z")

</div>

Thank you for the response. However, I have checked the system logs in the past before and I could not find anything related to that issue. Today I compared the running processes on both systems as well as watch the logs during a deployment. I found the following things:

- `ps aux | grep hab` is identical on both nodes. I don’t see any hint regarding update strategy. The java process is the one that won’t update

```auto
root 1651 0.2 0.4 218112 26284 ? Sl Oct01 31:02 /hab/pkgs/core/hab-sup/0.63.0/20180914030447/bin/hab-sup run
root 19843 0.0 0.0 37716 6088 ? S 18:36 0:00 nginx: master process /hab/pkgs/core/nginx/1.15.5/20181010011756/bin/nginx -c /hab/svc/www-core/config/nginx.conf
hab 19850 0.0 0.0 38028 4992 ? S 18:36 0:00 nginx: worker process
hab 19851 0.0 0.0 38028 4528 ? S 18:36 0:00 nginx: worker process
hab 20218 1.1 4.7 4091364 288564 ? Sl 18:56 0:45 java -Dvertx.disableFileCPResolving=true -Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.Log4j2LogDelegateFactory -jar src-0.3.13-fat.jar run groovy:com.superevilcompany.core.Server -conf /hab/svc/superevilcompany-core/config/conf.json

```

- journalctl only shows messages regarding deployments of the other package that works fine. There is nothing listed regarding the package that won’t update

So far I think I need a more sophisticated strategy to find out what really is going on.

Just double checked. It is the same habitat version that is running on both systems. (hab 0.63.0/20180914025124)

This is journalctl on `hab svc load` after I unloaded the service (no explicit install). Yet again I do not seen anything of interest here…

```auto
Oct 11 17:05:03 stack-01 hab[1108]: hab-sup(AG): The myorg/mypkg-core service was successfully loaded
Oct 11 17:05:06 stack-01 hab[1108]: hab-sup(MR): Starting myorg/mypkg-core
Oct 11 17:05:06 stack-01 hab[1108]: service-core.default(UCW): Watching user.toml
Oct 11 17:05:06 stack-01 hab[1108]: www-core.default(HK): Hooks compiled
Oct 11 17:05:06 stack-01 hab[1108]: service-core.default(HK): Hooks compiled
Oct 11 17:05:06 stack-01 hab[1108]: service-core.default(SR): Initializing
Oct 11 17:05:06 stack-01 hab[1108]: service-core.default(SV): Starting service as user=hab, group=hab
Oct 11 17:05:07 stack-01 hab[1108]: www-core.default(HK): Hooks compiled
Oct 11 17:05:07 stack-01 hab[1108]: service-core.default(HK): Hooks compiled

```

invoking install after unload installs the new version, which then runs fine if I invoke load again without a version number. It looks somewhat like when trying to load a private package when no hab token is present, however if that would be the case then the other package would not work as well… So, yeah… pretty strange…

---

<div class="post-metadata">

**Author:** ![eeyun1](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/eeyun1/32/2658_2.png) [@eeyun1](https://discourse.chef.io/u/eeyun1)\
**Post date:** [October 12, 2018, 3:36pm UTC](https://discourse.chef.io/t/how-to-debug-supervisor-which-does-not-update-a-certain-package-to-its-latest-version/19409/4 "2018-10-12T15:36:32Z")

</div>

Ah I definitely left something out of my previous response: you’re gonna want to use the `RUST_LOG=debug` envvar to get more useful output in the logs. That should give us a better idea of whats occurring for each service.

---

<div class="post-metadata">

**Author:** ![Steve\_Hummingbird](https://avatars.discourse-cdn.com/v4/letter/s/b487fb/32.png) [@Steve\_Hummingbird](https://discourse.chef.io/u/Steve_Hummingbird)\
**Post date:** [October 15, 2018, 10:13am UTC](https://discourse.chef.io/t/how-to-debug-supervisor-which-does-not-update-a-certain-package-to-its-latest-version/19409/5 "2018-10-15T10:13:29Z")

</div>

yeah, that makes sense 😃  
I manually modified the upstart script to include the env var and after a reboot it clearly shows that the service is not started with a update strategy:

`DEBUG 2018-10-15T09:35:15Z: habitat_sup::manager::service::spec: Writing service spec to '/hab/sup/default/specs/mypkg-core.spec': ServiceSpec { ident: PackageIdent { origin: "mypkg", name: "mypkg-core", version: None, release: None }, group: "default", application_environment: None, bldr_url: "https://bldr.habitat.sh", channel: "stable", topology: Standalone, update_strategy: None, binds: [], binding_mode: Strict, config_from: None, desired_state: Up, svc_encrypted_password: None, composite: None }`

However, checking the used recipe in chef manage shows that the service is configured to use strategy ‘at-once’

```
hab_service 'myorg/mypkg-core' do
  action :load
  strategy 'at-once'
  channel node.habitat['channel']
end

```

So either this is an issue with the chef habitat cookbook, or with habitat itself. However as this node has seen a few updates, this might also have happened along the way when upgrading. If there is any additional information I could gather that would be helpful, please do let me know. Otherwise I’ll just throw away that node and start with a clean state and see if I’ll run into this again.

Yet, probably chef should be able to figure out that a service is running with the wrong strategy and take care of that. After looking through the cookbook, I don’t think that there is any implementation which would check if a service is configured correctly. I’ll open an [issue](https://github.com/chef-cookbooks/habitat/issues/141) over at the repo. Maybe we can find a way to improve this.

---

<div class="post-metadata">

**Author:** ![eeyun1](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/eeyun1/32/2658_2.png) [@eeyun1](https://discourse.chef.io/u/eeyun1)\
**Post date:** [October 15, 2018, 5:29pm UTC](https://discourse.chef.io/t/how-to-debug-supervisor-which-does-not-update-a-certain-package-to-its-latest-version/19409/6 "2018-10-15T17:29:53Z")

</div>

That’s some damn good detective work! I haven’t heard anyone else reporting a bug like this so I bet you’re right, probably an issue with the cookbook. Thank you so much for following that through to the end!