Approach to testing core plans

Core Development Core Plans
1

Hello all!

I’m currently debating how best to test core plans. Yes, @eeyun and I are working on an automated testing framework, but in order to inform how we would automatically test something we need to first understand how we would do it manually.

My current approach when reviewing a PR to core plans is that, if it adds new behavior or changes existing behavior, it must be tested manually before being merged and promoted (or in some cases after merge, but before promotion to stable). This does slow down the release of core plans changes quite a bit, particularly when it pertains to something without a ready made or easy to write sample use case.

One thought is to require, for all core plans PRs, steps on how to test the new or changed behavior, including a sample app when necessary.

However, I’d love to get more input on this, I will use that input to form an RFC around this if necessary.

2

I have sometimes found other core plans that depend on the core plan being changed and tested the changes through them, but this is not possible for every situation.

3

Personally I’m not against obligating a test harness for all new plans. I can get some examples together for the current (planned) supported shape for those test harnesses to ease the understanding of new folks. For changes to packages that already exist - I think it’s reasonable to work with the contributor to determine what functionality may have changed and to document manual test steps in the PR which could then be translated to a test harness addition ticket on the backlog?

4

Thank you for the input. For now, I’m going to create a new RFC that states that pull requests must include manual steps to test that PR, with the caveat that those steps will be automated as much as possible when the automated testing framework is ready.

1 Like
5

RFC has been opened! https://github.com/habitat-sh/core-plans/pull/1281

6

Sounds like we’re asking for 2 types of testing:

  1. “Unit Testing”: Where the binary/things that are built actually can work. Possible for having do_check() sufficient? This would at least test if the binary/thing that was built before packaging works.

  2. “Functional Testing”: where the binary uses hab-sup, can it start up, configure the thing, run some health check, and more successfully.

7

Do you mind adding that to the RFC? If you have not already? Would be good to capture the conversation there as well.