I noticed that running dylan-test-suite-app ran fewer tests than running testworks-run --load libdylan-test-suite.so so I tried to figure out which tests hadn't been added to the top-level suite.
I ran dylan-test-suite both ways, generating a json report for each, and then tried to diff them with testworks-report app.json run.json --report diff and got reams and reams of output about checks, making it extremely difficult to know which tests were left out of app.json.
The primary thing I want from a test report diff is to know whether any tests in the OLD run failed in the NEW run and which tests, if any, have been added or removed (and their status).
Checks are too fine-grained to care about, in my opinion. Adding a check is uninteresting as long as the test still passes. Minor tweaks to the description could cause noisy diffs.
I noticed that running
dylan-test-suite-appran fewer tests than runningtestworks-run --load libdylan-test-suite.soso I tried to figure out which tests hadn't been added to the top-level suite.I ran
dylan-test-suiteboth ways, generating a json report for each, and then tried to diff them withtestworks-report app.json run.json --report diffand got reams and reams of output about checks, making it extremely difficult to know which tests were left out of app.json.The primary thing I want from a test report diff is to know whether any tests in the OLD run failed in the NEW run and which tests, if any, have been added or removed (and their status).
Checks are too fine-grained to care about, in my opinion. Adding a check is uninteresting as long as the test still passes. Minor tweaks to the description could cause noisy diffs.