main test still says UNKNOWN
If test is WARN, top-level should also be WARN
ok. So SKIP is working as expected wrt item, but parent level test status is not correct (UNKNOWN)
Ok. So the rollup is not correct
Let me check that.
For parent test, I think the conditions should be:
All item PASS, parent PASS
All item PASS or SKIP, parent PASS
one item WARN, rest PASS/SKIP, parent WARN
one item FAIL, parent FAIL
one item WAIVED, no fail, parent WAIVED
precedence is: FAIL, WARN, CHECK, PASS
The following rules need to apply to megatest for status for
parent tests with items
1 FAIL item parent test FAIL
no FAIL, 1 CHECK parent test CHECK
no FAIL, no CHECK, 1 WAIVED parent test WAIVED
no FAIL, no CHECK, no WAIVED, 1 WARN parent test WARN
no FAIL, no CHECK, no WAIVED, no WARN, 1 SKIP parent test PASS
all SKIP parent test SKIP
all PASS parent test PASS
Precedence order is:
FAIL => CHECK => WAIVED => WARN => PASS => SKIP
1 corner case:
If all items SKIP, parent is SKIP. So if total number of ITEMS and
total number if SKIP match, then parent is SKIP.
SKIP is a lower priority than PASS as it reflects a preemptive cut
of a PASS condition (nothing was actually checked/done) whereas PASS
implies actual work to QA.
<hr /><i>matt added on 2013-04-24 23:21:44 UTC:</i><br />
<verbatim>
The following rules need to apply to megatest for status for
parent tests with items
1 FAIL item parent test FAIL
no FAIL, 1 CHECK parent test CHECK
no FAIL, no CHECK, 1 WAIVED parent test WAIVED
no FAIL, no CHECK, no WAIVED, 1 WARN parent test WARN
no FAIL, no CHECK, no WAIVED, no WARN, 1 SKIP parent test PASS
all SKIP parent test SKIP
all PASS parent test PASS
Precedence order is:
FAIL => CHECK => WAIVED => WARN => PASS => SKIP
1 corner case:
If all items SKIP, parent is SKIP. So if total number of ITEMS and
total number if SKIP match, then parent is SKIP.
SKIP is a lower priority than PASS as it reflects a preemptive cut
of a PASS condition (nothing was actually checked/done) whereas PASS
implies actual work to QA.
</verbatim>
|