Some time ago I wrote a short blog post about equivalence partitioning and boundary value analysis. Now with new version of Conformiq Designer coming out very soon we have introduced quite a few new testing heuristics and coverage options that are based on equivalence class partitioning I wanted to write about how Conformiq can use these new heuristics for generating really good quality test cases.
Condition Coverage
Condition Coverage or Predicate Coverage guides Conformiq Designer to look for behaviors that cause each Boolean sub-expression evaluate both to true and false (a condition is a Boolean sub-expression that cannot be broken down into a simpler Boolean expression). For example, assume we have the following QML (Conformiq Designer modeling language) fragment in our system model:
if ((x < 0) && (y == 10)) { … }
In order to fulfill Condition Coverage criteria for the above example, Conformiq Designer will look for behaviors that cause both ‘(x < 0)’ and ‘(y == 10)’ to evaluate at least one time to true and one time to false. Assuming no other constraints for the values ‘x’ and ‘y’ in our system model, the Conformiq Designer tool could satisfy this coverage criteria with the following test cases:
- x = -1 and y = 0 causing ‘(x < 0)’ to evaluate to true and ‘(y == 10)’ to false.
- x = 0 and y = 10 causing ‘(x < 0)’ to evaluate to false and ‘(y == 10)’ to true.
The above tests satisfy Condition Coverage, however this test set does not satisfy Decision Coverage since neither case will cause the if statement to evaluate to true.
Decision Coverage
Decision Coverage guides Conformiq Designer to look for behaviors that cover every QML level decision (conditional branch such as then and else cases of an if statement) for all the possible outcomes (a decision is a Boolean expression composed of one or more conditions). In the above example the Decision Coverage would be satisfied by two test cases:
- x = -1 and y = 10 causing both ‘(x < 0)’ and ‘(y == 10)’ to evaluate to true causing if statement to evaluate to true, therefore taking the then branch.
- x = -1 and y = 0 causing ‘(x < 0)’ to evaluate to true and ‘(y == 10)’ to false causing if statement to evaluate to false, therefore taking the else branch.
Note again that the above tests satisfy Decision Coverage however this test set does not satisfy Condition Coverage because there is no test that causes ‘(x < 0)’ to evaluate to false. However the following test set would satisfy both Condition and Decision Coverage criteria, but also Condition/Decision Coverage:
- x = -1 and y = 10 causing both ‘(x < 0)’ and ‘(y == 10)’ to evaluate to true causing if statement to evaluate to true, therefore taking the then branch.
- x = 0 and y = 0 causing both ‘(x < 0)’ and ‘(y == 10)’ to evaluate to false causing if statement to evaluate to false, therefore taking the else branch.
Compound Condition Coverage
Compound Condition Coverage guides Conformiq Designer to look for behaviors that cover every QML level atomic condition branch (such as left and right hand sides of the Boolean and) at least once. Note that because conditional Boolean expressions (that is && and ||) use short-circuit evaluation for Boolean connectives, there are value combinations that cannot be meaningfully tested in general. For example, in the case of x && y, Conformiq Designer will not attempt to generate a test case where ‘x’ would be false and ‘y’ would be true. The reason is that the short-circuit evaluation rule will prevent ‘y’ from being evaluated after it has been found that ‘x’ evaluates to false, which makes it possible that the value of ‘y’ may depend on the assumption that ‘x’ evaluates to true. (Note that ‘x’ and ‘y’ can be both, e.g., method calls.)
Modified Condition/Decision Coverage (MC/DC)
MC/DC extends the Condition/Decision Coverage criteria with a requirement that each condition should affect the decision outcome separately. The idea of MC/DC is that for every atomic Boolean expression inside a compound conditional expression you have one test case that verifies that the value does not incorrectly flip from false to true, and one that verifies that it does not incorrectly flip from true to false. For example, take the expression
(a OR b OR c OR d)
The minimum set of MC/DC test cases would be:
- (false, false, false, false) — if any value flips to true, the result changes from false to true
- (true, false, false, false) — if a value flips from true to false, the overall result changes from true to false
- (false, true, false, false) — if a value flips from true to false, the overall result changes from true to false
- (false, false, true, false) — if a value flips from true to false, the overall result changes from true to false
- (false, false, false, true) — if a value flips from true to false, the overall result changes from true to false
MC/DC coverage criteria is used to guide Conformiq Designer to look for behaviors that cover every QML level Boolean conditional (conditional-and ‘&&’ and conditional-or ‘||’) and logical (logical-and ‘&’ and logical-or ‘|’) operation at least once. For example, assume one has the following QML fragment in a system model:
if (((x < 0) || (y == 10)) && (z > 100)) { … }
Assuming no other constraints for the values ‘x’, ‘y’ and ‘z’ in the system model, the Conformiq Designer tool could satisfy Condition and Decision Coverage (and Condition/Decision Coverage) criteria with the following test cases:
- x = -1, y = 10 and z = 101 causing ‘(x < 0)’, ‘(y == 10)’ and ‘(z > 100)’ all to evaluate to true causing if statement to evaluate to true, therefore taking the then branch.
- x = 0, y = 0 and z = 0 causing ‘(x < 0)’, ‘(y == 10)’ and ‘(z > 100)’ all to evaluate to false causing if statement to evaluate to false, therefore taking the else branch.
However, the above set does not satisfy MC/DC coverage since in the first test the value of condition ‘(y == 10)’ does not affect the outcome of the decision while in the second test the value of ‘(z > 100)’ does not affect the outcome. In order to also satisfy the MC/DC criteria, Conformiq Designer could, for example, select the following set of tests:
- x = -1, y = 0 and z = 101 ⇒ ‘(x < 0)’ = true, ‘(y == 10)’ = false and ‘(z > 100)’ = true ⇒ decision to evaluate to true.
- x = 0, y = 0 and z = 101 ⇒ ‘(x < 0)’ = false, ‘(y == 10)’ = false and ‘(z > 100)’ = true ⇒ decision to evaluate to false.
- x = 0, y = 10 and z = 101 ⇒ ‘(x < 0)’ = false, ‘(y == 10)’ = true and ‘(z > 100)’ = true ⇒ decision to evaluate to true.
- x = 0, y = 10 and z = 0 ⇒ ‘(x < 0)’ = false, ‘(y == 10)’ = true and ‘(z > 100)’ = false ⇒ decision to evaluate to false.
Boundary Value Analysis
Boundary Value Analysis guides Conformiq Designer to look for behaviors that cover the boundary value cases for all the arithmetic comparisons between integer values. Boundary value analysis is a technique to determine tests covering known areas of frequent problems at the boundaries of input ranges. In boundary value analysis, the boundaries partition the input domain and it is assumed that the system partitions the input into a set of domains in which the system’s behavior is similar. Due to this, it is also assumed that if an input from a domain causes an error, then all the inputs of that domain will cause a similar error and if an input from a domain does not cause an error, then all the inputs of that domain will fail to produce an error. Based on these assumptions, Conformiq Designer attempts to cover the structure of the input as follows:
- For every arithmetic comparison ‘x = y’ and ‘x != y’, cover cases ‘x < y’, ‘x = y’, and ‘x > y’.
- For every arithmetic comparison ‘x < y’, cover cases ‘x < y – 1’, ‘x = y – 1’, ‘x = y’, and ‘x > y’.
- For every arithmetic comparison ‘x > y’, cover cases ‘x < y’, ‘x = y’, ‘x = y + 1’, and ‘x > y + 1’.
- For every arithmetic comparison ‘x <= y’, cover cases ‘x <= y – 1’, ‘x = y’, ‘x = y + 1’, and ‘x > y + 1’.
- For every arithmetic comparison ‘x >= y’, cover cases ‘x < y – 1’, ‘x = y – 1’, ‘x = y’, and ‘x >= y + 1’.
The test design heuristics covered in this paper are only a few of those available to users in Conformiq Designer. For a full list of available test design heuristics, consult the Conformiq web site or contact a local Conformiq specialist.