Tackling Huge System Specs in Model-Based Testing

Conformiq’s system model based testing is based on the premise that you can create a (computer-readable) model of the intended behavior of the system under test. But what if the system is really big? This is a question that people often ask from us, and there are typically three concerns: (1) How long will it take to create the model? (2) Can our engineers handle its complexity? (3) Does Conformiq’s tooling scale to huge models? Read on for some answers.

If you test a large system using conventional methods (read: manual test design), you divide the challenge into chunks of manageable size. For example, different functions can be tested first separately, leaving only feature/function interaction testing to be carried out later. Internal components can be tested thoroughly first (unit or component testing), and different components can be tested by different teams. The same pattern applies to model-based testing of large systems for two main reasons:

• For large systems, creating, reviewing and maintaining a holistic, multi-component and multi-level system model becomes a daunting and unmanageable task for humans. In the same way as a single engineer cannot fully comprehend every detail of the implementation of a large system, a single testing team cannot create a model that covers every detail of the behavior of a large system under test because it ends up as a too heavy cognitive burden.
• Advanced model-based testing tools such as Conformiq Designer do not scale well to very large models because some of the employed algorithms have high time complexity curves.

Another question is how long it takes to create a complex model in Conformiq Designer, and of course it takes more time to create larger models, but an easily missed point is that models can be created incrementally. It is as a matter of fact a very common pattern that people create first simple models that describe only the positive aspects of some of the main functions, and then grow the model to include more variation of behavior and more information about negative use cases (error and exception handling cases). A second dimension of incrementality is that system model driven test generation tolerates specification changes well, making Conformiq’s model-based testing technology a good mate for agile and test driven development methods.

So, in summary:

• Large systems are handled by dividing the testing task into subtasks, both because of the limitations of human cognition as well as because of the scalability behavior of advanced model-based testing algorithms.
• Models are usually create incrementally, both on the axis of growing scope as well as on the axis of changing requirements.