Model-based testing saves you lots of time and money. It’s true, and believe me – I’m deeply convinced about that. Model-based testing helps you generate better tests faster, with better quality. Something else the test manager must be aware of: it’s fun to create models! Who would not prefer to create models rather than create tests manually? Someone who is having fun at work will always be more productive. And I have experienced that myself when my boss told me: “you should take a look at model-based testing” at the beginning of my career. Now I cannot imagine going back and creating tests manually.
In my work I review models coming from many different companies and application domains. And sometimes I see a problem especially with highly motivated people: models are starting to become really complicated. I think this is because they may spend too much time on modeling details that finally may not be really necessary to model. So when should you stop modeling?
I think it’s a matter of black box versus white box testing. The literature talks a lot about this, but the reality is: sometimes it’s easy to be drawn from black box into white box testing. In particular, it becomes tempting when you have a powerful test generation engine and language. You may wind up doing white box testing without even being aware of it.
Let me tell you a story about one case I recently encountered in enterprise IT. Someone had specified a really good model around testing a particular menu for a web application. In one part of this model, he wanted to model a pop-up menu. In other words: “when my mouse moves on the menu, the menu must appear, and when my mouse goes out of the menu, the menu must disappear.” For reasons which are unknown to us, the user did not use a pop-up menu in the model (even though it was available) but instead modeled it literally in this detail: “when my mouse moves on the menu, the menu must appear, when my mouse moves out from the bottom, left and right the menu must disappear” (and he forgot to specify the menu should also disappear when the mouse moves out from the top.) From such a model, a test generation engine will create special tests for all the particular cases. So he got four test cases to test the menu, with a funny one: “when my mouse goes from the top the menu must still be open.” Imagine you are a tester who just got these tests for manual execution. Actually the tester will not be able to tell the tests were automatically generated. These four tests are not really “interesting” or efficient – and one even looks wrong! Never forget that a model-based testing tool is a machine, and if you model at this level of detail, it will generate your tests and exploit that level of detail. A test generation engine will always assume that everything in a model is intended behavior and generate tests for all of it, it cannot judge what is correct, or more or less important, unless you (the human user) specify it.
Taking a step back in this case, the user was able to model even such detail as mouse movements – but how long did he spend to model this menu behavior, which did not include the right behavior for functional testing? How much extra time did he spend figuring out why he got this funny test, and where and how to fix it? What did he gain from spending this time? What is the chance that the application under test will fail these detailed tests? What is the value? When you model the functionality of your menu at this level of detail, you are doing more “white box” instead of “black box” testing. There is a big difference in what you can model and what you should model. Remember that model-based testing is most effective when used for black box testing. If you are modeling detail like this, then it should be stated at this level in your specification.
To conclude, this was just one example; there are many others. When you are entering details of functionality, always question yourself: Is it really relevant for my functionality? Is it still black box testing? At some point, it’s more a matter of: How long will I spend to create, correct, and maintain models at this level of detail and what is the real value I will get back from that? As I stated before generally, we do not model like this by intent – so once you notice you are spending “too much time” on getting tests to look better, take a step back and ask yourself: Am I making something more complicated than it really is?
Alexis Despeyroux is a Conformiq technical field application engineering specialist.