Test Automation

Combinatorial Test Data Generation with Conformiq

Suppose a system model states that when a message comes in, it is forwarded out unchanged. This particular message has a number of fields, some of them integers, some strings. For some reason there is cause to suspect that the forwarding feature in the real implementation might be flawed, so we would like to have a number of different message ...

Distributed Test Generation

As detailed in a blog post I wrote a couple of months ago, the core of Conformiq DesignerTM is a custom crafted  semantics driven, symbolic state space exploration  algorithm for test generation from system models (because this is really the only known solution that robustly generates both test inputs and outputs from a system model without user ...

Selecting the Right MBT Tool: Key Factors

Interest towards model-based testing has increased quite significantly over the years as people have started to reach limits of traditional approaches and at the same time started to see and understand the benefits that applying MBT can have to the quality assurance function. In this blog post, I’m outlining what is really important when you are ...

Test Generation Performance: Optimization

Test generation from system models is computationally very hard: Just generating input sequences that cover all the statements of a system model is theoretically an undecidable problem, meaning it can be never solved completely. This does not mean there couldn’t be an algorithm that handles most of the industrially relevant problem instances, but ...

Extended String Testing with RegEx

Quite recently we added a support for expressing ”regular expressions” in the Conformiq models. As this is quite an interesting topic, I’m detailing this feature a bit in this post. (more…)

MBT Users Report Triple Benefit & 59% Bug Reduction

According to the recent model-based testing user survey (see here), the respondents saw on the average a 59% reduction in escaped bugs, 17% reduction in testing costs, and 25% reduction in testing time. What do these figures mean? Let's say that a (software) product has 100 defects (that can manifest themselves) when it enters the testing ...

Conformiq CTO Presents Automated AUTOSAR Testing

Dr. Stephan Schulz, Conformiq's CTO, will give a presentation on "Automating AUTOSAR Functional Test Design with Model-Based Testing" at the SAE 2012 World Congress, to be held in Detroit, USA in April 2012. The SAE 2012 World Congress theme, Get Connected, is said to represent the new and diverse connections that will drive significant ...

Understanding Pairwise Test Generation

Combinatorial test data generators generate data tables for testing. The most basic, commonly used combinatorial data generation strategy is what is known as pairwise testing, all-pairs testing, covering arrays or Taguchi designs. (The term orthogonal array is sometimes used also, but it actually refers to a method for designing statistical ...

Testing Economics: Why Non-Defect Tests Matter

Let's assume (this obviously a much simplified setting) that you are planning to release a new product version that will generate $1mn in sales. There may or may not be one critical bug left in the version. If there actually is a critical bug, this will cost to you in terms of product replacements, customer supports and customer disloyalty total ...

Michael Mlynarski: Testing ROI Challenges

On his model-based testing blog, Michael Mlynarski writes: In this session we have discussed the ROI of software testing techniques. More than 20 experienced testers or test managers had difficulties on formulating the ROI of any testing technique. Two main arguments were about reducing the costs for bug-fixes in late stages of the development ...