István Forgács, PhD, was originally a researcher at the Hungarian Academy of Sciences. He published 30+ scientific articles in leading international journals and conference proceedings. He was a speaker at several academic and industrial conferences, co-author of the book “Practical Test Design” and “Agile Testing Foundations, ‘An ISTQB Foundation Level Agile Tester” guide, co-author of the exceptional book ‘Paradigm Shift in Software Testing will be published soon.
He is the co-host of the website https://test-design.org/ that is the only place testers can exercise test design with the help of mutation testing. He is a member of the TA WG and a former member of Agile WG of ISTQB. István is the creator and key contributor of the test-first, codeless test design automation tool Harmony.
How to Build a Test Automation Tool That Works in Agile
Test automation is a must in our agile world. However, in practice test automation usually fails due to time constraints. According to a recent Capgemini survey, only 22% of the test cases are automated within sprints. The cause is clear. Even if most test automation tool vendors state that test cases can be made before the implementation, it’s not true.
Our engaged team developed a new test automation tool Harmony, intending to be a real test first tool. Nowadays lots of such tools are codeless, thus manual testers can make executable test cases without programming knowledge. We extended BDD’s Gherkin language so that our Gherkin++ description can be compiled into executable test code. Here is an example:
WHEN Login name IS Hall AND Password IS 2@A9ih
WHEN Login button IS #pressed
THEN Message IS successful login
I think the example is self-explanatory. The test cases are executed by Selenium.
Ok, very good, the test cases are readable for both the machine and human, let’s try it. The most relevant validation is to use our tool for testing our tool. However, the method failed. Why? Even if we believed that our Gherkin++ description is implementation-independent, it was not true. After implementation, we had to modify the test cases, resulting in double work. On the other hand, when we started making automated test cases after implementation it was too slow. So we experienced why this 22% is a genuine number.
Thinking a lot, we realized that making tests directly from the requirement is the problem. In development, there are intermediate steps such as wireframe, mockup, UML. Test automation also needs some genuine implementation-independent intermediate test description. First, we selected use case testing that is the most widely used test design technique. Our tool became test first, however, this technique is not reliable enough to detect tricky bugs.
We invented a new test design technique, called action-state testing, by which so-called higher-order bugs can be found. Action state testing consists of abstract test steps. A step contains a user action, system response, and the arriving state. During test design, these steps are added one by one.
Harmony supports the tester by displaying the current state transition graph, based on which new steps can be created. When no new actions exist, the tool offers steps to fulfill the test selection that is 1-switch coverage.
The method proved to be very strong. It detected a fifth-order bug that was found by neither the speaker nor by 10 testers trying to find the bugs by exploratory testing. You can make the exercise to find this bug here
To summarize, by applying the tool the tests can be automated in-sprint and almost all bugs can be detected by which bugfix costs can be significantly reduced.