CEA List, France
About Nikolai Kosmatov
Researcher in formal verification and testing.
Advanced Test Coverage Criteria: Specify and Measure, Cover and Unmask
Automatic test data generation (ATG) is a major topic in software engineering. A large amount of research effort has been invested to automate white-box testing. While a wide range of different and sometimes heterogeneous code-coverage criteria have been proposed, testing techniques still lack a generic formalism to describe them all, and available test automation tools usually support only a small subset of them.
This tutorial brings participants to a journey into the world of white-box testing criteria and their automated support. We try to bridge the gap between the coverage criteria supported by state-of-the-art white-box ATG technologies, such as Dynamic Symbolic Execution, and advanced coverage criteria found in the literature. The tutorial is articulated around labels, a recent specication mechanism for test objectives, and their effective support in automated testing tools. Labels are generic enough to specify many common testing criteria, and amenable to ecient automation making it possible to cover the corresponding test objectives and to measure the coverage level of a given test suite. We propose several optimization techniques resulting in an effective support for labels in ATG tools. We also show how a combination of static analysis techniques can be efficiently applied to detect — unmask — infeasible test objectives that are responsible for waste of test generation effort and imprecise coverage measurement. We demonstrate the LTest toolset for test automation with ecient support of labels. Finally, we present a recent extension of labels, called HTOL (Hyperlabel Test Objectives Language), capable to encode even most advanced test coverage criteria (such as variants of MCDC, dataow criteria, non-interference properties, etc.), including hyperproperties. This tutorial is based on a series of recent research and tool implementation efforts (presented at ICST 2014, TAP 2014, ICST 2015, ICST 2017t, ICST 2017r, ICSE 2018, ISOLA 2018).