Hyper-parameter Tuning for Search-based Test Case Generation

Abstract

There are different techniques used to automatically generate adequate test cases for a software program. Among all, search-based test case generation, which is the application of meta-heuristic search for generating test cases, has been studied a lot in the literature, lately. Since, in theory, the performance of meta-heuristic search methods is highly dependent on their hyper-parameters, there is a need to study hyper-parameter tuning in this domain. In this study, we first studied the effectiveness of hyper-parameter tuning for search-based test generation exhaustively analyze 1,200 hyper-parameter configurations of a well-known search-based test generation tool for Java (EvoSuite). Finding out that some Java classes do not respond to changing hyper-parameters in terms of coverage, we were motivated to allocate tuning budget to a subset of classes rather than the entire set. We proposed a new metric (Tuning Gain''), which estimates how cost-effective tuning a particular class is. We then predict Tuning Gain'' using static features of source code classes. Finally, we prioritize classes for tuning, based on the estimated ``Tuning Gains'' and spend the tuning budget only on the highly-ranked classes. To evaluate our approach, we analyzed 250 classes of 19 projects from benchmarks such as SF110 and SBST 2018 tool competition. We used a tuning approach called Meta-GA and compared the tuning results with and without the proposed class prioritization. The results show that for a low tuning budget, prioritizing classes outperforms the alternatives in terms of extra covered branches (10 times more than a traditional global tuning). Moreover, we tried different approaches like changing subset size, shrinking search space size and giving high tuning budgets to examine the potentials of our tuning method.