A systematic mapping study of search-based software engineering for software product lines
Introduction
Search Based Software Engineering (SBSE) is an established yet young discipline that focuses on the application of search-based optimization techniques to software engineering problems [1]. In this article, we follow Harman et al. who consider SBSE techniques to primarily include metaheuristic search based optimization techniques and classical operations research techniques [1], [2]. Some examples of SBSE techniques are: evolutionary computation techniques1 (e.g. genetic algorithms), basic local searches (e.g. hill climbing, simulated annealing or random search) [4], and integer programming [1].
Software Product Lines (SPLs) are families of related systems whose members are distinguished by the set of features they provide [5], [6]. Variability is the capacity of software artifacts to vary and its effective management and realization lie at the core of successful SPL development [7]. Feature models are tree-like structures that establish the relations between features and have become the de facto standard for modeling variability [8], [9]. Over the last decade, extensive research and practice both in academia and industry attest to the substantial benefits of applying SPL practices [6], [10], [11]. Among the benefits are better customization, improved software reuse, and faster time to market.
Typical SPLs have a large number of features that are combined in complex feature relations yielding a large number of individual software systems that must be effectively and efficiently designed, implemented and managed. Precisely this fact is what makes SPL-related problems suitable for the application of SBSE techniques which are generic, flexible, robust, and have been shown to scale to large search spaces such as those that typically characterize SPLs (e.g. [12]). This recent realization has sparked a surge of research and application at the intersection of SBSE and SPLs, which has manifested with an increasing number of articles at many of the publication outlets of both research communities. This is precisely what prompted us to perform a systematic mapping study to provide an overview of the research at the intersection of these two fields [13], [14], [15]. In contrast with a systematic literature review whose goal is primarily to identify best practice [13], [15], [16], [17], our general goal was to identify the quantity and the type of research and results available, and thus highlight possible open research problems and opportunities, for both SBSE and SPL communities. More concretely we wanted to identify at what stages of the SPL development life cycle have SBSE techniques been used and which ones. We also wanted to find out the provenance, number and types of artifacts used as case studies as well as how they were empirically analyzed. And finally, which are the fora where the research work was published.
Our study corroborated the increasing and ongoing interest in applying SBSE techniques in SPLs. We found that the most common application is software testing, and the most common techniques are genetic algorithms and multi-objective evolutionary algorithms. Our study identified the need to improve the robustness of the empirical evaluation (e.g. more adequate statistical analysis) and the need for more extensive and robust tool support. We hope that this mapping study not only serves to highlight the main research topics at the intersection of SBSE and SPLs but that it also serves to encourage researchers to pursue work at the intersection of both areas.
The paper is structured as follows. Section 2 presents the process we followed for our systematic mapping study. It details the research questions addressed, how the search was performed, the classification scheme used, and how the data was extracted and analyzed. Section 3 presents the results we obtained for each research question. Section 4 contains a description of the results found along with open questions and avenues worth of further investigation. Section 5 summarizes the threats to validity we identified in our work and how they were addressed. Section 6 concisely describes the existing review studies and surveys of SPLs and SBSE. Section 7 summarizes the conclusions of our study and future work.
Section snippets
Systematic mapping study
Evidence-Based Software Engineering (EBSE) is an emerging software engineering area whose goal is “to provide the means by which current best evidence from research can be integrated with practical experience and human values in the decision making process regarding the development and maintenance of software” [18]. One of the approaches advocated by EBSE is systematic mapping studies whose goal is to provide an overview of the results available within an area by categorizing them along
Results
In this section we present and describe the results obtained in our systematic mapping study. As mentioned in Sections 2.2 Conduct search for primary sources, 2.3 Screening of papers for inclusion and exclusion, we first performed search queries in specialized repositories and search engines. These queries were performed between 31st July 2014 and 14th August 2014 and produced a total of 2630 hits. As a second step we sieved the articles based on the title, abstract and keywords, resulting in
Analysis and discussion
In this section we analyze the findings revealed by our systematic mapping study. For each one, we present an overview of some of the relevant primary sources followed by a concise discussion on open questions and potential areas for further research.
Threats to validity
We faced similar validity threats as any other systematic mapping study. A threat to validity is the selection of the search queries. We addressed this threat with a carefully chosen selection of search terms based on previous studies on the core research areas of our systematic mapping study, SPLs and SBSE. The SPL search terms were collected and aggregated from 12 systematic mapping studies and literature reviews. The SBSE search terms were based on three studies which we extensively
Related work
In this section we briefly summarize the surveys and studies carried out in either SPLs or in SBSE.
SBSE surveys. The survey by Harman et al. presents a general overview of SBSE techniques and the areas where it has been employed [1]. Freitas et al. performed a bibliometric analysis of SBSE [47]. Their goal was to identify trends in the number of publications, the publication fora, the authorship and collaborations among members of the SBSE community. In contrast with our work, they have a
Conclusions and future work
In this paper we present the results of the first systematic mapping study on the application of SBSE techniques to SPL problems. Our study corroborates the increasing interest in applying this type of techniques as shown by the number of recent publications. The most common application is for testing at the Domain Engineering level, for example in computing test suites in Combinatorial Interaction Testing. The most common techniques used are genetic algorithms and multi-objective evolutionary
Acknowledgements
This research is partially funded by the Austrian Science Fund (FWF) Projects P25289-N15, P25513-N15, and Lise Meitner Fellowship M1421-N15. We thank Javier Ferrer, Francisco Chicano, and Enrique Alba for their help with the earlier versions of this work.
References (79)
- et al.
Automated analysis of feature models 20 years later: a literature review
Inf. Syst.
(2010) - et al.
Using mapping studies as the basis for further research – a participant-observer case study
Inform. Softw. Technol.
(2011) - et al.
Search-based software engineering
Inform. Softw. Tech.
(2001) - et al.
Requirements engineering for software product lines: a systematic literature review
Inform. Softw. Technol.
(2010) - et al.
Requirements for product derivation support: results from a systematic literature review and an expert survey
Inform. Softw. Technol.
(2010) - et al.
A systematic mapping study of software product lines testing
Inform. Softw. Technol.
(2011) - et al.
Software product line testing – a systematic mapping study
Inform. Softw. Technol.
(2011) - et al.
A systematic review and an expert survey on capabilities supporting multi product lines
Inform. Softw. Technol.
(2012) - et al.
A systematic mapping study on software product line evolution: from legacy system reengineering to product line refactoring
Sci. Comput. Program.
(2013) - et al.
Combining service-orientation and software product line engineering: a systematic mapping study
Inform. Softw. Technol.
(2013)
Variability in quality attributes of service-based software systems: a systematic literature review
Inform. Softw. Technol.
The next release problem
Inform. Softw. Technol.
Improving the accuracy of coplimo to estimate the payoff of a software product line
Expert Syst. Appl.
Coherent clusters in source code
J. Syst. Softw.
On the reliability of mapping studies in software engineering
J. Syst. Softw.
A systematic review of evaluation of variability management approaches in software product lines
Inform. Softw. Tech.
Search-based software engineering: trends, techniques and applications
ACM Comput. Surv.
Formulating software engineering as a search problem
IEE Proc. – Softw.
Introduction to Evolutionary Computing
Scaling step-wise refinement
IEEE Trans. Softw. Eng.
Software Product Line Engineering: Foundations, Principles and Techniques
A taxonomy of variability realization techniques
Softw. Pract. Exper.
Software Product Lines in Action: The Best Industrial Practice in Product Line Engineering
GPGPU test suite minimisation: search based software engineering performance improvement using graphics cards
Emp. Softw. Eng.
Experimentation in Software Engineering
Evidence-based software engineering
A hitchhiker’s guide to statistical tests for assessing randomized algorithms in software engineering
Softw. Test. Verif. Reliab.
Agile software product lines: a systematic mapping study
Softw. Pract. Exper.
On strategies for testing software product lines: a systematic literature review
Inform. Softw. Technol.
Variability in software systems – a systematic literature review
IEEE Trans. Softw. Eng.
A systematic review of the application and empirical investigation of search-based test case generation
IEEE Trans. Soft. Eng.
Cited by (89)
A Monte Carlo tree search conceptual framework for feature model analyses
2023, Journal of Systems and SoftwareCitation Excerpt :Some statistical analyses require the construction of BDDs (e.g., determining the distribution of the number of features among all valid configurations or testing the uniformity of a random sampler on complex models with thousands of features) (Heradio et al., 2019, 2022a), which can be intractable (Thüm, 2020). Other approaches like genetic algorithms (Lopez-Herrejon et al., 2015b,a) and metaheuristics (Yadav et al., 2020) require to incorporate specific domain knowledge, and analyzing and inferring results from the final solutions which is not straightforward. In addition, these techniques offer little information about the intermediate steps of the analysis process that allows considering other valuable alternatives before obtaining the final solution.
On the use of artificial intelligence to deal with privacy in IoT systems: A systematic literature review
2022, Journal of Systems and SoftwareThe Symposium on Search-Based Software Eengineering: Past, Present and Future
2020, Information and Software TechnologyCitation Excerpt :The latest major survey that covered the whole area was published in 2012 by Harman et al. [2]. We also found multiple papers on specific areas of SBSE, such as Search Based Software Testing (SBST) [3,13,14], Search Based Software Design (SBSD) [6,15], Search Based Software Refactoring (SBSR) [10,16], Search Based Software Project Management [17], and SBSE for Software Product Lines (SPL) [18–20]. Some other literature reviews focus on specific tasks, such as mining repositories using SBSE [21], test data generation [5,22], genetic improvement of software [23], software requirements management [8,24,25], re-engineering of legacy applications into SPL [26], software module clustering [27], software project scheduling [28], cloud migration [29], and test resource allocation [30].
Automated Test Suite Generation for Software Product Lines Based onQuality-Diversity Optimization
2023, ACM Transactions on Software Engineering and MethodologyVariability-aware Behavioural Learning
2023, ACM International Conference Proceeding Series