A systematic mapping study of search-based software engineering for software product lines

https://doi.org/10.1016/j.infsof.2015.01.008Get rights and content

Abstract

Context

Search-Based Software Engineering (SBSE) is an emerging discipline that focuses on the application of search-based optimization techniques to software engineering problems. Software Product Lines (SPLs) are families of related software systems whose members are distinguished by the set of features each one provides. SPL development practices have proven benefits such as improved software reuse, better customization, and faster time to market. A typical SPL usually involves a large number of systems and features, a fact that makes them attractive for the application of SBSE techniques which are able to tackle problems that involve large search spaces.

Objective

The main objective of our work is to identify the quantity and the type of research on the application of SBSE techniques to SPL problems. More concretely, the SBSE techniques that have been used and at what stage of the SPL life cycle, the type of case studies employed and their empirical analysis, and the fora where the research has been published.

Method

A systematic mapping study was conducted with five research questions and assessed 77 publications from 2001, when the term SBSE was coined, until 2014.

Results

The most common application of SBSE techniques found was testing followed by product configuration, with genetic algorithms and multi-objective evolutionary algorithms being the two most commonly used techniques. Our study identified the need to improve the robustness of the empirical evaluation of existing research, a lack of extensive and robust tool support, and multiple avenues worthy of further investigation.

Conclusions

Our study attested the great synergy existing between both fields, corroborated the increasing and ongoing interest in research on the subject, and revealed challenging open research questions.

Introduction

Search Based Software Engineering (SBSE) is an established yet young discipline that focuses on the application of search-based optimization techniques to software engineering problems [1]. In this article, we follow Harman et al. who consider SBSE techniques to primarily include metaheuristic search based optimization techniques and classical operations research techniques [1], [2]. Some examples of SBSE techniques are: evolutionary computation techniques1 (e.g. genetic algorithms), basic local searches (e.g. hill climbing, simulated annealing or random search) [4], and integer programming [1].

Software Product Lines (SPLs) are families of related systems whose members are distinguished by the set of features they provide [5], [6]. Variability is the capacity of software artifacts to vary and its effective management and realization lie at the core of successful SPL development [7]. Feature models are tree-like structures that establish the relations between features and have become the de facto standard for modeling variability [8], [9]. Over the last decade, extensive research and practice both in academia and industry attest to the substantial benefits of applying SPL practices [6], [10], [11]. Among the benefits are better customization, improved software reuse, and faster time to market.

Typical SPLs have a large number of features that are combined in complex feature relations yielding a large number of individual software systems that must be effectively and efficiently designed, implemented and managed. Precisely this fact is what makes SPL-related problems suitable for the application of SBSE techniques which are generic, flexible, robust, and have been shown to scale to large search spaces such as those that typically characterize SPLs (e.g. [12]). This recent realization has sparked a surge of research and application at the intersection of SBSE and SPLs, which has manifested with an increasing number of articles at many of the publication outlets of both research communities. This is precisely what prompted us to perform a systematic mapping study to provide an overview of the research at the intersection of these two fields [13], [14], [15]. In contrast with a systematic literature review whose goal is primarily to identify best practice [13], [15], [16], [17], our general goal was to identify the quantity and the type of research and results available, and thus highlight possible open research problems and opportunities, for both SBSE and SPL communities. More concretely we wanted to identify at what stages of the SPL development life cycle have SBSE techniques been used and which ones. We also wanted to find out the provenance, number and types of artifacts used as case studies as well as how they were empirically analyzed. And finally, which are the fora where the research work was published.

Our study corroborated the increasing and ongoing interest in applying SBSE techniques in SPLs. We found that the most common application is software testing, and the most common techniques are genetic algorithms and multi-objective evolutionary algorithms. Our study identified the need to improve the robustness of the empirical evaluation (e.g. more adequate statistical analysis) and the need for more extensive and robust tool support. We hope that this mapping study not only serves to highlight the main research topics at the intersection of SBSE and SPLs but that it also serves to encourage researchers to pursue work at the intersection of both areas.

The paper is structured as follows. Section 2 presents the process we followed for our systematic mapping study. It details the research questions addressed, how the search was performed, the classification scheme used, and how the data was extracted and analyzed. Section 3 presents the results we obtained for each research question. Section 4 contains a description of the results found along with open questions and avenues worth of further investigation. Section 5 summarizes the threats to validity we identified in our work and how they were addressed. Section 6 concisely describes the existing review studies and surveys of SPLs and SBSE. Section 7 summarizes the conclusions of our study and future work.

Section snippets

Systematic mapping study

Evidence-Based Software Engineering (EBSE) is an emerging software engineering area whose goal is “to provide the means by which current best evidence from research can be integrated with practical experience and human values in the decision making process regarding the development and maintenance of software” [18]. One of the approaches advocated by EBSE is systematic mapping studies whose goal is to provide an overview of the results available within an area by categorizing them along

Results

In this section we present and describe the results obtained in our systematic mapping study. As mentioned in Sections 2.2 Conduct search for primary sources, 2.3 Screening of papers for inclusion and exclusion, we first performed search queries in specialized repositories and search engines. These queries were performed between 31st July 2014 and 14th August 2014 and produced a total of 2630 hits. As a second step we sieved the articles based on the title, abstract and keywords, resulting in

Analysis and discussion

In this section we analyze the findings revealed by our systematic mapping study. For each one, we present an overview of some of the relevant primary sources followed by a concise discussion on open questions and potential areas for further research.

Threats to validity

We faced similar validity threats as any other systematic mapping study. A threat to validity is the selection of the search queries. We addressed this threat with a carefully chosen selection of search terms based on previous studies on the core research areas of our systematic mapping study, SPLs and SBSE. The SPL search terms were collected and aggregated from 12 systematic mapping studies and literature reviews. The SBSE search terms were based on three studies which we extensively

Related work

In this section we briefly summarize the surveys and studies carried out in either SPLs or in SBSE.

SBSE surveys. The survey by Harman et al. presents a general overview of SBSE techniques and the areas where it has been employed [1]. Freitas et al. performed a bibliometric analysis of SBSE [47]. Their goal was to identify trends in the number of publications, the publication fora, the authorship and collaborations among members of the SBSE community. In contrast with our work, they have a

Conclusions and future work

In this paper we present the results of the first systematic mapping study on the application of SBSE techniques to SPL problems. Our study corroborates the increasing interest in applying this type of techniques as shown by the number of recent publications. The most common application is for testing at the Domain Engineering level, for example in computing test suites in Combinatorial Interaction Testing. The most common techniques used are genetic algorithms and multi-objective evolutionary

Acknowledgements

This research is partially funded by the Austrian Science Fund (FWF) Projects P25289-N15, P25513-N15, and Lise Meitner Fellowship M1421-N15. We thank Javier Ferrer, Francisco Chicano, and Enrique Alba for their help with the earlier versions of this work.

References (79)

  • S. Mahdavi-Hezavehi et al.

    Variability in quality attributes of service-based software systems: a systematic literature review

    Inform. Softw. Technol.

    (2013)
  • A.J. Bagnall et al.

    The next release problem

    Inform. Softw. Technol.

    (2001)
  • R. Heradio et al.

    Improving the accuracy of coplimo to estimate the payoff of a software product line

    Expert Syst. Appl.

    (2012)
  • S.S. Islam et al.

    Coherent clusters in source code

    J. Syst. Softw.

    (2014)
  • C. Wohlin et al.

    On the reliability of mapping studies in software engineering

    J. Syst. Softw.

    (2013)
  • L. Chen et al.

    A systematic review of evaluation of variability management approaches in software product lines

    Inform. Softw. Tech.

    (2011)
  • M. Harman et al.

    Search-based software engineering: trends, techniques and applications

    ACM Comput. Surv.

    (2012)
  • J.A. Clark et al.

    Formulating software engineering as a search problem

    IEE Proc. – Softw.

    (2003)
  • A. Eiben et al.

    Introduction to Evolutionary Computing

    (2003)
  • S. Luke, Essentials of Metaheuristics, Lulu, 2009...
  • D.S. Batory et al.

    Scaling step-wise refinement

    IEEE Trans. Softw. Eng.

    (2004)
  • K. Pohl et al.

    Software Product Line Engineering: Foundations, Principles and Techniques

    (2005)
  • M. Svahnberg et al.

    A taxonomy of variability realization techniques

    Softw. Pract. Exper.

    (2005)
  • K. Kang, S. Cohen, J. Hess, W. Novak, A. Peterson, Feature-Oriented Domain Analysis (FODA) Feasibility Study, Tech....
  • F.J. van d Linden et al.

    Software Product Lines in Action: The Best Industrial Practice in Product Line Engineering

    (2007)
  • S. Yoo et al.

    GPGPU test suite minimisation: search based software engineering performance improvement using graphics cards

    Emp. Softw. Eng.

    (2013)
  • B. Kitchenham, S. Charters, Guidelines for performing systematic literature reviews in software engineering. version...
  • K. Petersen, R. Feldt, S. Mujtaba, M. Mattsson, Systematic mapping studies in software engineering, in: EASE, British...
  • D. Budgen, M. Turner, P. Brereton, B. Kitchenham, Using mapping studies in software engineering, in: Proceedings of...
  • C. Wohlin et al.

    Experimentation in Software Engineering

    (2012)
  • B. Kitchenham et al.

    Evidence-based software engineering

  • A. Arcuri et al.

    A hitchhiker’s guide to statistical tests for assessing randomized algorithms in software engineering

    Softw. Test. Verif. Reliab.

    (2014)
  • J. Ferreira Bastos, P. Anselmo da Mota Silveira Neto, E. Santana de Almeida, S. Romero de Lemos Meira, Adopting...
  • I.F. da Silva et al.

    Agile software product lines: a systematic mapping study

    Softw. Pract. Exper.

    (2011)
  • I. do Carmo Machado et al.

    On strategies for testing software product lines: a systematic literature review

    Inform. Softw. Technol.

    (2014)
  • M. Galster et al.

    Variability in software systems – a systematic literature review

    IEEE Trans. Softw. Eng.

    (2014)
  • W. Afzal, R. Torkar, R. Feldt, A systematic mapping study on non-functional search-based software testing, in:...
  • S. Ali et al.

    A systematic review of the application and empirical investigation of search-based test case generation

    IEEE Trans. Soft. Eng.

    (2010)
  • Cited by (89)

    • A Monte Carlo tree search conceptual framework for feature model analyses

      2023, Journal of Systems and Software
      Citation Excerpt :

      Some statistical analyses require the construction of BDDs (e.g., determining the distribution of the number of features among all valid configurations or testing the uniformity of a random sampler on complex models with thousands of features) (Heradio et al., 2019, 2022a), which can be intractable (Thüm, 2020). Other approaches like genetic algorithms (Lopez-Herrejon et al., 2015b,a) and metaheuristics (Yadav et al., 2020) require to incorporate specific domain knowledge, and analyzing and inferring results from the final solutions which is not straightforward. In addition, these techniques offer little information about the intermediate steps of the analysis process that allows considering other valuable alternatives before obtaining the final solution.

    • The Symposium on Search-Based Software Eengineering: Past, Present and Future

      2020, Information and Software Technology
      Citation Excerpt :

      The latest major survey that covered the whole area was published in 2012 by Harman et al. [2]. We also found multiple papers on specific areas of SBSE, such as Search Based Software Testing (SBST) [3,13,14], Search Based Software Design (SBSD) [6,15], Search Based Software Refactoring (SBSR) [10,16], Search Based Software Project Management [17], and SBSE for Software Product Lines (SPL) [18–20]. Some other literature reviews focus on specific tasks, such as mining repositories using SBSE [21], test data generation [5,22], genetic improvement of software [23], software requirements management [8,24,25], re-engineering of legacy applications into SPL [26], software module clustering [27], software project scheduling [28], cloud migration [29], and test resource allocation [30].

    • Variability-aware Behavioural Learning

      2023, ACM International Conference Proceeding Series
    View all citing articles on Scopus
    View full text