GEML: A grammar-based evolutionary machine learning approach for design-pattern detection

https://doi.org/10.1016/j.jss.2021.110919Get rights and content

Highlights

  • Grammar-guided design pattern detection method based on evolutionary machine learning.

  • Readable detection rules are derived by a configurable context-free grammar.

  • Categorical and numerical code properties are simultaneously supported for detection.

  • One single configuration can outperform other learning proposals and reference tools.

  • GEML is publicly available as a demonstration tool, offering practical application.

Abstract

Design patterns (DPs) are recognised as a good practice in software development. However, the lack of appropriate documentation often hampers traceability, and their benefits are blurred among thousands of lines of code. Automatic methods for DP detection have become relevant but are usually based on the rigid analysis of either software metrics or specific properties of the source code. We propose GEML, a novel detection approach based on evolutionary machine learning using software properties of diverse nature. Firstly, GEML makes use of an evolutionary algorithm to extract those characteristics that better describe the DP, formulated in terms of human-readable rules, whose syntax is conformant with a context-free grammar. Secondly, a rule-based classifier is built to predict whether new code contains a hidden DP implementation. GEML has been validated over five DPs taken from a public repository recurrently adopted by machine learning studies. Then, we increase this number up to 15 diverse DPs, showing its effectiveness and robustness in terms of detection capability. An initial parameter study served to tune a parameter setup whose performance guarantees the general applicability of this approach without the need to adjust complex parameters to a specific pattern. Finally, a demonstration tool is also provided.

Introduction

Design patterns (DPs) are reusable template solutions that address recurrent software-design problems. The adoption of DPs is a best practice for programmers with the goal of improving the quality of software products — in terms of their maintainability, elegance, flexibility and understandability (Gamma et al., 1995). Given such benefits, and considering that manual inspection is an error-prone and time-consuming process, automatic Design Pattern Detection (DPD) has become a prominent area in the reverse-engineering research field (Bafandeh Mayvan et al., 2017). By automatically identifying the adoption of DPs, DPD techniques can improve the understanding of the design decisions in software systems, as well as the processes of redocumenting them, reimplementing them, and reusing them.

Different techniques have been proposed in the research literature to automate DPD, most of which search for particular structures in static code (Mayvan and Rasoolzadegan, 2017). In this case, the code structures defining DPs need to be predetermined by experts into a knowledge-base, as they usually are specific to the codebase of study. Having the expert defined these code structures may impose rigidity to the detection technique, and design patterns should be reinterpreted for particular contexts.

To reduce this limitation, machine learning (ML) techniques were proposed for DPD, e.g., Ferenc et al. (2005). These techniques are more easily adapted to different codebases because they learn from a collection of representative examples — and thus can recognise diverse implementations by simply replacing or extending such analysed collection of examples. In particular, ML-based approaches provide mechanisms to learn the structural and behavioural properties of the source code, as well as software metrics, that best describe the DP. Even so, despite the fact that DPs can be described considering these multiple aspects, current ML approaches consider either software metrics or code properties — structural, behavioural or both.

Unfortunately, the detection performance of existing ML-based approaches for DPD is affected by the design pattern under analysis, often requiring its specific parameter configuration. This characteristic makes them harder to adopt in practice: tuning parameters requires considerable effort by software engineers, since machine learning is often not their main area of expertise. Furthermore, the results provided by certain ML-based techniques like neural networks are difficult to understand and interpret by the human expert (Mori and Uchihira, 2019), thus making their recommendations less likely to be trusted (Dzindolet et al., 2003).

In this paper, we present GEML, a novel machine-learning-based approach for DPD from static code. More specifically, GEML is founded on associative classification (AC), a ML technique for which we have implemented a solution based on grammar-guided genetic programming (G3P). Our goal is to achieve the benefits of ML-based proposals while also addressing their limitations by using G3P as a basis of our approach. Like other ML-based proposals, GEML learns from DP examples, giving it the ability to capture diverse DP implementations. As for the limitations of previous ML proposals, GEML has been designed to promote extensibility, readability and flexibility.

For extensibility, GEML includes a customisable collection of design microstructures whose potential to identify DP instances is automatically determined by the evolutionary algorithm during the learning phase. For readability, GEML builds a rule-based classifier guided by a configurable context-free grammar that declares the syntax of the rules. Rules, as a well-established mechanism to encode human knowledge (Grosan and Abraham, 2011), describe the distinctive characteristics of DP instances in a more comprehensible way than the outcomes produced by black-box models (Kotsiantis et al., 2006). And for flexibility, GEML is applicable to each DP without requiring different algorithms or parameter configurations, since it searches for optimal rules to describe each particular DP.

In addition to these benefits, in our experiments, GEML also provided accuracy levels that outperformed other available techniques, providing competitive results even when very few samples are provided. Furthermore, GEML is able to maintain a stable behaviour with just a single general configuration, which we believe will make it more beneficial for software engineers in practical settings.

In short, GEML allows developers to execute it out-of-the-box without having to configure it for each DP, and customise its elements and parameters to gain detection power in particular scenarios. However, these desirable qualities would only make GEML beneficial in practice if it also provided (reasonably) similar performance as other existing techniques. Thus, we focus our evaluation on studying the effectiveness provided by GEML.

We perform several experiments in our evaluation. Firstly, we study GEML’s detection performance in depth through a sensitivity analysis. Although these kinds of analysis are laborious, they become essential to comprehend the internals of artificial intelligence techniques like those applied here. In particular, we aim to understand the extent to which each algorithmic component and its configuration contributed to detecting each individual DP. In this way we are able to determine the best detection capability offered by GEML. We also study the selection options and effectiveness of GEML if its configuration was customised in terms of the design microstructures eligible by the software engineer and software metrics – called operators – that describe the properties that best characterise each DP. Then, we also experimented with the alternative scenario in which software engineers did have the knowledge and time to customise it separately for each DP. As additional benefits of customising GEML individually for each DP, we expect that practitioners would also observe lower runtimes. Finally, we measure the detection performance of the general configuration of GEML. We anticipate that this would be the most common (and simpler) usage scenario for practitioners.

We also analyse how GEML behaves when the general configuration is set but the training conditions change. In this case, we consider up to 15 DPs – the highest number of DPs studied from a ML perspective – at the cost of reducing the number of available training samples. Even so, our results show that GEML is able to infer detection rules in all cases, not requiring any adaptation regarding its internal components and configuration. We also compare GEML’s detection performance to other DPD methods, including both ML and non-ML techniques. GEML provided higher accuracy and F1 score than MARPLE (Zanoni et al., 2015) – a well-known ML-based approach – for four out of the five DPs available for comparison (with up to 35% improvement in F1 for one of them). Against other non-ML-based techniques, GEML also recover more DP instances when validating with JHotDraw, a frequently studied project in DPD literature. Lastly, we analyse the strengths and weaknesses of GEML with respect to SSA and Ptidej, the two reference DPD tools most frequently used for comparative purposes. In this sense, GEML is highly competitive since it correctly identifies a higher number of DP implementations and support DPs whose detection is not available in these tools. GEML also overcomes some limitations of these tools, such as the absence of the classes implementing some roles of the DP and the excessive number of false positives. In short, the results of our evaluation show that GEML is a practical approach for DPD: it improves the effectiveness of current methods, while returning readable outcomes. Furthermore, the possibility of choosing the code properties more relevant for learning brings flexibility to the DPD process.

This paper provides the following contributions:

  • a novel DPD technique (GEML) based on machine learning with grammar-guided genetic programming that is able to provide a single configuration for detecting 15 DPs and returning human-readable rules;

  • the provision of a collection of design microstructures and metrics, in terms of operators that are derived by thecontext-free grammar (CFG) guiding the G3P algorithm and defined with both categorical and numerical values;

  • an analysis of which design microstructures and software metrics best represent each DP.

  • a experimental evaluation of GEML under different training and configuration scenarios, finding highly competitive results when compared against other ML and non-ML approaches;

  • a research demonstration tool to support software engineers in executing GEML and customising it for individual DPs.

The rest of the paper is organised as follows. Main concepts and terminology related to the applied techniques are introduced in Section 2. Section 3 presents the related work, and Section 4 provides a detailed description of our DPD model. Section 5 details the experimental methodology and framework. Then, the three experiments conducted are discussed in Sections 6 Experiment 1: Validation of the detection model, 7 Experiment 2: Comparison against DPD methods using P-Mart, 8 Experiment 3: Analysis of applicability. Section 9 describes the demonstration tool provided as additional material supporting this approach. Finally, threats to validity and concluding remarks are presented in Sections 10 Threats to validity, 11 Concluding remarks, respectively.

Section snippets

Theoretical background

Design patterns are descriptions of communicating objects and classes that are customised to solve a general design problem in a particular context (Gamma et al., 1995). They differ in the number and purpose of their defining roles, each one describing a specific task to be performed. Consequently, a DPD method does not only identify the code elements implementing the design pattern, but also the roles they play within the pattern structure. Notice that the definition of these elements and how

Related work

Multiple techniques have been proposed for automatic detection of structural, behavioural and creational design patterns, using both static and dynamic analysis of source code. Also, both types of techniques can be combined together depending on the pattern to be detected, e.g., by inspecting both class definitions and object collaborations (Ng et al., 2010, Lucia et al., 2018). Given that GEML performs static analysis, we mostly focused on these approaches for DP detection, with special

The GEML approach

Fig. 1 depicts the overall structure of GEML. Taking the approach as a black-box, the inputs required consist in the organisational repository containing patterns predicted in the past (i.e., samples), and the source code for which the detection process is carried out. After the execution of GEML, the detected DPs are returned. These patterns can be incorporated to the organisational repository for future detections, so that the detection model is progressively adapted to the specific

Experimental settings

We first pose the research questions that set out the objectives to be validated in this section. Then we present the data repositories and explain the methodology followed in the experiments. The last section details the experimental framework.

Experiment 1: Validation of the detection model

Experiment #1 is explained in this section. The internal elements of the detection model are analysed by means of an extensive parameter study (RQ1). Then, we also focus on the selection of the grammar operators, revealing the most recurrent microstructures and design elements to describe each DP (RQ2). Finally, the results are discussed with special emphasis on the effectiveness of the proposal (RQ3).

Experiment 2: Comparison against DPD methods using P-Mart

This section presents the results and analysis of Experiment #2, in which GEML is compared to other DPD studies considering a project available in P-Mart, JHotDraw, as the testing project. We provide results from 10 out of the 11 Gamma’s DPs available in this project, since the rest of P-Mart projects cannot provide training instances for the Prototype pattern. Three recent DPD methods are chosen for comparison: DePATOS, which detects structural patterns using a sub-graph isomorphism algorithm (

Experiment 3: Analysis of applicability

Experiment #3 is explained in this section on the basis of a practical scenario, and evaluated according to more qualitative aspects. The experimentation has been performed with a large number of DPs (15), showing that no adaptation is required to execute GEML when new DPs are introduced. The outcomes serve us to analyse how the change of the training repository might influence the behaviour of the proposed method (RQ4). Then, a comparison against reference non-ML-based DPD tools is provided

Demonstration tool

GEML is publicly available as a Java-based demonstration tool (see Additional Material) that allows engineers to detect DP implementations from their own projects without requiring any expertise in ML or evolutionary techniques. The tool provides basic graphical support for the whole DPD process, divided into the following three phases:

  • 1.

    Generation of candidates. The source code is analysed to extract an initial set of potential DP implementations (candidates).

  • 2.

    Learning of the detection model. The

Threats to validity

Internal threats are those related to aspects of the experimentation that cannot ensure the causality of the obtained results. Here, the stochastic nature of the algorithm, as well as its setup and parametrisation are internal threats to be considered. Therefore, all experiments are based on 30 independent executions. Furthermore, a parameter study was conducted to determine the best values. As for the construction of the detection model, a stratified 10-fold cross validation is performed to

Concluding remarks

GEML was introduced as a novel automatic approach for design pattern detection based on evolutionary machine learning. Knowledge from code repositories is extracted by means of G3P4DPD in form of association rules, a highly readable format to represent knowledge (Grosan and Abraham, 2011). The use of an extendable context-free grammar to declare the syntax of rules makes the learning process highly flexible and adaptable to new organisational environments and design patterns. The application of

Additional material

For replicability purposes, experimentation data, such as the generated DP instances, are available for download, as well as the results from the experimentation and statistical analysis, and the demonstration tool, from https://www.uco.es/kdis/sbse/geml/

CRediT authorship contribution statement

Rafael Barbudo: Conceptualization, Methodology, Software, Formal analysis, Software, Writing - original draft. Aurora Ramírez: Conceptualization, Methodology, Formal analysis, Writing - original draft. Francisco Servant: Conceptualization, Methodology, Writing - original draft. José Raúl Romero: Conceptualization, Methodology, Validation, Writing - original draft, Supervision.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

This work was supported by the Spanish Ministry of Economy and Competitiveness [project TIN2017-83445-P], the Spanish Ministry of Education under the FPU program [grant FPU17/00799] and the University of Córdoba, Spain [postdoctoral grant “Plan propio - mod. 2.4”].

Rafael Barbudo is a Ph.D. student of Computer Science at the University of Córdoba, Spain. He received both his B.S. and M.S. in Information and Computer Science at the University of Córdoba. His research interests are related to the application of artificial intelligence techniques to solve real world problems. In this vein, during the M.S. years, his research focused on the application of machine learning techniques to address software engineering problems, especially those concerning the

References (60)

  • Agrawal, R., Imieliński, T., Swami, A., 1993. Mining association rules between sets of items in large databases. In:...
  • Al-ObeidallahM.G. et al.

    A multiple phases approach for design patterns recovery based on structural and method signature features

    Int. J. Softw. Innov.

    (2018)
  • AlhusainS. et al.

    Towards machine learning based design pattern recognition

  • ArcuriA. et al.

    A hitchhiker’s guide to statistical tests for assessing randomized algorithms in software engineering

    Softw. Test. Verif. Reliab.

    (2014)
  • Arevalo, G., Buchli, F., Nierstrasz, O., 2004. Detecting implicit collaboration patterns. In: Proc. 11th Working...
  • BernardiM.L. et al.

    Design pattern detection using a DSL-driven graph matching approach

    J. Softw. Evol. Process

    (2014)
  • BinunA. et al.

    DPJF - Design pattern detection with high accuracy

  • ChidamberS.R. et al.

    A metrics suite for object oriented design

    IEEE Trans. Softw. Eng.

    (1994)
  • CordellaL.P. et al.

    A (sub)graph isomorphism algorithm for matching large graphs

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2004)
  • Dabain, H., Manzer, A., Tzerpos, V., 2015. Design pattern detection using FINDER. In: Proc. 30th Ann. ACM Symposium on...
  • Di MartinoB. et al.

    A rule-based procedure for automatic recognition of design patterns in UML diagrams

    Softw. Pract. Exper.

    (2016)
  • DongJ. et al.

    A matrix-based approach to recovering design patterns

    IEEE Trans. Syst., Man, Cybern. A, Syst. Humans

    (2009)
  • DwivediA.K. et al.

    Software design pattern mining using classification-based techniques

    Front. Comput. Sci.

    (2018)
  • EibenA.E. et al.

    Introduction to Evolutionary Computing

    (2015)
  • Ferenc, R., Beszedes, A., Fulop, L., Lele, J., 2005. Design pattern mining enhanced by machine learning. In: Proc. 21st...
  • Fontana, F., Caracciolo, A., Zanoni, M., 2012. DPB: A benchmark for design pattern detection tools. In: Proc. 16th...
  • FontanaF.A. et al.

    Using design pattern clues to improve the precision of design pattern detection tools

    J. Object Technol.

    (2011)
  • GammaE. et al.

    Design Patterns: Elements of Reusable Object-oriented Software

    (1995)
  • Gil, J.Y., Maman, I., 2005. Micro patterns in java code. In: Proc. 20th Ann. ACM SIGPLAN Conf. Object-Oriented...
  • GrosanC. et al.

    Rule-based expert systems

  • Cited by (0)

    Rafael Barbudo is a Ph.D. student of Computer Science at the University of Córdoba, Spain. He received both his B.S. and M.S. in Information and Computer Science at the University of Córdoba. His research interests are related to the application of artificial intelligence techniques to solve real world problems. In this vein, during the M.S. years, his research focused on the application of machine learning techniques to address software engineering problems, especially those concerning the design phases. His current research is focused on the development of tools that assist both the domain experts and the data scientists in the process of knowledge extraction by optimising and automating those repetitive and timeconsuming phases.

    Aurora Ramírez received the M.Sc. and Ph.D. degrees in Computer Science from the University of Córdoba, Córdoba, Spain, in 2012 and 2018, respectively. She is currently a postdoctoral researcher of the Knowledge Discovery and Intelligent Systems Research Laboratory from the same university, where she conducts her research in artificial intelligence applied to software engineering. Her research interests are search-based software engineering, data mining and knowledge discovery, explainable artificial intelligence, and bio-inspired computing. Her contributions include eight journal articles and more than 15 conference publications. She has also participated as member of the program committee of 10 international conferences, and acts as reviewer of journals like IEEE Transactions on Software Engineering, Applied Soft Computing, and Information Sciences, among others.

    Francisco Servant is assistant professor of Computer Science at Virginia Tech. His research seeks to provide practical and human-friendly techniques and tools that improve the effectiveness and efficiency of software development and maintenance tasks. His research interests include software development productivity, software quality, mining of software repositories, program comprehension, and software visualisation. He has published articles in these areas at top conferences such as the International Conference on Software Engineering (ICSE), and the International Symposium on the Foundations of Software Engineering (FSE). He has also performed research for large technology companies, such as Microsoft Research and DreamWorks Animation. Francisco received a Ph.D. in Software Engineering from the University of California, Irvine, under the supervision of James A. Jones. He also holds a M.S. in Information and Computer Sciences from the same university, supervised by André van der Hoek. He received the B.S. in Computer Science from the University of Granada, Spain. More information about Francisco’s research is available at http://www.fservant.com.

    José Raúl Romero received the M.Sc. and Ph.D. degrees in computer science from the University of Málaga, Spain, in 2003 and 2007, respectively. He is currently an Associate Professor with the Department of Computer Science and Numerical Analysis, University of Córdoba, Spain. He has published more than 100 papers in journals, books and scientific conferences, and collaborated in different research projects and excellence networks. His research interests include the development of intelligent systems for supporting the software development process, democratisation and automation of data science, and model-driven development and design of domain-specific modelling languages. He can be reached at http://www.jrromero.net/en.

    Editor: [GABRIELE BAVOTA].

    View full text