Yi Song
ABSTRACT
Failures that are not related to a specific fault can reduce the effectiveness of fault localization in multi-fault scenarios. To tackle this challenge, researchers and practitioners typically cluster failures (e.g., failed test cases) into several disjoint groups, with those caused by the same fault grouped together. In such a fault isolation process that requires input in a mathematical form, ranking-based failure proximity (R-proximity) is widely used to model failed test cases. In R-proximity, each failed test case is represented as a suspiciousness ranking list of program statements through a fingerprinting function (i.e., a risk evaluation formula, REF). Although many off-the-shelf REFs have been integrated into R-proximity, they were designed for single-fault localization originally. To the best of our knowledge, no REF has been developed to serve as a fingerprinting function of R-proximity in multi-fault scenarios. For better clustering failures in fault isolation, in this paper, we present a genetic programming-based framework along with a sophisticated fitness function, for evolving REFs with the goal of more properly representing failures in multi-fault scenarios. By using a small set of programs for training, we get a collection of REFs that can obtain good results applicable in a larger and more general scale of scenarios. The best one of them outperforms the state-of-the-art by 50.72% and 47.41% in faults number estimation and clustering effectiveness, respectively. Our framework is highly configurable for further use, and the evolved formulas can be directly applied in future failure representation tasks without any retraining.
- Rui Abreu, Peter Zoeteweij, and Arjan JC Van Gemund. 2006. An evaluation of similarity coefficients for software fault localization. In 2006 12th Pacific Rim International Symposium on Dependable Computing (PRDC’06). IEEE, 39–46.Google Scholar
Digital Library
- Wasif Afzal and Richard Torkar. 2011. On the application of genetic programming for software engineering predictive modeling: A systematic review. Expert Systems with Applications 38, 9 (2011), 11984–11997.Google ScholarDigital Library
- Gabin An, Juyeon Yoon, and Shin Yoo. 2021. Searching for multi-fault programs in defects4j. In International Symposium on Search Based Software Engineering. Springer, 153–158.Google ScholarDigital Library
- James H Andrews, Lionel C Briand, and Yvan Labiche. 2005. Is mutation an appropriate tool for testing experiments?. In Proceedings of the 27th international conference on Software engineering. 402–411.Google ScholarDigital Library
- James H Andrews, Lionel C Briand, Yvan Labiche, and Akbar Siami Namin. 2006. Using mutation analysis for assessing and comparing testing coverage criteria. IEEE Transactions on Software Engineering 32, 8 (2006), 608–624.Google ScholarDigital Library
- Arun Babu, Qingkai Shi, and Muhammad Ashfaq. 2020. Python script for performing mutation testing. Github Repository. https://github.com/arun-babu/mutate.pyGoogle Scholar
- H. L. Cao and S. J. Jiang. 2017. Multiple-Fault Localization Based on Chameleon Clustering. Tien Tzu Hsueh Pao/Acta Electronica Sinica 45, 2 (2017), 394–400.Google Scholar
- Mike Y Chen, Emre Kiciman, Eugene Fratkin, Armando Fox, and Eric Brewer. 2002. Pinpoint: Problem determination in large, dynamic internet services. In Proceedings International Conference on Dependable Systems and Networks. IEEE, 595–604.Google ScholarCross Ref
- Stephen L Chiu. 1994. Fuzzy model identification based on cluster estimation. Journal of Intelligent & fuzzy systems 2, 3 (1994), 267–278.Google ScholarDigital Library
- Yingnong Dang, Rongxin Wu, Hongyu Zhang, Dongmei Zhang, and Peter Nobel. 2012. Rebucket: A method for clustering duplicate crash reports based on call stack similarity. In 2012 34th International Conference on Software Engineering (ICSE). IEEE, 1084–1093.Google ScholarCross Ref
- Higor A de Souza, Marcos L Chaim, and Fabio Kon. 2016. Spectrum-based software fault localization: A survey of techniques, advances, and challenges. arXiv preprint arXiv:1607.04347(2016).Google Scholar
- Nicholas DiGiuseppe and James A Jones. 2011. On the influence of multiple faults on coverage-based fault localization. In Proceedings of the 2011 international symposium on software testing and analysis. 210–220.Google ScholarDigital Library
- Nicholas DiGiuseppe and James A Jones. 2012. Concept-based failure clustering. In Proceedings of the ACM SIGSOFT 20th international symposium on the foundations of software engineering. 1–4.Google ScholarDigital Library
- Nicholas DiGiuseppe and James A Jones. 2012. Software behavior and failure clustering: An empirical study of fault causality. In 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation. IEEE, 191–200.Google ScholarDigital Library
- Nicholas DiGiuseppe and James A Jones. 2015. Fault density, fault types, and spectra-based fault localization. Empirical Software Engineering 20, 4 (2015), 928–967.Google ScholarDigital Library
- Hyunsook Do and Gregg Rothermel. 2006. On the use of mutation faults in empirical assessments of test case prioritization techniques. IEEE Transactions on Software Engineering 32, 9 (2006), 733–752.Google ScholarDigital Library
- Chunrong Fang, Zhenyu Chen, Kun Wu, and Zhihong Zhao. 2014. Similarity-based test case prioritization using ordered sequences of program entities. Software Quality Journal 22, 2 (2014), 335–361.Google ScholarDigital Library
- Yang Feng, James Jones, Zhenyu Chen, and Chunrong Fang. 2018. An empirical study on software failure classification with multi-label and problem-transformation techniques. In 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST). IEEE, 320–330.Google ScholarCross Ref
- Meng Gao, Pengyu Li, Congcong Chen, and Yunsong Jiang. 2018. Research on software multiple fault localization method based on machine learning. In MATEC web of conferences, Vol. 232. EDP Sciences, 01060.Google Scholar
- Ruizhi Gao and W Eric Wong. 2019. MSeer—An Advanced Technique for Locating Multiple Bugs in Parallel. IEEE Transactions on Software Engineering 45, 03 (2019), 301–318.Google ScholarDigital Library
- Debolina Ghosh and Jagannath Singh. 2021. Spectrum-based multi-fault localization using Chaotic Genetic Algorithm. Information and Software Technology 133 (2021), 106512.Google ScholarCross Ref
- Mojdeh Golagha, Constantin Lehnhoff, Alexander Pretschner, and Hermann Ilmberger. 2019. Failure clustering without coverage. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis. 134–145.Google ScholarDigital Library
- Mary Jean Harrold, Gregg Rothermel, Kent Sayre, Rui Wu, and Liu Yi. 2000. An empirical investigation of the relationship between spectra differences and regression faults. Software Testing, Verification and Reliability 10, 3(2000), 171–194.Google ScholarCross Ref
- Wolfgang Högerle, Friedrich Steimann, and Marcus Frenkel. 2014. More debugging in parallel. In 2014 IEEE 25th International Symposium on Software Reliability Engineering. IEEE, 133–143.Google ScholarDigital Library
- Y Huang and A Flynt. 2018. Exploration of Common Clustering Methods and the Behavior of Certain Agreement Indices. Ball State Undergraduate Mathematics Exchange 12, 1 (2018), 35–50.Google Scholar
- Yanqin Huang, Junhua Wu, Yang Feng, Zhenyu Chen, and Zhihong Zhao. 2013. An empirical study on clustering for isolating bugs in fault localization. In 2013 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW). IEEE, 138–143.Google ScholarCross Ref
- Dennis Jeffrey, Neelam Gupta, and Rajiv Gupta. 2008. Fault localization using value replacement. In Proceedings of the 2008 international symposium on Software testing and analysis. 167–178.Google ScholarDigital Library
- James A Jones, James F Bowring, and Mary Jean Harrold. 2007. Debugging in parallel. In Proceedings of the 2007 international symposium on Software testing and analysis. 16–26.Google ScholarDigital Library
- James A Jones and Mary Jean Harrold. 2005. Empirical evaluation of the tarantula automatic fault-localization technique. In Proceedings of the 20th IEEE/ACM international Conference on Automated software engineering. 273–282.Google ScholarDigital Library
- James A Jones, Mary Jean Harrold, and John Stasko. 2002. Visualization of test information to assist fault localization. In Proceedings of the 24th International Conference on Software Engineering. ICSE 2002. IEEE, 467–477.Google ScholarCross Ref
- René Just, Darioush Jalali, and Michael D Ernst. 2014. Defects4J: A database of existing faults to enable controlled testing studies for Java programs. In Proceedings of the 2014 International Symposium on Software Testing and Analysis. 437–440.Google ScholarDigital Library
- René Just, Darioush Jalali, Laura Inozemtseva, Michael D Ernst, Reid Holmes, and Gordon Fraser. 2014. Are mutants a valid substitute for real faults in software testing?. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. 654–665.Google ScholarDigital Library
- Leonard Kaufman and Peter J Rousseeuw. 2009. Finding groups in data: an introduction to cluster analysis. Vol. 344. John Wiley & Sons.Google Scholar
- Si-Mohamed Lamraoui and Shin Nakajima. 2016. A formula-based approach for automatic fault localization of multi-fault programs. Journal of Information Processing 24, 1 (2016), 88–98.Google ScholarCross Ref
- Hua Jie Lee, Lee Naish, and Kotagiri Ramamohanarao. 2009. Study of the relationship of bug consistency with respect to performance of spectra metrics. In 2009 2nd IEEE International Conference on Computer Science and Information Technology. IEEE, 501–508.Google Scholar
- Yihao Li, Pan Liu, Xiao Zhao, Jiaqi Yan, and Xiaoyu Song. 2021. Evaluating the Fault-Focused Clustering Performance of Distance Metrics in Parallel Fault Localization: From an Omniscient Perspective. In 2021 7th International Symposium on System and Software Reliability (ISSSR). IEEE, 1–10.Google ScholarCross Ref
- Ben Liblit, Mayur Naik, Alice X Zheng, Alex Aiken, and Michael I Jordan. 2005. Scalable statistical bug isolation. Acm Sigplan Notices 40, 6 (2005), 15–26.Google ScholarDigital Library
- Chao Liu, Long Fei, Xifeng Yan, Jiawei Han, and Samuel P Midkiff. 2006. Statistical debugging: A hypothesis testing-based approach. IEEE Transactions on software engineering 32, 10 (2006), 831–848.Google ScholarDigital Library
- Chao Liu and Jiawei Han. 2006. Failure proximity: a fault localization-based approach. In Proceedings of the 14th ACM SIGSOFT international symposium on Foundations of software engineering. 46–56.Google ScholarDigital Library
- Chao Liu, Xifeng Yan, Long Fei, Jiawei Han, and Samuel P Midkiff. 2005. SOBER: statistical model-based bug localization. ACM SIGSOFT Software Engineering Notes 30, 5 (2005), 286–295.Google ScholarDigital Library
- Chao Liu, Xiangyu Zhang, and Jiawei Han. 2008. A systematic study of failure proximity. IEEE Transactions on Software Engineering 34, 6 (2008), 826–843.Google ScholarDigital Library
- Lee Naish, Hua Jie Lee, and Kotagiri Ramamohanarao. 2011. A model for spectra-based software diagnosis. ACM Transactions on software engineering and methodology (TOSEM) 20, 3(2011), 1–32.Google Scholar
- Yulei Pang, Xiaozhen Xue, and Akbar Siami Namin. 2015. Debugging in Parallel or Sequential: An Empirical Study. J. Softw. 10, 5 (2015), 566–576.Google ScholarCross Ref
- Mike Papadakis, Marinos Kintis, Jie Zhang, Yue Jia, Yves Le Traon, and Mark Harman. 2019. Mutation testing advances: an analysis and survey. In Advances in Computers. Vol. 112. Elsevier, 275–378.Google Scholar
- Mike Papadakis and Yves Le Traon. 2015. Metallaxis-FL: mutation-based fault localization. Software Testing, Verification and Reliability 25, 5-7(2015), 605–628.Google ScholarDigital Library
- Hanyu Pei, Beibei Yin, Min Xie, and Kai-Yuan Cai. 2021. Dynamic random testing with test case clustering and distance-based parameter adjustment. Information and Software Technology 131 (2021), 106470.Google ScholarCross Ref
- Andy Podgurski, David Leon, Patrick Francis, Wes Masri, Melinda Minch, Jiayang Sun, and Bin Wang. 2003. Automated support for classifying software failure reports. In 25th International Conference on Software Engineering, 2003. Proceedings. IEEE, 465–475.Google ScholarCross Ref
- Michael Pradel and Koushik Sen. 2018. Deepbugs: A learning approach to name-based bug detection. Proceedings of the ACM on Programming Languages 2, OOPSLA(2018), 1–25.Google ScholarDigital Library
- Thomas Reps, Thomas Ball, Manuvir Das, and James Larus. 1997. The use of program profiling for software maintenance with applications to the year 2000 problem. In Software Engineering—Esec/Fse’97. Springer, 432–449.Google Scholar
- SIR. 2018. The Software Infrastructure Repository. Retrieved November 2, 2018 from https://sir.csc.ncsu.edu/portal/index.phpGoogle Scholar
- Jeongju Sohn and Shin Yoo. 2017. Fluccs: Using code and change metrics to improve fault localization. In Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis. 273–283.Google ScholarDigital Library
- Friedrich Steimann and Marcus Frenkel. 2012. Improving coverage-based localization of multiple faults using algorithms from integer linear programming. In 2012 IEEE 23rd International Symposium on Software Reliability Engineering. IEEE, 121–130.Google ScholarDigital Library
- Pang-Ning Tan, Michael Steinbach, and Vipin Kumar. 2016. Introduction to data mining. Pearson Education India.Google Scholar
- Chung Man Tang, WK Chan, Yuen Tak Yu, and Zhenyu Zhang. 2017. Accuracy graphs of spectrum-based fault localization formulas. IEEE Transactions on Reliability 66, 2 (2017), 403–424.Google ScholarCross Ref
- Yuan Tian, Chengnian Sun, and David Lo. 2012. Improved duplicate bug report identification. In 2012 16th European conference on software maintenance and reengineering. IEEE, 385–390.Google ScholarDigital Library
- Jingxuan Tu, Xiaoyuan Xie, Tsong Yueh Chen, and Baowen Xu. 2019. On the analysis of spectrum based fault localization using hitting sets. Journal of Systems and Software 147 (2019), 106–123.Google ScholarCross Ref
- Jingxuan Tu, Xiaoyuan Xie, and Baowen Xu. 2016. Code coverage-based failure proximity without test oracles. In 2016 IEEE 40th Annual Computer Software and Applications Conference (COMPSAC), Vol. 1. IEEE, 133–142.Google ScholarCross Ref
- Rijnard van Tonder, John Kotheimer, and Claire Le Goues. 2018. Semantic crash bucketing. In 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 612–622.Google ScholarDigital Library
- Qing Wang, Shujian Wu, and Mingshu Li. 2008. Software defect prediction. Journal of Software 19, 7 (2008), 1565–1580.Google ScholarCross Ref
- Shaowei Wang and David Lo. 2016. Amalgam+: Composing rich information sources for accurate bug localization. Journal of Software: Evolution and Process 28, 10 (2016), 921–942.Google ScholarDigital Library
- Shaowei Wang, David Lo, Lingxiao Jiang, Hoong Chuin Lau, 2011. Search-based fault localization. In 2011 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011). IEEE, 556–559.Google ScholarDigital Library
- Yabin Wang, Ruizhi Gao, Zhenyu Chen, W Eric Wong, and Bin Luo. 2014. WAS: A weighted attribute-based strategy for cluster test selection. Journal of Systems and Software 98 (2014), 44–58.Google ScholarDigital Library
- W Eric Wong, Vidroha Debroy, Ruizhi Gao, and Yihao Li. 2013. The DStar method for effective software fault localization. IEEE Transactions on Reliability 63, 1 (2013), 290–308.Google ScholarCross Ref
- W Eric Wong, Vidroha Debroy, and Dianxiang Xu. 2011. Towards better fault localization: A crosstab-based statistical approach. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 42, 3 (2011), 378–396.Google ScholarDigital Library
- W Eric Wong, Ruizhi Gao, Yihao Li, Rui Abreu, and Franz Wotawa. 2016. A survey on software fault localization. IEEE Transactions on Software Engineering 42, 8 (2016), 707–740.Google ScholarDigital Library
- W Eric Wong, Yu Qi, Lei Zhao, and Kai-Yuan Cai. 2007. Effective fault localization using code coverage. In 31st Annual International Computer Software and Applications Conference (COMPSAC 2007), Vol. 1. IEEE, 449–456.Google ScholarDigital Library
- Junjie Wu, Hui Xiong, and Jian Chen. 2009. Adapting the right measures for k-means clustering. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining. 877–886.Google ScholarDigital Library
- Yong-Hao Wu, Zheng Li, Yong Liu, and Xiang Chen. 2020. FATOC: Bug Isolation Based Multi-Fault Localization by Using OPTICS Clustering. Journal of Computer Science and Technology 35, 5 (2020), 979–998.Google ScholarDigital Library
- Xin Xia, Yang Feng, David Lo, Zhenyu Chen, and Xinyu Wang. 2014. Towards more accurate multi-label software behavior learning. In 2014 Software Evolution Week-IEEE Conference on Software Maintenance, Reengineering, and Reverse Engineering (CSMR-WCRE). IEEE, 134–143.Google ScholarCross Ref
- Yan Xiaobo, Bin Liu, and Wang Shihai. 2018. An analysis on the negative effect of multiple-faults for spectrum-based fault localization. IEEE Access 7(2018), 2327–2347.Google ScholarCross Ref
- JY XIE, Ying ZHOU, MZ WANG, and Weiliang JIANG. 2017. New criteria for evaluating the validity of clustering. CAAI Transactions on Intelligent Systems 12, 6 (2017), 873–882.Google Scholar
- Xiaoyuan Xie, Tsong Yueh Chen, Fei-Ching Kuo, and Baowen Xu. 2013. A theoretical analysis of the risk evaluation formulas for spectrum-based fault localization. ACM Transactions on Software Engineering and Methodology (TOSEM) 22, 4(2013), 1–40.Google ScholarDigital Library
- Xiaoyuan Xie, Fei-Ching Kuo, Tsong Yueh Chen, Shin Yoo, and Mark Harman. 2013. Provably optimal and human-competitive results in sbse for spectrum based fault localisation. In International Symposium on Search Based Software Engineering. Springer, 224–238.Google ScholarDigital Library
- Xiaoyuan Xie and Baowen Xu. 2021. Essential Spectrum-based Fault Localization. Springer.Google Scholar
- Xiaofeng Xu, Vidroha Debroy, W Eric Wong, and Donghui Guo. 2011. Ties within fault localization rankings: Exposing and addressing the problem. International Journal of Software Engineering and Knowledge Engineering 21, 06(2011), 803–827.Google ScholarCross Ref
- Jifeng Xuan, Matias Martinez, Favio Demarco, Maxime Clement, Sebastian Lamelas Marcote, Thomas Durieux, Daniel Le Berre, and Martin Monperrus. 2016. Nopol: Automatic repair of conditional statement bugs in java programs. IEEE Transactions on Software Engineering 43, 1 (2016), 34–55.Google ScholarDigital Library
- Xiaozhen Xue and Akbar Siami Namin. 2013. How significant is the effect of fault interactions on coverage-based fault localizations?. In 2013 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. IEEE, 113–122.Google ScholarCross Ref
- Ronald R Yager and Dimitar P Filev. 1994. Approximate clustering via the mountain method. IEEE Transactions on Systems, Man, and Cybernetics 24, 8(1994), 1279–1284.Google ScholarCross Ref
- Shin Yoo. 2012. Evolving human competitive spectra-based fault localisation techniques. In International Symposium on Search Based Software Engineering. Springer, 244–258.Google ScholarDigital Library
- Shin Yoo, Xiaoyuan Xie, Fei-Ching Kuo, Tsong Yueh Chen, and Mark Harman. 2017. Human competitiveness of genetic programming in spectrum-based fault localisation: Theoretical and empirical analysis. ACM Transactions on Software Engineering and Methodology (TOSEM) 26, 1(2017), 1–30.Google ScholarDigital Library
- Juyeon Yoon and Shin Yoo. 2021. Enhancing Lexical Representation of Test Coverage for Failure Clustering. In International Workshop on Software Engineering Automation: A Natural Language Prospective. (NLP-SEA 2021). 1–7.Google Scholar
- Zhongxing Yu, Chenggang Bai, and Kai-Yuan Cai. 2015. Does the failing test execute a single or multiple faults? An approach to classifying failing tests. In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, Vol. 1. IEEE, 924–935.Google ScholarCross Ref
- Abubakar Zakari and Sai Peck Lee. 2019. Parallel debugging: An investigative study. Journal of Software: Evolution and Process 31, 11 (2019), e2178.Google ScholarDigital Library
- Abubakar Zakari, Sai Peck Lee, Rui Abreu, Babiker Hussien Ahmed, and Rasheed Abubakar Rasheed. 2020. Multiple fault localization of software programs: A systematic literature review. Information and Software Technology 124 (2020), 106312.Google ScholarCross Ref
- Abubakar Zakari, Sai Peck Lee, and Ibrahim Abaker Targio Hashem. 2019. A community-based fault isolation approach for effective simultaneous localization of faults. IEEE Access 7(2019), 50012–50030.Google ScholarCross Ref
Index Terms
- Evolving Ranking-Based Failure Proximities for Better Clustering in Fault Isolation
Recommendations
Fault isolation for nonisolated blocks
We consider circuits represented as interconnections of logic blocks. In such circuits, the goal of fault isolation is to identify which one of the blocks is faulty based on a faulty output response produced by the circuit. We study this issue and ...
A comprehensive empirical investigation on failure clustering in parallel debugging
AbstractThe clustering technique has attracted a lot of attention as a promising strategy for parallel debugging in multi-fault scenarios, this heuristic approach (i.e., failure indexing or fault isolation) enables developers to perform ...
Highlights- The first empirical study regarding the clustering in parallel debugging.
- A ...
Data mining-based methods for fault isolation with validated FMEA model ranking
FMEA (Failure Mode and Effects Analysis), which was developed to enhance the reliability of complex systems, is a standard method to characterize and document product and process problems and a systematic method for fault identification/isolation in ...
Comments