- author = "Nadia Alshahwan and Mark Harman and Alexandru Marginean",
- title = "Software Testing Research Challenges: An Industrial Perspective",
- booktitle = "16th IEEE International Conference on Software Testing, Verification and Validation (ICST 2023)",
- year = "2023",
- editor = "Sreedevi Sampath",
- pages = "1--10",
- address = "Dublin, Ireland",
- month = "16-20 " # apr,
- note = "Keynote",
- keywords = "genetic algorithms, genetic programming, Genetic Improvement, SBSE, Automated Software Engineering, Software Testing, Automated Program Repair, APR, Artificial Intelligence, AI, Automated Remediation, regression testing",
- isbn13 = "978-1-6654-5667-8",
-
URL = "
https://research.facebook.com/file/1235985840680898/Software-Testing-Research-Challenges--An-Industrial-Perspective.pdf",
-
URL = "
https://conf.researchr.org/track/icst-2023/icst-2023-keynotes",
-
DOI = "
doi:10.1109/ICST57152.2023.00008",
- size = "10 pages",
- abstract = "There have been rapid recent developments in automated software test design, repair and program improvement. Advances in artificial intelligence also have great potential impact to tackle software testing research problems. we highlight open research problems and challenges from an industrial perspective. This perspective draws on our experience at Meta Platforms, which has been actively involved in software testing research and development for approximately a decade. As we set out here, there are many exciting opportunities for software testing research to achieve the widest and deepest impact on software practice. With this overview of the research landscape from an industrial perspective, we aim to stimulate further interest in the deployment of software testing research. We hope to be able to collaborate with the scientific community on some of these research challenges.",
-
notes = "descretization of performance execution time may lead
to performance test flakiness
'need more research ... on small performance [changes] ... in noisy ... [and better] confidence intervals'
'Technologies that significantly impede developer velocity are typically discarded'
'expect rapid uptake of automated [unit] test generation' 'including the generation of suitable test oracles' and 'appropriate mocks'. 'need techniques for constructing realistic values for complex data types'
e2e = end-to-end testing. test carving. [incremental] 'Delta mutation [testing] is highly change aware'
'return on investment of test effort'
'In industrial settings, other non-functional criteria are also highly important, including memory consumption, power consumption, code footprint size, scroll performance, image rendering speed/quality, server to client latency, and CPU cycle consumption'
'fitness without requiring overall system rebuild'
'A/B testing surrogate[s]'
'to pay off technical debt'
'Automated software transplantation is an important special case of refactoring.' 'the donor system can thus be used as a test oracle' ... 'software transplantation is fully automatable.'
[generative AI] 'tendency to hallucinate renders it relatively unreliable on its own. However'... [SE has ground truth].
[Large Language Model] 'AI techniques that mimic existing [human] coding styles' Codex LLM.
'Replacing execution with prediction'
ChatGPT LLM. 'LLMs [like undergraduates] are prone to hallucination'
'use [LLM] AI for performance optimisation'
Importance of when/where to use technology. 'it is essential to deploy into the continuous integration system at a time of maximum relevance'. 'Maximum automation is achieved when human decision time is minimised.'
'fuzzing [fuzz testing] technology has achieved widespread deployment'
Testability Transformation
Instagram Product Foundation, Meta Platforms Inc. Facebook",
