- author = "Mark Harman and Peter O'Hearn and Shubho Sengupta",
- title = "Harden and Catch for Just-in-Time Assured {LLM}-Based Software Testing: Open Research Challenges",
- booktitle = "Foundations of Software Engineering, FSE 2025",
- year = "2025",
- editor = "Jingyue Li",
- address = "Trondheim",
- month = "23-27 " # jun,
- publisher = "ACM",
- note = "Invited Keynote",
- keywords = "genetic algorithms, genetic programming, genetic improvement, SBSE, SBST, SBFT, Software Engineering, Programming Languages, AI, Large Language Models, LLMs, ANN, JIT, Meta, Facebook, LLMSE, ACH, TestGenLLM",
-
URL = "
https://arxiv.org/pdf/2504.16472",
-
URL = "https://conf.researchr.org/info/fse-2025/keynotes",
- size = "17 pages",
- abstract = "Despite decades of research and practice in automated software testing, several fundamental concepts remain ill-defined and under-explored, yet offer enormous potential real-world impact. We show that these concepts raise exciting new challenges in the context of Large Language Models for software test generation. More specifically, we formally define and investigate the properties of hardening and catching tests. A hardening test is one that seeks to protect against future regressions, while a catching test is one that catches such a regression or a fault in new functionality introduced by a code change. Hardening tests can be generated at any time and may become catching tests when a future regression is caught. We also define and motivate the Catching Just-in-Time (JiTTest) Challenge, in which tests are generated just-in-time to catch new faults before they land into production. We show that any solution to Catching JiTTest generation can also be repurposed to catch latent faults in legacy code. We enumerate possible outcomes for hardening and catching tests and JiTTests, and discuss open research problems, deployment options, and initial results from our work on automated LLM-based hardening at Meta.",
-
notes = "real time jittest <= 8 hours. Fig 1: Hardening and
catching Venn Diagram.
Automatically generated test cases/asserts may fail in current build because they are designed for the new functionality, and so only pass in the revised build.
sec 6.1 'Plenty of Computation Time'
sec 6.3 [test]'oracle inference problem'
sec 6.4 'humans drizzle oracle information into software'
sec 6.4 [when creating test cases]'we have to treat executable text (the executable source code) as potentially suspicious, and focus on extraction of intended semantics from non-executable text' (eg comments)
Talk UCL 15 May 2025
https://conf.researchr.org/home/fse-2025
Product Compliance and Privacy team, Meta Platforms London, UK",
