FEAT-KD: learning concise representations for single and multi-target regression via TabNet knowledge distillation

Created by W.Langdon from gp-bibliography.bib Revision:1.9085

@InProceedings{Fong:2025:ICML,

• author = "Kei Sen Fong and Mehul Motani",
• title = "{FEAT-KD}: learning concise representations for single and multi-target regression via {TabNet} knowledge distillation",
• booktitle = "Proceedings of the 42nd International Conference on Machine Learning",
• year = "2025",
• articleno = "669",
• pages = "17378--17391",
• address = "Vancouver, Canada",
• publisher = "JMLR.org",
• keywords = "genetic algorithms, genetic programming",
• URL = "https://icml.cc/virtual/2025/poster/46151",
• URL = "https://dl.acm.org/doi/10.5555/3780338.3781007",
• abstract = "we propose a novel approach that combines the strengths of FEAT and Tab-Net through knowledge distillation (KD), which we term FEAT-KD. FEAT is an intrinsically interpretable machine learning (ML) algorithm that constructs a weighted linear combination of concisely-represented features discovered via genetic programming optimization, which can often be inefficient. FEAT-KD leverages TabNet's deep-learning-based optimization and feature selection mechanisms instead. FEAT-KD finds a weighted linear combination of concisely-represented, symbolic features that are derived from piece-wise distillation of a trained TabNet model. We analyse FEAT-KD on regression tasks from two perspectives: (i) compared to TabNet, FEAT-KD significantly reduces model complexity while retaining competitive predictive performance, effectively converting a black-box deep learning model into a more interpretable white-box representation, (ii) compared to FEAT, our method consistently outperforms in prediction accuracy, produces more compact models, and reduces the complexity of learned symbolic expressions. In addition, we demonstrate that FEAT-KD easily supports multi-target regression, in which the shared features contribute to the interpretability of the system. Our results suggest that FEAT-KD is a promising direction for interpretable ML, bridging the gap between deep learning's predictive power and the intrinsic transparency of symbolic models.",
• notes = "GECCO 2026 Hot of the press????",

}

Genetic Programming entries for Kei Sen Fong Mehul Motani

Citations