- author = "Esteban Real and Chen Liang and David R. So and Quoc V. Le",
- title = "{AutoML-Zero}: Evolving Machine Learning Algorithms From Scratch",
- howpublished = "ArXiv",
- year = "2020",
- month = "30 " # jun,
- note = "Accepted for publication at the 37th International Conference on Machine Learning (ICML 2020). Near camera-ready version",
- keywords = "genetic algorithms, genetic programming, program synthesis, ANN",
-
URL = "
https://arxiv.org/abs/2003.03384",
-
URL = "https://ai.googleblog.com/2020/07/automl-zero-evolving-code-that-learns.html",
- eprint = "2003.03384",
- archiveprefix = "arXiv",
- primaryclass = "cs.LG",
- size = "23 pages",
- abstract = "Machine learning research has advanced in multiple aspects, including model structures and learning methods. The effort to automate such research, known as AutoML, has also made significant progress. However, this progress has largely focused on the architecture of neural networks, where it has relied on sophisticated expert-designed layers as building blocks---or similarly restrictive search spaces. Our goal is to show that AutoML can go further: it is possible today to automatically discover complete machine learning algorithms just using basic mathematical operations as building blocks. We demonstrate this by introducing a novel framework that significantly reduces human bias through a generic search space. Despite the vastness of this space, evolutionary search can still discover two-layer neural networks trained by backpropagation. These simple neural networks can then be surpassed by evolving directly on tasks of interest, e.g. CIFAR-10 variants, where modern techniques emerge in the top algorithms, such as bilinear interactions, normalized gradients, and weight averaging. Moreover, evolution adapts algorithms to different task types: e.g., dropout-like techniques appear when little data is available. We believe these preliminary successes in discovering machine learning algorithms from scratch indicate a promising new direction for the field.",
-
notes = "See \cite{pmlr-v119-real20a}
code: https://github.com/google-research/google-research/tree/master/automl_zero#automl-zero
AutoML-Zero, the proposal to automatically search forML algorithms from scratch with minimal human design
A novel framework with open-sourced code1and a searchspace that combines only basic mathematical operations
Detailed results to show potential through the discoveryof nuanced ML algorithms using evolutionary search.
CIFAR-10, SVHN, ImageNet, Fashion MNIST
memory scalar, vector and matrix variables (e.g.s1,v1,m1)
Throughput of 2000 to 10000 algorithms/second/cpu core. functional equivalence checking. hurdles (ICML 2019). Comparison with random search: 'As the tasktype becomes more difficult, evolution vastly outperforms RS'
Blog: (Thursday, July 9, 2020) https://ai.googleblog.com/2020/07/automl-zero-evolving-code-that-learns.html 'Evolving Learning Algorithms from Scratch We use a variant of classic evolutionary methods to search the space of algorithms. These methods have proved useful in discovering computer programs since the 80s. Their simplicity and scalability makes them especially suitable for the discovery of learning algorithms.' https://en.wikipedia.org/wiki/Genetic_programming
Google Brain team",
