- author = "Esteban Real and Chen Liang and David R. So and Quoc V. Le",
- title = "{AutoML}-Zero: Evolving Machine Learning Algorithms From Scratch",
- booktitle = "Proceedings of the 37th International Conference on Machine Learning",
- year = "2020",
- editor = "Hal {Daume, III} and Aarti Singh",
- volume = "119",
- series = "Proceedings of Machine Learning Research",
- pages = "8007--8019",
- month = "13--18 " # jul,
- publisher = "PMLR",
- note = "Winner 2021 HUMIES",
- keywords = "genetic algorithms, genetic programming, linear genetic programming, memory",
-
URL = "
http://www.human-competitive.org/sites/default/files/automl_zero_humies_competition_entry.txt",
-
URL = "
http://www.human-competitive.org/sites/default/files/automl_zero_humies_competition_paper.pdf",
-
URL = "http://proceedings.mlr.press/v119/real20a/real20a.pdf",
-
URL = "http://proceedings.mlr.press/v119/real20a/real20a-supp.pdf",
-
URL = "http://proceedings.mlr.press/v119/real20a.html",
-
URL = "http://proceedings.mlr.press/v119/",
-
video_url = "http://www.human-competitive.org/sites/default/files/estebanreal.humies_video_automl_zero.mp4",
-
code_url = "https://github.com/google-research/google-research/tree/master/automl_zero",
- size = "13 pages",
- abstract = "Machine learning research has advanced in multiple aspects, including model structures and learning methods. The effort to automate such research, known as AutoML, has also made significant progress. However, this progress has largely focused on the architecture of neural networks, where it has relied on sophisticated expert-designed layers as building blocks—or similarly restrictive search spaces. Our goal is to show that AutoML can go further: it is possible today to automatically discover complete machine learning algorithms just using basic mathematical operations as building blocks. We demonstrate this by introducing a novel framework that significantly reduces human bias through a generic search space. Despite the vastness of this space, evolutionary search can still discover two-layer neural networks trained by backpropagation. These simple neural networks can then be surpassed by evolving directly on tasks of interest, e.g. CIFAR-10 variants, where modern techniques emerge in the top algorithms, such as bilinear interactions, normalized gradients, and weight averaging. Moreover, evolution adapts algorithms to different task types: e.g., dropout-like techniques appear when little data is available. We believe these preliminary successes in discovering machine learning algorithms from scratch indicate a promising new direction for the field.",
-
notes = "see also \cite{real2020automlzero}
Start from empty code. (1) set up, (2) predict and (3) learn 0:33. Evolve code directly, use only basic maths, 0:59 Gradients are not provided, they must be evolved if they are to be used. Fitness on 100 CIFAR10 binary classification tasks. AutoML-Zero discovers machine learning from the start. Supervised learning of binary classifier. 3:42 many human like algorithms discovered and the authors are able to categorise GP evolved code in modern ML terms. GP like linear GP sequence of instructions? Better than human can be explained in terms of recent published ANN papers/text books 4:28. Noisy ReLU (2010). Evolution at scale, one experiment evaluated one trillion algorithms 6:18 10000 alg per second per CPU core. parallel population Connection to nature 7:05
2021 HUMIES prize giving video https://www.youtube.com/watch?v=jrT0sfq6WjM 43:10 -- 49:59 Many improvements on ANN",
