- author = "Pedro Carvalho and Nuno Lourenco and Penousal Machado",
- title = "Evolving Learning Rate Optimizers for Deep Neural Networks",
- howpublished = "ArXiv",
- year = "2021",
- month = "23 " # mar,
- volume = "abs/2103.12623",
- keywords = "genetic algorithms, genetic programming, ANN",
-
URL = "
http://www.human-competitive.org/sites/default/files/carvalho-autolr.txt",
-
URL = "
http://www.human-competitive.org/sites/default/files/evolving_learning_rate_optimizers_for_deep_neural_networks.pdf",
-
URL = "https://arxiv.org/abs/2103.12623",
- size = "10 pages",
-
abstract = "Artificial Neural Networks (ANNs) became popular due
to their successful application difficult problems such
image and speech recognition. However, when
practitioners want to design an ANN they need to
undergo laborious process of selecting a set of
parameters and topology. Currently, there are several
state-of-the art methods that allow for the automatic
selection of some of these aspects. Learning Rate
optimisers are a set of such techniques that search for
good values of learning rates. Whilst these techniques
are effective and have yielded good results over the
years, they are general solution i.e. they do not
consider the characteristics of a specific network.
We propose a framework called AutoLR to automatically design Learning Rate Optimizers. Two versions of the system are detailed. The first one, Dynamic AutoLR, evolves static and dynamic learning rate optimizers based on the current epoch and the previous learning rate. The second version, Adaptive AutoLR, evolves adaptive optimizers that can fine tune the learning rate for each network weight which makes them generally more effective. The results are competitive with the best state of the art methods, even outperforming them in some scenarios. Furthermore, the system evolved a classifier, ADES, that appears to be novel and innovative since, to the best of our knowledge, it has a structure that differs from state of the art methods.",
- notes = "Entered 2021 HUMIES",
