Discovering Gated Recurrent Neural Network Architectures

Created by W.Langdon from gp-bibliography.bib Revision:1.8129

@InCollection{Rawal:2020:DNE,

• author = "Aditya Rawal and Jason Liang and Risto Miikkulainen",
• title = "Discovering Gated Recurrent Neural Network Architectures",
• booktitle = "Deep Neural Evolution: Deep Learning with Evolutionary Computation",
• publisher = "Springer, Singapore",
• year = "2020",
• editor = "Hitoshi Iba and Nasimul Noman",
• series = "Natural Computing Series",
• chapter = "9",
• pages = "233--251",
• keywords = "genetic algorithms, genetic programming, ANN, Deep Neuroevolution, LSTM",
• isbn13 = "978-981-15-3684-7",
• DOI = "doi:10.1007/978-981-15-3685-4_9",
• abstract = "Gated recurrent networks such as those composed of Long Short-Term Memory (LSTM) nodes have recently been used to improve state of the art in many sequential processing tasks such as speech recognition and machine translation. However, the basic structure of the LSTM node is essentially the same as when it was first conceived 25 years ago. Recently, evolutionary and reinforcement-learning mechanisms have been employed to create new variations of this structure. This chapter proposes a new method, evolution of a tree-based encoding of the gated memory nodes, and shows that it makes it possible to explore new variations more effectively than other methods. The method discovers nodes with multiple recurrent paths and multiple memory cells, which lead to significant improvement in the standard language modeling benchmark task. The chapter also shows how the search process can be sped up by training an LSTM network to estimate performance of candidate structures, and by encouraging exploration of novel solutions. Thus, evolutionary design of complex neural network structures promises to improve performance of deep learning architectures beyond human ability to do so.",
• notes = "Chapter 9, the structure of long short-term memory (LSTM) nodes is optimized using a genetic programming-based approach. It is shown that evolutionary optimisation of LSTM can result in design which is more complex and powerful compared to human design",

}

Genetic Programming entries for Aditya Rawal Jason Liang Risto Miikkulainen

Citations