1. the complete title of one (or more) paper(s) published in the open literature describing the work that the author claims describes a human-competitive result Paper 1: Searching for activation functions using a self-adaptive evolutionary algorithm Paper 2: Evolution of Activation Functions: An Empirical Investigation 2. the name, complete physical mailing address, e-mail address, and phone number of EACH author of EACH paper(s) Andrew Nader, 60 Harbord Street, Toronto ON M5S 3L1, email: Andrew.nader@mail.utoronto.ca; phone: +1647-8606340 Danielle Azar, Computer Science and Math Department, Lebanese American University, PO.Box 36, F. Byblos, Lebanon. Email: Danielle.azar@lau.edu.lb; phone: +1961-9547254#2408 3. the name of the corresponding author (i.e., the author to whom notices will be sent concerning the competition) Danielle Azar 4. the abstract of the paper(s) Paper 1 Abstract: The introduction of the ReLU function in neural network architectures yielded substantial improvements over sigmoidal activation functions and allowed for the training of deep networks. Ever since, the search for new activation functions in neural networks has been an active research topic. However, to the best of our knowledge, the design of new activation functions has mostly been done by hand. In this work, we propose the use of a self-adaptive evolutionary algorithm that searches for new activation functions using a genetic programming approach, and we compare the performance of the obtained activation functions to ReLU. We also analyze the shape of the obtained activations to see if they have any common traits such as monotonicity or piece-wise linearity, and we study the effects of the self-adaptation to see which operators perform well in the context of a search for new activation functions. We perform a thorough experimental study on datasets of different sizes and types, using different types of neural network architectures. We report favorable results obtained from the mean and standard deviation of the performance metrics over multiple runs. Paper 2 Abstract: The hyper-parameters of a neural network are traditionally designed through a time consuming process of trial and error that requires substantial expert knowledge. Neural Architecture Search (NAS) algorithms aim to take the human out of the loop by automatically finding a good set of hyper-parameters for the problem at hand. These algorithms have mostly focused on hyper-parameters such as the architectural configurations of the hidden layers and the connectivity of the hidden neurons, but there has been relatively little work on automating the search for completely new activation functions, which are one of the most crucial hyper-parameters to choose. There are some widely used activation functions nowadays which are simple and work well, but nonetheless, there has been some interest in finding better activation functions. The work in the literature has mostly focused on designing new activation functions by hand, or choosing from a set of predefined functions while this work presents an evolutionary algorithm to automate the search for completely new activation functions. We compare these new evolved activation functions to other existing and commonly used activation functions. The results are favorable and are obtained from averaging the performance of the activation functions found over 30 runs, with experiments being conducted on 10 different datasets and architectures to ensure the statistical robustness of the study. 5. a list containing one or more of the eight letters (A, B, C, D, E, F, G, or H) that correspond to the criteria (see above) that the author claims that the work satisfies (B) The result is equal to or better than a result that was accepted as a new scientific result at the time when it was published in a peer-reviewed scientific journal. and (D) The result is publishable in its own right as a new scientific result independent of the fact that the result was mechanically created. 6. a statement stating why the result satisfies the criteria that the contestant claims (see examples of statements of human-competitiveness as a guide to aid in constructing this part of the submission) Criterion B: The choice of activation function is arguably among the most important hyper-parameters for a neural network given its influence on the shape of the loss landscape. The improvements obtained by the introduction of the ReLU function have sparked a vibrant and active investigation in the design of activation functions, but the research has mostly been concerned with hand designing these functions. Our work focuses on the use of evolutionary algorithms to automatically discover completely new activation functions for a specific dataset and architecture. Our results are positive, with the evolutionary search finding multiple functions that outperform commonly used ones such as ReLU, Leaky ReLU, and ELU. Our work is statistically robust, with the algorithm being tested on 10 different classification and regression datasets where we report different error metrics along with corresponding statistical significance tests Criterion D: Manually designing and choosing hyper-parameters (including activation functions) involves an extremely tedious and time consuming trial and error process for the Machine Learning experts, when that time could be spent working on more productive topics. We believe that our work is interesting on its own right since it shows that it is possible to automate the search for good activation functions, thus giving the expert the ability to focus on more interesting problems. Our work is also interesting in its own right because we study the evolution of the shapes of the activation functions throughout the algorithm’s run to see which properties are being favored for which datasets, which can hopefully prove useful in understanding the behavior of neural networks and can help inform future evolutionary algorithm decision. What was also interesting was to see that some function properties that were believed to be good (such as monotonicity, upper unboundedness, etc.) did not prove favorable on all datasets. 7. a full citation of the paper (that is, author names; title, publication date; name of journal, conference, or book in which article appeared; name of editors, if applicable, of the journal or edited book; publisher name; publisher city; page numbers, if applicable) Andrew Nader and Danielle Azar. 2020. Searching for activation functions using a self-adaptive evolutionary algorithm. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference Companion (GECCO '20). Association for Computing Machinery, New York, NY, USA, 145–146. DOI:https://doi.org/10.1145/3377929.3389942 Andrew Nader and Danielle Azar. 2021. To appear in Transactions in Evolutionary Learning and Optimization. 8. a statement either that "any prize money, if any, is to be divided equally among the co-authors" OR a specific percentage breakdown as to how the prize money, if any, is to be divided among the co-authors Any prize money, if any, is to be divided equally among the co-authors. 9. a statement stating why the authors expect that their entry would be the "best” Deep Learning is taking over the world so any algorithm that improves it or gets it into production more quickly through automated mechanisms would be extremely useful. We proposed an algorithm that optimizes what is arguably the most difficult hyper-parameter namely the activation function. To the best of our knowledge, this is the first work that proposes an algorithm that searches for completely new activation functions. Most of the previously published work focused on the selection of functions from a predefined pool. Results show that the algorithm is well suited to being integrated as part of Neural Architecture Search (NAS) algorithms and propose the inclusion of the search for new activation functions in NAS algorithms that usually focus on evolving new network architectures. 10. An indication of the general type of genetic or evolutionary computation used, such as GA (genetic algorithms), GP (genetic programming), ES (evolution strategies), EP (evolutionary programming), LCS (learning classifier systems), GI (genetic improvement), GE (grammatical evolution), GEP (gene expression programming), DE (differential evolution), etc. The general type of genetic or evolutionary computation used is Genetic Programming. 11. The date of publication of each paper. If the date of publication is not on or before the deadline for submission, but instead, the paper has been unconditionally accepted for publication and is “in press” by the deadline for this competition, the entry must include a copy of the documentation establishing that the paper meets the "in press" requirement. Date of publication of Paper 1: July 2020 Paper 2 is not yet published. We are attaching the letter from the editor confirming acceptance of the manuscript.