- author = "Mina Basirat",
- title = "The Quest for the Golden Activation Function",
- school = "Institute of Visual Computing, Graz University of Technology",
- year = "2021",
- address = "Austria",
- month = "23 " # nov,
- keywords = "genetic algorithms, genetic programming, ANN",
-
URL = "
https://repository.tugraz.at/publications/bqg05-90x12",
-
URL = "https://tugraz.webex.com/tugraz/j.php?MTID=m8519d344ed8a6089b38983d9b43bbb2c",
-
URL = "https://repository.tugraz.at/theses/78345",
-
DOI = "
10.3217/bqg05-90x12",
- size = "149 pages",
-
abstract = "Activation functions are one of the key building
blocks of a Deep Neural Network. Therefore, choosing
the right activation function always matters; however,
that comes with much manual effort. Even when the
expert knows that a certain parameterised activation
function is the optimal choice for a specific task, the
training might result in a sub-optimal model. This may
happen if the parameter is either fixed or optimized in
such a way that it is highly dependent on the
parameter's initial value. Besides, the expert usually
wants to have an insight into how the choice of
activation function contributes to the learning
behavior of a neural network and if the resulting model
satisfies certain expectations. Interpreting how a
neural network is trained and reasoning about its
traits is a most challenging problem.
This thesis presents three parts dedicated to mentioned problems. In the first part of this thesis, we propose to learn task-specific activation functions using ideas from genetic programming for image classification tasks. In the second part of this thesis, we initially propose a parametrized activation function, i.e., L*ReLU, to deal with the fine-grained visual categorization tasks. The function's parameter is adjusted with an optimal Lipschitz constant which is given by data in a brute force manner. Then, to automatically estimate the parameter's value, we apply a gradient-free optimization technique avoiding local optimum values for the parameter. In the third part of this thesis, the learning behaviour of deep neural networks and the traits of activation functions are interpreted by proposing a geometric perspective on Information Plane Analysis for both general learning phenomena and robust learning.",
-
notes = "In english.
Institute of Computer Graphics and Vision ICG
TUGRAZonline UF 786 880 Supervisor: Peter M. Roth",
