Symbolic Regression for Reinforcement Learning in Continuous Spaces

Created by W.Langdon from gp-bibliography.bib Revision:1.8120

@PhdThesis{Alibekov:thesis,

• author = "Eduard Alibekov",
• title = "Symbolic Regression for Reinforcement Learning in Continuous Spaces",
• school = "F3 Faculty of Electrical Engineering, Department of Cybernetics, Czech Technical University in Prague",
• year = "2021",
• address = "Czech Republic",
• month = aug,
• keywords = "genetic algorithms, genetic programming, Single Node Genetic Programming, reinforcement learning, optimal control, function approximation,evolutionary optimization, symbolic regression, robotics, autonomous systems",
• URL = "https://cyber.felk.cvut.cz/news/eduard-alibekov-defended-his-ph-d-thesis/",
• URL = "http://hdl.handle.net/10467/98283",
• URL = "https://dspace.cvut.cz/handle/10467/98283",
• URL = "https://dspace.cvut.cz/bitstream/handle/10467/98283/F3-D-2021-Alibekov-Eduard-phd_ready.pdf",
• size = "134 pages",
• abstract = "Reinforcement Learning (RL) algorithms can optimally solve dynamic decision and control problems in engineering, economics, medicine, artificial intelligence, and other disciplines.However, state-of-the-art RL methods still have not solved the transition from a small set of discrete states to fully continuous spaces. They have to rely on numerical function approximators, such as radial basis functions or neural networks, to represent the value function or policy mappings. While these numerical approximators are well-developed, the choice of a suitable architecture is a difficult step that requires significant trial-and-error tuning. Moreover, numerical approximators frequently exhibit uncontrollable surface artifacts that damage the overall performance of the controlled system.

  Symbolic Regression (SR) is an evolutionary optimization technique that automatically, without human intervention, generates analytical expressions to fit numerical data. The method has gained attention in the scientific community not only for its ability to recover known physical laws, but also for suggesting yet unknown but physically plausible and interpretable relationships. Additionally, the analytical nature of the result approximators allows to unleash the full power of mathematical apparatus.

  This thesis aims to develop methods to integrate SR into RL in a fully continuous case. To accomplish this goal, the following original contributions to the field have been developed.

  (i) Introduction of policy derivation methods. Their main goal is to exploit the full potential of using continuous action spaces, contrary to the state-of-the-art discretised set of actions.

  (ii) Quasi-symbolic policy derivation (QSPD) algorithm, specifically designed to be used with a symbolic approximation of the value function. The goal of the proposed algorithm is to efficiently derive continuous policy out of symbolic approximator. The experimental evaluation indicated the superiority of QSPD over state-of-the-art methods.

  (iii) Design of a symbolic proxy-function concept. Such a function is successfully used to alleviate the negative impacts of approximation artifacts on policy derivation.

  (iv) Study on fitness criterion in the context of SR for RL. The analysis indicated a fundamental flaw with any other symmetric error functions, including commonly used mean squared error. Instead, a new error function procedure has been proposed alongside with a novel fitting procedure. The experimental evaluation indicated dramatic improvement of the approximation quality for both numerical and symbolic approximators.

  (v) Robust symbolic policy derivation (RSPD) algorithm, which adds an extra level of robustness against imperfections in symbolic approximators. The experimental evaluation demonstrated significant improvements in the reachability of the goal state.

  All these contributions are then combined into a single,efficient SR for RL (ESRL) framework. Such a framework is able to tackle high-dimensional, fully-continuous RL problems out-of-the-box. The proposed framework has been tested on three bench-marks: pendulum swing-up, magnetic manipulation, and high-dimensional drone strike benchmark.",

• notes = "https://starfos.tacr.cz/en/project/GA15-22731S

  Supervisor: Olga Stepankova Supervisor-specialist: Robert Babuska",

}

Genetic Programming entries for Eduard Alibekov

Citations