Multi-agent architecture for Multi‐objective optimization of Flexible Neural Tree

Abstract

In this paper, a multi-agent system is introduced to parallelize the Flexible Beta Basis Function Neural Network (FBBFNT)’ training as a response to the time cost challenge. Different agents are formed; a Structure Agent is designed for the FBBFNT structure optimization and a variable set of Parameter Agents is used for the FBBFNT parameter optimization. The main objectives of the FBBFNT learning process were the accuracy and the structure complexity. With the proposed multi-agent system, the main purpose is to reach a good balance between these objectives. For that, a multi-objective context was adopted which based on Pareto dominance. The agents use two algorithms: the Pareto dominance Extended Genetic Programming (PEGP) and the Pareto Multi-Dimensional Particle Swarm Optimization ($\mathit{PMD}\_\mathit{PSO}$) algorithms for the structure and parameter optimization, respectively. The proposed system is called Pareto Multi-Agent Flexible Neural Tree ($\mathit{PMA}\_\mathit{FNT}$).

To assess the effectiveness of $\mathit{PMA}\_\mathit{FNT}$, four benchmark real datasets of classification are tested. The results compared with some classifiers published in the literature.

Introduction

In Artificial Intelligence field, scientists have always meditated natural life and behaviors to invent and evolve intelligent systems. We can for instance mention the Artificial Neural Network (ANN) which was inspired from the human brain behavior, the Evolutionary Computation (EC) algorithms which were inspired from some natural phenomena and so on [1].

Thanks to its efficiency, the Multi-Layer ANN and in particular the well known ANN' topology has attracted more attention nowadays and it is now considered as a powerful system for complex search problems such as prediction [2], pattern recognition [3] and classification [4], [5].

In fact, the main weakness of the Neural Network training is the slow convergence to the near desired output, specially with large problems. To circumvent this weakness, many works hover around the objective of accelerating the NN training with ameliorating its performance. Some researchers focus on only training the output weights for a very large neural network [6], [7]. Others tried to find some alternatives to improve the MLPs performance. In this context, Evolutionary Computation EC has been considered as a good candidate for the ANN evolution [8]. EC includes Swarm Intelligence like Particle Swarm Optimization (PSO) [9], Evolutionary Algorithms such as Genetic Algorithm (GA) [10] as well as some Mimic Algorithms like Harmony Search (HS) [11] and so on.

In addition, it is not possible to predict an ideal ANN structure likely to resolve all the problems. So, a structure adaptive task is required to improve the Neural Network performance. Decreasing the complexity of the NN structure and increasing its flexibility has led to the establishment of a new ANN encoding based on the tree representation called Flexible Neural Tree (FNT) [3]. The FNT topology is adapted in our work using the Beta function as a transfer function (FBBFNT). Although the FBBFNTs could solve complex problems [12], this model suffers from high time-cost. In addition, real problems include a large set of features/inputs which increase the time complexity. So, we have considered parallelizing the learning process of the FBBFNT so that we can deal with real problems with respect to time cost.

On the other side, Multi-Agent System (MAS) is viewed as a new intelligent system inspired from the social system. In fact, as an intelligent system, the human being is a part of a social system in which he operates and interacts with other humans distributed in the same environment. Much in the same way, MAS distributes and coordinates a set of jobs, tasks and decisions between different entities, called agents, to build coherent and interactive systems [13].

In this work, the Multi-Agent System is used to optimize and parallelize the learning process of the evolving Flexible Beta Basis Function Neural Tree (FBBFNT) model. It uses a set of interacting agents; a Structure Agent for NN architecture/structure optimization and a set of Parameter Agents for NN parameters optimization.

Indeed, an agent is an independent and autonomous entity that has the ability to interact, cooperate, coordinate and negotiate with each other. For that, a communication process is disposed to ensure a good negotiation between agents. In our model, a negotiation protocol is implemented as a communication protocol. It ensures the exchange of information between agents which compete to find the optimal ANN for the treated problem. The negotiation strategy is based on two factors; the Agent Dominance Rate (ADR) and the Agent Trust Value (ATV). They ensure the evaluation of the agent performance in the multi-agent system.

To attain the optimal solution, this model takes into consideration two main objectives: the accuracy and the structure complexity of the neural network. In fact, the trade-off between these objectives is caused by their influence on the convergence and the effectiveness of the given solution. This conflict is implicitly pointed out in [14], [2]. A simple aggregation of the two objective functions has averted systems dealing with the conflict. However, these objectives have different impact on the NN and should ensure good balance between them. Consequently, we developed a multi-objective optimization to solve this trade-off. Indeed, the learning process adopted a multi-objective optimization based on the Pareto dominance. Learning agents use multi-objective evolutionary algorithms; the Pareto dominance Extended Genetic Programming algorithm (PEGP) and a Pareto Multi-Dimensional Particle Swarm Optimization algorithm (PMD_PSO) to optimize the FBBFNT structure and parameters respectively. According to the formerly mentioned, the model was called the Pareto Multi-Agent Flexible Neural Tree PMA_FNT. It will be described in Section 3. The functionalities and the training method of Structure and Parameter agents are detailed in 3.1 Pareto dominance extended genetic programming for structure agent, 3.2 Pareto multi-dimensional particle swarm optimization for parameter agent, respectively. The communication process including the strategy and the negotiation protocol used is described in Section 3.3. In Section 4, the experimentation results with real classification problems and a comparative study with other classifiers are presented. The final section introduces some concluding remarks.