A comparison of genetic programming and artificial neural networks in metamodeling of discrete-event simulation models
Introduction
DES models are widely used in the design and analysis of systems. In many instances model execution times of DES models can be computationally expensive. In such cases, its use in operational tasks such as design, sensitivity analysis and optimization can be significantly undermined. This shortcoming of DES has motivated development of methods that allow the creation of approximate models, i.e., metamodels of systems which sacrifice accuracy for computational gain. A metamodel refers to an approximate predictive model of system performance which is dependent on decision variables.
Wang and Shan [1] outline the techniques that can be used to build approximate response models for engineering design problems. Jin et al. [2] perform a comparative analysis of different techniques: polynomial regression (PR), Kriging (KG), multivariate adaptive regression splines (MARS), radial basis functions (RBF) on a range of test problems. The results of the study showed a dispersion of the observed performance of the techniques in terms of accuracy and problem structures, e.g., size and nonlinearity of the problem. While these articles focus on only deterministic problems, Li et al. [3] present a thorough comparison on stochastic problems considering artificial neural networks (ANNs) and support vector regression (SVR) in addition to KG, MARS and RBF. However, none of these studies considered GP [4] in metamodeling of stochastic problems to evolve symbolic expressions.
In this paper, a comparison between GP and ANNs is presented since ANNs have found frequent application in simulation metamodeling. These approaches can develop models without underlying assumptions or a priori knowledge about the relationship between the control factors and the performance. To build approximate models, they only require the information on system configuration and the corresponding performance, i.e., training data. The approximations are progressively improved using the information available from previously generated metamodels throughout the process to stimulate the search to find better metamodels. This can be a means for building highly accurate metamodels as they do not necessitate a simplification on the complexity of the systems studied.
In the evaluation of GP against ANN, DES models of three systems, which are different in the size of decision space, in the degree of variability and in the range of the performance measures, are used. Since there is neither assumptions nor information on the underlying functions of the performance of these systems, the uniform design (UD) [5] is used to sample the decision space of the problems. UDs are space-filling experimental designs which can be used to obtain training data when the underlying model is unknown [6]. Therefore, they are inherently suitable to use with metamodeling approaches, such as ANNs and GP.
In the remainder of this paper, first a literature review of simulation metamodeling is presented. Following, the GP approach is briefly introduced. Subsequently, in Section 4, the methodology of the study is given and lastly the results of the study are presented. The results focus on the accuracy and robustness of the methods across the problems both in the model building (training) and in the validation (test) stages as well as the computational requirements. The results show that GP is a very competitive metamodeling method, showing superior results in developing more generalized metamodels when compared to ANN.
Section snippets
Literature review
One of the earliest reported approaches for building approximations is the response surface methodology (RSM) [7]. In general, low order polynomials are used in conjunction with regression analysis to fit the system responses [8], [9], [10]. In polynomial regression (PR), a low order analytical function of decision variables with unknown coefficients is used. The model is estimated via regression analysis. The model coefficients are updated with respect to residual errors between the fitted
Symbolic regression with genetic programming
Genetic programming (GP) is a sub-branch of evolutionary algorithms (EAs)) which emulate the natural evolution of species. It has the capability to evolve programs of a domain via symbolic regression [4]. These programs can be interpreted as grammar rules, logic instructions, analytical functions, etc. This ability has led to attempts to solve a variety of problems such as in genetics [34], data mining [35] and chemistry [36].
Koza [4] first proposed the use of GP to find a symbolic regression
Simulation models used in the study
This section describes the problems used to compare the performance of GP and ANN for metamodeling DES. As mentioned earlier (see Section 2), the analysis considers ANNs since they are commonly applied in metamodeling due to their efficiency.
Simulation models of three common industrial systems are used in the empirical comparative study of GP and ANN: (i) an automated material handling system (AMHS) in a semiconductor manufacturing facility; (ii) a periodic review inventory problem (s,S)
Experiments and results
In this section, the results from the experiments will be provided. The configuration of the algorithms are analyzed in terms of (i) training and test performance and (ii) computational effort. The configuration of the algorithms is systematically varied to identify better algorithmic settings. Such a comparison can allow benchmarking of the competitiveness of GP against a predominant approach, i.e., ANNs, in the context of simulation-based metamodeling. To establish this, in the remainder of
Conclusion
This paper presented an empirical comparison between two methods, artificial neural networks (ANNs) and genetic programming (GP) in constructing metamodels of discrete-event simulation (DES) models. Both ANN and GP have the advantage that they do not require a priori assumptions on the form of the metamodel.
The comparative study was carried out using three different systems: an automated material handling system (AMHS) in semiconductor manufacturing, an (s,S) inventory model and a serial
Acknowledgment
This research is funded by Irish Research Council for Science, Engineering and Technology. We also like to take the opportunity to thank the anonymous reviewer for the constructive comments.
References (58)
- et al.
A systematic comparison of metamodeling techniques for simulation optimization in decision support systems
Applied Soft Computing
(2010) Response surface methodology for constrained simulation optimization: an overview
Simulation Modelling Practice and Theory
(2008)- et al.
A metamodeling methodology involving both qualitative and quantitative input factors
European Journal of Operational Research
(2003) - et al.
Using subsystem linear regression metamodels in stochastic simulation
European Journal of Operational Research
(2009) - et al.
Response surface analysis of a multi-product batch processing facility using a simulation metamodel
International Journal of Production Economics
(2006) - et al.
Regression metamodeling for the design of automated manufacturing system composed of parallel machines sharing a material handling resource
International Journal of Production Economics
(2004) - et al.
Real-time prediction of order flowtimes using support vector regression
Computers and Operations Research
(2008) - et al.
Metamodeling: radial basis functions, versus polynomials
European Journal of Operational Research
(2002) Kriging metamodeling in simulation: a review
European Journal of Operational Research
(2009)- et al.
Artificial neural networks for job shop simulation
Advanced Engineering Informatics
(2002)