A comparison of genetic programming and artificial neural networks in metamodeling of discrete-event simulation models
Introduction
DES models are widely used in the design and analysis of systems. In many instances model execution times of DES models can be computationally expensive. In such cases, its use in operational tasks such as design, sensitivity analysis and optimization can be significantly undermined. This shortcoming of DES has motivated development of methods that allow the creation of approximate models, i.e., metamodels of systems which sacrifice accuracy for computational gain. A metamodel refers to an approximate predictive model of system performance which is dependent on decision variables.
Wang and Shan [1] outline the techniques that can be used to build approximate response models for engineering design problems. Jin et al. [2] perform a comparative analysis of different techniques: polynomial regression (PR), Kriging (KG), multivariate adaptive regression splines (MARS), radial basis functions (RBF) on a range of test problems. The results of the study showed a dispersion of the observed performance of the techniques in terms of accuracy and problem structures, e.g., size and nonlinearity of the problem. While these articles focus on only deterministic problems, Li et al. [3] present a thorough comparison on stochastic problems considering artificial neural networks (ANNs) and support vector regression (SVR) in addition to KG, MARS and RBF. However, none of these studies considered GP [4] in metamodeling of stochastic problems to evolve symbolic expressions.
In this paper, a comparison between GP and ANNs is presented since ANNs have found frequent application in simulation metamodeling. These approaches can develop models without underlying assumptions or a priori knowledge about the relationship between the control factors and the performance. To build approximate models, they only require the information on system configuration and the corresponding performance, i.e., training data. The approximations are progressively improved using the information available from previously generated metamodels throughout the process to stimulate the search to find better metamodels. This can be a means for building highly accurate metamodels as they do not necessitate a simplification on the complexity of the systems studied.
In the evaluation of GP against ANN, DES models of three systems, which are different in the size of decision space, in the degree of variability and in the range of the performance measures, are used. Since there is neither assumptions nor information on the underlying functions of the performance of these systems, the uniform design (UD) [5] is used to sample the decision space of the problems. UDs are space-filling experimental designs which can be used to obtain training data when the underlying model is unknown [6]. Therefore, they are inherently suitable to use with metamodeling approaches, such as ANNs and GP.
In the remainder of this paper, first a literature review of simulation metamodeling is presented. Following, the GP approach is briefly introduced. Subsequently, in Section 4, the methodology of the study is given and lastly the results of the study are presented. The results focus on the accuracy and robustness of the methods across the problems both in the model building (training) and in the validation (test) stages as well as the computational requirements. The results show that GP is a very competitive metamodeling method, showing superior results in developing more generalized metamodels when compared to ANN.
Section snippets
Literature review
One of the earliest reported approaches for building approximations is the response surface methodology (RSM) [7]. In general, low order polynomials are used in conjunction with regression analysis to fit the system responses [8], [9], [10]. In polynomial regression (PR), a low order analytical function of decision variables with unknown coefficients is used. The model is estimated via regression analysis. The model coefficients are updated with respect to residual errors between the fitted
Symbolic regression with genetic programming
Genetic programming (GP) is a sub-branch of evolutionary algorithms (EAs)) which emulate the natural evolution of species. It has the capability to evolve programs of a domain via symbolic regression [4]. These programs can be interpreted as grammar rules, logic instructions, analytical functions, etc. This ability has led to attempts to solve a variety of problems such as in genetics [34], data mining [35] and chemistry [36].
Koza [4] first proposed the use of GP to find a symbolic regression
Simulation models used in the study
This section describes the problems used to compare the performance of GP and ANN for metamodeling DES. As mentioned earlier (see Section 2), the analysis considers ANNs since they are commonly applied in metamodeling due to their efficiency.
Simulation models of three common industrial systems are used in the empirical comparative study of GP and ANN: (i) an automated material handling system (AMHS) in a semiconductor manufacturing facility; (ii) a periodic review inventory problem (s,S)
Experiments and results
In this section, the results from the experiments will be provided. The configuration of the algorithms are analyzed in terms of (i) training and test performance and (ii) computational effort. The configuration of the algorithms is systematically varied to identify better algorithmic settings. Such a comparison can allow benchmarking of the competitiveness of GP against a predominant approach, i.e., ANNs, in the context of simulation-based metamodeling. To establish this, in the remainder of
Conclusion
This paper presented an empirical comparison between two methods, artificial neural networks (ANNs) and genetic programming (GP) in constructing metamodels of discrete-event simulation (DES) models. Both ANN and GP have the advantage that they do not require a priori assumptions on the form of the metamodel.
The comparative study was carried out using three different systems: an automated material handling system (AMHS) in semiconductor manufacturing, an (s,S) inventory model and a serial
Acknowledgment
This research is funded by Irish Research Council for Science, Engineering and Technology. We also like to take the opportunity to thank the anonymous reviewer for the constructive comments.
References (58)
- et al.
A systematic comparison of metamodeling techniques for simulation optimization in decision support systems
Applied Soft Computing
(2010) Response surface methodology for constrained simulation optimization: an overview
Simulation Modelling Practice and Theory
(2008)- et al.
A metamodeling methodology involving both qualitative and quantitative input factors
European Journal of Operational Research
(2003) - et al.
Using subsystem linear regression metamodels in stochastic simulation
European Journal of Operational Research
(2009) - et al.
Response surface analysis of a multi-product batch processing facility using a simulation metamodel
International Journal of Production Economics
(2006) - et al.
Regression metamodeling for the design of automated manufacturing system composed of parallel machines sharing a material handling resource
International Journal of Production Economics
(2004) - et al.
Real-time prediction of order flowtimes using support vector regression
Computers and Operations Research
(2008) - et al.
Metamodeling: radial basis functions, versus polynomials
European Journal of Operational Research
(2002) Kriging metamodeling in simulation: a review
European Journal of Operational Research
(2009)- et al.
Artificial neural networks for job shop simulation
Advanced Engineering Informatics
(2002)
A comparison of experimental designs in the development of a neural network simulation metamodel
Simulation Modelling Practice and Theory
Buffer allocation and performance modeling in asynchronous assembly system operations: an artificial neural network metamodeling approach
Applied Soft Computing
Process optimization via neural network metamodeling
International Journal of Production Economics
Neural network metamodeling for cycle time-throughput profiles in manufacturing
European Journal of Operational Research
Modeling manufacturing processes using a genetic programming-based fuzzy regression with detection of outliers
Information Science
Heat transfer correlations by symbolic regression
International Journal of Heat and Mass Transfer
Steady-state modelling of chemical process systems using genetic programming
Computers & Chemical Engineering
A multi-objective genetic programming approach to developing Pareto optimal decision trees
Decision Support Systems
Simulation metamodel development using uniform design and neural networks for automated material handling systems in semiconductor wafer fabrication
Simulation Modelling Practice and Theory
A modular simulation approach for automated material handling systems
Simulation Practice and Theory
Multiple response optimization in a fully automated fab: an integrated tool and vehicle dispatching strategy
Computers & Industrial Engineering
A new approach for buffer allocation in unreliable production lines
International Journal of Production Economics
Review of metamodeling techniques in support of engineering design optimization
Journal of Mechanical Design
Comparative studies of metamodelling techniques under multiple modelling criteria
Structural and Multidisciplinary Optimization
Genetic programming: on the programming of computers by means of natural selection
Uniform design: theory and application
Technometrics
Design and modeling for computer experiments
Estimating and validating nonlinear regression metamodels in simulation
Communications in Statistics—Simulation and Computation
Chaos optimization svr algorithm with application in prediction of regional logistics demand
Cited by (96)
Simulation optimization applied to production scheduling in the era of industry 4.0: A review and future roadmap
2024, Journal of Industrial Information IntegrationMetamodel optimization of a complex, rural–urban emergency medical services system
2022, Simulation Modelling Practice and TheoryCitation Excerpt :In recent years, metamodeling has been widely studied, and for EMS systems has been used to compare and optimize both response times and survival rates by changing ambulance locations [38] and dispatch policies [39]; these models were based on agent-based simulators. More generally, comparisons of different types of metamodels, such as regression splines, kriging, artificial neural networks, and random forests, as well as different sampling approaches, including space-filling designs and adaptive sampling, have been conducted in many contexts on a variety of simulators [40–45], including discrete event simulators [46,47]. Depending on the context of the problem and the ultimate goal, the recommended model and approach varies.
Artificial neural network-based decision support systems in manufacturing processes: A systematic literature review
2022, Computers and Industrial EngineeringMulti-objective optimization algorithm assisted by metamodels with applications in aerodynamics problems
2022, Applied Soft ComputingEvolutionary Learning Based Simulation Optimization for Stochastic Job Shop Scheduling Problems
2021, Applied Soft Computing