Reducing overfitting in manufacturing process modeling using a backward elimination based genetic programming
Introduction
Modeling of manufacturing processes is important to control variability in processing steps in a manufacturing line, and to understand process variables which control desired outputs. Physical models or empirical models are employed to represent manufacturing processes. Physical models are typically consisted of a set of partial differential equations, and they provides a fundamental understanding of relationships between process variables and desired outputs. They have been developed in fluid dispensing process [1], [2], injection molding process [3], and transfer molding process [4]. However, many manufacturing processes are too complex to develop accurate physical models.
Empirical modeling is a popular approach in development of process models by using experimental data. Statistical regression is a commonly used approach to develop empirical process models [5]. However, statistical regression models are accurate only in ranges in which they are developed, thus the uncertainty in this type of process modeling becomes fuzzy. Artificial neural networks [6], [7], [8] and fuzzy logic modeling techniques [9], [10], [11] have been used to develop process models in various manufacturing processes. These approaches have the capability to transform a non-linear mathematical model into a simple black-box structure, and have the advantage of modeling non-linearity. However, the approaches of neural networks and fuzzy logic modeling normally require a large amount of experimental data to develop models, which are usually not available in process designs. Also due to their lack of transparency, sensitivity studies of process parameters cannot be done easily.
Genetic programming (GP) is an evolutionary computational method which can be used to generate empirical models for real-world systems [12], [13]. Lakshminarayanan et al. [14], Madar et al. [15], McKay et al. [16] and Willis et al. [17] have demonstrated how GP can be used to generate polynomial models in process design. It uses the general outcome of GP to construct structures of polynomial models [18], [19], [20] based on tree representation, and then uses the approach of least squares algorithm to estimate contributions of branches of the tree. As a result, coefficients of polynomial models can be identified, and completed polynomial models can be produced. In applying GP in polynomial modeling based on real data, two drawbacks have been detected: 1) GP produces overfitted polynomial models which do not perform well on prediction of untrained data [21]; 2) GP generates overfitted polynomial models because trees growth through evolutions of GP which decreases the training error [22]. An overfitted polynomial model with an oversized polynomial decreases the transparent and interpretable of the model. Therefore a good polynomial model not only has a high fitness-of-goodness with respect to a set of training data, but also is simple and provides satisfactory prediction on untrained data. Polynomial models with fewer terms have appeal of simplicity, and have an economic advantage in terms of obtaining necessary information. Therefore insignificant terms in polynomial models are necessary to be eliminated.
To reduce overfitting in GP, a set of basis polynomials [23] has been designed on fitness functions to reduce statistical biases and to fit more flexibly data. Also cross validation has been applied to detect when over-fitting starts by checking saturation of improvement of GP [24], [25], [26], [27]. However the two approaches, which require large amount of data sets to perform polynomial modeling, cannot be applied on practical situation, since limited number of data sets are only available in practical situation. Regularization techniques [28], [29] have been applied to the error component of the fitness function in GP, which aims at reducing the tendency of overfitting by reducing number of terms on polynomial models. However, those regularization techniques may not be guaranteed that those terms in the generated polynomial models are statistically significant to processes. Therefore significant terms cannot be guaranteed to be included in polynomial models and insignificant terms may be included.
In this paper, we propose to incorporate GP with backward elimination (BE) [30] for eliminating insignificant terms in polynomial models. We call the hybridized algorithm as backward elimination based genetic programming, BE-GP. In the proposed BE-GP, the BE is performed to estimate coefficients of polynomial models of individuals in the GP. Each term in the polynomial model is checked whether it is statistically significant by BE. The insignificant term in the polynomial model is eliminated by BE until there is no insignificant term in the polynomial model. Resulting polynomial models produced by BE are returned as individuals to the GP. The performance of the proposed BE-GP has been evaluated by modeling three manufacturing processes, epoxy dispenser for electronic packaging, solder paste dispenser for electronic manufacturing, and punch press system for leadframe downset in IC packaging. Comparison of the proposed BE-GP and the existent methods for reducing overfitting in polynomial modeling [28], [29] is presented.
Section snippets
Process modeling using GP
In process design, the basic purpose is to translate the output of the process from the independent variables of the process. The output y can be described as:where xj with j = 1,2,…n are independent variables of the process, and the model f relates the y and all variables. The model f can be found by a set of experimental data with the corresponding values of the i-th variable vector and corresponding value of the i-th output .
Evaluation of backward elimination based genetic programming BE-GP
To evaluate the effectiveness of the BE-GP, it is employed to model three real-world manufacturing processes, a fluid dispenser for electronic packaging (in Section 3.1), a solder paste dispenser for electronic manufacturing (in Section 3.2), and a punch press system for leadframe downset in IC packaging (in Section 3.3). The modeling results of the BE-GP are compared with the GP discussed in Section 2.1. Since the mean absolute error MAE is used as the fitness function (7) in the GP discussed
Conclusion
In this paper, we have enhanced an ordinary GP by incorporating with a statistical method, backward elimination BE. The objective of the proposed BE-GP is to reduce overfitting in modeling manufacturing processes by eliminating insignificant terms in polynomial modeling. The proposed BE-GP compensates the ordinary GP that significant terms cannot be guaranteed to be kept and insignificant terms cannot be guaranteed to be eliminated in polynomial modeling. The performance of the proposed BE-GP
References (32)
- et al.
New product design via analysis of historical databases
Computers and Chemical Engineering
(2000) - et al.
Steady-state modeling of chemical processes using genetic programming
Computers and Chemical Engineering
(1997) - D.X. Chen, Modeling and off-line control of fluid dispensing for electronics packaging. PhD thesis, The University of...
- et al.
Technology development and basic theory study of fluid dispensing—a review
- et al.
A unified simulation of the filling and postfilling stages in injection molding, Part 1: formulation
Polymer Engineering and Science
(1991) - et al.
Three-dimensional simulation of microchip encapsulation process
Polymer Engineering and Science
(2000) Linear Regression Analysis
(2003)- et al.
Optimization of resistance spot welding process using Taguchi method and a neural network
Experimental Techniques
(2007) Artificial Neural Systems
(1989)- et al.
Intelligent process design system for the transfer moulding of electronic packages
International Journal of Production Research
(2004)
Application of fuzzy logic for modeling of waterjet depainting
Machining Science and Technology
Physical and fuzzy logic modeling of a flip-chip thermo-compression bonding process
Journal of Electronic Packaging
Process optimization using a fuzzy logic response surface method
IEEE Transactions on Components, Packaging and Manufacturing Technology—Part A
Genetic Programming: On the Programming of Computers by Means of Natural Evolution
Genetic Programming II: Automatic Discovery of Reusable Programs
Genetic programming for the identification of nonlinear input–output models
Industrial and Engineering Chemistry Research
Cited by (41)
Multi-generation multi-criteria feature construction using Genetic Programming
2023, Swarm and Evolutionary ComputationRevising testing of composite precursors – A new framework for data capture in complex multi-material systems
2022, Composites Part A: Applied Science and ManufacturingCitation Excerpt :This is a typical example of model’s overfitting and that was reported by numbers of researchers in various fields [32,33]. Governing equations of high-performance consolidation models may contain complex over-parameterised terms and are able to fit any given data set [34], which may lead to overfitting of the training input sets [35]. In such a case, the underlying patterns and relationships of the experimental data are not captured by the algorithm.
Optimisation and finite element simulation of the chopping process for chopper sugarcane harvesting
2018, Biosystems EngineeringCitation Excerpt :It evaluates the effect of removing one variable from a dataset according to the ANOVA results. This process deletes the variable whose loss gives the most statistically insignificant deterioration of the model fit, and repeats until no further variables can be deleted without a statistically significant loss of model fit (Bursac, Gauss, Williams, & Hosmer, 2008; Chan, Kwong, Dillon, & Tsim, 2011). In this study, the backward elimination was applied to eliminate insignificant coefficients in polynomial models.
A novel defect depth measurement method based on Nonlinear System Identification for pulsed thermographic inspection
2017, Mechanical Systems and Signal ProcessingA coefficient clustering analysis for damage assessment of composites based on pulsed thermographic inspection
2016, NDT and E InternationalHypoglycaemia detection using fuzzy inference system with intelligent optimiser
2014, Applied Soft Computing JournalCitation Excerpt :Overtraining refers to the reduction of the generalisation ability that can occur as a system is trained. In this paper, a validation strategy is employed to reduce the risk of overtraining [20,21,23]. This strategy is embedded in the training operation and the formulation of the fitness functions in the proposed intelligent optimiser.