Elsevier

Applied Soft Computing

Volume 11, Issue 2, March 2011, Pages 1648-1656
Applied Soft Computing

Reducing overfitting in manufacturing process modeling using a backward elimination based genetic programming

https://doi.org/10.1016/j.asoc.2010.04.022Get rights and content

Abstract

Genetic programming (GP) has demonstrated as an effective approach in polynomial modeling of manufacturing processes. However, polynomial models with redundant terms generated by GP may depict overfitting, while the developed models have good accuracy on trained data sets but relatively poor accuracy on testing data sets. In the literature, approaches of avoiding overfitting in GP are handled by limiting the number of terms in polynomial models. However, those approaches cannot guarantee terms in polynomial models produced by GP are statistically significant to manufacturing processes. In this paper, a statistical method, backward elimination (BE), is proposed to incorporate with GP, in order to eliminate insignificant terms in polynomial models. The performance of the proposed GP has been evaluated by modeling three real-world manufacturing processes, epoxy dispenser for electronic packaging, solder paste dispenser for electronic manufacturing, and punch press system for leadframe downset in IC packaging. Empirical results show that insignificant terms in the polynomial models can be eliminated by the proposed GP and also the polynomial models generated by the proposed GP can achieve results with better predictions than the other commonly used existent methods, which are commonly used in GP for avoiding overfitting in polynomial modeling.

Introduction

Modeling of manufacturing processes is important to control variability in processing steps in a manufacturing line, and to understand process variables which control desired outputs. Physical models or empirical models are employed to represent manufacturing processes. Physical models are typically consisted of a set of partial differential equations, and they provides a fundamental understanding of relationships between process variables and desired outputs. They have been developed in fluid dispensing process [1], [2], injection molding process [3], and transfer molding process [4]. However, many manufacturing processes are too complex to develop accurate physical models.

Empirical modeling is a popular approach in development of process models by using experimental data. Statistical regression is a commonly used approach to develop empirical process models [5]. However, statistical regression models are accurate only in ranges in which they are developed, thus the uncertainty in this type of process modeling becomes fuzzy. Artificial neural networks [6], [7], [8] and fuzzy logic modeling techniques [9], [10], [11] have been used to develop process models in various manufacturing processes. These approaches have the capability to transform a non-linear mathematical model into a simple black-box structure, and have the advantage of modeling non-linearity. However, the approaches of neural networks and fuzzy logic modeling normally require a large amount of experimental data to develop models, which are usually not available in process designs. Also due to their lack of transparency, sensitivity studies of process parameters cannot be done easily.

Genetic programming (GP) is an evolutionary computational method which can be used to generate empirical models for real-world systems [12], [13]. Lakshminarayanan et al. [14], Madar et al. [15], McKay et al. [16] and Willis et al. [17] have demonstrated how GP can be used to generate polynomial models in process design. It uses the general outcome of GP to construct structures of polynomial models [18], [19], [20] based on tree representation, and then uses the approach of least squares algorithm to estimate contributions of branches of the tree. As a result, coefficients of polynomial models can be identified, and completed polynomial models can be produced. In applying GP in polynomial modeling based on real data, two drawbacks have been detected: 1) GP produces overfitted polynomial models which do not perform well on prediction of untrained data [21]; 2) GP generates overfitted polynomial models because trees growth through evolutions of GP which decreases the training error [22]. An overfitted polynomial model with an oversized polynomial decreases the transparent and interpretable of the model. Therefore a good polynomial model not only has a high fitness-of-goodness with respect to a set of training data, but also is simple and provides satisfactory prediction on untrained data. Polynomial models with fewer terms have appeal of simplicity, and have an economic advantage in terms of obtaining necessary information. Therefore insignificant terms in polynomial models are necessary to be eliminated.

To reduce overfitting in GP, a set of basis polynomials [23] has been designed on fitness functions to reduce statistical biases and to fit more flexibly data. Also cross validation has been applied to detect when over-fitting starts by checking saturation of improvement of GP [24], [25], [26], [27]. However the two approaches, which require large amount of data sets to perform polynomial modeling, cannot be applied on practical situation, since limited number of data sets are only available in practical situation. Regularization techniques [28], [29] have been applied to the error component of the fitness function in GP, which aims at reducing the tendency of overfitting by reducing number of terms on polynomial models. However, those regularization techniques may not be guaranteed that those terms in the generated polynomial models are statistically significant to processes. Therefore significant terms cannot be guaranteed to be included in polynomial models and insignificant terms may be included.

In this paper, we propose to incorporate GP with backward elimination (BE) [30] for eliminating insignificant terms in polynomial models. We call the hybridized algorithm as backward elimination based genetic programming, BE-GP. In the proposed BE-GP, the BE is performed to estimate coefficients of polynomial models of individuals in the GP. Each term in the polynomial model is checked whether it is statistically significant by BE. The insignificant term in the polynomial model is eliminated by BE until there is no insignificant term in the polynomial model. Resulting polynomial models produced by BE are returned as individuals to the GP. The performance of the proposed BE-GP has been evaluated by modeling three manufacturing processes, epoxy dispenser for electronic packaging, solder paste dispenser for electronic manufacturing, and punch press system for leadframe downset in IC packaging. Comparison of the proposed BE-GP and the existent methods for reducing overfitting in polynomial modeling [28], [29] is presented.

Section snippets

Process modeling using GP

In process design, the basic purpose is to translate the output of the process from the independent variables of the process. The output y can be described as:y=f(x1,x2,xn)where xj with j = 1,2,…n are independent variables of the process, and the model f relates the y and all variables. The model f can be found by a set of experimental data D=xD,yD=xiD,yiDi=1N with the corresponding values of the i-th variable vector xiD=x1Di,x2Di,...,xnDiRn and corresponding value of the i-th output yiDR.

Evaluation of backward elimination based genetic programming BE-GP

To evaluate the effectiveness of the BE-GP, it is employed to model three real-world manufacturing processes, a fluid dispenser for electronic packaging (in Section 3.1), a solder paste dispenser for electronic manufacturing (in Section 3.2), and a punch press system for leadframe downset in IC packaging (in Section 3.3). The modeling results of the BE-GP are compared with the GP discussed in Section 2.1. Since the mean absolute error MAE is used as the fitness function (7) in the GP discussed

Conclusion

In this paper, we have enhanced an ordinary GP by incorporating with a statistical method, backward elimination BE. The objective of the proposed BE-GP is to reduce overfitting in modeling manufacturing processes by eliminating insignificant terms in polynomial modeling. The proposed BE-GP compensates the ordinary GP that significant terms cannot be guaranteed to be kept and insignificant terms cannot be guaranteed to be eliminated in polynomial modeling. The performance of the proposed BE-GP

References (32)

  • S. Lakshminarayanan et al.

    New product design via analysis of historical databases

    Computers and Chemical Engineering

    (2000)
  • B. McKay et al.

    Steady-state modeling of chemical processes using genetic programming

    Computers and Chemical Engineering

    (1997)
  • D.X. Chen, Modeling and off-line control of fluid dispensing for electronics packaging. PhD thesis, The University of...
  • J. Li et al.

    Technology development and basic theory study of fluid dispensing—a review

  • H.H. Chiang et al.

    A unified simulation of the filling and postfilling stages in injection molding, Part 1: formulation

    Polymer Engineering and Science

    (1991)
  • R. Han et al.

    Three-dimensional simulation of microchip encapsulation process

    Polymer Engineering and Science

    (2000)
  • G.A.F. Seber

    Linear Regression Analysis

    (2003)
  • H.L. Li et al.

    Optimization of resistance spot welding process using Taguchi method and a neural network

    Experimental Techniques

    (2007)
  • P.K. Simpson

    Artificial Neural Systems

    (1989)
  • K.W. Tong et al.

    Intelligent process design system for the transfer moulding of electronic packages

    International Journal of Production Research

    (2004)
  • K. Babets et al.

    Application of fuzzy logic for modeling of waterjet depainting

    Machining Science and Technology

    (2000)
  • S.Y. Kang et al.

    Physical and fuzzy logic modeling of a flip-chip thermo-compression bonding process

    Journal of Electronic Packaging

    (1993)
  • H. Xie et al.

    Process optimization using a fuzzy logic response surface method

    IEEE Transactions on Components, Packaging and Manufacturing Technology—Part A

    (1994)
  • J. Koza

    Genetic Programming: On the Programming of Computers by Means of Natural Evolution

    (1992)
  • J. Koza

    Genetic Programming II: Automatic Discovery of Reusable Programs

    (1994)
  • J. Madar et al.

    Genetic programming for the identification of nonlinear input–output models

    Industrial and Engineering Chemistry Research

    (2005)
  • Cited by (41)

    • Revising testing of composite precursors – A new framework for data capture in complex multi-material systems

      2022, Composites Part A: Applied Science and Manufacturing
      Citation Excerpt :

      This is a typical example of model’s overfitting and that was reported by numbers of researchers in various fields [32,33]. Governing equations of high-performance consolidation models may contain complex over-parameterised terms and are able to fit any given data set [34], which may lead to overfitting of the training input sets [35]. In such a case, the underlying patterns and relationships of the experimental data are not captured by the algorithm.

    • Optimisation and finite element simulation of the chopping process for chopper sugarcane harvesting

      2018, Biosystems Engineering
      Citation Excerpt :

      It evaluates the effect of removing one variable from a dataset according to the ANOVA results. This process deletes the variable whose loss gives the most statistically insignificant deterioration of the model fit, and repeats until no further variables can be deleted without a statistically significant loss of model fit (Bursac, Gauss, Williams, & Hosmer, 2008; Chan, Kwong, Dillon, & Tsim, 2011). In this study, the backward elimination was applied to eliminate insignificant coefficients in polynomial models.

    • Hypoglycaemia detection using fuzzy inference system with intelligent optimiser

      2014, Applied Soft Computing Journal
      Citation Excerpt :

      Overtraining refers to the reduction of the generalisation ability that can occur as a system is trained. In this paper, a validation strategy is employed to reduce the risk of overtraining [20,21,23]. This strategy is embedded in the training operation and the formulation of the fitness functions in the proposed intelligent optimiser.

    View all citing articles on Scopus
    View full text