Extending the boundaries of design optimization by integrating fast optimization techniques with machine-code-based, linear genetic programming

https://doi.org/10.1016/j.ins.2003.05.006Get rights and content

Abstract

Optimized models of complex physical systems are difficult to create and time consuming to optimize. The physical and business processes are often not well understood and are therefore difficult to model. The models of often too complex to be well optimized with available computational resources. Too often approximate, less than optimal models result. This work presents an approach to this problem that blends three well-tested components. First: We apply Linear Genetic Programming (LGP) to those portions of the system that are not well understood––for example, modeling data sets, such the control settings for industrial or chemical processes, geotechnical property prediction or UXO detection. LGP builds models inductively from known data about the physical system. The LGP approach we highlight is extremely fast and builds rapid to execute, high-precision models of a wide range of physical systems. Yet it requires few parameter adjustments and is very robust against overfitting. Second: We simulate those portions of the system––for example, the cost model for the processes––these are well understood with human built models. Finally: We optimize the resulting meta-model using Evolution Strategies (ES). ES is a fast, general-purpose optimizer that requires little pre-existing domain knowledge. We have developed this approach over a several years period and present results and examples that highlight where this approach can greatly improve the development and optimization of complex physical systems.

Introduction

Engineers frequently encounter problems that require them to estimate control or response settings for industrial or business processes that optimize one or more goals. Most optimization problems include two distinct parts: (1) A model of the process to be optimized; and (2) An optimizer that varies the control parameters of the model to derive optimal settings for those parameters.

For example, one of the research and development (R&D) case studies included here involves the control of an incinerator plant to achieve a high probability of environmental compliance and minimal cost. This required predictive models of the incinerator process, environmental regulations, and operating costs. It also required an optimizer that could combine the underlying models to calculate a real-time optimal response that satisfied the underlying constraints. Fig. 1 shows the relationship of the optimizer and the underlying models for this problem.

The incinerator example discussed above and the other case studies below did not yield to a simple constrained optimization approach or a well-designed neural network approach. The underlying physics of the problem were not well understood; so this problem was best solved by decomposing it into its constituent parts––the three underlying models (Fig. 1) and the optimizer.

This work is, therefore, concerned with complex optimization problems characterized by either of the following situations.

First: Engineers often understand the underlying processes quite well, but the software simulator they create for the process is slow. Deriving optimal settings for a slow simulator requires many calls to the simulator. This makes optimization inconvenient or completely impractical.

Our solution in this situation was to reverse engineer the existing software simulator using Linear Genetic Programming (LGP)––in effect, we simulated the simulator. Such “second-order” LGP simulations are frequently very accurate and almost always orders of magnitude faster than the hand-coded simulator. For example, for the Kodak Simulator, described below, LGP reverse engineered that simulator, reducing the time per simulation from hours to less than a second. As a result, an optimizer may be applied to the LGP-derived simulation quickly and conveniently.

Second: In the incinerator example given above, the cost and regulatory models were well understood, but the physics of the incinerator plant were not. However good quality plant operating data existed. This example highlights the second situation in which our approach consistently yields excellent results. LGP built a model of plant operation directly from the plant operation data. Combined with the cost and regulatory models to form a meta-model, the LGP model permits real-time optimization to achieve regulatory and cost goals.

For both of the above types of problems, the optimization and modeling tools should possess certain clearly definable characteristics:

  • the optimizer should make as few calls to the process model as possible, consistent with producing high-quality solutions;

  • the modeling tool should consistently produce high-precision models that execute quickly when called by the optimizer;

  • both the modeling and optimizing tools should be general-purpose tools. That is, they should be applicable to most problem domains with minimal customization and capable of producing good to excellent results across the whole range of problems that might be encountered; and


by integrating tools with the above characteristics, we have been able to improve problem-solving capabilities very significantly for both problem types above.

This work is organized as follows. We begin by introducing the Evolution Strategies with Completely Derandomized Self-Adaptation (ES-CDSA) algorithm as our optimization algorithm of choice. Next, we describe machine-code-based, LGP in detail and detail a three-year study from which we have concluded that machine-code-based, LGP is our modeling tool of choice for these types of applications. Finally, we suggest ways in which the integrated optimization and modeling strategy may be applied to design optimization problems.

Section snippets

Evolution strategies optimization

Evolution strategies (ES) was first developed in Germany in the 1960s. It is a very powerful, general-purpose, parameter optimization technique [25], [26], [27]. Although we refer in this work to ES, it is closely related to Fogel’s Evolutionary Programming (EP) [1], [7]. Our discussion here applies equally to ES and EP. For ease of reference, we will use the term “ES” to refer to both approaches.

ES uses a population-based learning algorithm. Each generation of possible solutions is formed by

Linear genetic programming

Genetic Programming (GP) is the automatic, computerized creation of computer programs to perform a selected task using Darwinian natural selection. GP developers give their computers examples of how they want the computer to perform a task. GP software then writes a computer program that performs the task described by the examples.

GP is a robust, dynamic, and quickly growing discipline. It has been applied to diverse problems with great success––equaling or exceeding the best human-created

Why machine-code-based, linear genetic programming?

At first glance, it is not at all obvious that machine-code, LGP is a strong candidate for the modeling algorithm of choice for the types of complex, high-dimensional problems at issue here. But over the past three years, a series of tests were performed on both synthetic and industrial data sets––many of them data sets on which other modeling tools had failed. The purpose of these tests was to assess machine-code, LGP’s performance as a general-purpose modeling tool.

In brief summary, the

Multiple linear genetic programming runs

GP is a stochastic algorithm. Accordingly, running it over and over with the same inputs usually produces a wide range of results, ranging from very bad to very good. For example, Fig. 2 shows the distribution of the results from 30 runs of LGP on the incinerator plant modeling problem mentioned in the introduction––the R2 value is used to measure the quality of the solution. The solutions ranged from a very poor R2 of 0.05 to an excellent R2 of 0.95.

Our investigation to date strongly suggests

Configuration issues in performing multiple LGP runs

Our investigation into exploiting the multiple run capability of machine-code-based LGP had two phases––largely defined by software versioning. Early versions of the Discipulus LGP software permitted multiple runs, but only with user-predefined parameter settings.

As a result, our early multiple run efforts (described below as our Phase I investigation) just chose a range of reasonable values for key parameters, estimated an appropriate termination criterion for the runs, and conducted a series

Investigation of machine-code-based, linear genetic programming––Phase I

We tested Versions 1.0 and 2.0 of the Discipulus LGP software on a number of problem domains during this first phase of our investigation. This Phase I investigation covered about two years and is reported in the next three sections.

Investigation of machine-code-based, linear genetic programming––Phase II

Phase II of our investigation started when we began using Version 3.0 of the LGP software [22]. As noted above, this new version automated many aspects of conducting multiple runs, including automatically randomizing run parameters, hillclimbing to optimize run parameters, automatic determination of the appropriate termination criterion for LGP for a particular problem domain, and automatic creation of team solutions.

Conclusion regarding empirical studies

The key results of the two phases of our empirical studies of the LGP algorithm are as follows.

First: The LGP software we used consistently produces excellent results on difficult, industrial modeling problems with little customization of the learning algorithm. Note: LGP did not always produce better results than all other algorithms studied. However, on every problem studied, LGP produced a model that was as good as, or better than, any other algorithm.

The performance of other learning

Integrated system analysis

This work is concerned with the building of a system comprised of integrated modeling and optimization tools. The integrated tool suite, comprised of (1) Machine-code-based LGP for creating predictive models, and (2) ES-CDSA, is expected to ably handle a wide range of the complex problems with which we are concerned.

The remainder of this paper is devoted to discussing two application areas for the integration of these tools using two of the problems mentioned above––the incinerator R&D problem

Optimizing the LGP-derived UXO models

The problem of UXO or land mines affects millions of acres world-wide and includes both training areas and former battlefields. The estimated cost for remediating the U.S. training ranges alone is at least $14 billion, and this number is likely understated [11]. The very real cost of clean-up (or non-clean-up) is the injury or death to people.

Currently, too large a portion of the resources available for responding to UXO challenges is expended by digging up sites where UXO’s are predicted, but

Summary and conclusions

We are in the early stages of building a comprehensive, integrated Optimization and Modeling system to handle complex industrial problems. We believe a combination of machine-code-based, LGP (for modeling) and ES-CDSA (for optimization) together provide the best combination of available tools and algorithms for this task.

By conceiving of design optimization projects as integrated modeling and optimization problems from the outset, we anticipate that engineers and researchers will be able to

Acknowledgements

The results presented in this work are part of a three-plus year collaborative effort between SAIC and Register Machine Learning Technologies to advance the state of the-art of evolutionary computation as applied to complex systems. Specifically thanked for funding and latitude from SAIC are Joseph W. Craver, John Aucella, and Janardan J. Patel. Dr. Gregory Flach, Dr. Frank Syms, and Mr. Robert Walton are gratefully acknowledged for providing the input data sets used in some of the work––as are

References (28)

  • T. Bläck et al.

    An overview of evolutionary algorithms for parameter optimization

    Evolutionary Computation

    (1993)
  • W. Banzhaf et al.

    Genetic Programming, an Introduction

    (1998)
  • L.M. Deschaine

    Tackling real-world environmental challenges with linear genetic programming

    PCAI Magazine

    (2000)
  • L.M. Deschaine, J.J. Patel, R.G. Guthrie, J.T. Grumski, M.J. Ades, Using linear genetic programming to develop a C/C++...
  • L.M. Deschaine, R.A. Hoover, J.N. Skibinski, J.J. Patel, F.D. Francone, P. Nordin, M.J. Ades, Using machine learning to...
  • L.V. Fausett, A neural network approach to modeling a waste incinerator facility, in: Society for Computer Simulation’s...
  • D.B. Fogel, Evolving artificial intelligence, PhD thesis, University of California, San Diego, CA,...
  • F. Francone, P. Nordin, W. Banzhaf, Benchmarking the generalization capabilities of a compiling genetic programming...
  • F. Francone, Comparison of Discipulus™ Genetic Programming Software with Alternative Modeling Tools, 2002. Available...
  • A. Fukunaga et al.

    A genome compiler for high performance genetic programming

  • Government Accounting Office, DOD Training Range Clean-up Cost Estimates are Likely Understated, Report to House of...
  • N. Hansen et al.

    Completely derandomized self-adaptation in evolution strategies

    Evolutionary Computation

    (2001)
  • Jefferson Proving Grounds, Jefferson Proving Grounds Phase IV Report: Graph ES-1, May, Report No: SFIM-AEC-ET-CR-99051,...
  • J. Koza et al.

    Genetic Programming III

    (1999)
  • Cited by (36)

    • Genetic programming for predictions of effectiveness of rolling dynamic compaction with dynamic cone penetrometer test results

      2019, Journal of Rock Mechanics and Geotechnical Engineering
      Citation Excerpt :

      Although these noneffective code segments coexist with the effective code, they are not connected to the data flow unlike in TGP, where the structural introns do not exist because all the program components have a connection with the root node (Brameier and Banzhaf, 2007). However, because of the imperative program structure in LGP, the structural introns can be detected efficiently and completely (Francone and Deschaine, 2004; Alavi et al., 2013). There is a special variant of LGP, named automatic induction of machine code by genetic programming (AIMGP), where the individuals are represented and manipulated as native binary machine code (Nordin, 1994; Banzhaf et al., 1998).

    • Modeling soil bulk density through a complete data scanning procedure: Heuristic alternatives

      2017, Journal of Hydrology
      Citation Excerpt :

      Nonetheless, parsimony pressure tool was used here to reduce the tree (program) size so that it does not over-fit the data by becoming excessively customized. There are different schools of thought on using parsimony, since it is a binding or forcing function on an otherwise freely (i.e. unconstrained with maximal degrees of freedom) search strategy (when the program is allowed to grow unrestricted) (Francone and Deschaine, 2004). GEP gives the highest weight (variable importance) to the OC and CCE.

    • An evolutionary computational approach for formulation of compression index of fine-grained soils

      2014, Engineering Applications of Artificial Intelligence
      Citation Excerpt :

      The enhanced speed of the linear variants of GP (e.g., LGP and MEP) permits conducting many runs in realistic timeframes. This leads to deriving consistent and high-precision models with little customization (Francone and Deschaine, 2004; Poli et al., 2007; Gandomi et al., 2011b). EP techniques such as GEP and MEP are the most common linear-based GP methods.

    • Numerical modeling of concrete strength under multiaxial confinement pressures using linear genetic programming

      2013, Automation in Construction
      Citation Excerpt :

      The initial and maximum program sizes were respectively set to optimal values of 80 and 256 bytes as trade-offs between the running time and the complexity of the evolved solutions. One level was considered for the other parameters based on some previously suggested values [35,36,55,56,58,59] and also after making several preliminary runs and observing the performance behavior. The number of demes presented in Table 3 is related to the way that the population of programs is divided.

    View all citing articles on Scopus
    View full text