Elsevier

Neurocomputing

Volume 246, 12 July 2017, Pages 25-44
Neurocomputing

Improving the prediction of material properties of concrete using Kaizen Programming with Simulated Annealing

https://doi.org/10.1016/j.neucom.2016.12.077Get rights and content

Abstract

Predicting the properties of materials like concrete has been proven a difficult task given the complex interactions among its components. Over the years, researchers have used Statistics, Machine Learning, and Evolutionary Computation to build models in an attempt to accurately predict such properties. High-quality models are often non-linear, justifying the study of nonlinear regression tools. In this paper, we employ a traditional multiple linear regression method by ordinary least squares to solve the task. However, the model is built upon nonlinear features automatically engineered by Kaizen Programming, a recently proposed hybrid method. Experimental results show that Kaizen Programming can find low-correlated features in an acceptable computational time. Such features build high-quality models with better predictive quality than results reported in the literature.

Introduction

The production of concrete is an important technique in Civil constructing, and High-Performance Concrete (HPC) is a material that has been widely used in structural applications such as bridges, high-rise buildings, and pavement construction. As HPC results in good workability, high-strength and low permeability, it has replaced high-strength concrete [1] in several applications. Such concrete characteristics are directly related to long-term durability, making HPC more reliable [2]. HPC’s compressive strength is considered its most important quality [3], [4], while the slump flow – the diameter of slumped fresh concrete1 – can be used as a measure for its workability [5].

The production of HPC aims at reducing the porosity within the hydrated cement, which is the major difference to conventional concrete, achieving high compressive strength or a low water-to-binder ratio [3]. In order to achieve such characteristics, several chemical and mineral ingredients must be applied to the mixture during production, which requires industrial know-how because of the interactions/relationships among them [1], [6].

Aiming at developing increasingly better materials, the concrete must be produced and tested after some time has elapsed. Naturally, this step is costly and time-consuming. Researchers, therefore, have investigated methods to model the material behavior [1], which can later be optimized to give the best mixture for a particular material property. Using the putatively optimized mixture, the material can be produced and analyzed, with the measurement results inserted into the dataset as a new example. Then a new model can be created and optimized, and the cycle repeats. This way, material engineers can obtain increasingly better mixtures.

Concrete compressive strength (CCS) has been predicted by linear or non-linear regression methods [7], but given its non-linear characteristics, Machine Learning (ML) techniques – mainly Artificial Neural Networks (ANNs [8]) – have been investigated [1], [9]. Slump has been successfully predicted by ANNs [5], [6] and Evolutionary Computation (EC) methods [3], [10].

ML techniques such as ANNs and Support Vector Machines (SVMs [11]) produce models that are considered black-boxes, i.e., they are hard (if not impossible) to understand. Therefore, EC algorithms such as Genetic Programming (GP, [12], [13]), have been employed to generate smaller models by means of Symbolic Regression (SR [14], [15], [16]) that are easier to interpret [3], [10], [17], [18]

According to Schmidt and Lipson [19]: “Symbolic regression is a method for searching the space of mathematical expressions while minimizing various error metrics. Unlike traditional linear and nonlinear regression methods that fit parameters to an equation of a given form, SR methods search both the parameters and the form of equations simultaneously. This process automatically forms mathematical equations that are amenable to human interpretation and help explicate observed phenomena.” Therefore, the main reason for choosing SR over other regression methods such as SVMs or ANNs is interpretability, while a welcome side effect can be the automatic discovery of the exact equation that generates the response value. Such an equation may not be discovered by other methods because they only search for optimized coefficients. For instance, suppose that the desired equation is y=xsin(3.1415+x). SVMs and ANNs can generate a model that approximates the response surface; however, they are unable to find that ground-truth equation because their approach is not to find formulae, but to optimize coefficients.

Generally, EC algorithms try to mimic the evolution of organisms through improving solutions in a series of generations. A group of individuals (a population) generates offspring with the help of evolutionary operators called crossover (which recombines the genetic material of two parent individuals) and mutation (which adds variability in the genetic material of the offspring). Each individual is a complete solution to the problem being investigated, and its quality (fitness) is a score of how well it solves the problem (given by the objective function). Selection procedures are employed to select the parents and, later, the group of individuals that will pass to the next generation. In Genetic Programming (GP), the EC algorithms mostly used for SR, an individual is a group of instructions (such as a computer program) that must be executed to solve the problem. Such programs, as explained before, can be interpreted as mathematical equations composed of operators and operands. Operators can be arithmetic functions such as + and * or others, while operands/arguments are the constants and variables to be used in the formulae, e.g., 1.0, 3.14, x, and y. A widespread representation for GP operates on a tree-based structure, with operators as internal (non-terminal) nodes and operands as leaf (terminal) nodes of the tree. Another widespread representation (linear GP, [20]) uses a sequence of instructions as its genetic material. Generally, GP must optimize the individuals in order to obtain the best possible program.

In an attempt to develop an SR method that requires a smaller number of objective function evaluations and still produces high-quality, interpretable models, De Melo proposed a hybrid approach named Kaizen Programming (KP, [21]), based on the Kaizen [22] and the Plan-Do-Check-Act (PDCA [23]) methodologies. The basic idea is to use global search algorithms to perform automatic feature engineering and statistical methods to build models in a cycle of feature generation followed by feature selection (see Fig. 1). These steps will be explained in more detail later.

Because the features have to be used in the same model, they must complement each other; thus, the approach is truly collaborative instead of competitive. KP is guided by the importance of the partial solutions, not by the quality of the complete solution, which makes the search more efficient. KP relies on deterministic and efficient methods to build the model, while most similar methods rely on evolution.

A similar approach was proposed by Icke and Bongard [25] using a technique named Fast Function Extraction (FFX, [26]) to create features for a linear regression model. There, a hybrid algorithm uses features generated by an FFX run, which are passed onto a GP system for another step in model building. The authors hypothesized that such an approach would increase the chances of GP to succeed by letting FFX extract informative features while GP would build more complex models out of them. In this case, GP is the method supposed to solve the problem; this is exactly the opposite idea of our approach.

Given the promising results of KP shown in [21], here we extend the preliminary study of de Melo and Banzhaf [27] for feature engineering on real-world datasets of HPC concrete. The main contributions of this paper are as follows:

  • 1.

    we present a novel KP implementation for feature generation applied to an SR problem. This implementation uses Simulated Annealing (SA [28]) to solve the combinatorial optimization problem;

  • 2.

    we explicitly deal with multicollinearity, trying to assure that the resulting set of features, for the training set, has a maximum user-defined correlation value;

  • 3.

    we use the Akaike Information Criterion (AIC, [29]) as cost (energy) function, which can be employed for model comparison and for reducing overfitting;

  • 4.

    we apply a post-processing procedure to fix unrealistic prediction values;

  • 5.

    we present several comparisons with EC, statistics, and ML methods on a series of experiments, including processing time.

The remainder of the paper is organized as follows. Section 2 presents many related works. Section 3 introduces Kaizen Programming for feature constructing. In Section 4, we describe the approach proposed in this work. Experimental results are shown in Section 5. Finally, Section 6 has a summary, conclusions, and future work.

Section snippets

Related work

In this section, we first present some work related to KP and then introduce related work from the literature that investigated feature construction to improve prediction of HPC compressive strength. Most work is based on EC algorithms such as GP. Unfortunately, the other authors did not use exactly the same dataset, but some important characteristics of the methods will be highlighted.

Kaizen Programming for feature construction

Kaizen Programming (KP), a hybrid approach based on a Kaizen event with PDCA methodology, is used to guide a continuous improvement process. Both Kaizen and PDCA have many tools to solve problems. KP is proposed as an abstraction of these two methodologies, not a simulation. We start this section introducing KP terms, and later use traditional metaheuristics, statistics, and ML terms to simplify the reading.

A real Kaizen event has experts that propose and test ideas to solve a business issue.

Kaizen Programming with Simulated Annealing

In this work, we replace tree-based GP by SA. SA works with a single solution and performs a global search by evaluating neighboring solutions stochastically. If a neighbor is better than the current solution, it replaces the latter. On the other hand, a worse solution can be accepted according to an acceptance function. Such behavior allows SA to escape from non-promising regions and explore the search space by moving to worse neighborhoods. The acceptance probability function considers a

Experimental results

This section presents experimental results of KP using SA coupled with OLS to perform feature engineering and model fitting for a regression problem.

Summary, conclusions, and future works

Kaizen Programming is a hybrid algorithm that uses a collaborative problem solving approach where partial solutions compose a complete solution. KP can use global optimization algorithms, statistics, and ML. The partial solutions are created by the experts that generate ideas (partial solutions) based on the current standard (best set of partial solutions), which is improved over the cycles. Ideas are evaluated by their contribution to help to solve the problem, and the most important ones are

Acknowledgments

This paper was supported by the Brazilian Government CNPq (Universal) grant (486950/2013-1) and CAPES (Science without Borders) grant (12180-13-0) to V.V.M., and Canada’s NSERC Discovery grant RGPIN 283304-2012 to W.B.

Vinícius Veloso de Melo is an associate professor at the Institute of Science and Technology-UNIFESP, São Paulo, Brazil. He obtained his B.Sc. in Computer Science from PUC-MG, Brazil, in 2002; his M.Sc. and Ph.D. in Computer Science from University of São Paulo, Brazil, in 2005 and 2009, respectively. He has published papers in international peer-reviewed journals and conferences. His current research interests are evolutionary computation, metaheuristics, and machine learning.

References (51)

  • M.F. Brameier et al.

    Linear Genetic Programming

    (2007)
  • B. Efron et al.

    Least angle regression

    Ann. Stat.

    (2004)
  • YehI.-C. et al.

    Knowledge discovery of concrete material using genetic operation trees

    Expert Syst. Appl.

    (2009)
  • S.M. Mousavi et al.

    A new predictive model for compressive strength of HPC using gene expression programming

    Adv. Eng. Softw.

    (2012)
  • V. Chandwani et al.

    Modeling slump of ready mix concrete using genetic algorithms assisted training of artificial neural networks

    Expert Syst. Appl.

    (2015)
  • ChengM.-Y. et al.

    High-performance concrete compressive strength prediction using genetic weighted pyramid operation tree (GWPOT)

    Eng. Appl. Artif. Intell.

    (2014)
  • ChenL. et al.

    Prediction of slump flow of high-performance concrete via parallel hyper-cubic gene-expression programming

    Eng. Appl. Artif. Intell.

    (2014)
  • T. Perkis

    Stack-based genetic programming

    Proceedings of the First IEEE Conference on Evolutionary Computation, 1994. IEEE World Congress on Computational Intelligence

    (1994)
  • L. Spector

    Autoconstructive evolution: push, pushgp, and pushpop

    Proceedings of the Genetic and Evolutionary Computation Conference (GECCO-2001)

    (2001)
  • M. Moini et al.

    Effect of mixture temperature on slump flow prediction of conventional concretes using artificial neural networks

    Aust. J. Civil Eng.

    (2012)
  • S. Haykin

    Neural Networks – A Comprehensive Foundation

    (2004)
  • TsaiH.-C. et al.

    Predicting high-strength concrete parameters using weighted genetic programming

    Eng. Comput.

    (2011)
  • M.A. Hearst et al.

    Support vector machines

    IEEE Intell. Syst. Appl.

    (1998)
  • J.R. Koza

    Genetic Programming – On the Programming of Computers by Means of Natural Selection

    (1993)
  • W. Banzhaf et al.

    Genetic Programming: An Introduction on the Automatic Evolution of Computer Programs and Its Applications

    (1998)
  • Cited by (0)

    Vinícius Veloso de Melo is an associate professor at the Institute of Science and Technology-UNIFESP, São Paulo, Brazil. He obtained his B.Sc. in Computer Science from PUC-MG, Brazil, in 2002; his M.Sc. and Ph.D. in Computer Science from University of São Paulo, Brazil, in 2005 and 2009, respectively. He has published papers in international peer-reviewed journals and conferences. His current research interests are evolutionary computation, metaheuristics, and machine learning.

    Wolfgang Banzhaf is the John R. Koza Chair in Genetic Programming at Michigan State University, USA. Previously, he was University Research Professor in the Department of Computer Science of Memorial University of Newfoundland, Canada, where he served as head of department 2003--2009 and 2012--2016. His research interests are in the field of bio-inspired computing, notably evolutionary computation and complex adaptive systems. Studies of self-organization and the field of Artificial Life are also of very much interest to him. Recently he has become more involved with network research as it applies to natural and man-made systems.

    View full text