Optimal depth estimation by combining focus measures using genetic programming
Introduction
Estimating three-dimensional (3D) shape of an object from its two-dimensional (2D) images is a fundamental problem in computer vision [1], [28], [39]. Broadly, 3D shape recovery algorithms based on the optical reflective model can be categorized into active and passive techniques. In active techniques, light rays are projected whereas, in passive methods, simply the reflection of light rays is captured without any projection. The Shape From Focus (SFF) is one of the passive methods, to estimate 3D structure of the object, based on the image focus analysis. It is a famous one in the paradigm of shape from X, where X denotes the cue used to infer the shape as stereo, motion, shading, de-focus, and focus. The SFF technique has been successfully utilized in many industrial applications, i.e. microelectronics [29], industrial inspection [33], medical diagnostics [5], 3D cameras [23], TFT–LCD color filter manufacturing [2], and comparison of polymers [26].
In SFF techniques, an image sequence is acquired by translating object along the optical axis. It is important to note that acquired images from lenses with limited depth of field have both the areas in and out of focus. However, it is possible to compute the well-focused image from the image sequence taken at different focus levels by computing the high frequency image contents. A criterion, usually known as focus measure, is used to compute the focus quality of each pixel in the image sequence and a focus volume is obtained. An initial depth map is extracted from the focus volume by maximizing the focus measure along the optical axis. In the literature, many focus measure operators are reported in spatial [10], [16], [36] and transform domains [18], [19], [25], [37], [42]. Once an initial depth map is computed, an approximation technique is applied to further refine these results [24], [28], [35]. Most of these techniques use a single focus measure to estimate the initial depth map. Due to the diverse nature of real images, it is difficult for a single focus measure to perform equally well under different scenarios. Moreover, it is hard to select a suitable focus measure for specific conditions. Another drawback with existing techniques is the error introduced in computing initial depth map also propagates to the approximation step. This demands to develop a new generalized optimal depth estimator that can effectively incorporate useful information from more than one focus measures.
In this work, we propose a novel idea of combining the initial depth and focus values extracted from various focus measures. Using this concept, the advantages of one focus measure can overcome the shortcomings of other focus measures. However, the problem is how to combine them in a best possible way. Under such circumstances, we introduce genetic programming (GP) based technique that optimally combines the initial information extracted from one or more focus measures. GP approach works on the principles of natural selection and recombination to search the space for possible solutions under a fitness criterion. Due to the flexibility of adjustable parameters, GP optimization technique [8], [13], [14], [17], [27] is widely used in the applications of image processing [31], pattern recognition [21], and computer vision [22]. In the proposed scheme, GP based Optimal Composite Depth (OCD) functions are developed using homogeneous and heterogeneous feature sets. In the first step, features consisting of initial focus and depth values are computed through existing focus measures. The useful feature information and the random constants are combined through arithmetic operators to develop the OCD function. The composite function is then used to compute the optimal depth map. The improved performance of the developed function is investigated using the synthetic and the real images. Experimental results demonstrate the superiority of the proposed GP based scheme over the conventional focus measures.
In the remainder of the paper, Section 2 describes the SFF scheme and presents a summary of existing focus measures and approximation techniques. Section 3 describes the effect of focus measures on the depth map based on experimental results. Section 4 describes the proposed GP based scheme in detail. Section 5 explains experimental setup and comparative analysis. Finally, Section 6 concludes this study.
Section snippets
Shape From Focus
Shape From Focus techniques retrieve the spatial information from multiple images of the same scene taken at different focus levels. In SFF, the objective is to find out the depth by measuring the distance of well-focused position of every object point from the camera lens. Once distances for all points of the object are known, the 3D shape can be recovered. Fig. 1 shows the basic geometry of image formation of focused and de-focused objects through the convex lens. Suppose the lens is
Focus measure vs depth map
In this section, we will discuss the effect of focus measures on the depth map of an object. Comparative studies of focus measure operators [9], [36], [37], [40] revealed that different focus measures provide different focus values and thus provide different depth values. Practically, it is hard to predict the suitable focus measure among a large list. Many factors including window size, mask size, noise level, illumination, contrast, affect the accuracy of the computed depth map. Some focus
Proposed GP-based scheme
Our aim is to develop GP based depth estimation function, F′: x → y, that maps the useful input information x to the target depth y-values. The proposed scheme is divided into Preprocessing module, GP module and Depth estimation module. In Preprocessing module, training data is formed by computing features of the known object. During GP process, optimal depth estimation function is developed using the training data. Depth estimation module is used to estimate the optimal depth map of the object.
Implementation details
For the development of OCD function, we prepared training data from the simulated object. A sequence of 97 images, each of size 360 × 360, of the simulated cone was generated synthetically by using the simulation software [35]. For each pixel, a homogeneous feature vector x1 is formed by applying F1 focus measure. In this way, feature vectors along with true depth values are used to construct training data of 129,600(=360 × 360) points. To avoid the over fitting problem and to reduce the training
Conclusion
In this paper, we developed an improved performance composite function for optimal depth estimation of real objects through GP based technique. The main advantage of the proposed scheme is that the useful information of individual focus measures is automatically selected and combined during GP evolution cycle. Another benefit is that this generic method does not depend on a specific focus measure. Moreover, the capability of proposed depth estimator can be enhanced by adding more useful
Acknowledgment
This work was supported by BioImaging Research Center at GIST.
References (44)
- et al.
Computer vision methods for optical microscopes
Image and Vision Computing
(2007) - et al.
Dynamic population variation in genetic programming
Information Sciences
(2009) - et al.
Routine high-return human-competitive automated problem-solving by means of genetic programming
Information Sciences
(2008) - et al.
A Bayes-spectral-entropy-based measure of camera focus using a discrete cosine transform
Pattern Recognition Letters
(2006) - et al.
Consideration of illumination effects and optimization of window size for accurate calculation of depth map for 3D shape recovery
Pattern Recognition
(2007) - et al.
A novel algorithm for estimation of depth map using image focus for 3D shape recovery in the presence of noise
Pattern Recognition
(2008) - et al.
Ensemble strategies with adaptive evolutionary programming
Information Sciences
(2010) - et al.
Measure of image sharpness using eigenvalues
Information Sciences
(2007) - et al.
A new focus measure method using moments
Image and Vision Computing
(2000) - et al.
A heuristic approach for finding best focused shape
IEEE Transactions on Circuits and Systems for Video Technology
(2005)
Application of three dimensional shape from image focus in LCD/TFT displays manufacturing
IEEE Transactions on Consumer Electronics
Shape from focus using multilayer feedforward neural networks
IEEE Transactions on Image Processing
Digital Image Processing
An automated microscope for cytologic research a preliminary evaluation
Journal of Histochemistry and Cytochemistry
Three-dimensional shape recovery from the focused-image surface
Optical Engineering
A relevance feedback method based on genetic programming for classification of remote sensing images
Information Sciences
A comparison of different focus functions for use in autofocus algorithms
Cytometry
Robot Vision
Combination and optimization of classifiers in gender classification using genetic programming
International Journal of Knowledge-Based and Intelligent Engineering Systems
Focusing
International Journal of Computer Vision
Size fair and homologous tree genetic programming crossovers
Genetic Programming and Evolvable Machines
Cited by (40)
Shape from focus using gradient of focus measure curve
2023, Optics and Lasers in EngineeringMultiscale fusion and aggregation PCNN for 3D shape recovery
2020, Information SciencesCitation Excerpt :For a simulated object, since the ground-truth is available, the discrepancies between the estimated depth maps and the true map is easy to calculate. We use three objective evaluation criteria: root mean squared error (RMSE), peak signal to noise ratio (PSNR) and correlation [25]. For real objects, it is not easy to evaluate a depth map quantitatively since the ground-truth map does not exist.
3D shape reconstruction from multifocus image fusion using a multidirectional modified Laplacian operator
2020, Pattern RecognitionCitation Excerpt :The initial depth map can be obtained from maximizing these focus measure values. After that, some reconstruction schemes, such as the Gaussian model [14] and genetic programming [15], will make use of the focus information in the image sequence to find the accurate depth map. Optical microscopy is an important tool for high precision analysis and measurement of microscale objects due to its high magnification.
A focus fusion framework with anisotropic depth map smoothing
2015, Pattern RecognitionCan-Evo-Ens: Classifier stacking based evolutionary ensemble system for prediction of human breast cancer using amino acid sequences
2015, Journal of Biomedical Informatics