Multi-objective Cartesian Genetic Programming optimization of morphological filters in navigation systems for Visually Impaired People

https://doi.org/10.1016/j.asoc.2020.106130Get rights and content

Highlights

  • The paper proposes NSGA2CGP, a new evolutionary multi-objective optimization method.

  • NSGA2CGP automatically generates low error x low complexity morphological filters.

  • The filters are used to fix the missing distances in depth images of RGB-D cameras.

  • The proposed error function R2MSESSIM achieved the best error x complexity trade-off.

  • NSGA2CGP filters achieved good performance when deployed in a real navigation system.

Abstract

Navigation systems for Visually Impaired People (VIP) have improved in the last decade, incorporating many features to ensure navigation safety. Such systems often use grayscale depth images to segment obstacles and paths according to distances. However, this approach has the common problem of unknown distances. While this can be solved with good quality morphological filters, these might be too complex and power demanding. Considering navigation systems for VIP rely on limited energy sources that have to run multiple tasks, fixing unknown distance areas without major impacts on power consumption is a definite concern. Multi-objective optimization algorithms might improve filters’ energy efficiency and output quality, which can be accomplished by means of different quality vs. complexity trade-offs. This study presents NSGA2CGP, a multi-objective optimization method that employs the NSGA-II algorithm on top of Cartesian Genetic Programming to optimize morphological filters for incomplete depth images used by navigation systems for VIP. Its goal is to minimize output errors and structuring element complexity, presenting several feasible alternatives combining different levels of filter quality and complexity—both of which affect power consumption. NSGA2CGP-optimized filters were deployed into an actual embedded platform, so as to experimentally measure power consumption and execution time. We also propose two new fitness functions based on existing approaches from literature. Results showed improvements in visual quality, performance, speed and power consumption, thanks to our proposed error function, proving NSGA2CGP as a solid method for developing and evolving efficient morphological filters.

Introduction

Visual impairment is a common condition throughout the world. Lack of vision, from moderate impairment to blindness, affects 253 million people, among which 36 million are fully blind, according to the World Health Organization [1]. Significant advances spanning several research areas are helping Visually Impaired People/Person (VIP) accomplish most daily tasks, such as using computers, smartphones, walk safely, etc. [2], [3].

Vision-based navigation systems use cameras to gather images of the VIP’s path and process them by means of computer vision operations [4]. The most common cameras used in VIP navigation systems are known as RGB-D cameras, which have built-in modules to generate environmental depth maps, by means of well-known methods such as structured light, time-of-flight and stereoscopy [5], [6], [7]. All of these methods, however, share the common problem of unknown pixel distance, which leads to areas of the depth map being set to zero distance, resulting in several navigation system issues, such as wrong segmentation. Depth maps can also be represented as grayscale images, with pixel colors defined according to distance, i.e., lighter grays for objects that are farther, darker grays for objects that are nearer (or vice-versa), and unknown distances simply displayed as black pixels.

Mathematical morphology and its operations can alleviate this problem by employing basic operations, such as dilation and erosion, with several shapes of structuring elements to handle a broad variety of images, containing several types of noises and/or imperfections [8], [9]. Morphological filters may be generated through genetic programming algorithms, which use the main concepts of evolution theory – chromosomes, genes, alleles, mutations etc.– to evolve individuals of a population (in this case, filters) and achieve optimal results. Cartesian Genetic Programming(CGP) is a particular type of Graph-Based Genetic Programming where chromosomes are represented by a 2D array of integers (genes). Each integer may reference other genes or terminals (raw inputs), but it also may reference a function to process those inputs, resulting in one or more outputs [10].

Hardware setups for navigation systems must be portable, imposing an important constraint: power. Since the equipment cannot remain permanently connected to a power outlet, navigation systems need to rely on limited power supplies such as batteries [9], [11], [12]. Thus, an efficient use of available energy resources is required. However, the complexity of morphological operations might be a concern, due to the fact that personal navigation systems have to rely on small batteries for power, and complex sets of morphological operations might consume more energy and it adds an additional constraint to morphological filters: they must fix all the missing depth pixels whilst consuming as minimum of energy as possible, turning the creation of morphological filters into a problem with two conflicting objectives.

Multi-Objective Optimization (MOO) algorithms, when applied to morphological filters, should be able to evaluate and rank every filter according its quality (low error rate) and complexity (low complexity) within a feasible trade-off between them. The literature describes many evolutionary multi-objective optimization algorithms, such as the Nondominated Sorting Genetic Algorithm II (NSGA-II). NSGA-II employs the principles of nondominated fronts and crowding-distance [13] by using an elitist approach to create multiple nondominated fronts of parents and offsprings, and sort them by means of crowding-distance computation. Since NSGA-II has an evolutionary approach, it is possible to adapt it to work with a variety of genetic programming algorithms like CGP and evolve optimized sets of solutions, thereby being a good choice for our approach. Also, NSGA-II was chosen due its popularity among MOO algorithms as it also was reported to have superior performance with large populations, providing high diversity fronts [14], [15], [16], [17], [18].

The lack of depth estimation translates to black pixels in depth images and it has a direct impact on vision-based navigation systems, compromising (partially or totally) a depth image and the segmentation process, leading users to unsafe directions and/or presenting them wrong or misplaced obstacles. Even so, there is also the concern about how complex and power consuming is to fix depth images due the limited power sources that navigation systems rely on. Thereby, the main problem addressed by this work is the process to fix depth images containing incomplete depth data and overcome the challenge of doing so in a efficient manner in terms of complexity, a multi-objective problem. This paper proposes a method for multi-objective optimization of morphological filters, the NSGA2CGP Cartesian genetic programming MOO algorithm. The main goal of NSGA2CGP is to provide means to implement and generate sets of feasible and optimized morphological filters for depth images considering two objectives: minimum error and minimum complexity, aiming energy efficient filters. This method was devised and then experimentally applied to a specific use case, consisting of depth image scenes from university environments. In addition to NSGA2CGP we also proposed and applied two new fitness functions to evaluate images and calculate the error between them. The main contribution of this work is to provide a novel method to generate low-error x low-complexity optimized morphological filters to fix unknown distance areas of depth images in the context of vision-based navigation systems for VIP with multiple feasible solutions that fits different processing capabilities through varying trade-offs between error and complexity.

This paper is structured as follows: Section 2 presents literature approaches related to this work. The proposed method, NSGA2CGP, is presented in Section 3, followed by experiments in Section 4. Finally, results and conclusion are presented in Sections 5 Results and discussion, 6 Conclusion, respectively.

Section snippets

Related works

The literature presents a shortlist of approaches that combine CGP and NSGA-II. Most of them are based on theoretical digital circuit problems, jointly applying these techniques to improve circuit efficiency, reduce instruction delay, or minimize error [19], [20]. They are improved for their own domain and were not conceived to work other domains such as images.

Kalkreuth and colleagues [21] proposed a modified version of the NSGA-II algorithm based on CGP to evolve small and efficient programs

NSGA2CGP

NSGA2CGP is a modified version of the existing NSGA-II MOO algorithm. It uses Cartesian genetic programming – instead of a classic genetic algorithm – to optimize morphological filters for depth maps produced by VIP navigation systems. The method is intended to automatically generate and optimize filters based on training sets containing images from possible navigation areas, providing developers with the ability to choose the filter best-suited to the hardware specifications (battery capacity,

Experiments

Once implemented, the NSGA2CGP method was submitted to an extensive series of experiments, in order to evaluate its behavior, fine-tune code and verify the quality of results. Thus, we carried out experiments to test if the method was able to optimize solutions to adequately filter grayscale depth images, reducing or eliminating black areas and minimizing their interference over the segmentation processes of a future navigation system.

Navigation systems developed for people with visual

Results and discussion

As presented in the previous section, experiments were conducted to evaluate and validate NSGA2CGP using multiple indicators. Raw training results for execution time and energy consumption, among others, were analyzed first, in order to choose candidates for the next measurements. Due to the numerous possible parameter combinations, here we chose to present only the best settings. Thus, Table 3 identifies each training setup and its parameters. Particularities such as error function are

Conclusion

This paper presented a multi-objective optimization method for depth image morphological filters, NSGA2CGP, using the NSGA-II algorithm and the Cartesian Genetic Programming technique. It was used to optimize error rates and structuring element complexity, so as to achieve a deployable, energy- and time-efficient feasible solution for VIP navigation systems.

The NSGA2CGP was constructed according to the specifications in Section 3 and was used to train and evolve multiple scenarios of

CRediT authorship contribution statement

Antonio Miguel Batista Dourado: Conceptualization, Methodology, Software, Validation, Investigation, Writing - original draft, Visualization, Project administration. Emerson Carlos Pedrino: Formal analysis, Writing - review & editing, Supervision.

Declaration of Competing Interest

No author associated with this paper has disclosed any potential or pertinent conflicts which may be perceived to have impending conflict with this work. For full disclosure statements refer to https://doi.org/10.1016/j.asoc.2020.106130.

Acknowledgments

This study was financed by the Coordination for the Improvement of Higher Education Personnel (CAPES), Brazil – Finance Code 001, and by the São Paulo Research Foundation (FAPESP), Brazil – Project 2017/26421-3. We also thank the Federal Institute of São Paulo and the PARA-D.V. nongovernmental organization, whose contribution made this work possible.

References (57)

  • SilbermanN. et al.

    Indoor scene segmentation using a structured light sensor

  • LiL.

    Time-of-Flight Camera–An IntroductionTech. Rep. January

    (2014)
  • LeeS. et al.

    Real-time disparity estimation algorithm for stereo camera systems

    IEEE Trans. Consum. Electron.

    (2011)
  • PedrinoE.C. et al.

    Real-time morphological pipeline architecture using high-capacity programmable logical devices

    J. Electron. Imaging

    (2007)
  • A.M.B. Dourado, E.C. Pedrino, Embedded navigation and classification system for assisting visually impaired people, in:...
  • MillerJ.F.

    Cartesian genetic programming

  • KatzB.F.G. et al.

    NAVIG: augmented reality guidance system for the visually impaired

    Virtual Real.

    (2012)
  • KumarA. et al.

    An electronic travel aid for navigation of visually impaired persons

  • DebK. et al.

    A fast and elitist multiobjective genetic algorithm: NSGA-II

    IEEE Trans. Evol. Comput.

    (2002)
  • ShehadehH.A. et al.

    Optimal placement of near ground VHF/UHF radio communication network as a multi objective problem

    Wirel. Pers. Commun.

    (2019)
  • KavunnikovaE.A. et al.

    Comparison of design optimization algorithms of a multiply fractured horizontal well

    J. Phys. Conf. Ser.

    (2019)
  • PooranianZ. et al.

    Economical and environmental operation of smart networked microgrids under uncertainties using NSGA-II

  • GarciaS. et al.

    Comparison of multi-objective evolutionary algorithms to solve the modular cell design problem for novel biocatalysis

    Processes

    (2019)
  • HilderJ. et al.

    Use of a multi-objective fitness function to improve cartesian genetic programming circuits

  • HrbacekR. et al.

    Automatic design of approximate circuits by means of multi-objective evolutionary algorithms

  • KalkreuthR. et al.

    More efficient evolution of small genetic programs in Cartesian Genetic Programming by using genotypie age

  • KaufmannP. et al.

    Combining local and global search: A multi-objective evolutionary algorithm for cartesian genetic programming

  • ParisP. et al.

    Automatic learning of image filters using Cartesian genetic programming

    Integr. Comput.-Aided Eng.

    (2015)
  • Cited by (10)

    • Multi-type decision fusion network for visual Q&A

      2021, Image and Vision Computing
      Citation Excerpt :

      It aims to answer the questions based on the visual evidences, which refer to the question-related object regions and relation information of an image. With the recent progress in VQA task and some associated vision or natural language tasks such as image caption [5] and image text etc., various real-world applications have arisen, such as the intelligent navigator for the visually impaired people, the image data retrieval tool based on the understanding of natural language and the chatbot for human in [3,6–8]. The humans The humans always answer various visual questions in different perspectives, which means they can search for a reasoning cue to answer the question.

    • A scheduling and planning method for geological disasters

      2021, Applied Soft Computing
      Citation Excerpt :

      The vehicles that park at the “Temporary stop” (37, 25, 33, 8, 18 and 35) and depots continue to serve customers. To illustrate the effectiveness of the algorithm in solving multi-objective EMS, the HACO algorithm was compared with widely used algorithms, the non-dominated sorting genetic algorithm II (NSGA-II) [43–45] and multi-objective ant colony optimization algorithm (MOACO) [46,47]. The pseudocodes of MACAO and NSGA-II are shown as follows.

    • Multi-objective genetic programming for feature learning in face recognition

      2021, Applied Soft Computing
      Citation Excerpt :

      In recent years, evolutionary multi-objective optimisation algorithms have been widely applied to solve many real-world tasks, such as network planning [17], bound-constrained real-world problems [18], and spread spectrum radar polyphase code design problem [19]. As evolutionary algorithms, multi-objective GP algorithms have been proposed for symbolic regression and modelling [20], and morphological filters optimisation [21]. It can also be found that multi-objective GP has been applied for feature extraction and construction [22–24].

    View all citing articles on Scopus
    View full text