Multi-objective Cartesian Genetic Programming optimization of morphological filters in navigation systems for Visually Impaired People
Introduction
Visual impairment is a common condition throughout the world. Lack of vision, from moderate impairment to blindness, affects 253 million people, among which 36 million are fully blind, according to the World Health Organization [1]. Significant advances spanning several research areas are helping Visually Impaired People/Person (VIP) accomplish most daily tasks, such as using computers, smartphones, walk safely, etc. [2], [3].
Vision-based navigation systems use cameras to gather images of the VIP’s path and process them by means of computer vision operations [4]. The most common cameras used in VIP navigation systems are known as RGB-D cameras, which have built-in modules to generate environmental depth maps, by means of well-known methods such as structured light, time-of-flight and stereoscopy [5], [6], [7]. All of these methods, however, share the common problem of unknown pixel distance, which leads to areas of the depth map being set to zero distance, resulting in several navigation system issues, such as wrong segmentation. Depth maps can also be represented as grayscale images, with pixel colors defined according to distance, i.e., lighter grays for objects that are farther, darker grays for objects that are nearer (or vice-versa), and unknown distances simply displayed as black pixels.
Mathematical morphology and its operations can alleviate this problem by employing basic operations, such as dilation and erosion, with several shapes of structuring elements to handle a broad variety of images, containing several types of noises and/or imperfections [8], [9]. Morphological filters may be generated through genetic programming algorithms, which use the main concepts of evolution theory – chromosomes, genes, alleles, mutations etc.– to evolve individuals of a population (in this case, filters) and achieve optimal results. Cartesian Genetic Programming(CGP) is a particular type of Graph-Based Genetic Programming where chromosomes are represented by a 2D array of integers (genes). Each integer may reference other genes or terminals (raw inputs), but it also may reference a function to process those inputs, resulting in one or more outputs [10].
Hardware setups for navigation systems must be portable, imposing an important constraint: power. Since the equipment cannot remain permanently connected to a power outlet, navigation systems need to rely on limited power supplies such as batteries [9], [11], [12]. Thus, an efficient use of available energy resources is required. However, the complexity of morphological operations might be a concern, due to the fact that personal navigation systems have to rely on small batteries for power, and complex sets of morphological operations might consume more energy and it adds an additional constraint to morphological filters: they must fix all the missing depth pixels whilst consuming as minimum of energy as possible, turning the creation of morphological filters into a problem with two conflicting objectives.
Multi-Objective Optimization (MOO) algorithms, when applied to morphological filters, should be able to evaluate and rank every filter according its quality (low error rate) and complexity (low complexity) within a feasible trade-off between them. The literature describes many evolutionary multi-objective optimization algorithms, such as the Nondominated Sorting Genetic Algorithm II (NSGA-II). NSGA-II employs the principles of nondominated fronts and crowding-distance [13] by using an elitist approach to create multiple nondominated fronts of parents and offsprings, and sort them by means of crowding-distance computation. Since NSGA-II has an evolutionary approach, it is possible to adapt it to work with a variety of genetic programming algorithms like CGP and evolve optimized sets of solutions, thereby being a good choice for our approach. Also, NSGA-II was chosen due its popularity among MOO algorithms as it also was reported to have superior performance with large populations, providing high diversity fronts [14], [15], [16], [17], [18].
The lack of depth estimation translates to black pixels in depth images and it has a direct impact on vision-based navigation systems, compromising (partially or totally) a depth image and the segmentation process, leading users to unsafe directions and/or presenting them wrong or misplaced obstacles. Even so, there is also the concern about how complex and power consuming is to fix depth images due the limited power sources that navigation systems rely on. Thereby, the main problem addressed by this work is the process to fix depth images containing incomplete depth data and overcome the challenge of doing so in a efficient manner in terms of complexity, a multi-objective problem. This paper proposes a method for multi-objective optimization of morphological filters, the NSGA2CGP Cartesian genetic programming MOO algorithm. The main goal of NSGA2CGP is to provide means to implement and generate sets of feasible and optimized morphological filters for depth images considering two objectives: minimum error and minimum complexity, aiming energy efficient filters. This method was devised and then experimentally applied to a specific use case, consisting of depth image scenes from university environments. In addition to NSGA2CGP we also proposed and applied two new fitness functions to evaluate images and calculate the error between them. The main contribution of this work is to provide a novel method to generate low-error x low-complexity optimized morphological filters to fix unknown distance areas of depth images in the context of vision-based navigation systems for VIP with multiple feasible solutions that fits different processing capabilities through varying trade-offs between error and complexity.
This paper is structured as follows: Section 2 presents literature approaches related to this work. The proposed method, NSGA2CGP, is presented in Section 3, followed by experiments in Section 4. Finally, results and conclusion are presented in Sections 5 Results and discussion, 6 Conclusion, respectively.
Section snippets
Related works
The literature presents a shortlist of approaches that combine CGP and NSGA-II. Most of them are based on theoretical digital circuit problems, jointly applying these techniques to improve circuit efficiency, reduce instruction delay, or minimize error [19], [20]. They are improved for their own domain and were not conceived to work other domains such as images.
Kalkreuth and colleagues [21] proposed a modified version of the NSGA-II algorithm based on CGP to evolve small and efficient programs
NSGA2CGP
NSGA2CGP is a modified version of the existing NSGA-II MOO algorithm. It uses Cartesian genetic programming – instead of a classic genetic algorithm – to optimize morphological filters for depth maps produced by VIP navigation systems. The method is intended to automatically generate and optimize filters based on training sets containing images from possible navigation areas, providing developers with the ability to choose the filter best-suited to the hardware specifications (battery capacity,
Experiments
Once implemented, the NSGA2CGP method was submitted to an extensive series of experiments, in order to evaluate its behavior, fine-tune code and verify the quality of results. Thus, we carried out experiments to test if the method was able to optimize solutions to adequately filter grayscale depth images, reducing or eliminating black areas and minimizing their interference over the segmentation processes of a future navigation system.
Navigation systems developed for people with visual
Results and discussion
As presented in the previous section, experiments were conducted to evaluate and validate NSGA2CGP using multiple indicators. Raw training results for execution time and energy consumption, among others, were analyzed first, in order to choose candidates for the next measurements. Due to the numerous possible parameter combinations, here we chose to present only the best settings. Thus, Table 3 identifies each training setup and its parameters. Particularities such as error function are
Conclusion
This paper presented a multi-objective optimization method for depth image morphological filters, NSGA2CGP, using the NSGA-II algorithm and the Cartesian Genetic Programming technique. It was used to optimize error rates and structuring element complexity, so as to achieve a deployable, energy- and time-efficient feasible solution for VIP navigation systems.
The NSGA2CGP was constructed according to the specifications in Section 3 and was used to train and evolve multiple scenarios of
CRediT authorship contribution statement
Antonio Miguel Batista Dourado: Conceptualization, Methodology, Software, Validation, Investigation, Writing - original draft, Visualization, Project administration. Emerson Carlos Pedrino: Formal analysis, Writing - review & editing, Supervision.
Declaration of Competing Interest
No author associated with this paper has disclosed any potential or pertinent conflicts which may be perceived to have impending conflict with this work. For full disclosure statements refer to https://doi.org/10.1016/j.asoc.2020.106130.
Acknowledgments
This study was financed by the Coordination for the Improvement of Higher Education Personnel (CAPES), Brazil – Finance Code 001, and by the São Paulo Research Foundation (FAPESP), Brazil – Project 2017/26421-3. We also thank the Federal Institute of São Paulo and the PARA-D.V. nongovernmental organization, whose contribution made this work possible.
References (57)
- et al.
A mathematical model for project scheduling and material ordering problem with sustainability considerations: A case study in Iran
Comput. Ind. Eng.
(2019) - et al.
Color image segmentation based on multiobjective artificial bee colony optimization
Appl. Soft Comput. J.
(2015) - et al.
Multi-objective nature-inspired clustering and classification techniques for image segmentation
Appl. Soft Comput. J.
(2011) - et al.
Structure guided fusion for depth map inpainting
Pattern Recognit. Lett.
(2013) - et al.
A new method for inpainting of depth maps from time-of-flight sensors based on a modified closing by reconstruction algorithm
J. Vis. Commun. Image Represent.
(2017) - et al.
Automatic acquisition of hierarchical mathematical morphology procedures by genetic algorithms
Image Vis. Comput.
(1999) Blindness and vision impairment
- et al.
Increasing usability when interacting through screen readers
Univ. Access Inf. Soc.
(2004) - et al.
Slide rule : Making mobile touch screens accessible to blind people using multi-touch interaction techniques
Design
(2008) - et al.
Navigation systems for individuals with visual impairment: A survey
Indoor scene segmentation using a structured light sensor
Time-of-Flight Camera–An IntroductionTech. Rep. January
Real-time disparity estimation algorithm for stereo camera systems
IEEE Trans. Consum. Electron.
Real-time morphological pipeline architecture using high-capacity programmable logical devices
J. Electron. Imaging
Cartesian genetic programming
NAVIG: augmented reality guidance system for the visually impaired
Virtual Real.
An electronic travel aid for navigation of visually impaired persons
A fast and elitist multiobjective genetic algorithm: NSGA-II
IEEE Trans. Evol. Comput.
Optimal placement of near ground VHF/UHF radio communication network as a multi objective problem
Wirel. Pers. Commun.
Comparison of design optimization algorithms of a multiply fractured horizontal well
J. Phys. Conf. Ser.
Economical and environmental operation of smart networked microgrids under uncertainties using NSGA-II
Comparison of multi-objective evolutionary algorithms to solve the modular cell design problem for novel biocatalysis
Processes
Use of a multi-objective fitness function to improve cartesian genetic programming circuits
Automatic design of approximate circuits by means of multi-objective evolutionary algorithms
More efficient evolution of small genetic programs in Cartesian Genetic Programming by using genotypie age
Combining local and global search: A multi-objective evolutionary algorithm for cartesian genetic programming
Automatic learning of image filters using Cartesian genetic programming
Integr. Comput.-Aided Eng.
Cited by (10)
Towards interactive customization of multimodal embedded navigation systems for visually impaired people
2023, International Journal of Human Computer StudiesMulti-type decision fusion network for visual Q&A
2021, Image and Vision ComputingCitation Excerpt :It aims to answer the questions based on the visual evidences, which refer to the question-related object regions and relation information of an image. With the recent progress in VQA task and some associated vision or natural language tasks such as image caption [5] and image text etc., various real-world applications have arisen, such as the intelligent navigator for the visually impaired people, the image data retrieval tool based on the understanding of natural language and the chatbot for human in [3,6–8]. The humans The humans always answer various visual questions in different perspectives, which means they can search for a reasoning cue to answer the question.
A scheduling and planning method for geological disasters
2021, Applied Soft ComputingCitation Excerpt :The vehicles that park at the “Temporary stop” (37, 25, 33, 8, 18 and 35) and depots continue to serve customers. To illustrate the effectiveness of the algorithm in solving multi-objective EMS, the HACO algorithm was compared with widely used algorithms, the non-dominated sorting genetic algorithm II (NSGA-II) [43–45] and multi-objective ant colony optimization algorithm (MOACO) [46,47]. The pseudocodes of MACAO and NSGA-II are shown as follows.
Multi-objective genetic programming for feature learning in face recognition
2021, Applied Soft ComputingCitation Excerpt :In recent years, evolutionary multi-objective optimisation algorithms have been widely applied to solve many real-world tasks, such as network planning [17], bound-constrained real-world problems [18], and spread spectrum radar polyphase code design problem [19]. As evolutionary algorithms, multi-objective GP algorithms have been proposed for symbolic regression and modelling [20], and morphological filters optimisation [21]. It can also be found that multi-objective GP has been applied for feature extraction and construction [22–24].
Multi-obstacle aware smart navigation system for visually impaired people in fog connected IoT-cloud environment
2022, Health Informatics Journal