Is depth information and optical flow helpful for visual control?

Introduction

With this contribution, we investigated whether depth and motion provide an evolutionary advantage compared to color alone. As a test environment, we have used The Open Racing Car Simulator (TORCS) [1]. Simulated evolution [2] was used to evolve control algorithms for the racing car. These algorithms use screen grabs from the racing car simulator. The screen grabs are processed using elementary computer vision operators. The output of these algorithms control the steering wheel as well as gas/brakes of the car [3]. OpenCV [4, 5], an open source library for computer vision, was used for image processing. Genetic programming [6–8] was used to evolve visual algorithms. We will see that significantly better solutions are found if color, depth, and optical flow are all available.

This article is structured as follows. The next section gives a brief introduction to the visual system. It is followed by the description of the racing car simulator, explanation how data from the simulator is used to compute optical flow, and a brief introduction to genetic programming. Related work on visual control using genetic programming is discussed as well as how genetic programming is used to evolve visual control algorithms. Finally, the study’s results and conclusions are given.

The visual system

The human visual system was shaped through natural evolution [9, 10]. Visual processing starts with light entering the eye. This light is measured by two different types of receptors inside the retina [11, 12]: rods and cones. The rods mediate vision when little light is available. They have a much higher sensitivity than the cones. Cones are in charge of color vision. Three different types of cones exist, which absorb light mainly in the red, green, and blue parts of the visual spectrum.

Some preprocessing occurs inside the retina. Information flows from the retinal receptors to the retinal ganglion cells. This information exits the eye at the blind spot, passes through the lateral geniculate nucleus, and then reaches the primary visual cortex or area V1. Area V1 is highly structured [13]. Cells within ocular dominance segments respond primarily to stimuli from one or the other eye. V1 contains columns with orientation-sensitive cells. Blobs with color or lightness-sensitive cells also exist. Visual information is analyzed using a retinotopic map with respect to different aspects. Indeed, the entire visual cortex is highly structured.

Color, shape, and motion appear to be processed by separate visual areas [14, 15]. Color is a product of the brain. It is processed in visual area V4. Shape is processed in V3, and motion is processed in V5. A dedicated area for face and object recognition also exists. It is also interesting that color and motion are not perceived synchronously. Moutoussis and Zeki [16] demonstrated that color is perceived earlier than motion. The brain appears to bind visual attributes that are perceived together.

TORCS

TORCS [1] is an open-source racing car three-dimensional (3D) simulation. A sample screenshot is shown in Figure 1. We have used this simulator as a test environment to evaluate whether depth information and optical flow are helpful for visual control. Currently, Berhard Wymann maintains the TORCS project. Its original creators were Eric Espié and Christophe Guionneau. The TORCS simulator provides several different racing tracks. A player can choose among different cars when playing the game. Several different opponents are available to race against. A split-screen mode is also available. Up to four human players are supported.

Figure 1:

The Open Racing Car Simulator.

Supported controls are a joystick, mouse, and keyboard. Some steering wheels are also supported. The game features realistic 3D graphics, lighting, smoke, and skid marks. Game physics include simulation of collisions, tire and wheel properties, and a simple damage model. It even includes a simple aerodynamic model with slip-streaming and ground effects. TORCS was used in several different scientific competitions [17, 18]. For these competitions, participants are developing artificial intelligence methods that drive the racing car along its track. Usually, a client-server architecture is used and competition participants develop a client that sends its control commands to the TORCS server.

We have modified TORCS slightly to use it for our experiments. We extract screen grabs from the graphics buffer. A dense depth map is extracted form the so-called Z-buffer of the graphics library [19], i.e. both color and depth are readily available form the graphics context. In addition to color and depth, we also provide optical flow to the visual control algorithms. How we compute optical flow is described in the next section.

Computing optical flow

Optical flow describes the pattern of motion of objects that are seen on the screen. This information is very helpful for visual control. We would like to estimate a dense flow field, i.e. optical flow for each image pixel. Let v(x, y, t)=(v_x, v_y) be the optical flow, estimated for image pixel (x, y) in an image taken at time t, then the image content in the vicinity of pixel (x, y) will be found at position (x+v_xΔt, y+v_yΔt) in an image taken at time t+Δt provided that the object is moving with constant velocity across the screen.

Several different methods have been developed to compute optical flow from visual input [20–23]. In recent years, the accuracy of the methods has improved considerably. Many methods for computing optical flow are based on partial derivatives. However, block-based methods are also used. Block-based methods search for pixels within an area of a given pixel in a subsequent image to determine image motion.

Estimating optical flow from two subsequent images is an expensive image operation. Therefore, we have used the depth map and the known motion of the race car to compute optical flow. The depth map is readily available from the graphics library. It is a by-product of rendering the scene. The depth map contains, for each image pixel, the distance from the object to the camera along the Z-axis. The motion of the race car is available directly from the race car simulator.

Let d(x, y)=d_x,y be the depth map of the current image shown on the screen. We assume that all screen coordinates are specified relative to the center of the screen. Let f be the focal length of the camera. The location of an object of the scene that is shown at image pixel (x, y) has coordinates (X_S, Y_S, Z_S) inside the camera coordinate system centered on the car driver.

The coordinates (X_S, Y_S, Y_S) are relative to the viewer sitting inside the car, i.e. these are eye coordinates.

Let R be the inverse 3×3 rotation matrix that describes the rotatory motion of the racing car from one time step of the simulation to the next. Let D be the inverse vector that describes the translatory motion of the racing car from one time step of the simulation to the next. Hence, after the racing car has moved, the point (X_S, Y_S, Z_S) will have moved to a location (X′S, Y′S, Z′S) relative to the eye of the driver.

The coordinate (X′S, Y′S, Z′S) can be projected onto the screen using the known focal length of the camera. Let (x′, y′) be the screen coordinates of (X′S, Y′S, Z′S), then we have

Optical flow can then be computed by subtracting the screen coordinate before the racing car has moved from the screen coordinate after the car has moved.

As the depth map and the known motion of the racing car are correct, the optical flow will also be correct. Figure 2 shows the computed optical flow for a sample image.

It would be possible to compute optical flow directly from the input images using an algorithm based on partial derivatives or using block based methods. However, this would take considerably more computing resources and also would have the disadvantage that the estimated optical flow would not be 100% correct for all image pixels.

Figure 2:

Computing optical flow from game angine data.

(A) input image, (B) depth map, and (C) optical flow computed using the depth map and the known ego-motion of the car.

Next, we will describe genetic programming, which we have used to evolve visual control algorithms.

Genetic programming

Genetic programming [6–8] is an evolutionary algorithm. Evolutionary algorithms use simulated evolution to solve optimization problems. Such algorithms work with a population of individuals. Each individual represents a possible solution to the optimization problem. Darwinian selection is used to select above average individuals to create an offspring population, i.e. a new generation of individuals.

In genetic programming, individuals are represented as trees. The fitness of an individual describes how well this individual solves the given problem. The main operators of an evolutionary algorithms are selection, reproduction, and variation. Above-average individuals are selected to create offspring. For our experiments, we have used four genetic operators: reproduction, mutation, ephemeral random constant (ERC) mutation, and crossover. Each genetic operator is applied with a certain probability p_rep, p_mut, p_{ERC mut}, and p_cross respectively. These four probabilities sum to 1.

Individuals of the first generation are created using the so-called ramped half and half initialization [6]. To create an offspring, one genetic operator is randomly selected (using the four probabilities). The reproduction operator simply creates a copy of the genetic material of the individual, i.e. the tree. Mutation, ERC mutation, and crossover create offspring that are similar but not identical to their parents. Depending on the type of operator, one or two parents are selected from the population. Typically, tournament selection is used to select new parents. For tournament selection, n_T individuals are selected with uniform probability from the population. These n_T individuals form a tournament. The individual with highest fitness is the winner of the tournament and becomes a parent. This parent will then create offspring using one of the genetic operators.

The genetic operators are illustrated in Figure 3. Figure 3A shows the reproduction operator that is applied with probability p_rep. Figure 3B shows the mutation operator that is applied with probability p_mut. Figure 3C shows the ERC mutation operator that is applied with probability p_{ERC mut}. Figure 3D shows the crossover operator that is applied with probability p_cross.

Figure 3:

Genetic operators.

(A) reproduction, (B) mutation, (C) ERC mutation, and (D) crossover.

If the reproduction operator is applied, then a copy of the parent individual is created. Next, a node of the tree is randomly selected. Internal nodes are selected with probability 0.9, whereas external nodes are selected with probability 0.1. Finally, the selected node is replaced with a randomly generated sub-tree. The method that is used to create this sub-tree is the same method that is used to create the individuals of the first generation. For ERC-mutation, a single node which contains a so called ephemeral random constant (ERC) is selected. All ERCs located within this subtree are mutated. We originally intended to use Gaussian mutation (like an evolution strategy [24]) to slightly alter this ERC value, i.e. v:=v·e^0.01z where v is the original value of the constant and z is a normally distributed random value with mean 0 and standard deviation 1. However, we have actually used v:=v·0.01 which pulls the ERC towards zero and may also change the sign of the constant. For crossover, two parent individuals are selected. Next, two nodes are randomly selected (one for each tree). Again, internal nodes are selected with probability 0.9, whereas external nodes are selected with probability 0.1. Then the two sub-trees (together with the selected nodes) are exchanged between the two individuals. Trees are limited to a maximum depth of 17.

Whenever an offspring is created, it is inserted into the next generation of individuals. The process of selecting a genetic operator and creating offspring continues until the next generation is filled. Usually, an evolutionary run is terminated after a certain number of generations have been created. The individual with highest fitness that was found during all these generations is the solution that solves our problem best.

For our experiments, we have used the Evolutionary Computation Library ECJ developed by Luke [25]. Development of ECJ started in 1998 and is a mature library for evolutionary computation.

Visual control using genetic programming

Our racing car is controlled through visual input alone. We only use the images obtained from screen grabs and the optical flow. Data from the game engine is not used to control the car. It is only used to compute optical flow as described above.

Genetic programming has been used by Winkeler and Manjunath [26] for object detection. Johnson et al. [27] used it to evolve visual routines. Ebner and Tiede [28] have previously evolved controllers for TORCS using genetic programming. However, for this work, input was taken directly from the game engine and not from screen grabs. Koutník et al. [29] have used an evolutionary algorithm to evolve compressed encodings of a recursive neural network to control a racing car in TORCS. Tanev and Shimohara [30, 31] have used a genetic algorithm to evolve parameters that will control an actual scale model of a car using visual input from an overhead camera.

Other researchers have used Atari video games for training game players [32]. Hausknecht et al. [33, 34] evaluated neuro-evolutionary methods for general game playing of Atari video games. They found that HyperNEAT was the only neuro-evolution algorithm able to play based on raw-pixel input from the games. Mnih et al. [35, 36] created a deep neural network that was trained using reinforcement learning. It was able to achieve a level comparable to human players. Deep learning in combination with Monte-Carlo tree search planning was used by Guo et al. [37]. Parker and Bryant [38, 39] evolved controllers for Quake II, which used only visual input.

Materials and methods

We are using strongly typed genetic programming [40] to evolve two trees. The first tree is used to control the steering wheel. The second tree is used to control the velocity of the racing car. The terminal symbols are shown in Table 1. We work with two return types: float and image. The only terminal symbol returning a floating point value is an ERC. An ERC is a random floating point value from the range [0, 1]. Once a node with an ERC is created, it stays constant throughout the life of the node. However, it may be modified by the ERC mutation operator.

The remaining terminal symbols provide access to visual information obtained via screen grabs from the game engine. This screen grab is scaled down to one third of its original size. All pixel values are transformed to the range [0, 1]. All terminal symbols returning image data provide single band images: red channel (imageR), green channel (imageG), blue channel (imageB), cyan channel (imageC), magenta channel (imageM), yellow channel (imageY), and gray channel (imageGray). The depth map of this input image is available through the terminal symbol (depthMap). Optical flow is computed using the depth map and the known ego-motion of the car, which is available from the game engine. As optical flow is a two-dimensional vector, the x-component of this vector is made available through the terminal opticalFlowX, and the y-component is made available through the terminal opticalFlowY. All image data is downscaled to one third of this size of the original image. Pixel values are scaled to the range [0, 1].

The set of elementary functions is shown in Tables 2 and 3. Table 2 shows elementary functions that return a floating point value. Table 3 shows elementary functions that return an entire image. We have used standard arithmetic functions such as addition and multiplication, computation of minimum and maximum. We have also included functions that search for maximum and minimum values inside the image. These functions return either the x or the y coordinate of the position where the extremum was found. A Gaussian filter is also available. If we apply a Gaussian filter to a gray-scale input image and then a function that locates the maximum, we can locate the brightest point in the image. The function extractNAME extracts smaller regions from the input image. The size of the region is one third of the image. The location of the region can be specified through the parameters of the function. All bands (see terminal symbols) are available for this extraction operation.

This functions is very useful to control the steering wheel of the car. This is illustrated in Figure 4. Suppose we extract a region from the depth map from the left-hand side of the image and we extract another region from the depth map from the right-hand side of the image. If we apply the average function, which computes the average depth within these two areas, then we can compare both average depths to control the steering wheel. This simple control algorithm will drive the car in the direction where more space is available.

Figure 4:

Extracting sub-regions from the visual input is helpful for visual control.

(A) Sample tree which evaluates information from the depth map, (B) input image, and (C) depth map with regions.

We have carried out four sets of experiments to evaluate whether depth information is helpful in controlling the racing car. For our experiments, we used the same basic set of elementary functions but varied the input information that was made available to the evolved individuals. For experiment A, only color information was available. For experiment B, only the depth map was available. For experiment C, only optical flow was provided. Finally, for experiment D, all of the visual information (color, depth, and optical flow) was provided. The terminal symbols and elementary functions for the four experiments are shown in Table 4. Note that some of the functions listed under “All” accept both floating point values and images as input.

The track that we have used for all of our experiments is shown in Figure 5. The red arrow illustrates the direction in which the race will start. A path taken by an evolved individual is shown overlaid on this track (green line). The end of this path is marked with a green cross. At this point, the evolved driver lost control of its car and crashed into the border of the track.

Figure 5:

Track used for our experiments.

(A) Track and (B) path taken by an evolved individual (green line). Driving direction (red arrow).

The task is to evolve visual controllers that will drive the car along the track. Therefore, fitness is computed by considering distance traveled along the track. In addition, the damage attained is also used for the fitness computation. Individuals that stay away from the border of track and manage to avoid damage will receive higher fitness values. Let d be the distance traveled along the track (in meters). Let a be the amount of damage attained (with range [0, 1], where 1 is a completely damaged car). Then fitness f_i of individual i is given as

A controller is disqualified if it (a) drives along the track in the wrong direction or (b) does not use the steering wheel, i.e. the first tree returns a constant value or (c) does not use the gas pedal, i.e. the second tree returns a constant value. If the controller is disqualified, it will be penalized with a fitness value of –50. Otherwise, it will receive a fitness of d·(1–a)². This fitness value is doubled if the controller turns the steering wheel to the left as well as to the right.

For each experiment, 10 runs with different initializations of the random number generator were performed. For each run, a population of 200 individuals was evolved for 99 generations. Individuals were selected using tournament selection with n_T=7. Crossover was applied with probability p_cross=0.2; mutation was applied with probability p_mut=0.4; ERC mutation was applied with probability p_ERC–mut=0.3; reproduction was applied with probability p_rep=0.1. Ramped half and half initialization was used to initialize the individuals of the first generation with depth ranging from 2 to 6.

Results

Figure 6 shows the best fitness values obtained for all four experiments. By far the highest fitness values were reached when only depth information was available. It is clear (as we have described above) that this type of information is helpful in controlling the car. Optical flow also seemed to be helpful. Indeed, it is well known that bees use optical flow to achieve centering behavior [41, 42]. This ability (comparison of lateral optical flow) has also been used for visual control in robotics [43, 44].

Figure 6:

Best fitness values obtained for all four experiments.

Ten runs were conducted for each experiment. Depth information seems to provide an evolutionary advantage. For some of the runs, it produced exceptionally high-fitness individuals. Optical flow also seems to provide an evolutionary advantage.

Figure 7 shows the average best fitness for all four experiments. Using only color information resulted, on average, in less average best fitness compared to using depth information, optical flow, or all three after 99 generations. Average best fitness at generation 99 is summarized in Table 5. This table also shows the overall best fitness obtained in all of the 40 runs (10 per experiment) that we have carried out.

Figure 7:

Average best fitness for all four experiments.

We have used a Mann-Whitney U test to compare these averages, as shown in Table 6. We confirmed that the maximum fitness at the end of 99 generations was not normally distributed using a Kolmogorov-Smirnov test. Results, when using depth information or optical flow, were not significantly better compared to when only visual information was used. Average fitness improved considerably when depth information or optical flow was used for visual control. However, the difference was not significantly different because the Mann-Whitney U test compares results on an ordinal scale.

Only when all visual data (color, depth, and optical flow) is combined do we get significantly better results than when either visual input is used alone. Interestingly, the human brain evaluates its visual input with respect to color (in V4) and motion (in V5) [11, 45]. Ocular dominance columns are found in V1. Depth information can be computed from disparity information, i.e. a lateral offset of visual information between the two eyes [46]. The human visual system somehow combines this visual information in higher areas to achieve visual control.

Conclusion

We have used an evolutionary algorithm (genetic programming) to evolve image-processing algorithms to control a racing car. These algorithms are able to process various types of visual information: color, depth information, or optical flow. Each individual consists of two trees. The first tree is used to control the steering wheel and the second tree is used to control the acceleration of the car. Several elementary visual operations such as Gaussian smoothing, addition, multiplication, threshold, or locating a maximum or minimum response are also provided. These are all elementary operations that can be performed easily by a network of spiking neurons. In our experiments, we found that significantly better results in driving a racing car along its track are obtained when color, depth, and optical flow are provided together.