EEG-Based Emotion Recognition Using Deep Learning and M3GP

Abstract

This paper presents the proposal of a method to recognize emotional states through EEG analysis. The novelty of this work lies in its feature improvement strategy, based on multiclass genetic programming with multidimensional populations (M3GP), which builds features by implementing an evolutionary technique that selects, combines, deletes, and constructs the most suitable features to ease the classification process of the learning method. In this way, the problem data can be mapped into a more favorable search space that best defines each class. After implementing the M3GP, the results showed an increment of 14.76% in the recognition rate without changing any settings in the learning method. The tests were performed on a biometric EEG dataset (BED), designed to evoke emotions and record the cerebral cortex’s electrical response; this dataset implements a low cost device to collect the EEG signals, allowing greater viability for the application of the results. The proposed methodology achieves a mean classification rate of 92.1%, and simplifies the feature management process by increasing the separability of the spectral features.

1. Introduction

Machine Learning and EEG

The development of techniques that facilitate human behavior analysis has been progressed notably in the last decade, which has driven the development of machine learning tools and signal processing technology that can perform complex analyses of our environments. This has led to a diversification of analysis techniques and even the development of hardware to analyze our environmental signals, such as intelligent voice assistants and wearable devices that can give a detailed follow-up of our vital signs. Similarly, this development has allowed certain fields, such as the analysis of electroencephalography (EEG) signals, to also notably develop in areas that have previously been restricted by costs and technological limitations.

EEG signal analysis is a non-invasive technique that is widely used to analyze brain activity. Traditionally, the analysis of these signals has focused on the diagnosis and monitoring of medical conditions [1]. However, the interest in this signal has diversified, and machine learning (ML) has played a significant role in EEG signal processing and analysis because it has allowed the development of analysis techniques that successfully perform recognition and classification tasks. Promising results have been achieved in fields as varied as the analysis of mental illness, object manipulation, and cognitive processes [2,3,4,5,6].

A central aspect for the study of EEG signals is the premise of obtaining information directly from the cognitive processes of a person and implementing ML to process traits that allow us to identify, analyze, associate, and even control these processes through cognitive responses [3,7,8]. When analyzing the EEG signals, it is expected that the data are closely related to the phenomenon and cannot be manipulated. However, it is also a technique that is highly susceptible to experimental noise, so signal processing methodologies need to be carefully established.

The novelty of our proposal lies in applying the ML tool M3GP to the conversion of the search space through selecting, combining, deleting, and constructing the most suitable features to ease the classification process of the learning method. This process allows us to improve the classification stage without changing the architectures, which are often obtained through extensive processes.

The literature shows that the community has proposed diverse EEG data processing architectures; some use large and complex neural networks since their features require it [9] and others argue that low complexity architectures could handle the problem but use distinct spectral feature extractions [10,11]. The proposed methodology enables the improvement of the recognition rate for this type of analysis without changing the topology.

Several techniques have been developed for signal processing and feature selection, which have been traditionally based on spectral components [12], since they reduce the computational load for ML processes [11]. Recent literature suggests that automatic feature selection can be achieved using ML algorithms, thereby reducing the signal processing stage and allowing an ML strategy to choose the best-fit features [13,14]. This proposal is based on implementing MP3G to transform and select the most suitable features by considering a spectral stage instead of performing a greedy search.

This work focuses on the BED dataset [15], which was developed to evoke and register emotional stimuli using low cost devices that can help to generalize the results of this type of research and also proposes precise experimentation strategies and performs a rigorous signal treatment process. State-of-the-art (SOA) works, as reported in [16], have demonstrated the viability and importance of implementing new and accessible methodologies that seek the greater applicability of this type of research.

2. Materials and Methods

2.1. Emotions in Machine Learning

Russell’s arousal and valence levels [17] are used to measure how positive and intense an emotion is. This model subdivides the emotional responses into high arousal–high valence (HAHV), high arousal–low valence (HALV), low arousal–low valence (LALV), and low arousal–high valence (LAHV), as can be seen in Figure 1. We used this four subdivision model to create the classes our the ML process.

2.2. Dataset

This work used the BED, which is a dataset that is specialized for the analysis of emotional states. Although there are other datasets available, such as DEAP or SEED [18,19], BED was conceived and designed with the specific purpose of studying the brain’s response to an emotional stimulus; it is also a novel dataset that demonstrates an extensive and well-designed experimental process. In addition, BED was developed using low cost signal acquisition equipment, specifically the Emotiv EPOC+ [20] headband, which allows the experimental process to have a greater applicability.

The BED implements a methodology of evoking emotional responses through audiovisual stimuli followed by an arousal and valence analysis. The dataset contains metadata that allow for the association of the expected responses from each stimulus. Even more important is that it reports the experience of the user: each experiment is associated with a survey that records the actual level of arousal and valence experienced by the users (scored on the scale 0–4). We relied on the responses provided by the users to associate each experiment to one of the Russell model classes, as shown in Figure 2. In this way, since we were working according to the user responses, we were able to ensure that we were working with the appropriate information.

The dataset contains 21 healthy participants (18 male and 3 female), aged between 23 and 47 years old. The dataset emphasizes that it was created with the specific purpose of evoking emotions using Visual Evoked Potentials (VEP). The experiment implements a prolonged rest period to mitigate against the fatigue or satiety of the participants by dividing the experimental sessions into two days.

Emotiv EPOC+ is a low cost wireless headset that records EEG signals using 16 contact sensors at 256 Hz. BED considers the 14 channels that are closely aligned with the modified combinatorial nomenclature (MCN) of the 10–20 system (AF3, F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8, and AF4) and the M1 and M2 mastoid locations are used as the references. Using this device was one of the main aspects of this analysis exercise since it is a portable device and a viable alternative to much more expensive equipment. The computational resources and software implemented in the development of this project are: Python Scikit Learn libraries, Python 3.7.6, NumPy 1.18.5, Pillow 8.3.1, imutils 0.5.4, TensorFlow GPU 1.15.0, keras 2.2.4, matplotlib 3.1.1, SciPy 1.2.0., Intel i7 (4th gen), 3.6 GHz, 16 RAM, and Nvidia GeForce 1080.

2.3. Feature Extraction

This work implemented two feature selection strategies. First, we used a configuration similar to that presented in [15] to obtain our reference by using the features already contained in the dataset. Second, we implemented a feature selection and transformation through a genetic programming (GP) [21] algorithm called M3GP [22], which helped us to improve the recognition rate.

Spectral Features Arrangement

This work uses 518,154 spectral features contained in the dataset (mel-frequency cepstral (MFCC), autoregression reflection coefficients (ARRC), and spectral features (SPEC)) distributed into four categories, as presented in Figure 2. Recent advances have proven that artificial intelligence (AI) and ML have numerous applications in all engineering fields for processing and analyzing signals [23,24,25].

2.4. Feature Transformation by M3GP

M3GP is an ML tool based on genetic programming that improves the search space in terms of locally maximizing the distance between classes and thus, increasing the performance of the classification and recognition algorithms. Our method used a variant of tree-based GP called multidimensional multiclass GP with multidimensional populations (M3GP) [22], which is a wrapper approach for supervised classification and regression [26]. M3GP enables a transformation of the form k from ${\mathbb{R}}^p\to {\mathbb{R}}^d$ into $p,d\in \mathbb{N}$ using a special tree representation (see Figure 3), in essence mapping the p input features of the problem to a new feature space of size d. Afterward, M3GP applies the Mahalanobis distance classifier [27] to measure the quality of each transformation based on classification accuracy. In other words, M3GP is a wrapper-based GP classifier for multiclass problems that performs a feature treatment of the input data, thereby obtaining new data features with which the classification method can achieve better results.

The evolutionary process of M3GP enables the transformation of the input feature space, such that the transformed data of each class can be grouped into a single cluster, as in Figure 4. To achieve this, M3GP uses a tree representation that allows it to perform the mapping $k:{\mathbb{R}}^p\to {\mathbb{R}}^d$. The representation is the same as in regular tree-based GP, except that the root node of the tree exists only to define the number of dimensions d of the new space. Each branch stemming directly from the root performs the mapping in one of the d dimensions (see Figure 3). The initial population can start with a single feature; to search for simpler and smaller transformations, the increase in features occurs through the genetic operators.

As with any evolutionary algorithm, M3GP involves a population that may contain individual features of several different dimensions, as shown in Figure 5. M3GP includes special genetic operators that can add or remove dimensions. It is assumed that the selection process is sufficient to discard the individuals with the least useful dimensions (also referred to as "features") whilst maintaining the best dimensions within the population.

2.4.1. Initial Population

M3GP starts the search with a random population where all features have only one dimension, see Figure 5. This ensures that the evolutionary search begins by looking for simple one-dimensional solutions before moving on to higher dimension solutions, which could also be more complex. If the problem has a high number of features, it is recommended to start with more features. For the experiments presented in this work, the number of starting dimensions was 10.

2.4.2. Mutation

During the breeding phase, whenever mutation is the chosen genetic operator, one of three actions is performed with equal probability: (1) standard sub-tree mutation, where a new randomly created tree replaces a randomly chosen branch of the parent (excluding the root node); (2) adding a new randomly created tree as a new branch of the root node, effectively adding one dimension to the parent tree; and (3) randomly removing a complete branch of the root node, effectively removing one dimension from the parent tree (see Figure 6).

In M3GP, mutation is the only way of adding dimensions. Therefore, it has a probability of $0.5$ of guaranteeing a proper search for the best number of dimensions. Previous results have shown that this high mutation probability (compared to most GP algorithms) works best with M3GP [22].

2.4.3. Crossover

Whenever crossover is chosen, one of two actions is performed with equal probability: (1) standard sub-tree crossover, where a random node is chosen from each of the parents and the respective branches are swapped (excluding the root node); (2) the swapping of dimensions, where a randomly selected complete branch of the root node is chosen from each parent and swapped, effectively swapping how the input data are transformed into a new feature dimension. The second event is just a variant of the first, where the crossing nodes are guaranteed to belong to the root node (see Figure 7).

2.4.4. Pruning

Mutation, as described above, allows M3GP to easily add dimensions to the evolved solutions. However, some dimensions may degrade the individual’s fitness and so, they need to be removed. Mutation can also remove dimensions but it does so randomly and is blind to fitness, as described above. We removed the detrimental dimensions by pruning the best individual after the breeding phase to maintain the simplicity and stochasticity of the genetic operators.

The pruning procedure removes the first dimension and re-evaluates the tree. If the fitness improves, the pruned tree replaces the original and moves on to the pruning of the next dimension. Otherwise, the pruned tree is discarded and the original tree moves on to the pruning of the next dimension. The procedure stops after pruning the last dimension. Pruning applied to the whole population has a higher computational cost but controls the growth of dimensions; nevertheless, to offset the computational cost, we only pruned the best solution found in the population for each generation.

2.4.5. Elitism

This criterion ensures the survival of the individuals from one generation to the next. M3GP does not allow the best individual of any generation to be lost and always copies it to the next generation. Let us recall that this individual is highly optimized because it went through the pruning process. Another elitist criterion occurs when two solutions have the same accuracy: only the smallest is conserved, thereby simplifying the transformation tree.

3. Neural Network Configuration

The implemented deep learning (DL) topology and configuration was extracted from the experimental process in [11], which determined that this kind of low-complexity neural network achieves an excellent performance without increasing the computational burden. The topology can be observed in Figure 8 and

Table 1. The experimentation parameters considered in our DL architecture.

Parameters	Configuration
Train/Test	70/30
Validation	10-Fold
Alpha value	0.0002
Maximum iterations	200
Functions	ReLU, Tanh, and Logistic
Output layer	Softmax

. The network uses the one-dimensional convolutional structure to perform a depthwise convolution that acts separately on the channels, followed by a pointwise convolution that mixes the channels [28]. The four emotional states define the four classes, as in the emotions model (see Figure 1). Also, this work considers ReLU, Tanh, and logistics functions to observe performance variations and demonstrate how they are affected by the change in traits. The proposed methodology uses the same topology but distinct features composition in the training process as can be observed in Figure 9. This allowed us to observe whether we could improve the classification rate by using the same topology with distinct feature configurations.

4. Results

We performed tests with various neural network configurations, using ReLU, Tanh, and logistic functions, to demonstrate the performance improvement when implementing the M3GP. The performance of the network before M3GP is shown in

Table 2. The best results before the M3GP transformation of the dataset.

Function	AUC	CA	F1	Precision
ReLU	0.880	0.650	0.651	$0.661\pm 0.021$
Tanh	0.880	0.655	0.655	$0.657\pm 0.027$
Logistic	0.748	0.478	0.477	$0.482\pm 0.011$

, where it can be observed that the classification accuracy was around 65.9% for the cases of the ReLU and Tanh functions. In contrast, the logistic function achieved an inferior performance, which was expected since it does not usually perform well on non-binary systems. However, it was in our interest to show the impact of M3GP on the grouping of the elements.

The improvements achieved in the classification rates can be seen in

Table 3. The best results after the M3GP transformation of the dataset.

Function	AUC	CA	F1	Precision
ReLU	0.991	0.920	0.920	$0.921\pm 0.008$
Tanh	0.972	0.859	0.859	$0.865\pm 0.012$
Logistic	0.692	0.457	0.425	$0.457\pm 0.180$

. It can be seen that the ReLU and Tanh functions had an improvement of 26% and 20.8%, respectively, and each class classification improvement is represented in Figure 10 and Figure 11. The logistic function showed a decrease in the recognition rate since it considers a binary scenario, which M3GP makes less feasible; this can be better appreciated in Figure 12. The best training and testing cross-validations for the ReLU function can be observed in Figure 13 and the training confusion matrix values can be observed in Figure 14.

4.1. Standalone M3GP

The main reason for using M3GP was not to obtain a classifier with superior results to the SOA algorithms, but to obtain a feature transformation that could improve the results obtained by the SOA algorithms.

M3GP was run with the settings shown in

Table 4. M3GP setup settings.

Runs	10
Population Size	500 individuals
Generations	100 generations
Initial Dimension	10 dimensions
Initialization	6-depth full initialization [21]
Data	70% Training −30% Tests
Operator Probabilities	Crossover $p_c=0.5$, Mutation $p_{\mu }=0.5$
Function Set	$+\left(plus\right),-\left(minus\right),\times \left(times\right),÷$ protected as in [21], sine, cosine, exponential, logarithm (abs), power${}^2$, power${}^3$, square root (positive), tangent (tan), hyperbolic tangent (tanh)
Terminal Set	Ephemeral random constants $[0,1]$
Bloat Control	17-depth limit [21]
Selection	Lexicographic tournament [29] of size 5
Elitism	Keep best individual

with the spectral feature (MFCC, ARRC, and SPEC) arrangement of the BED dataset. The best results, presented in

Table 5. The accuracy obtained by M3GP, the number of root node features of the best solution found in each dataset, and the original features of each set.

M3GP	MFCC	ARRC	SPEC
Train	94.4	59.9	46.3
Test	39.1	39.6	40
Root Node Features	63	23	13
Original Features	168	168	224

, were not significantly different from the SOA methods shown in

Table 6. Performance comparisons between SOA techniques reported in [30]. Similar EEG-based works have reported an overall recognition rate of 82.9%.

Classifier	Average Performance
Neural Network [31,32,33]	85.80%
Support Vector Machine [34,35,36]	77.80%
K-Nearest Neighbor [33,37,38]	88.94%
Multi-layer Perceptron [38,39,40]	78.16%
Bayes [41,42,43]	69.62%
Extreme Learning Machine [41]	87.10%
K-Means [43]	78.06%
Linear Discriminant Analysis [42]	71.30%
Gaussian Process [44]	71.30%

. There are many reasons for the low performance of well-known algorithms (e.g., the number of features is too high, there are not enough samples, there is an unbalanced dataset) and when considering all possibilities, users may be tempted to apply some form of data treatment, but not letting the evolutionary process use all available data can affect the performance of the evolutionary algorithms by removing the possibility of letting the natural selection process find the ideal treatment.

To improve on the performance of the SOA methods, we used the features generated by M3GP in Figure 15. This kind of transfer learning was used in [45] with success and the best training transformation found in M3GP was used to transform the dataset into a new one (M3GP tree in Section 4.2), considering that these new features contain more information to simplify the learning process of the SOA methods. The chosen dataset was MFCC, with a matrix of 1481 × 168 becoming a matrix of 1481 × 63 through the M3GP transformation. The improved result is shown in Table 3.

4.2. Transformation Tree

Root node(X23, X24, times(X139,X85), X147, X116, X6, X80, X118, X42, X67, X2, X167, times(X161,tanh(tanh(X130))), X56, X122, X158, X74, X111, X109, X121, X154, X75, X84, X19, X43, X149, X12, X139, X102, X96, X36, X112, X28, X45, X33, X29, plus(X152,X21), X166, X27, X31, X15, X159, X38, squareroot(X86), X34, X89, X126, X65, X161, X82, X123, X151, X136, X156, X103, X168, X162, X51, X57, X87, X155, X132, sine(X30)). Note that each X is an original feature of the BED MFCC dataset.

5. Discussion

The implementation of M3GP in the feature generation stage and the pattern recognition stage through deep learning allowed us to obtain competitive results. We carried out a similar test to that reported in [15] but implementing an architecture that we had previously tested in other benchmarks [12,28]. These results were still not significant compared to the SOA algorithms, although we obtained similar results to the original study (see Table 6). However, the results of maintaining the same architecture and changing the feature selection stage via a transformation of features using M3GP improved to the results obtained in [15] by 20.9% and even exceeded some of the SOA methods.

M3GP has shown competitive results in different benchmarks [22,26,46], especially regarding multiclass problems; but in this case, it was not able to obtain competitive results with EEG from the BED dataset. It also showed a high overfitting problem in the best solution found for the MFCC data. However, this experiment was not wasteful since, based on the results obtained in [45], M3GP feature transformations can still be used as feature optimizers by transferring the knowledge obtained from the evolutionary process to the transformation of features. The effect of the feature can be seen in

Table 7. The percentage of classification improvement.

Fuction	AUC %	CA %	F1 %	Precision %
ReLU	11.1	27	26.9	26
Tanh	9.2	20.4	20.4	20.8
Logistic	−5.6	−2.1	−5.2	−2.5

.

An interesting result was the overfitting problem when using the Mahalanobis classifier being so high, as observed in Table 5. No other classifiers show any signs of overfitting when working with the transformed dataset, which makes the overfitting problem data independent.

6. Conclusions

A central aspect of the study of mental states through EEG signals is that it does not leave a clear marker on which to focus, unlike certain diseases, such as epilepsy. This means that we often cannot clearly define whether a stimulus exists or not when it begins or ends and which part of the signal is liked to the stimuli. By implementing M3GP, we presupposed that we could accentuate the differences between the behavior of the signals. As shown, once the spectral coefficients of the signals have been extracted, they are processed to increase the distance between them and facilitate the classification process.

This proposal reduced the complexity of the recognition process by reducing features through an evolutionary process that selects those features that best define each of the classes. Traditionally, this analysis process extracts the spectral features of the signals to carry out the pattern search process. However, these features could be linked to diverse phenomena that are inherent to the signals, such as artefact noise, or various physiological phenomena, such as closing eyes, or cognitive processes that are simply not associated with the experimental process. M3GP allowed us, through a selection process, to obtain only those individuals that best fit the description of each of the classes and thus, allowed us to improve the classification rate.

Another essential aspect of this work is the implementation of the BED dataset, which was conceived and built to study users’ emotional reactions. BED explores the use of low cost devices in a formal study, thereby promoting and establishing some basis for applicability in less controlled environments.

Our future work will based on integrating the feature selection process into the network architecture configuration, i.e., implementing the neat GP concept in the generalization and optimization of network operation [9,47]. To our knowledge, DL has shown outstanding performances when working with EEG signals; however, the community continues to propose architectures that can efficiently solve this type of problem. We hypothesize that we can establish the criteria for the network features and topology when implementing an evolutionary strategy. Another proposal for future work that is linked to this model lies in implementing GP as an AutoML methodology, with the premise that GP could be used to autonomously select features and DL topology (currently, this process is performed through digital signal processing by extracting spectral coefficients).