Elsevier

Pattern Recognition Letters

Volume 133, May 2020, Pages 272-279
Pattern Recognition Letters

A novel fitness function in genetic programming to handle unbalanced emotion recognition data

https://doi.org/10.1016/j.patrec.2020.03.005Get rights and content

Highlights

  • A novel fitness function in genetic programming to handle unbalanced emotion recognition data.

  • A genre-wise classification of emotions is performed targeting happy, sad, horror, and neutral genre in the participants.

  • Response of different age groups for emotions is analyzed.

Abstract

In the area of behavioral psychology, real-time emotion recognition by using physiological stimuli is an active topic of interest. This research considers the recognition of two class of emotions i.e., positive and negative emotions using EEG signals in response to happy, horror, sad, and neutral genres. In a noise-free framework for data acquisition of 50 participants, NeuroSky MindWave 2 is used. The dataset collected is unbalanced i.e., there are more instances of positive classes than negative ones. Therefore, accuracy is not a useful metric to assess the results of the unbalanced dataset because of biased results. So, the primary goal of this research is to address the issue of unbalanced emotion recognition dataset classification, for which we are proposing a novel fitness function known as Gap score (G score), which learns about both the classes by giving them equal importance and being unbiased. The genetic programming (GP) framework in which we implemented G score is named as G-score GP (GGP). The second goal is to assess how distinct genres affect human emotion recognition process and to identify an age group that is more active emotionally when their emotions are elicited. Experiments were conducted on EEG data acquired with a single-channel EEG device. We have compared the performance of GGP for the classification of emotions with state-of-the-art methods. The analysis shows that GGP provides 87.61% classification accuracy by using EEG. In compliance with the self-reported feelings, brain signals of 26 to 35 years of age group provided the highest emotion recognition rate.

Introduction

Emotion determination and prediction form a well-researched area in the field of psychology [16]. Emotions play a vital role in people’s interaction with each other as it affects the contextual comprehension of texts expression or graphics type. For effective human-machine interaction, the affective interactions between humans and computers need to consider how emotions can be acknowledged and manifested during human-computer interaction (HCI). Emotional assessment is one of the central mechanisms in HCI studies to implement emotional intelligence [19]. Two different class of emotions, i.e., positive and negative, are considered in this research.

In the past, different approaches are used for emotion recognition that is widely divided into four types. The first type of approach relies on data from the questionnaire [8]. The second approach considers speech [1] and facial expression [12]. In these techniques, images or videos are used to record emotions also multi-view facial images are considered for facial expression. The third type is based on periphery physiological signals [9]. In this category, signals are recorded from skin temperature (SC), pulse rate, and heart rate that is measured by electrocardiogram (ECG). The fourth type is based on brain stimuli using electroencephalograph (EEG), electrocorticography (ECoG), and functional magnetic resonance imaging (fMRI). All conventional emotion prediction models for psychological analysis tend to rely on self-assessment forms and human inputs. Humans are adept at concealing and suppressing emotions. Thus, this brings forth the need to develop a new emotion recognition approach.

There are two significant challenges in the field of recognizing emotions. The first challenge is the suppression and manipulation of emotions. It is impossible to falsify brain activity with the help of recording brain signals. This helps us in achieving the right level of accuracy in the classification of emotions as the use of EEG is non-invasive, fast, and inexpensive, making it a preferred method to study the response of the brain to emotional stimuli [6]. EEG signals are recorded using electrodes, which record the electrical activity of the brain from the scalp. We use the frequency band 0–40 Hz to examine healthy brain activity. EEG inputs consist of ample frequency ranges from 0 Hz to 100 Hz, and in literature band 0–40 Hz is used for examination of human brain-behavior [3]. Second, the collected emotion recognition dataset is unbalanced because there are more instances of positive emotion than negative ones. Datasets can be termed as an unbalanced if one of its classes is represented by only a small number of training instances (called the minority class) while the other classes make up the majority [3]. Due to the influence of the larger majority class on traditional training criteria in the fitness function, the classifier results tend to have good accuracy on the majority class but has insufficient accuracy on the minority classes [11]. Therefore it is significant and vital to develop a framework which deals with these scenarios.

Genetic Programming is an evolutionary approach that is utilized for solving the problem of data classification [3], [4]. Therefore, in this paper, we introduce a novel fitness function, Gap score (G-score), to evaluate the GP framework to address the issue of unbalanced emotion recognition data classification. G-score handles unbalanced data by learning about both the classes by giving them equal importance i.e., by being unbiased with the classes. In this research work fast fourier transformation (FFT) is used for extracting features from EEG signals, which transforms a signal from the time domain into the frequency domain. For classification, we intend to improve on state-of-the-art methods by proposing the GGP framework to recognize emotion [5]. To illustrate the dominance of our proposed GGP method, we partitioned the data into 50-50, 60-40, 70-30 training-testing partition, and also 10-fold cross-validation analysis has been done. The average, minimum, and maximum accuracy of the proposed model is evaluated to compare our work with other state-of-the-art-methods. We also determine the human behavior analysis in terms of genre-wise classification, and also we predicted the age group response to emotions.

The main contribution of this paper is as follows:

  • A novel fitness function is proposed to handle unbalanced emotion recognition data in GP.

  • A new EEG signal dataset for emotion recognition is established with a single-channel EEG headset (NeuroSky MindWave 2).

  • Classification of the genre-wise class of emotion and responses of different age groups is analyzed.

  • To the best of our knowledge, it is for the first time that the GGP framework is used for recognizing emotion in response to emotional videos that are used to evoke different emotions as external stimuli.

The rest of this paper is organized as follows. Section 2 presents a background that contains fast fourier transformation, genetic programming, and fitness function explanation. Section 3 describes the proposed Gap Score fitness function. Section 4 is devoted to the experimental settings, followed by the experimental results in Section 5. Section 6 contains the conclusion.

Section snippets

Background

This section describe the statutory background for this approach, which includes the feature extraction process fast fourier transformation (FFT) [20], genetic programming classifier (GP) [14], and fitness function. To execute the GGP framework, the first need is to extract features from the EEG signals for that we used FFT. After the features are extracted, we used those features as the input to the GGP model to form a GP tree. After this, we evaluate the fitness of the GP tree, and then the

Proposed gap score fitness function

This section explains the details of the proposed fitness function. Due to the unbalanced nature of the emotion recognition dataset, most ML models fail to classify between positive and negative cases properly, even though they have high accuracy. As observed in the previous section, accuracy gives biased results with unbalanced data, which makes it inefficient to work as the fitness function for GP. To resolve the issues related to the classification of unbalanced data, we propose a new

Experimental settings

In this section, we describe the experimental set-up for emotion recognition using EEG. The participant pools, description of the dataset and experiment procedure is described next.

Experimental results

This section presents the experimental results of the proposed G-score Genetic Programming (GGP) for emotion recognition by analysis of EEG Signal along with three different state-of-the-art classifiers Multilayer Perceptron (MLP) [6], K-nearest neighbors (K-NN) [1], [2], and Support Vector Machine (SVM) [13]. All these algorithms are used to classify human emotions into two class of emotions that is positive and negative emotions. MLP, K-NN, SVM, and GGP framework as a classifier was

Conclusion

We proposed a novel fitness function termed as G-score for addressing the classification of an unbalanced emotion recognition dataset. A novel GGP framework, which is brain signals based human emotion recognition system, is analyzing the emotional conditions of participants through brain wave assessment. To validate our results, we performed multiple experiments, which are valuable in comparing our GGP approach with the existing methods. These results indicate a benefit in terms of accuracy and

Declaration of Competing Interest

The author do not have any conflict of interest.

References (20)

There are more references available in the full text version of this article.

Cited by (21)

  • A multi-task hybrid emotion recognition network based on EEG signals

    2023, Biomedical Signal Processing and Control
  • Emotion Recognition Based on Brain Connectivity Reservoir and Valence Lateralization for Cyber-Physical-Social Systems

    2022, Pattern Recognition Letters
    Citation Excerpt :

    Wager et al. [29] conducted a quantitative meta-analysis on emotion neuroimaging, and concluded that both men and women are in line with VL. Acharya et al. [30] confirmed that the evaluation of the effects of different types of emotions on the left and right brain activities confirmed the correctness of VL in adults. Owing to most of the participants of emotion lateralization are adults, Watling and Damaskinou [31] chose participants from 5 to 12 years old to evaluate the relationship between lateralization and emotion, discovering that VL also exists in minors.

  • A LSTM based deep learning network for recognizing emotions using wireless brainwave driven system

    2021, Expert Systems with Applications
    Citation Excerpt :

    The third method is based on using physiological signals to recognize emotion. Recently, recognizing emotions using periphery physiological signals has gained much attention in this field (Acharya et al., 2020). The physiological signal provides a better understanding of the participant’s reactions generated at the time of experiments.

  • An enhanced fitness function to recognize unbalanced human emotions data

    2021, Expert Systems with Applications
    Citation Excerpt :

    Emotion recognition and prediction form a well-researched area in the field of psychology (Cowie et al., 2001). Recently, emotion recognition has drawn growing attention as an evolving research area (Acharya, Goel, Asthana, & Bhardwaj, 2020; Chen, Jiang, & Zhang, 2019; Hossain & Muhammad, 2019). Promising results for application areas like online gaming (Kim, Lee, & Provost, 2013), emotion-aware health care systems (Chen, Zhang, Qiu, Guizani, & Hao, 2018), effectiveness based learning programs (Arroyo et al., 2009), digital advertisements, and online shopping (Hossain & Muhammad, 2017) are there.

View all citing articles on Scopus

Editor: Shiguang Shan, Ph.D.

View full text