An enhanced fitness function to recognize unbalanced human emotions data

Highlights

• An enhanced distance score (eD-score) fitness function is proposed to handle unbalanced dataset classification.

• A new EEG signal dataset for emotional clips is created with a portable, computer-efficient single-channel EEG headset.

• A novel eDGP framework is proposed for the classification of unbalanced emotion recognition data.

• Analysis of human behavior for age, genre, gender, and attention level is performed.

Abstract

In cognitive science and human-computer interaction, automatic human emotion recognition using physiological stimuli is a key technology. This research considers two-class (positive and negative) of emotions recognition using electroencephalogram (EEG) signals in response to an emotional clip from the genres happy, amusement, sad, and horror. This paper introduces an enhanced fitness function named as eD-score to recognize emotions using EEG signals. The primary goal of this research is to assess how genres affect human emotions. We also analyzed human behaviour based on age and gender responsiveness. We have compared the performance of Multilayer Perceptron (MLP), K-nearest neighbors (KNN), Support Vector Machine (SVM), D-score Genetic Programming (DGP), and enhanced D-score Genetic Programming (eDGP) for classification of emotions. The analysis shows that for two class of emotion eDGP provides classification accuracy as 83.33%, 84.69%, 85.88%, and 87.61% for 50-50, 60-40, 70-30, and 10-fold cross-validations. Generalizability and reliability of this approach is evaluated by applying the proposed approach to publicly available EEG datasets DEAP and SEED. When participants in this research are exposed to amusement genre, their reaction is positive emotion. In compliance with the self-reported feelings, brain signals of 26–35 years of age group provided the highest emotional identification. Among genders, females are more emotionally active as compared to males. These results affirmed the potential use of our method for recognizing emotions.

Introduction

Emotion recognition and prediction form a well-researched area in the field of psychology (Cowie et al., 2001). Recently, emotion recognition has drawn growing attention as an evolving research area (Acharya et al., 2020, Chen et al., 2019, Hossain and Muhammad, 2019). Promising results for application areas like online gaming (Kim, Lee, & Provost, 2013), emotion-aware health care systems (Chen, Zhang, Qiu, Guizani, & Hao, 2018), effectiveness based learning programs (Arroyo et al., 2009), digital advertisements, and online shopping (Hossain & Muhammad, 2017) are there. Response to stimulation lasting for seconds or minutes is termed as emotion, the mood is a condition of emotion lasting for hours and days, and a person is inclined to experience certain feelings during that time (Picard, Vyzas, & Healey, 2001). Emotions are an integral element of humans because they motivate action and contribute significance to nearly all human experiences (Sourina, Wang, Liu, & Nguyen, 2011). In the communication between people also, emotions play a crucial part as it affects the contextual comprehension of texts expression or graphics type (Wagner, Kim, & André, 2005).

Based on individual differences in temperament, a dimensional model is proposed to describe their emotional state and attention level (Eerola and Vuoskoski, 2011, Gable and Harmon-Jones, 2010). Another reason is that distinct attention levels can have distinct effects on emotional experiences. A beneficial method for regulating mind and body is attention. The relationships between emotion and attention is a very popular research area for several purposes. One reason is the apparent fact that our emotions can regulate the level of attention (Holland and Gallagher, 1999, Kuusinen et al., 2018, Gordon et al., 2018). While recognizing the emotions of a person, their attention level can also be analyzed. In the traditional way of teaching face to face, instructions were given generally, and by observing students expressions, attentiveness of students can be checked. In the case of online learning; however, these problems become more intense because the instructor finds it hard to distinguish the student’s attention as they are at remote locations and taking courses online. Thus, indirect attention measurement is a highly helpful method that provides teacher feedback in real-time. Learning outcomes are directly affected by the attentiveness of learners (Klimesch et al., 1998, Chen et al., 2019). This study is also addressing the effect of emotions on attention level so that it can be identified that under what circumstances the attention level is enhanced or drops in the participants. As a result, elevated levels of attention offers new ways to learn more and to work more productively also.

In affective computing field, research on recognizing emotions are either based on self-assessment manikin (SAM) that contains subjective measures (De Moor et al., 2006, Bradley and Lang, 1994) or by analyzing physiological and behavioral signals. On the other hand, significant research has been reported using images (Busso et al., 2004, Mollahosseini et al., 2016), speech (Anagnostopoulos, Iliou, & Giannoukos, 2015), audio (Kim & André, 2008), and facial expression (Zheng, 2014). All conventional emotion prediction models for psychological analysis tend to rely on self-assessment forms and human inputs (Kartelj, Filipović, & Milutinović, 2012). To recognize emotions from images, ImageNet was used with the convolutional neural network (CNN) in which static images are considered, and the concept of transfer learning was applied (Ng, Nguyen, Vonikakis, & Winkler, 2015). Kim et al. (2013) used Deep belief network to recognize emotions from speech and video, where he proposed a feature selection technique to enhance the accuracy of the model. Multi-view facial images are also considered for facial expression based emotion recognition by Zhang et al. (2016). Although several previous works are published on audio-visual recognition in the literature, most of them suffer from low accuracy of recognition.

Recently, to recognize emotions using periphery physiological signals has gained much attention in this field (Nakisa et al., 2018, Huang et al., 2019, Zhang et al., 2020). In this category, signals are recorded from brain using electroencephalograph (EEG) and functional magnetic resonance imaging (fMRI), skin temperature (SC), pulse rate, and heart rate that is measured by electrocardiogram (ECG) and electrocorticography (ECoG) (Li and Chen, 2006, Dahl et al., 2014). Among these physiological signals brain, stimuli based signals have gained more popularity. EEG signals have been used widely in medicine to study the brain. EEG is believed to be the finest way of recording data in various modalities owing to its distinctive elements when interacting with emotional conditions (Lokannavar, Lahane, Gangurde, & Chidre, 2015).

It is already proven that humans are adept at concealing and suppressing emotions. Due to which the works published in literature based on the self-assessment test achieved low accuracy in recognizing emotions. Thus, this brings forth the need to develop a new emotion recognition approach. In this study an enhanced fitness function in genetic programming (GP) (Devarriya, Gulati, Mansharamani, Sakalle, & Bhardwaj, 2020) approach to recognize emotions in response to emotional clips is proposed. Emotional movie clips are perceived as a compelling medium for eliciting human emotions, as they are described as the language of emotions (Liang, Oba, & Ishii, 2019). A deep connection among emotion of humans and emotional movie clips is described as the part in the brain of humans that feel and interpret emotions are near together (Pouyanfar & Sameti, 2014).

The challenge in the field of recognizing emotion is the manipulation of emotions. The exact process of recognizing emotions is very time consuming, so there is a need to automate this process, which can help a person to utilize his/her potential to the maximum. It is impossible to falsify brain activity with the help of recording brain signals. The use of portable EEG device is non-invasive, fast, and inexpensive, making it a preferred method to study the response of the brain to emotional stimuli (Acharya et al., 2020) this helps us in achieving a right level of accuracy in the classification of emotions also. EEG signals are recorded using electrodes placed on the scalp of the brain. EEG inputs consist of sample frequency ranges from 0.5 Hz to 100 Hz, and in literature band 0.5 Hz–40 Hz is used for examination of human brain-behavior (Bhatti, Majid, Anwar, & Khan, 2016). EEG signals are classified as five leading frequency bands in this study: Delta, Theta, Alpha, Beta, and Gamma. The other problems in this emotion recognitions dataset is that the dataset is unbalanced. In literature, it is already proven that it is more difficult to classify an unbalanced dataset.

The dataset collected for emotion recognition was unbalanced, as there are more instances of positive emotions than negative ones. An unbalanced dataset is one in which one class have a higher number of samples called Majority class, and other classes have very less number of samples called minority class (Beyan annd Fisher, 2015). In most frameworks, accuracy is used as a measure to assess the results (Acharya, Billimoria, Srivastava, Goel, & Bhardwaj, 2020). However, accuracy provides misleading results in the context of classification of unbalanced dataset (Devarriya et al., 2020). As, a result, the classifier produces higher accuracy for majority class as it influences the classifier because of more number of samples, and lower accuracy for minority classes. A framework that can cope with such situations is therefore required. In the past few years evolutionary algorithms have gained importance in the research field among the researchers for handling classification problems (Chawla et al., 2004, Garcı et al., 2012, Beyan annd Fisher, 2015).

Genetic Programming (GP) (Koza, 1992) inspires the proposed approach to handle classification of unbalanced dataset classification. GP is widely used for different types of classification problems like object detection (Kishore, Patnaik, Mani, & Agrawal, 2000), in medical field (Bhardwaj, Sakalle, Bhardwaj, & Tiwari, 2019), for feature extraction (Bhardwaj et al., 2015, Bhardwaj and Tiwari, 2013, Bhardwaj and Tiwari, 2013), (Guo, Rivero, Dorado, Munteanu, & Pazos, 2011), in game theory (Hargraves and Paris, 1987, Bhowan et al., 2013). GP uses the basic concept of creating multiple functions known as a population and then randomly combine them to create programs. Therefore, in this paper, we introduce an enhanced version of D-score (Devarriya et al., 2020) named as eD-score, to address the issue of unbalanced emotion recognition dataset by using the GP framework. D-score is a fitness function which handles unbalanced dataset being unswayed. Instead of traditional machine learning algorithms which are more lean towards majority classes, D-score gives both the class equal value even though minority classes have less number of samples. The framework in which we implemented eD-score is eDGP.

To show the supremacy of our eDGP we also analyzed it over the two publicly available benchmark dataset DEAP (Koelstra et al., 2011) and SEED (Zheng & Lu, 2015) and compare our work with other state-of-the-art methods as shown in Table 16 (Bhatti et al., 2016, Koelstra et al., 2010, Bastos-Filho et al., 2012). Fast Fourier Transformation is used for feature extraction. We aim to enhance classification results on the state-of-the-art techniques by using an enhanced DGP approach to recognize unbalanced emotions. To illustrate the dominance of our method, we use the 50-50, 60-40, 70-30, 10-fold cross-validation for confusion matrix and Mann Whitney test. Also, evaluated the maximum, minimum, and average classification accuracy of our model. In terms of the social behaviour analysis genre-wise classification of emotions, age group response to emotions, attention level of participants, and gender-based responsiveness is analyzed. The contribution of this paper can be summarized as below:

• An enhanced distance score (eD-score) fitness function is proposed to handle unbalanced dataset classification.

• A new EEG signal dataset for emotions is created with a portable, single-channel EEG headset (NeuroSky MindWave2).

• A novel eDGP framework is proposed for the classification of unbalanced emotion recognition data.

• A genre-wise classification, age-group, and gender-based responsiveness of emotions is performed.

• Analysis of attention level in participants and its correlation with emotions is done.

The rest of the paper is organized as follows: Section 2 contains related work followed by the background description in Section 3. The proposed methodology is explained in Section 4. Section 5 illustrates the experimental settings followed by performance parameters described in Section 6. Results and discussion is presented in Section 7, and Section 8 contains conclusion.