A genetic programming-based feature selection and fusion for facial expression recognition

https://doi.org/10.1016/j.asoc.2021.107173Get rights and content

Highlights

  • Genetic programming-based programs learns to discriminate pairs of expressions.

  • Feature selection and fusion are incorporated in the learning process.

  • Feature selection and fusion are performed differently for each pair of expressions.

  • Three geometric and texture-based features are tested using the proposed framework.

  • Obtained results, on posed and spontaneous FER datasets, are very encouraging.

Abstract

Emotion recognition has become one of the most active research areas in pattern recognition due to the emergence of human–machine interaction systems. Describing facial expression is a very challenging problem since it relies on the quality of the face representation. A multitude of features have been proposed in the literature to describe facial expression. None of these features is universal for accurately capturing all the emotions since facial expressions vary according to the person, gender and type of emotion (posed or spontaneous). Therefore, some research works have considered combining several features to enhance the recognition rate. But they faced significant problems because of information redundancy and high dimensionality of the resulting features. In this work, we propose a genetic programming framework for feature selection and fusion for facial expression recognition, which we called GPFER. The main component of this framework is a tree-based genetic program with a three functional layers (feature selection, feature fusion and classification). The proposed genetic program is a binary classifier that performs discriminative feature selection and fusion differently for each pair of expression classes. The final emotion is captured by performing a unique tournament elimination between all the classes using the binary programs. Three different geometric and texture features were fused using the proposed GPFER. The obtained results, on four posed and spontaneous facial expression datasets (DISFA, DISFA+, CK+ and MUG), show that the proposed facial expression recognition method has outperformed, or achieved a comparable performance to the state-of-the-art methods.

Introduction

Recognition of human emotions has long been the subject of active research area. A wide range of human interaction applications have to decipher the facial emotional state. Unlike other non-verbal gesture, the emotional state of the face can be relied to several expressions. Most research has focused on posed facial expressions and reached high level of efficiency recognizing human emotions [1], [2]. However, some posed expressions are still very hard to discriminate such as disgust and sadness emotions. Quite fewer works have done advances interpreting spontaneous facial emotions. There are several factors affecting the precision of facial expression recognition (FER) systems on spontaneous or posed expressions, including prominent facial feature selection, feature fusion and classifier design. Since, FER applications have to deal with natural emotions, our goal in this work is to develop a system that can achieve accurate recognition rates on posed as well as on spontaneous facial expressions. Extracting efficient facial features is crucial towards facial emotion recognition. Commonly, two types of features are used to discriminate facial emotions: geometric and appearance features. Geometric features give clues about shape and position of face components, whereas appearance based features contain information about the furrows, bulges, wrinkles, etc. Appearance features contain micro-patterns which provide important information about the facial expressions. But one major drawback with them is the difficulty to generalize across different persons. Although geometric features are noise sensitive and difficult to extract, they proved to be sufficient to give accurate facial expression recognition results [3]. Moreover, He et al. [4]

demonstrated that geometric features are more effective than appearance ones in most cases. However, geometric-based FER methods still have difficulties discriminating some expressions. As an illustration, Fig. 1 shows two faces displaying disgust emotional state that were miss-classified as sadness expressions using geometric based features and correctly recognized using local binary patterns (LBP) according to a study presented in [5]. Indeed, micro-patterns, captured by LBP features, are able to offset the weakness of geometric based features by capturing micro-variations caused by wrinkles, which are useful to separate the disgust and sadness emotional states. Therefore, facial geometry distortions, given by geometrical features, are complementary with textural information captured by appearance ones. In other words, considering geometrical and appearance feature fusion can be an interesting way to design more discriminative features to deal with FER challenges.

Feature fusion based methods face the problem of high dimensionality which can affect the quality of the facial emotion recognition. Indeed, dealing with large number of features can increase the computational time and overwhelm classifiers with unnecessary or redundant information. In this case, a rigorous feature selection step is necessary. To carry out selection of a good feature subset, several factors must be considered. First, feature selection cannot be performed in the same way for spontaneous and posed expressions. Indeed, spontaneous facial muscle movements have been proven significantly different from deliberate ones. According to Ekman and P [6], zygomatic major is the only interacted muscle in posed smiles. However, muscles around the eyes (i.e. orbicularis oculi) are also contracted during genuine smiles. Moreover, Namba et al. [7] studied the difference in action units (AUs) [8] between involuntary and real emotions. For instance, the three most commonly seen AUs for genuine disgust are squinting eyes and raising the upper lip. On the other side, glare and raising the chin are often spotted in posed disgust. The eyebrow raiser, the lips part, the jaw drop and the upper lid raiser are hardly observed in genuine expressions comparing to acted ones as specified in [7]. In other words, for the same emotional state, spontaneous AUs differ from posed ones. Second, AUs discriminating between expressions may change from one couple of expressions to another even within the same expression category (spontaneous or posed). Therefore, performing a static selection method and choosing relatively the best features subset to implement the prediction for all the expressions may not be efficient. In fact, choosing the average does not always mean choosing the best. Although the selected subset has shown good results in most expressions, it may perform poorly in others. Thus, for better training and emotion detection, choosing the right and effective features is crucial as irrelevant and noisy features may mislead and negatively affect the recognition system.

To sum up, two main challenges facing research in the field of facial expression recognition were raised. First, is it possible to fuse hybrid features (geometric, texture, etc.), within the framework of facial expression recognition, without information redundancy? Second, how to design a feature selection mechanism to solve the problem of expressions that are commonly miss-classified by FER systems? The aim of this work is to explore soft computing techniques, such as genetic programming, to enhance the way features are selected and fused for more accurate facial expression recognition. The main contributions of this work are threefold:

  • We propose a genetic programming (GP) based framework for posed and spontaneous facial expression recognition allowing to combine hybrid facial features, then we test it on the fusion of geometric and appearance features.

  • The feature selection and fusion, in this work, are performed in a binary way: the most prominent features are selected and fused differently for each pair of expressions.

  • To the best of our knowledge, we are the first to propose a genetic programming based classifiers for facial emotion recognition incorporating simultaneous feature selection and fusion within the evolutionary learning process.

The rest of this paper is organized as follows. We present a brief literature review on feature fusion and selection based methods for facial expression recognition in Section 2. The proposed method is detailed in Section 3. The results are presented and discussed in Section 4. Finally, in Section 5, we conclude the proposed work and present some ideas for future studies.

Section snippets

Related work

A basic FER framework have to perform emotion recognition by representing the facial emotional states through an appropriate set of features, while ensuring a trade-off between recognition accuracy and computation time. Geometric and texture features has been widely used, mostly separately, for FER purposes. There has been some researches on feature fusion in emotion recognition over recent years. In [9], the authors proposed a hybrid method combining geometric and appearance features. This

Proposed method

This section provides a detailed description of the proposed method for spontaneous and posed facial emotion recognition. The general flowchart of the suggested method is illustrated in Fig. 2. Indeed, a face detection step followed by a feature extraction step are performed. In the extraction step, geometric and texture features are extracted and fed to genetically evolved programs. Indeed, a binary genetic program learns to select the most discriminating features and to fuse them specifically

Experimental results

In this section, the proposed facial emotion recognition method is tested and compared to several relevant FER methods. The proposed method has been implemented using the Anaconda Python distribution and the DEAP1 evolutionary computation framework. The experiments have been executed on a PC with Windows 10 as an operating

Conclusion and future work

In this work, a robust method for automatic facial expression recognition, is proposed. Genetic programming-based binary programs, which incorporate feature selection and fusion in the learning process, are proposed to discriminate between pairs of expression classes. The overall expression recognition is performed using a unique tournament elimination between the learned binary classifiers. The suggested method selects and combines differently linear, eccentricity and LBP features for each

CRediT authorship contribution statement

Haythem Ghazouani: Conceptualization, Methodology, Software, Validation, Writing.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

References (47)

  • M. He, S. Wang, Z. Liu, X. Chen, Analyses of the differences between posed and spontaneous facial expressions, in: 2013...
  • SenD.S. et al.

    Facial emotion classification using concatenated geometric and textural features

    Multimedia Tools Appl.

    (2019)
  • EkmanF.W. et al.

    Felt, false, and miserable smiles

    J. Nonverbal Behav.

    (1982)
  • NambaS. et al.

    Spontaneous facial expressions are different from posed facial expressions: Morphological properties and dynamic sequences

    Curr. Psychol.

    (2016)
  • Paul EkmanS. et al.

    Facial Action Coding System: A Technique for the Measurement of Facial Movement

    (1978)
  • ChenJ. et al.

    Facial expression recognition using geometric and appearance features

  • H. Sadeghi, A. Raie, M. Mohammadi, Facial expression recognition using geometric normalization and appearance...
  • GharsalliS. et al.

    Various fusion schemes to recognize simulated and spontaneous emotions

  • C. Turan, K. Lam, X. He, Facial expression recognition with emotion-based feature fusion, in: 2015 Asia-Pacific Signal...
  • MahmoodA. et al.

    Recognition of facial expressions under varying conditions using dual-feature fusion

    Math. Probl. Eng.

    (2019)
  • J. de Andrade Fernandes Junior, L.N. Matos, M.G. dos Santos Aragão, Geometrical approaches for facial expression...
  • Mark A. HallA.

    Correlation-based feature selection for machine learning

    (1999)
  • M. Hall, Correlation-based feature selection for discrete and numeric class machine learning, in: Proceedings of the...
  • Cited by (0)

    View full text