Machine recognition of emotions is an essential prerequisite for natural and intelligent interactions between humans and computers. Existing research has demonstrated the potential of electroencephalogram (EEG)-based emotion recognition. In addition to information from neural sources, EEG signals also contain a mixture of signals originating from sensorimotor, cognitive and affective activities in addition to artifacts. Facial expressions, eye and head movements generate electrical activities that have higher amplitude than the EEG signals and are the main source of artifacts. Even though artifacts are considered a nuisance to brain-computer interfaces, in affective brain-computer interfaces (aBCI), behavioral artifacts can be valuable for emotion recognition.

Our research aims to identify the inter-modality interaction between EEG signals and facial behaviors to improve EEG-based emotion recognition performance. The interaction between these two modalities can be explained by signal interference and joint emotional variations. For the first time, we designed a specific protocol and collected data to isolate the effects of expressions, subjective feelings, and stimuli in these signals. We recorded a precisely synchronized multimodal database with visual, olfactory and mimicry stimuli, including genuine and acted expressions of emotions. This dataset enabled us to design and evaluate the emotion recognition methods in different scenarios and in the presence of between participant variances. We used our novel dataset in addition to the existing datasets (DEAP and MAHNOB) for within- and cross-corpus evaluation.