UNIGE document Chapitre d'actes
previous document  unige:98499  next document
add to browser collection
Title

Large-scale Affective Content Analysis: Combining Media Content Features and Facial Reactions

Authors
McDuff, Daniel
Published in 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017). Washington (DC, USA): IEEE. 2017, p. 339-345
Abstract We present a novel multimodal fusion model for affective content analysis, combining visual, audio and deep visual-sentiment descriptors from the media content with automated facial action measurements from naturalistic responses to the media. We collected a dataset of 48,867 facial responses to 384 media clips and extracted a rich feature set from the facial responses and media content. The stimulus videos were validated to be informative, inspiring, persuasive, sentimental or amusing. By combining the features, we were able to obtain a classification accuracy of 63% (weighted F1-score: 0.62) for a five-class task. This was a significant improvement over using the media content features alone. By analyzing the feature sets independently, we found that states of informed and persuaded were difficult to differentiate from facial responses alone due to the presence of similar sets of action units in each state (AU 2 occurring frequently in both cases). Facial actions were beneficial in differentiating between amused and informed states whereas media content features alone performed less well due to similarities in the visual and audio make up of the content. We highlight examples of content and reactions from each class. This is the first affective content analysis based on reactions of 10,000s of people.
Keywords Large-scale affective content analysisMedia content featuresFacial reactionsMultimodal fusion modelDeep visual-sentiment descriptorsAutomated facial action measurementsMedia clipsFacial responsesAU-2
Identifiers
ISBN: 978-1-5090-4023-0
Full text
Structures
Research groups Affective sciences
Computer Vision and Multimedia Laboratory
Multimodal Interaction Group
Project FNS: Ambizione
Citation
(ISO format)
MCDUFF, Daniel, SOLEYMANI, Mohammad. Large-scale Affective Content Analysis: Combining Media Content Features and Facial Reactions. In: 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017). Washington (DC, USA). [s.l.] : IEEE, 2017. p. 339-345. https://archive-ouverte.unige.ch/unige:98499

98 hits

86 downloads

Update

Deposited on : 2017-11-02

Export document
Format :
Citation style :