UNIGE document Chapitre d'actes
previous document  unige:102216  next document
add to browser collection
Title

Multimodal Analysis of Image Search Intent: Intent Recognition in Image Search from User Behavior and Visual Content
Authors
Soleymani, Mohammad
Riegler, Michael
Halvorsen, Pål
Published in ICMR '17 Proceedings of the 2017 ACM on International Conference on Multimedia Retrieval. Bucharest (Romania) - June 06 - 09, 2017 - ACM Press. 2017
Abstract Users search for multimedia content with different underlying motivations or intentions. Study of user search intentions is an emerging topic in information retrieval since understanding why a user is searching for a content is crucial for satisfying the user's need. In this paper, we aimed at automatically recognizing a user's intent for image search in the early stage of a search session. We designed seven different search scenarios under the intent conditions of finding items, re-finding items and entertainment. We collected facial expressions, physiological responses, eye gaze and implicit user interactions from 51 participants who performed seven different search tasks on a custom-built image retrieval platform. We analyzed the users' spontaneous and explicit reactions under different intent conditions. Finally, we trained machine learning models to predict users' search intentions from the visual content of the visited images, the user interactions and the spontaneous responses. After fusing the visual and user interaction features, our system achieved the F-1 score of 0.722 for classifying three classes in a userindependent cross-validation. We found that eye gaze and implicit user interactions, including mouse movements and keystrokes are the most informative features. Given that the most promising results are obtained by modalities that can be captured unobtrusively and online, the results demonstrate the feasibility of deploying such methods for improving multimedia retrieval platforms.
Keywords Multimedia — User interaction — Intent — Emotion — Experiment — Eye gaze — Facial expression — Computer vision
Identifiers
ISBN: 978-1-4503-4701-3
Full text
Proceedings chapter (Published version) (4.2 MB) - public document Free access
Other version: http://dl.acm.org/citation.cfm?doid=3078971.3078995
Structures
Faculté des sciences / Département d'informatique
Centres et instituts / Centre universitaire d'informatique
Centres et instituts / Centre interfacultaire en sciences affectives
Research groups Affective sciences
Computer Vision and Multimedia Laboratory
Multimodal Interaction Group
Project
Swiss National Science Foundation: Ambizione
Citation
(ISO format) 		SOLEYMANI, Mohammad, RIEGLER, Michael, HALVORSEN, Pål. Multimodal Analysis of Image Search Intent: Intent Recognition in Image Search from User Behavior and Visual Content. In: ICMR '17 Proceedings of the 2017 ACM on International Conference on Multimedia Retrieval. Bucharest (Romania). [s.l.] : ACM Press, 2017. doi: 10.1145/3078971.3078995 https://archive-ouverte.unige.ch/unige:102216

545 hits

90 downloads

Contact an author

Update request

Update

Deposited on : 2018-02-15

Export document
Format :
Citation style :