Emotions play a pivotal role in viewers' content selection and use. The main aim of this study is to detect and estimate affective characteristics of videos based on the content and viewers' response. These emotional characterizations can be used to tag the content. Implicit or automated tagging of videos using emotions help recommendation and retrieval systems to improve their performance. The analysis and evaluations directions in this thesis are twofold: first, methodology and results of emotion recognition methods employed to detect emotion in response to videos are presented. Second, methodology and results of emotional understanding of multimedia using content analysis are provided. In conclusion, promising results have been obtained in emotional tagging of videos. However, emotional understanding of multimedia is a challenging task and with the current state of the art methods a universal solution to detect and tag all different content which suits all the users is not possible.