The focus of multimodal emotion recognition has often been on the analysis of several fusion strategies. However, little attention has been paid to the efect of emotional cues, such as physiological and audio cues, on external annotations used to generate the Ground Truths (GTs). In our study, we analyze this efect by collecting six continuous arousal annotations for three groups of emotional cues: speech only, heartbeat sound only and their combination. Our results indicate signifcant diferences between the three groups of annotations, thus giving three distinct cue-specifc GTs. The relevance of these GTs is estimated by training multimodal machine learning models to regress speech, heart rate and their multimodal fusion on arousal. Our analysis shows that a cue(s)-specifc GT is better predicted by the corresponding modality(s). In addition, the fusion of several emotional cues for the defnition of GTs allows to reach a similar performance for both unimodal models and multimodal fusion. In conclusion, our results indicates that heart rate is an efcient cue for the generation of a physiological GT; and that combining several emotional cues for GTs generation is as important as performing input multimodal fusion for emotion prediction.