The literature points to a large distributed brain network involved in the estimation of time. Among these regions, the role of the insular cortex is still poorly understood. At the confluence of emotional, interoceptive, and environmental signals, this brain structure has been proposed to underlie awareness of the passage of time and emotion related time dilation. Yet, this assumption has not been tested so far. This study aimed at exploring how a lesion of the insula affects subjective duration, either in an emotional context or in a non-emotional context. Twenty-one patients with a stroke affecting the insula, either left or right, were studied for their perception of sub and supra second durations. A verbal estimation task and a temporal bisection task were used with either pure tones or neutral and emotional sounds lasting between 300 and 1500 ms and presented monaurally. Results revealed that patients with a right insular lesion, showed less temporal sensitivity than both control participants and patients with a left insular lesion. Unexpectedly, emotional effects were similar in patients and control participants. Altogether, these results suggest a specific role of the right insula in the discrimination of durations, but not in emotion related temporal distortion. In addition, an ear × emotion interaction in control participants suggests that temporal processing of positive and negative sounds may be lateralized in the brain.