Impressions play an important role in both human-human and human-virtual agent interaction. However, detecting impressions from physiological and nonverbal cues have not been explored yet. This work intends to recognize impressions within the warmth and competence dimensions, through multimodal physiological signals and behaviors. Eye gaze, facial expressions, gestures, heart rate and other physiological parameters will be used for impression recognition. In addition, the temporal unfolding of expressions between the two agents will be considered. In this way, the nonverbal cues that contribute towards the formation of impressions will be learned. Behaviors which favor a specific impression will then be adapted for a virtual agent to effectively interact with users. The resulting work will consist of an affective loop between a virtual agent and human envisioning the usage at museums.