Virtual and robotic agents are becoming increasingly prominent, taking on a variety of everyday life roles (i.e., assistants, tutors, coaches, companions). Displaying social and affective behaviour is a necessary requirement when agents need to interact and collaborate with humans. Nevertheless, current agent prototypes lack important skills, such as recognising human emotions, adapting to them and expressing appropriate affective states. This dissertation addresses issues related to these challenges. First, research is surveyed which investigates the use of psychophysiology for affect recognition. Next, empirical work is presented which investigates the multimodal expression of emotions through robotic embodiment. Finally, a use-case is presented, where a virtual agent takes the role of a companion for older adults. An evaluation study is discussed, highlighting the effects of the agent's socio-affective capabilities on interactions and long-term user-engagement. The dissertation concludes with a set of guidelines for the design of natural, believable, effective and acceptable human-agent interactions.