Older adults often desire to remain in their homes for as long as possible, and Socially Assistive Robots (SARs) can play a role in supporting this goal. However, the acceptance and adoption rates of SARs remain relatively low, suggesting that current designs may not fully address all user needs. Field studies in Human-Robot Interaction, particularly those involving multiple end-users, remain limited. Nevertheless, such studies are crucial for identifying factors that shape the user experience with SARs, potentially improving their acceptance and adoption in healthcare settings. Therefore, this study aims to explore user perspectives, referred to as factors, that could guide design considerations for SAR development. We conducted a field study with 90 participants across Italy, Switzerland, and the Netherlands to identify these factors and their implications for improving the SAR user experience for older adults and their formal and informal caregivers. SARs were placed in the homes of older adults, and interviews were conducted with the three groups of primary end-users, at the beginning, midpoint, and end of the two-to six-week trial period. We initially focused on four factors (personalisation, interactivity, embodiment, and ethical considerations), identified in earlier design phases of the related 3-year Guardian project. Our findings confirmed the importance of these factors while uncovering additional ones. Personalisation and interactivity emerged as the most important ones among these factors. Based on our insights, we recommend involving all primary end-users in SAR research and design process and prioritising field studies to refine design elements. In conclusion, our study identified six factors for SAR design that can enhance the user experience: personalisation, interactivity, embodiment, ethical considerations, connectedness, and dignity. These findings provide valuable guidance for developing SARs that may better address the needs of older adults and their caregivers.