Purpose This study aimed (1) to examine the role of potential correlates of HRQoL in a large representative sample of older adults, and (2) to investigate whether the relationships between HRQoL and potential factors differ as a function of HRQoL component (physical vs. mental) and/or age cohort (young-old vs. old–old). Methods This cross-sectional study included 802 older adults aged 60–79 years old. HRQoL was assessed using the SF-36 questionnaire. Functional fitness was assessed using the Senior Fitness Test. Physical activity was measured via the Baecke questionnaire. Demographic information, mental and health features were obtained through questionnaires. Results A multiple regression analysis showed that BMI (β = −0.15, p = 0.001), body strength (β = 0.21, p < 0.001), aerobic endurance (β = 0.29, p < 0.001), physical activity (β = 0.11, p = 0.007), depressive symptoms (β = −0.19, p < 0.001), falls (β = −0.19, p < 0.001), and living alone (β = −0.16, p < 0.001) were all significantly related to HRQoL-SF-36 total score. The positive relation with aerobic endurance was significantly higher for the physical component of HRQoL, while the negative relation with living alone was significantly higher for the mental component. The positive relation of HRQoL with physical activity was significantly higher in old–old compared to young-old adults. Conclusion This data suggest that body composition, functional fitness, psycho-social factors, and falls are important correlates of HRQoL in old age. There are HRQoL-component and age-cohort differences regarding these correlates, underlying the need for specific strategies at the community level to promote HRQoL in older adults.