Background: People more socioeconomically vulnerable tend to have a poorer health-related quality of life (HRQoL). Studies are trying to analyse the factors that may condition this relationship, including physical activity (PA), which may influence the relationship between socioeconomic status (SES) and HRQoL. This study aimed to analyse the relationship between SES and HRQoL through specific domains of PA. Methods: A total of 381 adults (≥18 years) from the Autonomous Region of Madeira completed the measurements. Mediation analyses using bootstrapping methods adjusted for confounding variables were performed to relate SES and HRQoL; the latter was evaluated using the physical component score (PCS) of the SF-12, the mental component score (MCS) of the SF-12, and the total score in the SF-12 questionnaire (SF-12 score), through physical activity. Results: Educational level was positively related to PCS and SF-12 score. Leisure-time PA (not including sports PA) and PA at work, as single mediating variables, did not mediate the relationship between SES and HRQoL. The total PA suppressed the socioeconomic gradient of HRQoL by 8–10%, 39–46%, and 15–16%, respectively, for the PCS, MCS, and the SF-12 score; sports PA mediated the relationship by 13–16%, 50%, and 15–21%, respectively. Conclusions: The results suggest that sports PA contributes to reducing the socioeconomic gradient of HRQoL.