Pathogen strains may differ in virulence because they attain different loads in their hosts, or because they induce different disease-causing mechanisms independent of their load. In evolutionary ecology, the latter is referred to as "per-parasite pathogenicity". Using viral load and CD4+ T-cell measures from 2014 HIV-1 subtype B-infected individuals enrolled in the Swiss HIV Cohort Study, we investigated if virulence-measured as the rate of decline of CD4+ T cells-and per-parasite pathogenicity are heritable from donor to recipient. We estimated heritability by donor-recipient regressions applied to 196 previously identified transmission pairs, and by phylogenetic mixed models applied to a phylogenetic tree inferred from HIV pol sequences. Regressing the CD4+ T-cell declines and per-parasite pathogenicities of the transmission pairs did not yield heritability estimates significantly different from zero. With the phylogenetic mixed model, however, our best estimate for the heritability of the CD4+ T-cell decline is 17% (5-30%), and that of the per-parasite pathogenicity is 17% (4-29%). Further, we confirm that the set-point viral load is heritable, and estimate a heritability of 29% (12-46%). Interestingly, the pattern of evolution of all these traits differs significantly from neutrality, and is most consistent with stabilizing selection for the set-point viral load, and with directional selection for the CD4+ T-cell decline and the per-parasite pathogenicity. Our analysis shows that the viral genotype affects virulence mainly by modulating the per-parasite pathogenicity, while the indirect effect via the set-point viral load is minor.