Background: Referees’ activity can be an instrument for training students with an impact on health-related physical fitness (PF). This study aimed to investigate the differences in PF and body composition between students without sports practice (G1), students with regular sports practice (G2), and student referees in team invasion games (G3). Methods: This study followed a cross-sectional design. The sample comprised 45 male students between 14 and 20 years old (16.40 ± 1.85). Three groups (G1, G2, and G3) of 15 participants each were selected. PF was assessed by a 20 m shuttle run, change-of-direction test, and standing long jump. Body composition was determined by body mass index (BMI; kg/m2) and the percentage of body fat (%BF) predicted by skinfold. Results: After controlling for age as a covariate, the block of variables used to characterize PF was statistically different between sports practice groups favoring student referees (p < 0.001; r = 0.26). Similar results were achieved for body composition (i.e., BMI and %BF) (p < 0.001; r = 0.17). However, when the dependent variables were evaluated separately, there were only differences between groups in %BF (p = 0.007; r = 0.21). Student referees had statistically significantly lower values than the remaining groups. Conclusion: Refereeing activity benefits PF related to health and performance, including body composition. This study confirms the benefits related to health in children and adolescents who are involved in refereeing activity.