The prevalence of low semen quality and the incidence of testicular cancer have been steadily increasing over the past decades in different parts of the World. Although these conditions may have a genetic or epigenetic origin, there is growing evidence that multiple environmental and lifestyle factors can act alone or in combination to induce adverse effects. Exposure to these factors may occur as early as during fetal life, via the mother, and directly throughout adulthood after full spermatogenic capacity is reached. This review aims at providing an overview of past and current trends in semen quality and its relevance to fertility as well as a barometer of men's general health. The focus will be on recent epidemiological studies of young men from the general population highlighting geographic variations in Europe. The impact of some lifestyle and environmental factors will be discussed with their role in both fetal life and adulthood. These factors include smoking, alcohol consumption, psychological stress, exposure to electromagnetic radiation, and Endocrine Disrupting Chemicals (EDCs). Finally, the challenges in investigating the influence of environmental factors on semen quality in a fast changing world are presented.