The nature of the rocks exposed to weathering and erosion on continents exerts an important control on weathering feedbacks and the supply of nutrients to the oceans. It also reflects the prevailing tectonic regime responsible for the formation of continents. How the chemical and lithological compositions of the continents evolved through time is, however, still a matter of debate. We use an extensive compilation of terrigenous sediment compositions to better constrain the nature of rocks at the surface of continents at 3.25 Gyr and 250 Myr ago. Specifically, we use geochemical ratios that are sensitive indicators of komatiite, mafic, and felsic rocks in the provenance of the sediments. Our results show that the average Al2O3/TiO2 ratio of fine-grained terrigenous sediments decreased slightly over time from 26.2 ± 1.3 in the Archean to 22.1 ± 1.1 (2SE) in the Phanerozoic. In contrast, in the same time interval, the average Zr/TiO2 ratio stayed nearly constant at ~245. Considering the distinct behaviors of Al, Ti and Zr during sedimentary processes, we find that hydrodynamic mineral sorting had a minor effect on the chemical composition of Archean fine-grained sediments, but could have been more effective during periods of supercontinents. We show that the compositions of Phanerozoic sediments (Al2O3/TiO2, Zr/TiO2, La/Sc, Th/Sc, Ni/Co, Cr/ Sc) are best explained with igneous rocks at the surface of continents consisting of 76 ± 8 wt% felsic, 14 ± 6 wt% Arc-basalts and 10 ± 2 wt% within-plate basalts, most likely in the form of continental flood basalts. Applying the same mass-balance calculations to the Paleoarchean suggests continental landmasses with 65 ± 7 wt% felsic, 25 ± 6 wt% mafic and 11 ± 3 wt% ultramafic rocks (all 2SE), likely in the form of komatiites. The presence of volumetrically abundant felsic rocks at the surface of continents (as evident from the sediment record) as well as at mid-crustal levels (as evident from presently exposed igneous rock record) in Paleoarchean cratons is currently best explained with the onset of subduction magmatism before 3.25 Gyr.