GWAS have generally focused on common variants from genotyping arrays or rare protein-coding variants from exome sequencing. Here, we use whole-genome sequencing data to evaluate the contribution to and architecture of rare non-coding variants for three commonly studied anthropometric traits: height, BMI and waist-hip ratio adjusted for BMI. Analysing 447,461 individuals in the UK Biobank for discovery and 225,515 individuals in All of Us for replication, we identify 90 rare and low-frequency single variant associations, including two independent rare variants upstream of IGF2BP2 that substantially reduce waist-hip ratio adjusted for BMI, but have distinct effects on other adiposity traits. We further identify 135 coding variant aggregates. For example, UBR3 protein-truncating variants are associated with a 2.7 kg/m2 increase in BMI. We additionally identify 51 non-coding variant aggregate associations, including one in the 5'UTR of FGF18 associated with up to 6 cm effects on height. We show that 97% of rare variant associations occur near GWAS-identified loci, demonstrating convergence of rare and common variant associations. Finally, we show that ultra rare variants explain a small fraction of heritability compared to common variants for these traits, that heritability is largely shared across ancestries, and that it concentrates around common variant loci.