Studying the genetic basis of gene expression and chromatin organization is key to characterizing the effect of genetic variability on the function and structure of the human genome. Here we unravel how genetic variation perturbs gene regulation using a dataset combining activity of regulatory elements, gene expression, and genetic variants across 317 individuals and two cell types. We show that variability in regulatory activity is structured at the intra- and interchromosomal levels within 12,583 cis-regulatory domains and 30 trans-regulatory hubs that highly reflect the local (that is, topologically associating domains) and global (that is, open and closed chromatin compartments) nuclear chromatin organization. These structures delimit cell type-specific regulatory networks that control gene expression and coexpression and mediate the genetic effects of cis- and trans-acting regulatory variants on genes.