Complex architectural rearrangements are associated to the control of the HoxD genes in different cell types; yet, how they are implemented in single cells remains unknown. By use of polymer models, we dissect the locus 3D structure at the single DNA molecule level in mouse embryonic stem and cortical neuronal cells, as the HoxD cluster changes from a poised to a silent state. Our model describes published Hi-C, 3-way 4C, and FISH data with high accuracy and is validated against independent 4C data on the Nsi-SB 0.5-Mb duplication and on triple contacts. It reveals the mode of action of compartmentalization on the regulation of the HoxD genes that have gene- and cell-type-specific multi-way interactions with their regulatory elements and high cell-to-cell variability. It shows that TADs and higher-order 3D structures, such as metaTADs, associate with distinct combinations of epigenetic factors, including but not limited to CCCTC-binding factor (CTCF) and histone marks.