Inherited retinal degeneration due to loss of photoreceptor cells is a leading cause of human blindness. These cells possess a photosensitive outer segment linked to the cell body through the connecting cilium (CC). While structural defects of the CC have been associated with retinal degeneration, its nanoscale molecular composition, assembly, and function are barely known. Here, using expansion microscopy and electron microscopy, we reveal the molecular architecture of the CC and demonstrate that microtubules are linked together by a CC inner scaffold containing POC5, CENTRIN, and FAM161A. Dissecting CC inner scaffold assembly during photoreceptor development in mouse revealed that it acts as a structural zipper, progressively bridging microtubule doublets and straightening the CC. Furthermore, we show that Fam161a disruption in mouse leads to specific CC inner scaffold loss and triggers microtubule doublet spreading, prior to outer segment collapse and photoreceptor degeneration, suggesting a molecular mechanism for a subtype of retinitis pigmentosa.