Accurate chromosome segregation requires the assembly of kinetochores, multiprotein complexes that assemble on the centromere of each sister chromatid. A key step in this process involves binding of the constitutive centromere-associated network (CCAN) to CENP-A, the histone H3 variant that constitutes centromeric nucleosomes. This network is proposed to operate as a persistent structural scaffold for assembly of the outer kinetochore during mitosis. Here, we show by fluorescence resonance energy transfer (FRET) that the N-terminus of CENP-N lies in close proximity to the N-terminus of CENP-A in vivo, consistent with in vitro data showing direct binding of CENP-N to CENP-A. Furthermore, we demonstrate in living cells that CENP-N is bound to kinetochores during S phase and G2, but is largely absent from kinetochores during mitosis and G1. By measuring the dynamics of kinetochore binding, we reveal that CENP-N undergoes rapid exchange in G1 until the middle of S phase when it becomes stably associated with kinetochores. The majority of CENP-N is loaded during S phase and dissociates again during G2. We propose a model in which CENP-N functions as a fidelity factor during centromeric replication and reveal that the CCAN network is considerably more dynamic than previously appreciated.